Quality Risk Management Demystified at CMC Strategy Forum

The prospect of conducting risk assessments, as part of the quality risk management (QRM) strategy promoted by the International Conference on Harmonization (ICH) Q9 guideline, tends to evoke blank stares and confusion. It also prompts many questions: Which of the many tools—PHA, HACCP, FMEA, etc.—should we use? How could we possibly conduct a line-by-line analysis of every risk known to bioprocessing? Is this really useful?

The CMC Strategy Forum on Practical Applications of Quality Risk Management, held in Bethesda, MD, on July 27 and 28, tackled these challenges and questions through a hands-on workshop and plenty of lively discussion.

Laura Bush

In the workshop, small groups used the preliminary hazard analysis (PHA) method to assess the risks involved in operating a large-scale production bioreactor for a monoclonal antibody. As the groups considered inputs such as duration, media, and temperature, they identified the possible hazards involved and the harms those hazards could cause, and then assigned a severity score to each harm. They also assigned a separate risk score to each potential harm, based on the likelihood that it would occur. Then they multiplied those numbers to reach an overall risk score, addressed the controls used to prevent or mitigate the event, and decided whether or not the risk would be acceptable (Table 1).

Table 1. A partial risk assessment worksheet created during a fictitious exercise for the operation of a production bioreactor, using the preliminary hazard analysis (PHA) method. Each potential harm was assigned a severity score (on a scale of 1–9), and each potential hazardous situation was given an occurrence score, based on the likelihood that it would occur, taking into account the controls used to prevent or mitigate the event. The two scores were multiplied to reach an overall risk index.

After the breakout sessions, everyone understood the concepts, terminology, and process much better, and had a good appreciation for the challenges involved in conducting risk assessments. They also had a lot of questions, which were discussed with the overall group.

HOW TO ACCOUNT FOR CONTROLS

One common question raised during the risk assessment exercise was how to account for the controls that mitigate risks.

Several speakers stressed that the effectiveness of controls should never affect how one ranks the severity of a potential harm. "Severity rankings are dependent on harm only," said Joe Siemiatkoski of Biogen Idec. "You should never lower a severity score by taking credit for controls."

For example, if a given harm, such as bioreactor contamination, is considered high, say 9 on a scale of 1 to 10, that severity ranking should not be lowered just because numerous measures are in place to prevent contamination. Instead, those controls would lower the ranking number assigned to the likelihood that such a harm would occur. Then, when the total risk is assessed by multiplying the severity and likelihood scores, its overall risk ranking would not be high, and thus additional controls may not be warranted.

Dan Weese of Amgen explained that this approach is important for documenting the potential risks to any given unit operation. "You may have a risk where you are confident that a later step will take care of it, but what if a later step is removed?" he said. "You need the risk to be flagged as severe in the earlier unit operation, so you don't lose your record of it."

THE MEANING OF SEVERITY RANKINGS: HOW TO ASSESS POTENTIAL PATIENT HARM

Participants also struggled to understand the meaning of severity rankings. The ICH Q9 guideline instructs the industry to consider harm in terms of patients. For some examples, however, like the cell culture step discussed in the workshop, it is difficult to envision any problem—from contamination to aggregation—that wouldn't be resolved before the product reaches the patient. Some cited improper glycosylation, but others pointed out that for many products, particularly monoclonal antibodies, even carbohydrate structures would not necessarily affect product safety or efficacy.

The solution, many speakers said, is to use in-process product quality as a surrogate for patient harm when assigning severity rankings.

"The first time we piloted a risk assessment during our implementation of ICH Q9, we attempted to use a severity scoring criteria that included harm to patients, and the highest severity score included serious injury or death," said Weese. "That didn't work for upstream assessments begun in the middle of Phase 2, because in many cases, you just don't know how a potential failure in a unit operation might cause direct harm to a patient. But we can estimate the potential impact on product quality attributes." As a result, he said, they decided it made more sense to use the potential impact on a product quality attribute as a surrogate for potential harm to patients.

The connection to patient risk would be clearer, some said, if clinical staff participated in risk assessments. "We include clinical staff when evaluating the need for product recalls, so I would think companies could benefit from including clinical staff when assessing risks," said Kevin O'Donnell of the Irish Medicines Board. "Clinical staff do see things that we don't in terms of risk to patients."

A few people responded that clinical staff participate in an earlier stage in the process. "We get input from clinicians in an earlier step, when we are determining critical quality attributes," said Kathy Francissen of Genentech.

Others questioned whether any clinician could really be expected to predict the patient impact of a manufacturing change. "Unless you have a very strong event, you are not going to get a clear linkage, because of extreme number of variables involved," said Wassim Nashabeh of Genentech.

Steven Kozlowski of the US Food and Drug Administration's Center for Drug Evaluation and Research (CDER) asked if anyone has attempted to tackle this problem through data mining, using multivariate data analysis to compare clinical outcomes or adverse event reporting to product attributes, perhaps by linking those data to lot numbers. "If we did this, would we find clusters of data that would be useful?" he asked.

Stefanie Pluschkell of Pfizer said her company is trying to do just that. "It's complex, because the databases [of adverse events and product lots] are not always directly linked," she said, adding that it also can be difficult to determine which specific adverse events to look at, because they can be diverse. "But I am reasonably hopeful that this effort will be useful," she said.

SCIENTISTS AND SUBJECTIVITY

A number of participants also questioned the real value of risk-ranking scores, given their subjectivity. "Now may be the time to improve the science of our decision-making, to increase our confidence in the results of these risk assessment exercises," said O'Donnell of the Irish Medicines Board. "There are other disciplines, such as experimental psychology, that we should look at. Much research work has been carried out and that research can be helpful to us when we are carrying out quality risk management exercises."

"Subjectivity will not disappear, and we should not strive for that," cautioned Rohin Mhatre of Biogen Idec. "The whole process is very subjective, from deciding how many experiments to run, to which parameters to study. We should try to build a good rationale for our risk assessments, but not aim for objectivity."

Nancy Waites of the FDA's Center for Biologics (CBER) agreed. "We're all scientists and we don't like ambiguity," she said. "But you can't think of everything. You need to be comfortable with not knowing everything."

Terry Ocheltree of CDER also concurred. "I don't think anyone at FDA expects you to remove all subjectivity," he said. "What is important is to have the tools and use them correctly. But that's not to say that everyone must run risk assessments in the same way."

THE RIGHT TEAM AND THE RIGHT FACILITATOR

Various speakers stressed the importance of assembling the right team to conduct a risk assessment. "The scoring is as good as who is in the room," said Amgen's Weese, emphasizing the importance of including staff from all key departments. "Our rules say that if certain people are not present, we will not proceed," he added.

But don't underestimate the importance of a good facilitator, several people said. "It's not just a question of involving the right subject matter experts, but also about the competence of the facilitator," said Keith Webber of CDER.

In particular, managing the people and personalities involved can be tricky. "Often we will have one very outspoken person who dominates, especially if that person is an expert in the topic under discussion," said Weese. "We need to ensure we hear from everyone else."

Vince Hamner of Talecris agreed, adding that those who disagree don't always voice their views. "Sometimes you get someone who sits back and doesn't say anything, then suddenly speaks up at the end," he said.

A good facilitator, Weese added, can help the group reach decisions, particularly when doubt leads a group to score everything conservatively. "If every risk number is extremely high, we are not achieving anything," he said.

But many also pointed out that there are few people in the biotech industry trained in risk management. Some suggested turning to the medical devices industry, because medical device manufacturers have been conducting risk assessments for decades.

ARE RISK ASSESSMENT EXERCISES WORTH THE EFFORT?

Conducting risk assessments is a lot of work, however, as the group exercise made clear. Some raised the question of whether it is worth the effort, because it seems that in most cases, the risks identified are the usual suspects.

"The majority are the usual suspects," acknowledged Richard O'Keeffe of Amgen. "But you are documenting what previously might only have been in people's heads. One of the benefits comes from systematic documentation."

"But it seems we are making it out to be more than it should be," countered Mhatre of Biogen Idec. "I hope we are not taking it so far that it just becomes an exercise and we lose all the practicality of it, like what happened years ago with process validation."

Sally Seaver of Seaver Associates agreed that getting real value out of risk assessments requires doing them properly, and not letting them become a rote exercise. "I think if risk assessments are done well there will be surprises—we will identify some unexpected risks," she said.

Furthermore, the work involved in formalizing risk management processes should get easier with time, said Genentech's Nashabeh. "In the early stages, you layer these things onto what you are already doing," he said. "If we get to a point where we integrate this into existing systems, we may save resources, but we are not there yet."

Julia Edwards of Genentech agreed. "Conducting risk assessments needs to be integrated into your quality systems, not an additional activity," she said.

Weese added that additional benefits are gained when risk assessment work is standardized throughout a company. "What we have done with Q9 is to establish a common framework across molecules, and across departments," he said. "That's the big benefit, in addition to reducing overall risk."

The cross-functional dialogue that occurs naturally during risk assessments is another big gain, Weese said. He cited an example where staff from different groups found out important information about a process during a risk assessment exercise. "'You are really doing that?' one group asked another. 'You can't do that!' That was really important," said Weese. "That exercise clearly paid for itself."

Genentech's Edwards gave a presentation in which she showed how her company is applying risk assessments to multi-use facilities, and then leveraging that work at other sites. The biggest gains, she said, can be achieved by using that information to file comparability protocols with the FDA. "By leveraging integrated quality risk management in regulatory submissions, we can effectively say, 'if we make this change at one site without a problem, we should also be able to make it at a different site,' since QRM allows us to account for site-specific considerations," she said.

Risk assessment tools also can be used to get upper management to recognize and formally accept risks, several participants said. Nadine Ritter of Biologics Consulting Group cited the example of a project manager who used a form listing the risks of changing an analytical method. The form required a signature from senior management, stating that they accepted responsibility for the decision. "Then they would re-think it carefully," she said. "That was brilliant."

Yet making the business case for quality risk management work is not easy, said Krista Terry of Genentech, because many of the gains relate to preventing problems, and it is difficult to assign a dollar value to such benefits. "On the operations side, for example, we have seen a decrease in the number of discrepancies, perhaps because staff are more aware of the importance of following procedures," she said. "It's more about working toward cost avoidance."

REGULATORY REQUIREMENTS

Applying quality risk management principles is not a regulatory requirement in the US, but it is in Europe, and may become one in other jurisdictions too.

"Under Chapter 1 of the EU GMPs, it is a requirement for companies to have a quality risk management program as part of the quality assurance system. This has been in place since July of last year," said O'Donnell of the Irish Medicines Board. O'Donnell went on to explain that although additional guidance on quality risk management is available in Annex 20 to the EU GMP guide, that guidance is voluntary at this time, meaning that companies are not required to comply with it.

Anthony Ridgway of Health Canada also foresees that this requirement eventually could be adopted in Canada and internationally. "ICH Q9 clearly states that it is not intended to influence regulatory requirements, however, if doing risk assessments becomes a routine part of good manufacturing practices, it could potentially get included in a future version of GMP regulations," he said.

And all the regulators at the meeting said they saw value in quality risk management.

"Conducting risk assessments is not a regulatory requirement [in the US], but companies that have risk management programs in place tend to have a good handle on their processes," said Waites of CBER. "They communicate well among themselves and often catch problems earlier."

Waites also noted that inspectors would look at the outcome of risk management efforts, not the process. "We won't fault you for ranking a risk as a 3 rather than a 7, but the outcome of ranking something too low might be inadequate validation," she said. "And we do give out 483s for a lot of things that wouldn't have occurred if you had done a risk assessment," CDER's Kozlowski said that regulatory agencies should also use risk-based approaches. "One of the most common complaints we get from industry is that we pay too much attention to lower risks," he said. "So if we apply the concept of risk management, it will benefit everyone."

Some companies have also made effective use of quality risk management in regulatory filings. Examples mentioned included filings for process and site changes, and even expanded comparability protocols. "I've seen it used in cases where a legacy process needed to be updated, such as for introducing new equipment or operating principles," said Patricia Hughes of CDER. "It's useful for improving processes."

Hughes also mentioned the example of changing a facility from single- to multi-product production. "This is especially useful in cases where you have very difficult cleaning validation issues for potent products, or different cell lines used in a multi-product facility," she said. "Risk analysis helps you understand the risks involved, and determine how to handle them."

SUMMARY

The discussions, presentations, and group workshop at the recent CMC Strategy Forum made it much easier to understand how to tackle the challenges involved in conducting quality risk assessments as part of a quality risk management program.

The discussions clarified how to think about potential harms, how to account for controls that prevent or mitigate risks, and how to handle the subjectivity of scoring. Speakers also stressed the importance of skilled facilitation, and of assembling a crossfunctional team for risk assessment exercises to ensure that all critical groups involved in a given unit operation have input.

No one doubted that conducting quality risk assessments is time-consuming and a lot of work. But the benefits, most said, are worth it. Apart from the obvious gain of reducing risk, and avoiding future compliance failures and their related costs, some said that the results could be used to justify manufacturing changes and even to support expanded comparability protocols. In addition, many participants said the process itself was useful, by fostering cross-functional dialogue and recording institutional knowledge. And although quality risk management is not yet a regulatory requirement in the US, it is already codified into European regulations, and that trend may expand.