An Update on the Quality Metrics Initiative

In October 2005, Janet Woodcock stated FDA wanted “a maximally efficient, agile, flexible, pharmaceutical manufacturing sector that reliably produces high quality drug products without extensive regulatory oversight” (1). The groundwork to achieve this realization started with the adoption of Title VII, Sections 705 and 706 of the Food and Drug Administration Safety and Innovation Act (FDASIA) in July 2012. It has been almost 10 years since Dr. Woodcock revealed her vision and almost three years since the industry began to hear rumblings that FDA was going to establish quality metrics that would be used to determine the suitability of a manufacturer’s ability to provide quality products to the patient. So how far have FDA and the industry come in establishing a set of quality metrics applicable to the pharmaceutical industry?

The aforementioned sections of FDASIA gave FDA the ability to establish criteria to perform risk-based inspections of bio/pharmaceutical manufacturers and to request certain documents be provided in advance or, in some cases, in lieu of inspections. On Feb. 12, 2013, FDA published a notice in the Federal Register asking for assistance from industry in developing a strategic plan and quality metrics to prevent drug shortages (2). FDA’s objective seems to be to reduce companies’ regulatory burden allowing for innovation while maintaining responsibility of protecting the public health.

Industry responds

In response to the Federal Register notice and during the subsequent three-year review period, several organizations have made recommendations to the agency on what they perceived as fair, reasonable, and equitable quality metrics. Organizations that independently participated in the development of quality metrics include the Parenteral Drug Association (PDA), the International Society for Pharmaceutical Engineering (ISPE), the Generic Pharmaceutical Association (GPhA), Pharmaceutical Researchers and Manufacturers of America (PhRMA), and Consumer Healthcare Products Association (CHPA). These organizations held conferences, wrote white papers, and polled their members to try and define what specific quality metrics would be appropriate for the intended purpose. This effort culminated in a meeting facilitated by the Brookings Institute in May 2014. Representatives from the above organizations, covering all aspects of the pharmaceutical industry, met to try and come to a consensus on exactly what quality metrics could be and how they could be used as an indicator of a company’s ability to continuously provide high quality medicine to patients. The outcome of the Brookings meeting resulted in a set of proposed consensus metrics that included lot acceptance rate, product complaint rate, confirmed out-of-specification rate, and recall rate.

The proceedings from the meeting clarified that the “consensus set of metrics are somewhat rudimentary, and provide limited information about the culture of quality at a given organization” (3). According to the published proceedings, “many [participants] remarked that a strong quality culture is a critical component in driving the system and processes that underpin the quality control and assurance infrastructure at an organization. However, quality culture is also difficult to capture through metrics.”

It was at this point where industry took off in different directions; PDA and ISPE have been the most engaged organizations in subsequent efforts. ISPE designed and asked members to participate in a pilot program for quality metrics, while PDA held a conference on quality metrics that focused on how to define a mature quality culture.

The metrics used in the ISPE Quality Metrics Pilot Program include the consensus metrics from the Brookings meetings as well as others. Some of the unique metrics used by ISPE in their pilot include timeliness of annual product quality reviews, recurring deviations rate, corrective action and preventive action (CAPA) effectiveness rate, process capability, and quality culture (4). The results of the pilot program are expected to be available in April 2015. The preliminary findings from the program seem to indicate the collection, formatting, and submission of the metrics to  FDA may be of a significant resource burden to collect the metrics.

Traditional, enhanced, and other quality systems

A conference held in December 2014 by PDA (2) and FDA focused on quality culture metrics (5). The hypothesis for the meeting was that mature quality attributes have a strong relationship to positive quality culture behaviors. The consensus from the meeting was that measuring a quality culture is subjected and defined by a set of behaviors, beliefs, values, attitudes, and governance and that mature quality attributes go beyond traditional quality systems in creating a framework for a strong quality culture. PDA divided quality systems into three types: Traditional, Enhanced, and Other. PDA defined traditional quality systems by traditional metrics including deviations, complaints, CAPA, etc. Enhanced quality systems were defined as those that had advanced programs such as risk management, knowledge management, quality by design, quality manual, etc. The other quality systems were much more evolved and have programs like shared quality goals, rewards and recognition programs, and cost of quality awareness programs, etc. Conference attendees defined the top five mature quality attributes as the following:
• Program to show how employee’s specific goals contribute to overall quality goals
• Program to measure, share, and discuss product quality performance and improvement from shop floor to executive management
• Continuous improvement program/plans with active support of CEO and corporate management of quality management systems (QMS)
• Program that establishes quality system maturity model and action plan and tracking to measure progress
• Internal survey measuring a company/site quality culture.
Meanwhile, FDA presented the following metrics for measuring a quality system:
• Lot acceptance rate (the number of lots rejected by the establishment in a year divided by the number of lots attempted by the same establishment in the same year)
• Right-first-time rate (the number of lots with at least one deviation by the establishment in a year divided by the number of lots attempted by the same establishment in the same year)
• Product quality complaint rate (the number of complaints received by the manufacturer of the product concerning any actual or potential failure of a unit of drug product to meet any of its specifications, divided by the total number of lots released by the manufacturer of the product in the same year)
• Invalidated out-of-specification (OOS) rate (the number of OOS test results invalidated by the establishment, or contracted establishment in a year divided by the total number of tests performed by the establishment in the same year).

The developing future of quality metrics
So what is the future of quality metrics? Perhaps it is still too early to tell. Xavier University (6) held meetings on the topic of quality in March 2014 and again in March 2015. ISPE has yet to reveal the results of their pilot program, and PDA has yet to publish the results of their surveys from the December 2014 conference. The most important part of the puzzle, FDA’s guidance on quality metrics, which is due to be published in 2015, is still missing. But whatever happens, it is important to keep in mind that there are no perfect quality metrics. The concept of unintended consequences needs to be addressed so everyone is on a level playing field. Data trending can be more valuable in determining the robustness of a quality system than direct comparisons, and an open, honest quality culture will drive the integrity of the quality metrics.

References
1. J. Wechsler, Pharm. Technol. 29 (12), pp. 36-42 (2005).
2. FDA, Food and Drug Administration Drug Shortages Task Force and Strategic Plan; Request for Comments, Federal Register, Vol. 78, No. 29 pp. 9928-9929 (Feb. 12, 2013), www.gpo.gov/fdsys/pkg/FR-2013-02-12/html/2013-03198.htm.
3. Brookings Institute, Measuring Pharmaceutical Quality through Manufacturing Metrics and Risk-Based Assessment, Meeting Summary (Engelberg Center for Health Care Reform at Brookings, May 1-2, 2014).
4. A complete listing of the metrics used in the ISPE Pilot can be found at www.ispe.org/quality%20metrics%20defined.
5. PDA, Quality Metrics, www.pda.org.
6. Xavier University, FDA/Industry Collaborative Approach to Quality:  With the Patient in Mind (Xavier University, Cincinnati, OH, April 12, 2014.

Article Details
BioPharm International
Vol. 28, No. 5
Pages: 41-41
Citation: When referring to this article, please cite it as S. Schniepp, “An update on the Quality Metrics Initiative,” BioPharm International 28 (5) 2015.