Information Management Systems Enable Manufacturing Facilities to Ramp Up Productivity

The pharmaceutical industry has a record of making good use of technology in product development, most notably for replacing "wet" research with computer databases that enable researchers to test many combinations of chemical or biological compounds in simulated environments. This computerized research has improved product development's pace and efficiency exponentially as companies face increasing pressure to put more products in the pipeline.

Further down the line, however, in its manufacturing facilities – in the pharma "kitchens" – the industry is still mired in the mid-20th century.¹ Computer technology's spread has stalled at machine-level data collection for tracking basic processing information. The pharma industry has lagged in using IT on a broader scale to automate and streamline time-consuming manufacturing processes, specifically its batch record systems and for maintaining process quality control. Limited automation, electronic record keeping, and information collection contribute to pharma's poor manufacturing efficiency record — and to a costly and increasingly crippling regulatory reporting infrastructure. Pharma industry equipment utilization hovers below 40 percent, which would be an unacceptable figure in most industries.² Industry statistics for batch quality failures range from 5 to 15 percent.³ As a comparison, the semiconductor market — which dealt with similar process problems 25 years ago — today, has waste significantly below one percent.⁴

Pharma manufacturing's primary efficiency issues are:

• Poor process understanding and control, which causes lower yields and production delays due to subsequent quality control procedures⁵

• Inefficient, voluminous, paper-based production workflow, which produces very slow set up, production, verification, etc.⁶

• Chronic uncertainty of demand in the consumer market.⁷

These factors are a recipe for supply-chain inefficiency that has yielded average inventory turns of three to five versus 50 for world-class manufacturers in other industries.⁸

Dave Femia

Electronic batch record (EBR) and process analytical technology (PAT) information systems are key elements of a new paradigm in pharmaceutical manufacturing to eliminate these inefficiencies and transform manufacturing processes from a liability into an asset for meeting regulatory and market demands. EBR and PAT systems can create a common data framework that turns plant floor data into a strategic-level tool. The information they generate can help production managers increase yield while helping analysts and executives root out inefficiency, plan capacity usage, and meet regulatory reporting requirements without digging through stacks of paper that manual batch record systems generate.

However, this new paradigm will be a challenge to pharma executives because it is not a simple matter of extending conventional corporate IT systems and management approaches down to the manufacturing plant level. Reporting requirements and public safety regulations make IT systems reliability a much higher priority at the manufacturing level than in the general corporate IT environment. Outages and glitches common in corporate infrastructures are a much more serious issue in pharmaceutical manufacturing. If a pharma company can't document every step it takes to set up, process, and verify a batch, US and European regulations compel them to scrap it. EBR systems, for example, must collect data constantly because even one break in the electronic record chain can turn a million-dollar batch of medication into that quarter's biggest loss. For PAT to provide real-time quality control and key process parameter recording, its underlying computing infrastructure must be free of crashes and service interruptions.

Figure 1. An Electronic Batch Record System An electronic batch record (EBR) system draws data from many systems in pharma manufacturing plants. The network infrastructures between and within the systems are breeding grounds for outages and system interruptions and must be factored into plans for implementing an EBR system.

That need — for much higher reliability than conventional IT infrastructures provide — demands a new mindset from pharma executives, many of whom have solid backgrounds in corporate and scientific IT, but not in manufacturing IT. The mainstream business approach of consolidating IT systems at corporate data centers does not apply to manufacturing. Manufacturing data collection systems must be situated in relatively close proximity to the process, not at a remote data center. Plant-level IT staffs, however, don't have the resources or expertise to maintain the ultra-high reliability – 99.999 percent uptime – infrastructure that this new pharma manufacturing IT paradigm demands.

The challenge for pharma executives is to implement at the plant-level technologies and management practices that eliminate crashes, service interruptions, and performance degradations, while not increasing the plant IT budget, overburdening the existing staff, and risking regulatory non-compliance. As they move to meet the reliability challenge, pharma executives must understand the language of providing 99.999 percent uptime. There are many IT product and service vendors that tout phrases like "high-availability," "high reliability," and "continuous availability" for their products and services, but few of them provide 99.999 percent uptime. This article defines continuous availability as a system (hardware or software component) or integrated group of systems that is designed, manufactured, and delivered with an availability of 99.999 percent uptime or more, and is maintained at this level through an ongoing quality management process.

PHARMA'S RELIABILITY NEED

While most industries consider 97 to 98 percent information system availability excellent performance, the remaining two or three percent can represent huge financial losses and risk of non-compliance for pharmaceutical EBR and PAT implementations.⁹ Failure rates as low as two to three percent are unacceptable because they break the process monitoring and record chain, which results in spoiled batches with all the accompanying losses. Without uninterrupted operation, EBR and PAT solutions will not pay off as a cost-effective, risk-based approach to electronic record compliance and the manufacturing efficiencies that go with it. Therefore, the IT infrastructure that supports these applications must exceed standard corporate benchmarks of reliability, availability, and uptime.

From the outside, a continuously available manufacturing infrastructure resembles a conventional IT environment. It consists of servers running applications, databases, storage, and networking components. The servers are linked with storage arrays by router-based networks running protocols over hard-wired and (more recently) wireless connections. In a manufacturing environment, interfaces with plant floor systems pull in data from individual process execution systems.

The primary difference between a conventional and a continuously available infrastructure is their approach to downtime. The prevailing mindset in conventional corporate infrastructures focuses on recovery from errors and failures, an approach known as (return to operations, or RTO). Recovery-oriented solutions assume downtime, even if it's only a few minutes during failover from

one server to another. IT configurations that depend on recovery are typically not appropriate for pharmaceutical EBR and PAT deployments.

Continuous availability's focus, however, is on symptom detection and error prevention, which aligns nicely with recent risk-based guidance by US and European regulators and supported by leading industry associations like the International Society for Pharmaceutical Engineering.^10,11,12 Continuously available infrastructures are built with redundancy and error detection that prevents failure and supports a Six-Sigma, quality management system (QMS) approach to the IT environment that aligns with mandates from regulatory agencies in a cost-effective and efficient manner. True high availability requires looking at the entire infrastructure from both design and operational perspectives.

Selecting the proper technology is a good place to start building an ultra-high reliability infrastructure. A comparison between fault-tolerant computing and clusters illustrates the difference between recovery-oriented and prevention-oriented solutions, and why the latter is better suited to pharmaceutical manufacturing.

FAULT-TOLERANT COMPUTING FOR ELECTRONIC RECORD SYSTEMS

Server clusters are what most often comes to mind when a company considers a high availability solution for supporting EBR or PAT systems. In clusters, pairs of servers linked by clustering software operate as a primary and a backup. If the primary server fails, the software shifts the processing load to the backup server.

The different flavors of various clustering technologies suffer from common weaknesses: complexity, cost, and unproven reliability. To meet pharma manufacturing's QMS requirements, each "computer system" must be properly revision controlled and validated. Clustering software requires duplicate control records and custom scripting. This leads to requirements for ongoing operational qualification (OQ) demands to test the cluster's functionality, in addition to the simple installation qualification and maintenance required for typical hardware components such as a storage system or network device.¹³ Additionally, clusters require two servers, increasing management demand and cost. Finally, clusters run on enterprise-class versions of operating systems, not the relatively inexpensive versions. The cost of the additional compliance activities described earlier adds up quickly. Finally, a software cluster is a configured system, not one that a company can plug in and run. Configuration can be expensive and laborious.

As clusters' weaknesses have surfaced, pharma companies have begun exploring fault tolerant servers that are essentially two servers operating in lockstep inside a single chassis. Until recently, fault-tolerant computers have been available only in the realm of "big iron"— highly funded corporate data centers that have the budgets and staffs to run large legacy systems. In the last few years, however, the market has seen the emergence of cost-effective, fault-tolerant computers that run the commercially available operating systems used in pharma manufacturing and other lower-cost environments. Such infrastructure components can be purchased off the shelf with built-in, factory-tested high availability features. Like their network and storage counterparts, they only require IQ, significantly simplifying qualification and maintenance. Furthermore, only one version of the operating system is required. Some lower-end versions even run the less expensive server operating system configurations (as opposed to the enterprise versions). This form of fault tolerance is ideally suited for the regulatory, cost, and management burden needs in pharma manufacturing.

Consider the example of a major US-based pharmaceutical company that decided to overhaul its IT infrastructure by implementing an EBR system to replace its paper-based one. This company learned that first-hand conventional reliability solutions didn't meet the pharma manufacturing standard.

The company decided that an EBR system would streamline regulatory compliance, eliminate the overhead expense of keeping paper records, and improve production efficiency. For the system to deliver these benefits, however, it needed to provide an uninterrupted stream of data. During proof-of-concept tests with conventional networking technology, the company's IT staff soon discovered the EBR system's reliability limitations. Routine issues like application and operating system crashes, reboots, and scheduled downtime all but erased the EBR system's benefits. The tests also uncovered reliability problems in unexpected places. Server and driver hardware weren't solid enough to resist errors and downtime. Third-party device drivers, which provided interfaces to server peripherals and communications lines, caused the operating system to crash.

The plant IT team chose an ultra-high availability architecture based on fault-tolerant servers. The servers' continuous availability features and remote monitoring capabilities supported an off-the-shelf Windows server applications the company used with none of the extensive customizations clustering would have required. The new system has met or exceeded the computer industry's defining measure of 99.999 percent availability in real-world customer installations. The emphasis on device driver reliability, in particular, counteracted an oft-overlooked root cause of operating system instability with third-party device drivers.

Therefore fault-tolerant computing and infrastructure design are important parts of the solution, but they are only one-half of the picture. Infrastructure management is the backbone of continuous availability — and perhaps the highest barrier to it.

MAINTAINING A CONTINUOUSLY AVAILABLE INFRASTRUCTURE

Although most pharma companies can probably afford to build one, the ongoing cost of maintaining a highly available infrastructure is beyond most plant-level IT staff and budgets, even at larger companies. The constant diligence needed to guard against failures extends to even routine tasks performed hundreds of times per day at most companies.

For example, consider the plugging of a laptop into the network. That's not a risky act at most companies, but for a pharma manufacturer trying to maintain 99.999 percent uptime, it could be a disaster. Just as a natural ecosystem is affected when a new species is introduced, so is a continuously available ecosystem. Plugging an unauthorized laptop into the network may introduce anomalies into an infrastructure — such as a virus. While losing a set of computers to a virus is disruptive in any business, a virus in a pharma manufacturing facility can crash record collection or process control data collection and force the company to scrap a batch that could be worth millions of dollars. Even seemingly innocuous tasks such as upgrading applications, applying patches, and routine maintenance can result in slowdowns and crashes.

Corporate data centers in many industries, including major life science corporations, leverage outside service providers to maintain computing infrastructures and systems compliance based on negotiated service level agreements. Yet, this practice has not yet made it to the pharma manufacturing environment. Recent research has determined the root cause of this is trust. Pharma manufacturing IT people know that such service providers do not understand the details of their environment and their operations, as well as the regulatory mandates they face. These same IT people understand, however, that high availability and QMS are synonymous. They know what they must do to meet their QMS needs, but are bandwidth and budget constrained.

So, compliance requires high availability, and high availability requires a robust QMS for IT infrastructure. But resources are limited at the plant — time and money for robustness seem out of reach. And the core competency of a pharma manufacturer is not IT. What is the solution?

A new breed of service providers is emerging that join the industry knowledge, manufacturing IT experience, and high availability deployment and maintenance skills together. They supplement the manufacturer's IT staff and regulatory processes to provide a robust QMS capability to pharma manufacturers. From design (or analysis of existing infrastructures, people, and processes) to deployment, verification, and on-going management of the infrastructure, such services can be obtained within budgetary constraints. Services such as patch evaluation and deployment, IQ of components, and operational and performance screening (as a preliminary check prior to OQ and PQ) can be outsourced to relieve the burden of valuable internal compliance resources and reduce time-to-delivery and cost of IT capabilities to the production managers. And, most importantly, high availability can be achieved through services that can remotely monitor, detect, and help prevent slow-downs or outages while providing capabilities to maintain proper configuration control and release management of upgrades and end-of-life scenarios.

WORRY ABOUT THE RIGHT THINGS

Vendors who define availability in terms of a single network element such as clustered servers, raid storage, or backup/recovery software are only addressing part of the high availability solution. True continuous availability is not only beyond the capability of clusters; it requires looking at the compute, network, and storage environment as a whole. Continuous availability does not begin and end with any single component but treats the entire flow of information across these components as a potential problem, including the software applications. And while it is vitally important to leverage fault-tolerant components in an infrastructure based on appropriate risk evaluation, the monitoring of the entire environment to prevent failures and to facilitate the lifecycle management of the components and stored information is also a crucial component of the infrastructure.

EBR and PAT applications offer enormous benefits to pharma companies. As regulations grow more complex, reporting more demanding, and markets more volatile, plant-level IT needs to give their pharma companies new measures of flexibility to improve efficiency and hem in costs.14 Companies need continuous availability to make these applications deliver their business value, but they shouldn't be worrying about providing the required high availability. High availability is its own competence, with its own specialists. Better to use them as a cost-effective and reliable solution than try to grow internal capabilities that takes time and money, with the high potential that they can easily move to a competitor.

Dave Femia is the director of the life science practice at Stratus Technologies 111 Powdermill Rd., Maynard, MA. 61754, (978).461.7000.

REFERENCES

1. DePalma, A,"In Search of Consistency," pharmamanufacturing.com 03/18/2003, http://www.pharmamanufacturing.com/articles/2003/71.html.

2. "Quality by Design: A Challenge to the Pharma Industry," www.fda.gov/ohrms/dockets/ac/02/briefing/3869B1_07_Camp.ppt, CAMP Member Companies March 2002

3. "Continuous Quality Assurance for Improved Business & Process Performance in Pharmaceuticals (Proceedings)," IQPC, London, 7/1/2005.

4. "Continuous Quality Assurance for Improved Business & Process Performance in Pharmaceuticals (Proceedings)," IQPC, London, 7/1/2005.

5. "Continuous Quality Assurance for Improved Business & Process Performance in Pharmaceuticals (Proceedings)," IQPC, London, 7/1/2005.

6. Bartholomew, D, "MES Goes Prime Time," pharmamanufacturing.com, 10/20/2004 http://www.pharmamanufacturing.com/articles/2004/225.html.

7. Martin, R,"Pharmaceutical and Life Sciences Industry Outlook: Growth and Efficiency Enabled by Demand-Driven Supply Networks," AMR, 9/17/2004.

8. "Continuous Quality Assurance for Improved Business & Process Performance in Pharmaceuticals (Proceedings)," IQPC, London, 7/1/2005

9. "Best Practices for Continuous Application Availability," Donna Scott, Gartner, 12/2/2004, Data Center Conference.

10. Pharmaceutical cGMPs for the 21st Century — A Risk-Based Approach, Final Report, Fall 2004. http://www.fda.gov/cder/gmp/gmp2004/GMP_finalreport2004.htm

11. "ICH Q9, Note for Guidance Quality Risk Management" (draft), European Medicines Agency, May 2005.

12. "A Risk-Based Approach to Compliant Electronic Records and Signatures," ISPE/GAMP, February 2005

13. "GAMP Guide for Validation of Automated Systems, GAMP4," ISPE/GAMP, December 2001.

14. Larson, K,"Changing Pharmaceutical Industry Must Leverage Manufacturing IT," pharmamanufacturing.com, 12/31/2003 http://www.pharmamanufacturing.com/articles/2004/197.html.