INTEGRATING BMS and PCS Systems in the GxP Environment

Multi-use, hybrid facilities are becoming the standard in the life sciences industry rather than the exception. Most facilities are designed to serve more than one purpose. While housing a manufacturing area may be a building's most important function, it may include a group of labs or warehousing space. These multi-purpose facilities create an interesting challenge. This article examines these challenges and potential solutions.

As an example, consider a facility with process and packaging lines, offices, a warehouse, some common spaces, and a cafeteria. The owner's challenge is to simultaneously manufacture the product, access and store the data, provide a comfortable environment, and comply with GxP regulations.

In theory, a single system can be used to accomplish these goals. Otherwise, the building management system (BMS) and the process control system (PCS) must be integrated. We will discuss the single-system scenario as well as five schemes for integrating BMS and PCS systems.

SINGLE SYSTEM

The differences between BMS and PCS systems make it extremely challenging to integrate them into a single control system operating the process line and managing the environment. However, single systems are being explored and implemented with varying degrees of success. BMS is designed to control the heating, ventilation, and air conditioning (HVAC). It is "fit for purpose" with specialized algorithms and built-in functions specifically designed to streamline HVAC installation and operation. Other utilities are often controlled by the BMS but, BMS systems struggle with process control. They are not efficient in batch processes, and they are not designed for applications where sub-second sampling rates are required.

PCSs such as programmable logic controllers (PLCs) and distributed control systems (DCSs) are designed with process control in mind. They have software designed for controlling batch recipes, and they are equipped to handle high-scan-rate applications. However, it is difficult and costly to apply these systems to basic HVAC applications due to their hardware and software design.

For example, in office spaces, variable air volume (VAV) boxes provide conditioned air to each zone. The BMS has controllers with built-in flow transmitters and electric actuators that are designed to mount onto the VAV box's damper shaft in the plenum and control it based on temperature sensing in that zone. Additionally, the algorithms that control all the VAV applications are embedded in the controller and need only be selected in each application; no "programming" is required. This is a specialized, small point-count application that requires application-specific control to work effectively.

A PCS would struggle to handle this application efficiently and cost effectively. A PLC or multi-point controller is necessary, and a separate actuator would be needed for damper control. Ladder logic programming must be created to offer the appropriate control for the VAV box. Even if you find a ladder-logic programmer who understands this application, you will still lose the benefit of the specialized algorithms and functions already built into the BMS controller. The cost of the BMS application likely would be much less than the poorly adapted PCS solution.

Finally, there is the issue of validation. According to the current Good Automated Manufacturing Practices Guide (GAMP 4) of International Society for Pharmaceutical Engineering (ISPE), programmable controllers such as PLCs require Level 5 validation (the most stringent) since the code is untested. Most BMS controllers are configurable and require only Level 3-4 validation. The software objects are fully tested and only require configuration to make them application-specific. In short, one size does not fit all applications.

INTEGRATE TWO SYSTEMS VIA S95

As you can see, the differences between BMS and PCS systems make the single-system approach very difficult to implement. System integration is the most effective solution, allowing interaction and data passing between the two systems so they act more like a single system.

Facility integration often includes security and safety systems. Although these are outside the scope of this article, the concept of system integration still applies. In most cases, security and safety can be integrated into the BMS system, depending upon the owner's facility operation plan and requirements.

S95 is a standard developed by the Instrumentation, Systems, and Automation Society (ISA, www.isa.org) to offer some consistency to the architectures and terminologies used in the automation industy.

We will use the S95 standard (Figure 1) as a framework for our discussion of system integration. S95 divides control architecture into five distinct layers that perform separate functions before sending data to higher levels of the hierarchy. S95's five levels are:

• Level 0: discrete manufacturing and control (instrumentation)

• Level 1: discrete process devices (field bus controllers)

• Level 2: automation network (supervisory controllers)

• Level 3: operations information network (user interfaces and application servers)

• Level 4: business process information network (ERP, business systems, etc.).

Figure 1. ISA’s S95 Architecture

This article will discuss the pros and cons of pursuing system integration at each of the five levels defined by S95:

• field bus instrumentation integration

• field bus controller integration

• supervisory level integration

• enterprise level computer protocol integration

• database integration.

As you examine the diagrams representing these strategies, you will see the same set of building blocks, but the main connection between the two systems moves further to the right, indicating integration at higher levels in the S95 architecture.

S95 was created as a standard for process control, but BMS solution providers also are applying their solutions to this architecture.

Field Bus Instrumentation Integration (S95 Level 0)

There are two main ways of integrating the BMS and PCS at the field bus level using instrumentation. The first way is to use dual sensors sending separate signals to the two systems (Figure 2). One sensor will be validated, and the data will be sent to the PCS for room condition monitoring. The other sensor will typically remain unvalidated and will provide data to the BMS for control. This approach is widely used in the industry but creates some challenges. First, it can be difficult to ensure that the two sensors remain consistent with each other and remain identically calibrated. If the validated sensor is gathering environmental data for the production records, that data must match the data used to control the room.

Figure 2. Field Bus Instrumentation Integration with Two Sensors

Table 1. Disadvantages and Advantages of Field Bus Instrumentation Integration with Two Sensors

The second method uses a single sensor (with a transmitter) and splits the signal between the two systems. This approach requires validating the sensor, the transmitter, and the splitter (Figure 3). One side of the splitter goes to the PCS for record retention, while the data on the other side of the splitter goes to the BMS for control.

Figure 3. Field Bus Instrumentation Integration with One Sensor

Table 2. Disadvantages and Advantages of Field Bus Instrumentation Integration with One Sensor

Field Bus Controller Integration (S95 Level 1)

Another method of integrating the systems at the field bus level is passing data between controllers. This is not always possible since the BMS and PCS systems use different communication protocols. At the field bus level (Level 0, 1), BMS systems typically use one of the following protocols: proprietary RS-485, open RS-485 (N2), LONworks, or BACNet (MSTP). In comparison, a PCS typically uses the following protocols: ModBus (the most widely used), ProfiBus, DeviceNET, or ControlNet.

There is almost no commonality between the two sets of protocols. Drivers are available that allow BMS systems to communicate with one or more of the PCS protocols. However, it is best to be cautious of such solutions, as the drivers may not provide all the data necessary for seamless integration (Figure 4). ModBus requires a master-slave arrangement, and you must ensure that two masters do not exist on a single network. However, many of BMS-ModBus integration tools must be set up as masters, and the PCS system probably includes a ModBus master.

Figure 4. Field Bus Controller Integration

Table 3. Disadvantages and Advantages of Field Bus Controller Integration

Open and interoperable protocols, such as LONworks and BACnet, are the most desirable solution since they offer maximum flexibility to the facility owner. However, a complete facility-integration strategy must address systems and devices employing proprietary protocols (due to incumbency or application specifications). The main concept to keep in mind is that protocols are a means to an end. They simply exchange data between devices and applications; they do not provide automation or control.

Supervisory Level Integration (S95 Level 2)

Integration possibilities also exist at the supervisory level (Level 2 in the S95 architecture). This level acts as a gateway to the building IT network, handles global calls, and monitors systems traffic to ensure that the system is being a good IT citizen. Most supervisory controllers offer functions such as global point passing, global trending and buffering, and global alarm management.

Level 2 integration schemes pass data from the BMS supervisory controller directly to the PCS supervisory controller (or vice versa) over a common protocol or via a gateway. In the case shown in Figure 5, a network communication module (NCM) with a network port that speaks ABDH (Allen Bradley Data Highway) or DH+ (Data Highway Plus) and can pass data directly to the Control Logix PLC. If the BMS does not have this capability, a gateway device can table the data, convert it to the appropriate protocol, and send it to the PLC. Obviously, a direct path is better because it is faster, more reliable, and less costly to set up and maintain.

Figure 5. Supervisory Level Integration

Table 4. Disadvantages and Advantages of Supervisory Level Integration

ELCPI (S95 Level 2-3)

Enterprise level computer protocol integration (ELCPI) uses computer workstations (PCs) to share data. There are a few options available to pass data from PC to PC. The most widely used and most robust solution in the industry today is object linking and embedding (OLE) for process control (OPC). This Microsoft technology has become an industry standard.

OPC took over where dynamic data exchange (DDE) left off and provides a relatively seamless path between two OPC-compliant software packages. By utilizing component object model (COMM) and distributed COM (DCOM) technologies, OPC (representing info at the data presentation level on the ISO diagram), can act as a client or server application to present data (DA OPC), alarms and events (A&E OPC), and, in some cases, to collect trends (historical OPC). Alternatively, the most advanced systems support web services and can transfer data directly using embedded extensible markup language (XML).

Figure 6 shows data being passed between an OPC client-server on the Rockwell Automation PCS directly to the Metasys for Validated Environments workstation, which has a built-in OPC server. Data from Levels 0-2 have risen through the architecture and reside at Level 3, the enterprise level. These data are available to be shared, and there is no reference to which protocol is used because OPC represents data obtained by the protocol driver inside the application.

Figure 6. Enterprise Level Computer Protocol Integration

Table 5. Disadvantages and Advantages of Enterprise Level Computer Protocol Integration

Database Integration

The highest degree of integration is at the database level. Database integration allows data to be gathered independently and then exchanged (Figure 7). Typically, open database connectivity (ODBC), a Microsoft technology, can be used to exchange data between relational databases. Databases that support this exchange include MS Sequel and Oracle. Not all original equipment manufacturers (OEMs) support open, relational database structures. When dealing with OEMs that offer only proprietary database solutions, budget extra money for database integration.

Another very common method of database integration is creating a single master database integrating all system data. The advantage of this approach is the reduction in operational costs and maintenance. The key here is ensuring that data is kept intact and secure throughout the integration. Integration is pointless if the data are not reliable or they are not accessible to FDA auditors.

Figure 7. Database Integration

Table 6. Disadvantages and Advantages of Database Integration

FACILITY INTEGRATION EXAMPLE

Let us apply these alternatives to the previously introduced model facility, which housed production and packaging lines, some warehouse space, offices, common spaces, and a cafeteria. Assuming that this facility has been designed to provide effective physical system boundaries, our goal is to attain the level of integration needed to satisfy facility control and operational requirements while addressing FDA's data retrieval and records retention rules — all at a reasonable cost. As you can imagine, this can be a daunting task, but it is almost always achievable.

We will implement a PLC solution for the process line and a BMS for HVAC. This will give us excellent control of both the production process and HVAC by systems that are fit for purpose. It will also allow us to take advantage of the process-specific algorithms designed to make system operation easier.

In addition, this gives us the best physical system separation for defining system boundaries. When a single system on a single network is forced to cover multiple applications across the entire facility, it can be difficult to draw the system boundaries. Physically separating the GMP network and the non-GMP network creates clear system boundaries.

We must determine how our systems need to communicate to accomplish each of the following goals:

• integrating environmental data to lot and batch production data

• integrating alarm data to the process floor

• sending environmental data to the data historian for record retention.

Then we can identify the best integration approach and implement the solution in a way that provides maximum reliability while minimizing (or at least mitigating) risk of data loss, FDA observations of noncompliance, or production shutdowns.

The non-GMP BMS is not linked with any other system. It is a stand-alone application that controls the environment in the building's cafeteria, hallways, offices, and interstitial spaces. In some scenarios, this non-GMP BMS is integrated with other non-GMP systems, such as security and access control systems and fire monitoring systems, but this paper will focus primarily on the GMP side of the facility and network.

Integrating Environmental and Production Data

In most production environments, it is critical to be able to associate historical environmental data with process data to demonstrate that the environment was within specification during batch and lot production and storage. This can be accomplished in many ways. Three of the five integration approaches described above would be acceptable solutions (field bus instrumentation integration, ELCPI integration, or database integration), but which one is most effective?

Many owners use the field bus instrumentation integration solution because it is simple and fairly dependable. However, utilizing two sensors and sending the data from one sensor to the PCS for monitoring and records retention and the data from the other sensor to the BMS for control is expensive when you consider the cost of the industrial sensors and transmitters required and the lifecycle and operational cost associated with keeping two sets of sensors calibrated and maintained. Using one sensor and a qualified splitter that sends the data to the BMS for control and the PCS for monitoring and records retention is a more cost-effective solution, but remember that the splitter must be calibrated per your standard operating procedures.

The ELCPI option involves passing critical data between the BMS and PCS workstations via a common protocol. Since it usually happens that a common protocol does not exist, this integration requires a protocol converter or integration device. The integration device is simply an industrial PC that houses software drivers for both the BMS and the PCS protocols. The PCS and the BMS are connected to each side of the PC, typically via a serial connection. The integration device tables the data on both sides, enabling data exchange.

Finally, one could go to the top level of the architecture and use an integrated database. This differs from the previous approach because each system gathers and stores its own data before bringing it together. This can be very tricky because it requires organizing both databases identically. In most cases, one of the databases (most likely the PCS database) is assigned as the master database and data from the subservient database data is queried, copied, and imported into the master database via a reporting tool. Alternatively, that system can input its data directly into the master database, eliminating the subservient database and eliminating the steps involved in combining the data. The only disadvantage could arise if you need to separate the data for any reason.

Despite these issues, ELPCI integration can be the most effective solution for tying environmental data to lot and batch production data. Once this method of data transfer is set up correctly and properly tested, it is typically a reliable solution with a low cost of ownership.

Alarm Integration

Depending on how critical the environment is to the production process, the production line may need to be shut down if the environment goes outside of the control parameters. Production of a bulk powder product open to the environment during one or more production steps is an example. In this case the temperature and, more importantly, humidity are critical to keeping the product granular and avoiding clumping. In this example, the BMS can be set up to direct alarms to many locations including the following:

• production operator's or manager's email account

• production operator's or manager's pager

• traffic light alarm enunciator

• directly to the PCS (for warning or shutdown depending on criticality).

The first three options can all be handled by a BMS's alarm-management software. The last option requires one of the following integration strategies: field bus controller integration, supervisory controller integration, or ELPCI integration.

Depending on the process protocol, field bus controller integration can be unreliable. This would be the most efficient way to handle the integration (since the data would not have to travel up the BMS network, only to be sent back down the process network), but the availability of efficient integration tools is limited. In most cases, a gateway device would be required to make this work.

At the next level of system integration, you can use Ethernet and, in some cases, OPC and XML technologies, which are supported by both BMS and process OEMs. Most manufacturers of BMS and PCS supervisory devices communicate via Ethernet, which provides a common, open protocol for communication. However, many of these same devices use proprietary packages to send data via Ethernet connections, making communication with other devices challenging. Knowing that both systems support Ethernet communications does not ensure easy integration. In our example of supervisory controller integration, the network port passes data between the BMS and PCS. It must be noted that this is a vendor-specific solution, but it is probably the most reliable and effective one for this task.

With ELPCI, the BMS alarm data can be sent to the operator workstation, transferred to the PCS workstation, and then routed back down to the appropriate controller to affect the warning or shutdown. Workstation-to-workstation integration is typically performed through OPC or XML, industry standards supported by all reputable system manufacturers. This is a longer path, but it can be a reliable and predictable solution.

Record Retention

This integration is only required if the BMS is a separate system without its own data historian. Utilizing the PCS data historian for all GxP data is a common approach. It allows the owner to maintain only one historian for both major systems while ensuring environmental data are matched to production data for internal and external audits. Supervisory-level, enterprise-level, and database-level integration are possible solutions.

Each of these integration strategies are operation-critical, since they all involve sending data for proof of validated production conditions. Therefore, it is imperative that these integrations are robust, dependable, and can be validated.

Supervisory-level integration involves sending data from the BMS supervisory controller via a common protocol to either a supervisory controller on the PCS or to a PCS workstation (typically via OPC or XML). Not all BMS solution providers support this approach. Be certain to verify that your BMS vendor has a network port that can send data to the PCS in a language and protocol the PCS can understand.

Enterprise-level integration involves sending raw data from the BMS operator workstation or network controller using a common protocol such as OPC or XML. These data are then gathered in the PCS database for eventual transfer to the data historian. The advantage of this approach is the availability of the data from both processes at the PCS workstation. As a bonus, only one database must be maintained.

Database integration is probably the best solution for this application. The BMS will always send the control data to a database for its own use. A good BMS should offer a non-proprietary, albeit secure, database. As mentioned earlier, ODBC is typically used to share data between relational databases. Assuming that both the BMS and the historian support ODBC, this is a reasonable setup. The only disadvantage is the need to maintain two databases.

CAVEAT EMPTOR

From the owner's perspective, integration is a tool to be used only when needed to enhance control, ensure data acquisition, reduce operation costs, or assist in validation boundary definitions. However, successful integration can generate fabulous results and create a unique, showcase atmosphere that builds team pride. It also can be fraught with danger and expense. Many integration projects turn into funded science projects, so buyers must be cautious. Knowledge of integration options is the best way to protect yourself from dishonest or uninformed system integrators.

Addressing the following points can help ensure a successful integration project:

• Define the need. Why? What's the benefit?

• Prove the technology. Can it be done with existing, proven technology?

• Establish the budget. How much is it worth to my company?

• Evaluate the system integrator. Who can provide this for me?

• Prototype the solution. Make them show you.

• Execute the successful prototype. Look, it worked again!

• Verify the success. Is it working like I expected?

• Document for future use. Can we use this again?

Good luck and happy integrations.

Jeffrey Bredeson is business development director for Johnson Controls Life Sciences Group, 1255 N. Senate Ave., Indianapolis, IN 46202, 317.917.5172, fax 317.638.6146, jeffrey.s.bredeson@jci.com.

Dario Sala is director of system solutions - Europe for Johnson Controls Life Sciences Group, Via Monfalcone 15 Milano, Italy 20132, 011.39.02.28042.304, fax 011.39.02.28042.221, dario.sala@jci.com.

Greg Weddle is manager of global critical environments for Johnson Controls Life Sciences Group, 1255 N. Senate Ave., Indianapolis, IN 46202, 317.917.5170, fax 317.638.6146, gregory.b.weddle@jci.com.