Quality Control: Operator Error: Is It Really the Root Cause of Performance Problems?

This article explores unintended consequences of automatically using operator error as the root cause of performance deviations. In addition, the article describes how to conduct a root-cause analysis of training events using a cause-and-effect diagram. The diagram is slightly modified to encompass the "4-M" checklist—man, machine, methods, and materials—while integrating questions and suggestions focused on training system elements.

At the conclusion of performance training events, there is usually some form of assessment of the knowledge gained or a demonstration of skills learned. Then, after the formal attendee documentation is completed, trainees return to work. If a performance problem occurs soon thereafter, an investigation is launched to determine what happened. More often than not, the root cause is documented as "operator error," and the corrective action is retraining. How many times have you heard of this scenario?

What's wrong with using operator error as a root cause? First, it draws regulators' attention to weaknesses in your current training program and the management of the training program. In addition, it identifies flaws in the organization's corrective actions and preventive actions (CAPA) program. Here's why: the label "operator error" sends a blatant message that training wasn't effective and that operations are not in control. That in turn leads to the impression that the CAPA program is weak because the firm conducts inadequate root-cause investigations.

Therefore, companies must recognize that identifying operator error as a root cause strongly suggests to external regulators that things are not right, and that the real root cause needs to be more thoroughly analyzed and effectively addressed by the organization to demonstrate that the training and CAPA systems are performing as designed.

HOW REGULATORS REACT TO OPERATOR ERROR

When regulators are onsite conducting an inspection or any type of visit that leads to a review of the training or CAPA program, potential follow-up questions are always asked regarding operator errors. The first and obvious question is: "Why wasn't the training effective?" This question can lead to scrutiny of the effectiveness of the training system and an exploration into the types of knowledge checks or learning assessments that are conducted. Be prepared to produce the approved training procedures.

Table 1. Elements of a typical corrective action–preventive action (CAPA) program

The next potential question suggests a lack of confidence in the organization's ability to perform tasks correctly on the production floor. Subsequent questions follow, including: "How many times does operator error occur?" "Is it with a specific operator?" "Is it with a specific procedure?" and "What is management doing about this?" Be prepared to explain the course of action taken to remedy the situation.

HOW QUALITY ASSURANCE GROUPS REACT TO OPERATOR ERROR

An industry trend is slowly emerging: Not only is operator error signaling an alarm for external investigators, but internal quality assurance (QA) groups, as well, are starting to reject operator error as the root cause of an investigation, especially if it is a repeat issue leading to the uncomfortable questions listed above.

Quick Recap

NEED TO GET TO THE REAL ROOT CAUSE OF THE PROBLEM

Companies facing these questions from either regulators or their QA groups, or both, should conduct a root-cause analysis of their training programs; they should also consider reviewing the effectiveness of their overall CAPA programs in revealing and correcting root causes of performance problems. The following paragraphs describe how to conduct a root-cause analysis of performance training as part of the CAPA program (Table 1).

The Cause-and-Effect Diagram and the 4-M Checklist

A cause-and-effect diagram is an effective tool for identifying potential root causes of a performance problem. This tool helps managers methodically analyze the factors of a process and visualize their contributing role to the situation. Grouping these factors into the 4-M categories—man, machine, methods, and materials—provides a handy checklist to ensure that all categories are considered. (A fifth category, measurements–metrics, is also used by some organizations.¹ ) The four categories can be translated in the following manner: man refers to people (both trainer and trainees); machine is the tools; methods are the procedures, written instructions, activities, etc.; and materials are the training materials (Table 2). A proper root-cause analysis must evaluate all these groups to uncover the cause of the problem.

Table 2. Training program elements in the 4 M categories

A cause-and-effect diagram is also known as a "fishbone diagram" because it resembles a fishbone. Each bone represents one of the 4 Ms, and the head represents the potential root cause. Figure 1 depicts a 4-M fishbone diagram with the matching training elements.

FIRST M, MAN: OPERATORS, TRAINEES, TRAINERS, SUPERVISORS AND MANAGEMENT

Operators and Trainees

Let's start with the first M, "man," and focus on the operators and trainees. For operators and trainees, it must be determined what is causing the performance discrepancies. Performance discrepancies occur when actual performance does not match expected performance.² For example, a deviation from written procedure or batch record has occurred. The overarching question to address is whether the discrepancy is a deficiency in the employee's knowledge or skills, or whether other factors or a combination of factors are involved, such as training quality or management reinforcement. Determine if there is a lack of procedural understanding, or a case of poorly worded work instructions with ambiguous steps for decision-making. Consider if the training event involved a new procedure with new concepts for which proficiency might have required more time than originally planned. Check training documentation to see if prerequisite knowledge was required, or if trainee skills were missing. Did on-the-job training (OJT) include supervisor or peer involvement to verify that the new procedure was understood? Perhaps this is truly a one-time mistake that has not occurred before. Determine if the operator is assigned to the correct role. Was the trainee able to perform the procedure in the past but has now forgotten how to do it? Consider how often the operator is expected to perform the task. Explore how often trainees are receiving specific feedback regarding acceptable standards, and how often they are practicing the skills needed to perform.

Table 3. Possible courses of action for addressing performance problems

If it is determined that there is a knowledge or skill deficiency, then a decision should be made about whether refresher training or retraining is appropriate. When the evaluation determines that the operator lacks physical strength, required skill, or mental capability to complete the task correctly, then retraining or refresher training wastes time and money and raises false expectations. In that case, either change the nature of the task or redirect the operator to another, more suitable position. If all else fails, you may need to explore an exit package (Table 3).

Trainers

Are trainers qualified as subject-matter experts? Find out what is the pace of their delivery. Is it too slow, too fast, too technical, or too full of jargon for initial training? Review course evaluations, if applicable. Are the standards of acceptable performance being communicated and demonstrated? Are trainers providing enough practice time with constructive feedback during training? Determine if trainers are following a leader's guide or a lesson plan that includes a delivery checklist or on-the-job training checklist to ensure complete and consistent training. The analysis may need to include interviews with random operators to collect information on actual training practices and compare those practices to written training procedures. For example, in one instance it was discovered that one trainer taught his group to do things one way, while another trainer instructed her group differently.

Supervisors and Management

Supervisors and managers play a critical role in the effectiveness of training programs, and in the performance of operator and trainee. To evaluate the potential role of supervisors and managers in root causes, explore the post-training reinforcement that management provides. Mary Broad and John Newstrom, authors of Transfer of Training, concluded in their field research that the number one barrier of transfer of training is the absence of reinforcement from management back on the job.³ Are managers scheduling adequate time for training? Are they providing sufficiently frequent feedback? Include second- and third-shift training practices, in which there is frequently less managerial presence. Is the feedback constructive and specific? Find out if, how, when, and how often feedback is delivered. Is it delivered only once a year during performance reviews, or only during disciplinary sessions? Feedback delivered at these times comes too late—it's after the fact. For behavior and performance to improve, managers must provide timely and specific feedback about what task, action, or behavior needs attention; how the improvement is to occur; and what is the acceptable level of quality.

SECOND M, MACHINE: TOOLS

Tools can be anything from a physical tool itself, such as a computer, to a form or a checklist. What kinds of job aids are available besides the standard operating procedures (SOP)? Are instructions for completing forms inadequate or confusing? Is the physical equipment reliable and qualified, or is it new and not yet part of a routine work practice? Are there assessment tools, such as knowledge checks and follow-up monitoring, for the knowledge transfer? Determine if these tools are available, valid, and reliable. Do the conditions under which the operators are trained match the conditions under which they are expected to perform? Unless the tool matches what is to be measured, you will not have substantial data to validate the effectiveness of training. By matching the performance conditions called for in the assessment or knowledge check with the conditions of the learning objectives, you will ensure that the objectives were in fact met, thus providing evaluation data of the effectiveness.⁴ If the problem involves a lack of tools, investigate whether reduced budgets have prevented restocking. If there is a lack of, or poorly stocked and stored, equipment and tools, investigate whether there are improper housekeeping practices.²

THIRD M, METHODS: PROCEDURES

The first set of procedures to review is the training program or system procedures. Do they describe how to develop, deliver, and assess training, or are they inadequate and incomplete?⁵ Consider the date trained versus the date performed. Investigate how much time elapsed before the operator put the skills to use. Training loses its effectiveness unless it's used immediately. Also consider how the training was delivered: via hands-on or read-and-understood-only training? Via group or classroom training as opposed to one-on-one? The bulk of today's training involves procedures, and effective procedure training is critical. At the conclusion, you want the operator to perform the procedure—not memorize it—so move quickly to the application by providing demonstrations and exercises that require operators to perform the activity described in the procedure.⁶ Allowing trainees to practice new skills and procedures engages them in the learning process; it not only increases their retention, but it promotes faster transfer of the learned skill back on the job.⁷ What if the root cause is something else that is causing non-compliance with a procedure or work instruction? The analysis should explore barriers to performance. If the barrier is a lack of authority, then determine if there are unclear roles and responsibilities. If it's a lack of time, look at scheduling conflicts.

Figure 1

But what if the situation is a case of "We've always done it this way?" Then the analysis should determine whether operators were ever able to perform the approved procedure. Explore whether the procedure is accurate. Is the SOP author a subject matter expert (SME)? Determine if the procedure was written with support from a lead operator. Confirm if the draft procedure was challenged using a dry run or a field test with those responsible for execution. Verify if proposed changes are stalled in change control awaiting effective release long after training is completed.

FOURTH M, MATERIALS: TRAINING MATERIALS

The analysis should evaluate the quality of training materials. Start with the objectives. Are they measurable? Assess the training design and confirm if delivery includes activities, exercises, and demonstrations. Evaluate if content is up-to-date, accurate, and approved by QA when required. Is the sequence logical and easy-to-follow? Determine if sufficient time is allocated to each training item, and if there are opportunities to practice the new skills. Are assessments or knowledge checks required and included in the leader's guide? Confirm if course evaluations are being completed.

KEY ELEMENT OF CAPA SYSTEM

Identifying the root cause is a key element of a CAPA system, so that appropriate action is taken to eliminate the cause or source of the problem and to prevent further recurrence. By examining the problem using the 4 Ms and a fishbone diagram, you avoid the temptation of selecting the first apparent cause as the root cause. The 4 Ms and the fishbone diagram force you to generate all potential causes, and they foster a thorough analysis of the contributing factors, not just the obvious ones; this ensures that the corrective and preventive actions are not just a quick fix.

SUMMARY

Thoroughly addressing persistent operator-error performance problems through thoughtful analysis allows companies to identify the real root cause of a problem; it also enables them to take the proper corrective action to maintain and improve compliance performance and manage risk effectively. The alternative is explaining to an FDA investigator, or to your management, why repeat training is the appropriate corrective action even when numerous deviations indicate it is not fixing the problem, or preventing it from recurring in the future.

Vivian Bringslimark is a senior consultant with PAREXEL Consulting, 19 Cobblers Mill Road, Sandy Hook, CT 06482, 203.470.8363, fax: 203.270.6519, vivian.bringslimark@PAREXEL.com

REFERENCES

1. Suzaki K. The new shop floor management: empowering people for continuous improvement. New York: Free Press; 1993. p. 391.

2. Mager R., Pipe P. Analyzing performance problems. Belmont, CA: Lake Publishing; 1984.

3. Broad ML, Newstrom JW. Transfer of training: action-packed strategies to ensure high payoff from training investments. Scranton, PA: Addison-Wesley Perseus Press; 1992.

4. Mager R. Measuring instructional results. Belmont, CA: Lake Publishing; 1984.

5. Bringslimark V. If training is so easy, why isn't everyone in compliance? BioPharm Int. 2004;17(1):46–53.

6. Clark RC. Developing technical training: a structured approach for the development of classroom and computer based instructional materials. Phoenix, AZ: Buzzards Bay Press; 1994. p. 41.

7. Silberman A. Active training. New York: Lexington Books; 1990. p. 95.

SUGGESTED READING

Association for the Advancement of Medical Instrumentation; The quality system compendium: GMP requirements and industry practice. Arlington, VA; 1998.

Brookfield S. Understanding and facilitating adult learning. San Francisco: Jossey-Bass; 1986.

Carr, C. Smart training: the manager's guide to training for improved performance. New York: McGraw-Hill; 1992.

Jacobs RL, Jones MJ. Structured on-the-job training: unleashing employee expertise in the workplace. San Francisco: Berrett-Koehler; 1995.

Mager R. Preparing instructional objectives. Belmont, CA: Lake Publishing; 1984.

Rothwell WJ, Kazanas HC. Improving on-the-job training: how to establish and operate a comprehensive OJT program. San Francisco: Jossey-Bass; 1994.

Senge P, Kleiner A, Roberts C, Ross R, Smith B. The fifth discipline fieldbook. New York: Doubleday; 1994. p. 127.