A Signal-to-Noise Ratio is No Subsitute for a Brain

Orchard,Terry;

A Signal-to-Noise Ratio is No Subsitute for a Brain

March 1, 2006

By Terry Orchard

Publication

Article

BioPharm InternationalBioPharm International-03-01-2006

Volume 19

Issue 3

Pages: 40–45

Signal-to-noise ratios are useful in robust engineering to design products and processes that consistently deliver on target.

Dr. W. Edwards Deming, the renowned quality guru who led the Japanese post-war industrial revival, often told his followers that, "The control chart is no substitute for a brain."

Terry Orchard

I was reminded of this teaching when reading the article by James McAllister, "Stop Rejecting Good Batches—Use a Signal-to-Noise Transformation," which appeared in the July 2005 issue of BioPharm International.

This is not meant as a criticism of McAllister's article—far from it. McAllister's article was valuable and stimulating. Rather, this article elaborates on the use of control charts and describes the relevance of the signal-to-noise ratio McAllister proposed.

Bespak Europe

THE USE OF CONTROL CHARTS

McAllister's article focused on the following four points:

1. When batch-to-batch variability is high, using an X-bar chart to track the batch means often results in the majority of the means falling outside the 3-sigma limits.

2. The transformation to a signal-to-noise ratio (S/N) leads to 3-sigma limits that contain most of the S/N values.

3. The S/N combines infor-mation on the replicates' variability and the degree off-target.

4. The S/N values are Normally distributed, whereas the means are not.

McAllister's article presented three replicate results for 41 sodium dodecyl sulphate polyacrylamide gel electrophoresis (SDS-PAGE) analyses. The variation among the 41 means was much larger than it was among the three replicates, and as a result, an X-bar chart was of no value. A routine solution to this problem is using an individual–moving range–R chart (I–MR–R) chart. An I–MR–R chart displays the individual means, moving ranges of the means, and ranges of the three replicates in separate panels. The component charts can be produced separately. With a package such as Minitab, the I-chart of means will require calculating the means before the chart can be produced.

Figure 1. Individual chart of mean shot volume for 213 batches

Data with a similar pattern often are seen in the volume or weight of drug dispensed by metered dose devices. For these data, the test results for accepting batches are the means of the quantity delivered by a sample of 50 to 125 valves. In such cases, it is appropriate to use control charts of the means to manage the processes. As with the SDS-PAGE data described by McAllister, almost all the means would fall outside control chart limits that failed to allow for the large batch-to-batch variability.

An example of a typical control chart is shown in Figure 1, which shows the means of 213 batches as calculated from three replicate measurements on a sample of 125 valves. Because the average moving-range estimate of sigma is reduced by the clustering of batch means, we used the standard deviation of the 213 means for the "3-sigma" control chart limits instead of relying on the Minitab default.

Figure 2. Individual chart of McAllister S/N for 213 batches

The chart of S/N_T shown in Figure 2 uses a value of 50 for the target; the two charts are very similar. The S/N_T value on the upper limit of 39.3 has a volume precisely on the target and a small standard deviation of 0.542.

The signal-to-noise ratio used by McAllister is set out in the following equation:.

in which T is the target.

For convenience, our laboratory reports of the volume or weight of drug dispensed present a concise summary of the information on the performance of each batch. This summary consists of the mean, the coefficient of variation, and the minimum and maximum of 375 individual measurements. This summary provides all the relevant information contained in the individual three replicate measurements of 125 packs.

The absence of raw data is not a problem because we can calculate S²_T by modifying Equation (1) as follows:

The standard deviation S_x can be estimated by multiplying the coefficient of variation (CV) by the mean or by dividing the range (R) by a constant (d₂) that depends on sample size. Values of d₂ are available in reference books such as Wheeler and Chambers (1992). Some values of d₂ are: 1.128 for n = 2, 1.693 for n = 3, 3.078 for n = 10 and 3.735 for n = 20. For a sample of 375 (3 repeats x 125 valves), the value of d₂ is 5.90.

Presenting S/N_T in the form of Equation 2 may make it easier to understand what the signal-to-noise ratio tells us about the process. We can see that the term 20 * log₁₀ T, which = 33.979 if T = 50, can be regarded as a baseline.

The second term,

makes a positive contribution to S/N_Twhen

is less than 1.0, and a negative contribution when it is greater than 1.0. Thus, high values of S/N_T require the mean to be "on-target" and the variation between replicate measurements to be small.

Because these data have large differences between means and small differences between replicates,

can be expected to dominate the second term. The differences between the mean and the target range from –2.7 to +0.5, and the values of S_x range from 0.4 to 1.4. If S_x² = 0.5, the difference between the mean and the target has to be less than 0.7 for the second term to make a positive contribution.

Because

has a large effect on the value of S/N_T, there is little difference between the two control charts in Figures 1 and 2. There is only a small chance that the chart of Xbar will lead to the rejection of good batches that could be prevented by using a S/N_T chart. From Equation (2), we can see that high values of S/N_T, particularly those that exceed the upper limit of the control chart, are "good" because they result from means that are on target with very small differences between the replicates.

As McAllister pointed out, signal-to- noise ratios are particularly useful in robust engineering, where the goal is to design products and processes that consistently deliver "on target." But combining control chart information on replicate variability with the extent to which the mean is off-target is not necessarily beneficial. A very high or low batch mean indicates a shift in the process, which could result from equipment, raw material, or the operator. A high range or standard deviation is more likely to result from a large measurement error. The combined information in an S/N_T chart can tell us that there is a problem, but identifying the possible causes requires an examination of the mean and standard deviation used to calculate S/N_T.

We also need to be aware that when individual responses or measurements are combined, it is possible for opposite differences to cancel and for one of the terms to dominate. The dominance of the

term in S/N_T makes it impossible to detect repeatability problems, and some of the acceptable S/N_T values may have very high standard deviations of the replicates. This is easy to demonstrate using Equation (2). For example, the very low S/N_T of 13.97 for McAllister's batch EE results from a mean of 83.67 and a range of 2. We can calculate that a range of 0 for batch EE would only increase S/N_T to 13.98. However, a mean around the overall average of 93.8 and a range of 12, fifteen times the average, will result in S/N_T= 19.67, a value so far within the limits that no warning bell would ring.

McAllister pointed to the more normal distribution of S/N_T as a benefit. The more normal distribution is not surprising because a log transformation reduces extreme values. However, the transformation is not necessarily beneficial. Extreme values are what control charts seek to detect and thus we should use our brains to decide if we want to use a transformation that reduces extreme values.

We also need to be aware that extreme values that are not typical for a process that is "in-control" should be excluded from the data used to calculate the control chart limits. For this reason, it is often worthwhile to use an outlier test to verify extreme values. The book by Iglewicz and Hoaglin is a good source of methods. A test flagged two of McAllister's SDS-PAGE means as possible outliers. These were 84.733 and 83.667. Removing them produced a distribution of raw data that, while not as close to normal as the S/N_T, were not significantly different to normal.

Figure 3. Means and ranges of three replicates

CONCLUSIONS

In conclusion, although using control charts of a signal-to-noise ratio is a novel suggestion that may be useful, combining off-target and repeatability variability hides information. The problems associated with large batch-to-batch variability can be solved easily by simple standard procedures. Calculating control chart limits with the two outliers removed resulted in the charts of the means and ranges presented in Figure 3 and the S/N_T chart presented in Figure 4. The separate Xbar and R charts use the same data and require about the same amount of work to produce. Those charts indicate the presence of three very low means and one very high range that are worth investigating.

Figure 4. Individual chart of T S/N ratio

Terry Orchard is statistician at Bespak Europe Blackhill Drive, Milton Keynes, MK12 5TS, UK +44.1908.525262, fax: +44.1908.525260, terry.orchard@bespak.co.uk