Out-of-Trend Identification and Removal in Stability Modelling and Regression Analysis

Outliers in regression modelling may cause incorrect and invalid results to occur when predicting stability. A clearly defined out-of-trend (OOT) protocol is needed to correctly and consistently identify and remove outliers from expiry and stability modelling and prediction where technically warranted. OOT is a point (measurement) in a regression analysis that has statistically greater error at a defined risk factor from a regression line or multiple-factor regression model than other determinations; it is a time-dependent result that falls outside a predicted statistical interval (see Figure 1). Simply put, an OOT event is an outlier in a regression analysis. This article provides an overview of OOT from literature, guidance, and best analytical practice and the procedure to be used during stability modelling and analysis.  

Figure 1: Out of trend (OOT) illustration and influence of OOT.

Figure 1: Out of trend (OOT) illustration and influence of OOT.

OOT points are considered to be non-representative of the test sample and are due to analytical, transcription, or other sources of error. Failure to remove OOT point(s) if they exist will produce calculated rates of change that will not be representative of the drug product nor drug substance.  The following are indications that OOTs are present in the stability analysis:

• Points do not line up on the regression line

• Confidence intervals of the fit are excessively wide

• Root-mean-squared error (RMSE) of the residuals has excessively expanded well beyond the characterized analytical error

• Expiry from one time point to another has a large amount of difference

• R2 has a large amount of change with and without the OOT time point.

“OOT stability data can be described as a result or sequence of results that are within specification limits but are unexpected, given the typical analytical and sampling variation and a measured characteristic’s normal change over time (e.g., an increase in degradation product on stability)” (1).

Regression analysis is normally used to determine change over time and associated 95% confidence limits relative to rates of change and expiry. International Council on Harmonization (ICH) Q1A(R2) Stability Testing of New Drug Substances and Product (2) states, “the nature of any degradation relationship will determine whether the data should be transformed for linear regression analysis. Usually the relationship can be represented by a linear, quadratic, or cubic function on an arithmetic or logarithmic scale. Statistical methods should be employed to test the goodness of fit of the data on all batches and combined batches (where appropriate) to the assumed degradation line or curve.”

OOT evaluation and elimination should be used for the following applications and prediction:

• Shelf-life estimation

• Storage evaluation

• Impurity formation and trending

• In-process monitoring and prediction

• Tracking and trending of lot performance.

Likely Root Causes for OOT
The following are typical possible sources and mechanisms for OOT events that may occur during stability evaluation:

• Sample selection and sample handling errors

• Dilution and sample-preparation errors

• Sample materials and plate errors

• Temperature, reaction time, and pH effects errors

• Vendor and lot variation on critical reagents

• Flow rates and process time errors

• Analyst errors

• Instrument variation and calibration error

• Nonstandard test procedures and not following the method standard operating procedure (SOP)

• Drift in standards or reference materials

• Stability of test samples or critical reagents

• Calibration or compensation errors

• Interaction and composite errors

• Expiry of bulk materials.

Historical Approaches to OOT
The following are typical historical approaches to OOT identification and removal, though they are not recommended approaches. They are not considered to be statistically sound procedures for OOT identification and removal. Using procedures that are not statistically sound may remove time points from a stability analysis that cannot be defended nor justified upon review. There is no statistical basis for the following definitions of OOT and these do not take into account process nor method variation:

• The difference between consecutive results is outside of half the difference between the prior result and the specification

• The result is outside ±5% of initial result

• The result is outside ±3% of previous result

• The result is outside ±5% of the mean of all previous results.

Closed-Loop Approach to OOT identification and removal
A best-practice approach to OOT determination and removal is to see it as a part of a closed-loop control system during stability monitoring and expiry prediction (see Figure 2). The five steps to a closed loop system for OOT are:

• Addition of new time points and data

• OOT identification

• OOT determination and point removal where warranted

• OOT verification and evaluation of OOT influence

• Stability and performance prediction.

Figure 2: Closed loop out of trend (OOT) identification and resolution.

Figure 2: Closed loop out of trend (OOT) identification and resolution.

Addition of New Time Points and Data, Closed Loop Step 1
As each new time point is added to the stability analysis, the time point should be checked for OOT potential. If they are within the criteria for OOT identification, then rates of change, expiry, etc. are determined. OOT identification, determination, and verification are used if new time points appear to be suspect.

OOT Identification, Closed Loop Step 2
Some CMC teams recommend the percent change from point-to-point as well as from the initial time point to indicate an OOT (3). For example, more than a 5% change from baseline may be considered a possible OOT event (4).

Analytically, there are four methods to identify a point as OOT: visually, outlier boxplot of the residuals, multivariate Jackknife distances, and control chart of the residuals (see Figures 3-6). Jackknife distances are the most sensitive in identification of OOT points in a regression analysis as they include and remove each time point in the analysis to evaluate their influence in the model. Once an OOT has been identified, the next step is to test it to determine if the OOT will be removed.  Root cause as to why the point is OOT is a secondary investigation once the point has been determined to be OOT. Once the point is determined to be a possible OOT, the point is tested statistically to determine if it is or is not OOT.

Figure 3: Visual analysis.

Figure 3: Visual analysis.

Figure 4: Outlier box plot of residuals.

Figure 4: Outlier box plot of residuals.

OOT Determination, Closed Loop Step 3
The following is the recommended procedure for OOT outlier determination (see Figures 7 and 8):

• Exclude and hide the suspected OOT point in the data analysis.  

• Fit a linear regression line with the potential OOT time point excluded.  

• Save the predicted response (concentration) from the linear fit.

• Calculate the difference (Delta) at each time point.  

• Calculate a z score for each time point (see Equation 1).

z score= (Measurement-predicted)/stdev(residuals with point removed)
[Eq. 1]

Figure 6: Control chart of residuals.

Figure 5: Multivariate Jackknife distances.

Once the z score for the OOT has been calculated, it can be compared with a risk threshold.  A k-sigma of 2.576 (or 99% risk) is used to set the limit for OOT detection. Abs(z)> 2.576 (99%) is OOT, so in this example, z is -27.238; therefore, it is OOT. A z score with a limit is the best method of OOT detection. The key difference with this procedure and other z score procedures written in literature is the z score is evaluated with the point removed. This correctly scales the residual error so that the influence of the OOT point is not included in the residual error and the OOT time point can be correctly evaluated based on the other measurements error.
 

OOT Verification
To verify the influence of the OOT, the following measures are recommended:

• Change in R2

• Change in RMSE

• Change in expiry calculation.  

Comparing with or without the OOT time point will verify the influence of the time point and confirm the need for removal. If the change in the three identified measures is trivial, then the OOT has not been verified and its removal is not warranted. Differences in R2, RMSE, or expiry of 3% or less are generally not practically important to drug-substance or drug-product expiry or stability evaluation. Verification is performed by including, then removing, the OOT point in the stability evaluation and then measuring the change in the key performance metrics of the fit and the prediction. Also, RMSE error can be compared to the repeatability of the analytical method to determine if the residual error is primarily due to analytical measurement error.

Figure 7: Out of trend (OOT) determination.

Figure 7: Out of trend (OOT) determination.

Figure 8: Stability analysis with out of trend (OOT) removed.

Figure 8: Stability analysis with out of trend (OOT) removed.
A control chart of the residuals with the OOT time point excluded will be a secondary confirmation of OOT identification and removal as an outlier (see Figure 9). Residual points from the regression fit are plotted onto a control chart, points that are within the control limit may be due to random or analytical method variation, points outside of the limits confirm points are likely to be OOT.  Remember to exclude the OOT point when building the chart so it does not influence the limits.

Figure 9: Control chart of residuals.

Figure 9: Control chart of residuals.

Stability Prediction
Once the OOT point has been removed and verified, stability prediction can then be performed per ICH Q1E (5). OOT points should always be included in the plot to indicate the measurement but tagged as OOT to indicate the point was included in graph but not included in the analysis. Including it in the plots will provide full disclosure as to all observations at all time points. Once a point has been determined to be OOT, it does not render the analysis as additional data are added to the stability prediction.

Single Factor, Single Batch,and Multiple-Factor/Multiple-Batch OOT
The technique for identifying and removing an OOT point described in this paper works equally well for single-batch/single-factor versus multiple-batch/multiple-factor stability modelling and expiry prediction. The example provided was for a single-batch, single-factor analysis. For the multiple-factor and/or multiple-batch condition, the same approach would be used. The model would be fit with the OOT removed and then the model would be saved, z scores would be calculated for all time points, and then OOT would be determined based on a 2.576 (99%) threshold.

Conclusion
Having a well-defined and statistically valid OOT protocol is a powerful addition to any stability program and needs a clearly defined SOP to consistently apply the logic to day-to-day stability evaluation. Including an OOT protocol to stability testing and data analysis will produce more statistically reliable stability determination and expiry prediction. OOT determination based on the protocol described in this paper opens the door for the automation of OOT determination and removal from any stability analysis and may be the best approach to systematically evaluate all time points in stability analysis.

References
1. PhRMA CMC Statistics and Stability Expert Teams, “A Review of the Potential Regulatory Issue and Various Approaches,” Pharmaceutical Technology 27 (2003).
2. ICH, Q1A(R2) Stability Testing Of New Drug Substances (ICH, February 2003).
3. T-J, Torbovska, “Method for Identification of Out-of-Trend Stability Results,” Pharmaceutical Technology 37 (2003).  
4. PhRMA CMC Statistics and Stability Expert Teams, “Identification of Out-of-Trend Stability Results, Part II,”  Pharmaceutical Technology (Oct 2, 2005).
5. ICH, Q1E Evaluation for Stability Data (ICH, February 2003).

Article DetailsBioPharm International
Vol. 29, No. 1
Pages: 50–55

Citation
When referring to this article, please cite it as T. Little, "Out-of-Trend Identification and Removal in Stability Modelling and Regression Analysis," BioPharm International29 (1) 2016.