Data Integrity Matters | What do you do with Orphan Data Created during Data Acquisition?

Halting a submitted chromatographic Sample Set is more than pressing ‘stop’

For analytical laboratories, a Sample Set or Sequence is a means of submitting multiple samples for analysis into an instrument. For many techniques, including chromatography where “relative” quantitation is the only possible method of calculation, it is essential that samples are submitted along with the appropriate known concentration standards in the same sequence.

Chromatographic runs may also include a series of System Suitability samples at the beginning of the run, to be sure that the instrument is capable of creating accurate results.

When everything goes according to plan, once you have run System Readiness or Equilibration injections to ensure the system is equilibrated and ready, most laboratories would submit the entire Sample Set to acquire and store the data for later processing. Less likely is that the Sample Set would be set to Run and Process, or Run, Process and Report, although this may be perfectly feasible in laboratories with simple and very robust separation methods.

Run-only is the most generally used mode, such that all the data is collected and saved first, allowing the analyst to ensure that the default integration method is suitable for this specific run, or to optimize the processing parameters first before creating and saving accurately integrated results.

This approach has a disadvantage in that you have committed to run the entire sequence — but have yet to gather information about the System’s Suitability for the analysis.

Another common approach is to first run the System Suitability samples (either as a separate sequence or by running just selected entries at the beginning of the Sample Set), process the System Suitability samples, and evaluate the performance of the system before submitting the samples for analysis. If the System Suitability injections indicate that the instrument is not performing, it would be foolhardy to submit samples that you know will need to be invalidated, regardless of test result (i.e., in spec or out of spec). Many laboratories will require approval of the system performance before allowing the remaining standards and samples to be analyzed.

To assure only quality results are created such a phased approach — performing separation and data acquisition, only if initial System Suitability passes — would seem a sensible approach. However, sometimes tests for System Suitability will be acquired during the entire run by interspersing known control solutions between the samples and standards. In this situation, only when the run is completed and the System Suitability injections are evaluated, can you be sure the chromatographic system is suitable for use.

Choosing to stop a run

Halting a chromatographic analysis where the system is not performing correctly or when another error is evident in the separation is a valuable and scientific way to avoid creating unusable data that needs to be subsequently invalidated following a defined analytical lab error /anomalous result SOP.

A valid reason to halt a run might be to avoid clearly obvious errors; e.g., the standards do not have the required numbers of peaks, the analyst noted that vials were placed out of order, additional urgent samples need to be added to the sequence and the sequence restarted, or the wrong detection method was chosen. Most Chromatography Data Systems will have a means to abort a run, and may have the opportunity for the analyst to note a reason for that. This action will be recorded in the data records, the audit trail, or in other log/notification tools. It may require noting in an instrument log book, a deviation document, or in a laboratory book. Critical pieces of evidence are the data “collected so far” since obvious errors or mistakes in these partially completed results will assist in justifying the decision to end the run.

However, sequences may also end without human intervention due to errors in the instrument, tripping a pressure alarm, a leak in the system, a communication failure, a lamp failure, or a power outage. These would not be initiated by the user, and so may not be recorded in the same way that a user action would be, with the opportunity to document a reason. Depending on how the run was stopped, where in the sequence, and what needs to be done to reactivate it, the laboratory will have a written procedure about either restarting from the point of failure, or repeating the samples from the beginning, including whether to repeat the System Suitability testing.

Warning signs

Quality auditors, however, may have concerns about the uncontrolled and undocumented stopping of a chromatographic run before it has completed, as was seen in the below FDA’s inspection observations:

1. Failure to adequately investigate and document out-of-specification results according to a procedure
2. C) “Your OOS investigation procedure 036/—/QS/QA permits an analyst to abort a chromatographic run if an apparent OOS is observed prior to completing analysis of all samples scheduled to be injected in the sequence. Your quality control (QC) manager confirmed that analysts abort HPLC analyses if they “expect to invalidate” them later for an assignable cause. For example, you aborted the HPLC sequence of (b)(4)API batch (b)(4)while observing the chromatographic run on the screen (“online monitoring”) in which an individual unknown impurity tested at (b)(4)% (specification: NMT (b)(4)%). There was no machine malfunction (e.g., unstable system) that would justify aborting the automated analysis. 

Our investigators documented approximately 248 instances of aborted sequences.

The observation begins by noting entirely correctly that producing a result that does not meet specification (an OOS result) is certainly no reason to end the analysis. However, the confirmation from the QC manager would be an appropriate response, if there really was a reason to believe the data could truly not be deemed accurate and would need to be invalidated with an assignable cause. In this case, the curveball is that one of the results in that sequence (once the data was processed) was seen to be Out of Specification.  This should still be non-consequential if the reason for stopping the analysis was bona fide, documented, and independent of the result. In fact, if there was an assignable error in the analysis, then Out of Spec results may be an expected outcome entirely due to the error.

The challenge here is that we are looking at two distinct and separate procedures:

• Aborting an automated analysis for an observed error
• OOS investigations procedure

Any action attempting to ignore or hide an OOS result raises suspicion about a human “motive” or bias for stopping the analysis. As mentioned before, the critical piece of evidence is in the data “collected so far,” the partially completed raw data or results, and potentially the metadata and audit trails, which should clearly be able to support the documented decision to end the run.

The inspection report’s observation goes on to note:

Your SOP was inadequate.

When performing a sample preparation, it may be possible to identify an obvious manual error at the time of the mistake. In such a limited instance, it can be appropriate to discontinue the sample preparation, immediately document the deviation, and justify a new sample preparation.

However, it is not appropriate to stop an in-progress automated analysis because of an assumption that an earlier error may be causing an OOS result. Obtaining an unexpected result does not constitute an “assignable cause” and the assumption of such a cause is not a valid basis for interrupting an analysis. The automated analytical sequence should be allowed to proceed to completion, irrespective of the appearance of undesirable analytical results on the computer screen.”

Clearly the decision to halt an analysis must be clear and unbiased, regardless of the disposition of the test results. However, I don’t believe that we would agree with the final sentence — that all automated sequences should be allowed to continue to completion; neither the separation and data acquisition stages, nor insisting that all results (however unscientific they are and with whatever errors are in there), proceed to the final processed result.

Do you keep the data?

Would you take that another step and say that all processed results, regardless of whether they are usable or unusable, should be entered into LIMS or SAP?

When laboratories are working to the U.S. GMP regulation, 21 CFR Part 211.194a clearly says that all data created and all calculations performed should be available, therefore dealing with the data in partially completed analyses should be considered through procedures. If users are continually aborting runs for trivial or suspect reasons, the laboratory manager or QA oversight manager should address this.

But in many non-regulated analytical laboratories — conversely — aborting chromatographic sequences would be considered a normal part of everyday life. So, a blanket ban on aborting analyses is simply not realistic.

I will discuss this further in a later post when we look at the total Invalidation Rate (both OOS and IS results) vs. the Invalidation of OOS Rate alone (IOOSR), one of the FDA’s Quality Metrics initiatives measures.

Read more articles in Heather Longden’s blog series, Data Integrity Matters.