Using target measurement uncertainty to determine fitness for purpose

The methods an analytical laboratory uses must be validated to be fit for purpose. The fitness for purpose of a quantitative method used to determine the concentration of a substance when assessing compliance to requirements can be described by the maximum measurement uncertainty. This is called the target measurement uncertainty. Acceptance criteria for precision and bias in the method validation are then established in terms of the target measurement uncertainty. The target measurement uncertainty can be decided by following a process which involves determining the required concentration range of the measurand; determining the acceptable level of risks of incorrect decisions of compliance; developing a suitable decision rule, with guard bands if appropriate; using the probability of making an incorrect decision of compliance based on the decision rule; and assessing the impact of bias. A key participant in this process is the end user of the data, the laboratory customer. This paper presents the concepts concerning target measurement uncertainty introduced in recently published international guidelines to the practicing analytical chemist who is not generally familiar with these concepts. Three examples are used to illustrate the process.     

Principles of the EURACHEM/CITAC guide “Use of uncertainty information in compliance assessment”

Many, possibly most, analytical measurements are carried out to assess compliance with a specification or a regulation, for example in the control contaminants in food or the detection of performance enhancing substances in sport. When making an assessment of compliance the presence of unavoidable measurement uncertainty introduces the risk of making incorrect decisions, that is of accepting a batch of material which is outside the specification or rejecting one that is within. This often leads to controversy over whether or not the compliance decision is correct. How to make reliable assessment decisions is described in the EURACHEM/CITAC Guide “Use of uncertainty information in compliance assessment”. The key is the use of decision rules that lead to an unambiguous interpretation of the measurement result and its uncertainty. These decision rules need to be designed to ensure that requirements of the specification or regulation are met and that the risk of making an incorrect decision is acceptable. Ideally they should form part of the specification or regulation. Presented at the Measurement Uncertainty Symposium, April 2008, Berlin, Germany.     

Models to estimate overall analytical measurements uncertainty: assumptions, comparisons and applications

Evaluation of analytical results reliability is of core importance as crucial decisions are taken with them. From the various methodologies to evaluate the fitness of purpose of analytical methods, overall measurement uncertainty estimation is more and more applied. Overall measurement uncertainty allows to combine simultaneously the remaining systematic influences to the random sources of uncertainty and allows assessing the reliability of results generated by analytical methods. However there are various interpretations on how to estimate overall measurement uncertainty, and thus various models for estimating it. Each model together with its assumptions has great impacts on the risks to abusively declare that analytical methods are suitable for their intended purpose. This review paper aims at (i) summarizing the various models used to estimate overall measurement uncertainty, (ii) provide their pros and cons, (iii) review the main areas of application and (iv) as a conclusion provide some recommendations when evaluating overall measurement uncertainty.     

Traceability in perspective

Results that reference SI units rarely pose problems in chemical measurement because traceable standards, with uncertainties derived from a chain of calibrations from the SI, are readily available at the analyst??s bench. These uncertainties are nearly always far smaller than that required for fitness for purpose in the analytical result. Moreover, the greater part of the uncertainty in a typical result is not derived from primary measurements traceable to the SI but from recovery problems and matrix effects. Even so, the incidence of wildly inaccurate results stems not from this uncertainty but from ??blunders??, deviations from the correct procedure. Attention to traceability beyond that employed by any competent analyst therefore cannot reduce the uncertainty. Furthermore, there is no rational reason to reduce the uncertainty if the result is already fit for purpose. The current focus on traceability is distracting analysts from the more pressing task of eliminating blunders.     

Total error and uncertainty: Friends or foes?

Guidelines ISO 17025 and ISO 15189 aim to improve the quality-assurance scheme of laboratories. Reliable analytical results are of central importance due to the critical decisions that are taken with them. ISO 17025 and ISO 15189 therefore require that analytical methods be validated and that laboratories can routinely provide the measurement uncertainty of the results of measurements. To evaluate the fitness of purpose of analytical methods, total error is increasingly applied to assess the reliability of results generated by analytical methods. However, the ISO requirement to estimate measurement uncertainty seems opposed to the concept of total error, leading to delays in laboratories implementing ISO 17025 and ISO 15189 and confusion for the analysts. This article therefore aims to clarify the divergences between total error and measurement uncertainty, but also to discuss their main similarities and emphasize their implementation.     

A global approach to method validation and measurement uncertainty

The enforcement of legal limits for food safety raises the question of decision-making in the context of uncertain measurements. It also puts the question of demonstrating that measurement technique that is used is fit for the purpose of controlling legal limits. A recent European Commision (EC) decision gives some indications how to deal with this question. In the meantime, the implementation of quality systems in analytical laboratories is now a reality. While these requirements deeply modified the organization of the laboratories, it has also improved the quality of the results. The goal of this communication is to describe how two fundamental requirements of ISO 17025 standard, i.e. validation of the methods and estimation of the uncertainty of measurements, can give a way to check whether an analytical method is correctly fit for the purpose of controlling legal limits. Both these requirements are not independent and it will be shown how they can be combined. A recent approach based on the “accuracy profile” of a method was applied to the determination of acrylamide and illustrates how uncertainty can be simply derived from the data collected for validating the method. Moreover, by basing on the β-expectation tolerance interval introduced by Mee [Technometrics (1984) 26(3): 251–253], it is possible to unambiguously demonstrate the fitness for purpose of a method. Remembering that the expression of uncertainty of the measurement is also a requirement for accredited laboratories, it is shown that the uncertainty can be easily related to the trueness and precision issuing from the data collected to build the method accuracy profile. The example presented here consists in validating a method for the determination of acrylamide in pig plasma by liquid chromatography–mass spectromery (LC–MS). Concentrations are expressed as mg/l and instrumental response is peak surface. The calibration experimental design included 5×5×2 measurements and namely consisted in preparing duplicate standard solutions at five concentration levels ranging from 10 to about 5000 mg/l. This was repeated for 5 days. The validation experimental design was similar.     

Multi-analyte optimisation of uncertainty in infant food analysis

The Optimised Uncertainty (OU) methodology has been developed to optimise multi-analyte situations. It has then been applied to a retail survey of infant food for trace elements, classifying the food as compliant or non-compliant with the regulatory thresholds or specification limits that are appropriate for each element. The large-scale survey of infant foods was successfully adapted to allow the estimation of uncertainties, from both primary sampling and chemical analysis, for elemental concentrations in infant formula (milk) and wet meals. The analytes included in this investigation comprised both contaminants (Pb and Cd) and elements essential for child development (Zn and Cu). Optimisation of the measurement process for a 'single analyte' demonstrated the potential financial benefits of optimising future surveys for a false compliance scenario. Uncertainty estimates for the measurement of elemental concentrations in infant formula were dominated by uncertainty from the analytical method. Large potential savings (up to pounds 575,000 per batch) are predicted for both Pb and Zn by increasing the expenditure on chemical analysis to the optimal level. In comparison the uncertainty estimates for elemental concentration in wet meals showed a dominance of sampling as a source of uncertainty for Cd and Cu due to the increased heterogeneity. The feasibility of 'multi-analyte' optimisation is demonstrated for the case study of infant milk. Single analyte optimisation of the four analytes for a false compliance scenario indicated a decrease in expectations of financial loss of between 99% and 8%. An overall decrease in the total expectation of financial loss of 99% is indicated following multi-analyte optimisation.     

Optimised uncertainty in food analysis: application and comparison between four contrasting 'analyte-commodity' combinations

The optimised uncertainty (OU) methodology is applied across a range of analyte-commodity combinations. The commodities and respective analytes under investigation were chosen to encompass a range of input factors: measurement costs (sampling and analytical), sampling uncertainties, analytical uncertainties and potential consequence costs which may be incurred as a result of misclassification. Two types of misclassification are identified-false compliance and false non-compliance. These terms can be used across a wide range of foodstuffs that have regulations requiring either minimum compositional requirements, maximum contaminant allowances or compositional specifications. The latter refers to foodstuffs with regulations that state an allowable tolerance around the compositional specification, i.e. the upper specification limit (USL) and the lower specification limit (LSL). The traditional OU methodology has been adapted so that it is applicable in these cases and has been successfully applied in practice. The Newton-Raphson method has been used to determine the optimal uncertainty value for the two case studies in which analyte concentration is assessed against a 'single threshold' regulatory requirement. This numerical method was shown to give a value of the optimal uncertainty that is practically identical to that given by the previously used method of visual inspection. The expectation of financial loss was reduced by an average of 65% over the four commodities by the application of the OU methodology, showing the benefit of the method.     

Do we need to consider metrological meanings of different measurement uncertainty estimations?

Along the years, several approaches for measurement uncertainty estimation have been suggested. Emphasis has been put on the general metrological interpretation of measurement uncertainty, but not on its different meanings when it is associated to given conditions of measurement where analytical work is performed and errors are originated. Three different definitions for uncertainty are proposed for reproducibility and intermediate precision conditions of measurement. These definitions inherit features from the VIM 3 definition of measurement uncertainty. It is argued that if a high performance laboratory keeps errors under control with proper validation and quality assurance programs, measurement uncertainty from intermediate precision condition of measurement is justified as a suitable estimation of its capability to attribute values to a measurand. Alternatively, a laboratory that does not keep errors under control should use uncertainty from reproducibility condition of measurement as the cost of its imperfections. Selection of information sources for measurement uncertainty estimation should be in harmony with its metrological meaning.     

Traceability and uncertainty – A comparison of their application in chemical and physical measurement

 Establishment of the traceability and the evaluation of the uncertainty of the result of a measurement are essential in order to establish its comparability and fitness for purpose. There are both similarities and differences in the way that the concepts of traceability and uncertainty have been utilised in physical and chemical measurement. The International Committee of Weights and Measures (CIPM) have only in the last decade set up programmes in chemical metrology similar to those that have been in existence for physical metrology for over a century. However, analytical chemists over that same period have also developed techniques, based on the concepts of traceability and uncertainty, to ensure that their results are comparable and fit for purpose. This paper contrasts these developments in physical and chemical metrology and identifies areas where these two disciplines can learn from each other.     

Do we really need to account for run bias when producing analytical results with stated uncertainty? Comment on 'Treatment of bias in estimating measurement uncertainty' by G. E. O'Donnell and D. B. Hibbert

Treatment of bias is an important issue relating to analytical quality. Recently, G. E. O'Donnell and D. B. Hibbert (Analyst, 2005, 130, 721) recommended to always correct analytical results for 'run bias' determined by a single analysis of a certified reference material (CRM) in each analytical run. In the authors' opinion, this is necessary for the results obtained to be comparable from run to run. It is argued here that such a recommendation is logically inconsistent and stems from misinterpretation of measurement uncertainty as being estimated under repeatability conditions. The fundamental principle underlying the measurement uncertainty methodology is that all relevant sources of error should be taken into account, which results in overall uncertainty assessment and thus provides a means for a global comparability of measurement and test results. The local, i.e. run-to-run, comparability is not a factor if analytical results are interpreted on the basis of their associated uncertainty.     

The assessment of compliance with legal limits. Part 2. How to consider both type I and type II errors when working in the concentration domain

Considering the uncertainty of measurement when assessing compliance with reference values given in compositional specifications and statutory limits is still a controversial matter. In theory, assessing compliance requires considering both type I (false positive) and type II (false negative) errors. The more the concentration of the analyte in the sample under investigation is close to the allowed concentration limit, the more critical it is to consider both types of errors. This paper describes how this could be done. The matter is discussed in the light of the most recent literature information.     

Quantifying the measurement uncertainty of results from environmental analytical methods

The Eurachem-CITAC Guide Quantifying Uncertainty in Analytical Measurement was put into practice in a public laboratory devoted to environmental analytical measurements. In doing so due regard was given to the provisions of ISO 17025 and an attempt was made to base the entire estimation of measurement uncertainty on available data from the literature or from previously performed validation studies. Most environmental analytical procedures laid down in national or international standards are the result of cooperative efforts and put into effect as part of a compromise between all parties involved, public and private, that also encompasses environmental standards and statutory limits. Central to many procedures is the focus on the measurement of environmental effects rather than on individual chemical species. In this situation it is particularly important to understand the measurement process well enough to produce a realistic uncertainty statement. Environmental analytical methods will be examined as far as necessary, but reference will also be made to analytical methods in general and to physical measurement methods where appropriate. This paper describes ways and means of quantifying uncertainty for frequently practised methods of environmental analysis. It will be shown that operationally defined measurands are no obstacle to the estimation process as described in the Eurachem/CITAC Guide if it is accepted that the dominating component of uncertainty comes from the actual practice of the method as a reproducibility standard deviation.     

Quantifying the measurement uncertainty of results from environmental analytical methods

The Eurachem–CITAC Guide Quantifying Uncertainty in Analytical Measurement was put into practice in a public laboratory devoted to environmental analytical measurements. In doing so due regard was given to the provisions of ISO 17025 and an attempt was made to base the entire estimation of measurement uncertainty on available data from the literature or from previously performed validation studies. Most environmental analytical procedures laid down in national or international standards are the result of cooperative efforts and put into effect as part of a compromise between all parties involved, public and private, that also encompasses environmental standards and statutory limits. Central to many procedures is the focus on the measurement of environmental effects rather than on individual chemical species. In this situation it is particularly important to understand the measurement process well enough to produce a realistic uncertainty statement. Environmental analytical methods will be examined as far as necessary, but reference will also be made to analytical methods in general and to physical measurement methods where appropriate. This paper describes ways and means of quantifying uncertainty for frequently practised methods of environmental analysis. It will be shown that operationally defined measurands are no obstacle to the estimation process as described in the Eurachem/CITAC Guide if it is accepted that the dominating component of uncertainty comes from the actual practice of the method as a reproducibility standard deviation.     

Uncertainty in analytical results from solid materials with electrothermal atomic absorption spectrometry: a comparison of methods

The combined uncertainty in the analytical results of solid materials for two methods (ET-AAS, analysis after prior sample digestion and direct solid sampling) are derived by applying the Guide to the Expression of Uncertainty in Measurement from the International Standards Organization. For the analysis of solid materials, generally, three uncertainty components must be considered: (i) those in the calibration, (ii) those in the unknown sample measurement and (iii) those in the analytical quality control (AQC) process. The expanded uncertainty limits for the content of cadmium and lead from analytical data of biological samples are calculated with the derived statistical estimates. For both methods the expanded uncertainty intervals are generally of similar width, if all sources of uncertainty are included. The relative uncertainty limits for the determination of cadmium range from 6% to 10%, and for the determination of lead they range from 8% to 16%. However, the different uncertainty components contribute to different degrees. Though with the calibration based on reference solutions (digestion method) the respective contribution may be negligible (precision < 3%), the uncertainty from a calibration based directly on a certified reference material (CRM) (solid sampling) may contribute significantly (precision about 10%). In contrast to that, the required AQC measurement (if the calibration is based on reference solutions) contributes an additional uncertainty component, though for the CRM calibration the AQC is “built-in”. For both methods, the uncertainty in the certified content of the CRM, which is used for AQC, must be considered. The estimation of the uncertainty components is shown to be a suitable tool for the experimental design in order to obtain a small uncertainty in the analytical result.     

On the assessment of compliance with legal limits Part 1: signal and concentration domains

Taking into account the uncertainty of measurement when assessing compliance of a given sample with compositional specifications or statutory limits is an open question. Rigorous assessment should be performed within the signal domain, and by considering both α and β errors. Working within the concentration domain through a calibration function, which is affected by additional experimental uncertainties, involves a different degree of uncertainty and can sometimes lead to unreliable verdicts. The matter is discussed, and illustrated with the aid of some worked examples, each using multiple simulated data sets, obtained by adding an arbitrarily chosen Gaussian noise to representative response/concentration functional relationships. Received: 12 October 2000 Accepted: 12 July 2001     

Considering uncertainty of measurement when assessing compliance or non-compliance with reference values given in compositional specifications and statutory limits: a proposal

Considering the uncertainty of measurement (UOM) is mandatory when assessing compliance with reference values given in compositional specifications and statutory limits, but the matter is still open to question. Working in the signal or concentration domains and considering false negative together with false positive errors are the main points of debate. Frequently, the available approaches look too simplified for being accurate or too complex (since more rigorously formulated from a theoretical point of view) for being largely acceptable. In the Authors opinion, assessing compliance with reference values given in compositional specifications and statutory limits is a problem similar to that of estimating the limit of detection. This allows proposing a simple operational approach based on well-known and accepted assumptions and approximations. This proposal, described in the light of the most recent literature information, is aimed to stimulate a critical discussion in view of evaluating possible corrections to the generally accepted approach.Papers published in this section do not necessarily reflect the opinion of the Editors, the Editorial Board and the Publisher.     

Usage of the uncertainty of measurement by accredited calibration laboratories when stating compliance

First, concepts to state compliance with specifications are presented when taking into account the uncertainty of measurement. Then, the methods used by accredited calibration laboratories, especially within the Deutscher Kalibrierdienst (DKD), are introduced and compared with these methods and concepts. Compliance can only be stated with some probability (risk), which can be calculated by integration, if the measurement results and its probability density functions are known. A scheme for the calculation of the risks by the Monte Carlo method leads to zones of correct and false acceptance, and correct and false rejection. The user of the device under test or calibration has to decide which risk is acceptable. A statement of compliance should only be made if the measurement capability index is not smaller than about 2. The correspondence of probabilities of compliance and guard bands is shown. The rules of ISO 14253-1 and DKD-5 are equivalent to the usage of a guard band of special width. Finally, the performance criteria for comparisons in calibration are treated briefly. Their use has some similarity to the decisions made on compliance. Presented at the Measurement Uncertainty Symposium, Berlin, Germany, April 2008.     

Lifetime of the traceability chain in chemical measurement

Since the uncertainty of each link in the traceability chain (measuring analytical instrument, reference material or other measurement standard) changes over the course of time, the chain lifetime is limited. The lifetime in chemical analysis is dependent on the calibration intervals of the measuring equipment and the shelf-life of the certified reference materials (CRMs) used for the calibration of the equipment. It is shown that the ordinary least squares technique, used for treatment of the calibration data, is correct only when uncertainties in the certified values of the measurement standards or CRMs are negligible. If these uncertainties increase (for example, close to the end of the calibration interval or shelf-life), they are able to influence significantly the calibration and measurement results. In such cases regression analysis of the calibration data should take into account that not only the response values are subjects to errors, but also the certified values. As an end-point criterion of the traceability chain destruction, the requirement that the uncertainty of a measurement standard should be a source of less then one-third of the uncertainty in the measurement result is applicable. An example from analytical practice based on the data of interlaboratory comparisons of ethanol determination in beer is discussed. Received: 5 October 2000 Accepted: 3 December 2000     

New advances in method validation and measurement uncertainty aimed at improving the quality of chemical data

The implementation of quality systems in analytical laboratories has now, in general, been achieved. While this requirement significantly modified the way that the laboratories were run, it has also improved the quality of the results. The key idea is to use analytical procedures which produce results that fulfil the users needs and actually help when making decisions. This paper presents the implications of quality systems on the conception and development of an analytical procedure. It introduces the concept of the lifecycle of a method as a model that can be used to organize the selection, development, validation and routine application of a method. It underlines the importance of method validation, and presents a recent approach based on the accuracy profile to illustrate how validation must be fully integrated into the basic design of the method. Thanks to the -expectation tolerance interval introduced by Mee (Technometrics (1984) 26(3):251–253), it is possible to unambiguously demonstrate the fitness for purpose of a new method. Remembering that it is also a requirement for accredited laboratories to express the measurement uncertainty, the authors show that uncertainty can be easily related to the trueness and precision of the data collected when building the method accuracy profile.