Uncertainty evaluation in proficiency testing: state-of-the-art, challenges, and perspectives

The evaluation of measurement uncertainty, and that of uncertainty statements of participating laboratories will be a challenge to be met in the coming years. The publication of ISO 17025 has led to the situation that testing laboratories should, to a certain extent, meet the same requirements regarding measurement uncertainty and traceability. As a consequence, proficiency test organizers should deal with the issues measurement uncertainty and traceability as well. Two common statistical models used in proficiency testing are revisited to explore the options to include the evaluation of the measurement uncertainty of the PTRV (proficiency test reference value). Furthermore, the use of this PTRV and its uncertainty estimate for assessing the uncertainty statements of the participants for the two models will be discussed. It is concluded that in analogy to Key Comparisons it is feasible to implement proficiency tests in such a way, that the new requirements can be met. Received: 29 September 2000 Accepted: 3 December 2000     

Draft radioanalytical water and soil performance testing: Uncertainty acceptance criteria derived from historical mixed-analyte performance evaluation program results

Radioanalytical performance testing programs provide radioassay laboratories, regulators and the public performance-based evidence that measurement capabilities are in control. Performance acceptance criteria that combine aspects of measurements difference from a known value and the associated combined uncertainty establishes a quantitative statement of statistical confidence. However, there is need for a reasonable upper limit of the reported measurement uncertainty. Evaluation of thousands of historical U.S. DOE Mixed-Analyte Performance Evaluation Program measurement results for 17 radionuclides in soil and water samples provides predictive expectations for future measurement results and a statistical basis for establishing reasonable upper limits for reported measurement uncertainties for performance evaluation programs.     

Using uncertainty functions to predict and specify the performance of analytical methods

In both European legislation relating to the testing of food and the recommendations of the Codex Alimentarius Commission, there is a movement away from specifying particular analytical methods towards specifying performance criteria to which any methods used must adhere. This ‘criteria approach’ has hitherto been based on the features traditionally used to describe analytical performance. This paper proposes replacing the traditional features, namely accuracy, applicability, detection limit and limit of determination, linearity, precision, recovery, selectivity and sensitivity, with a single specification, the uncertainty function, which tells us how the uncertainty varies with concentration. The uncertainty function can be used in two ways, either as a ‘fitness function’, which describes the uncertainty that is fit for purpose, or as a ‘characteristic function’ that describes the performance of a defined method applied to a defined range of test materials. Analytical chemists reporting the outcome of method validations are encouraged to do so in future in terms of the uncertainty function. When no uncertainty function is available, existing traditional information can be used to define one that is suitable for ‘off-the-shelf’ method selection. Some illustrative examples of the use of these functions in methods selection are appended.     

Measurement of near zero concentration: recording and reporting results that fall close to or below the detection limit

Issues relating to the recording and reporting of analytical data obtained where the concentration of analyte is around or below the detection limit are discussed. The following recommendations are proposed. Analytical results should be recorded by the analyst exactly as they occur, including any negative results, and such records retained for an appropriate length of time. For the purposes of quality assurance in the laboratory (including method validation, internal quality control, and proficiency testing), negative results should be used as they stand. Analytical results reported to a customer should be accompanied by a statement of uncertainty including, in the present context, uncertainty at low concentrations of analyte. The method of editing of reported results must be a contractual matter between the analyst and the customer, but a statement of the procedure used should accompany the results and should be explicit. Normally such editing should be restricted to setting negative results to zero. The customer should be encouraged to pass on the statement to all end users. Data intended for the public domain should be accompanied by a statement detailing the uncertainty, the method of editing, and the location of the unedited data. Most types of statistical processing of datasets containing low concentrations of analyte should be undertaken on the unedited data.     

Interlaboratory comparison of test results of an ordinal or nominal binary property: analysis of variation

Applications of a new statistical method Ordinal Analysis of Variance (ORDANOVA) for interlaboratory comparisons of measurement or test results of semi-quantitative (ordinal) and qualitative (binary) properties are discussed. ORDANOVA can be helpful for validation of measurement or test methods, proficiency testing of laboratories, development of reference materials with certified semi-quantitative and qualitative properties, that is, probably in every field where ANOVA is applied for quantitative properties. A statistics and criteria are proposed for performance assessment of laboratories active in semi-quantitative and qualitative testing and for other purposes of statistical analysis of such test results.     

Homogeneity testing of reference materials

 Homogeneity testing is of the highest importance for the certification of reference materials, as it should demonstrate the validity of the certified values and their uncertainties in the analysis of individual units or portions thereof. However, the conclusions drawn from the results of these studies may often be questioned. It is proposed to improve this situation by quantifying the effect of homogeneity studies in terms of their impact on the uncertainty of certified values. Here it should be noted that the between-units variability directly affects the certified uncertainty, whereas the within-unit inhomogeneity only defines the minimum representative test portion. Received: 18 September 1997 · Accepted: 21 November 1997     

New tools: Expert systems for uncertainty budgets

An important part of quality assurance in any analytical laboratory is the production of comprehensive results integrating uncertainty measurements. Many testing laboratories face the problem that the expenditure required to evaluate small uncertainties (high precision and high accuracy) is often uneconomic. In most cases an uncertainty of high reliability has to be calculated from only a few data (one calibration, few replications, etc.). This problem can be solved by an expert system. To achieve this the analytical procedure has to be structured into a dialouge and divided into parts. The uncertainty has to be calculated for each part of the procedure. Addition of the individual uncertainties results in the combined and expanded uncertainty. During the dialouge the system should advise the analyst how to get an efficient and effective calculation of uncertainty. All calculations, mathematical and statistical procedures have to be surveyable but running the system should not be too time consuming for economic reasons. Within the scope of the EURECA-project initiated by the Eidgenössische Materialprüfungs- und Forschungsanstalt (EMPA), St. Gallen, Switzerland, expert system software is being developed in cooperation with other research institutes and manufacturers of analytical instruments. Using this software it will be possible to calculate the uncertainty for analytical procedures such as titration, atomic emission spectrometry (ICP-OES), atomic absorption spectrometry (AAS) and gas- and liquid chromatography (GC, HPLC).     

The relationships between phenolic content, pollen diversity, physicochemical information and radical scavenging activity in honey

Honey is rich in different secondary plant metabolites acting as natural antioxidants and contributing to human health. Radical scavenging activity (RSA) is related to antioxidant activity, while the correlation between the phenolic content and RSA is often weak. Consequently, exclusive information on phenolics is often insufficient to qualify the RSA and the health promoting effects of honey. The paper deals with a case study of honey samples originating from the alpine areas of Italy's Lombardia and Veneto regions and realized by standard physicochemical and statistical analytical methods. In pure honey, the total phenolic content and the RSA were measured in spectrophotometric tests with the 2,2-diphenyl-1-picrylhydrazyl (DPPH·) free radical and Folin-Ciocalteu assays, respectively. Melissopalynological data was used to qualify pollen diversity through rank-frequency curves separating the samples into two groups. On the basis of physicochemical data, the samples were analyzed through multivariate classification and ranking procedures resulting in the identification of an outlier. Elimination of the outlier produced a high correlation between the total phenolic content and RSA in the two pollen diversity groups. The case study suggests that, after disregarding outliers, the RSA activity can be satisfactorily qualified on the basis of phenolics with pollen diversity as a covariate.     

田口方法在X射线荧光分析时曲线校正中的应用

根据田口玄一博士在测量工程学中提出的信噪比概念,计算了X荧光分析中某一标准曲线极限信噪比。当日常测试中系统的信噪比大于极限值时,不需要对曲线进行校正;当日常测试中系统的信噪比小于极限值时,应对曲线进行校正．以保证该测量系统工作的稳定性。     

The correlation coefficient attacks again

The use of the correlation coefficient for testing the linearity of calibration curves is performed according to the ANOVA checking of the lack-of-fit. The procedure is illustrated from a case study.Papers published in this section do not necessarily reflect the opinion of the Editors,the Editorial Board and the Publisher. Apart from exceptional circumstances, they are not submitted to the usual referee procedure and go essentially unaltered.     

Uncertainty in the quantification of pentachlorophenol in wood processing wastewaters by SPME-GC-MS

Pentachlorophenol is a trace contaminant with toxicological effects in environment. When dealing with the analysis and quantification of trace amounts in complex matrices, procedures such as extraction/pre-concentration are often needed and chromatographic determination are often operated close to the limit of detection, which contribute to the increase of the uncertainty of the measurement. This matter is of crucial importance when the results obtained approach the legal limits and is frequently ignored in scientific work. The aim of this study was the estimation of the global uncertainty associated to the determination of pentachlorophenol (PCP) in aqueous samples, by gas chromatography with mass spectrometric detection (GC-MS), after solid-phase microextraction (SPME). It was concluded that, while in the range of 5 to 40 μg/L the uncertainty did not exceed 20%, in the vicinity of the limit of detection (0.75 μg/L) it raised up to 64%.     

Testing green coffee for ochratoxin A, part III: performance of ochratoxin A sampling plan

Green coffee shipments are often inspected for ochratoxin A (OTA) and classified into good or bad categories depending on whether the OTA estimates are above or below a defined regulatory limit. Because of the uncertainty associated with the sampling, sample preparation, and analytical steps of an OTA test procedure, some shipments of green coffee will be misclassified. The misclassification of lots leads to some good lots being rejected (sellers' risk) and some bad lots being accepted (buyers' risk) by an OTA sampling plan. Reducing the uncertainty of an OTA test procedure and using an accept/reject limit less than the regulatory limit can reduce the magnitude of one or both risks. The uncertainty of the OTA test procedure is most effectively reduced by increasing sample size (or increasing the number of samples analyzed), because the sampling step is the largest source of uncertainty in the OTA test procedure. The effects of increasing sample size and changing the sample accept/reject limit relative to the regulatory limit on the performance of OTA sampling plans for green coffee were investigated. For a given accept/reject limit of 5 microg/kg, increasing sample size increased the percentage of lots accepted at concentrations below the regulatory limit and increased the percentage of lots rejected at concentrations above the regulatory limit. As a result, increasing sample size reduced both the number of good lots rejected (sellers' risk) and the number of bad lots accepted (buyers' risk). For a given sample size (1 kg), decreasing the sample accept/reject limit from 5 to 2 microg/kg relative to a fixed regulatory limit of 5 microg/kg decreased the percentage of lots accepted and increased the percentage of lots rejected at all OTA concentrations. As a result, decreasing the accept/reject limit below the regulatory limit increased the number of good lots rejected (sellers' risk), but decreased the number of bad lots accepted (buyers' risk).     

About acceptance and rejection zones as defined in the EURACHEM/CITAC Guide (2007) “Use of uncertainty information in compliance assessment”

Considering measurement uncertainty is mandatory in assessing conformance to legal or compositional limits, and specific guidelines are available issued by ASME, ISO and Eurachem/CITAC. However, differences between ISO and EURACHEM/CITAC wordings could induce some perplexities in the most careful readers. Possible problems arise from considering that, before performing a test, it should be decided whether it is to be a test for conformity or a test for non-conformity. This choice could perhaps require some renaming of acceptance/rejection zones as defined in the EURACHEM/CITAC Guide. A tentative solution is discussed in this contribution.     

Usage of the uncertainty of measurement by accredited calibration laboratories when stating compliance

First, concepts to state compliance with specifications are presented when taking into account the uncertainty of measurement. Then, the methods used by accredited calibration laboratories, especially within the Deutscher Kalibrierdienst (DKD), are introduced and compared with these methods and concepts. Compliance can only be stated with some probability (risk), which can be calculated by integration, if the measurement results and its probability density functions are known. A scheme for the calculation of the risks by the Monte Carlo method leads to zones of correct and false acceptance, and correct and false rejection. The user of the device under test or calibration has to decide which risk is acceptable. A statement of compliance should only be made if the measurement capability index is not smaller than about 2. The correspondence of probabilities of compliance and guard bands is shown. The rules of ISO 14253-1 and DKD-5 are equivalent to the usage of a guard band of special width. Finally, the performance criteria for comparisons in calibration are treated briefly. Their use has some similarity to the decisions made on compliance. Presented at the Measurement Uncertainty Symposium, Berlin, Germany, April 2008.     

The evaluation of the scoring systems: the fixed effects model under known variances

A number of scoring systems for proficiency testing and interlaboratory comparison are in use by the metrology community. The choice of scoring system for a given study is often based on the study coordinator’s experience and anecdotal knowledge, perhaps attributable to a historic lack of detailed and formal explanation about the foundation of these systems. This has influenced the development of new scoring systems, some of them departing from the well-established hypothesis testing theory. Often, different scoring systems give different results not because one may be better than the others but because, as they are documented, the user cannot control the confidence level of the test. We present a formal evaluation of seven of these systems under the fixed effects model assuming known variances. Under these sound assumptions, the systems analyzed all have the same statistical properties. Furthermore, these systems are all members of a family of systems based on strictly increasing functions in which the statistical decision problem is invariant. Under the fixed effects model with known variances, no unbiased scoring system can provide greater statistical power than the members of this family of systems. We apply these results to the lead content of water example provided in International Standard ISO 13528:2015 “Statistical methods for use in proficiency testing by interlaboratory comparisons.”     

Global test for metabolic pathway differences between conditions

In many metabolomics applications there is a need to compare metabolite levels between different conditions, e.g., case versus control. There exist many statistical methods to perform such comparisons but only few of these explicitly take into account the fact that metabolites are connected in pathways or modules. Such a priori information on pathway structure can alleviate problems in, e.g., testing on individual metabolite level. In gene-expression analysis, Goeman's global test is used to this extent to determine whether a group of genes has a different expression pattern under changed conditions. We examined if this test can be generalized to metabolomics data. The goal is to determine if the behavior of a group of metabolites, belonging to the same pathway, is significantly related to a particular outcome of interest, e.g., case/control or environmental conditions. The results show that the global test can indeed be used in such situations. This is illustrated with extensive intracellular metabolomics data from Escherichia coli and Saccharomyces cerevisiae under different environmental conditions.     

Measurement uncertainty

Measurement uncertainty is a statistical parameter which describes the possible fluctuations of the result of a measurement. It is not a mere repeatability but it is at least as high as the intra-laboratory reproducibility. If it is an attribute of a general analytical test procedure it is at least as high as the inter-laboratory reproducibility. Measurement uncertainty can be determined by the addition of the variances of the individual steps of the test procedure or by an approach which starts with one of the above-mentioned reproducibilities. Any measurement uncertainty should be kept low but it is objectionable to state too low a value, e.g. by falsely reporting mere repeatability data instead of properly determined uncertainty data. Some good working principles can help to obtain low measurement uncertainties.     

A case study in the practical estimation of measurement uncertainty

This case study is written for analytical laboratories, in order to give support to the implementation of the concept of measurement uncertainty for routine measurements. The aim is to provide a practical, understandable and common way of performing measurement uncertainty calculations, based mainly on pre-existing quality control and validation data. Practical examples taken directly from environmental laboratory monitoring are presented and explained. However, the approach is very general and should be applicable to most testing laboratories in the chemical field. Following the protocol of evaluation illustrated in the case study, it is possible to ensure that most relevant uncertainty components associated with the method are covered. Contributions associated with sampling, homogenisation, sub-sampling, and so on, are, however, excluded.     

A practitioner's approach to the assessment of the quality of sampling, sample preparation, and subsampling

 The methodology of evaluating the performance of sampling, sample preparation, and subsampling is reviewed. The requirements to be set for a successful experiment are revisited. The central role of the reference method is explained, and so is the choice of the parameters and the measurement methods. Based on the principles of the "Guide to the expression of uncertainty in measurement" (GUM), a statistical model is developed that demonstrates the influence of the experimental design on the outcome of the assessment experiment. This relationship is often overlooked in practice, as it is hardly mentioned in written standards dealing with this kind of quality assessments. The statistical framework thus developed covers the statistical procedures commonly appearing in written standards. Finally, the issue of testing the significance of the bias obtained from the experiment is discussed. Received: 14 June 1997 · Accepted: 2 September 1997