Integration of metrological principles and performance evaluation in a proficiency testing scheme in support of the Council Directive 98/83/EC

Proficiency testing as a means of external quality assessment plays the role of independent evidence of laboratories’ performance. To enable laboratories to fulfil the requirements stated in legislation, methodology for evaluation of laboratories’ performance in proficiency testing schemes should incorporate principles of measurement results which are fit for intended use and incorporate evaluation of laboratories’ performances based on independent reference value. A proficiency testing scheme was designed to support Drinking Water Directive (98/83/EC) specifically. The methodology for performance evaluation, which takes into account a “fitness for purpose”-based standard deviation for proficiency assessment, is proposed and discussed in terms of requirements of the Drinking Water Directive. A ζ′-score, modified by application of target uncertainty was developed in a way that fulfils requirements defined in the legislation. As an illustration, results are reported for nitrate concentration in water. The approach presented can also be applied to other fields of measurements.     

Visualisation of proficiency test exercise results in Kiri plots

A novel graphical method (‘Kiri plots’) for the presentation of proficiency test exercise results is presented. The Kiri plot visualises the evaluation of the proficiency test results based on three statistical tests (the z score, the zeta score and the relative uncertainty outlier test) by defining six zones including a central “in agreement” zone.     

Performance of uncertainty evaluation strategies in a food proficiency scheme

A study of the performance of different uncertainty evaluation strategies among 163 voluntary respondents from food proficiency schemes is presented. Strategies included use of: single-laboratory validation data, quality control data, past proficiency testing data, reproducibility data, a measurement equation and the dispersion of replicate observations on the test material. Most performed reasonably well, but the dispersion of replicate observations underestimated uncertainty by a factor of approximately 3. Intended compliance with accreditation requirements was associated with significantly improved uncertainty evaluation performance, while intended compliance with the ISO “Guide to the expression of uncertainty in measurement” had no significant effect. Substituting estimates based on the Horwitz or Horwitz–Thompson models or on PT target standard deviation for the respondents’ own estimates of uncertainty led to a marked reduction in poor zeta scores and significant improvement in dispersion of zeta scores.     

Approaching target uncertainty in proficiency testing schemes: experience in the field of water measurement

Good correspondence between evaluations of the proficiency testing data for water measurements according to different scoring schemes was found as a result of implementation of the measurement quality according to the target uncertainty defined in the Drinking Water Directive. The accuracy of the evaluation is determined by the uncertainty of the assigned value, and this uncertainty should be in correspondence with the target uncertainty. Modification of the zeta-score by the thorough application of the target uncertainty is presented. It is convenient for generalised presentation of the proficiency testing data. The allowed bias is included as a linear term on the measured value scale, as it is treated in uncertainty analysis. The importance of implementation of the target measurement uncertainty in other fields of the routine measurements is indicated.     

Confidence in testing services – new expectations, new rules, new challenges. 5th Eurolab Symposium offered important information and stimulating discussions

The 5th Eurolab Symposium on 4th and 5th October in Edinburgh was the highlight event of Eurolab in 2000, Eurolab’s 10th anniversary year. Under the title: ”Confidence in Testing Services – new expectations, new rules, new challenges” it reflected recent changes and developments in the laboratory scene and focussed on new tasks and challenges. The most essential changes for laboratories are connected with the new standard ISO/IEC 17025 ”General requirements for the competence of testing and calibration laboratories”. Keywords in this respect are uncertainty of measurement and traceability to SI units, requirements on personnel, proficiency testing, customer orientation, and management systems.     

The practice of proficiency testing in China

This paper briefly summarises the current situation for proficiency testing (PT) in China, outlines the policy for PT of China’s national accreditation committee for laboratories (CNACL), and exemplifies activities of the CNACL’s metal working group. Received: 9 December 2000 Accepted: 14 December 2000     

From GUM to alternative methods for measurement uncertainty evaluation

Since the advent of the Guide to the expression of Uncertainty in Measurement (GUM) in 1995 laying the principles of uncertainty evaluation numerous projects have been carried out to develop alternative practical methods that are easier to implement namely when it is impossible to model the measurement process for technical or economical aspects. In this paper, the author presents the recent evolution of measurement uncertainty evaluation methods. The evaluation of measurement uncertainty can be presented according to two axes based on intralaboratory and interlaboratory approaches. The intralaboratory approach includes “the modelling approach” (application of the procedure described in section 8 of the GUM, known as GUM uncertainty framework) and “the single laboratory validation approach”. The interlaboratory approaches are based on collaborative studies and they are respectively named “interlaboratory validation approach” and “proficiency testing approach”.     

Uncertainty evaluation in proficiency testing: state-of-the-art, challenges, and perspectives

The evaluation of measurement uncertainty, and that of uncertainty statements of participating laboratories will be a challenge to be met in the coming years. The publication of ISO 17025 has led to the situation that testing laboratories should, to a certain extent, meet the same requirements regarding measurement uncertainty and traceability. As a consequence, proficiency test organizers should deal with the issues measurement uncertainty and traceability as well. Two common statistical models used in proficiency testing are revisited to explore the options to include the evaluation of the measurement uncertainty of the PTRV (proficiency test reference value). Furthermore, the use of this PTRV and its uncertainty estimate for assessing the uncertainty statements of the participants for the two models will be discussed. It is concluded that in analogy to Key Comparisons it is feasible to implement proficiency tests in such a way, that the new requirements can be met. Received: 29 September 2000 Accepted: 3 December 2000     

Metrological traceability is not always a straight line

The two most important concepts in metrology are certainly “traceability to standards” and “measurement uncertainty evaluation”. So far the questions related to these concepts have been reasonably solved in the metrology of “classical quantities”, but for the introduction of metrological concepts in new fields, such as chemistry and biology, a lot of problems remain and must be solved in order to support international arrangements. In this presentation, the authors want to develop the strategy implemented at Laboratoire national de métrologie et d’essais (LNE) in metrology in chemistry and biology. The strategy is based on: (1) pure solutions for calibration of analytical instruments, (2) use of certified reference materials (matrix reference materials), and (3) participation to proficiency testing schemes. Examples will be presented in organic and inorganic chemistry. For laboratory medicine, proficiency testing providers play an important role in the organization of External Quality Assessment Schemes. For the time being, the reference value or the assigned value of the comparison is calculated with the results obtained by the participants. This assigned value is not often traceable to SI units. One of the methods suggested by LNE is to ensure the metrological traceability to SI units of the assigned value for the more critical quantities carried on analytes by implementing the Joint committee for traceability in laboratory medicine reference methods.     

ISO/IEC 17043: the new International Standard for proficiency testing

The revision of ISO/IEC Guide 43 is well underway by CASCO Working Group 28, with the author as Convener. The ballot on the committee draft was successfully completed in June 2008. The Working Group met in September to resolve the CD comments and to recommend that the document should advance as an ISO/IEC DIS. The document is in harmony with ISO/IEC 17025 (2005), ILAC G13 (2007), and the IUPAC Harmonized Protocol (2006) for proficiency testing. Major changes from Guide 43 include considerations for inspection bodies, requirements for metrological traceability of assigned values for calibration schemes, limitations on subcontracting, requirements for reporting the uncertainty of assigned values, and for including method information and technical commentary in scheme reports. Minor changes include requirements for equipment used in the manufacture and testing of proficiency test items, considerations for handling laboratories’ reported uncertainties, and the issuance of certificates of participation. These changes have not generated any substantial objections from the CASCO membership and liaisons.     

Using mixture models for bump-hunting in the results of proficiency tests

The interpretation of the results of proficiency tests by the use of mixture models is described. The data are interpreted as a sample from a mixture of several normal populations. The calculation of the statistics (the means, variances and proportions of each component) is accomplished by means of the ‘EM’ algorithm. The method has several advantages over those previously advanced, principally that the algorithm is fast and easy to execute. Examples from proficiency testing are discussed.     

Matrix materials for proficiency testing: optimization of a procedure for spiking soil with gamma-emitting radionuclides

The IAEA Reference Materials Group of the Chemistry Unit, Agency’s Laboratories Seibersdorf, has developed and optimized a procedure for spiking some environmental matrices with gamma-emitting radionuclides. This paper describes the spiking procedure, homogeneity testing of the spiked material, and assignment of property values and their associated uncertainties for the radionuclides ⁵⁴Mn, ⁶⁵Zn, ⁶⁰Co, ¹⁰⁹Cd, ¹³⁴Cs, ¹³⁷Cs, ²¹⁰Pb, and ²⁴¹Am. This procedure has already been successfully used in an IAEA proficiency test on the determination of ¹³⁷Cs and ²¹⁰Pb in spiked soil and has been found to be appropriate for production of soil materials for proficiency testing and internal quality control samples. The main advantage of this procedure is a low uncertainty arising from heterogeneity, which was found to be less than 1.2% for all the analytes studied.      

On the use of the ‘uncertainty budget’ to detect dominant terms in the evaluation of measurement uncertainty

In the evaluation of measurement uncertainty, the uncertainty budget is usually used to identify dominant terms that contribute to the uncertainty of the output estimate. Although a feature of the GUF method, it is also recommended as a qualitative tool in MCM by using ‘nonlinear’ equivalents of uncertainty contributions and sensitivity coefficients. In this paper, the use of ‘linear’ and ‘nonlinear’ parameters is discussed. It is shown that when and only when the standard uncertainty of the output estimate is nearly equal to the square root of the sum of the squares of the individual uncertainty contributions, will the latter be a reliable tool to detect the degree of contribution of each input quantity to the measurand uncertainty.     

Using uncertainty functions to predict and specify the performance of analytical methods

In both European legislation relating to the testing of food and the recommendations of the Codex Alimentarius Commission, there is a movement away from specifying particular analytical methods towards specifying performance criteria to which any methods used must adhere. This ‘criteria approach’ has hitherto been based on the features traditionally used to describe analytical performance. This paper proposes replacing the traditional features, namely accuracy, applicability, detection limit and limit of determination, linearity, precision, recovery, selectivity and sensitivity, with a single specification, the uncertainty function, which tells us how the uncertainty varies with concentration. The uncertainty function can be used in two ways, either as a ‘fitness function’, which describes the uncertainty that is fit for purpose, or as a ‘characteristic function’ that describes the performance of a defined method applied to a defined range of test materials. Analytical chemists reporting the outcome of method validations are encouraged to do so in future in terms of the uncertainty function. When no uncertainty function is available, existing traditional information can be used to define one that is suitable for ‘off-the-shelf’ method selection. Some illustrative examples of the use of these functions in methods selection are appended.     

The content of chlorides in concrete designs: a probabilistic approach to compliance assessment with due regard to the measurement uncertainty

A major factor for the durability of concrete is the protection against corrosion of reinforcing steel. One out of some corrosion agents are dissolved chlorides. Therefore, their contents are limited in the raw materials and subject to a compliance assessment. The concrete standard EN 206-1 provides both a semi-probabilistic calculation method for the design of concrete as regards the input of chlorides through the components and a decision rule for compliance assessment. This paper describes a full probabilistic approach to the compliance assessment taking into account the heterogeneity of the components and the uncertainty of testing. The content of Cl⁻ is defined as the sum of gamma-distributed parameters. The uncertainty data are taken from the test procedures and from effects due to sampling procedures. The proposed approach is quantified for a high-performance and for a simple concrete. It allows calculating upper limits of chloride contents according to the client’s requirements for conformity and/or for producer’s/consumer’s risks. Additionally discussed are the particular financial consequences of incorrect decisions. Major results of the application of the probabilistic approach are that the upper limits of the chloride contents are comparable to those of the semi-probabilistic one and that the risks for high-performance concretes are not negligible. The decision rule given in EN 206-1 is easy to apply but not unambiguous. The probabilistic approach permits to define concepts for decision rules based on the probability of conformity or on the appropriate risks. For risk-based concepts, the individual performance of the testing laboratory can be taken into account.     

Laboratory assessment by combined z score values in proficiency tests: experience gained through the European Union proficiency tests for pesticide residues in fruits and vegetables

The obligation for accredited laboratories to participate in proficiency tests under ISO 17025, performing multiresidue methods (MRMs) for pesticide residues, involves the reporting of a large number of individual z scores making the evaluation of the overall performance of the laboratories difficult. It entails, time and again, the need for ways to summarise the laboratory’s overall assessment into a unique combined index. In addition, the need for ways to continually evaluate the performance of the laboratory over the years is equally acknowledged. For these reasons, following 14 years of the European Union Reference Laboratory for Pesticide Residues in Fruits and Vegetables (EUPT-FV), useful formulas have been designed to globally evaluate the assessment of the participating laboratories. The aim is to achieve a formula which is easy to understand, which can be applied and which fits the purposes of long-term evaluation detecting positive and negative trends. Moreover, consideration is needed for a fair compensation of bad results in MRM, taking into account the large number of compounds that are covered. It is therefore important to be aware of the difficulties in getting satisfactory values from a wide range of compounds. This work presents an evaluation of the main well-established combined z score formulas together with those new ones developed here which have been applied to the European proficiency test results (EUPTs) over the years. Previous formulas such as the rescaled sum of z score (RSZ), the sum squared of z score (SSZ) and the relative laboratory performance (RLP) are compared with the newer ones: the sum of weighted z scores (SWZ) and the sum of squared z scores (SZ2). By means of formula comparisons, conclusions on the advantages, drawbacks and the most fit-for-purpose approach are achieved.     

The determination of an accepted reference value from proficiency data with stated uncertainties

Proficiency data with stated uncertainties represent a unique opportunity for testing that the reported uncertainties are consistent with the Guide to the expression of uncertainty in measurement (GUM). In most proficiency tests, however, this opportunity is forfeited, because proficiency data are processed without regard to their uncertainties. In this paper we present alternative approaches for determining a reference value as the weighted mean of all mutually consistent results and their stated uncertainties. Using an accepted reference value each reported uncertainty estimate can be expressed as an E _n number, but a value of confirms its validity only if the uncertainty of the reference value is negligible in comparison.Reference values calculated for results from an International Measurement Evaluation Programme (IMEP-9) by “bottom up” as well as “top down” methods were practically identical, although the first strategy yielded the lowest uncertainty. A plot of individual coefficients of variation (CV) versus E _n numbers helps interpretation of the proficiency data, which could be used to validate relative uncertainties down to <1%.     

A new certified reference material for benzene measurement in air on a sorbent tube: development and proficiency testing

A certified matrix reference material (CRM) for the measurement of benzene in ambient air has been developed at Laboratoire National de Métrologie et d’Essais. The production of these CRMs was conducted using a gravimetric method fully traceable to the International System of Units. The CRMs were prepared by sampling an accurate mass of a gaseous primary reference material of benzene, using a high-precision laminar flowmeter and a mass flow controller, with a PerkinElmer sampler filled with Carbopack™ X sorbent. The relative standard deviations obtained for the preparation of a batch of 20 tubes loaded with 500 ng of benzene were below 0.2%. Each CRM is considered independent from the others and with its own certified value and an expanded uncertainty estimated to be within 0.5%, lower than the uncertainties of benzene CRMs already available worldwide. The stability of these materials was also established up to 12 months. These CRMs were implemented during proficiency testing, to evaluate the analytical performances of seven French laboratories involved in benzene air monitoring.     

Proficiency tests using four batches of green alga with controlled levels of cadmium

A set of test materials of the green alga Chlorella vulgaris with different levels of naturally bound Cd and about the same levels of other 13 essential or trace elements (BIOMA 1–4) was produced for proficiency testing of laboratories involved in elemental analysis of food. Criteria of the laboratories’ performance, such as the z-score and combination scores RSZ and SSZ were evaluated and discussed in terms of their suitability for proficiency testing using a set of test materials with the same matrix. The use of the test materials as a set of internal reference materials has also been suggested. Received: 23 May 1997 / Revised: 1 August 1997 / Accepted: 11 August 1997     

The impact of measurement uncertainty on the producer’s and user’s risks, on classification and conformity assessment: an example based on tests on some construction products

The reliability of test results and subsequent classification statements or product certification depend on the variability of the product’s properties and on the validity of the test procedures used. With an emphasis on measurement uncertainty, producer’s and user’s risks as well as probabilities of conformance, conformity and classification are calculated exemplarily for two requirements for mineral aggregates used in construction. An important methodological basis is an international draft document on measurement uncertainty in conformity assessment. The mathematical instruments given are applied and further developed to a risk scenario for product classification. The results from a classification point of view show that the reliability of test results for acid-soluble sulphates is mostly acceptable and both the producer’s and user’s risks are quite small. In contrast, the magnesium sulphate index test produces results which are hardly usable for classification and certification purposes or for risk management in production. Product certification bodies should generally have an appropriate approach when dealing with results where precision data are poor.