Estimates of uncertainty of measurement from proficiency testing data: a case study

The results obtained by a laboratory over a number of proficiency testing/external quality assessment schemes (PT/EQAS) rounds can give information on the uncertainty of its measurements for a given test, provided that conditions such as full coverage of the routine analytical range, traceability, and small uncertainty of the assigned values (compared to the spread of the results) are met and provided that systematic deviations and any other sources of uncertainty are considered. As organisers of the Italian EQAS (ITEQAS) in occupational and environmental laboratory medicine, we tested this hypothesis using as model data from well-performing laboratories taking part in ITEQAS for lead in blood over the last 2 years. We also investigated how different PT/EQAS features (frequency of trials and number of samples) would affect a laboratory estimate of its uncertainty. Such information can be helpful in improving PT/EQAS organisation and define, for a given test: (a) the state of the art of the uncertainty of current measurement procedures, (b) identify needs for improvement of analytical methodologies and (c) set targets for acceptable uncertainty values.Presented at the Eurachem PT Workshop September 2005, Portorož, Slovenia.Papers published in this section do not necessarily reflect the opinion of the Editors, the Editorial Board and the Publisher.     

Evaluation of proficiency testing results taking into account their uncertainties

A new composite score for the evaluation of performance of proficiency testing participants is proposed. The score is based on a combination of the z-score, uncertainty of a participant’s measurement result and uncertainty of the proficiency testing scheme’s assigned value. The use of such a composite score will allow evaluation not only of the participant’s ability to determine an analyte in corresponding matrix, but also their understanding of the uncertainty in the obtained analytical result. The score may be helpful for the laboratory’s quality system and for laboratory accreditation according to ISO 17025.     

Statistical treatment of proficiency testing data

 There are three stages to evaluating a laboratory's results in an interlaboratory proficiency test: establishing the correct result for the test item, determining an evaluation statistic for the particular result, and establishing an acceptable range. There are a wide variety of procedures for accomplishing these three stages and a correspondingly wide variety of statistical techniques in use. Currently in North America the largest number of laboratory proficiency test programs are in the clinical laboratory field, followed by programs for environmental laboratories that test drinking water and waste water. Proficiency testing in both of these fields is under the jurisdiction of the federal government and other regulatory and accreditation agencies. Many of the statistical procedures are specified in the regulations, to assure comparability of different programs and a fair evaluation of performance. In this article statistical procedures recommended in International Organization for Standardization Guide 43, Part 1, are discussed and compared with current practices in North America. Received: 22 April 1998 · Accepted: 12 May 1998     

The determination of an accepted reference value from proficiency data with stated uncertainties

Proficiency data with stated uncertainties represent a unique opportunity for testing that the reported uncertainties are consistent with the Guide to the expression of uncertainty in measurement (GUM). In most proficiency tests, however, this opportunity is forfeited, because proficiency data are processed without regard to their uncertainties. In this paper we present alternative approaches for determining a reference value as the weighted mean of all mutually consistent results and their stated uncertainties. Using an accepted reference value each reported uncertainty estimate can be expressed as an E _n number, but a value of confirms its validity only if the uncertainty of the reference value is negligible in comparison.Reference values calculated for results from an International Measurement Evaluation Programme (IMEP-9) by “bottom up” as well as “top down” methods were practically identical, although the first strategy yielded the lowest uncertainty. A plot of individual coefficients of variation (CV) versus E _n numbers helps interpretation of the proficiency data, which could be used to validate relative uncertainties down to <1%.     

The application of data from proficiency testing to laboratory accreditation according to ISO 17025

Current methods of testing laboratories for their proficiency in reporting correct measurement results are liable to substantial errors of the second kind. This means that laboratories with deflated uncertainties are accepted as proficient, even though their reported measurement results pave the way for erroneous conclusions. Only by using E _n numbers based on an accepted reference value with the lowest possible uncertainty can the risk of recognising incorrect measurement results be kept at an acceptable level. Based on an actual set of proficiency test (PT)-data for the concentration of Pb in water, this paper compares PT results obtained by methods using E _n numbers with methods based on z-scores. Kaj Heydorn is a technical assessor at The Danish Accreditation and Metrology Fund (DANAK).     

Analytical performance is improved by regular participation in proficiency testing: an analysis of data from the Aquacheck proficiency testing scheme

Analysis of historical data from the Aquacheck proficiency testing scheme demonstrates improvement in the performance of participants correlated to the frequency of participation. Analysis of the spread of results returned by all participants, defined by the percentage relative standard deviation (RSD) and the performance (Z) scores awarded to the individual participants, indicates that the overall performance for the group of participants is consistent. However, when the frequency of participation is analysed, it can be clearly seen that those laboratories who participated in the majority of the distributions available reported 10–15% more acceptable (Z-scores between +2 and ?2) results than those labs who took part with a greatly reduced frequency. When ‘more difficult’ analytes are considered, those laboratories who participated in greater than 75% of available rounds reported up to 30% more acceptable results than those laboratories who participated in 25% or fewer, of the available rounds.     

A proficiency testing scheme to evaluate the effectiveness of laboratory sample reduction of a soil sample

Accreditation and Quality Assurance - Although proficiency testing (PT) schemes in analytical chemistry are intended to test the entire analytical process, the reality is that some aspects of the...     

Interpretation of interlaboratory comparison results to evaluate laboratory proficiency

Guidelines are given for the evaluation of proficiency test (PT) results in order to increase the effectivity of PT participation. For better understanding, some statistical background is given along with some examples to show the effects of the choices made by the PT provider. The calculation method of the assigned value and the selection of the standard deviation both affect the z-score that is used by the participating laboratory to judge the quality of its performance in the PT. Therefore, the participating laboratory is advised to use the PT results with care and, if necessary, to recalculate the z-scores. Finally, advice is given on how not to follow up bad PT results along with some valuable steps that could be part of an effective follow-up procedure.     

Improved evaluation of measurement uncertainty from sampling by inclusion of between-sampler bias using sampling proficiency testing

A realistic estimate of the uncertainty of a measurement result is essential for its reliable interpretation. Recent methods for such estimation include the contribution to uncertainty from the sampling process, but they only include the random and not the systematic effects. Sampling Proficiency Tests (SPTs) have been used previously to assess the performance of samplers, but the results can also be used to evaluate measurement uncertainty, including the systematic effects. A new SPT conducted on the determination of moisture in fresh butter is used to exemplify how SPT results can be used not only to score samplers but also to estimate uncertainty. The comparison between uncertainty evaluated within- and between-samplers is used to demonstrate that sampling bias is causing the estimates of expanded relative uncertainty to rise by over a factor of two (from 0.39% to 0.87%) in this case. General criteria are given for the experimental design and the sampling target that are required to apply this approach to measurements on any material.     

Control charts to evaluate long-term performance in proficiency tests

As proficiency tests (PTs) provider, the European Union Reference Laboratory for Chemical Elements in Food of Animal Origin (EURL-CEFAO) yearly organizes exercises on the determination of chemical elements within the area of its mandate. The scheme is addressed to the National Reference Laboratories (NRLs) of the European Union Member States. Each PT is planned giving priority to the matrices and elements for which maximum levels are set in Commission Regulation No. 1881/2006 and following amendments as well as to those mainly considered in National Residues Control Plans. Samples to be distributed are prepared in the Laboratory’s facilities using materials purchased on the market and often adjusting the concentration levels of the analytes around their maximum levels. In the 2006–2011 period, PTs were planned as a long-term programme pursuing the general objective of providing the NRLs with a scheme that can give them the opportunity to check and improve the performance of their analytical methods and to verify the effectiveness of any corrective actions through the repetition of the same matrix. Control charts give participants the possibility of a long-term follow-up of their performance promoting the improvement of Quality Control. To this aim, Shewhart and Cusum control charts for z-scores are prepared according to ISO 13528:2005 and updated after each PT for each laboratory. z-scores are assigned using values of σ _pEURL-CEFAO that correspond to the level of performance appropriate to the demand. Cusum charts are also evaluated by EURL-CEFAO in order to identify problems that may cause a bias in the measurement method.     

Experiences from the assessment of proficiency testing providers

Following a pilot project in 2001–2002, UKAS accredited nine PT providers. During the course of the pilot and at subsequent surveillance visits, UKAS has gained experience in assessing organisations against the criteria detailed within ISO/IEC Guide 43-1:1997 and ILAC G13:2000. This article explores some of the key aspects that arose during the pilot and how UKAS has accommodated the addition of the accreditation of PT providers in to its range of services.Presented at the Eurachem PT Workshop September 2005, Portorož, Slovenia     

First proficiency testing to evaluate the ability of European Union National Reference Laboratories to detect staphylococcal enterotoxins in milk products

The European Commission has designed a network of European Union-National Reference Laboratories (EU-NRLs), coordinated by a Community Reference Laboratory (CRL), for control of hygiene of milk and milk products (Council Directive 92/46/ECC). As a common contaminant of milk and milk products such as cheese, staphylococcal enterotoxins are often involved in human outbreaks and should be monitored regularly. The main tasks of the EU-CRLs were to select and transfer to the EU-NRLs a reference method for detection of enterotoxins, and to set up proficiency testing to evaluate the competency of the European laboratory network. The first interlaboratory exercise was performed on samples of freeze-dried cheese inoculated with 2 levels of staphylococcal enterotoxins (0.1 and 0.25 ng/g) and on an uninoculated control. These levels were chosen considering the EU regulation for staphylococcal enterotoxins in milk and milk products and the limit of detection of the enzyme-linked immunosorbent assay test recommended in the reference method. The trial was conducted according to the recommendations of ISO Guide 43. Results produced by laboratories were compiled and compared through statistical analysis. Except for data from 2 laboratories for the uninoculated control and cheese inoculated at 0.1 ng/g, all laboratories produced satisfactory results, showing the ability of the EU-NRL network to monitor the enterotoxin contaminant.     

Normalisation of data from allergens proficiency tests

The problem of allergen analysis using ELISA kits from different commercial products giving significantly different results is widely acknowledged. The effect on proficiency testing results is that different assigned values have to be generated for the different kits used. Some experimental Food Analysis Performance Assessment Scheme (FAPAS) proficiency tests aimed to establish whether the use of a standardised calibrant could be used to normalise the complete data set without recourse to differentiation. Three recent FAPAS proficiency tests (2776 peanut, 2778 soya and 2781 gluten) sent out a second spiked sample, in addition to the usual spiked and unspiked samples. Further analysis of the data was undertaken after the completion of the tests. The ratio of the submitted results for the two spiked samples yielded complete data sets which could be tested for normality of the distribution. Where the raw data for each individual test sample was clearly non-normal and multi-modal, the ratio data yielded a much more normal and symmetrical distribution. The use of one of the test samples as a single-point calibrant has some limitations but the principle of applying a standardisation clearly works. The development of internationally recognised sets of certified reference calibration standards for use by allergens testing laboratories would greatly benefit the analysis.     

Harmonisation of proficiency testing schemes

The harmonisation of proficiency testing (PT) schemes has been under debate for a long time. There are obvious reasons why harmonisation of the practices in PT would be beneficial. In many areas, there is still a belief that further harmonisation of practices in PT would improve the comparability of measurement data. In particular when two laboratories are to be compared that have not participated in a single PT, problems arise which allegedly can be overcome by further harmonisation of PT schemes. In practice, however, parties involved in PT are not always embracing the idea of harmonisation. With the results of two European projects in mind, a discussion is given on harmonisation aspects, and some considerations are given that may help to decide in practice whether harmonisation is likely to solve particular problems. The first project, the European Proficiency Testing Network (EPTN), is concerned with further harmonisation. The second European project (COEPT) aims at providing a basis to assess equivalence across proficiency tests, and explores the conditions under which such an assessment is feasible.

     

Harmonisation of proficiency testing schemes

The harmonisation of proficiency testing (PT) schemes has been under debate for a long time. There are obvious reasons why harmonisation of the practices in PT would be beneficial. In many areas, there is still a belief that further harmonisation of practices in PT would improve the comparability of measurement data. In particular when two laboratories are to be compared that have not participated in a single PT, problems arise which allegedly can be overcome by further harmonisation of PT schemes. In practice, however, parties involved in PT are not always embracing the idea of harmonisation. With the results of two European projects in mind, a discussion is given on harmonisation aspects, and some considerations are given that may help to decide in practice whether harmonisation is likely to solve particular problems. The first project, the European Proficiency Testing Network (EPTN), is concerned with further harmonisation. The second European project (COEPT) aims at providing a basis to assess equivalence across proficiency tests, and explores the conditions under which such an assessment is feasible.