Performance evaluation for proficiency testing with a limited number of participants

Proficiency testing (PT) is an essential tool for laboratories to assess their competency. Also, participation in PT has become one of the mandatory requirements for laboratory to seek accreditation according to ISO/IEC 17025. For this reason, the effectiveness of performance evaluation by PT scheme is of great concern for the participants and for accreditation bodies as well. In practice, owing to unavailability of other appropriate alternatives, PT scheme providers may have to choose using consensus values to evaluate the performance of participants. However, such consensus values approach was not recommended by relevant international guidelines for PT schemes with limited number of participants. With the use of Monte Carlo simulation technique, this study attempted to investigate the effectiveness of using consensus values for performance evaluation in PT schemes with limited number of participants. The simulation process was schemed according to the statistical model provided by ISO 5725-1 for laboratory measurement results, which covered components like method bias, laboratory bias, and measurement precision. The effectiveness of the consensus value approach was expressed as the percentage of participants in a simulation run could get the same evaluation result, either satisfactory or unsatisfactory, against the “true value.” The findings indicated that the number of participants, choice of consensus values, mass fraction of analyte, method bias, laboratory bias, and measurement repeatability of participating laboratories would all affect the effectiveness of the consensus value approach but at different extent. However, under certain circumstances, use of consensus value could still be considered as an acceptable approach for performance evaluation even the number of participants was limited. Some of the findings were further verified using real data from PT schemes where appropriate certified reference materials or reliable reference values were available.     

Metrological traceability is not always a straight line

The two most important concepts in metrology are certainly “traceability to standards” and “measurement uncertainty evaluation”. So far the questions related to these concepts have been reasonably solved in the metrology of “classical quantities”, but for the introduction of metrological concepts in new fields, such as chemistry and biology, a lot of problems remain and must be solved in order to support international arrangements. In this presentation, the authors want to develop the strategy implemented at Laboratoire national de métrologie et d’essais (LNE) in metrology in chemistry and biology. The strategy is based on: (1) pure solutions for calibration of analytical instruments, (2) use of certified reference materials (matrix reference materials), and (3) participation to proficiency testing schemes. Examples will be presented in organic and inorganic chemistry. For laboratory medicine, proficiency testing providers play an important role in the organization of External Quality Assessment Schemes. For the time being, the reference value or the assigned value of the comparison is calculated with the results obtained by the participants. This assigned value is not often traceable to SI units. One of the methods suggested by LNE is to ensure the metrological traceability to SI units of the assigned value for the more critical quantities carried on analytes by implementing the Joint committee for traceability in laboratory medicine reference methods.     

Evaluation of the use of consensus values in proficiency testing programmes

Proficiency testing (PT) is an essential tool used by laboratory accreditation bodies to assess the competency of laboratories. Because of limited resources of PT providers or for other reasons, the assigned reference value used in the calculation of z-score values has usually been derived from some sort of consensus value obtained by central tendency estimators such as the arithmetic mean or robust mean. However, if the assigned reference value deviates significantly from the ‘true value’ of the analyte in the test material, laboratories’ performance will be evaluated incorrectly. This paper evaluates the use of consensus values in proficiency testing programmes using the Monte Carlo simulation technique. The results indicated that the deviation of the assigned value from the true value could be as large as 40%, depending on the parameters of the proficiency testing programmes under investigation such as sample homogeneity, number of participant laboratories, concentration level, method precision and laboratory bias. To study how these parameters affect the degree of discrepancy between the consensus value and the true value, a fractional factorial design was also applied. The findings indicate that the number of participating laboratories and the distribution of laboratory bias were the prime two factors affecting the deviation of the consensus value from the true value.     

Determination of Assigned Values for Cadmium,Cobalt and Manganese in Drinking Water Proficiency Testing

In proficiency tests the selection of an appropriate assigned value is one of the most critical points. Determination of the assigned value for proficiency test can be done in 2 ways: based on reference value and consensus value. The use of such a consensus value as an assigned value is a commonplace in proficiency testing, because there is seldom at present a cost effective alternative. However, the consensus value has a weakness in that it could be significantly different from the true value due largely inexperienced laboratories. Therefore in this work, the reference values were used as the assigned values for all analytes of interest. The reference values in spiked drinking water samples were produced by the Metrology in Chemistry Laboratory of Research Center for Chemistry - Indonesian Institute of Sciences (RCChem-LIPI) with traceable to SI units through standard solutions from the National Institute Standard and Technology, USA. The reference values of cobalt and manganese were obtained by measurement using two from three comparable methods of Graphite Furnace-Atomic Absorption Spectrometry (GF-AAS), Inductively Coupled Plasma-Optical Emission Spectrometry (ICP-OES) and Inductively Coupled Plasma-Mass Spectrometry (ICP-MS) while that for cadmium was obtained by a primary method of ICP-Isotope Dilution Mass Spectrometry (ICP-IDMS).The obtained reference values were used to evaluate the laboratory performance result in a proficiency testing scheme of cadmium, cobalt and manganese in drinking water which was provided in 2014. All fifty nine laboratories reported cadmium and cobalt at low concentration levels which were about more than 82% showed satisfactory performance results. Similarly, the results of manganese were also comparable at high level concentrations.     

An estimation of the measurement uncertainty for an optical emission spectrometric method

 A validation procedure based on the ISO/IEC 17025 standard was used to demonstrate the long-term stability of a calibration process and to assess the measurement uncertainty of a standard test method for optical emission vacuum spectrometric analysis of carbon and low-alloy steel (ASTM E 415–99a). The validation was used to provide documented evidence that the selected method fulfils the requirements and that the method is ”fit for purpose”. A test for drift was applied to determine statistically whether the analytical results vary systematically with time. The accuracy and traceability of the optimised method were tested by an analysis of closely matched matrix certified reference materials (CRMs). The measurement uncertainty estimations took account of the precision study, the bias and its uncertainty, and the qualification of uncertainties not considered in the overall performance studies. Received: 2 November 2002 Accepted: 2 January 2003 Acknowledgement The author expresses gratitude to Dr. Aleš Fajgelj for helpful discussions during the 3rd Central European Conference on Reference Materials and Measurements. Presented at CERMM-3, Central European Reference Materials and Measurements Conference: The function of reference materials in the measurement process, May 30–June 1, 2002, Rogaška Slatina, Slovenia Correspondence to T. Drglin     

Development of a reference measurement procedure for the determination of methylmercury in fish products

The development of an analytical procedure for speciation analysis of methylmercury in fish products is presented. The method is based on high-performance liquid chromatography hyphenated to inductively coupled plasma-mass spectrometry. The metrological approach is stressed out in this paper, in order to provide reliable and comparable results. A complete uncertainty budget has been evaluated and the method has been validated by the use of a certified reference material. Moreover, the detection could rely on the isotope dilution mass spectrometry, a powerful strategy capable of highly accurate results traceable to the “Système International d’Unités” and recognised by the “Comité Consultatif pour la Quantité de Matière” as a primary method of measurement. Presented at MEFNM 2008, September 2008, Budapest, Hungary.     

Establishment of SI-traceable reference values for the content of various elements in the IMEP-14 sediment sample

For the first time in the International Measurement Evaluation Programme (IMEP)-14, a sediment sample was offered to analytical laboratories to perform measurements of As, Cd, Cr, Cu, Fe, Pb, Hg, Ni, U and Zn. In line with IMEP policy, the results were presented according to the certified / assigned reference values established by several reference laboratories around the world. The certification campaign is described in detail. Isotope dilution mass spectrometry was applied as a primary method of measurement, whenever possible, to achieve SI-traceable results. For reference measurements of As, Fe, Hg and Zn, k₀-neutron activation analysis and Zeeman atomic absorption spectrometry were applied. The reference values (ranges) were characterised as ”certified” (for Cd, Cr, Pb, Ni and U) or ”assigned” (for As, Cu, Fe, Hg and Zn) according to IMEP policy. The measurement uncertainty of the certified / assured reference values was calculated according to the ISO/BIPM Guide. Received: 7 June 2001-10-27 Accepted: 19 August 2001     

Analysis of 19-norandrosterone in human urine by gas chromatography–isotope-dilution mass spectrometry: method adopted by LGC for participation in the Comité Consultatif pour la Quantité de Matière (CCQM) Pilot Study P68

The method used at LGC for analysis of “total” 19-norandrosterone (19-norandrosterone glucuronide plus “free” 19-norandrosterone) in urine for the Comité Consultatif pour la Quantité de Matière Pilot Study (CCQM-P68) is described. The analytical method used was a modified version of the method developed at the National Measurement Institute of Australia, which used a hydrolysis and derivatisation procedure first described by the German Sports University. This method is routinely used by World Anti-Doping Agency-accredited laboratories for sports drug testing. The main modifications made to the method were the use of 19-norandrosterone glucuronide as a calibration standard and 19-norandrosterone glucuronide-d4 as an isotopically labelled internal standard, and the use of a bench-top quadrupole gas chromatograph–mass spectrometer. The results produced by LGC (2.14 ± 0.15 ng g⁻¹ expanded uncertainty, coverage factor k = 2) were in excellent agreement with those from other participating national metrology institutes and thus further validates the exact-matching isotope-dilution mass spectrometric procedures used at LGC for a wide range of reference measurement applications, including measurement of ng g⁻¹ levels of steroids in a biological matrix.     

Reference measurement procedure for Δ9-tetrahydrocannabinol in serum

An isotope dilution gas chromatography/mass spectrometry (ID-GC/MS) reference measurement procedure for Δ9-tetrahydrocannabinol (THC) in serum was developed and validated. The method complies with the concept of a ratio primary reference measurement procedure. The uncertainty was determined for two concentrations of THC in serum (1 ng/mL and 2.4 ng/mL). The calculation procedure is based on the Guide to the Expression of Uncertainty in Measurement (GUM). The relative expanded uncertainty was found to be less than 2% for both concentration levels, corresponding to a 95% confidence interval. For the reference method, it was shown that the measurement of THC within the concentration range covered by the current threshold values is very accurate. The method has the potential to provide traceability for the methods used in practical forensics.     

A comparison of location estimators for interlaboratory data contaminated with value and uncertainty outliers

While estimation of measurement uncertainty (MU) is increasingly acknowledged as an essential component of the chemical measurement process, there is little agreement on how best to use even nominally well-estimated MU. There are philosophical and practical issues involved in defining what is “best” for a given data set; however, there is remarkably little guidance on how well different MU-using estimators perform with imperfect data. This report characterizes the bias, efficiency, and robustness properties for several commonly used or recently proposed estimators of true location, μ, using “Monte Carlo” (MC) evaluation of “measurement” data sets drawn from well-defined distributions. These synthetic models address a number of issues pertinent to interlaboratory comparisons studies. While the MC results do not provide specific guidance on “which estimator is best” for any given set of real data, they do provide broad insight into the expected relative performance within broadly defined scenarios. Perhaps the broadest and most emphatic guidance from the present study is that (1) well-estimated measurement uncertainties can be used to improve the reliability of location determination and (2) some approaches to using measurement uncertainties are better than others. The traditional inverse squared uncertainty-weighted estimators perform well only in the absence of unrepresentative values (value outliers) or underestimated uncertainties (uncertainty outliers); even modest contamination by such outliers may result in relatively inaccurate estimates. In contrast, some inverse total variance-weighted-estimators and probability density function area-based estimators perform well for all scenarios evaluated, including underestimated uncertainties, extreme value outliers, and asymmetric contamination.     

Metrological reference values for estimating measurement bias in clinical laboratory sciences

Clinical laboratory quantities are measured for monitoring or diagnostic purposes. In both cases, a modification of the measurement bias can generate a false interpretation of measurement results. On the other hand, in clinical laboratory sciences, one of the most frequently used metrological reference value for calculating measurement bias is a conventional value that corresponds to the called consensus value. But this selection probably is not the best one, and some clinical and biological considerations should be made to decide the requirement for maximum permissible measurement bias and to decide the more appropriate metrological reference value. In the clinical laboratory, the medical relevance of the measurement bias of any measurement system in use depends on the measurement bias with which the biological reference values were produced. This dependence is due to the necessity that, for interpretation purposes, the current measurement bias should be the same that is included in the biological reference limits. For this reason, it is necessary to control the changes of bias during the life span of a measuring system. Several scenarios are described for the different estimations of bias.     

Consensus values for NIST biological and environmental Standard Reference Materials

The National Institute of Standards and Technology (NIST, formerly the National Bureau of Standards or NBS) has produced numerous Standard Reference Materials (SRM) for use in biological and environmental analytical chemistry. The value listed on the “NIST Certificate of Analysis” is the present best estimate of the “true” concentration of that element and is not expected to deviate from that concentration by more than the stated uncertainty. However, NIST does not certify the elemental concentration of every constituent and the number of elements reported in the NIST programs tends to be limited. Numerous analysts have published concentration data on these reference materials. Major journals in analytical chemistry, books, proceedings and “technical reports” have been surveyed to collect these available literature values. A standard statistical approach has been employed to evaluate the compiled data. Our methodology has been developed in a series of previous papers. Some subjective criteria are first used to reject aberrant data. Following these eliminations, an initial arithmetic mean and standard deviation (S.D.) are computed from remaining data for each element. All data now outside two S.D. from the initial mean are dropped and a second mean and S.D. recalculated. These final means and associated S.D. are reported as “consensus values” in our tables. Received: 25 April 1997 / Revised: 24 July 1977 / Accepted: 25 July 1997     

A practical procedure for assigning a reference value and uncertainty in the frame of an interlaboratory comparison

A method is suggested for the calculation of a reference value and its uncertainty to be used in the frame of an interlaboratory comparison (ILC). It is assumed that the reference value of the measurand is determined independently from the ILC round. It is derived from a limited set of measurement results obtained from one or several expert laboratories. The procedure involves three stages: (1) check of the experimental data and possible corrections; (2) check of the consistency of data, and possibly increase of the uncertainties in order to attain internal consistency; (3) choice between fully, partially or un-weighted mean.     

MALDI-TOF MS stability study of model poly(p-phenylene terephthalamide)s

In the present study, we address the possibility of matrix-assisted laser desorption/ionization (MALDI)–time-of-flight MS analysis-induced chain fragmentation in poly(p-phenylene terephthalamide) (PPD-T) by considering two possible sources: (1) grinding-induced fragmentation resulting from the evaporation–grinding MALDI sample preparation method (E-G method) and (2) in-source/metastable fragmentation induced by the MALDI laser. An analysis of variance (ANOVA) statistical study found, with a high probability, that obtaining MALDI spectra with the effective laser area as large as possible (the “fanned-out” setting) did not cause any chain fragmentation due to the E-G MALDI sample preparation method, even when three additional grinding steps were used. However, the effect of laser fluence was less clear. A significant effect of laser fluence was observed for lower mass oligomers (<1,400 Da), but there was essentially no effect for higher mass species up to our limit of ANOVA measurement (∼2,300 Da). Plausible explanations are presented to explain these observations. The most likely scenario is that “unexpected” end-group modifications occur during PPD-T synthesis, producing small quantities of low mass species, which are amplified by the MALDI-EG extraction procedure.     

The role of different soil sample digestion methods on trace elements analysis: a comparison of ICP-MS and INAA measurement results

The measurement of trace-element concentration in soil, sediment and waste, is generally a combination of a digestion procedure for dissolution of elements and a subsequent measurement of the dissolved elements. “Partial” and “total” digestion methods can be used in environmental monitoring activities. To compare measurement results obtained by different methods, it is crucial to determine and to maintain control of the bias of the results obtained by these methods. In this paper, ICP-MS results obtained after matrix digestion with modified aqua regia (HCl+HNO₃+H₂O₂) method and two “total” digestion methods (microwave aqua regia+HF and HNO₃+HF) are compared with those obtained by instrumental neutron activation analysis, a non-destructive analytical method for the determination of the total mass concentrations of inorganic components in environmental matrices. The comparison was carried out on eight agricultural soil samples collected in one test area and measured by k₀-INAA and ICP-MS to determine As, Co, Cr, Sb and Zn mass concentration. The bias of results for As, Cd, Co, Cr, Cu, Ni, Pb, Sb and Zn of the three digestion methods were assessed using selected measurement standards. This paper highlights that the digestion procedure is an integral part of the measurement and can affect the measurement result in environmental analysis.     

Metrological role of neutron activation analysis. IA. Inherent characteristics of relative INAA as a primary ratio method of measurement

In a series of three papers, the inherent characteristics of relative instrumental neutron activation analysis (INAA) as a primary ratio method of measurement, the unique functions of parametric INAA as an ideal back-up method of the relative INAA, and the valuable role of INAA in characterization of sampling behavior of individual elements in certified reference materials (CRMs) will be discussed. In this paper, the first of the series, the uncertainty evaluation and the traceability of values measured by neutron activation analysis (NAA), especially instrumental NAA (INAA), will be described to demonstrate the method at its ”the state-of-the-art” level can meet CCQM criteria for a primary ratio method. The scope and examples will be given. Received: 19 March 2001 Accepted: 2 October 2001     

Validation of multi-element isotope dilution ICPMS for the analysis of basalts

In this study we have validated a newly developed multi-element isotope dilution (ID) ICPMS method for the simultaneous analysis of up to 12 trace elements in geological samples. By evaluating the analytical uncertainty of individual components using certified reference materials we have quantified the overall analytical uncertainty of the multi-element ID ICPMS method at 1–2%. Individual components include sampling/weighing, purity of reagents, purity of spike solutions, calibration of spikes, determination of isotopic ratios, instrumental sources of error, correction of mass discrimination effect, values of constants, and operator bias. We have used the ID-determined trace elements for internal standardization to improve indirectly the analysis of 14 other (mainly mono-isotopic trace elements) by external calibration. The overall analytical uncertainty for those data is about 2–3%.In addition, we have analyzed USGS and MPI-DING geological reference materials (BHVO-1, BHVO-2, KL2-G, ML3B-G) to quantify the overall bias of the measurement procedure. Trace element analysis of geological reference materials yielded results that agree mostly within about 2–3% relative to the reference values. Since these results match the conclusions obtained by the investigation of the overall analytical uncertainty, we take this as a measure for the validity of multi-element ID ICPMS.     

The Japanese traceability system and the role of uncertainty evaluation in measurement

 The new traceability system of measurement standards based on the Japanese Measurement Law has been established since November 1993. Some reference materials such as metal standard solutions, pH standard solutions and standard gas mixtures are included in the system together with relevant physical quantities. In this system, primary measurement standard instruments or primary reference materials are designated by the regulation for each quantity. For the practical dissemination of each quantity, accreditation of calibration bodies is recognized by the steering committee under the supervision of the government. In the course of assessment of a candidate calibration body, the concepts of ISO/IEC Guide 25 and ISO/IEC Guide 58 are effectively introduced. For the estimation of reliability, the concept of how to introduce the statistical approach is effectively considered. The method of uncertainty evaluation described in the ISO document entitled "Guide to the expression of uncertainty in measurement" is adopted.     

Economy of proficiency testing: reference versus consensus values

Many proficiency tests are operated with a consensus value derived from the participants’ results. Apart from technical issues, one of the reasons often mentioned is that proficiency tests operated with consensus values would be cheaper than those using reference values obtained from a priori characterisation measurements. The economy of a proficiency test must of course be balanced by the need of the participants, and the quality of the comparison in general. The proficiency tests selected in this study had both a reference value and a consensus value, one of which was used for assessing the performance of the participating laboratories. In this work, both a technical and an economical assessment of how the comparisons were operated is made. From the evaluation, it follows that usually the use of consensus values does not necessarily reduce the costs of a proficiency test. However, frequently it may be observed that the quality of the assessment of the laboratories is better with a reference value. Received: 11 October 2000 Accepted: 3 January 2001