Visualisation of proficiency test exercise results in Kiri plots

A novel graphical method (‘Kiri plots’) for the presentation of proficiency test exercise results is presented. The Kiri plot visualises the evaluation of the proficiency test results based on three statistical tests (the z score, the zeta score and the relative uncertainty outlier test) by defining six zones including a central “in agreement” zone.     

A new approach for proficiency test exercise data evaluation

In this paper, a new data evaluation method for proficiency test exercises consisting of a combination of a z-test, a zeta test and an uncertainty outlier test is presented. This new method is compared with eight other evaluation methods (both measurement uncertainty using and measurement uncertainty ignoring) in common use and/or recommended by ISO 13528. The data set used to test the evaluation methods is real data and consists of the ⁹⁵Nb results of the National Physical Laboratory Environmental Radioactivity Proficiency Test Exercise 2007. The evaluation of 14 out of 32 results were affected by the choice of method.     

Evaluation of proficiency testing results taking into account their uncertainties

A new composite score for the evaluation of performance of proficiency testing participants is proposed. The score is based on a combination of the z-score, uncertainty of a participant’s measurement result and uncertainty of the proficiency testing scheme’s assigned value. The use of such a composite score will allow evaluation not only of the participant’s ability to determine an analyte in corresponding matrix, but also their understanding of the uncertainty in the obtained analytical result. The score may be helpful for the laboratory’s quality system and for laboratory accreditation according to ISO 17025.     

Use of uncertainty estimates as reported by participants in proficiency testing for the evaluation of their results: pros and cons

The aim of this paper is to study the measurement uncertainties reported in proficiency tests (PTs) using examples from PTs in the environmental sector and to compare the obtained measurement uncertainty estimates using different approaches. In addition, the paper focusses on the differences between the z-score and the zeta score. Since the year 2000, the Finnish Environment Institute has asked participants to report analytical methods as well as measurement uncertainties in connection with PT results. The measurement uncertainties of the assigned value have also been evaluated. On the basis of the results, the measurement uncertainties reported by the participants varied greatly. Participants often reported underestimated measurement uncertainties, but overestimated uncertainties were also reported. At the moment, it seems as if performance assessment should be based on the z-score because of a number of significant over- and underestimated measurement uncertainties. The zeta score should be used for information and educational purposes mainly.     

Comparability of measurement results for pesticide residues in foodstuffs: an open issue?

The dispersion of results from proficiency tests for the analysis of pesticide residues in foodstuffs suggests that improvements in the compatibility of measurement results are needed. Currently observed divergences can make the evaluation conclusion on foodstuffs compliance with certain legislation dependent on the consulted laboratory. This work discusses the origin and impact of this lack of compatibility, following the metrological concepts presented at the latest version of the “International Vocabulary of Metrology” (VIM3), thus allowing for a clear diagnostic of the problem. The reporting of results from different measurement methods uncorrected for the observed analyte recovery makes them traceable to different “operationally defined measurement procedures” (VIM3) and, therefore, not comparable. When results from different measurement methods are reported corrected for analyte recovery, R, and R is different for spiked and incurred residues, measurement results may be not compatible if this effect is not considered on the uncertainty budget. This discussion is illustrated with metrological models for any possible combination of “measurement performance” and “maximum residue level”. These models are complemented with experimental data of the analysis of pesticide residues in a sample of ginseng powder from a proficiency test. The adopted experimental design allowed the identification of additional threats to metrological compatibility in this field. Solutions to the faced problem are discussed for practicability and impact on regulatory issues. The use of a universal “reference measurement procedure” proves to be the most feasible way of ensuring comparability of measurements in this field.     

Integration of metrological principles and performance evaluation in a proficiency testing scheme in support of the Council Directive 98/83/EC

Proficiency testing as a means of external quality assessment plays the role of independent evidence of laboratories’ performance. To enable laboratories to fulfil the requirements stated in legislation, methodology for evaluation of laboratories’ performance in proficiency testing schemes should incorporate principles of measurement results which are fit for intended use and incorporate evaluation of laboratories’ performances based on independent reference value. A proficiency testing scheme was designed to support Drinking Water Directive (98/83/EC) specifically. The methodology for performance evaluation, which takes into account a “fitness for purpose”-based standard deviation for proficiency assessment, is proposed and discussed in terms of requirements of the Drinking Water Directive. A ζ′-score, modified by application of target uncertainty was developed in a way that fulfils requirements defined in the legislation. As an illustration, results are reported for nitrate concentration in water. The approach presented can also be applied to other fields of measurements.     

Development of metrological models for internal standard single-point and multi-point calibrations for the assessment of tear gas weapons compliance with legislation

Tear gases are the most widely used non-lethal weapons, both by security forces and by the general public. The pepper spray, whose active agent is capsaicin, is the only self-defence aerosol allowed in Portugal, where capsaicin concentration must be below 5 g/100 mL. The cost-effective evaluation of the compliance of self-defence weapons with legislation involves the use of two measurement procedures with increasing quantitative capability. Samples are first assessed by preliminary measurement procedure based on single-point GC–MS calibration. Whenever the measurement uncertainty from this assessment makes evaluation inconclusive, the evaluation of sample compliance using multi-point GC–MS calibration is performed. Metrological models including sound criteria for the evaluation of sample compliance with legislation were developed for both measurement procedures. Such models include the evaluation of the impact of instrumental performance, calibration model, sample dilution and standards preparation on measurement uncertainty. The relative expanded uncertainty, in the studied range (capsaicin 3–7 g/100 mL), of measurements supported in single-point calibrations ranged from 10 to 22% and the ones supported on multi-point calibrations from 8 to 12% depending on capsaicin concentration and daily GC–MS repeatability. Measurements are fit for the intended use since they present a relative expanded uncertainty smaller than a target value of 30, or 15%, for measurements supported in single- or multi-point calibrations, respectively.     

Determination of the standard deviation for proficiency assessment from past participant’s performances

The ??uncertainty function?? introduced by Thompson et al. estimates the reproducibility standard deviation as a function of concentration or mass fraction. This model was successfully applied to data derived from three proficiency testing schemes aiming at the quantification of cadmium, lead and mercury in blood and urine. This model allows the estimation of standard deviation for the performance assessment for proficiency testing rounds.     

Uncertainty estimation to evaluate mass balances on a combustion system

Mass balances of ash and potassium for a fluidized bed combustor were performed incorporating measurement uncertainties. The total output mass of ash or a chemical element should be equal to the mass in the input fuel; however, this is not often achieved. A realistic estimation of recovery uncertainty can support the reliability of a mass balance. Estimation of uncertainty helps to establish a reliable evaluation of the recovery ratio of ash mass and elemental mass. This may clarify whether any apparent lack in closing the mass balance can be attributed to uncertainties. The evaluation of measurement uncertainty for different matrices, namely coal, biomass, sand and ashes from different streams was based on internal quality control data and external quality control data, namely analysis of samples from proficiency tests or use of a certified reference material. The evaluation of intermediate precision and trueness allowed the estimation of measurement uncertainty. Due to the different physic and chemical characteristics of the studied matrices, the uncertainty of precision was evaluated using R-charts of data obtained from the analysis of duplicates for the majority of samples. This allowed evaluating sample heterogeneity effects. The instrumental acceptance criterion was also considered and included in the combined uncertainty. The trueness was evaluated using data from several proficiency tests and from analysis of a certified reference material or sample spiking. Statistically significant bias was included.     

From GUM to alternative methods for measurement uncertainty evaluation

Since the advent of the Guide to the expression of Uncertainty in Measurement (GUM) in 1995 laying the principles of uncertainty evaluation numerous projects have been carried out to develop alternative practical methods that are easier to implement namely when it is impossible to model the measurement process for technical or economical aspects. In this paper, the author presents the recent evolution of measurement uncertainty evaluation methods. The evaluation of measurement uncertainty can be presented according to two axes based on intralaboratory and interlaboratory approaches. The intralaboratory approach includes “the modelling approach” (application of the procedure described in section 8 of the GUM, known as GUM uncertainty framework) and “the single laboratory validation approach”. The interlaboratory approaches are based on collaborative studies and they are respectively named “interlaboratory validation approach” and “proficiency testing approach”.     

Proficiency testing in the light of a new rationale in metrology

The novel proposed definition of measurement result in the international metrology vocabulary requires a revision of standards and guidelines for proficiency testing (PT), and a new approach to processing proficiency data is needed to test the ability of laboratories to present not only unbiased quantity values, but reliable estimates of their uncertainty. Hence, an accepted reference value with the smallest possible uncertainty is needed to ascertain the proficiency of laboratories reporting results with lower than average uncertainty. A strategy based on the T-statistic is proposed leading to an accepted reference value that fully reflects the uncertainties reported by participants in a PT scheme and permits calculation of E _n numbers to distinguish whether or not measurement results are consistent with the accepted definition of the measurand. The strategy is applied to PT data from a recent international laboratory intercomparison of uranium isotopic ratios.     

Regular participation in proficiency testing provides long term improvements in laboratory performance: an assessment of data over time

Data from various proficiency testing schemes, operated by LGC Standards, was collated and reviewed to determine whether regular involvement within third party, proficiency testing, either over a prolonged period of time or via multiple participation, had improved the performance of those laboratories taking part. Three statistical evaluations were applied to historical PT results, a review of improvements over time, demonstrated by the evaluation of satisfactory performance scores achieved, a review of the relative robust standard deviations obtained from multiple methods and a focused review of one analyst participating in proficiency testing over a prolonged period of time. In each case the data indicated that long term participation and evaluation via proficiency testing had resulted in consistent and sustained improvements in laboratory performance.     

Assessment of the determination of heavy metals in organic soil improvers by ICP–OES

Directive 86/278/EEC sets maximum levels of heavy metals in sewage sludge used in agriculture to regulate the risk of using these organic soil improvers. The assessment of the compliance of soil improvers with this legislation should be supported on measurements with adequately low uncertainty. This work presents a strategy for assessing the performance of the determination of aqua regia extractable Cu, Zn and Cr in organic soil improvers and urban sewage sludges following EN 13650 standard. The measurement procedure validation involves checking the adequacy of the linear weighted regression model for ICP–OES calibration, the determination of the limit of quantification, the assessment of measurement repeatability, intermediate precision and trueness, and the evaluation of the measurement uncertainty using the differential approach. Routine tests quality control, including the estimated measurement uncertainty, is checked through the analysis of control standards, equivalent to calibrators, and reference materials from proficiency tests. Since the estimated relative expanded uncertainty is smaller than the defined target value (40 %), measurements are fit for assessing compliance of the mass fraction of heavy metals in organic soil improvers with the council directive.     

Proficiency testing and external quality assurance: crossing borders and disciplines

Proficiency testing and external quality assurance of medical laboratories is now entering its sixth decade. These activities comprise a broad range of applications including: providing participants and public health authorities with estimates of measurement uncertainty and national infrastructure; providing education; provision of a practical basis for accreditation and regulatory compliance. All branches of medical laboratory science have employed external quality assurance as a basis for improvement and comparability. The opportunities and challenges reviewed here include: the proper establishment of multiple target values in comparison to a system of traceability to reference or definitive methods; the problems of matrix effects and commutability of patient and proficiency test samples; generating information on laboratory infrastructure and trends in analytical technique and performance; providing education and setting goals for laboratory improvement; problems of specimen distribution; application of Internet technology; the role of programs in legal mandates and accreditation. Received: 24 April 2002 Accepted: 11 July 2002     

Approaching target uncertainty in proficiency testing schemes: experience in the field of water measurement

Good correspondence between evaluations of the proficiency testing data for water measurements according to different scoring schemes was found as a result of implementation of the measurement quality according to the target uncertainty defined in the Drinking Water Directive. The accuracy of the evaluation is determined by the uncertainty of the assigned value, and this uncertainty should be in correspondence with the target uncertainty. Modification of the zeta-score by the thorough application of the target uncertainty is presented. It is convenient for generalised presentation of the proficiency testing data. The allowed bias is included as a linear term on the measured value scale, as it is treated in uncertainty analysis. The importance of implementation of the target measurement uncertainty in other fields of the routine measurements is indicated.     

Measurement uncertainty of food carotenoid determination

For consistent interpretation of an analytical method result it is necessary to evaluate the confidence that can be placed in it, in the form of a measurement uncertainty estimate. The Guide to the expression of Uncertainty in Measurement issued by ISO establishes rules for evaluating and expressing uncertainty. Carotenoid determination in food is a complex analytical process involving several mass transfer steps (extraction, evaporation, saponification, etc.), making difficult the application of these guidelines. The ISO guide was interpreted for analytical chemistry by EURACHEM, which includes the possibility of using intra- and interlaboratory information. Measurement uncertainty was estimated based on laboratory validation data, including precision and method performance studies, and also, based on laboratory participation in proficiency tests. These methods of uncertainty estimation were applied to analytical results of different food matrices of fruits and vegetables. Measurement uncertainty of food carotenoid determination was 10–30% of the composition value in the great majority of cases. Higher values were found for measurements near instrumental quantification limits (e.g. 75% for β-cryptoxanthin, and 99% for lutein, in pear) or when sample chromatograms presented interferences with the analyte peak (e.g. 44% for α-carotene in orange). Lower relative expanded measurement uncertainty values (3–13%) were obtained for food matrices/analytes not requiring the saponification step. Based on these results, the saponification step should be avoided if food carotenoids are not present in the ester form. Food carotenoid content should be expressed taking into account the measurement uncertainty; therefore the maximum number of significant figures of a result should be 2.     

Measurement uncertainty evaluation and in-house method validation of the herbicide iodosulfuron-methyl-sodium in water samples by using HPLC analysis

A method for separation and quantitative determination of the iodosulfuron-methyl-sodium in water samples by high-performance liquid chromatography (HPLC) was developed and in-house validated in order to demonstrate its performance for monitoring of heterogeneous photocatalytic elimination of the herbicide iodosulfuron-methyl-sodium from water. Surface and ground water samples were used to demonstrate its selectivity, detection and quantification limits, linearity, trueness and precision. In addition, stability of iodosulfuron-methyl-sodium was studied in function of temperature and time. Method accuracy was quantified through measurement uncertainty estimate based on method validation data. The paper gives practical and easy to follow guidance on how uncertainty estimates can be obtained from method validation experiments. It shows that, if properly planned and executed, key precision and trueness studies undertaken for validation purposes can also provide much of the data needed to produce an estimate of measurement uncertainty. Our analytical protocol allowed us to quantify iodosulfuron-methyl-sodium in ground water and surface water in concentration level between 2.50–50.0 μmol L⁻¹ with satisfactory recoveries (99–104%) and repeatability lower or equal than 0.3% for all the matrices. We also estimated within-laboratory reproducibility over 3-month period, which was 0.7%. We proved that the method was selective for determination of iodosulfuron-methyl-sodium in the relevant matrices. Measurement uncertainty of results was evaluated to be 4.0% with 95% confidence level. After validation and measurement uncertainty evaluation steps, results obtained showed that the method can be applied to efficiently monitor heterogenous photocatalytic degradation of the herbicide iodosulfuron-methyl-sodium.     

Principles of the EURACHEM/CITAC guide “Use of uncertainty information in compliance assessment”

Many, possibly most, analytical measurements are carried out to assess compliance with a specification or a regulation, for example in the control contaminants in food or the detection of performance enhancing substances in sport. When making an assessment of compliance the presence of unavoidable measurement uncertainty introduces the risk of making incorrect decisions, that is of accepting a batch of material which is outside the specification or rejecting one that is within. This often leads to controversy over whether or not the compliance decision is correct. How to make reliable assessment decisions is described in the EURACHEM/CITAC Guide “Use of uncertainty information in compliance assessment”. The key is the use of decision rules that lead to an unambiguous interpretation of the measurement result and its uncertainty. These decision rules need to be designed to ensure that requirements of the specification or regulation are met and that the risk of making an incorrect decision is acceptable. Ideally they should form part of the specification or regulation. Presented at the Measurement Uncertainty Symposium, April 2008, Berlin, Germany.     

The determination of an accepted reference value from proficiency data with stated uncertainties

Proficiency data with stated uncertainties represent a unique opportunity for testing that the reported uncertainties are consistent with the Guide to the expression of uncertainty in measurement (GUM). In most proficiency tests, however, this opportunity is forfeited, because proficiency data are processed without regard to their uncertainties. In this paper we present alternative approaches for determining a reference value as the weighted mean of all mutually consistent results and their stated uncertainties. Using an accepted reference value each reported uncertainty estimate can be expressed as an E _n number, but a value of confirms its validity only if the uncertainty of the reference value is negligible in comparison.Reference values calculated for results from an International Measurement Evaluation Programme (IMEP-9) by “bottom up” as well as “top down” methods were practically identical, although the first strategy yielded the lowest uncertainty. A plot of individual coefficients of variation (CV) versus E _n numbers helps interpretation of the proficiency data, which could be used to validate relative uncertainties down to <1%.     

Evaluation of food emergency response laboratories’ capability for <Superscript>210</Superscript>Po analysis using proficiency test material with verifiable traceability

Measurement capability and data comparability are essential for emergency response when analytical data from cooperative laboratories are used for risk assessment and post incident decision making. In this study, the current capability of food emergency response laboratories for the analysis of ²¹⁰Po in water was evaluated using a proficiency test scheme in compliance with ISO-43 and ILAC G13 guidelines, which comprises a test sample preparation and verification protocol and an insightful statistical data evaluation. The results of performance evaluations on relative bias, value trueness, precision, false positive detection, minimum detection limit, and limit of quantification, are presented.