Visualisation of proficiency test exercise results in Kiri plots

A novel graphical method (‘Kiri plots’) for the presentation of proficiency test exercise results is presented. The Kiri plot visualises the evaluation of the proficiency test results based on three statistical tests (the z score, the zeta score and the relative uncertainty outlier test) by defining six zones including a central “in agreement” zone.     

A new approach in assessing microbiological results in water analysis proficiency testing

Due to the relatively large spread in the results of microbiological proficiency tests, the z-scores are often not able to detect zero or low results as being ‘bad’ results. This paper describes an adapted z-score based on the average or the standard deviation of the 50% ‘highest’ results. The combination of the adapted z-scores of four samples enables a better judgement of the performance of each laboratory. Papers published in this section do not necessarily reflect the opinion of the Editors, the Editorial Board or the Publisher.     

Use of uncertainty estimates as reported by participants in proficiency testing for the evaluation of their results: pros and cons

The aim of this paper is to study the measurement uncertainties reported in proficiency tests (PTs) using examples from PTs in the environmental sector and to compare the obtained measurement uncertainty estimates using different approaches. In addition, the paper focusses on the differences between the z-score and the zeta score. Since the year 2000, the Finnish Environment Institute has asked participants to report analytical methods as well as measurement uncertainties in connection with PT results. The measurement uncertainties of the assigned value have also been evaluated. On the basis of the results, the measurement uncertainties reported by the participants varied greatly. Participants often reported underestimated measurement uncertainties, but overestimated uncertainties were also reported. At the moment, it seems as if performance assessment should be based on the z-score because of a number of significant over- and underestimated measurement uncertainties. The zeta score should be used for information and educational purposes mainly.     

Performance of uncertainty evaluation strategies in a food proficiency scheme

A study of the performance of different uncertainty evaluation strategies among 163 voluntary respondents from food proficiency schemes is presented. Strategies included use of: single-laboratory validation data, quality control data, past proficiency testing data, reproducibility data, a measurement equation and the dispersion of replicate observations on the test material. Most performed reasonably well, but the dispersion of replicate observations underestimated uncertainty by a factor of approximately 3. Intended compliance with accreditation requirements was associated with significantly improved uncertainty evaluation performance, while intended compliance with the ISO “Guide to the expression of uncertainty in measurement” had no significant effect. Substituting estimates based on the Horwitz or Horwitz–Thompson models or on PT target standard deviation for the respondents’ own estimates of uncertainty led to a marked reduction in poor zeta scores and significant improvement in dispersion of zeta scores.     

Approaches to uncertainty evaluation based on proficiency testing schemes in chemical measurements

Since the advent of the Guide to the Expression of Uncertainty in Measurement (GUM) founding the principles of uncertainty evaluation, numerous projects have been carried out to develop alternative practical methods when it is impossible to model technical or economical aspects of the measurement process. These methods can use all the experimental data available to the laboratories, such as repeatability, reproducibility, quality-control charts, etc. The studies presented in this paper compare the results obtained by the modelling method from GUM with the uncertainties found by applying alternative methods. They show two examples, one in the field of environmental monitoring, the other in the biomedical field, based on the exploitation of PT schemes results. Presented at BERM-11, October 2007, Tsukuba, Japan.     

Comparison of different statistical methods for evaluation of proficiency test data

In analytical chemistry, proficiency testing usually consists in tests that laboratories conduct under routine conditions and report the result to the PT provider who then converts the result to a score which helps the participant to assess the accuracy of the result. The aim of this work is to show PT providers, accreditations bodies, and participating laboratories that different scoring results can be achieved depending on the evaluation system selected. The influence of different evaluation techniques on the results of an interlaboratory comparison for determination of gold in precious metals alloys was investigated. Results from 19 participating laboratories were evaluated by means of the three procedures: (1) classical statistical approach—outliers detection; (2) robust methods—(2A) robust procedure and (2B) ISO 13528; and (3) fitness for purpose. Evaluation of the same PT data revealed very interesting issues depending on the different scoring systems that were used and the robustness of the statistical methods used for detecting outliers. As a general rule, laboratories with scoring Z > 2 offered clearly poorer performance in robust approaches than classical ones. In order to support this first evidence, we evaluated a second data set with results from 24 laboratories (mercury from soil samples) by means of the four mentioned approaches. Selection and comparison of different scoring systems must be done very carefully, because sometimes they are not the best approach for studying the data population or the more appropriate one for evaluating the distribution of the data. Finally it should be taken into account that sometimes the robust scoring systems are not always suitable for evaluating the results of some PT schemes.     

The application of data from proficiency testing to laboratory accreditation according to ISO 17025

Current methods of testing laboratories for their proficiency in reporting correct measurement results are liable to substantial errors of the second kind. This means that laboratories with deflated uncertainties are accepted as proficient, even though their reported measurement results pave the way for erroneous conclusions. Only by using E _n numbers based on an accepted reference value with the lowest possible uncertainty can the risk of recognising incorrect measurement results be kept at an acceptable level. Based on an actual set of proficiency test (PT)-data for the concentration of Pb in water, this paper compares PT results obtained by methods using E _n numbers with methods based on z-scores. Kaj Heydorn is a technical assessor at The Danish Accreditation and Metrology Fund (DANAK).     

A model to set measurement quality objectives and to establish measurement uncertainty expectations in analytical chemistry laboratories using ASTM proficiency test data

A model is presented that correlates historical proficiency test data as the log of interlaboratory standard deviations versus the log of analyte concentrations, independent of analyte (measurand) or matrix. Analytical chemistry laboratories can use this model to set their internal measurement quality objectives and to apply the uncertainty budget process to assign the maximum allowable variation in each major step in their bias-free measurement systems. Laboratories that are compliant with this model are able to pass future proficiency tests and demonstrate competence to laboratory clients and ISO 17025 accreditation bodies. Electronic supplementary material to this paper can be obtained by using the Springer LINK server located at http://dx.doi.org/ 10.1007/s007690100398-y. Received: 31 March 2001 Accepted: 11 September 2001     

Long-term proficiency testing for the UK Acid Waters Monitoring Network

A series of proficiency test rounds to support the UK Acid Waters Monitoring Network has been conducted since the inception of the Network in 1988. These tests provide an illustration of the accuracy and comparability of analysis for laboratories that supply data to the Network. A summary of the tests is reported. Performance with respect to accuracy targets defined for the Network has been shown to be satisfactory for the majority of measurands of interest in studies of surface water acidification.     

The determination of an accepted reference value from proficiency data with stated uncertainties

Proficiency data with stated uncertainties represent a unique opportunity for testing that the reported uncertainties are consistent with the Guide to the expression of uncertainty in measurement (GUM). In most proficiency tests, however, this opportunity is forfeited, because proficiency data are processed without regard to their uncertainties. In this paper we present alternative approaches for determining a reference value as the weighted mean of all mutually consistent results and their stated uncertainties. Using an accepted reference value each reported uncertainty estimate can be expressed as an E _n number, but a value of confirms its validity only if the uncertainty of the reference value is negligible in comparison.Reference values calculated for results from an International Measurement Evaluation Programme (IMEP-9) by “bottom up” as well as “top down” methods were practically identical, although the first strategy yielded the lowest uncertainty. A plot of individual coefficients of variation (CV) versus E _n numbers helps interpretation of the proficiency data, which could be used to validate relative uncertainties down to <1%.     

Lessons from the first international proficiency test for the detection of spores from the honey bee pathogen Paenibacillus larvae

We report on the first proficiency test in the context of honey bee disease testing on a broad international scale. Honey samples were distributed to 12 participating laboratories for isolation and identification of spores of the etiological agent of American foulbrood, Paenibacillus larvae. Of the 11 laboratories responding to this proficiency testing event, only 6 (54%) provided data that were in full agreement with the previously determined status of the honey samples; eight (72%) laboratories fell within the threshold for qualification as defined a priori in this proficiency test (=level of agreement of at least 90%). Some lessons can now be drawn from the organizational point of view and will certainly improve new initiatives to be taken.     

Interpretation of interlaboratory comparison results to evaluate laboratory proficiency

Guidelines are given for the evaluation of proficiency test (PT) results in order to increase the effectivity of PT participation. For better understanding, some statistical background is given along with some examples to show the effects of the choices made by the PT provider. The calculation method of the assigned value and the selection of the standard deviation both affect the z-score that is used by the participating laboratory to judge the quality of its performance in the PT. Therefore, the participating laboratory is advised to use the PT results with care and, if necessary, to recalculate the z-scores. Finally, advice is given on how not to follow up bad PT results along with some valuable steps that could be part of an effective follow-up procedure.     

First proficiency test for the determination of NORM in contaminated soil from the oil field

The Atomic Energy Commission of Syria (AECS), in cooperation with the Arabic Atomic Energy Agency (AAEA), has initiated the first proficiency test exercise for the determination of naturally occurring radioactive materials (NORM) in contaminated soil from the oil field. The soil sample was collected from one of the most highly radioactively contaminated lagoons with production water in the Syrian oil fields, which was then prepared, characterised and certified according to a standard procedure. Samples were dispatched to laboratories from eight Arab countries; Kuwait, Egypt, Yemen, Tunisia, Jordan, the Kingdom of Saudi Arabia, Libya and Syria. The results were evaluated using three statistical criteria; z-score, the U _test score and the relative bias. These statistical methods were used to evaluate the performance of each laboratory, in addition to the overall evaluation for each radionuclide. This evaluation has indicated that 57% and 86% of the results passed the criteria set for precision and accuracy applied for this test in relation to ²²⁶Ra and ²²⁸Ra, respectively. These two radionuclides are considered to be the most important radionuclides in the oil industry.     

Performance indication improvement for a proficiency testing

The paper discusses peculiarities of Z scores and E _n numbers, which are most often used for the treatment of proficiency test data. The important conditions of proper usage of these performance indicators and their improvement are suggested on the basis of systematic approach, on the idea of accuracy classification, and on some principles of optimality borrowed from information theory. The author believes that this paper may be of interest and practical value for all those engaged in applied metrology, specifically in the field of developing of and participating in proficiency testing programs, and in the activity connected with accrediting testing and calibration laboratories.Presented at 15th International Conference on Quality in Israel, November 2004     

A new spline approach for data fitting

This paper presents an extension of the Modified Spline Technique (MST) formulation for data fitting named power spline, which gives consistent results for sets of data of concave and convex curves. This method is based on a technique which couples an implicit formulation of the maximum likelihood principle to the spline method, making the method suitable to fit data when no physical model is available. The MST method proved to be better than to the spline method and the extended spline fit technique (EST), because it provided accurate results for the first and second derivatives in sets of data where the EST solution developed inaccuracies. The EST method is a formulation that couples Least Squares to the spline method. There are, however, some sets of data where the MST method would show inconsistent solutions for the first and second derivatives. The power spline method eliminates these problems for concave or convex curves. Another improvement on the method is a more flexible choice for the interval boundaries.¹     

A CRL–IRMM joint proficiency test for trace elements in fish tissue

The Community Reference Laboratory (CRL) for residues and contaminants in food of animal origin at the Istituto Superiore di Sanità (ISS) plans and undertakes on a regular basis Proficiency Tests Schemes (PTSs) focused on the residues under its responsibility, as prescribed by the Council Directive 96/23/EC of April 29, 1996. In consideration of the benefits ensuing from a constant monitoring of the daily activities of the National Reference Laboratories (NRLs) for residues in the Member States, the ISS–CRL frequently prepares test materials to promote comparability of data and support harmonisation of routine analytical procedures. When target values for the analytes are set, the PTSs also provide objective assessment and improvement of the analytical trueness in the daily work.In this context, an interlaboratory comparison for trace elements in tuna fish matrix was organised in cooperation with the Institute for Reference Materials and Measurements (IRMM), Joint Research Centre, European Commission. Freeze-dried tuna fish samples were sent out to the NRLs for chemical elements in the Member States. Participants were requested to carry out at least three independent measurements of the elements As, Cd, Cr, Cu, Fe, Hg, Pb and Zn. The assessment of the proficiency was based on the z-score approach and a relative bias better than or equal to 10% of the target value was deemed to be an acceptable performance for this kind of measurements. On this basis, more than 50% of the laboratories returned accurate results for As, Cu, Fe, Hg and Zn, whereas the analysis of Cd, Cr and Pb appeared to be more problematic.     

A Bayesian approach to assess data from radionuclide activity analyses in environmental samples

A Bayesian statistical approach is introduced to assess experimental data from the analyses of radionuclide activity concentration in environmental samples (low activities). A theoretical model has been developed that allows the use of known prior information about the value of the measurand (activity), together with the experimental value determined through the measurement. The model has been applied to data of the Inter-laboratory Proficiency Test organised periodically among Spanish environmental radioactivity laboratories that are producing the radiochemical results for the Spanish radioactive monitoring network. A global improvement of laboratories performance is produced when this prior information is taken into account. The prior information used in this methodology is an interval within which the activity is known to be contained, but it could be extended to any other experimental quantity with a different type of prior information available.     

Proficiency testing in the light of a new rationale in metrology

The novel proposed definition of measurement result in the international metrology vocabulary requires a revision of standards and guidelines for proficiency testing (PT), and a new approach to processing proficiency data is needed to test the ability of laboratories to present not only unbiased quantity values, but reliable estimates of their uncertainty. Hence, an accepted reference value with the smallest possible uncertainty is needed to ascertain the proficiency of laboratories reporting results with lower than average uncertainty. A strategy based on the T-statistic is proposed leading to an accepted reference value that fully reflects the uncertainties reported by participants in a PT scheme and permits calculation of E _n numbers to distinguish whether or not measurement results are consistent with the accepted definition of the measurand. The strategy is applied to PT data from a recent international laboratory intercomparison of uranium isotopic ratios.     

鸡蛋中氟虫腈砜能力验证样品单元内与单元间均匀性评价案例与模型分析

对鸡蛋中氟虫腈砜能力验证样品进行单元内均匀性和单元间均匀性评价。对能力验证涉及的检测方法进行方法学考察,按照预设的模型,在单元内和单元间取样检测。使用单元内子样结果均方差和方法重复性均方差构建单因素方差分析模型,完成了单元内均匀性评价。对单元间和单元内均匀性评价的方差分析模型中组间和组内方差的来源及代表性进行了讨论。结果表明鸡蛋中氟虫腈砜样品的单元内均匀性和单元间均匀性满足要求。开展单元内均匀性评价有利于掌握能力验证方法性能,确认检测最小取样量,识别不均匀性来源,增加单元间均匀性评价的可靠性。建立的单元内均匀性评价模型克服了标准模型中多次转化测量数据难以获得的问题,可在各类能力验证项目中推广应用。