Evaluation of the use of consensus values in proficiency testing programmes

Proficiency testing (PT) is an essential tool used by laboratory accreditation bodies to assess the competency of laboratories. Because of limited resources of PT providers or for other reasons, the assigned reference value used in the calculation of z-score values has usually been derived from some sort of consensus value obtained by central tendency estimators such as the arithmetic mean or robust mean. However, if the assigned reference value deviates significantly from the ‘true value’ of the analyte in the test material, laboratories’ performance will be evaluated incorrectly. This paper evaluates the use of consensus values in proficiency testing programmes using the Monte Carlo simulation technique. The results indicated that the deviation of the assigned value from the true value could be as large as 40%, depending on the parameters of the proficiency testing programmes under investigation such as sample homogeneity, number of participant laboratories, concentration level, method precision and laboratory bias. To study how these parameters affect the degree of discrepancy between the consensus value and the true value, a fractional factorial design was also applied. The findings indicate that the number of participating laboratories and the distribution of laboratory bias were the prime two factors affecting the deviation of the consensus value from the true value.     

玩具油漆涂层中有机锡检测能力验证样品研制及应用分析

制备了用于可迁移有机锡检测能力验证的玩具油漆涂层样品,并开展了相关测定能力验证计划,采用迭代法、四分位法和敏感分析法3种不同统计方法对23家实验室数据进行统计分析,并对不满意结果的原因进行了分析。结果表明所研制样品的均匀性和稳定性良好,能满足能力验证要求。不同统计方法对实验室能力评定结果存在差异,但指定值差异很小,而能力评定标准差差异较大,运用卡方检验能选择出更合理的统计方法。     

An inter-laboratory comparison study for the determination of copper and lead from the wastewater of printed circuit board manufacturing industry in Pakistan

This inter-laboratory comparison study was arranged for 28 laboratories from different public and private sector organizations in Pakistan having wastewater testing capabilities aimed at improving the quality and comparability of test results. This national inter-laboratory study was started in December 2003 and completed in July 2004. Laboratories were invited to analyze the wastewater collected from printed circuit board (PCB) industry for lead and copper contamination. The samples fulfill the criteria for homogeneity and stability as done by the reference laboratory. The results obtained from participating laboratories were analyzed in terms of Hampel Test for outliers, while the performance evaluation of the participating laboratories was done on the basis of Z-score. An assigned value derived from the participant's results was compared with a reference value provided by a reference laboratory. Overall >50% of the participating laboratories have shown good performance in this PT-program     

The challenges and benefits of implementing the requirements of ISO/IEC 17043 by PT/EQA providers

ISO/IEC 17043 Conformity assessment??general requirements for proficiency testing is intended to replace previous international guides used to assess the competence of proficiency testing (PT) providers. It expands on the requirements of previous guides and is intended to accommodate PT providers of calibration laboratories and testing laboratories handling both qualitative and quantitative data. QMP-LS is an office-based external quality assessment provider for medical laboratories in Ontario, Canada and operates 46 different PT schemes for approximately 250 diagnostic tests. In 2010, these schemes were accredited to ISO/IEC 17043. Schemes included tests from the following disciplines: chemistry, hematology, microbiology, transfusion medicine, cytology, histology and genetics. Thirty of the schemes were qualitative. The challenges and benefits of implementing ISO/IEC 17043 are discussed, with particular emphasis on clauses addressing the following requirements: statistical design, determination of assigned value, homogeneity and stability testing, packaging, labeling and distribution, performance evaluation and subcontracting services.     

Proficiency testing in food microbiology: experience from implementation of ISO/IEC 17043 and ISO/TS 22117

The proficiency testing (PT) scheme ??AQUA?? for food microbiology was organised by the Istituto Zooprofilattico Sperimentale delle Venezie (IZSVe) according to ISO/IEC 17043 and ISO/TS 22117. This paper describes the IZSVe experience on the application of the above-mentioned standards for the PTs, with focus on the Enterobacteriaceae enumeration one. Freeze-dried food matrices contaminated with American Type Culture Collection bacterial strains were used as test samples for each microbiological PT organised by IZSVe. The sample homogeneity and stability were verified prior to distribution to participants and throughout the PT, respectively. The participating laboratories analysed samples using their routine methods, and results were transmitted to IZSVe. Data and methods used by each participating laboratory were analysed in order to evaluate the laboratory performance. With reference to the Enterobacteriaceae PT, the test samples were homogeneous and stable. In addition, most laboratory results were obtained using equivalent test methods. Statistical approaches applied to analyse data generated from all participating laboratories revealed similar outcomes as no significant outlying count and only 5?% of unacceptable results were observed. Finally, the z-score, with the standard deviation that does not vary from round toround, was applied to compare and to evaluate the performance of each laboratory over time highlighting possible persistent trends over several rounds.     

Capability of laboratories to determine core nutrients in foods: results of an international proficiency test

A proficiency test (PT) to assess the capabilities of laboratories to determine nutrients in a biscuit sample was carried out in September 2009. The need for such interlaboratory comparison arose from an increasing nutrition labeling requirements of different countries around the world. Forty-eight laboratories worldwide participated in this PT program for the determination of total lipids, saturated fat, trans-fat, protein, sugars, total dietary fiber (TDF), ash and sodium in food. This program was organized under the auspices of the Asia-Pacific Laboratory Accreditation Cooperation (APLAC). APLAC considers PT programs as one of the objective means of assessing the performance of the accredited laboratories. The program is one of the APLAC PT series whose primary purposes are to establish mutual agreement on the equivalence of the operation of APLAC member laboratories and to provide an opportunity to identify testing deficiency so that corrective actions can be taken, if necessary. The results of this program indicated that the consensus mean values estimated by robust statistics were in good agreement with the corresponding values obtained in the homogeneity tests except for trans-fat and TDF. The relative standard deviations of participant results for trans-fat and TDF were also found to be higher (19 and 24%, respectively) than the other test parameters. The relative standard deviations for other test parameters were less than 10%. One-half of the laboratories obtained satisfactory z-scores (| z | < 3) (\left| z \right| < 3) for all their reported results. Although participants were instructed to provide the measurement uncertainties of their reported results, only 27.5% of them had done so. Albeit many of the measurement uncertainties appear to be reasonable, some of them were found to vary widely, indicating that some laboratories still have difficulties in estimating measurement uncertainties. Although the overall measurement capability can be considered to be satisfactory, there is still room for improvement of analytical procedures.     

Performance evaluation for proficiency testing with a limited number of participants

Proficiency testing (PT) is an essential tool for laboratories to assess their competency. Also, participation in PT has become one of the mandatory requirements for laboratory to seek accreditation according to ISO/IEC 17025. For this reason, the effectiveness of performance evaluation by PT scheme is of great concern for the participants and for accreditation bodies as well. In practice, owing to unavailability of other appropriate alternatives, PT scheme providers may have to choose using consensus values to evaluate the performance of participants. However, such consensus values approach was not recommended by relevant international guidelines for PT schemes with limited number of participants. With the use of Monte Carlo simulation technique, this study attempted to investigate the effectiveness of using consensus values for performance evaluation in PT schemes with limited number of participants. The simulation process was schemed according to the statistical model provided by ISO 5725-1 for laboratory measurement results, which covered components like method bias, laboratory bias, and measurement precision. The effectiveness of the consensus value approach was expressed as the percentage of participants in a simulation run could get the same evaluation result, either satisfactory or unsatisfactory, against the “true value.” The findings indicated that the number of participants, choice of consensus values, mass fraction of analyte, method bias, laboratory bias, and measurement repeatability of participating laboratories would all affect the effectiveness of the consensus value approach but at different extent. However, under certain circumstances, use of consensus value could still be considered as an acceptable approach for performance evaluation even the number of participants was limited. Some of the findings were further verified using real data from PT schemes where appropriate certified reference materials or reliable reference values were available.     

Performance evaluation of polycyclic aromatic hydrocarbons analysis in sediment from proficiency testing with metrological reference values

This paper describes a metrological approach to evaluate the measurement capability of laboratories participating in two proficiency testing (PT) programmes involving the analysis of five polycyclic aromatic hydrocarbons (PAHs) in sediment samples. Reference values of PAHs in the programmes for performance assessment were obtained from an accurate isotope dilution gas chromatography mass spectrometry (ID-GCMS) method which was thoroughly validated and verified. Isotope dilution mass spectrometry (IDMS) technique usually has a well-defined measurement uncertainty budget and a traceability link to an International System of Units. Provision of the metrological reference values in PT enables the establishment of a technical platform to assess the actual competence of the participating laboratories in sediment PAHs analysis. Results of the PT programmes showed that about 80 % of the laboratories employed gas chromatography in their analyses and the remaining used liquid chromatography. Irrespective of the techniques being used, however, the majority of the participating laboratories were observed to underestimate values in which the mean values of the five reported PAHs were less than those of the ID-GCMS-derived reference values by 13–20 %. Only 41–44 % of the participating laboratories were able to achieve satisfactory z-scores. The present study revealed that the reinforcement of the capability for accurate measurement of PAHs in sediment samples in laboratories worldwide should be addressed.     

Note on the bimodal distribution of PT results

The practitioner’s report "Treatment of bimodality in proficiency test of pH in bioethanol matrix" by Sarmanho et al. deals with a proficiency test (PT) for the measurement of pH in bioethanol. One group of 19 participating laboratories used an electrode that contained saturated LiCl and the other one KCl. The results of the LiCl group are biased, and those of the KCl group are unbiased as compared with the assigned reference value \(y_\mathrm{{RM}}\), and hence, the distribution of the laboratory means is bimodal. In order to deal with this situation, Sarmanho et al. present and carry out two different methods of fitting the bimodal distribution of the laboratory means: a fit by a mixture of two normal distributions and a Gaussian kernel estimation. Since this rather advanced statistical analysis does not help to solve the practical problem, some recomendations of ISO 17043 are discussed that might be useful for the coordinator of the PT and as well for the participating laboratories.     

Preparation of in-house reference material of benzylpenicillin in milk and results of a Brazilian proficiency testing scheme

Two milk test materials containing benzylpenicillin and a benzylpenicillin-free material were prepared and used for the operation of a Brazilian proficiency testing (PT) scheme according to the requirements of ISO/IEC 17043. The PT scheme was designed to include laboratories in charge of confirmatory analysis as well as screening analysis. Sets of sample vials, benzylpenicillin-free and spiked items containing 0.2 g of lyophilized milk, were distributed to each participant. Of 20 participants who reported screening data, 18 accomplished satisfying results, providing evidence of the capability of Brazilian laboratories to produce reliable qualitative information. The assigned value (robust average calculated from 6 results) and its uncertainty were (8.28 ± 0.52) μg/L. Participants’ performance was evaluated using z-scores. A small number of participants were able to report quantitative benzylpenicillin results, and consequently care should be taken in the interpretation of the laboratory’s performance, considering the statistical nature of z-scores. Long-term stability testing on remaining PT items showed that a stable benzylpenicillin quality control material was produced and can be used for internal quality control or validation purposes.     

Proficiency testing program for the determination of total arsenic, cadmium, and lead in seawater shrimp

The proficiency testing (PT) program for 97 worldwide laboratories for determining total arsenic, cadmium, and lead in seawater shrimp under the auspices of the Asia-Pacific Laboratory Accreditation Cooperation (APLAC) is discussed. The program is one of the APLAC PT series whose primary purposes are to establish mutual agreement on the equivalence of the operation of APLAC member laboratories and to take corrective actions if testing deficiencies are identified. Pooled data for Cd and Pb were normally distributed with interlaboratory variations of 21.9 and 34.8%, respectively. The corresponding consensus mean values estimated by robust statistics were in good agreement with those obtained in the homogeneity tests. However, a bimodal distribution was observed from the determination of total As, in which 14 out of 74 participants reported much smaller values (0.482-6.4 mg/kg) as compared with the mean values of 60.9 mg/kg in the homogeneity test. The use of consensus mean is known to have significant deviation from the true value in bi- or multimodal distribution. Therefore, the mode value, a better estimate of central tendency, was chosen to assess participants' performance for total As. Estimates of the overall uncertainty from participants varied in this program, and some were recommended to acquire more comprehensive exposure toward important criteria as stipulated in ISO/IEC 17025.     

Breakfast cereal sampling study for nutritional elements

The National Institute of Standards and Technology (NIST) has established a Dietary Supplement Laboratory Quality Assurance Program (DSQAP) in collaboration with the National Institutes of Health Office of Dietary Supplements (NIH-ODS). The DSQAP invites laboratories twice annually to participate in interlaboratory studies where participants elect to measure concentrations of nutritional and/or toxic elements as well as active and/or marker compounds. One of these studies was designed to determine the effects of material granularity and sample processing techniques on measurement variability (precision) as well as to provide participating laboratories information on their performance relative to the NIST assigned values (bias) and to the other participants (concordance). Participants were asked to determine the mass fractions of Ca, Fe, and Zn, in mg/kg, in six breakfast cereal samples. Cereal samples consisted of three ground materials (homogenized wheat, wheat, and rice), two flake materials (wheat and rice) and a partially crushed material (a wheat/rice mixture). In general, approximately 25 % of the laboratories processed and analyzed the suite of six cereal materials with adequate to exemplary measurement precision. Over half of the laboratories (60 %) experienced measurement issues related to only a particular type of cereal matrix or for only a single element. A small number (15 %) of laboratories experienced significant sample processing or measurement problems. Future studies planned by the DSQAP may be designed to use commercial products to aid laboratories with their sampling and analytical techniques.

Establishment of an assigned value and its uncertainty for tumour markers in proficiency testing in China

Proficiency testing (PT) is an important aspect of clinical laboratory quality control for tumour markers. To evaluate the performance of an individual laboratory, the assigned value and the standard uncertainty of each tumour marker must be determined. However, no data about the standard uncertainty of the assigned value of any analyte in PT is available in China. The aim of this paper is to introduce a new approach for establishment of assigned value and its standard uncertainty for tumour markers. Five sets of control materials at different concentrations were assigned to each laboratory that participated in PT for tumour markers in 2012. We collected the data from each laboratory using the Clinet-EQA reporting system V1.5 and Clinet-EQA evaluation system V1.0 and developed a software program to calculate the assigned value and its uncertainty according to ISO 13528 (Statistical methods for use in proficiency testing by interlaboratory comparisons. ISO 13528. International Standards Organisation, Geneva, 2005). The assigned value and inter-laboratory CV calculated by the new approach was compared with that obtained by the current approach used in PT in our country. No significant difference between the two outcomes was found in our study. Most assigned values had a standard uncertainty that can meet the criterion: $ u_{X} \le \, 0.3\hat{\sigma } $ u X ≤ 0.3 σ ? , which meant the new approach for establishment of the assigned value was acceptable. It is expected that the new approach to determination of the assigned value and its standard uncertainty for PT for tumour markers in China will enable participants to evaluate their measurement uncertainty.     

Development of a certified reference material for the determination of acrylamide in potato chips

A certified reference material (CRM), KRISS CRM 108-10-003, has been developed for analysis of acrylamide in potato chips, as a representative of carbohydrate-rich food cooked in high-temperature oil. The material was prepared by grinding commercially available potato chips to a paste which was then homogenized, bottled in 15-g units, and stored at ?70 °C. Certification, homogeneity and stability testing, were carried out by liquid chromatography–isotope-dilution mass spectrometry (ID-LC–MS). A single ID-LC–MS measurement was performed for each of 10 selected units for certification and homogeneity assessment. The mean measurement result for the 10 bottles, 0.455?±?0.012 mg?kg^?1, was assigned as the certified value of the CRM. The between-bottle homogeneity was 0.8% of the certified value. The within-bottle homogeneity, tested by measuring three replicate sub-samples from each of three randomly selected bottles, was similar to the between-bottle homogeneity. The stability of the CRM under storage conditions (?70 °C) was tested for 21 months and no change in the acrylamide content was observed within the measurement uncertainty. Stability of the CRM at –20 °C (storage at user’s site) and room temperature (for regular use and transportation) was also tested. Also presented is the newly designed procedure for evaluating the uncertainty of the certified value for the characterization scheme used in this study.

Measurement of PAHs in environmental matrices: results from an interlaboratory comparison on the different steps of the measurement procedure

Proficiency testing (PT) is becoming a feature of laboratory accreditation and the PT results are used to assess the technical competence of the participating laboratories. ISPRA (former APAT) plays an important role in supporting the Italian laboratories belonging to the network of the Regional Environmental Agencies to improve the quality of their analytical measurements. As a consequence, ISPRA organized an interlaboratory comparison to assess the performance of the laboratories on PAH measurement procedure. The interlaboratory comparison was drawn separating the different steps of the measurement (from the extraction to the instrumental measurement). Two matrix reference materials: (1) a polluted soil and (2) an extract reference material of the same polluted soil and a “blind” PAH mixture standard stock solution were distributed to 59 Italian laboratories. The results of interlaboratory comparison showed a significant dispersion of the PAH measurements that masks the effects of the different extraction and cleanup procedures used, but it is consistent with the results of other European interlaboratory comparisons.     

Comparability and compatibility of proficiency testing results in schemes with a limited number of participants

Comparability and compatibility of proficiency testing (PT) results are discussed for schemes with a limited number of participants (less than 20–30) based on the use of reference materials (RMs) as test items. Since PT results are a kind of measurement/analysis/test result, their comparability is a property conditioned by traceability to measurement standards applied in the measurement process. At the same time, metrological traceability of the certified value of the RM (sent to PT participants as test items) is also important, since the PT results are compared with the RM certified value. The RM position in the calibration hierarchy of measurement standards sets the degree of comparability for PT results, which can be assessed in the scheme. However, this assessment is influenced by commutability (adequacy or match) of the matrix RM used for PT and routine samples. Compatibility of PT results is a characteristic of the collective (group) performance of the laboratories participating in PT that can be expressed as closeness of the distribution of the PT results to the distribution of the RM data. Achieving quality-of-measurement/analysis/test results in the framework of the concept “tested once, accepted everywhere” requires both comparability and compatibility of the test results.     

Evaluation of proficiency tests in microbiological analysis: enumeration of aerobic microorganisms

South Korea’s Ministry of Food and Drug Safety (MFDS) has been developing programs for the inspection and accreditation of food sanitation inspection institutions. Food sanitation inspection institutions such as MFDS regional offices, the Research Institute of Public Health and Environment and authorized private service providers in South Korea must participate in proficiency testing (PT) programs to comply with the Food Sanitation Act and MFDS Notification No. 2012-112. As the PT provider, the MFDS annually plans various microbiological and chemical PT programs for foods, cosmetics, and pharmaceutical products in accordance with ISO/IEC 17043. The aim of this project was to evaluate the performance of microbiological PT programs to ensure the quality of their routine test results. The test materials used were freeze-dried BioBalls from BTF Pty Ltd. Homogeneity and stability were investigated to assess the adequacy of the selected test materials. This project also contains data from inter-laboratory comparisons organized by MFDS in 2011 and 2012. More than 50 laboratories attended the PT program and submitted their results. Laboratory results were rated with z-scores according to the international standard ISO 13528. The results from 2011 and 2012 revealed that all participating laboratories had similar levels of proficiency. Most of the participants received a rating of “Satisfactory.” Moreover, the percentage of participants who received a rating of “Unsatisfactory” decreased from 3.5 % in 2011 to 2.0 % in 2012.     

The effect of the choice of method for determining assigned value on the assessment of performance in melting point analysis

The determination of melting point is a fundamental test in the Pharmaceutical industry, since it is one of the simplest techniques for the identification of a chemical substance. The melting point provides information on both identity and purity of a chemical substance and for that reason is a key test in the PHARMASSURE proficiency testing (PT) scheme. The PT scheme assesses participant??s determination of melting point, using chemicals of high purity and basing the assigned value on the robust consensus mean (median). In recent rounds, melting point reference standards have been provided as the test material and a reference value used as the assigned value for PT assessment. Comparison of the PT results over a number of rounds, using test materials with a wide range of melting points, shows the overall performance of the participant group is worse in rounds where a reference material and associated reference assigned value are used for performance assessment. When participants were assessed against the reference assigned value, a positive bias was observed in the participant??s results. Detailed information regarding the methodology used demonstrated that the majority of participants use the same analytical method, EU.Ph.2.2.14 (Council of Europe, Strasbourg, 2011), for the determination of melting point although this procedure allows flexibility in key methodological parameters, such as heating ramp rate, which may fail to ensure consistent performance across the group of participant laboratories.     

Bias analysis of proficiency testing programme results

A proficiency testing programme might involve a great number of participating laboratories coming from different countries or regions, and normally they analysed the same test materials using their own routine analytical methods. Hence, the results of a proficiency testing programme may contain valuable information which could serve purposes other than just performance evaluation. This study attempted to extract information from the results of a proficiency testing programme for the purposes of educating the participating laboratories as suggested by ISO/IEC 17043. The “bias analysis” approach introduced in this study was based on the statistical model of measurement and the nature of bias in chemical analysis. With this approach, the participating laboratories could estimate the bias associated with different settings of experimental conditions according to the statistics of subset distribution of the reported results from the participating laboratories. This would be useful for them to review the analytical procedures they used and modify their methods if needed. The approach was applied to the analysis of data obtained from a number of past proficiency testing programmes, and the findings were discussed in this paper.