Performance of commercial soil laboratories in a proficiency test program in Brazil

A soil proficiency test (PT) was administered to 50 participant laboratories in which two sets of samples, consisting of 20 yearly PT samples and 5 ‘blind’ samples in clients’ names were analyzed for pH, organic matter, total acidity, extractable calcium, magnesium, potassium and phosphorus by the laboratories. Our objective was to determine whether laboratories take extra care to analyze clients’ samples as they do with regular PT samples. The analytical data were evaluated essentially by the procedure described in the international harmonized protocol for proficiency testing of analytical chemistry laboratories. Performance of participant laboratories was assessed by z-scores and summary z-scores statistics involving sum of squared z-scores interpreted as chi-square ( c_n² ) \left( {\chi_{n}^{2} } \right) distribution for zero-centered z-scores with unit variance. From 8 750 determinations, outliers and stragglers accounted for less than 2% of the entire data. Over 93% of the data were satisfactory, whereas between 2 and 4% were either unsatisfactory or questionable in both the PT and ‘blind’ tests. On the basis of sum of squared z-scores interpreted from c_n² \chi_{n}^{2} distribution table, between 30 and 40% of the laboratories had more than 90% probability of having their measurement data within the robust mean and standard deviation for each soil parameter, while another 30–42% of the laboratories had less than 50% probability of having measurement data within the robust mean and standard deviation. Overall, 21 laboratories (42%) were ranked in Class A either in the PT or ‘blind’ tests out of which 12 of them (57%) retained this ranking in both tests. Fourteen laboratories (28%) were ranked in Class C in either the PT or ‘blind’ tests with only 5 of them (36%) consistently ranking in this class in both tests.     

Proficiency testing: binary data analysis

A method for evaluating qualitative proficiency testing (PT) of laboratories conducting binary tests is proposed. The method is based on the scale-invariant item response model proposed by the authors in earlier publications. We consider the case where the laboratories under the PT conduct test consisting of a set of test items/species presenting different, but unknown beforehand levels of difficulty when trying to detect a particular property of theirs, and we need to evaluate/compare both the intrinsic abilities of the participating laboratories and the level of difficulty of the test items. We assume that the responses to different test items do not affect one another and discuss how to get and interpret the most likely estimates/scores. The method is illustrated by the example presented in a recent publication by our colleagues from QuoData GmbH and can be considered as an alternative to that proposed in their publication method of scoring.     

Using Bayes methods and mixture models in inter-laboratory studies with outliers

Inter-laboratory studies (especially so-called key comparisons) are conducted to evaluate both national and international equivalence of measurement. In these studies, a reference value of some measurand (the quantity intended to be measured) is developed and results for all laboratories are compared to this single value. How to determine the reference value is not completely obvious if there are observations and/or laboratories that could be considered outliers. Since ignoring results from one or more participating laboratories is untenable in practical terms, developing methods that are robust to the possibility that a small fraction of the laboratories produces observations unlike those from the others is critical. This paper outlines two Bayesian methods of analyzing inter-laboratory data that have been proposed in the literature and suggests three modifications of one that are more robust to outliers. A simulation study is conducted to compare the five methods.     

Official proficiency tests of the organisation for the prohibition of chemical weapons: current status and future directions

The Organisation for the Prohibition of Chemical Weapons (OPCW) has been organising and conducting Official Proficiency Tests (PT) since 1996 in accordance with ILAC-G13 to certify laboratories for the analysis of authentic samples under the provision of Chemical Weapons Convention. The tests are part of a mechanism to ensure that there are laboratories that have proven competence in the analysis of chemicals related to the Convention. Laboratories that have successfully completed the tests are designated by the Director General of the OPCW for analysis of authentic samples. To maintain the Designated Laboratory status, a laboratory must take and pass at least one of the two proficiency tests offered per calendar year. Unlike many proficiency tests, the OPCW PT is qualitative, that is the laboratories must determine if any of a very large set (essentially infinite) of chemicals relevant to the Convention are present in the samples. The tests are organised with the assistance of two laboratories, one preparing the test samples, and the other evaluating the test results. The paper provides an overview of the current status of these PTs and outlines salient features regarding procedure and scope of tests, selection of assisting laboratories, scoring and performance rating of participating labs. The emerging issues are also briefly discussed.     

An inter-laboratory comparison study for the determination of copper and lead from the wastewater of printed circuit board manufacturing industry in Pakistan

This inter-laboratory comparison study was arranged for 28 laboratories from different public and private sector organizations in Pakistan having wastewater testing capabilities aimed at improving the quality and comparability of test results. This national inter-laboratory study was started in December 2003 and completed in July 2004. Laboratories were invited to analyze the wastewater collected from printed circuit board (PCB) industry for lead and copper contamination. The samples fulfill the criteria for homogeneity and stability as done by the reference laboratory. The results obtained from participating laboratories were analyzed in terms of Hampel Test for outliers, while the performance evaluation of the participating laboratories was done on the basis of Z-score. An assigned value derived from the participant's results was compared with a reference value provided by a reference laboratory. Overall >50% of the participating laboratories have shown good performance in this PT-program     

Evaluation of the use of consensus values in proficiency testing programmes

Proficiency testing (PT) is an essential tool used by laboratory accreditation bodies to assess the competency of laboratories. Because of limited resources of PT providers or for other reasons, the assigned reference value used in the calculation of z-score values has usually been derived from some sort of consensus value obtained by central tendency estimators such as the arithmetic mean or robust mean. However, if the assigned reference value deviates significantly from the ‘true value’ of the analyte in the test material, laboratories’ performance will be evaluated incorrectly. This paper evaluates the use of consensus values in proficiency testing programmes using the Monte Carlo simulation technique. The results indicated that the deviation of the assigned value from the true value could be as large as 40%, depending on the parameters of the proficiency testing programmes under investigation such as sample homogeneity, number of participant laboratories, concentration level, method precision and laboratory bias. To study how these parameters affect the degree of discrepancy between the consensus value and the true value, a fractional factorial design was also applied. The findings indicate that the number of participating laboratories and the distribution of laboratory bias were the prime two factors affecting the deviation of the consensus value from the true value.     

Comparability and compatibility of proficiency testing results in schemes with a limited number of participants

Comparability and compatibility of proficiency testing (PT) results are discussed for schemes with a limited number of participants (less than 20–30) based on the use of reference materials (RMs) as test items. Since PT results are a kind of measurement/analysis/test result, their comparability is a property conditioned by traceability to measurement standards applied in the measurement process. At the same time, metrological traceability of the certified value of the RM (sent to PT participants as test items) is also important, since the PT results are compared with the RM certified value. The RM position in the calibration hierarchy of measurement standards sets the degree of comparability for PT results, which can be assessed in the scheme. However, this assessment is influenced by commutability (adequacy or match) of the matrix RM used for PT and routine samples. Compatibility of PT results is a characteristic of the collective (group) performance of the laboratories participating in PT that can be expressed as closeness of the distribution of the PT results to the distribution of the RM data. Achieving quality-of-measurement/analysis/test results in the framework of the concept “tested once, accepted everywhere” requires both comparability and compatibility of the test results.     

Development of a proficiency testing scheme for a limited number of participants in the field of natural water analysis

A proficiency testing (PT) scheme was developed for a limited number of analytical laboratories participating in the analysis of natural water in Israel. Three fit-for-intended-use reference materials (RMs) were prepared for a pilot PT from natural water: RM-blank and two in-house reference materials (IHRMs) with seven analytes fortified to achieve different levels of analyte concentrations. The measurands for the PT participants were the traceable spike values certified in the IHRMs, i.e., added mass concentrations of the analytes. The RM-blank and IHRMs were found to be homogeneous and stable over 11 days, the time necessary for the experimental part of the PT. The RMs were distributed to the PT participants as unknown test items similar to routine samples. The test results were evaluated using their deviations from the IHRM-certified values. Eight Israeli laboratories took part in the interlaboratory comparison. Individual laboratory performance and metrological compatibility of the PT results of the participants, as a local group of laboratories, were evaluated for every analyte.     

Quantitation of 35S promoter in maize DNA extracts from genetically modified organisms using real-time polymerase chain reaction, part 2: interlaboratory study

The European Committee for Standardization (CEN) and the European Network of GMO Working Laboratories have proposed development of a modular strategy for stepwise validation of complex analytical techniques. When applied to the quantitation of genetically modified organisms (GMOs) in food products, the instrumental quantitation step of the technique is separately validated from the DNA extraction step to better control the sources of uncertainty and facilitate the validation of GMO-specific polymerase chain reaction (PCR) tests. This paper presents the results of an interlaboratory study on the quantitation step of the method standardized by CEN for the detection of a regulatory element commonly inserted in GMO maize-based foods. This is focused on the quantitation of P35S promoter through using the quantitative real-time PCR (QRT-PCR). Fifteen French laboratories participated in the interlaboratory study of the P35S quantitation operating procedure on DNA extract samples using either the thermal cycler ABI Prism 7700 (Applied Biosystems, Foster City, CA) or Light Cycler (Roche Diagnostics, Indianapolis, IN). Attention was focused on DNA extract samples used to calibrate the method and unknown extract samples. Data were processed according to the recommendations of ISO 5725 standard. Performance criteria, obtained using the robust algorithm, were compared to the classic data processing after rejection of outliers by the Cochran and Grubbs tests. Two laboratories were detected as outliers by the Grubbs test. The robust precision criteria gave values between the classical values estimated before and after rejection of the outliers. Using the robust method, the relative expanded uncertainty by the quantitation method is about 20% for a 1% Bt176 content, whereas it can reach 40% for a 0.1% Bt176. The performances of the quantitation assay are relevant to the application of the European regulation, which has an accepted tolerance interval of about +/-50%. These data were fitted to a power model (r2 = 0.96). Thanks to this model, it is possible to propose an estimation of uncertainty of the QRT-PCR quantitation step and an uncertainty budget depending on the analytical conditions.     

Performance evaluation for proficiency testing with a limited number of participants

Proficiency testing (PT) is an essential tool for laboratories to assess their competency. Also, participation in PT has become one of the mandatory requirements for laboratory to seek accreditation according to ISO/IEC 17025. For this reason, the effectiveness of performance evaluation by PT scheme is of great concern for the participants and for accreditation bodies as well. In practice, owing to unavailability of other appropriate alternatives, PT scheme providers may have to choose using consensus values to evaluate the performance of participants. However, such consensus values approach was not recommended by relevant international guidelines for PT schemes with limited number of participants. With the use of Monte Carlo simulation technique, this study attempted to investigate the effectiveness of using consensus values for performance evaluation in PT schemes with limited number of participants. The simulation process was schemed according to the statistical model provided by ISO 5725-1 for laboratory measurement results, which covered components like method bias, laboratory bias, and measurement precision. The effectiveness of the consensus value approach was expressed as the percentage of participants in a simulation run could get the same evaluation result, either satisfactory or unsatisfactory, against the “true value.” The findings indicated that the number of participants, choice of consensus values, mass fraction of analyte, method bias, laboratory bias, and measurement repeatability of participating laboratories would all affect the effectiveness of the consensus value approach but at different extent. However, under certain circumstances, use of consensus value could still be considered as an acceptable approach for performance evaluation even the number of participants was limited. Some of the findings were further verified using real data from PT schemes where appropriate certified reference materials or reliable reference values were available.     

Environmental analysis in the Czech Republic: organisation of proficiency testing and its use in laboratory assessment

The history, origin, and development of a system for monitoring and assessing water and other environmental laboratories in the Czech Republic is described. The system started in 1991 and has matured to its present complexity with similarities to the accreditation systems found in other countries. Differences from internationally recognized procedures are being corrected step by step. During the first year of its existence ASLAB, as part of its brief, organised proficiency testing (PT) programs for fifty laboratories. Today the total number of regularly participating laboratories exceeds 700 from the Czech Republic, the Slovak Republic, and Germany. This paper describes the ASLAB PT system, discusses some experiences with its use, and describes the use of PT results in assessment of the competence of laboratories. Received: 12 October 2000 Accepted: 7 January 2001     

Proficiency testing in food microbiology: experience from implementation of ISO/IEC 17043 and ISO/TS 22117

The proficiency testing (PT) scheme ??AQUA?? for food microbiology was organised by the Istituto Zooprofilattico Sperimentale delle Venezie (IZSVe) according to ISO/IEC 17043 and ISO/TS 22117. This paper describes the IZSVe experience on the application of the above-mentioned standards for the PTs, with focus on the Enterobacteriaceae enumeration one. Freeze-dried food matrices contaminated with American Type Culture Collection bacterial strains were used as test samples for each microbiological PT organised by IZSVe. The sample homogeneity and stability were verified prior to distribution to participants and throughout the PT, respectively. The participating laboratories analysed samples using their routine methods, and results were transmitted to IZSVe. Data and methods used by each participating laboratory were analysed in order to evaluate the laboratory performance. With reference to the Enterobacteriaceae PT, the test samples were homogeneous and stable. In addition, most laboratory results were obtained using equivalent test methods. Statistical approaches applied to analyse data generated from all participating laboratories revealed similar outcomes as no significant outlying count and only 5?% of unacceptable results were observed. Finally, the z-score, with the standard deviation that does not vary from round toround, was applied to compare and to evaluate the performance of each laboratory over time highlighting possible persistent trends over several rounds.     

Proficiency testing of chloride content in different types of Portland cement

The accurate determination of chloride content in cement is a prerequisite for the production of high-quality concrete due to the damaging influence of free chloride ions on durability and safety of concrete constructions. The results for total chloride content in four different types of Portland cement provided by testing laboratories participating in an inter-laboratory comparison are presented. The data sets were evaluated by using different statistical methods. The laboratories exceeding critical values were signaled and sources of error were investigated. The precision parameters of the method were determined and the variance of the entire data set was apportioned among the sources of variation. The systematic differences between participating laboratories and heterogeneity of the cement samples were identified as major sources of variability of results. The ANOVA determined the intermediate precision parameters, which showed a low value for random errors resulting in good repeatability and high impact of sample and inter-laboratory difference on reproducibility of the method.Presented at the Eurachem PT Workshop September 2005, Portorož, Slovenia     

Evaluation of the long-term performance of water-analyzing laboratories

After 7 years of successful operation, the database of the proficiency testing (PT) scheme for water analysis organized by the Institute for Agrobiotechnology (IFA), Tulln contains nearly 4,000 data sets from over 300 interlaboratory comparison samples. About 70 analytical parameters (major ions, metals, trace ions, herbicides, volatile halogenated hydrocarbons, organochlorine pesticides and polycyclic aromatic hydrocarbons) were covered by the PT scheme. The data were evaluated using robust statistics in order to determine a set of coefficients of variation (CV) for each analytical parameter. Concentration and time dependence of the CV were checked. The CV were combined to obtain standard deviations for proficiency assessment (Z-score criteria). Furthermore, a viewer program was designed to facilitate monitoring of the analytical performance of participating laboratories.     

Performance evaluation of polycyclic aromatic hydrocarbons analysis in sediment from proficiency testing with metrological reference values

This paper describes a metrological approach to evaluate the measurement capability of laboratories participating in two proficiency testing (PT) programmes involving the analysis of five polycyclic aromatic hydrocarbons (PAHs) in sediment samples. Reference values of PAHs in the programmes for performance assessment were obtained from an accurate isotope dilution gas chromatography mass spectrometry (ID-GCMS) method which was thoroughly validated and verified. Isotope dilution mass spectrometry (IDMS) technique usually has a well-defined measurement uncertainty budget and a traceability link to an International System of Units. Provision of the metrological reference values in PT enables the establishment of a technical platform to assess the actual competence of the participating laboratories in sediment PAHs analysis. Results of the PT programmes showed that about 80 % of the laboratories employed gas chromatography in their analyses and the remaining used liquid chromatography. Irrespective of the techniques being used, however, the majority of the participating laboratories were observed to underestimate values in which the mean values of the five reported PAHs were less than those of the ID-GCMS-derived reference values by 13–20 %. Only 41–44 % of the participating laboratories were able to achieve satisfactory z-scores. The present study revealed that the reinforcement of the capability for accurate measurement of PAHs in sediment samples in laboratories worldwide should be addressed.     

Measurement of PAHs in environmental matrices: results from an interlaboratory comparison on the different steps of the measurement procedure

Proficiency testing (PT) is becoming a feature of laboratory accreditation and the PT results are used to assess the technical competence of the participating laboratories. ISPRA (former APAT) plays an important role in supporting the Italian laboratories belonging to the network of the Regional Environmental Agencies to improve the quality of their analytical measurements. As a consequence, ISPRA organized an interlaboratory comparison to assess the performance of the laboratories on PAH measurement procedure. The interlaboratory comparison was drawn separating the different steps of the measurement (from the extraction to the instrumental measurement). Two matrix reference materials: (1) a polluted soil and (2) an extract reference material of the same polluted soil and a “blind” PAH mixture standard stock solution were distributed to 59 Italian laboratories. The results of interlaboratory comparison showed a significant dispersion of the PAH measurements that masks the effects of the different extraction and cleanup procedures used, but it is consistent with the results of other European interlaboratory comparisons.     

Capability of laboratories to determine core nutrients in foods: results of an international proficiency test

A proficiency test (PT) to assess the capabilities of laboratories to determine nutrients in a biscuit sample was carried out in September 2009. The need for such interlaboratory comparison arose from an increasing nutrition labeling requirements of different countries around the world. Forty-eight laboratories worldwide participated in this PT program for the determination of total lipids, saturated fat, trans-fat, protein, sugars, total dietary fiber (TDF), ash and sodium in food. This program was organized under the auspices of the Asia-Pacific Laboratory Accreditation Cooperation (APLAC). APLAC considers PT programs as one of the objective means of assessing the performance of the accredited laboratories. The program is one of the APLAC PT series whose primary purposes are to establish mutual agreement on the equivalence of the operation of APLAC member laboratories and to provide an opportunity to identify testing deficiency so that corrective actions can be taken, if necessary. The results of this program indicated that the consensus mean values estimated by robust statistics were in good agreement with the corresponding values obtained in the homogeneity tests except for trans-fat and TDF. The relative standard deviations of participant results for trans-fat and TDF were also found to be higher (19 and 24%, respectively) than the other test parameters. The relative standard deviations for other test parameters were less than 10%. One-half of the laboratories obtained satisfactory z-scores (| z | < 3) (\left| z \right| < 3) for all their reported results. Although participants were instructed to provide the measurement uncertainties of their reported results, only 27.5% of them had done so. Albeit many of the measurement uncertainties appear to be reasonable, some of them were found to vary widely, indicating that some laboratories still have difficulties in estimating measurement uncertainties. Although the overall measurement capability can be considered to be satisfactory, there is still room for improvement of analytical procedures.     

Determination of POPs in environmental matrices – proficiency tests for Polish laboratories

Thirty Polish laboratories participated in proficiency tests (PT) for determination of persistent organic pollutants – POPs (PCBs – polychlorinated biphenyls, PAHs – polyaromatic hydrocarbons and pesticides) in solid environmental matrices (sediments, soil). Participants used their in-house method. The aim of the exercises was to obtain information about the “state of the art” of POPs analysis in environmental samples. The CV (coefficient of variation) of analytical results submitted varied from 45 to 65%, with an outlier rate 2 to 7%. The averages of results for all analytes were in good agreement with reference values. Statistical treatment of results was made. Comparison of the each result with the certified values characterizing the material has been done. Additionally, z-scores were calculated taking into account the uncertainty of a certified value and the acceptable deviation from the certified value in the range of −20% to +10% (in accordance with recommendations in the newest directives of European Union (EU) [1]); For rejecting outliers Hampel test has been applied.     

Providing a traceable assigned value in a proficiency testing programme on the determination of benzoic acid in orange juice

A proficiency testing (PT) programme on the determination of benzoic acid in orange juice was organised by the Chemical Metrology Laboratory, Health Sciences Authority, Singapore, for the local food testing laboratories. The PT programme used a metrologically traceable assigned value determined by isotope dilution mass spectrometry to assess the performance of the participating laboratories. This paper discusses the reasons and approach to applying metrological principles to determine the assigned value of a PT programme. The procedure undertaken by the Laboratory to prepare a batch of PT sample, conduct homogeneity and stability testings, as well as value assignment, is presented. The discussions also include the estimation of the measurement uncertainties arising from the characterisation, testing for homogeneity and testing for stability of the PT sample. The Coordinator of the PT programme used the assigned value and its associated measurement uncertainty, together with those of the participating laboratories to assess their performances. Statistical tools applied in the performance evaluations are also presented.     

Robustified least squares support vector classification

Support vector machine (SVM) algorithms are a popular class of techniques to perform classification. However, outliers in the data can result in bad global misclassification percentages. In this paper, we propose a method to identify such outliers in the SVM framework. A specific robust classification algorithm is proposed adjusting the least squares SVM (LS‐SVM). This yields better classification performance for heavily tailed data and data containing outliers. Copyright © 2009 John Wiley & Sons, Ltd.