The application of data from proficiency testing to laboratory accreditation according to ISO 17025

Current methods of testing laboratories for their proficiency in reporting correct measurement results are liable to substantial errors of the second kind. This means that laboratories with deflated uncertainties are accepted as proficient, even though their reported measurement results pave the way for erroneous conclusions. Only by using E _n numbers based on an accepted reference value with the lowest possible uncertainty can the risk of recognising incorrect measurement results be kept at an acceptable level. Based on an actual set of proficiency test (PT)-data for the concentration of Pb in water, this paper compares PT results obtained by methods using E _n numbers with methods based on z-scores. Kaj Heydorn is a technical assessor at The Danish Accreditation and Metrology Fund (DANAK).     

Proficiency testing in the light of a new rationale in metrology

The novel proposed definition of measurement result in the international metrology vocabulary requires a revision of standards and guidelines for proficiency testing (PT), and a new approach to processing proficiency data is needed to test the ability of laboratories to present not only unbiased quantity values, but reliable estimates of their uncertainty. Hence, an accepted reference value with the smallest possible uncertainty is needed to ascertain the proficiency of laboratories reporting results with lower than average uncertainty. A strategy based on the T-statistic is proposed leading to an accepted reference value that fully reflects the uncertainties reported by participants in a PT scheme and permits calculation of E _n numbers to distinguish whether or not measurement results are consistent with the accepted definition of the measurand. The strategy is applied to PT data from a recent international laboratory intercomparison of uranium isotopic ratios.     

Performance indication improvement for a proficiency testing

The paper discusses peculiarities of Z scores and E _n numbers, which are most often used for the treatment of proficiency test data. The important conditions of proper usage of these performance indicators and their improvement are suggested on the basis of systematic approach, on the idea of accuracy classification, and on some principles of optimality borrowed from information theory. The author believes that this paper may be of interest and practical value for all those engaged in applied metrology, specifically in the field of developing of and participating in proficiency testing programs, and in the activity connected with accrediting testing and calibration laboratories.Presented at 15th International Conference on Quality in Israel, November 2004     

Capability of laboratories to determine core nutrients in foods: results of an international proficiency test

A proficiency test (PT) to assess the capabilities of laboratories to determine nutrients in a biscuit sample was carried out in September 2009. The need for such interlaboratory comparison arose from an increasing nutrition labeling requirements of different countries around the world. Forty-eight laboratories worldwide participated in this PT program for the determination of total lipids, saturated fat, trans-fat, protein, sugars, total dietary fiber (TDF), ash and sodium in food. This program was organized under the auspices of the Asia-Pacific Laboratory Accreditation Cooperation (APLAC). APLAC considers PT programs as one of the objective means of assessing the performance of the accredited laboratories. The program is one of the APLAC PT series whose primary purposes are to establish mutual agreement on the equivalence of the operation of APLAC member laboratories and to provide an opportunity to identify testing deficiency so that corrective actions can be taken, if necessary. The results of this program indicated that the consensus mean values estimated by robust statistics were in good agreement with the corresponding values obtained in the homogeneity tests except for trans-fat and TDF. The relative standard deviations of participant results for trans-fat and TDF were also found to be higher (19 and 24%, respectively) than the other test parameters. The relative standard deviations for other test parameters were less than 10%. One-half of the laboratories obtained satisfactory z-scores (| z | < 3) (\left| z \right| < 3) for all their reported results. Although participants were instructed to provide the measurement uncertainties of their reported results, only 27.5% of them had done so. Albeit many of the measurement uncertainties appear to be reasonable, some of them were found to vary widely, indicating that some laboratories still have difficulties in estimating measurement uncertainties. Although the overall measurement capability can be considered to be satisfactory, there is still room for improvement of analytical procedures.     

Certification of cabbage and carnation samples and their use in an international proficiency study

Summary The preparation of carnation straw (Dianthus) and white cabbage (Brassica oleracea) as reference plant materials for chemical analysis and the testing of their homogeneity is described. The elemental content has been determined and evaluated by the Comité Inter-Instituts (CII) using different analytical procedures. The CII is an organisation grouping about 30 selected European laboratories which prepares, on a regular basis, several, very different reference plant materials. At the moment about 24 different plant samples are in stock. Recommended values for the two samples used in this study, are given for ash content, N, P, K, Ca, Mg, Na, Al, Fe and Mn. For 3 years these materials have been used as home references in the International Plant-analysis Exchange (IPE), a continuous proficiency testing scheme, organized by the Wageningen Agricultural University (The Netherlands), in which 230 laboratories worldwide are participating. The values found by the IPE laboratories are compared with the values recommended by CII and are mostly in good agreement with each other.     

Performance of commercial soil laboratories in a proficiency test program in Brazil

A soil proficiency test (PT) was administered to 50 participant laboratories in which two sets of samples, consisting of 20 yearly PT samples and 5 ‘blind’ samples in clients’ names were analyzed for pH, organic matter, total acidity, extractable calcium, magnesium, potassium and phosphorus by the laboratories. Our objective was to determine whether laboratories take extra care to analyze clients’ samples as they do with regular PT samples. The analytical data were evaluated essentially by the procedure described in the international harmonized protocol for proficiency testing of analytical chemistry laboratories. Performance of participant laboratories was assessed by z-scores and summary z-scores statistics involving sum of squared z-scores interpreted as chi-square ( c_n² ) \left( {\chi_{n}^{2} } \right) distribution for zero-centered z-scores with unit variance. From 8 750 determinations, outliers and stragglers accounted for less than 2% of the entire data. Over 93% of the data were satisfactory, whereas between 2 and 4% were either unsatisfactory or questionable in both the PT and ‘blind’ tests. On the basis of sum of squared z-scores interpreted from c_n² \chi_{n}^{2} distribution table, between 30 and 40% of the laboratories had more than 90% probability of having their measurement data within the robust mean and standard deviation for each soil parameter, while another 30–42% of the laboratories had less than 50% probability of having measurement data within the robust mean and standard deviation. Overall, 21 laboratories (42%) were ranked in Class A either in the PT or ‘blind’ tests out of which 12 of them (57%) retained this ranking in both tests. Fourteen laboratories (28%) were ranked in Class C in either the PT or ‘blind’ tests with only 5 of them (36%) consistently ranking in this class in both tests.     

QUASIMEME results of laboratories involved in the German Marine Monitoring Programme for the North and Baltic Seas

This case study is written for laboratories in the German Marine Monitoring Program (GMMP) for the North and Baltic Seas in order to give support to the assessment of proficiency testing results for their external quality assurance. A second issue is to provide a practical, understandable and common way for the calculation of laboratory performance parameters, so-called combination scores. In this paper, the GMMP laboratory proficiency testing results of QUASIMEME (Quality Assurance of Information for Marine Environmental Monitoring in Europe) Laboratory Performance Studies for the last 5 years were reviewed. Combination scores for organochlorine pesticides and chlorinated biphenyls in marine sediment are presented and explained. However, the assessment approach is very general and should be applicable to most testing laboratories in the chemical field.Presented at the Eurachem PT Workshop September 2005, Portorož, Slovenia     

Detection of genetic modification in cured tobacco leaf: proficiency testing using polymerase chain reaction-based methods

The Cooperation Centre for Scientific Research Relative to Tobacco (CORESTA; Paris, France) "Task Force Genetically Modified Tobacco-Detection Methods" investigated the performance of qualitative and quantitative methods based on the polymerase chain reaction (PCR) for the detection and quantitation of genetically modified (GM) tobacco. In the 4 successful rounds of proficiency testing, the cauliflower mosaic virus 35S RNA promoter (CaMV 35S) and the Agrobacterium tumefaciens nopaline synthase terminator (NOS) were selected as target sequences. Blind-coded reference materials containing from 0.1 to 5.0% and from 0.15 to 4% GM tobacco were used in 2 rounds of qualitative and quantitative PCR, respectively. Eighteen laboratories from 10 countries participated in this study. Considering all methods and 2 rounds, the different laboratories were able to detect GM tobacco at the 0.1% level in 46 out of 58 tests in qualitative assays. The results of the proficiency test indicate that both end point screening and real-time quantitative methods are suitable for the detection of genetically modified organisms in tobacco leaf samples having a GM content of 0.1% or higher. The CORESTA proficiency study represents a first step towards the interlaboratory evaluation of accuracy and precision of PCR-based GM tobacco detection, which may lead to the harmonization of analytical procedures and to the enhancement of comparability of testing results produced by different laboratories.     

Normalisation of data from allergens proficiency tests

The problem of allergen analysis using ELISA kits from different commercial products giving significantly different results is widely acknowledged. The effect on proficiency testing results is that different assigned values have to be generated for the different kits used. Some experimental Food Analysis Performance Assessment Scheme (FAPAS) proficiency tests aimed to establish whether the use of a standardised calibrant could be used to normalise the complete data set without recourse to differentiation. Three recent FAPAS proficiency tests (2776 peanut, 2778 soya and 2781 gluten) sent out a second spiked sample, in addition to the usual spiked and unspiked samples. Further analysis of the data was undertaken after the completion of the tests. The ratio of the submitted results for the two spiked samples yielded complete data sets which could be tested for normality of the distribution. Where the raw data for each individual test sample was clearly non-normal and multi-modal, the ratio data yielded a much more normal and symmetrical distribution. The use of one of the test samples as a single-point calibrant has some limitations but the principle of applying a standardisation clearly works. The development of internationally recognised sets of certified reference calibration standards for use by allergens testing laboratories would greatly benefit the analysis.     

Accreditation for United States environmental laboratories

 There is currently no national program for recognition of United States environmental laboratories. This situation should change sometime in 1998 as environmental testing laboratories should be able to apply for accreditation in the United States via a system developed by the National Environmental Laboratory Accreditation Conference (NELAC). NELAC is an organization whose purpose is to establish performance standards for accrediting environmental laboratories. Laboratories to be accredited will be required to demonstrate qualifications of testing personnel, pass on-site inspections, perform satisfactorily on proficiency test samples, and meet specified quality assurance standards.     

Evaluation of the use of consensus values in proficiency testing programmes

Proficiency testing (PT) is an essential tool used by laboratory accreditation bodies to assess the competency of laboratories. Because of limited resources of PT providers or for other reasons, the assigned reference value used in the calculation of z-score values has usually been derived from some sort of consensus value obtained by central tendency estimators such as the arithmetic mean or robust mean. However, if the assigned reference value deviates significantly from the ‘true value’ of the analyte in the test material, laboratories’ performance will be evaluated incorrectly. This paper evaluates the use of consensus values in proficiency testing programmes using the Monte Carlo simulation technique. The results indicated that the deviation of the assigned value from the true value could be as large as 40%, depending on the parameters of the proficiency testing programmes under investigation such as sample homogeneity, number of participant laboratories, concentration level, method precision and laboratory bias. To study how these parameters affect the degree of discrepancy between the consensus value and the true value, a fractional factorial design was also applied. The findings indicate that the number of participating laboratories and the distribution of laboratory bias were the prime two factors affecting the deviation of the consensus value from the true value.     

Assessment of performance of laboratories in determining acrylamide in crispbread

A proficiency testing round was undertaken to assess the performance of laboratories to measure acrylamide in a sample of crispbread. Retail samples of crispbread were ground to a fine powder and after thorough mixing were packed in 40 g units for distribution. Ten samples were selected at random and analyzed in duplicate for acrylamide by liquid chromatography/mass spectrometry (LC/MS). Standard statistical tests showed that the material was homogeneous for the purposes of proficiency testing. Test samples were distributed to 55 laboratories in 16 countries in Europe, North America, Australia, and the Middle East. The results were analyzed by standard proficiency testing statistical procedures, and laboratories were awarded z-scores on the basis of their reported results. Based on a target standard deviation (sigmap value) taken from the Horwitz equation, for a robust mean value of 1.2 mg/kg acrylamide, satisfactory results (z-score within +/- 2 for those between 0.8 and 1.6 mg/kg) were obtained by 86% of the 37 laboratories that returned results. Only 1 laboratory was unsatisfactory and 4 had questionable results. About equal numbers of laboratories used gas chromatography (GC)/MS and LC/MS procedures with about 25% using MS/MS and one using GC with electron capture detection. There was no evident trend in performance or bias in results. GC/MS and LC/MS data were evenly distributed across the population of laboratories reporting results.     

水中易释放氰化物能力验证样品的研制及应用

根据国家地表水环境质量监测网监测任务要求和环境监测的实际需要,制备了水中易释放氰化物能力验证样品。通过均匀性、稳定性检验以及量值一致性评价,研制的样品均匀性良好,在3℃~6℃冷藏避光保存条件下1年内稳定,样品配制值与多家实验室协作测定结果一致。探讨了样品在能力验证活动中的应用,共有来自全国11个省的32家实验室参加了水中易释放氰化物的能力验证计划,实验室满意率在80%以上,实验室结果出现有问题或不满意主要是由于样品前处理以及检测过程质量控制不当导致的。经检测及实验室反馈的数据验证,该能力验证样品能够应用于能力验证活动。     

Performance of Canadian clinical laboratories processing throat culture proficiency testing surveys

Proficiency testing results obtained from simulated throat swab cultures sent between 1999 and 2013 were analyzed to evaluate the ability of participants to process, analyze, and properly report results for these cultures. Eight percent of all pathogen-positive samples were reported as false negatives, and 11 % of the pathogen-negative ones were reported as false positives. More than 90 % of the participants achieved acceptable grades in each of the proficiency testing samples containing Streptococcus pyogenes. Laboratories were less successful in reporting other significant pathogens including group C and G streptococcus and Arcanobacterium haemolyticum, although an overall increase in performance was observed for each subsequent survey. Most laboratories recognized other respiratory pathogens as not being agents of pharyngitis; a common issue with these specimens was the suggestion of the organism’s clinical significance when their presence was reported. The presence of small colony-forming β-hemolytic streptococcus (Streptococcus anginosus group) proved to be challenging for many laboratories as 66 % of all unacceptable grades for pathogen-negative throat swabs occurred in these PT samples. Moreover, performance in this group did not improve with subsequent surveys. Results obtained in this study support the added value of clinical relevancy challenges for proficiency testing as laboratories that may not have problems with the analytical aspect of the testing process struggle with the appropriate interpretation of results.     

An inter-laboratory comparison study for the determination of copper and lead from the wastewater of printed circuit board manufacturing industry in Pakistan

This inter-laboratory comparison study was arranged for 28 laboratories from different public and private sector organizations in Pakistan having wastewater testing capabilities aimed at improving the quality and comparability of test results. This national inter-laboratory study was started in December 2003 and completed in July 2004. Laboratories were invited to analyze the wastewater collected from printed circuit board (PCB) industry for lead and copper contamination. The samples fulfill the criteria for homogeneity and stability as done by the reference laboratory. The results obtained from participating laboratories were analyzed in terms of Hampel Test for outliers, while the performance evaluation of the participating laboratories was done on the basis of Z-score. An assigned value derived from the participant's results was compared with a reference value provided by a reference laboratory. Overall >50% of the participating laboratories have shown good performance in this PT-program     

Bias analysis of proficiency testing programme results

A proficiency testing programme might involve a great number of participating laboratories coming from different countries or regions, and normally they analysed the same test materials using their own routine analytical methods. Hence, the results of a proficiency testing programme may contain valuable information which could serve purposes other than just performance evaluation. This study attempted to extract information from the results of a proficiency testing programme for the purposes of educating the participating laboratories as suggested by ISO/IEC 17043. The “bias analysis” approach introduced in this study was based on the statistical model of measurement and the nature of bias in chemical analysis. With this approach, the participating laboratories could estimate the bias associated with different settings of experimental conditions according to the statistics of subset distribution of the reported results from the participating laboratories. This would be useful for them to review the analytical procedures they used and modify their methods if needed. The approach was applied to the analysis of data obtained from a number of past proficiency testing programmes, and the findings were discussed in this paper.     

Quality issues in proficiency testing

 Proficiency testing is a means of assessing the ability of laboratories to competently perform specific tests and/or measurements. It supplements a laboratory's own internal quality control procedure by providing an additional external audit of their testing capability and provides laboratories with a sound basis for continuous improvement. It is also a means towards achieving comparability of measurement between laboratories. Participation is one of the few ways in which a laboratory can compare its performance with that of other laboratories. Good performance in proficiency testing schemes provides independent evidence and hence reassurance to the laboratory and its clients that its procedures, test methods and other laboratory operations are under control. For test results to have any credibility, they must be traceable to a standard of measurement, preferably in terms of SI units, and must be accompanied by a statement of uncertainty. Analytical chemists are coming to realise that this is just as true in their field as it is for physical measurements, and applies equally to proficiency testing results and laboratory test reports. Recent approaches toward ensuring the quality and comparability of proficiency testing schemes and the means of evaluating proficiency test results are described. These have led to the drafting of guidelines and subsequently to the development of international requirements for the competence of scheme providers. Received: 2 January 1999 · Accepted: 7 April 1999     

Determination of Assigned Values for Cadmium,Cobalt and Manganese in Drinking Water Proficiency Testing

In proficiency tests the selection of an appropriate assigned value is one of the most critical points. Determination of the assigned value for proficiency test can be done in 2 ways: based on reference value and consensus value. The use of such a consensus value as an assigned value is a commonplace in proficiency testing, because there is seldom at present a cost effective alternative. However, the consensus value has a weakness in that it could be significantly different from the true value due largely inexperienced laboratories. Therefore in this work, the reference values were used as the assigned values for all analytes of interest. The reference values in spiked drinking water samples were produced by the Metrology in Chemistry Laboratory of Research Center for Chemistry - Indonesian Institute of Sciences (RCChem-LIPI) with traceable to SI units through standard solutions from the National Institute Standard and Technology, USA. The reference values of cobalt and manganese were obtained by measurement using two from three comparable methods of Graphite Furnace-Atomic Absorption Spectrometry (GF-AAS), Inductively Coupled Plasma-Optical Emission Spectrometry (ICP-OES) and Inductively Coupled Plasma-Mass Spectrometry (ICP-MS) while that for cadmium was obtained by a primary method of ICP-Isotope Dilution Mass Spectrometry (ICP-IDMS).The obtained reference values were used to evaluate the laboratory performance result in a proficiency testing scheme of cadmium, cobalt and manganese in drinking water which was provided in 2014. All fifty nine laboratories reported cadmium and cobalt at low concentration levels which were about more than 82% showed satisfactory performance results. Similarly, the results of manganese were also comparable at high level concentrations.     

Proficiency testing in food microbiology: experience from implementation of ISO/IEC 17043 and ISO/TS 22117

The proficiency testing (PT) scheme ??AQUA?? for food microbiology was organised by the Istituto Zooprofilattico Sperimentale delle Venezie (IZSVe) according to ISO/IEC 17043 and ISO/TS 22117. This paper describes the IZSVe experience on the application of the above-mentioned standards for the PTs, with focus on the Enterobacteriaceae enumeration one. Freeze-dried food matrices contaminated with American Type Culture Collection bacterial strains were used as test samples for each microbiological PT organised by IZSVe. The sample homogeneity and stability were verified prior to distribution to participants and throughout the PT, respectively. The participating laboratories analysed samples using their routine methods, and results were transmitted to IZSVe. Data and methods used by each participating laboratory were analysed in order to evaluate the laboratory performance. With reference to the Enterobacteriaceae PT, the test samples were homogeneous and stable. In addition, most laboratory results were obtained using equivalent test methods. Statistical approaches applied to analyse data generated from all participating laboratories revealed similar outcomes as no significant outlying count and only 5?% of unacceptable results were observed. Finally, the z-score, with the standard deviation that does not vary from round toround, was applied to compare and to evaluate the performance of each laboratory over time highlighting possible persistent trends over several rounds.     

Proficiency tests using four batches of green alga with controlled levels of cadmium

A set of test materials of the green alga Chlorella vulgaris with different levels of naturally bound Cd and about the same levels of other 13 essential or trace elements (BIOMA 1–4) was produced for proficiency testing of laboratories involved in elemental analysis of food. Criteria of the laboratories’ performance, such as the z-score and combination scores RSZ and SSZ were evaluated and discussed in terms of their suitability for proficiency testing using a set of test materials with the same matrix. The use of the test materials as a set of internal reference materials has also been suggested. Received: 23 May 1997 / Revised: 1 August 1997 / Accepted: 11 August 1997