Proficiency testing in food microbiology: experience from implementation of ISO/IEC 17043 and ISO/TS 22117

The proficiency testing (PT) scheme ??AQUA?? for food microbiology was organised by the Istituto Zooprofilattico Sperimentale delle Venezie (IZSVe) according to ISO/IEC 17043 and ISO/TS 22117. This paper describes the IZSVe experience on the application of the above-mentioned standards for the PTs, with focus on the Enterobacteriaceae enumeration one. Freeze-dried food matrices contaminated with American Type Culture Collection bacterial strains were used as test samples for each microbiological PT organised by IZSVe. The sample homogeneity and stability were verified prior to distribution to participants and throughout the PT, respectively. The participating laboratories analysed samples using their routine methods, and results were transmitted to IZSVe. Data and methods used by each participating laboratory were analysed in order to evaluate the laboratory performance. With reference to the Enterobacteriaceae PT, the test samples were homogeneous and stable. In addition, most laboratory results were obtained using equivalent test methods. Statistical approaches applied to analyse data generated from all participating laboratories revealed similar outcomes as no significant outlying count and only 5?% of unacceptable results were observed. Finally, the z-score, with the standard deviation that does not vary from round toround, was applied to compare and to evaluate the performance of each laboratory over time highlighting possible persistent trends over several rounds.     

Providing a traceable assigned value in a proficiency testing programme on the determination of benzoic acid in orange juice

A proficiency testing (PT) programme on the determination of benzoic acid in orange juice was organised by the Chemical Metrology Laboratory, Health Sciences Authority, Singapore, for the local food testing laboratories. The PT programme used a metrologically traceable assigned value determined by isotope dilution mass spectrometry to assess the performance of the participating laboratories. This paper discusses the reasons and approach to applying metrological principles to determine the assigned value of a PT programme. The procedure undertaken by the Laboratory to prepare a batch of PT sample, conduct homogeneity and stability testings, as well as value assignment, is presented. The discussions also include the estimation of the measurement uncertainties arising from the characterisation, testing for homogeneity and testing for stability of the PT sample. The Coordinator of the PT programme used the assigned value and its associated measurement uncertainty, together with those of the participating laboratories to assess their performances. Statistical tools applied in the performance evaluations are also presented.     

Development of a proficiency testing scheme for a limited number of participants in the field of natural water analysis

A proficiency testing (PT) scheme was developed for a limited number of analytical laboratories participating in the analysis of natural water in Israel. Three fit-for-intended-use reference materials (RMs) were prepared for a pilot PT from natural water: RM-blank and two in-house reference materials (IHRMs) with seven analytes fortified to achieve different levels of analyte concentrations. The measurands for the PT participants were the traceable spike values certified in the IHRMs, i.e., added mass concentrations of the analytes. The RM-blank and IHRMs were found to be homogeneous and stable over 11 days, the time necessary for the experimental part of the PT. The RMs were distributed to the PT participants as unknown test items similar to routine samples. The test results were evaluated using their deviations from the IHRM-certified values. Eight Israeli laboratories took part in the interlaboratory comparison. Individual laboratory performance and metrological compatibility of the PT results of the participants, as a local group of laboratories, were evaluated for every analyte.     

The influence of a quality system according to ISO/IEC 17025 on the performance of Brazilian laboratories in proficiency testing in the environmental area

This paper presents a performance analysis of laboratories in proficiency testing (PT) with the main objective of evaluating whether having accreditation to ISO/IEC 17025 or not having an implemented quality system has an influence on the performance of these laboratories. This research was conducted with data from an accredited PT scheme of physical–chemical water analyses. Nine metals were considered in the analysis, with a total of 896 results. The performance indicator was z-score. The proposed analysis was conducted through statistical methods of analysis of variance, Dunnett’s test, Chi-square and cross-tabulation. It was verified that there is a significant difference between the performance of accredited laboratories, recognised laboratories, and laboratories that do not have an implemented quality system. It was established that an accredited or recognised quality system has a positive influence on the laboratory performance, resulting in a lower z-score (in module) and in a larger percentage of satisfactory results.     

Evaluation of the use of consensus values in proficiency testing programmes

Proficiency testing (PT) is an essential tool used by laboratory accreditation bodies to assess the competency of laboratories. Because of limited resources of PT providers or for other reasons, the assigned reference value used in the calculation of z-score values has usually been derived from some sort of consensus value obtained by central tendency estimators such as the arithmetic mean or robust mean. However, if the assigned reference value deviates significantly from the ‘true value’ of the analyte in the test material, laboratories’ performance will be evaluated incorrectly. This paper evaluates the use of consensus values in proficiency testing programmes using the Monte Carlo simulation technique. The results indicated that the deviation of the assigned value from the true value could be as large as 40%, depending on the parameters of the proficiency testing programmes under investigation such as sample homogeneity, number of participant laboratories, concentration level, method precision and laboratory bias. To study how these parameters affect the degree of discrepancy between the consensus value and the true value, a fractional factorial design was also applied. The findings indicate that the number of participating laboratories and the distribution of laboratory bias were the prime two factors affecting the deviation of the consensus value from the true value.     

Accuracy and reliability of radiometric measurements – results of a laboratory proficiency test with labelled sediment

 In 1995, a laboratory proficiency test was carried out in Germany with labelled sediment which was analyzed by 40 participants (operators of nuclear power plants, officially authorized institutions of the German Federal States, and laboratories of research centers). Preparation and certification of the labelled material were described in detail. The deviations of the laboratory means from the corresponding certified reference values of ten radionuclides were determined. Outliers of type 2 were recognized by means of the Grubbs test and eliminated. The consistency of laboratory means and reference values was tested by a modified t test. To describe the reliability of the measurements performed, a radioanalytical quality index which has been defined elsewhere was used. By use of this index, the radioanalytical quality of laboratory means and the means of laboratory means could be evaluated and a very realistic picture of the "state-of-the-practice" of the participating laboratories obtained. In some cases, it was possible to identify systematic errors and to explain their causes by physical phenomena. Received: 11 June 1996 Accepted: 14 September 1996     

Comparability and compatibility of proficiency testing results in schemes with a limited number of participants

Comparability and compatibility of proficiency testing (PT) results are discussed for schemes with a limited number of participants (less than 20–30) based on the use of reference materials (RMs) as test items. Since PT results are a kind of measurement/analysis/test result, their comparability is a property conditioned by traceability to measurement standards applied in the measurement process. At the same time, metrological traceability of the certified value of the RM (sent to PT participants as test items) is also important, since the PT results are compared with the RM certified value. The RM position in the calibration hierarchy of measurement standards sets the degree of comparability for PT results, which can be assessed in the scheme. However, this assessment is influenced by commutability (adequacy or match) of the matrix RM used for PT and routine samples. Compatibility of PT results is a characteristic of the collective (group) performance of the laboratories participating in PT that can be expressed as closeness of the distribution of the PT results to the distribution of the RM data. Achieving quality-of-measurement/analysis/test results in the framework of the concept “tested once, accepted everywhere” requires both comparability and compatibility of the test results.     

Official proficiency tests of the organisation for the prohibition of chemical weapons: current status and future directions

The Organisation for the Prohibition of Chemical Weapons (OPCW) has been organising and conducting Official Proficiency Tests (PT) since 1996 in accordance with ILAC-G13 to certify laboratories for the analysis of authentic samples under the provision of Chemical Weapons Convention. The tests are part of a mechanism to ensure that there are laboratories that have proven competence in the analysis of chemicals related to the Convention. Laboratories that have successfully completed the tests are designated by the Director General of the OPCW for analysis of authentic samples. To maintain the Designated Laboratory status, a laboratory must take and pass at least one of the two proficiency tests offered per calendar year. Unlike many proficiency tests, the OPCW PT is qualitative, that is the laboratories must determine if any of a very large set (essentially infinite) of chemicals relevant to the Convention are present in the samples. The tests are organised with the assistance of two laboratories, one preparing the test samples, and the other evaluating the test results. The paper provides an overview of the current status of these PTs and outlines salient features regarding procedure and scope of tests, selection of assisting laboratories, scoring and performance rating of participating labs. The emerging issues are also briefly discussed.     

Uses and abuses of PT

Proficiency testing (PT) is a very valuable quality tool for laboratories making all kinds of analytical measurements, in chemistry, microbiology and laboratory medicine. Compared with other quality tools, which are largely concerned with INPUTS (e.g. staff training, calibration, documentation), PT is concerned with OUTPUTS (i.e. the actual results produced by the laboratory). The quality of routine measurements and results, which would normally be reported to customers, both internal and external, should be demonstrated. This is a major reason for many laboratories participating in PT, even if they are not accredited, or use very few, if any, other quality tools. This is important, for it enables the laboratory manager to measure the efficacy of the overall quality system, and to determine if the inputs to this are appropriate. This should be done in a spirit of learning—it was discussed at the previous workshop whether PT schemes are teachers or policemen, and the consensus was that MOST are teachers (or at least try to be). Learning from mistakes, investigating problems, implementing corrective actions and then observing whether thisis successful in the next PT represents what many in our profession would regard as a proper and effective use of PT. Although this may be the intended primary use of PT by laboratories, we must not forget that PT has other valid uses. In particular, the use of PT to demonstrate competence to third parties—accreditation bodies, regulators, and customers—has become more important in the last few years. However, this development, however valuable, can lead to abuse of PT scheme data. This paper will highlight briefly some examples of both proper use and abuse of PT. These are taken from my experience with PT schemes over many years.     

The current status and future of medical laboratory quality regulation and accreditation in Ghana

High-quality and reliable laboratory services are important components of effective and well-functioning health systems. Accurate, reliable and timely medical laboratory testing is crucial to patient care and disease surveillance. Unfortunately, in many sub-Saharan African countries, medical laboratory systems are adversely affected by the unavailability of medical laboratories, poor laboratory infrastructure and lack of well-trained personnel [1]. Quality in the laboratory is only achieved in a systematic way through the implementation of a quality management system. The results of the study showed that approximately 60?% of the 78 respondents were unaware of the requirements of ISO 15189:2007. A trial of proficiency testing, termed ??blind proficiency testing??, was carried out in which 19 laboratories determined the concentrations of urea and cholesterol in a proficiency testing material. Of the 19 laboratories that determined the concentration of urea, 63?% produced satisfactory results with scores between ?2 and +2. Similarly, 63?% of the participating laboratories obtained satisfactory z scores for cholesterol determination. Some of the laboratories that obtained satisfactory scores for urea determination had unsatisfactory scores for cholesterol determination and vice versa. It is recommended that the Ghanaian government pass a law and establish a standard to regulate medical laboratories in Ghana in order to improve quality in a significant way.     

Comparison of different statistical methods for evaluation of proficiency test data

In analytical chemistry, proficiency testing usually consists in tests that laboratories conduct under routine conditions and report the result to the PT provider who then converts the result to a score which helps the participant to assess the accuracy of the result. The aim of this work is to show PT providers, accreditations bodies, and participating laboratories that different scoring results can be achieved depending on the evaluation system selected. The influence of different evaluation techniques on the results of an interlaboratory comparison for determination of gold in precious metals alloys was investigated. Results from 19 participating laboratories were evaluated by means of the three procedures: (1) classical statistical approach—outliers detection; (2) robust methods—(2A) robust procedure and (2B) ISO 13528; and (3) fitness for purpose. Evaluation of the same PT data revealed very interesting issues depending on the different scoring systems that were used and the robustness of the statistical methods used for detecting outliers. As a general rule, laboratories with scoring Z > 2 offered clearly poorer performance in robust approaches than classical ones. In order to support this first evidence, we evaluated a second data set with results from 24 laboratories (mercury from soil samples) by means of the four mentioned approaches. Selection and comparison of different scoring systems must be done very carefully, because sometimes they are not the best approach for studying the data population or the more appropriate one for evaluating the distribution of the data. Finally it should be taken into account that sometimes the robust scoring systems are not always suitable for evaluating the results of some PT schemes.     

Regular participation in proficiency testing provides long term improvements in laboratory performance: an assessment of data over time

Data from various proficiency testing schemes, operated by LGC Standards, was collated and reviewed to determine whether regular involvement within third party, proficiency testing, either over a prolonged period of time or via multiple participation, had improved the performance of those laboratories taking part. Three statistical evaluations were applied to historical PT results, a review of improvements over time, demonstrated by the evaluation of satisfactory performance scores achieved, a review of the relative robust standard deviations obtained from multiple methods and a focused review of one analyst participating in proficiency testing over a prolonged period of time. In each case the data indicated that long term participation and evaluation via proficiency testing had resulted in consistent and sustained improvements in laboratory performance.     

Proficiency testing: binary data analysis

A method for evaluating qualitative proficiency testing (PT) of laboratories conducting binary tests is proposed. The method is based on the scale-invariant item response model proposed by the authors in earlier publications. We consider the case where the laboratories under the PT conduct test consisting of a set of test items/species presenting different, but unknown beforehand levels of difficulty when trying to detect a particular property of theirs, and we need to evaluate/compare both the intrinsic abilities of the participating laboratories and the level of difficulty of the test items. We assume that the responses to different test items do not affect one another and discuss how to get and interpret the most likely estimates/scores. The method is illustrated by the example presented in a recent publication by our colleagues from QuoData GmbH and can be considered as an alternative to that proposed in their publication method of scoring.     

Evaluation of proficiency tests in microbiological analysis: enumeration of aerobic microorganisms

South Korea’s Ministry of Food and Drug Safety (MFDS) has been developing programs for the inspection and accreditation of food sanitation inspection institutions. Food sanitation inspection institutions such as MFDS regional offices, the Research Institute of Public Health and Environment and authorized private service providers in South Korea must participate in proficiency testing (PT) programs to comply with the Food Sanitation Act and MFDS Notification No. 2012-112. As the PT provider, the MFDS annually plans various microbiological and chemical PT programs for foods, cosmetics, and pharmaceutical products in accordance with ISO/IEC 17043. The aim of this project was to evaluate the performance of microbiological PT programs to ensure the quality of their routine test results. The test materials used were freeze-dried BioBalls from BTF Pty Ltd. Homogeneity and stability were investigated to assess the adequacy of the selected test materials. This project also contains data from inter-laboratory comparisons organized by MFDS in 2011 and 2012. More than 50 laboratories attended the PT program and submitted their results. Laboratory results were rated with z-scores according to the international standard ISO 13528. The results from 2011 and 2012 revealed that all participating laboratories had similar levels of proficiency. Most of the participants received a rating of “Satisfactory.” Moreover, the percentage of participants who received a rating of “Unsatisfactory” decreased from 3.5 % in 2011 to 2.0 % in 2012.     

Interpretation of interlaboratory comparison results to evaluate laboratory proficiency

Guidelines are given for the evaluation of proficiency test (PT) results in order to increase the effectivity of PT participation. For better understanding, some statistical background is given along with some examples to show the effects of the choices made by the PT provider. The calculation method of the assigned value and the selection of the standard deviation both affect the z-score that is used by the participating laboratory to judge the quality of its performance in the PT. Therefore, the participating laboratory is advised to use the PT results with care and, if necessary, to recalculate the z-scores. Finally, advice is given on how not to follow up bad PT results along with some valuable steps that could be part of an effective follow-up procedure.     

Proficiency testing in the light of a new rationale in metrology

The novel proposed definition of measurement result in the international metrology vocabulary requires a revision of standards and guidelines for proficiency testing (PT), and a new approach to processing proficiency data is needed to test the ability of laboratories to present not only unbiased quantity values, but reliable estimates of their uncertainty. Hence, an accepted reference value with the smallest possible uncertainty is needed to ascertain the proficiency of laboratories reporting results with lower than average uncertainty. A strategy based on the T-statistic is proposed leading to an accepted reference value that fully reflects the uncertainties reported by participants in a PT scheme and permits calculation of E _n numbers to distinguish whether or not measurement results are consistent with the accepted definition of the measurand. The strategy is applied to PT data from a recent international laboratory intercomparison of uranium isotopic ratios.     

Bias analysis of proficiency testing programme results

A proficiency testing programme might involve a great number of participating laboratories coming from different countries or regions, and normally they analysed the same test materials using their own routine analytical methods. Hence, the results of a proficiency testing programme may contain valuable information which could serve purposes other than just performance evaluation. This study attempted to extract information from the results of a proficiency testing programme for the purposes of educating the participating laboratories as suggested by ISO/IEC 17043. The “bias analysis” approach introduced in this study was based on the statistical model of measurement and the nature of bias in chemical analysis. With this approach, the participating laboratories could estimate the bias associated with different settings of experimental conditions according to the statistics of subset distribution of the reported results from the participating laboratories. This would be useful for them to review the analytical procedures they used and modify their methods if needed. The approach was applied to the analysis of data obtained from a number of past proficiency testing programmes, and the findings were discussed in this paper.     

玩具油漆涂层中有机锡检测能力验证样品研制及应用分析

制备了用于可迁移有机锡检测能力验证的玩具油漆涂层样品,并开展了相关测定能力验证计划,采用迭代法、四分位法和敏感分析法3种不同统计方法对23家实验室数据进行统计分析,并对不满意结果的原因进行了分析。结果表明所研制样品的均匀性和稳定性良好,能满足能力验证要求。不同统计方法对实验室能力评定结果存在差异,但指定值差异很小,而能力评定标准差差异较大,运用卡方检验能选择出更合理的统计方法。     

French laboratory proficiency testing program for food microbiology

The proficiency testing program in food microbiology (Réseau d'Analyses et d'Echanges en Microbiologie des Aliments; RAEMA), created in 1988, currently includes 440 participating laboratories. The program establishes proficiency in detection of Salmonella and Listeria monocytogenes, as well as quantitation of aerobic microorganisms, Enterobacteriaceae, coliforms, Escherichia coli, Clostridium perfringens, coagulase-positive Staphylococcus, and Listeria monocytogenes. Twice a year, 5 test samples are sent to participants to assess their precision and trueness for enumeration and detection of microorganisms. Results show an increasing involvement of food microbiology laboratories in quality assurance programs and use of standard and validated analytical methods. However, the percentage of laboratories obtaining questionable and unsatisfactory microbiological results remains relatively constant.     

Proficiency testing program for the determination of total arsenic, cadmium, and lead in seawater shrimp

The proficiency testing (PT) program for 97 worldwide laboratories for determining total arsenic, cadmium, and lead in seawater shrimp under the auspices of the Asia-Pacific Laboratory Accreditation Cooperation (APLAC) is discussed. The program is one of the APLAC PT series whose primary purposes are to establish mutual agreement on the equivalence of the operation of APLAC member laboratories and to take corrective actions if testing deficiencies are identified. Pooled data for Cd and Pb were normally distributed with interlaboratory variations of 21.9 and 34.8%, respectively. The corresponding consensus mean values estimated by robust statistics were in good agreement with those obtained in the homogeneity tests. However, a bimodal distribution was observed from the determination of total As, in which 14 out of 74 participants reported much smaller values (0.482-6.4 mg/kg) as compared with the mean values of 60.9 mg/kg in the homogeneity test. The use of consensus mean is known to have significant deviation from the true value in bi- or multimodal distribution. Therefore, the mode value, a better estimate of central tendency, was chosen to assess participants' performance for total As. Estimates of the overall uncertainty from participants varied in this program, and some were recommended to acquire more comprehensive exposure toward important criteria as stipulated in ISO/IEC 17025.