Evaluation of proficiency tests in microbiological analysis: enumeration of aerobic microorganisms

South Korea’s Ministry of Food and Drug Safety (MFDS) has been developing programs for the inspection and accreditation of food sanitation inspection institutions. Food sanitation inspection institutions such as MFDS regional offices, the Research Institute of Public Health and Environment and authorized private service providers in South Korea must participate in proficiency testing (PT) programs to comply with the Food Sanitation Act and MFDS Notification No. 2012-112. As the PT provider, the MFDS annually plans various microbiological and chemical PT programs for foods, cosmetics, and pharmaceutical products in accordance with ISO/IEC 17043. The aim of this project was to evaluate the performance of microbiological PT programs to ensure the quality of their routine test results. The test materials used were freeze-dried BioBalls from BTF Pty Ltd. Homogeneity and stability were investigated to assess the adequacy of the selected test materials. This project also contains data from inter-laboratory comparisons organized by MFDS in 2011 and 2012. More than 50 laboratories attended the PT program and submitted their results. Laboratory results were rated with z-scores according to the international standard ISO 13528. The results from 2011 and 2012 revealed that all participating laboratories had similar levels of proficiency. Most of the participants received a rating of “Satisfactory.” Moreover, the percentage of participants who received a rating of “Unsatisfactory” decreased from 3.5 % in 2011 to 2.0 % in 2012.     

Proficiency test on incurred and spiked pesticide residues in cereals

A proficiency test on incurred and spiked pesticide residues in wheat was organised in 2008. The test material was grown in 2007 and treated in the field with 14 pesticides formulations containing the active substances, alpha-cypermethrin, bifentrin, carbendazim, chlormequat, chlorpyrifos-methyl, difenconazole, epoxiconazole, glyphosate, iprodione, malathion, pirimicarb, prochloraz, spiroxamin and trifloxystrobin. After harvest, the test material was additionally spiked in the laboratory with three pesticides, that where the residues were too low, and axozystrobin. In total, 72 laboratories submitted results and z-scores were calculated for all laboratories and pesticides, except for glyphosate where only five laboratories submitted results and summed weighted z-scores were calculated for the laboratories with a sufficient scope. For several pesticides, the submitted results were strongly depending on the extraction procedure and consequently the assigned values were calculated based on part of the results. Acceptable z-scores were obtained by 56–97% of the participants.     

Evaluation of the performance of 57 Japanese participating laboratories by two types of z-scores in proficiency test for the quantification of pesticide residues in brown rice

A proficiency test for the analysis of pesticide residues in brown rice was carried out to support upgrading in analytical skills of participant laboratories. Brown rice containing three target pesticides (etofenprox, fenitrothion, and isoprothiolane) was used as the test samples. The test samples were distributed to the 57 participants and analyzed by appropriate analytical methods chosen by each participant. It was shown that there was no significant difference among the reported values obtained by different types of analytical method. The analytical results obtained by National Metrology Institute of Japan (NMIJ) were 3 % to 10 % greater than those obtained by participants. The results reported by the participant were evaluated by using two types of z-scores, that is, one was the score based on the consensus values calculated from the analytical results of participants, and the other one was the score based on the reference values obtained by NMIJ with high reliability. Acceptable z-scores based on the consensus values and NMIJ reference values were achieved by 87 % to 89 % and 79 % to 94 % of the participants, respectively. Graphical Abstract

Distribution of z and z_NMIJ-scores for isoprothiolane     

Performance evaluation of polycyclic aromatic hydrocarbons analysis in sediment from proficiency testing with metrological reference values

This paper describes a metrological approach to evaluate the measurement capability of laboratories participating in two proficiency testing (PT) programmes involving the analysis of five polycyclic aromatic hydrocarbons (PAHs) in sediment samples. Reference values of PAHs in the programmes for performance assessment were obtained from an accurate isotope dilution gas chromatography mass spectrometry (ID-GCMS) method which was thoroughly validated and verified. Isotope dilution mass spectrometry (IDMS) technique usually has a well-defined measurement uncertainty budget and a traceability link to an International System of Units. Provision of the metrological reference values in PT enables the establishment of a technical platform to assess the actual competence of the participating laboratories in sediment PAHs analysis. Results of the PT programmes showed that about 80 % of the laboratories employed gas chromatography in their analyses and the remaining used liquid chromatography. Irrespective of the techniques being used, however, the majority of the participating laboratories were observed to underestimate values in which the mean values of the five reported PAHs were less than those of the ID-GCMS-derived reference values by 13–20 %. Only 41–44 % of the participating laboratories were able to achieve satisfactory z-scores. The present study revealed that the reinforcement of the capability for accurate measurement of PAHs in sediment samples in laboratories worldwide should be addressed.     

A proficiency test scheme for the quality control of black tea and development of quality control material for internal quality control

A proficiency test (PT) was organized for quality control analysis of black tea. Test materials for the analyses of total content of powder tea, moisture, total ash, acid-insoluble ash, water-soluble ash, alkalinity of water-soluble ash, water extract, crude fiber and caffeine were sent to the participant laboratories (n:43) in 2009. The assigned value, standard deviation of the parameters and z-scores of the participant laboratories were calculated using the data reported by the participants. The percentage of the reported results for analytes was found to be ranged from 67.5 to 100%. Acceptable z-scores were achieved by 80.5–97.5% of the participant laboratories. More than 15% of unacceptable results were obtained for acid-insoluble ash and caffeine analyses. The quality control material for quality control analysis of black tea was developed from the remaining material.     

Performance of commercial soil laboratories in a proficiency test program in Brazil

A soil proficiency test (PT) was administered to 50 participant laboratories in which two sets of samples, consisting of 20 yearly PT samples and 5 ‘blind’ samples in clients’ names were analyzed for pH, organic matter, total acidity, extractable calcium, magnesium, potassium and phosphorus by the laboratories. Our objective was to determine whether laboratories take extra care to analyze clients’ samples as they do with regular PT samples. The analytical data were evaluated essentially by the procedure described in the international harmonized protocol for proficiency testing of analytical chemistry laboratories. Performance of participant laboratories was assessed by z-scores and summary z-scores statistics involving sum of squared z-scores interpreted as chi-square ( c_n² ) \left( {\chi_{n}^{2} } \right) distribution for zero-centered z-scores with unit variance. From 8 750 determinations, outliers and stragglers accounted for less than 2% of the entire data. Over 93% of the data were satisfactory, whereas between 2 and 4% were either unsatisfactory or questionable in both the PT and ‘blind’ tests. On the basis of sum of squared z-scores interpreted from c_n² \chi_{n}^{2} distribution table, between 30 and 40% of the laboratories had more than 90% probability of having their measurement data within the robust mean and standard deviation for each soil parameter, while another 30–42% of the laboratories had less than 50% probability of having measurement data within the robust mean and standard deviation. Overall, 21 laboratories (42%) were ranked in Class A either in the PT or ‘blind’ tests out of which 12 of them (57%) retained this ranking in both tests. Fourteen laboratories (28%) were ranked in Class C in either the PT or ‘blind’ tests with only 5 of them (36%) consistently ranking in this class in both tests.     

Performance of Canadian clinical laboratories processing throat culture proficiency testing surveys

Proficiency testing results obtained from simulated throat swab cultures sent between 1999 and 2013 were analyzed to evaluate the ability of participants to process, analyze, and properly report results for these cultures. Eight percent of all pathogen-positive samples were reported as false negatives, and 11 % of the pathogen-negative ones were reported as false positives. More than 90 % of the participants achieved acceptable grades in each of the proficiency testing samples containing Streptococcus pyogenes. Laboratories were less successful in reporting other significant pathogens including group C and G streptococcus and Arcanobacterium haemolyticum, although an overall increase in performance was observed for each subsequent survey. Most laboratories recognized other respiratory pathogens as not being agents of pharyngitis; a common issue with these specimens was the suggestion of the organism’s clinical significance when their presence was reported. The presence of small colony-forming β-hemolytic streptococcus (Streptococcus anginosus group) proved to be challenging for many laboratories as 66 % of all unacceptable grades for pathogen-negative throat swabs occurred in these PT samples. Moreover, performance in this group did not improve with subsequent surveys. Results obtained in this study support the added value of clinical relevancy challenges for proficiency testing as laboratories that may not have problems with the analytical aspect of the testing process struggle with the appropriate interpretation of results.     

Control charts to evaluate long-term performance in proficiency tests

As proficiency tests (PTs) provider, the European Union Reference Laboratory for Chemical Elements in Food of Animal Origin (EURL-CEFAO) yearly organizes exercises on the determination of chemical elements within the area of its mandate. The scheme is addressed to the National Reference Laboratories (NRLs) of the European Union Member States. Each PT is planned giving priority to the matrices and elements for which maximum levels are set in Commission Regulation No. 1881/2006 and following amendments as well as to those mainly considered in National Residues Control Plans. Samples to be distributed are prepared in the Laboratory’s facilities using materials purchased on the market and often adjusting the concentration levels of the analytes around their maximum levels. In the 2006–2011 period, PTs were planned as a long-term programme pursuing the general objective of providing the NRLs with a scheme that can give them the opportunity to check and improve the performance of their analytical methods and to verify the effectiveness of any corrective actions through the repetition of the same matrix. Control charts give participants the possibility of a long-term follow-up of their performance promoting the improvement of Quality Control. To this aim, Shewhart and Cusum control charts for z-scores are prepared according to ISO 13528:2005 and updated after each PT for each laboratory. z-scores are assigned using values of σ _pEURL-CEFAO that correspond to the level of performance appropriate to the demand. Cusum charts are also evaluated by EURL-CEFAO in order to identify problems that may cause a bias in the measurement method.     

Development of a proficiency testing scheme for a limited number of participants in the field of natural water analysis

A proficiency testing (PT) scheme was developed for a limited number of analytical laboratories participating in the analysis of natural water in Israel. Three fit-for-intended-use reference materials (RMs) were prepared for a pilot PT from natural water: RM-blank and two in-house reference materials (IHRMs) with seven analytes fortified to achieve different levels of analyte concentrations. The measurands for the PT participants were the traceable spike values certified in the IHRMs, i.e., added mass concentrations of the analytes. The RM-blank and IHRMs were found to be homogeneous and stable over 11 days, the time necessary for the experimental part of the PT. The RMs were distributed to the PT participants as unknown test items similar to routine samples. The test results were evaluated using their deviations from the IHRM-certified values. Eight Israeli laboratories took part in the interlaboratory comparison. Individual laboratory performance and metrological compatibility of the PT results of the participants, as a local group of laboratories, were evaluated for every analyte.     

Proficiency testing schemes for therapeutics and toxicology

 This report addresses the proposition: current proficiency testing (PT) schemes for therapeutics and toxicology can easily engender complacency. The proposition was based on the premise that there are substantial between-laboratory differences and objectives in the supply of services for the measurement of drugs, as a result of which it is difficult to design PT schemes which adequately cover the requirements of all laboratories. Thus, acceptable performance in a PT scheme may lead participants to consider that they have tested their procedures more rigorously than is the case. Four areas, validity of the PT scheme, PT material, frequency of testing and acceptance criteria are examined. Received: 15 April 2000 · Accepted: 15 April 2000     

Proficiency assessment of laboratories undertaking hard coal analysis

The Institute for Chemical Processing of Coal in Zabrze, Poland, organized a proficiency testing round to determine the mass fraction of ash, total sulfur and total carbon and the value of gross calorific value. Forty-four participants from Polish heat and power plants, research entities and independent laboratories reported results. The performance of the participants was evaluated using z and E _n scores. Over 85 % of the participants reported satisfactory results indicating the high level of proficiency of the Polish laboratories in the field.     

The influence of a quality system according to ISO/IEC 17025 on the performance of Brazilian laboratories in proficiency testing in the environmental area

This paper presents a performance analysis of laboratories in proficiency testing (PT) with the main objective of evaluating whether having accreditation to ISO/IEC 17025 or not having an implemented quality system has an influence on the performance of these laboratories. This research was conducted with data from an accredited PT scheme of physical–chemical water analyses. Nine metals were considered in the analysis, with a total of 896 results. The performance indicator was z-score. The proposed analysis was conducted through statistical methods of analysis of variance, Dunnett’s test, Chi-square and cross-tabulation. It was verified that there is a significant difference between the performance of accredited laboratories, recognised laboratories, and laboratories that do not have an implemented quality system. It was established that an accredited or recognised quality system has a positive influence on the laboratory performance, resulting in a lower z-score (in module) and in a larger percentage of satisfactory results.     

Proficiency testing in food microbiology: experience from implementation of ISO/IEC 17043 and ISO/TS 22117

The proficiency testing (PT) scheme ??AQUA?? for food microbiology was organised by the Istituto Zooprofilattico Sperimentale delle Venezie (IZSVe) according to ISO/IEC 17043 and ISO/TS 22117. This paper describes the IZSVe experience on the application of the above-mentioned standards for the PTs, with focus on the Enterobacteriaceae enumeration one. Freeze-dried food matrices contaminated with American Type Culture Collection bacterial strains were used as test samples for each microbiological PT organised by IZSVe. The sample homogeneity and stability were verified prior to distribution to participants and throughout the PT, respectively. The participating laboratories analysed samples using their routine methods, and results were transmitted to IZSVe. Data and methods used by each participating laboratory were analysed in order to evaluate the laboratory performance. With reference to the Enterobacteriaceae PT, the test samples were homogeneous and stable. In addition, most laboratory results were obtained using equivalent test methods. Statistical approaches applied to analyse data generated from all participating laboratories revealed similar outcomes as no significant outlying count and only 5?% of unacceptable results were observed. Finally, the z-score, with the standard deviation that does not vary from round toround, was applied to compare and to evaluate the performance of each laboratory over time highlighting possible persistent trends over several rounds.     

Key metrological issues in proficiency testing––response to “Metrological compatibility-a key issue in further accreditation” by K. Heydorn

A discussion of proficiency testing (PT) topics started by Heydorn (Accred Qual Assur 15:643–645, 2010) is continued in the present paper. The role of PT in the accreditation of testing/analytical laboratories, the use of consensus values (average or weighted average, median, observed standard deviation, etc.) and a metrological background of PT schemes are discussed. It is shown that metrological traceability, comparability, and compatibility, as well as commutability of a reference material, are the key issues of any PT scheme that applies certified reference material as test items. Metrological compatibility of PT results in such schemes is a property demonstrating the closeness of the PT results to the certified value in comparison with the measurement uncertainty of their difference. The metrological background is especially important for the selection and use of PT schemes for a limited number of participants (fewer than 30) as detailed in IUPAC/CITAC Guide on the topic published in 2010 in Pure Appl Chem 82(5):1099–1135.     

An interlaboratory study as useful tool for proficiency testing of chemical oxygen demand measurements using solid substrates and liquid samples with high suspended solid content

In 2008, the first Proficiency Testing Scheme of Chemical Oxygen Demand (1^stCOD-PT^ADG) was conducted to assess the results obtained for different research groups whose field work is mainly anaerobic digestion. This study was performed using four samples, two solid samples as raw materials and two solid samples to prepare high concentration suspended solid solutions. Invitations were sent to a large number of laboratories, mainly to anaerobic digestion research groups. Finally, thirty labs from sixteen countries agreed to participate, but for different reasons four participants could not send any data. In total, twenty-six results were reported to the COD-PT coordinator.This study showed the importance of continuous participation in proficiency testing (PT) schemes in order to compare the results obtained. Taking into account the lack of a general standard method and high quality certified reference materials (CRMs), the traceability of COD determination is not currently easy to check. In addition, the spread of participants’ results obtained was high and pointed to the advisability of using consensus values due to their unreliability. Therefore, the theoretical oxygen demand (ThOD) values were considered as assigned values for all the samples analysed. On the other hand, in this PT the established standard deviation (ESD) has been determined by the Horwitz modified function.Participants of this 1^stCOD-PT^ADG were asked to give a short report on the analytical method used. Although all the participants used potassium dichromate as their oxidant reagent, their experimental procedures were very different. With the purpose of comparing the results obtained, the different experimental conditions used were classified into five methods, corresponding to two main categories, open and closed reflux. The performance of laboratories was expressed by the z-score, whose value is considered satisfactory when z-score ≤±2. The overall analytical data evaluation showed that 64% of z-scores obtained were outside the accepted limits.     

The application of data from proficiency testing to laboratory accreditation according to ISO 17025

Current methods of testing laboratories for their proficiency in reporting correct measurement results are liable to substantial errors of the second kind. This means that laboratories with deflated uncertainties are accepted as proficient, even though their reported measurement results pave the way for erroneous conclusions. Only by using E _n numbers based on an accepted reference value with the lowest possible uncertainty can the risk of recognising incorrect measurement results be kept at an acceptable level. Based on an actual set of proficiency test (PT)-data for the concentration of Pb in water, this paper compares PT results obtained by methods using E _n numbers with methods based on z-scores. Kaj Heydorn is a technical assessor at The Danish Accreditation and Metrology Fund (DANAK).     

Analytical performance is improved by regular participation in proficiency testing: an analysis of data from the Aquacheck proficiency testing scheme

Analysis of historical data from the Aquacheck proficiency testing scheme demonstrates improvement in the performance of participants correlated to the frequency of participation. Analysis of the spread of results returned by all participants, defined by the percentage relative standard deviation (RSD) and the performance (Z) scores awarded to the individual participants, indicates that the overall performance for the group of participants is consistent. However, when the frequency of participation is analysed, it can be clearly seen that those laboratories who participated in the majority of the distributions available reported 10–15% more acceptable (Z-scores between +2 and ?2) results than those labs who took part with a greatly reduced frequency. When ‘more difficult’ analytes are considered, those laboratories who participated in greater than 75% of available rounds reported up to 30% more acceptable results than those laboratories who participated in 25% or fewer, of the available rounds.     

Do proficiency tests always verify laboratories’ performance? The case of FAPAS PT 0270

In this paper, the case of FAPAS PT 0270 “Doramectin and Oxfendazole in Sheep Liver” is discussed. During evaluation of the data received from participants (determination of total, oxidised oxfendazole residue and calculation of the sum of oxfendazole and oxfendazole sulfone residues), significant differences were observed between the results obtained by use of two analytical approaches. This phenomenon can be explained by the route of oxfendazole metabolism, which results in the presence of fenbendazole in the sample. This was not predicted by the provider; consequently, not all the necessary tests on the material were conducted. Due to the high uncertainty of the z-scores in this test, the results of the PT cannot be used for purposes of evaluation, and the benefits of participation in PT 0270 are questionable.     

Preparation of liquid milk for proficiency test and internal quality control for chemical elements in food

Laboratories dealing with the analysis of food of animal origin have to face the shortage of both appropriate certified reference materials and pertinent proficiency tests (PTs). Therefore, internal quality control (IQC) materials have to be prepared in-house. The combination matrix/analytes/concentration, as well as the physical state, is a key issue. The European Union Reference Laboratory for Chemical Elements in Food of Animal Origin (EURL-CEFAO), within its activity as PT provider, has set a procedure to prepare test items of liquid milk spiked with arsenic, cadmium and lead at adequate values of concentration. This work describes in detail the steps of this procedure. Homogeneity and stability of the PT test items thus prepared were evaluated by the EURL-CEFAO using a sensitive and accurate method accredited according to ISO/IEC 17025. The analyte concentrations in spiked milk were found stable up to 18 months which is of particular interest for cadmium due to the low value of spiking (roughly 5 μg/kg). The outcome proves that the procedure used to prepare liquid milk spiked with chemical elements is suitable to produce in-house reference materials stable enough for analytical purposes. Furthermore, this procedure is easy to perform and can be applied by routine laboratories to produce samples for their IQC.     

Application of ISO 13528 robust statistical method for external quality assessment of blood glucose measurements in China

External quality assessment (EQA) including extensive participants often result in abnormal distribution. Robust statistical methods are insensitive to slight deviation for a given probability model and can estimate the population parameters utilizing robust algorithm. The aim of this study was to evaluate the blood glucose testing performance of EQA in China using ISO 13528 robust statistical method. A total of 4571 participants measured two lots of materials at different concentrations and submitted the results through the Clinet-EQA reporting system V1.5. Data were collected and analyzed by Clinet-EQA evaluating system and SPSS 13.0. Participants were classified by method principles they adopted. Acceptability of relative differences between each result and robust mean of the group was calculated using biological variation desirable quality specification for allowable total error. The kernel density plots and Youden Plot were also applied to show the distribution of z-scores. The robust means of the two lots were 6.31 mmol/L and 15.71 mmol/L, and the robust standard deviations were 0.23 mmol/L and 0.5 mmol/L, respectively. Acceptability of relative differences was more than 90 % among all groups. About 91.5% and 89.7 % of the participants out of the two lots fulfilled the condition of |z| ≤ 2. In addition, 87.2 % participants fell inside the confidence ellipse for probability levels of 95 % in Youden Plot. According to the statistical results, we can conclude that most participants are qualified for performing blood glucose test and ISO 13528 robust statistical method is appropriate for EQA result analysis.