Proficiency testing in analytical chemistry, microbiology and laboratory medicine: working discussions on current practice and future directions

A summary of the working group (WG) discussions on proficiency testing (PT) and external quality assessment (EQA) held at the Eurachem Workshop, Rome, 5–7 October 2008 is provided. The eight WG’s covered a range of issues concerned with current practice and future directions; how frequently should laboratories participate in PT/EQA? (WG1); developments in PT/EQA within the EU—what is required in future? (WG2); what issues do developing countries face with regards to PT/EQA? (WG3), what issues are specific to microbiology PT/EQA? (WG4); what new fields are emerging for PT/EQA? (WG5); what will be the impact of the new ISO/IEC 17043 standard? (WG6); do current PT/EQA schemes meet the needs of participants? (WG7); and what are the issues that affect the quality of proficiency test items? (WG8). Delegates with different backgrounds were on each WG in order to capture a range of views and experience from a number of different sectors. Working group representatives included PT/EQA providers, participants in PT/EQA schemes and end users of PT results such as accreditation bodies and regulatory authorities, from countries around the world.     

Proficiency testing in analytical chemistry, microbiology and laboratory medicine: working group discussions on current practice and future directions

A summary of the working group (WG) discussions on proficiency testing (PT) and external quality assessment (EQA) held at the joint EURACHEM/CITAC/EQALM workshop, Bracknell, UK, 16–18 February 2003 is provided. The nine WGs covered a range of issues concerned with current practice and future directions; PT/EQA as a tool for regulators (WG1); PT/EQA as a tool for accreditation (WG2); evaluation of performance and uncertainty (WG3); frequency of PT/EQA participation (WG4); selection of appropriate PT/EQA schemes (WG5); added value of PT/EQA and cost benefit evaluation (WG6); global harmonisation and rationalisation (WG7); new technical areas and challenges in PT/EQA (WG8); and accreditation of PT/EQA providers (WG9). Participants with different backgrounds were on each WG in order to capture a range of views and experience from different sectors. The discussions reflected on the keynote lectures and built, in many cases, on discussions at previous workshops in 2000 and 2002.     

The challenges and benefits of implementing the requirements of ISO/IEC 17043 by PT/EQA providers

ISO/IEC 17043 Conformity assessment??general requirements for proficiency testing is intended to replace previous international guides used to assess the competence of proficiency testing (PT) providers. It expands on the requirements of previous guides and is intended to accommodate PT providers of calibration laboratories and testing laboratories handling both qualitative and quantitative data. QMP-LS is an office-based external quality assessment provider for medical laboratories in Ontario, Canada and operates 46 different PT schemes for approximately 250 diagnostic tests. In 2010, these schemes were accredited to ISO/IEC 17043. Schemes included tests from the following disciplines: chemistry, hematology, microbiology, transfusion medicine, cytology, histology and genetics. Thirty of the schemes were qualitative. The challenges and benefits of implementing ISO/IEC 17043 are discussed, with particular emphasis on clauses addressing the following requirements: statistical design, determination of assigned value, homogeneity and stability testing, packaging, labeling and distribution, performance evaluation and subcontracting services.     

Accrediting PT/EQA providers to ISO/IEC 17043

After many years and having several different attempts for the accreditation of proficiency testing provider (PT provider), there is finally one stand-alone standard defining the requirements for the competence of PT providers and therefore an internationally harmonised basis for the accreditation of proficiency test providers. Since February 2010, the ISO/IEC 17043:2010 has replaced ISO/IEC Guide 43:1997 and ILAC G 13:2007. The philosophy of the standard about subcontracting work is different to this of the standards mostly used for accreditation like ISO/IEC 17025:2005 or ISO/IEC 17020:2004, etc. Besides the planning of the proficiency tests (PT), the performance evaluation and the authorisation of the PT reports the ISO/IEC 17043:2010 allows subcontracting for the rest of the work when providing PTs. This is a challenge for the assessors to judge about the competence of a PT provider. In numerous paragraphs, the standard sets very detailed requirements. Nevertheless, there is room for interpretation. For these cases, for example, contracts for subcontractors, procedure for the advisory board, minimum requirements for PT certificates, etc., some proposals are given to enable harmonised approach for the assessment of PT providers.     

Surveys on the accreditation of providers of proficiency testing and external quality assessment schemes

Two surveys among providers of proficiency testing (PT) and external quality assessment (EQA) schemes were carried out during 2004 and 2005. The main objectives were to explore the current status of accreditation/certification and collect the providers’ views. Information based on the response from 160 providers in 32 countries reveals a strong tendency towards accreditation of PT/EQA. It is shown that this type of accreditation is based on several combinations of normative documents, hence illustrating a lack of harmonisation of national accreditation bodies. The surveys also show that schemes are operated under considerably different conditions and that providers’ competence may or may not be underpinned by other certification and/or accreditation. This paper elaborates on a number of issues related to PT/EQA accreditation, including customers’ views, normative documents, providers’ experience from the accreditation process, views expressed by international organisations, and effects of accreditation on participation fees, quality and availability.     

Proficiency testing in analytical chemistry, microbiology, and laboratory medicine – working group discussions on current status, problems, and future directions

Working group (WG) discussions on proficiency testing (PT) held at the joint Eurachem/ EQALM workshop, Borås, Sweden, 24–26 September 2000 are summarized. The discussions focused on aspects of PT and accreditation (WG 1), general aspects of PT in analytical chemistry (WG 2), microbiology (WG 3), and laboratory medicine (WG 4), incorporation of measurement uncertainty into PT schemes (WG 5), international harmonization of PT schemes (WG 6), and the role of PT in the international structure of chemical measurement (WG 7). Current status, problems and future directions are identified. Each WG contained a majority of participants experienced in the subject being covered by that WG, and a few participants with different expertise. This was done to promote cross-fertilization of ideas between sectors, a key objective of the workshop. The WG issues reflected the content of the keynote lectures and some issues were covered from different perspectives by more than one group.     

Development of a standard for the organization of the interlaboratory comparisons in microbiology of food

The ILAC G13 Guide and the ISO IEC Guide 43-1 are the common, general and horizontal bases for accreditation of providers of proficiency testing and interlaboratory comparisons used by several accreditation bodies. Despite their widespread use, these guides omit specific technical requirements, and sometimes even elements of quality management, clearly defined for the organization of proficiency-testing schemes for microbiology of food. The Sub-Committee 9 “Microbiology of food” of ISO TC34 has created a working group (WG4) to establish a standard detailing such specific requirements for microbiology of food. This technical standard will describe those specific requirements of proficiency testing which organizers of proficiency-testing schemes and any subcontractors must satisfy in addition to the requirements of the ILAC G13 Guide and the ISO IEC Guide 43-1 to achieve accreditation or other recognition.     

Proficiency testing schemes for the assessment of Legionella PCR methodologies

Standard operating procedures used for the detection of bacteria in environmental samples are primarily based on bacterial growth on specific culture media and confirmation by biochemical and/or immunological tests. In the case of Legionella, isolation on BCYE-?? medium is the standard method, although it presents a number of drawbacks, and for this reason, the implementation of molecular methods, mainly those based on PCR, has increased over the last years. Following the ISO/IEC 17025, laboratories need an external evaluation of their work to assure the quality of the results they are producing, and the participation in proficiency testing (PT) schemes is compulsory. For those water-testing laboratories using PCR methods for Legionella, we have developed a PT scheme accredited according to ISO/IEC 17043. The preparation and the statistical analysis of the results are performed following this standard and the ISO 13528. The used samples have a very rapid and easy to use format, consisting of tablets with inactivated freeze-dried Legionella cells or freeze-dried Legionella DNA. In this PT scheme, participants evaluate both Legionella pneumophila and Legionella spp. detection systems and control the whole PCR process from the water sample concentration until the PCR results.     

Review of the new edition of ISO 13528

The international standard ISO 13528 is an important reference document for proficiency testing providers and testing laboratories alike, which provides guidance on the use of statistical methods in proficiency testing. With a view to bringing the document in harmony with the ISO/IEC 17043:2010, the first edition of ISO 13528 was revised and the new edition was published in September 2015. This paper attempted to review and discuss the major changes introduced in the new edition of ISO 13528, particularly the part on statistical design with respect to the objective of proficiency testing scheme and the considerations on assigned value determination and performance scoring. Moreover, procedures provided in the new edition of ISO 13528 for qualitative proficiency testing schemes and various robust statistical methods would also be briefly discussed in the paper.     

Identification of possible technical problems in determination of the major inorganic constituents of brown-rice flour by evaluating proficiency test results

To support skill upgrading in analysis of inorganic constituents of environmental and food samples, the National Metrology Institute of Japan (NMIJ) and the National Food Research Institute (NFRI) have organized a proficiency test (PT) of determination of Mn, Fe, Cu, Zn, As, and Cd in brown-rice flour based on the international standard (ISO/IEC 17043:2010). One hundred and thirty-three sets of reports were assessed by use of the E _n -number and z-score approaches in accordance with ISO/IEC 17043 and the international harmonized protocol for PT. The PT results and analytical procedures, reported in detail, were reviewed, and possible technical reasons for questionable or unsatisfactory results are discussed.

Evaluation of proficiency tests in microbiological analysis: enumeration of aerobic microorganisms

South Korea’s Ministry of Food and Drug Safety (MFDS) has been developing programs for the inspection and accreditation of food sanitation inspection institutions. Food sanitation inspection institutions such as MFDS regional offices, the Research Institute of Public Health and Environment and authorized private service providers in South Korea must participate in proficiency testing (PT) programs to comply with the Food Sanitation Act and MFDS Notification No. 2012-112. As the PT provider, the MFDS annually plans various microbiological and chemical PT programs for foods, cosmetics, and pharmaceutical products in accordance with ISO/IEC 17043. The aim of this project was to evaluate the performance of microbiological PT programs to ensure the quality of their routine test results. The test materials used were freeze-dried BioBalls from BTF Pty Ltd. Homogeneity and stability were investigated to assess the adequacy of the selected test materials. This project also contains data from inter-laboratory comparisons organized by MFDS in 2011 and 2012. More than 50 laboratories attended the PT program and submitted their results. Laboratory results were rated with z-scores according to the international standard ISO 13528. The results from 2011 and 2012 revealed that all participating laboratories had similar levels of proficiency. Most of the participants received a rating of “Satisfactory.” Moreover, the percentage of participants who received a rating of “Unsatisfactory” decreased from 3.5 % in 2011 to 2.0 % in 2012.     

Addressing issues associated with the development and management of PT programs and with their optimum use

Over 200 participants from 54 countries took part in working group (WG) discussions on proficiency testing (PT) held in conjunction with the Global Odyssey 2002 conference on 24–26 February, 2002 in Atlanta, Georgia, USA. Each WG addressed a series of questions focusing on the role of PT in laboratory quality management (WG1), alternatives to traditional PT (WG2), international aspects of PT programs (WG3), quality management of PT programs (WG4), how to assess laboratory performance (WG5), and developing and funding PT for resource-limited countries and regions (WG6). Each large WG was further subdivided into smaller groups to allow maximum participation, which was one objective of the WGs. Some subgroups addressed the same questions, but in most cases they drew very similar conclusions. Individual participants were surprised to discover the truly global nature of some problems with PT, while at the same time how specific differences among countries or regions can either enhance or hinder opportunities to implement PT programs. Received: 13 April 2002 Accepted: 25 April 2002     

Experiences from the assessment of proficiency testing providers

Following a pilot project in 2001–2002, UKAS accredited nine PT providers. During the course of the pilot and at subsequent surveillance visits, UKAS has gained experience in assessing organisations against the criteria detailed within ISO/IEC Guide 43-1:1997 and ILAC G13:2000. This article explores some of the key aspects that arose during the pilot and how UKAS has accommodated the addition of the accreditation of PT providers in to its range of services.Presented at the Eurachem PT Workshop September 2005, Portorož, Slovenia     

Preparation of in-house reference material of benzylpenicillin in milk and results of a Brazilian proficiency testing scheme

Two milk test materials containing benzylpenicillin and a benzylpenicillin-free material were prepared and used for the operation of a Brazilian proficiency testing (PT) scheme according to the requirements of ISO/IEC 17043. The PT scheme was designed to include laboratories in charge of confirmatory analysis as well as screening analysis. Sets of sample vials, benzylpenicillin-free and spiked items containing 0.2 g of lyophilized milk, were distributed to each participant. Of 20 participants who reported screening data, 18 accomplished satisfying results, providing evidence of the capability of Brazilian laboratories to produce reliable qualitative information. The assigned value (robust average calculated from 6 results) and its uncertainty were (8.28 ± 0.52) μg/L. Participants’ performance was evaluated using z-scores. A small number of participants were able to report quantitative benzylpenicillin results, and consequently care should be taken in the interpretation of the laboratory’s performance, considering the statistical nature of z-scores. Long-term stability testing on remaining PT items showed that a stable benzylpenicillin quality control material was produced and can be used for internal quality control or validation purposes.     

Proficiency testing in food microbiology: experience from implementation of ISO/IEC 17043 and ISO/TS 22117

The proficiency testing (PT) scheme ??AQUA?? for food microbiology was organised by the Istituto Zooprofilattico Sperimentale delle Venezie (IZSVe) according to ISO/IEC 17043 and ISO/TS 22117. This paper describes the IZSVe experience on the application of the above-mentioned standards for the PTs, with focus on the Enterobacteriaceae enumeration one. Freeze-dried food matrices contaminated with American Type Culture Collection bacterial strains were used as test samples for each microbiological PT organised by IZSVe. The sample homogeneity and stability were verified prior to distribution to participants and throughout the PT, respectively. The participating laboratories analysed samples using their routine methods, and results were transmitted to IZSVe. Data and methods used by each participating laboratory were analysed in order to evaluate the laboratory performance. With reference to the Enterobacteriaceae PT, the test samples were homogeneous and stable. In addition, most laboratory results were obtained using equivalent test methods. Statistical approaches applied to analyse data generated from all participating laboratories revealed similar outcomes as no significant outlying count and only 5?% of unacceptable results were observed. Finally, the z-score, with the standard deviation that does not vary from round toround, was applied to compare and to evaluate the performance of each laboratory over time highlighting possible persistent trends over several rounds.     

The use of multiple correspondence analysis for the evaluation of inter-laboratory comparisons

Participation in inter-laboratory comparisons (ILC) is one of the recommended means of external quality control according to ISO/IEC 17025:2005. Providers of ILC or proficiency test (PT) schemes collect, besides the measurement results on the test samples, information on the sample treatment and measurement procedure. The objective of this paper is to evaluate in a non-traditional way, using numerical and non-numerical data provided by the participants in IMEP-20 (trace elements in tuna fish), the additional information concerning the applied analytical methods and the accreditation/nomination status. Arsenic was taken as an example. The basic statistical procedure for the evaluation of questionnaire information was the multiple correspondence analysis (MCA). Univariate clustering techniques were applied for the categorization of the numerical data (measurement values). The methodology of the evaluation of supplementary non-numeric information used in this paper might serve (a) to providers of ILC (PT) schemes to modify/improve their questionnaires and (b) to give laboratories better guidance in the methods used for the determination of various analytes in different matrices. This paper is meant serve as a guide for the possible interpretation of the questionnaires accompanying ILC schemas. Presented at the 3rd International Conference on Metrology, November 2006, Tel Aviv, Israel     

Performance evaluation for proficiency testing with a limited number of participants

Proficiency testing (PT) is an essential tool for laboratories to assess their competency. Also, participation in PT has become one of the mandatory requirements for laboratory to seek accreditation according to ISO/IEC 17025. For this reason, the effectiveness of performance evaluation by PT scheme is of great concern for the participants and for accreditation bodies as well. In practice, owing to unavailability of other appropriate alternatives, PT scheme providers may have to choose using consensus values to evaluate the performance of participants. However, such consensus values approach was not recommended by relevant international guidelines for PT schemes with limited number of participants. With the use of Monte Carlo simulation technique, this study attempted to investigate the effectiveness of using consensus values for performance evaluation in PT schemes with limited number of participants. The simulation process was schemed according to the statistical model provided by ISO 5725-1 for laboratory measurement results, which covered components like method bias, laboratory bias, and measurement precision. The effectiveness of the consensus value approach was expressed as the percentage of participants in a simulation run could get the same evaluation result, either satisfactory or unsatisfactory, against the “true value.” The findings indicated that the number of participants, choice of consensus values, mass fraction of analyte, method bias, laboratory bias, and measurement repeatability of participating laboratories would all affect the effectiveness of the consensus value approach but at different extent. However, under certain circumstances, use of consensus value could still be considered as an acceptable approach for performance evaluation even the number of participants was limited. Some of the findings were further verified using real data from PT schemes where appropriate certified reference materials or reliable reference values were available.     

Proficiency testing in analytical chemistry,microbiology and laboratory medicine – working group discussions on current status,problems and future directions

Working group (WG) discussions on proficiency testing and external quality assessment held at the Eurachem workshop, Portorož, Slovenia 26–27 Sept 2005 are summarised. The discussions focused on performance criteria (WG 1), different aspects of accreditation (WGs 2–3), the revised international harmonised protocol (WG 4), pre- and post-analytical schemes (WG 5), Internet applications (WG 6), experience from the CoEPT project (WG 7), and future aspects (WG 8). Current status, problems and future directions were identified. The WG contained a mix of participants with different expertise. This was done to promote cross-fertilisation of ideas between sectors. The WG issues reflected the content of the keynote lectures and some issues were covered from different perspectives by more than one WG.Presented at the Eurachem PT Workshop September 2005, Portorož, Slovenia.     

National External Quality Assessment Schemes for microbiology, parasitology, and virology in Europe

Quality assurance is an important aspect of laboratory management. One of the activities involved is the participation in external quality assessment (EQA) schemes by the clinical laboratory. These EQA schemes should be organised according to well-defined guidelines, such as the ESO/IEC Guide 43. The present work provides an inventory of the EQA schemes organised in Europe. The schemes are related to the fields of bacteriology, parasitology and virology. For each field various surveys have been organised. Data such as the number of participants, frequency of survey and number of samples in a survey are presented. The nature and way of manufacturing the control material is specified.     

Note on the bimodal distribution of PT results

The practitioner’s report "Treatment of bimodality in proficiency test of pH in bioethanol matrix" by Sarmanho et al. deals with a proficiency test (PT) for the measurement of pH in bioethanol. One group of 19 participating laboratories used an electrode that contained saturated LiCl and the other one KCl. The results of the LiCl group are biased, and those of the KCl group are unbiased as compared with the assigned reference value \(y_\mathrm{{RM}}\), and hence, the distribution of the laboratory means is bimodal. In order to deal with this situation, Sarmanho et al. present and carry out two different methods of fitting the bimodal distribution of the laboratory means: a fit by a mixture of two normal distributions and a Gaussian kernel estimation. Since this rather advanced statistical analysis does not help to solve the practical problem, some recomendations of ISO 17043 are discussed that might be useful for the coordinator of the PT and as well for the participating laboratories.