Interlinkages and recognition of proficiency-testing schemes

 Because proficiency testing (PT) is increasingly used for the accreditation of testing laboratories and as a tool for backing up existing multilateral recognition arrangements between accreditation bodies, the question of performance and comparability of the proficiency-test providers arises. In this paper different approaches to assess the equivalence of European PT schemes and the competence of their providers are presented. As a first step a workshop is proposed to agree on a pilot study. The final aim is to create confidence in the existing PT schemes in Europe and to use them as common European tools.     

Do we need to accredit proficiency testing schemes?

 Proficiency testing (PT) is being increasingly used as an important quality assurance tool for laboratories. The subject of quality of the providers of PT schemes has been discussed increasingly in recent years. Some countries have implemented systems for the accreditation of PT schemes. This paper looks at the background to the accreditation of PT schemes, the likely mechanisms which could be employed for accreditation, and some of the practical aspects.     

Accreditation of PT providers – is it worth the money?

This topic is being presented from two perspectives, namely the views of an accreditation body and those of an accredited provider of external quality assessment schemes. The first perspective, from an accreditation body, is based on the experience of Australia's national laboratory accreditation body (NATA), which has accredited fourteen proficiency testing (PT) providers and is currently processing an application from another. The second perspective is based on the experience of EQUALIS, the national quality assurance organisation for laboratory medicine in Sweden, which was granted accreditation in 2002.Presented at 5^th Eurachem Workshop on Proficiency testing in analytical chemistry, microbiology and laboratory medicine, Portorož (Slovenia) 25–27 Sept 2005     

Proficiency testing in analytical chemistry, microbiology and laboratory medicine: working group discussions on current practice and future directions

A summary of the working group (WG) discussions on proficiency testing (PT) and external quality assessment (EQA) held at the joint EURACHEM/CITAC/EQALM workshop, Bracknell, UK, 16–18 February 2003 is provided. The nine WGs covered a range of issues concerned with current practice and future directions; PT/EQA as a tool for regulators (WG1); PT/EQA as a tool for accreditation (WG2); evaluation of performance and uncertainty (WG3); frequency of PT/EQA participation (WG4); selection of appropriate PT/EQA schemes (WG5); added value of PT/EQA and cost benefit evaluation (WG6); global harmonisation and rationalisation (WG7); new technical areas and challenges in PT/EQA (WG8); and accreditation of PT/EQA providers (WG9). Participants with different backgrounds were on each WG in order to capture a range of views and experience from different sectors. The discussions reflected on the keynote lectures and built, in many cases, on discussions at previous workshops in 2000 and 2002.     

Addressing issues associated with the development and management of PT programs and with their optimum use

Over 200 participants from 54 countries took part in working group (WG) discussions on proficiency testing (PT) held in conjunction with the Global Odyssey 2002 conference on 24–26 February, 2002 in Atlanta, Georgia, USA. Each WG addressed a series of questions focusing on the role of PT in laboratory quality management (WG1), alternatives to traditional PT (WG2), international aspects of PT programs (WG3), quality management of PT programs (WG4), how to assess laboratory performance (WG5), and developing and funding PT for resource-limited countries and regions (WG6). Each large WG was further subdivided into smaller groups to allow maximum participation, which was one objective of the WGs. Some subgroups addressed the same questions, but in most cases they drew very similar conclusions. Individual participants were surprised to discover the truly global nature of some problems with PT, while at the same time how specific differences among countries or regions can either enhance or hinder opportunities to implement PT programs. Received: 13 April 2002 Accepted: 25 April 2002     

Aspects of auditing in clinical laboratories

Quality and patient safety are terms that both providers and recipients of healthcare are very familiar with. Accreditation is another term that is closely linked to quality and patient safety. Audit is a systematic, independent, and documented process for obtaining evidence and evaluating it objectively to determine the extent to which audit criteria are fulfilled. Accreditation and audit are integral components of the same process. Three different types of audit are well recognized—internal, external, and co-operative. Reading of relevant documents, observation of laboratory practices, and asking open-ended probing questions are important auditing techniques. For auditing to be successful, experienced, qualified, and well trained auditors are essential. Furthermore, the auditor should be open-minded, not prejudiced, a team player and effective communicator, both in writing and verbally. In many instances, the emphasis for seeking laboratory accreditation has shifted from building quality systems—to produce reliable results and ensure patient safety—to just passing the inspection. Recently, the emphasis for laboratory quality improvement has been placed on pre and post-analytical processes in preference to analytical quality. The analytical quality of laboratory results is still far from ideal and it may be detrimental if less emphasis is placed on this aspect of laboratory medicine. Auditing or on-site inspection as a regulatory tool does not work or present a realistic picture of laboratory quality. A continuous quality improvement approach will help laboratories to build quality into their systems. Presented at the Conference “Excellence in Laboratory Medicine”, November 2007, Al Ain, United Arab Emirates.     

Proficiency testing in analytical chemistry,microbiology and laboratory medicine – working group discussions on current status,problems and future directions

Working group (WG) discussions on proficiency testing and external quality assessment held at the Eurachem workshop, Portorož, Slovenia 26–27 Sept 2005 are summarised. The discussions focused on performance criteria (WG 1), different aspects of accreditation (WGs 2–3), the revised international harmonised protocol (WG 4), pre- and post-analytical schemes (WG 5), Internet applications (WG 6), experience from the CoEPT project (WG 7), and future aspects (WG 8). Current status, problems and future directions were identified. The WG contained a mix of participants with different expertise. This was done to promote cross-fertilisation of ideas between sectors. The WG issues reflected the content of the keynote lectures and some issues were covered from different perspectives by more than one WG.Presented at the Eurachem PT Workshop September 2005, Portorož, Slovenia.     

Comparability of analytical results obtained in proficiency testing based on a metrological approach

A “yes–no” type of criterion is proposed for the assessment of comparability of proficiency testing (PT) results when the PT scheme is based on a metrological approach, i.e. on the use of a reference material as the test sample, etc. The criterion tests a null hypothesis concerning the insignificance of a bias of the mean of the results from a traceable value certified in the reference material used for the PT. Reliability of such assessment is determined by the probabilities of not rejecting the null hypothesis when it is true, and rejecting it when it is false (the alternative hypothesis is true). It is shown that a number of chemical, metrological and statistical reasons should be taken into account for careful formulation of the hypotheses, enabling the avoidance of an erroneous assessment of the comparability. The criterion can be helpful for PT providers and laboratory accreditation bodies in analysis of PT results.     

Nonparametric assessment of comparability of analytical results obtained in proficiency testing based on a metrological approach

A nonparametric sign test is implemented for assessment of comparability of proficiency testing (PT) results when their distribution differs from the normal or other known distribution. It allows testing the null hypothesis about insignificance of the bias of median of results obtained in PT from the traceable certified value of the reference material used in PT as test items, i.e., the hypothesis stating that comparability of the PT results is successful. Probability of type I error of rejecting the null hypothesis when it is true, and probability of type II error of it not rejecting when it is false (the alternative hypothesis about unsuccessful comparability is true) are considered. The test can be helpful for PT providers and laboratory accreditation bodies in analysis of PT results when the comparability criterion developed for a normal results distribution (Accred. Qual. Assur. 10:466–470) is not applicable.     

Estimation of the uncertainty of mass measurements from in-house calibrated analytical balances

The uncertainty evaluation of mass measurements when using “in-house” calibrated analytical balances is revisited according to the Guide to the expression of Uncertainty Measurement (GUM). The calibration of analytical balances is discussed according to the guidelines of several bodies such as ASTM, UKAS and DKD/PTB. The remainder components of uncertainty can be estimated from the balance data sheet specifications.     

Accrediting PT/EQA providers to ISO/IEC 17043

After many years and having several different attempts for the accreditation of proficiency testing provider (PT provider), there is finally one stand-alone standard defining the requirements for the competence of PT providers and therefore an internationally harmonised basis for the accreditation of proficiency test providers. Since February 2010, the ISO/IEC 17043:2010 has replaced ISO/IEC Guide 43:1997 and ILAC G 13:2007. The philosophy of the standard about subcontracting work is different to this of the standards mostly used for accreditation like ISO/IEC 17025:2005 or ISO/IEC 17020:2004, etc. Besides the planning of the proficiency tests (PT), the performance evaluation and the authorisation of the PT reports the ISO/IEC 17043:2010 allows subcontracting for the rest of the work when providing PTs. This is a challenge for the assessors to judge about the competence of a PT provider. In numerous paragraphs, the standard sets very detailed requirements. Nevertheless, there is room for interpretation. For these cases, for example, contracts for subcontractors, procedure for the advisory board, minimum requirements for PT certificates, etc., some proposals are given to enable harmonised approach for the assessment of PT providers.     

The correlation of laboratory performance in proficiency testing with other QA characteristics

Although it seems self-evident that proficiency testing (PT) and accreditation can be expected to improve quality, their relative benefits remain uncertain as does their efficacy. The study reported here examines the following issues: (a) Why do laboratories take part in PT schemes? (b) How does participation in PT fit in with a laboratory's overall quality assurance (QA) system? (c) Is there a link between a laboratory's performance in specific PT and it's QA system? (d) How does PT performance change with time and how do laboratories respond to poor performance? The overall conclusion is that there is no evidence from the present study that laboratories with third-party assessment (accreditation and certification) perform any better in PT than laboratories without. The validity of this conclusion and its significance for the future design and operation of such schemes requires further investigation. In particular, study is required of the degree to which good performance in open PT correlates with blind PT performance, where laboratories are not aware that the samples being analysed are part of a quality assessment exercise.     

Proficiency testing in analytical chemistry, microbiology and laboratory medicine: working discussions on current practice and future directions

A summary of the working group (WG) discussions on proficiency testing (PT) and external quality assessment (EQA) held at the EURACHEM Workshop, Istanbul, 3?C6 October 2011, is provided. The six WGs covered a range of issues concerned with current practice and future directions; implementing the requirements of ISO/IEC 17043 by PT/EQA providers (WG1); accrediting PT/EQA providers to ISO/IEC 17043 (WG2); pre- and post-analytical aspects in PT/EQA (WG3); evaluating participant performance in qualitative PT/EQA schemes (WG4); establishing PT/EQA schemes in developing countries (WG5); and establishing acceptability criteria in microbiology PT/EQA schemes (WG6). Delegates with different backgrounds were on each WG in order to capture a range of views and experience from a number of different sectors. Working group representatives included PT/EQA providers, participants in PT/EQA schemes and end-users of PT results such as accreditation bodies and regulatory authorities, from countries around the world.     

Environmental analysis in the Czech Republic: organisation of proficiency testing and its use in laboratory assessment

The history, origin, and development of a system for monitoring and assessing water and other environmental laboratories in the Czech Republic is described. The system started in 1991 and has matured to its present complexity with similarities to the accreditation systems found in other countries. Differences from internationally recognized procedures are being corrected step by step. During the first year of its existence ASLAB, as part of its brief, organised proficiency testing (PT) programs for fifty laboratories. Today the total number of regularly participating laboratories exceeds 700 from the Czech Republic, the Slovak Republic, and Germany. This paper describes the ASLAB PT system, discusses some experiences with its use, and describes the use of PT results in assessment of the competence of laboratories. Received: 12 October 2000 Accepted: 7 January 2001     

Performance evaluation for proficiency testing with a limited number of participants

Proficiency testing (PT) is an essential tool for laboratories to assess their competency. Also, participation in PT has become one of the mandatory requirements for laboratory to seek accreditation according to ISO/IEC 17025. For this reason, the effectiveness of performance evaluation by PT scheme is of great concern for the participants and for accreditation bodies as well. In practice, owing to unavailability of other appropriate alternatives, PT scheme providers may have to choose using consensus values to evaluate the performance of participants. However, such consensus values approach was not recommended by relevant international guidelines for PT schemes with limited number of participants. With the use of Monte Carlo simulation technique, this study attempted to investigate the effectiveness of using consensus values for performance evaluation in PT schemes with limited number of participants. The simulation process was schemed according to the statistical model provided by ISO 5725-1 for laboratory measurement results, which covered components like method bias, laboratory bias, and measurement precision. The effectiveness of the consensus value approach was expressed as the percentage of participants in a simulation run could get the same evaluation result, either satisfactory or unsatisfactory, against the “true value.” The findings indicated that the number of participants, choice of consensus values, mass fraction of analyte, method bias, laboratory bias, and measurement repeatability of participating laboratories would all affect the effectiveness of the consensus value approach but at different extent. However, under certain circumstances, use of consensus value could still be considered as an acceptable approach for performance evaluation even the number of participants was limited. Some of the findings were further verified using real data from PT schemes where appropriate certified reference materials or reliable reference values were available.     

Accreditation to ISO 17025:2005 for the Radioactivity Metrology Group of the UK’s National Physical Laboratory

In the mid 1990s, the National Physical Laboratory (NPL) took the decision to seek external accreditation to the then UK national accreditation standard (M10, M10 supplement and M11) through the NPL’s National Measurement Accreditation Service (NAMAS). This paper details the reasoning behind that initial decision and, in particular, how this impinged on the day-to-day activities of the NPL’s Radioactivity Metrology Group (RMG). In the intervening decade, the accreditation standard has changed considerably; accreditation is now to the international standards ISO 9001:2000 (Quality Management Systems: Requirements) and ISO 17025:2005 (General Requirements for the Competence of Testing and Calibration Laboratories); accreditation is now carried out by a wholly separate successor organization to NAMAS, the United Kingdom Accreditation Service (UKAS). To meet the new accreditation requirements the RMG: realigned it’s scope of work; streamlined and consolidated written procedures, references and appendices; centralized the collection of written procedures, and clarified the document identification system. Future developments will include efforts for RMG accreditation for conducting proficiency tests and providing reference materials.     

Proficiency testing in analytical chemistry, microbiology, and laboratory medicine – working group discussions on current status, problems, and future directions

Working group (WG) discussions on proficiency testing (PT) held at the joint Eurachem/ EQALM workshop, Borås, Sweden, 24–26 September 2000 are summarized. The discussions focused on aspects of PT and accreditation (WG 1), general aspects of PT in analytical chemistry (WG 2), microbiology (WG 3), and laboratory medicine (WG 4), incorporation of measurement uncertainty into PT schemes (WG 5), international harmonization of PT schemes (WG 6), and the role of PT in the international structure of chemical measurement (WG 7). Current status, problems and future directions are identified. Each WG contained a majority of participants experienced in the subject being covered by that WG, and a few participants with different expertise. This was done to promote cross-fertilization of ideas between sectors, a key objective of the workshop. The WG issues reflected the content of the keynote lectures and some issues were covered from different perspectives by more than one group.     

Accreditation of proficiency-testing schemes in The Netherlands

Proficiency testing by laboratories, national accreditation bodies, and other third parties is becoming more and more considered as a standard and integral part of the quality control system. Therefore, it is of utmost importance that the quality of the provided proficiency-testing (PT) service is outstanding. If PT-schemes are set up in order to help laboratories monitor and improve their quality, PT-schemes need not only be of high quality themselves, but the organizer also needs to be able to demonstrate this. In The Netherlands formal accreditation of the organization of proficiency testing schemes is used as a tool to guarantee high quality schemes and also to enable organizers to demonstrate their competence. Since 1996, the Dutch Council for Accreditation (RvA) has used the ISO-Guide 43–1 to assess PT-organizers in The Netherlands. From January 2000, the ISO-guide 43–1 was replaced by the ILAC G13 document for assessing organizers. Up till now, four institutes have been accredited by the RvA for the organization of PT-schemes.     

Evaluation of the use of consensus values in proficiency testing programmes

Proficiency testing (PT) is an essential tool used by laboratory accreditation bodies to assess the competency of laboratories. Because of limited resources of PT providers or for other reasons, the assigned reference value used in the calculation of z-score values has usually been derived from some sort of consensus value obtained by central tendency estimators such as the arithmetic mean or robust mean. However, if the assigned reference value deviates significantly from the ‘true value’ of the analyte in the test material, laboratories’ performance will be evaluated incorrectly. This paper evaluates the use of consensus values in proficiency testing programmes using the Monte Carlo simulation technique. The results indicated that the deviation of the assigned value from the true value could be as large as 40%, depending on the parameters of the proficiency testing programmes under investigation such as sample homogeneity, number of participant laboratories, concentration level, method precision and laboratory bias. To study how these parameters affect the degree of discrepancy between the consensus value and the true value, a fractional factorial design was also applied. The findings indicate that the number of participating laboratories and the distribution of laboratory bias were the prime two factors affecting the deviation of the consensus value from the true value.     

The challenges and benefits of implementing the requirements of ISO/IEC 17043 by PT/EQA providers

ISO/IEC 17043 Conformity assessment??general requirements for proficiency testing is intended to replace previous international guides used to assess the competence of proficiency testing (PT) providers. It expands on the requirements of previous guides and is intended to accommodate PT providers of calibration laboratories and testing laboratories handling both qualitative and quantitative data. QMP-LS is an office-based external quality assessment provider for medical laboratories in Ontario, Canada and operates 46 different PT schemes for approximately 250 diagnostic tests. In 2010, these schemes were accredited to ISO/IEC 17043. Schemes included tests from the following disciplines: chemistry, hematology, microbiology, transfusion medicine, cytology, histology and genetics. Thirty of the schemes were qualitative. The challenges and benefits of implementing ISO/IEC 17043 are discussed, with particular emphasis on clauses addressing the following requirements: statistical design, determination of assigned value, homogeneity and stability testing, packaging, labeling and distribution, performance evaluation and subcontracting services.