Key metrological issues in proficiency testing––response to “Metrological compatibility-a key issue in further accreditation” by K. Heydorn

A discussion of proficiency testing (PT) topics started by Heydorn (Accred Qual Assur 15:643–645, 2010) is continued in the present paper. The role of PT in the accreditation of testing/analytical laboratories, the use of consensus values (average or weighted average, median, observed standard deviation, etc.) and a metrological background of PT schemes are discussed. It is shown that metrological traceability, comparability, and compatibility, as well as commutability of a reference material, are the key issues of any PT scheme that applies certified reference material as test items. Metrological compatibility of PT results in such schemes is a property demonstrating the closeness of the PT results to the certified value in comparison with the measurement uncertainty of their difference. The metrological background is especially important for the selection and use of PT schemes for a limited number of participants (fewer than 30) as detailed in IUPAC/CITAC Guide on the topic published in 2010 in Pure Appl Chem 82(5):1099–1135.     

External quality assurance programmes in medical laboratories

 Medical laboratories have a long tradition of external quality assessment. Starting from pure quality control of laboratory performances, most schemes have evolved to a powerful tool for improving quality of clinical outcome of results. External quality assurance in medical laboratories not only includes laboratory performance evaluation, but also evaluation of method performance, post-marked vigilance, training and help. In the future, the quality of programmes must further be improved by accreditation of schemes and by using electronic data interchange. Received: 9 December 2000 Accepted: 14 December 2000     

Quality specifications for evaluation and comparison of performance among external quality assessment schemes in occupational and environmental laboratory medicine

Quality specifications (QS) are proposed for lead in blood and for aluminium, copper, selenium and zinc in serum as part of the aim to set standards of performance for laboratories so that results can be demonstrated to be fit for the purpose to which they are applied. The QS were established taking account of the analytical state-of-the-art, physiological variations in the concentrations of the analyte and the clinical purpose for which the assay is to be used. A procedure was devised that uses these QS to give equivalence of assessment among external quality assessment schemes (EQAS), thus avoiding conflicting information which has been demonstrated in the past. Advantages of this procedure are: to provide direct comparison of performance of laboratories taking part in different schemes, to provide equivalence of assessment of laboratory performance necessary to establish mutual recognition agreements, and to demonstrate the fitness for purpose of results from participants.Presented at the Eurachem PT Workshop September 2005, Portorož, Slovenia     

Performance evaluation for proficiency testing with a limited number of participants

Proficiency testing (PT) is an essential tool for laboratories to assess their competency. Also, participation in PT has become one of the mandatory requirements for laboratory to seek accreditation according to ISO/IEC 17025. For this reason, the effectiveness of performance evaluation by PT scheme is of great concern for the participants and for accreditation bodies as well. In practice, owing to unavailability of other appropriate alternatives, PT scheme providers may have to choose using consensus values to evaluate the performance of participants. However, such consensus values approach was not recommended by relevant international guidelines for PT schemes with limited number of participants. With the use of Monte Carlo simulation technique, this study attempted to investigate the effectiveness of using consensus values for performance evaluation in PT schemes with limited number of participants. The simulation process was schemed according to the statistical model provided by ISO 5725-1 for laboratory measurement results, which covered components like method bias, laboratory bias, and measurement precision. The effectiveness of the consensus value approach was expressed as the percentage of participants in a simulation run could get the same evaluation result, either satisfactory or unsatisfactory, against the “true value.” The findings indicated that the number of participants, choice of consensus values, mass fraction of analyte, method bias, laboratory bias, and measurement repeatability of participating laboratories would all affect the effectiveness of the consensus value approach but at different extent. However, under certain circumstances, use of consensus value could still be considered as an acceptable approach for performance evaluation even the number of participants was limited. Some of the findings were further verified using real data from PT schemes where appropriate certified reference materials or reliable reference values were available.     

Accreditation of proficiency-testing schemes in The Netherlands

Proficiency testing by laboratories, national accreditation bodies, and other third parties is becoming more and more considered as a standard and integral part of the quality control system. Therefore, it is of utmost importance that the quality of the provided proficiency-testing (PT) service is outstanding. If PT-schemes are set up in order to help laboratories monitor and improve their quality, PT-schemes need not only be of high quality themselves, but the organizer also needs to be able to demonstrate this. In The Netherlands formal accreditation of the organization of proficiency testing schemes is used as a tool to guarantee high quality schemes and also to enable organizers to demonstrate their competence. Since 1996, the Dutch Council for Accreditation (RvA) has used the ISO-Guide 43–1 to assess PT-organizers in The Netherlands. From January 2000, the ISO-guide 43–1 was replaced by the ILAC G13 document for assessing organizers. Up till now, four institutes have been accredited by the RvA for the organization of PT-schemes.     

Comparability of PT schemes – what did we learn from COEPT?

The use of proficiency testing schemes (PTS) by laboratories as an integral part of their quality system has been increasing in recent years. Accreditation bodies, regulators and the laboratories’ customers are increasingly using results from PTS in their relationship with laboratories. There are many PTS available in Europe in analytical chemistry; EPTIS indicates over 400.The comparability of these PTS is now a real issue, as many organisers of PTS move into new markets. The COEPT project has systematically demonstrated (in four technical sectors – water, soil, food and occupational hygiene), that there are many similarities between PTS in each sector. For example, nearly all use the z-score as a performance index. One significant difference between many PTS is the value used for the term s in the z-score equation, and this gives a range of evaluations for the same data point. Despite this, the agreement between PTS in the same sector for the evaluation of data is approximately 85%. COEPT has given us a basis for establishing the comparability of PTS and showing us where further harmonisation could occur.Presented at the Eurachem PT Workshop September 2005, Portorož, Slovenia.     

The correlation of laboratory performance in proficiency testing with other QA characteristics

Although it seems self-evident that proficiency testing (PT) and accreditation can be expected to improve quality, their relative benefits remain uncertain as does their efficacy. The study reported here examines the following issues: (a) Why do laboratories take part in PT schemes? (b) How does participation in PT fit in with a laboratory's overall quality assurance (QA) system? (c) Is there a link between a laboratory's performance in specific PT and it's QA system? (d) How does PT performance change with time and how do laboratories respond to poor performance? The overall conclusion is that there is no evidence from the present study that laboratories with third-party assessment (accreditation and certification) perform any better in PT than laboratories without. The validity of this conclusion and its significance for the future design and operation of such schemes requires further investigation. In particular, study is required of the degree to which good performance in open PT correlates with blind PT performance, where laboratories are not aware that the samples being analysed are part of a quality assessment exercise.     

Accreditation of PT providers – is it worth the money?

This topic is being presented from two perspectives, namely the views of an accreditation body and those of an accredited provider of external quality assessment schemes. The first perspective, from an accreditation body, is based on the experience of Australia's national laboratory accreditation body (NATA), which has accredited fourteen proficiency testing (PT) providers and is currently processing an application from another. The second perspective is based on the experience of EQUALIS, the national quality assurance organisation for laboratory medicine in Sweden, which was granted accreditation in 2002.Presented at 5^th Eurachem Workshop on Proficiency testing in analytical chemistry, microbiology and laboratory medicine, Portorož (Slovenia) 25–27 Sept 2005     

Quality assurance in the view of a commercial analytical laboratory

 The necessity for analytical quality assurance is primarily a feature of the analytical process itself. With the full establishment of the EU domestic market, it is also becoming a legal necessity for an increasing number of analytical laboratories. The requirements which laboratories will need to fulfil are stipulated in DIN EN 45 001. Accredited testing laboratories must in fact provide evidence that they work solely in accordance with this standard. National and EU commissions, which are legislative authorities, tend therefore to specify analytical methods, e.g. in the form of regulations or appendices thereto, intended to ensure that results from different laboratories will be comparable and hence will stand up in a court of law. The analytical quality assurance system (AQS), introduced by the Baden-Württemberg Ministry for the Environment in 1984, obliges laboratories to regularly participate in collaborative studies and thereby demonstrate their ability to provide suitably accurate analyses. This alone, however, does not sufficiently demonstrate the competence of a laboratory. Only personal appraisal of the laboratory by an auditor, together with the successful analysis of a sample provided by the same and performed under his observation, can provide proof of the competence of the laboratory. From an analytical point of view, the competence of a laboratory must be regarded as the decisive factor. Competence can only be attained through analytical quality assurance, which thus must be demanded of all laboratories. Received: 4 October 1996 Accepted: 15 January 1997     

The effects of quality-management systems on external quality-assessment performance in Finnish clinical chemistry laboratories

The effects of the implementation of quality-management systems and technological changes on the performance of 23 Finnish clinical chemistry laboratories were studied in external quality-assessment (EQA) schemes organized by Labquality Ltd., Finland, during 1993 to 1999. The investigated serum analytes were sodium, calcium, glucose, cholesterol, and alanine aminotransferase. According to the results, the improvements in analytical quality due to the quality programs were rather small. The effects to the proportions of satisfactory results in terms of precision and deviation from the target value were 5–15% (95% confidence intervals for the proportions ±1–5%). At the same time the laboratories were implementing quality-management systems new technology and characteristics of quality-control materials affected the EQA performance. Both improvements and deteriorations in analytical quality due to these factors were identified. The effects and their total outcomes differed substantially between the five analytes.     

Do we need to accredit proficiency testing schemes?

 Proficiency testing (PT) is being increasingly used as an important quality assurance tool for laboratories. The subject of quality of the providers of PT schemes has been discussed increasingly in recent years. Some countries have implemented systems for the accreditation of PT schemes. This paper looks at the background to the accreditation of PT schemes, the likely mechanisms which could be employed for accreditation, and some of the practical aspects.     

Comparison of different statistical methods for evaluation of proficiency test data

In analytical chemistry, proficiency testing usually consists in tests that laboratories conduct under routine conditions and report the result to the PT provider who then converts the result to a score which helps the participant to assess the accuracy of the result. The aim of this work is to show PT providers, accreditations bodies, and participating laboratories that different scoring results can be achieved depending on the evaluation system selected. The influence of different evaluation techniques on the results of an interlaboratory comparison for determination of gold in precious metals alloys was investigated. Results from 19 participating laboratories were evaluated by means of the three procedures: (1) classical statistical approach—outliers detection; (2) robust methods—(2A) robust procedure and (2B) ISO 13528; and (3) fitness for purpose. Evaluation of the same PT data revealed very interesting issues depending on the different scoring systems that were used and the robustness of the statistical methods used for detecting outliers. As a general rule, laboratories with scoring Z > 2 offered clearly poorer performance in robust approaches than classical ones. In order to support this first evidence, we evaluated a second data set with results from 24 laboratories (mercury from soil samples) by means of the four mentioned approaches. Selection and comparison of different scoring systems must be done very carefully, because sometimes they are not the best approach for studying the data population or the more appropriate one for evaluating the distribution of the data. Finally it should be taken into account that sometimes the robust scoring systems are not always suitable for evaluating the results of some PT schemes.     

Proficiency testing schemes for therapeutics and toxicology

 This report addresses the proposition: current proficiency testing (PT) schemes for therapeutics and toxicology can easily engender complacency. The proposition was based on the premise that there are substantial between-laboratory differences and objectives in the supply of services for the measurement of drugs, as a result of which it is difficult to design PT schemes which adequately cover the requirements of all laboratories. Thus, acceptable performance in a PT scheme may lead participants to consider that they have tested their procedures more rigorously than is the case. Four areas, validity of the PT scheme, PT material, frequency of testing and acceptance criteria are examined. Received: 15 April 2000 · Accepted: 15 April 2000     

Estimates of uncertainty of measurement from proficiency testing data: a case study

The results obtained by a laboratory over a number of proficiency testing/external quality assessment schemes (PT/EQAS) rounds can give information on the uncertainty of its measurements for a given test, provided that conditions such as full coverage of the routine analytical range, traceability, and small uncertainty of the assigned values (compared to the spread of the results) are met and provided that systematic deviations and any other sources of uncertainty are considered. As organisers of the Italian EQAS (ITEQAS) in occupational and environmental laboratory medicine, we tested this hypothesis using as model data from well-performing laboratories taking part in ITEQAS for lead in blood over the last 2 years. We also investigated how different PT/EQAS features (frequency of trials and number of samples) would affect a laboratory estimate of its uncertainty. Such information can be helpful in improving PT/EQAS organisation and define, for a given test: (a) the state of the art of the uncertainty of current measurement procedures, (b) identify needs for improvement of analytical methodologies and (c) set targets for acceptable uncertainty values.Presented at the Eurachem PT Workshop September 2005, Portorož, Slovenia.Papers published in this section do not necessarily reflect the opinion of the Editors, the Editorial Board and the Publisher.     

External quality assessment schemes for clinical laboratories

 The clinical routine laboratory generally utilizes cheap, easy and rapid measurement procedures ("methods") in order to meet the requirement for the production of many analytical results (500–3000 per day) of hundreds of different types. The measurement procedures are optimized for the analysis of native patients' samples, but are frequently sensitive to deviation of the composition of the matrix from that in normal fresh samples. The inherent lack of stability of patient samples means that control samples need to be stabilized. The method of stabilization is critical. Furthermore, the method of "spiking" samples with pathological material is a matter of concern. Generally, minimally processed patients' samples should be used in external quality assessment (EQA) schemes. Consensus values are currently the most popular for use as a guide to the best results from participating laboratories in EQA schemes; these often work fairly well. However, the uncertainty and traceability of this type of value is unknown, and in some cases may even be misleading, tending to preserve bad routine methods when these are dominant in the participating laboratories. Reference measurement procedure (RMP) values are recommended to provide scientifically based information, to facilitate the proper choice of methods in the routine laboratories, and to validate the suitability of control materials in EQA schemes. The present paper provides selected examples from a study comparing consensus values with RMP values on lyophilized sera, and also presents results on a fresh frozen thawed serum for the study of commutability. Received: 8 November 1995 Accepted: 8 May 1996     

Different approaches to legal requirements on validation of test methods for official control of foodstuffs and of veterinary drug residues and element contaminants in the EC

 In order to ensure food consumer protection as well as to avoid barriers to trade and unnecessary duplications of laboratory tests and to gain mutual recognition of results of analyses, the quality of laboratories and test results has to be guaranteed. For this purpose, the EC Council and the Commission have introducedprovisions – on measures for quality assurance for official laboratories concerning the analyses of foodstuffs on the one hand and animals and fresh meat on the other, – on the validation of test methods to obtain results of sufficient accuracy. This article deals with legal requirements in the European Union on basic principles of laboratory quality assurance for official notification to the EC Commission and on method validation concerning official laboratories. Widespread discussions and activities on measurement uncertainty are in progress, and the European validation standards for official purposes may serve as a basis for world-wide efforts on quality harmonization of analytical results. Although much time has already been spent, definitions and requirements have to be revised and further additions have to be made.     

Comparability and compatibility of proficiency testing results in schemes with a limited number of participants

Comparability and compatibility of proficiency testing (PT) results are discussed for schemes with a limited number of participants (less than 20–30) based on the use of reference materials (RMs) as test items. Since PT results are a kind of measurement/analysis/test result, their comparability is a property conditioned by traceability to measurement standards applied in the measurement process. At the same time, metrological traceability of the certified value of the RM (sent to PT participants as test items) is also important, since the PT results are compared with the RM certified value. The RM position in the calibration hierarchy of measurement standards sets the degree of comparability for PT results, which can be assessed in the scheme. However, this assessment is influenced by commutability (adequacy or match) of the matrix RM used for PT and routine samples. Compatibility of PT results is a characteristic of the collective (group) performance of the laboratories participating in PT that can be expressed as closeness of the distribution of the PT results to the distribution of the RM data. Achieving quality-of-measurement/analysis/test results in the framework of the concept “tested once, accepted everywhere” requires both comparability and compatibility of the test results.     

Quality issues in proficiency testing

 Proficiency testing is a means of assessing the ability of laboratories to competently perform specific tests and/or measurements. It supplements a laboratory's own internal quality control procedure by providing an additional external audit of their testing capability and provides laboratories with a sound basis for continuous improvement. It is also a means towards achieving comparability of measurement between laboratories. Participation is one of the few ways in which a laboratory can compare its performance with that of other laboratories. Good performance in proficiency testing schemes provides independent evidence and hence reassurance to the laboratory and its clients that its procedures, test methods and other laboratory operations are under control. For test results to have any credibility, they must be traceable to a standard of measurement, preferably in terms of SI units, and must be accompanied by a statement of uncertainty. Analytical chemists are coming to realise that this is just as true in their field as it is for physical measurements, and applies equally to proficiency testing results and laboratory test reports. Recent approaches toward ensuring the quality and comparability of proficiency testing schemes and the means of evaluating proficiency test results are described. These have led to the drafting of guidelines and subsequently to the development of international requirements for the competence of scheme providers. Received: 2 January 1999 · Accepted: 7 April 1999     

The Italian way: Accreditation and continuous quality improvement in clinical laboratories

 The transition from quality assurance of the analytical phase to the quality management of total testing in clinical laboratories is still at an early stage. But it has begun. Accreditation through voluntary, educational and professional schemes, like the Clinical Pathology Accreditation scheme, is a useful tool for following defined standards of practice and having these independently confirmed on the basis of a peer review. Approved clinical laboratories can obtain a hallmark of performance and offer reassurance to users of their services. However, accreditation does not guarantee an error-free service; it is not the final step, but an important stage in the improvement process. Quality is a journey and continuous quality improvement is the paradigm for better addressing our efforts to satisfy customers' expectations for the desired health outcomes related to a high-quality laboratory service.     

Interlinkages and recognition of proficiency-testing schemes

 Because proficiency testing (PT) is increasingly used for the accreditation of testing laboratories and as a tool for backing up existing multilateral recognition arrangements between accreditation bodies, the question of performance and comparability of the proficiency-test providers arises. In this paper different approaches to assess the equivalence of European PT schemes and the competence of their providers are presented. As a first step a workshop is proposed to agree on a pilot study. The final aim is to create confidence in the existing PT schemes in Europe and to use them as common European tools.