Confidence in testing services – new expectations, new rules, new challenges. 5th Eurolab Symposium offered important information and stimulating discussions

The 5th Eurolab Symposium on 4th and 5th October in Edinburgh was the highlight event of Eurolab in 2000, Eurolab’s 10th anniversary year. Under the title: ”Confidence in Testing Services – new expectations, new rules, new challenges” it reflected recent changes and developments in the laboratory scene and focussed on new tasks and challenges. The most essential changes for laboratories are connected with the new standard ISO/IEC 17025 ”General requirements for the competence of testing and calibration laboratories”. Keywords in this respect are uncertainty of measurement and traceability to SI units, requirements on personnel, proficiency testing, customer orientation, and management systems.     

Proficiency tests using four batches of green alga with controlled levels of cadmium

A set of test materials of the green alga Chlorella vulgaris with different levels of naturally bound Cd and about the same levels of other 13 essential or trace elements (BIOMA 1–4) was produced for proficiency testing of laboratories involved in elemental analysis of food. Criteria of the laboratories’ performance, such as the z-score and combination scores RSZ and SSZ were evaluated and discussed in terms of their suitability for proficiency testing using a set of test materials with the same matrix. The use of the test materials as a set of internal reference materials has also been suggested. Received: 23 May 1997 / Revised: 1 August 1997 / Accepted: 11 August 1997     

Approaching target uncertainty in proficiency testing schemes: experience in the field of water measurement

Good correspondence between evaluations of the proficiency testing data for water measurements according to different scoring schemes was found as a result of implementation of the measurement quality according to the target uncertainty defined in the Drinking Water Directive. The accuracy of the evaluation is determined by the uncertainty of the assigned value, and this uncertainty should be in correspondence with the target uncertainty. Modification of the zeta-score by the thorough application of the target uncertainty is presented. It is convenient for generalised presentation of the proficiency testing data. The allowed bias is included as a linear term on the measured value scale, as it is treated in uncertainty analysis. The importance of implementation of the target measurement uncertainty in other fields of the routine measurements is indicated.     

Evaluation of the use of consensus values in proficiency testing programmes

Proficiency testing (PT) is an essential tool used by laboratory accreditation bodies to assess the competency of laboratories. Because of limited resources of PT providers or for other reasons, the assigned reference value used in the calculation of z-score values has usually been derived from some sort of consensus value obtained by central tendency estimators such as the arithmetic mean or robust mean. However, if the assigned reference value deviates significantly from the ‘true value’ of the analyte in the test material, laboratories’ performance will be evaluated incorrectly. This paper evaluates the use of consensus values in proficiency testing programmes using the Monte Carlo simulation technique. The results indicated that the deviation of the assigned value from the true value could be as large as 40%, depending on the parameters of the proficiency testing programmes under investigation such as sample homogeneity, number of participant laboratories, concentration level, method precision and laboratory bias. To study how these parameters affect the degree of discrepancy between the consensus value and the true value, a fractional factorial design was also applied. The findings indicate that the number of participating laboratories and the distribution of laboratory bias were the prime two factors affecting the deviation of the consensus value from the true value.     

Requirements vs acceptability in proficiency testing schemes and other interlaboratory comparisons

Many laboratories take part in proficiency testing schemes, external quality assessment programmes and other interlaboratory comparisons. These have many similarities but also important differences in their modus operandi and evaluation of performance of participating laboratories. This paper attempts to highlight both the similarities and differences. It also puts particular emphasis on requirements called ”target values for uncertainty” and their meaning. Received: 24 January 2001 Accepted: 25 January 2001     

ISO/IEC 17043: the new International Standard for proficiency testing

The revision of ISO/IEC Guide 43 is well underway by CASCO Working Group 28, with the author as Convener. The ballot on the committee draft was successfully completed in June 2008. The Working Group met in September to resolve the CD comments and to recommend that the document should advance as an ISO/IEC DIS. The document is in harmony with ISO/IEC 17025 (2005), ILAC G13 (2007), and the IUPAC Harmonized Protocol (2006) for proficiency testing. Major changes from Guide 43 include considerations for inspection bodies, requirements for metrological traceability of assigned values for calibration schemes, limitations on subcontracting, requirements for reporting the uncertainty of assigned values, and for including method information and technical commentary in scheme reports. Minor changes include requirements for equipment used in the manufacture and testing of proficiency test items, considerations for handling laboratories’ reported uncertainties, and the issuance of certificates of participation. These changes have not generated any substantial objections from the CASCO membership and liaisons.     

Aspects of proficiency testing studies of trace elements in environmental samples with a focus on laboratory performance

Four proficiency tests of environmentally significant elements in soil and water are described. The effects of sample matrix, analyte, sample preparation procedure, extraction method efficiency and instrumental technique on laboratory performance were assessed. Participants’ performance was evaluated using z scores and E _n scores. One study included dried sediment and sediment digest as test samples. By testing both dried sediment and sediment digest in the same study, differences in laboratories’ results due to the extraction method or instrumental technique were evaluated. Cadmium and nickel were the elements with the most unsatisfactory z scores for dried sediment compared to sediment digest. The second study had as test samples river water and river water fortified with biosolid. For an evaluation of the efficiency of participants’ extraction method, the results reported for the analysis of river water were compared to the results for analysis of the same river water fortified with biosolid. No consensus could be found between participants’ results for Cr and Ni in this study as the extraction method employed by them produced results that were too variable. The last two studies included saline and fresh water samples. The analysis of fresh and saline water samples with similar analyte concentrations assisted laboratories in developing their methods for trace elements in saline water. For Cr, Pb and Se, the assigned values were reference values traceable to SI, measured using isotope dilution inductively coupled plasma mass spectrometry. Saline water containing low concentrations of analytes presented difficulties to some laboratories. Arsenic and selenium were the analytes most difficult to analyse in all types of environmental samples.     

Performance of uncertainty evaluation strategies in a food proficiency scheme

A study of the performance of different uncertainty evaluation strategies among 163 voluntary respondents from food proficiency schemes is presented. Strategies included use of: single-laboratory validation data, quality control data, past proficiency testing data, reproducibility data, a measurement equation and the dispersion of replicate observations on the test material. Most performed reasonably well, but the dispersion of replicate observations underestimated uncertainty by a factor of approximately 3. Intended compliance with accreditation requirements was associated with significantly improved uncertainty evaluation performance, while intended compliance with the ISO “Guide to the expression of uncertainty in measurement” had no significant effect. Substituting estimates based on the Horwitz or Horwitz–Thompson models or on PT target standard deviation for the respondents’ own estimates of uncertainty led to a marked reduction in poor zeta scores and significant improvement in dispersion of zeta scores.     

Comparability of PT schemes – what did we learn from COEPT?

The use of proficiency testing schemes (PTS) by laboratories as an integral part of their quality system has been increasing in recent years. Accreditation bodies, regulators and the laboratories’ customers are increasingly using results from PTS in their relationship with laboratories. There are many PTS available in Europe in analytical chemistry; EPTIS indicates over 400.The comparability of these PTS is now a real issue, as many organisers of PTS move into new markets. The COEPT project has systematically demonstrated (in four technical sectors – water, soil, food and occupational hygiene), that there are many similarities between PTS in each sector. For example, nearly all use the z-score as a performance index. One significant difference between many PTS is the value used for the term s in the z-score equation, and this gives a range of evaluations for the same data point. Despite this, the agreement between PTS in the same sector for the evaluation of data is approximately 85%. COEPT has given us a basis for establishing the comparability of PTS and showing us where further harmonisation could occur.Presented at the Eurachem PT Workshop September 2005, Portorož, Slovenia.     

The third edition of ISO Guide 34: what were the drivers for the revision and what is new?

After the International Laboratory Accreditation Cooperation (ILAC) had taken in 2004, the resolution to conduct accreditation of producers of reference materials according to ISO Guide 34 ‘General requirements for the competence of reference material producers’ in combination with ISO/IEC 17025 ‘General requirements for the competence of testing and calibration laboratories’, ISO/REMCO, the ISO Committee on Reference Materials, decided in 2005 to revise ISO Guide 34 to align it closer with ISO/IEC 17025 and to clarify certain issues for accreditors and producers seeking accreditation without adding new requirements. Moreover, the publication in 2007 of ISO/IEC Guide 99 ‘International vocabulary of metrology—Basic and general concepts and associated terms (VIM)’ triggered additional adaptations of the guide.     

Evaluation of proficiency testing results taking into account their uncertainties

A new composite score for the evaluation of performance of proficiency testing participants is proposed. The score is based on a combination of the z-score, uncertainty of a participant’s measurement result and uncertainty of the proficiency testing scheme’s assigned value. The use of such a composite score will allow evaluation not only of the participant’s ability to determine an analyte in corresponding matrix, but also their understanding of the uncertainty in the obtained analytical result. The score may be helpful for the laboratory’s quality system and for laboratory accreditation according to ISO 17025.     

Regulatory compliance vs. NEXUS quality for laboratory tests

In the U.S., all clinical laboratory testing is regulated by the Clinical Laboratory Improvement Amendments (CLIA) (). The CLIA link test quality and adherence to a body of testing regulations intended to ensure accurate, reliable, and timely patient test results. The goal of the CLIA legislation was to ensure a minimum, fundamental level of quality. In the context of “NEXUS,” quality must “go beyond getting the ‘right’ answer on the ‘right’ patient that can be interpreted against ‘right’ reference values. CLIA regulations with specific minimum, performance requirements, or safeguards, are designed to prevent testing errors. The US Institute of Medicine found that testing processes fail as a result of human error, lack of documentation, and lack of test management. In the latest (2004) interpretations of CLIA regulations, the minimum quality control requirement continues to be analyzing at least two external, liquid quality control materials per test per day. In 1995, we proposed that the responsibility for achieving quality test results shifts from the sole purview of the laboratory director to an “alliance” of laboratory professionals, manufacturers, and regulators. The EQC (equivalent quality control) concept as proposed is a positive step in achieving this alliance. With the obvious lack of scientific and statistical robustness, EQC falls far short of ensuring quality. Achieving the “NEXUS Vision” for quality laboratory testing will not come solely from laboratory professionals. The NEXUS is about how to ensure the full-quality assessment of the testing process – pre-analytical, analytical, and post-analytical.Presented at the 10th Conference Quality in the Spotlight, March 2005, Antwerp, Belgium.     

Visualisation of proficiency test exercise results in Kiri plots

A novel graphical method (‘Kiri plots’) for the presentation of proficiency test exercise results is presented. The Kiri plot visualises the evaluation of the proficiency test results based on three statistical tests (the z score, the zeta score and the relative uncertainty outlier test) by defining six zones including a central “in agreement” zone.     

Economy of proficiency testing: reference versus consensus values

Many proficiency tests are operated with a consensus value derived from the participants’ results. Apart from technical issues, one of the reasons often mentioned is that proficiency tests operated with consensus values would be cheaper than those using reference values obtained from a priori characterisation measurements. The economy of a proficiency test must of course be balanced by the need of the participants, and the quality of the comparison in general. The proficiency tests selected in this study had both a reference value and a consensus value, one of which was used for assessing the performance of the participating laboratories. In this work, both a technical and an economical assessment of how the comparisons were operated is made. From the evaluation, it follows that usually the use of consensus values does not necessarily reduce the costs of a proficiency test. However, frequently it may be observed that the quality of the assessment of the laboratories is better with a reference value. Received: 11 October 2000 Accepted: 3 January 2001     

Standardisation of a European measurement method for the determination of mercury in deposition: results of the field trial campaign and determination of a measurement uncertainty and working range

A standard method for the measurement of mercury in deposition is currently being finalised by Working Group 25 of the European Committee for Standardisation’s Technical Committee 264 ‘Air Quality’, in response to the requirements of the European Union’s Fourth Air Quality Daughter Directive. This paper reports the results of a field measurement programme which was undertaken to assess the uncertainty of the proposed standard method, define its working range and determine its compliance with the required data quality objectives of the Fourth Air Quality Daughter Directive.     

An interlaboratory comparison on whole water samples

The European Water Framework Directive 2000/60/EC requires monitoring of organic priority pollutants in so-called whole water samples, i.e. in aqueous non-filtered samples that contain natural colloidal and suspended particulate matter. Colloids and suspended particles in the liquid phase constitute a challenge for sample homogeneity and stability. Within the joint research project ENV08 “Traceable measurements for monitoring critical pollutants under the European Water Framework Directive 2000/60/EC”, whole water test materials were developed by spiking defined amounts of aqueous slurries of ultra-finely milled contaminated soil or sediment and aqueous solutions of humic acid into a natural mineral water matrix. This paper presents the results of an European-wide interlaboratory comparison (ILC) using this type of test materials. Target analytes were tributyltin, polybrominated diphenyl ethers and polycyclic aromatic hydrocarbons in the ng/L concentration range. Results of the ILC indicate that the produced materials are sufficiently homogeneous and stable to serve as samples for, e.g. proficiency testing or method validation. To our knowledge, this is the first time that ready-to-use water materials with a defined amount of suspended particulate and colloidal matter have been applied as test samples in an interlaboratory exercise. These samples meet the requirements of the European Water Framework Directive. Previous proficiency testing schemes mainly employed filtered water samples fortified with a spike of the target analyte in a water-miscible organic solvent.     

Evaluation of laboratory performance in IMEP water interlaboratory comparisons

The aim of International Evaluation Programme (IMEP) is to present objectively the quality of chemical measurements. Participants in IMEP compare their reported measurement results with independent external certified reference values with demonstrated traceability and uncertainty, as evaluated according to international guidelines. Three major interlaboratory comparisons (ILCs), IMEP-6, IMEP-9 and IMEP-12, on trace elements in water were carried out from 1994 to 2000. Participants' results for Cd, Fe and Pb concentrations from these three different IMEP water ILCs were compared by means of suitable performance indicators. The performance evaluation criteria were set according to the requirements stated in the Council Directive 98/83/EC of 3 November 1998 on the quality of water intended for human consumption. Two different performance indicators were selected for evaluation of the individual participants' results: the commonly used z-score and the not so well-known E _n number. The group performance indicator is based on the variation of z-scores. To assess the individual measurement performance, not only the deviation of the reported measurement values from the certified reference value, but also criteria for maximum and minimum acceptable uncertainties were taken into account. The participants' performance is also reviewed by means of using a simple graphical tool, called “Naji-plots”.     

Laboratory assessment by combined z score values in proficiency tests: experience gained through the European Union proficiency tests for pesticide residues in fruits and vegetables

The obligation for accredited laboratories to participate in proficiency tests under ISO 17025, performing multiresidue methods (MRMs) for pesticide residues, involves the reporting of a large number of individual z scores making the evaluation of the overall performance of the laboratories difficult. It entails, time and again, the need for ways to summarise the laboratory’s overall assessment into a unique combined index. In addition, the need for ways to continually evaluate the performance of the laboratory over the years is equally acknowledged. For these reasons, following 14 years of the European Union Reference Laboratory for Pesticide Residues in Fruits and Vegetables (EUPT-FV), useful formulas have been designed to globally evaluate the assessment of the participating laboratories. The aim is to achieve a formula which is easy to understand, which can be applied and which fits the purposes of long-term evaluation detecting positive and negative trends. Moreover, consideration is needed for a fair compensation of bad results in MRM, taking into account the large number of compounds that are covered. It is therefore important to be aware of the difficulties in getting satisfactory values from a wide range of compounds. This work presents an evaluation of the main well-established combined z score formulas together with those new ones developed here which have been applied to the European proficiency test results (EUPTs) over the years. Previous formulas such as the rescaled sum of z score (RSZ), the sum squared of z score (SSZ) and the relative laboratory performance (RLP) are compared with the newer ones: the sum of weighted z scores (SWZ) and the sum of squared z scores (SZ2). By means of formula comparisons, conclusions on the advantages, drawbacks and the most fit-for-purpose approach are achieved.     

Quality issues in proficiency testing

 Proficiency testing is a means of assessing the ability of laboratories to competently perform specific tests and/or measurements. It supplements a laboratory's own internal quality control procedure by providing an additional external audit of their testing capability and provides laboratories with a sound basis for continuous improvement. It is also a means towards achieving comparability of measurement between laboratories. Participation is one of the few ways in which a laboratory can compare its performance with that of other laboratories. Good performance in proficiency testing schemes provides independent evidence and hence reassurance to the laboratory and its clients that its procedures, test methods and other laboratory operations are under control. For test results to have any credibility, they must be traceable to a standard of measurement, preferably in terms of SI units, and must be accompanied by a statement of uncertainty. Analytical chemists are coming to realise that this is just as true in their field as it is for physical measurements, and applies equally to proficiency testing results and laboratory test reports. Recent approaches toward ensuring the quality and comparability of proficiency testing schemes and the means of evaluating proficiency test results are described. These have led to the drafting of guidelines and subsequently to the development of international requirements for the competence of scheme providers. Received: 2 January 1999 · Accepted: 7 April 1999     

The challenges and benefits of implementing the requirements of ISO/IEC 17043 by PT/EQA providers

ISO/IEC 17043 Conformity assessment??general requirements for proficiency testing is intended to replace previous international guides used to assess the competence of proficiency testing (PT) providers. It expands on the requirements of previous guides and is intended to accommodate PT providers of calibration laboratories and testing laboratories handling both qualitative and quantitative data. QMP-LS is an office-based external quality assessment provider for medical laboratories in Ontario, Canada and operates 46 different PT schemes for approximately 250 diagnostic tests. In 2010, these schemes were accredited to ISO/IEC 17043. Schemes included tests from the following disciplines: chemistry, hematology, microbiology, transfusion medicine, cytology, histology and genetics. Thirty of the schemes were qualitative. The challenges and benefits of implementing ISO/IEC 17043 are discussed, with particular emphasis on clauses addressing the following requirements: statistical design, determination of assigned value, homogeneity and stability testing, packaging, labeling and distribution, performance evaluation and subcontracting services.