Proficiency tests for laboratories: a systematic review

Laboratories analytical results must be reliable. Proficiency tests (PT) main objective is to provide independent demonstrations of laboratory competence. The demand for these activities is increasing on the world scenario, as well as its importance. The main objective of this study is to identify the important publications of the PT theme from 2005 to 2012 based on a systematic review procedure. The method proposed reached a total of 113 papers published in indexed journals and another 34 additional references, including standards, guidelines and recommendations of international or regional accreditation body cooperation, international standards organization and international metrological institutes. All selected references were clustered based on its approach, and then the main practices were presented. The approaches identified were related to performance assessment, calculation method for performance assessment, use of PT for validation and/or estimation of measurement uncertainty and management of PT. Results indicate some opportunities to develop researches, such as: project management related to PT, the importance of analyses of data probability distribution function when consensus value is used, criteria to select the parameter to homogeneity and stability tests and to explore the link between PT, method validation and measurement uncertainty, among others.     

Estimation of measurement uncertainty in food microbiology: The ISO approach

Given the current interest in measurement uncertainty (MU) in food microbiology, in particular for laboratory accreditation purposes, and the need to have harmonized reference documents specifically in that area at the international level, ISO is conducting works to meet this need. An ISO Technical Specification (ISO/TS 19036) is being prepared on MU estimation for quantitative determinations. A global approach has been chosen, based on the reproducibility standard deviation of the final result of the measurement process. Three possibilities are envisaged for the estimation of the reproducibility standard deviation, in a decreasing order of preference: The intra-laboratory standard deviation, the inter-laboratory standard deviation derived from method validation, and the inter-laboratory standard deviation derived from proficiency testing.The uncertainty of qualitative determinations is still under investigation, and will be covered by a separate ISO publication. Presented at AOAC Europe/Eurachem Symposium March 2005, Brussels, Belgium     

Performance evaluation for proficiency testing with a limited number of participants

Proficiency testing (PT) is an essential tool for laboratories to assess their competency. Also, participation in PT has become one of the mandatory requirements for laboratory to seek accreditation according to ISO/IEC 17025. For this reason, the effectiveness of performance evaluation by PT scheme is of great concern for the participants and for accreditation bodies as well. In practice, owing to unavailability of other appropriate alternatives, PT scheme providers may have to choose using consensus values to evaluate the performance of participants. However, such consensus values approach was not recommended by relevant international guidelines for PT schemes with limited number of participants. With the use of Monte Carlo simulation technique, this study attempted to investigate the effectiveness of using consensus values for performance evaluation in PT schemes with limited number of participants. The simulation process was schemed according to the statistical model provided by ISO 5725-1 for laboratory measurement results, which covered components like method bias, laboratory bias, and measurement precision. The effectiveness of the consensus value approach was expressed as the percentage of participants in a simulation run could get the same evaluation result, either satisfactory or unsatisfactory, against the “true value.” The findings indicated that the number of participants, choice of consensus values, mass fraction of analyte, method bias, laboratory bias, and measurement repeatability of participating laboratories would all affect the effectiveness of the consensus value approach but at different extent. However, under certain circumstances, use of consensus value could still be considered as an acceptable approach for performance evaluation even the number of participants was limited. Some of the findings were further verified using real data from PT schemes where appropriate certified reference materials or reliable reference values were available.     

Quantifying the measurement uncertainty of results from environmental analytical methods

The Eurachem-CITAC Guide Quantifying Uncertainty in Analytical Measurement was put into practice in a public laboratory devoted to environmental analytical measurements. In doing so due regard was given to the provisions of ISO 17025 and an attempt was made to base the entire estimation of measurement uncertainty on available data from the literature or from previously performed validation studies. Most environmental analytical procedures laid down in national or international standards are the result of cooperative efforts and put into effect as part of a compromise between all parties involved, public and private, that also encompasses environmental standards and statutory limits. Central to many procedures is the focus on the measurement of environmental effects rather than on individual chemical species. In this situation it is particularly important to understand the measurement process well enough to produce a realistic uncertainty statement. Environmental analytical methods will be examined as far as necessary, but reference will also be made to analytical methods in general and to physical measurement methods where appropriate. This paper describes ways and means of quantifying uncertainty for frequently practised methods of environmental analysis. It will be shown that operationally defined measurands are no obstacle to the estimation process as described in the Eurachem/CITAC Guide if it is accepted that the dominating component of uncertainty comes from the actual practice of the method as a reproducibility standard deviation.     

Proficiency testing in analytical chemistry, microbiology, and laboratory medicine – working group discussions on current status, problems, and future directions

Working group (WG) discussions on proficiency testing (PT) held at the joint Eurachem/ EQALM workshop, Borås, Sweden, 24–26 September 2000 are summarized. The discussions focused on aspects of PT and accreditation (WG 1), general aspects of PT in analytical chemistry (WG 2), microbiology (WG 3), and laboratory medicine (WG 4), incorporation of measurement uncertainty into PT schemes (WG 5), international harmonization of PT schemes (WG 6), and the role of PT in the international structure of chemical measurement (WG 7). Current status, problems and future directions are identified. Each WG contained a majority of participants experienced in the subject being covered by that WG, and a few participants with different expertise. This was done to promote cross-fertilization of ideas between sectors, a key objective of the workshop. The WG issues reflected the content of the keynote lectures and some issues were covered from different perspectives by more than one group.     

Quantifying the measurement uncertainty of results from environmental analytical methods

The Eurachem–CITAC Guide Quantifying Uncertainty in Analytical Measurement was put into practice in a public laboratory devoted to environmental analytical measurements. In doing so due regard was given to the provisions of ISO 17025 and an attempt was made to base the entire estimation of measurement uncertainty on available data from the literature or from previously performed validation studies. Most environmental analytical procedures laid down in national or international standards are the result of cooperative efforts and put into effect as part of a compromise between all parties involved, public and private, that also encompasses environmental standards and statutory limits. Central to many procedures is the focus on the measurement of environmental effects rather than on individual chemical species. In this situation it is particularly important to understand the measurement process well enough to produce a realistic uncertainty statement. Environmental analytical methods will be examined as far as necessary, but reference will also be made to analytical methods in general and to physical measurement methods where appropriate. This paper describes ways and means of quantifying uncertainty for frequently practised methods of environmental analysis. It will be shown that operationally defined measurands are no obstacle to the estimation process as described in the Eurachem/CITAC Guide if it is accepted that the dominating component of uncertainty comes from the actual practice of the method as a reproducibility standard deviation.     

Using target measurement uncertainty to determine fitness for purpose

The methods an analytical laboratory uses must be validated to be fit for purpose. The fitness for purpose of a quantitative method used to determine the concentration of a substance when assessing compliance to requirements can be described by the maximum measurement uncertainty. This is called the target measurement uncertainty. Acceptance criteria for precision and bias in the method validation are then established in terms of the target measurement uncertainty. The target measurement uncertainty can be decided by following a process which involves determining the required concentration range of the measurand; determining the acceptable level of risks of incorrect decisions of compliance; developing a suitable decision rule, with guard bands if appropriate; using the probability of making an incorrect decision of compliance based on the decision rule; and assessing the impact of bias. A key participant in this process is the end user of the data, the laboratory customer. This paper presents the concepts concerning target measurement uncertainty introduced in recently published international guidelines to the practicing analytical chemist who is not generally familiar with these concepts. Three examples are used to illustrate the process.     

Performance of uncertainty evaluation strategies in a food proficiency scheme

A study of the performance of different uncertainty evaluation strategies among 163 voluntary respondents from food proficiency schemes is presented. Strategies included use of: single-laboratory validation data, quality control data, past proficiency testing data, reproducibility data, a measurement equation and the dispersion of replicate observations on the test material. Most performed reasonably well, but the dispersion of replicate observations underestimated uncertainty by a factor of approximately 3. Intended compliance with accreditation requirements was associated with significantly improved uncertainty evaluation performance, while intended compliance with the ISO “Guide to the expression of uncertainty in measurement” had no significant effect. Substituting estimates based on the Horwitz or Horwitz–Thompson models or on PT target standard deviation for the respondents’ own estimates of uncertainty led to a marked reduction in poor zeta scores and significant improvement in dispersion of zeta scores.     

Assessment of reproducibility and uncertainty of food microbiology methods: statistical approach of a multi-site laboratory

The validation of food microbiology methods and assessment of measurement uncertainty are required for laboratories operating EN ISO/IEC 17025 accreditation systems. This paper aims to describes the statistical approach adopted by a multi-site laboratory to satisfy these requirements and to routinely check the performance of the methods. A validation protocol was performed, reflecting a great variability of experimental conditions, represented by the period of time during which determinations were made, the different laboratories concerned, the large number of technicians involved, the differences in the level of contamination of the matrix analyzed and the operating conditions (equipment, reagents, culture media, environmental conditions, etc.). Despite the very high variability of the experimental conditions, the values of repeatability and reproducibility obtained for the methods were lower than those stated in the respective regulations. In addition, the top-down approach adopted in this study is to be considered effective overtime and allows to randomize all variable factors, including the effects of sensitivity and specificity, in order to simulate the conditions of real reproducibility. A specific Excel spreadsheet was also developed for the routine expression of analytical results. This spreadsheet represents a very useful tool for operators to calculate and routinely check the results, as well as to assess the uncertainty of measurement. This particular quality assessment system is enabling the multi-site laboratory to implement the accuracy, reliability and comparability of the analytical results and to ensure the analytical control of data output.     

A simplified approach to the estimation of analytical measurement uncertainty

The present study summarizes the measurement uncertainty estimations carried out in Nestlé Research Center since 2002. These estimations cover a wide range of analyses of commercial and regulatory interests. In a first part, this study shows that method validation data (repeatability, trueness and intermediate reproducibility) can be used to provide a good estimation of measurement uncertainty.In a second part, measurement uncertainty is compared to collaborative trials data. These data can be used for measurement uncertainty estimation as far as the in-house validation performances are comparable to the method validation performances obtained in the collaborative trial.Based on these two main observations, the aim of this study is to easily estimate the measurement uncertainty using validation data.     

A comparison in the evaluation of measurement uncertainty in analytical chemistry testing between the use of quality control data and a regression analysis

The evaluation of measurement uncertainties has been widely applied to the calibration of measurement instruments, whereas its application to tests, despite increasing requirements, is a more recent phenomenon. The generalization of the evaluation of measurement uncertainties to tests has been a gradual process, in line with changes in the requirements of the normative framework that regulates the accreditation of tests laboratories and also as the perceived good practices have evolved. The sole identification of the relevant sources of uncertainty was followed by the requirement to provide a simplified estimate of the measurement uncertainty, and it is now an accepted requirement to properly evaluate the expanded measurement uncertainty associated with any tests. In this study, the evaluation of measurement uncertainty associated with the determination of sulfate in water will be attempted using a procedure that includes linear regression, with the regression parameters provided with associated uncertainties, and a Monte Carlo method applied as a validation tool of the conventional mainstream evaluation method, concerning the approximations in terms of linearization of the model and the assumed shape of the output distribution introduced by this approach.     

A validation concept for purity determination and assay in the pharmaceutical industry using measurement uncertainty and statistical process control

 The analytical chemists in process development in the pharmaceutical industry have to solve the difficult problem of producing high quality methods for purity determination and assay within a short time without a clear definition of the substance to be analyzed. Therefore the quality management is very difficult. The ideal situation would be that every method is validated before use. This is not possible because this would delay the development process. A process-type quality development approach with an estimation type fast validation (measurement uncertainty) is therefore suggested. The quality management process consists of the estimation of measurement uncertainty for early project status. Statistical process control (SPC) is started directly after measurement uncertainty estimation and a classical validation for the end of the project. By this approach a process is defined that allows a fast and cost-efficient way of supporting the development process with the appropriate quality at the end of the process and provides the transparency needed in the development process. The procedure presented tries to solve the problem of the parallelism between the two development processes (chemical and analytical development) by speeding up the analytical development process initially. Received: 25 March 1997 · Accepted: 17 May 1997     

Flexibilization of the scope of accreditation: an important asset for food and water microbiology laboratories

Beltest, the Belgian accreditation body, has investigated flexibilization of the scope of accreditation for chemistry laboratories and food and water microbiology laboratories. This flexibilization, synonymous with test-type accreditation, allows a laboratory to add new test methods or retry previous test methods without having to undergo a new audit by Beltest. It has been used for nearly ten years by German and Swiss accreditation bodies. Flexibilization permits the validation of methods and results, given that the competence of the particular laboratory is already well established. This new concept in microbiology allows client’s needs to be adequately met, and facilitates the quick establishment of a method in several laboratories at once in case of a public health crisis. The first laboratory to participate at this investigation on the flexibilization concept, as a test of the concept, was the Belgian reference laboratory for food microbiology.     

Benefits arising from proficiency testing schemes: The comparison of analytical methods

 The primary objective of proficiency testing (PT) is in the provision of information and support to participating laboratories, to enable them to monitor and improve the quality of their measurements. However, other benefits can be obtained from PT. These include the comparison of data for a given measurement by different methods, the validation of new methods, and the provision of information for laboratories' customers and accreditation bodies. This paper considers the subject of method comparison, and highlights some of the approaches which can be followed, as well as the practical use to which this can be put, to benefit the analytical community more widely. This is illustrated by a case study concerning the measurement of haze in beer. In this study the United Kingdom Institute of Brewing (IoB) conducted a survey of participants in the Brewing Analytes Proficiency Scheme (BAPS). From the survey data taken together with data from the BAPS scheme, the IoB is now in a position to give guidance on the use of particular instruments and procedures, as well as consider changes to the scope of the BAPS scheme to provide greater benefits for participants concerned with measuring haze. Received: 3 March 1998 · Accepted: 9 June 1998     

Meeting the measurement uncertainty and traceability requirements of ISO/IEC standard 17025 in chemical analysis

The new laboratory accreditation standard, ISO/IEC 17025, reflects current thinking on good measurement practice by requiring more explicit and more demanding attention to a number of activities. These include client interactions, method validation, traceability, and measurement uncertainty. Since the publication of the standard in 1999 there has been extensive debate about its interpretation. It is the author's view that if good quality practices are already in place and if the new requirements are introduced in a manner that is fit for purpose, the additional work required to comply with the new requirements can be expected to be modest. The paper argues that the rigour required in addressing the issues should be driven by customer requirements and the factors that need to be considered in this regard are discussed. The issues addressed include the benefits, interim arrangements, specifying the analytical requirement, establishing traceability, evaluating the uncertainty and reporting the information.     

Estimates of uncertainty of measurement from proficiency testing data: a case study

The results obtained by a laboratory over a number of proficiency testing/external quality assessment schemes (PT/EQAS) rounds can give information on the uncertainty of its measurements for a given test, provided that conditions such as full coverage of the routine analytical range, traceability, and small uncertainty of the assigned values (compared to the spread of the results) are met and provided that systematic deviations and any other sources of uncertainty are considered. As organisers of the Italian EQAS (ITEQAS) in occupational and environmental laboratory medicine, we tested this hypothesis using as model data from well-performing laboratories taking part in ITEQAS for lead in blood over the last 2 years. We also investigated how different PT/EQAS features (frequency of trials and number of samples) would affect a laboratory estimate of its uncertainty. Such information can be helpful in improving PT/EQAS organisation and define, for a given test: (a) the state of the art of the uncertainty of current measurement procedures, (b) identify needs for improvement of analytical methodologies and (c) set targets for acceptable uncertainty values.Presented at the Eurachem PT Workshop September 2005, Portorož, Slovenia.Papers published in this section do not necessarily reflect the opinion of the Editors, the Editorial Board and the Publisher.     

Automated ion-selective measurement of lithium in serum. A practical approach to result-level verification in a two-way method validation

 The Quality Assurance Department of Medix Diacor Labservice evaluated a two-way method validation procedure for serum lithium quantification in therapeutic drug monitoring In the process of a company fusion and rationalization of two considerably large production lines, three independent ion-selective electrode (ISE) methods were surveyed, among many others. While tailoring the new medical laboratory production, subcontracting from a collaborating company was discontinued. Likewise, modernization of the ISE instrumentation was unavoidable to increase throughput and effectiveness. It was important that the new result levels should be comparable both with the former subcontractor's levels and with the levels reported from the previously existing instrumentation. The aim of this study was to evaluate the most crucial performance characteristics of a novel lithium method in comparison to the two ISE test methods being withdrawn. The standardized lithium test method was inspected in terms of linear measurement range, analytical variation, bias, past and on-going proficiency testing, in addition to method comparison, to achieve the desired analytical goals. Fulfilling the accreditation requirements in terms of the introduced method validation parameters is discussed. Received: 19 April 2000 / Accepted: 26 July 2000     

Establishment of an assigned value and its uncertainty for tumour markers in proficiency testing in China

Proficiency testing (PT) is an important aspect of clinical laboratory quality control for tumour markers. To evaluate the performance of an individual laboratory, the assigned value and the standard uncertainty of each tumour marker must be determined. However, no data about the standard uncertainty of the assigned value of any analyte in PT is available in China. The aim of this paper is to introduce a new approach for establishment of assigned value and its standard uncertainty for tumour markers. Five sets of control materials at different concentrations were assigned to each laboratory that participated in PT for tumour markers in 2012. We collected the data from each laboratory using the Clinet-EQA reporting system V1.5 and Clinet-EQA evaluation system V1.0 and developed a software program to calculate the assigned value and its uncertainty according to ISO 13528 (Statistical methods for use in proficiency testing by interlaboratory comparisons. ISO 13528. International Standards Organisation, Geneva, 2005). The assigned value and inter-laboratory CV calculated by the new approach was compared with that obtained by the current approach used in PT in our country. No significant difference between the two outcomes was found in our study. Most assigned values had a standard uncertainty that can meet the criterion: $ u_{X} \le \, 0.3\hat{\sigma } $ u X ≤ 0.3 σ ? , which meant the new approach for establishment of the assigned value was acceptable. It is expected that the new approach to determination of the assigned value and its standard uncertainty for PT for tumour markers in China will enable participants to evaluate their measurement uncertainty.     

Measurement uncertainty and doping control in sport

In a series of alleged cases of doping in sport, the assessment of (non-) compliance with an established threshold value has been under debate. From several cases witnessed, it has been concluded that the work of some of the laboratories responsible for doping control does not comply with present quality requirements, most notably those of ISO/IEC 17025. In all cases studied, the observed value is just above the threshold, and a credible statement of measurement uncertainty is lacking. National accreditation bodies should pay specific attention to the latter issue, as the uncertainty statement is part of the material evidence of a (alleged) doping offence, and, as such, an important integral part of the accreditation status of a laboratory and its scope. The fight against doping is based on proper concepts and principles, but it has been noted that in some cases practice does not always follow the requirements set by the system.     

From GUM to alternative methods for measurement uncertainty evaluation

Since the advent of the Guide to the expression of Uncertainty in Measurement (GUM) in 1995 laying the principles of uncertainty evaluation numerous projects have been carried out to develop alternative practical methods that are easier to implement namely when it is impossible to model the measurement process for technical or economical aspects. In this paper, the author presents the recent evolution of measurement uncertainty evaluation methods. The evaluation of measurement uncertainty can be presented according to two axes based on intralaboratory and interlaboratory approaches. The intralaboratory approach includes “the modelling approach” (application of the procedure described in section 8 of the GUM, known as GUM uncertainty framework) and “the single laboratory validation approach”. The interlaboratory approaches are based on collaborative studies and they are respectively named “interlaboratory validation approach” and “proficiency testing approach”.