Laboratory effects models for interlaboratory comparisons

The statistical analysis of results from inter-laboratory comparisons (for example Key Comparisons, or Supplemental Comparisons) produces an estimate of the measurand (reference value) and statements of equivalence of the results from the participating laboratories. Methods to estimate the reference value have been proposed that rest on the idea of finding a so-called consistent subset of laboratories, that is, eliminating allegedly outlying participants. We propose an alternative statistical model that accommodates all participant data and incorporates the dispersion of the measurement values obtained by different laboratories into the total uncertainty of the various estimates. This model recognizes the fact that the dispersion of values between laboratories often is substantially larger than the measurement uncertainties provided by the participating laboratories. We illustrate the methods on data from key comparison CCQM–K25.     

Evaluation of the use of consensus values in proficiency testing programmes

Proficiency testing (PT) is an essential tool used by laboratory accreditation bodies to assess the competency of laboratories. Because of limited resources of PT providers or for other reasons, the assigned reference value used in the calculation of z-score values has usually been derived from some sort of consensus value obtained by central tendency estimators such as the arithmetic mean or robust mean. However, if the assigned reference value deviates significantly from the ‘true value’ of the analyte in the test material, laboratories’ performance will be evaluated incorrectly. This paper evaluates the use of consensus values in proficiency testing programmes using the Monte Carlo simulation technique. The results indicated that the deviation of the assigned value from the true value could be as large as 40%, depending on the parameters of the proficiency testing programmes under investigation such as sample homogeneity, number of participant laboratories, concentration level, method precision and laboratory bias. To study how these parameters affect the degree of discrepancy between the consensus value and the true value, a fractional factorial design was also applied. The findings indicate that the number of participating laboratories and the distribution of laboratory bias were the prime two factors affecting the deviation of the consensus value from the true value.     

Using Bayes methods and mixture models in inter-laboratory studies with outliers

Inter-laboratory studies (especially so-called key comparisons) are conducted to evaluate both national and international equivalence of measurement. In these studies, a reference value of some measurand (the quantity intended to be measured) is developed and results for all laboratories are compared to this single value. How to determine the reference value is not completely obvious if there are observations and/or laboratories that could be considered outliers. Since ignoring results from one or more participating laboratories is untenable in practical terms, developing methods that are robust to the possibility that a small fraction of the laboratories produces observations unlike those from the others is critical. This paper outlines two Bayesian methods of analyzing inter-laboratory data that have been proposed in the literature and suggests three modifications of one that are more robust to outliers. A simulation study is conducted to compare the five methods.     

Proficiency testing of chloride content in different types of Portland cement

The accurate determination of chloride content in cement is a prerequisite for the production of high-quality concrete due to the damaging influence of free chloride ions on durability and safety of concrete constructions. The results for total chloride content in four different types of Portland cement provided by testing laboratories participating in an inter-laboratory comparison are presented. The data sets were evaluated by using different statistical methods. The laboratories exceeding critical values were signaled and sources of error were investigated. The precision parameters of the method were determined and the variance of the entire data set was apportioned among the sources of variation. The systematic differences between participating laboratories and heterogeneity of the cement samples were identified as major sources of variability of results. The ANOVA determined the intermediate precision parameters, which showed a low value for random errors resulting in good repeatability and high impact of sample and inter-laboratory difference on reproducibility of the method.Presented at the Eurachem PT Workshop September 2005, Portorož, Slovenia     

Proficiency testing in food microbiology: experience from implementation of ISO/IEC 17043 and ISO/TS 22117

The proficiency testing (PT) scheme ??AQUA?? for food microbiology was organised by the Istituto Zooprofilattico Sperimentale delle Venezie (IZSVe) according to ISO/IEC 17043 and ISO/TS 22117. This paper describes the IZSVe experience on the application of the above-mentioned standards for the PTs, with focus on the Enterobacteriaceae enumeration one. Freeze-dried food matrices contaminated with American Type Culture Collection bacterial strains were used as test samples for each microbiological PT organised by IZSVe. The sample homogeneity and stability were verified prior to distribution to participants and throughout the PT, respectively. The participating laboratories analysed samples using their routine methods, and results were transmitted to IZSVe. Data and methods used by each participating laboratory were analysed in order to evaluate the laboratory performance. With reference to the Enterobacteriaceae PT, the test samples were homogeneous and stable. In addition, most laboratory results were obtained using equivalent test methods. Statistical approaches applied to analyse data generated from all participating laboratories revealed similar outcomes as no significant outlying count and only 5?% of unacceptable results were observed. Finally, the z-score, with the standard deviation that does not vary from round toround, was applied to compare and to evaluate the performance of each laboratory over time highlighting possible persistent trends over several rounds.     

An inter-laboratory comparison for determination of cortisol in saliva

An inter-laboratory comparison study for cortisol in saliva is reported in the present study. Nine laboratories representing four different methods participated in the study. Each laboratory measured five blind samples prepared from natural saliva spiked with a pure certified reference material in the range 0–70 nmol/L. The average observed values were established as means of eight laboratories, because one laboratory was excluded as an outlier. The natural content of cortisol in the pooled saliva was 6.8 nmol/L. The recovery of the certified reference material was 83%–131% (164% for the outlying laboratory). For five laboratories the 95%-confidence intervals of the recoveries did not cover 100%. The standard deviations for a concentration of 10 nmol/L were estimated for each laboratory. They were 0.64–1.67 nmol/L (6.26 nmol/L for the outlying laboratory). The present study emphasizes the need for an external proficiency-testing scheme and a certified reference material for cortisol in saliva. Received: 8 May 2002 Accepted: 3 September 2002     

Economy of proficiency testing: reference versus consensus values

Many proficiency tests are operated with a consensus value derived from the participants’ results. Apart from technical issues, one of the reasons often mentioned is that proficiency tests operated with consensus values would be cheaper than those using reference values obtained from a priori characterisation measurements. The economy of a proficiency test must of course be balanced by the need of the participants, and the quality of the comparison in general. The proficiency tests selected in this study had both a reference value and a consensus value, one of which was used for assessing the performance of the participating laboratories. In this work, both a technical and an economical assessment of how the comparisons were operated is made. From the evaluation, it follows that usually the use of consensus values does not necessarily reduce the costs of a proficiency test. However, frequently it may be observed that the quality of the assessment of the laboratories is better with a reference value. Received: 11 October 2000 Accepted: 3 January 2001     

Performance evaluation for proficiency testing with a limited number of participants

Proficiency testing (PT) is an essential tool for laboratories to assess their competency. Also, participation in PT has become one of the mandatory requirements for laboratory to seek accreditation according to ISO/IEC 17025. For this reason, the effectiveness of performance evaluation by PT scheme is of great concern for the participants and for accreditation bodies as well. In practice, owing to unavailability of other appropriate alternatives, PT scheme providers may have to choose using consensus values to evaluate the performance of participants. However, such consensus values approach was not recommended by relevant international guidelines for PT schemes with limited number of participants. With the use of Monte Carlo simulation technique, this study attempted to investigate the effectiveness of using consensus values for performance evaluation in PT schemes with limited number of participants. The simulation process was schemed according to the statistical model provided by ISO 5725-1 for laboratory measurement results, which covered components like method bias, laboratory bias, and measurement precision. The effectiveness of the consensus value approach was expressed as the percentage of participants in a simulation run could get the same evaluation result, either satisfactory or unsatisfactory, against the “true value.” The findings indicated that the number of participants, choice of consensus values, mass fraction of analyte, method bias, laboratory bias, and measurement repeatability of participating laboratories would all affect the effectiveness of the consensus value approach but at different extent. However, under certain circumstances, use of consensus value could still be considered as an acceptable approach for performance evaluation even the number of participants was limited. Some of the findings were further verified using real data from PT schemes where appropriate certified reference materials or reliable reference values were available.     

Analysis of interlaboratory performance in the determination of total selenium in water

This study explored the performance of experienced laboratories in the analysis for total selenium in water by a variety of analytical methods. The goal of the study was to examine intra- and interlaboratory variability. Replicates (n = 7) of 7 sample types that included a reference material of known Se concentration, natural waters, and treated wastewaters were submitted to 7 laboratories with prequalified Se analytical experience. Results of the study indicated wide ranges in minimum and maximum results, distinct differences in laboratory precision, and routine reporting of numerical results below statistical limits of quantitation. Hydride generation as a sample introduction technique demonstrated superior performance. In general, the study supports a caution advisory about using low-level Se data, especially results lower than about 10 micrograms Se/L, without quantitating the statistical uncertainty of the data. Because this study used data from samples that were submitted in bulk to participating laboratories prequalified for Se analytical expertise and experience, it can be considered a best-case demonstration of performance.     

Inter-laboratory validation of the fluorescent protein phosphatase inhibition assay to determine diarrhetic shellfish toxins: intercomparison with liquid chromatography and mouse bioassay

Toxic episodes of diarrhetic shellfish toxins (DSP) in shellfish harvesting areas have serious economic and public health implications, where fluorescent protein phosphatase inhibition assay (FPPIA) may be a highly useful tool for monitoring purposes. This paper presents results from the first inter-laboratory study to validate the assay. Three laboratories participated in the design and development of the inter-laboratory work. Standard solutions and spiked samples of the main toxin, okadaic acid, were used at the beginning of the validation exercise to avoid cross-inhibition of other toxins that would otherwise deteriorate the quantitative significance of the data. HPLC with fluorimetric detection of okadaic acid was also submitted to inter-laboratory validation to be subsequently used as a quantitative reference method. FPPIA results from spiked samples were free of systematic bias in any laboratory and determinations repeated over 3 days showed that the classic “repeatability” was the main within-laboratory source of variability (15-26% R.S.D. depending on the sample).After the inter-laboratory validation of both HPLC and FPPIA methods, 83 samples of mussel hepatopancreas collected during a toxic DSP episode were analyzed over 9 weeks. Toxic levels determined with FPPIA were in line with mouse bioassay results, highlighting the lack of false negative results of the FPPIA test: 98.7% of samples whose concentration of okadaic acid equivalents was over 0.8 μg/g hep., provided positive bioassay results within 24 h of observation time. The reliability and the quantitativeness of the FPPIA method in naturally contaminated samples was demonstrated by intercomparison with mouse bioassay and HPLC.     

Inter-laboratory comparison of liquid chromatographic techniques and enzyme-linked immunosorbent assay for the determination of surfactants in wastewaters

Seven laboratories participated in an inter-laboratory comparison exercise within the framework of the PRISTINE, SANDRINE and INEXsPORT European Union Projects. Solid-phase extraction (SPE) methodologies were used for the extraction of target analytes from wastewaters. The analytical strategies were based on liquid chromatography (LC) coupled to mass spectrometric (MS) or to fluorescent (FL) detection in all cases with the exception of one laboratory using a test-tube enzyme-linked immunosorbent assay kit. Samples were spiked with the surfactants nonylphenolpolyglycol ether, coconut diethanolamide, linear alkylbenzene sulfonate, nonylphenolpolyglycol ether sulfate, alkylpolyglycol ether and secondary alkane sulfonate. After enrichment on previously conditioned SPE cartridges, the SPE cartridges were distributed among the participating laboratories without the information about the amount of spiked surfactants. In addition, SPE cartridges loaded with a real-world environmental sample containing a tannery wastewater were also analyzed. The results of the programme showed that SPE followed by LC-MS techniques are reliable for the surfactants determination at submicrogram to microgram per liter levels in wastewaters. Inter-laboratory precision values were calculated as the reproducibility relative standard deviation (RSD(R)) which was determined from the reproducibility standard deviation (sR) and the average concentration at a particular concentration level. When data from all laboratories were pooled, the RSD(R) values ranged from 5.1 to 28.3% for the determination of target analytes. The most accurate result corresponded to that given for linear alkylbenzene sulfonates. Taking into account that different methodologies were used (including non-chromatographic techniques) and the complexity of the samples analyzed, it can be considered that acceptable reproducibility values were obtained in this inter-laboratory study.     

Development of a proficiency testing scheme for a limited number of participants in the field of natural water analysis

A proficiency testing (PT) scheme was developed for a limited number of analytical laboratories participating in the analysis of natural water in Israel. Three fit-for-intended-use reference materials (RMs) were prepared for a pilot PT from natural water: RM-blank and two in-house reference materials (IHRMs) with seven analytes fortified to achieve different levels of analyte concentrations. The measurands for the PT participants were the traceable spike values certified in the IHRMs, i.e., added mass concentrations of the analytes. The RM-blank and IHRMs were found to be homogeneous and stable over 11 days, the time necessary for the experimental part of the PT. The RMs were distributed to the PT participants as unknown test items similar to routine samples. The test results were evaluated using their deviations from the IHRM-certified values. Eight Israeli laboratories took part in the interlaboratory comparison. Individual laboratory performance and metrological compatibility of the PT results of the participants, as a local group of laboratories, were evaluated for every analyte.     

Inter-laboratory comparison of methods used for analysing polycyclic aromatic hydrocarbons (PAHs) in soil samples

Summary In view of the investigation of contaminated model sites in Baden-Württemberg an inter-laboratory comparison was organized for the methods used for analysing polycyclic aromatic hydrocarbons (PAHs). The laboratories performing chemical and physical analyses at the model sites participated at the request of the Landesanstalt für Umweltschutz, Baden-Württemberg. This inter-laboratory comparison was meant as a first laboratory evaluation, and not as the search for the best method available. Furthermore, it was to show the difficulties in comparing results from different laboratories, which have analysed identical samples. The results of the inter-laboratory comparison between the nine laboratories showed a reproducibility coefficient of variation, for the total of the 16 PAHs according to the EPA, between 48.8 and 61.1% in the concentration range of 5 to 55 mg/kg in the soil samples. One of the laboratories reported significantly lower values compared with the true value. This was probably due to an ill-suited (sample) preparation method. Although the results of the inter-laboratory comparison are, in general, satisfactory, the reproducibility can be improved by standardizing the analytical method for the determination of PAHs in solid environmental samples.     

Providing a traceable assigned value in a proficiency testing programme on the determination of benzoic acid in orange juice

A proficiency testing (PT) programme on the determination of benzoic acid in orange juice was organised by the Chemical Metrology Laboratory, Health Sciences Authority, Singapore, for the local food testing laboratories. The PT programme used a metrologically traceable assigned value determined by isotope dilution mass spectrometry to assess the performance of the participating laboratories. This paper discusses the reasons and approach to applying metrological principles to determine the assigned value of a PT programme. The procedure undertaken by the Laboratory to prepare a batch of PT sample, conduct homogeneity and stability testings, as well as value assignment, is presented. The discussions also include the estimation of the measurement uncertainties arising from the characterisation, testing for homogeneity and testing for stability of the PT sample. The Coordinator of the PT programme used the assigned value and its associated measurement uncertainty, together with those of the participating laboratories to assess their performances. Statistical tools applied in the performance evaluations are also presented.     

Determination of Assigned Values for Cadmium,Cobalt and Manganese in Drinking Water Proficiency Testing

In proficiency tests the selection of an appropriate assigned value is one of the most critical points. Determination of the assigned value for proficiency test can be done in 2 ways: based on reference value and consensus value. The use of such a consensus value as an assigned value is a commonplace in proficiency testing, because there is seldom at present a cost effective alternative. However, the consensus value has a weakness in that it could be significantly different from the true value due largely inexperienced laboratories. Therefore in this work, the reference values were used as the assigned values for all analytes of interest. The reference values in spiked drinking water samples were produced by the Metrology in Chemistry Laboratory of Research Center for Chemistry - Indonesian Institute of Sciences (RCChem-LIPI) with traceable to SI units through standard solutions from the National Institute Standard and Technology, USA. The reference values of cobalt and manganese were obtained by measurement using two from three comparable methods of Graphite Furnace-Atomic Absorption Spectrometry (GF-AAS), Inductively Coupled Plasma-Optical Emission Spectrometry (ICP-OES) and Inductively Coupled Plasma-Mass Spectrometry (ICP-MS) while that for cadmium was obtained by a primary method of ICP-Isotope Dilution Mass Spectrometry (ICP-IDMS).The obtained reference values were used to evaluate the laboratory performance result in a proficiency testing scheme of cadmium, cobalt and manganese in drinking water which was provided in 2014. All fifty nine laboratories reported cadmium and cobalt at low concentration levels which were about more than 82% showed satisfactory performance results. Similarly, the results of manganese were also comparable at high level concentrations.     

Interlaboratory system to ensure and improve the quality of glassware calibration and use in a large laboratory

This paper describes the implementation and methodology of an interlaboratory system that ensures the quality of glassware calibration and use in a large laboratory. The interlaboratory system involves periodic comparisons between laboratories with evaluations and improvements made over time. Two similar items are calibrated in each exercise according to a detailed calibration procedure. The reference value is traceable to the international system supplied by a metrology laboratory. The results are evaluated as normalized errors and analyzed by Youden graphs. The calibration procedure is presented. An interlaboratory experiment is described in which 7 participating laboratories performed calibrations of 2 volumetric flasks. The reported results, the interlaboratory evaluation, and the actions taken are presented.     

An inter-laboratory study of detection limits in the analysis of water and wastewater for organo-chlorine pesticides by liquid/liquid extraction and gas chromatography-electron capture detector (USEPA Method 608)

An inter-laboratory study was conducted to assess the Kaiser-Currie Model (KCM) for the determination of detection limits. Six laboratories participated in the analysis of samples prepared from distilled water, some containing organo-chlorine pesticides at a concentration of zero and other with a greater than zero concentration. The study consisted of three phases, the first of which was a study to assess the longer term variability of distilled water samples containing no organo-chlorine pesticides prepared by the participating laboratory analysed over a six month period. A second phase consisted of replicates of distilled water samples containing organo-chlorine pesticides prepared at a single concentration greater than zero by the laboratory and were analysed over several days. Finally, a third phase consisted of twelve distilled water samples, eleven containing organo-chlorine pesticides at a concentration of greater than zero and one with a concentration of zero prepared by a third party. Estimated detection limits were determined and then compared to the observed detection limits. Only in a minority of cases, where the distribution of results from samples containing a concentration of zero was normally distributed, did Currie's L _C accurately predict a concentration which corresponded to a 1% false positive rate in distilled water samples with a zero concentration of the study analyte. The USEPA's MDL performed more poorly. In the majority of cases, when any non-zero results were obtained from distilled water samples containing a concentration of zero, they were not normally distributed. Contrary to expectation, false negatives and false positives rarely occurred simultaneously on any given day. The variability between days of analysis and the use of noise reducing techniques proved to be a significant source of the observed non-normal distribution of distilled water samples. Conventional procedures based on the KCM and their underlying analytical and statistical assumptions did not provide useful predictions of laboratory sensitivity in most cases in this study.     

Evaluation of the results from an inter-laboratory comparison study of the determination of acrylamide in crispbread and butter cookies

Analytical methods currently employed for determination of acrylamide (AA) in two carbohydrate-rich food samples, crispbread and butter cookies, obtained commercially, and native and spiked bread extract samples have been evaluated in a collaborative study. The objective of the study was to obtain information about the performance of the participating laboratories when analysing samples with an AA content close to the limit of quantification (LOQ) and at a higher AA level, and to investigate the influence of sample-preparation procedures on the results of the analysis. For this purpose an aqueous native extract of white bread crumb, a fortified extract, and AA standard solutions, the analyte content of which were not disclosed to the participants, were included in the study. A total of 62 laboratories, applying seven different measurement techniques and a broad spectrum of analyte extraction and sample-preparation procedures reported their analytical results. Because the measurement data were not normally distributed, they were evaluated by application of robust statistics. The relative performance of the laboratories was highlighted by calculation of z-scores. For the crispbread sample, especially, a large percentage of the calculated z-scores were outside the satisfactory range. From their distribution it became obvious that one of the analytical techniques might be biased, if not applied correctly. Consequently, the impact of the applied methods was examined in more detail. Information about the analytical technique, extraction solvent, quantity weighed, calibration method, clean-up, and the experience of the participating laboratories were extracted from the analytical protocols and transcribed into a data matrix which was evaluated by multifactor analysis of variance. The applied measurement technique seems to have a statistically significant influence on the analytical results.     

Inter-laboratory trials for analysis of perfluorooctanesulfonate and perfluorooctanoate in water samples: Performance and recommendations

The ISO 25101 (International Organization for Standardization, Geneva) describes a new international standard method for the determination of perfluorooctanesulfonate (PFOS) and perfluorooctanoate (PFOA) in unfiltered samples of drinking and surface waters. The method is based on the extraction of target analytes by solid phase extraction, solvent elution, and determination by high performance liquid chromatography–tandem mass spectrometry (HPLC–MS/MS). For the determination of the performance of this method, more than 20 laboratories from 9 different countries participated in an inter-laboratory trial in 2006. In addition, inter-laboratory trials were conducted in 2008 and 2009 for the analysis of perfluoroalkylsubstances (PFASs), including PFOS and PFOA, in water samples by following the protocols of Japanese Industrial Standard (JIS). Overall, the repeatability coefficients of variation (i.e., within-laboratory precision) for PFOS and PFOA in all water samples were between 3 and 11%, showing a adequate precision of the ISO and JIS methods. The reproducibility coefficients of variation (i.e., between-laboratory precision) were found to vary within a range of 7–31% for surface water and 20–40% for wastewater. The recoveries of PFOS and PFOA, as a measure of accuracy, varied from 84 to 100% for surface water and from 84 to 100% for wastewater among the samples with acceptable criteria for internal standards recovery. The determined concentrations of PFASs in samples compared well with the “true” values. The results of the inter-laboratory trial confirmed that the analytical methods are robust and reliable and can be used as a standard method for the analysis of target compounds in water samples.