Comparability of analytical results obtained in proficiency testing based on a metrological approach

A “yes–no” type of criterion is proposed for the assessment of comparability of proficiency testing (PT) results when the PT scheme is based on a metrological approach, i.e. on the use of a reference material as the test sample, etc. The criterion tests a null hypothesis concerning the insignificance of a bias of the mean of the results from a traceable value certified in the reference material used for the PT. Reliability of such assessment is determined by the probabilities of not rejecting the null hypothesis when it is true, and rejecting it when it is false (the alternative hypothesis is true). It is shown that a number of chemical, metrological and statistical reasons should be taken into account for careful formulation of the hypotheses, enabling the avoidance of an erroneous assessment of the comparability. The criterion can be helpful for PT providers and laboratory accreditation bodies in analysis of PT results.     

Introduction of a new critical p value correction method for statistical significance analysis of metabonomics data

Nuclear magnetic resonance (NMR) spectroscopy-based metabonomics is of growing importance for discovery of human disease biomarkers. Identification and validation of disease biomarkers using statistical significance analysis (SSA) is critical for translation to clinical practice. SSA is performed by assessing a null hypothesis test using a derivative of the Student's t test, e.g., a Welch's t test. Choosing how to correct the significance level for rejecting null hypotheses in the case of multiple testing to maintain a constant family-wise type I error rate is a common problem in such tests. The multiple testing problem arises because the likelihood of falsely rejecting the null hypothesis, i.e., a false positive, grows as the number of tests applied to the same data set increases. Several methods have been introduced to address this problem. Bonferroni correction (BC) assumes all variables are independent and therefore sacrifices sensitivity for detecting true positives in partially dependent data sets. False discovery rate (FDR) methods are more sensitive than BC but uniformly ascribe highest stringency to lowest p value variables. Here, we introduce standard deviation step down (SDSD), which is more sensitive and appropriate than BC for partially dependent data sets. Sensitivity and type I error rate of SDSD can be adjusted based on the degree of variable dependency. SDSD generates fundamentally different profiles of critical p values compared with FDR methods potentially leading to reduced type II error rates. SDSD is increasingly sensitive for more concentrated metabolites. SDSD is demonstrated using NMR-based metabonomics data collected on three different breast cancer cell line extracts.     

Comparability and compatibility of proficiency testing results in schemes with a limited number of participants

Comparability and compatibility of proficiency testing (PT) results are discussed for schemes with a limited number of participants (less than 20–30) based on the use of reference materials (RMs) as test items. Since PT results are a kind of measurement/analysis/test result, their comparability is a property conditioned by traceability to measurement standards applied in the measurement process. At the same time, metrological traceability of the certified value of the RM (sent to PT participants as test items) is also important, since the PT results are compared with the RM certified value. The RM position in the calibration hierarchy of measurement standards sets the degree of comparability for PT results, which can be assessed in the scheme. However, this assessment is influenced by commutability (adequacy or match) of the matrix RM used for PT and routine samples. Compatibility of PT results is a characteristic of the collective (group) performance of the laboratories participating in PT that can be expressed as closeness of the distribution of the PT results to the distribution of the RM data. Achieving quality-of-measurement/analysis/test results in the framework of the concept “tested once, accepted everywhere” requires both comparability and compatibility of the test results.     

Local comparability of proficiency testing results: determination of concrete slump and compressive strength values

A proficiency testing (PT) scheme is developed for comparability assessment of results of concrete slump and compressive strength determination. The scheme is based on preparing of a test portion/sample of a concrete in-house reference material (IHRM) at a reference laboratory (RL) in the same conditions for every PT participant. Therefore, in this scheme IHRM instability is not relevant as a source of measurement/test uncertainty, while intra- and between-samples inhomogeneity parameters are evaluated using the results of RL testing of the samples taken at the beginning, the middle and the end of the PT experiment. The IHRM assigned slump and compressive strength values are calculated as averaged RL results. Their uncertainties include the measurement/test uncertainty components and the components arising from the material inhomogeneity. The test results of 25 PT participants were compared with the IHRM assigned values taking into account both the uncertainties of the assigned values and the measurement/test uncertainties of the participants. Since traceability of the IHRM assigned values to the international measurement standards and SI units cannot be stated, local comparability of the results is assessed. It is shown, that comparability of the slump and compressive strength determination results is satisfactory, while uncertainty evaluation for slump results requires additional efforts.     

Tutorial on evaluation of type I and type II errors in chemical analyses: From the analytical detection to authentication of products and process control

Uncertainty is inherent in all experimental determinations. Nevertheless, these measurements are used to make decisions including the performance of the own measurement systems. The link between the decision and the true implicit system that generates the data (measurement system, production process, category of samples, etc.) is a representation of this uncertainty as a probability distribution. This representation leads to the probabilistic formalization of the possibility of making errors. In the context of regulations established by official agencies, it is important to use these statistical decision methods in some cases because the own norm makes them mandatory and, in general, because this is the way of reasonably evaluating whether a working hypothesis is rejected on the basis of the experimental data.The aim of the present tutorial is to introduce some ideas and basic methods for the critical analysis of experimental data. With this goal, the basic elements of the Neyman-Pearson theory of hypothesis testing are formally introduced in connection with the common problems in chemical analysis and, if this is the case, their relation to the norms of regulatory agencies. The notion of decision with ‘enough quality’ is modelled when explicitly considering: (1) the null, H₀, and alternative, H₁, hypotheses. (2) The significance level of the test, which is the probability, α, of rejecting H₀ when it is true, and the power of the test, 1 − β, β being the probability of accepting H₀ when it is false. (3) The difference between H₀ and H₁ that has to be detected with experimental data. (4) The needed sample size. These four concepts should be explicitly defined for each problem and, under the usual assumption of normal distribution of the data, the mathematical relations among these concepts are shown, which allow the analyst to design a decision rule with pre-set values of α and β.To illustrate the unifying character of this inferential methodology, several situations are exposed along the tutorial: the design of a hypothesis test to decide on the performance characteristics of analytical methods, the capability of detection of both quantitative and qualitative analytical methods (including its generalization to the case of multivariate and/or multiway signals), the analytical sensitivity with multivariate signals, the class-modelling and the process control.     

Harmonisation of proficiency testing schemes

The harmonisation of proficiency testing (PT) schemes has been under debate for a long time. There are obvious reasons why harmonisation of the practices in PT would be beneficial. In many areas, there is still a belief that further harmonisation of practices in PT would improve the comparability of measurement data. In particular when two laboratories are to be compared that have not participated in a single PT, problems arise which allegedly can be overcome by further harmonisation of PT schemes. In practice, however, parties involved in PT are not always embracing the idea of harmonisation. With the results of two European projects in mind, a discussion is given on harmonisation aspects, and some considerations are given that may help to decide in practice whether harmonisation is likely to solve particular problems. The first project, the European Proficiency Testing Network (EPTN), is concerned with further harmonisation. The second European project (COEPT) aims at providing a basis to assess equivalence across proficiency tests, and explores the conditions under which such an assessment is feasible.     

Key metrological issues in proficiency testing––response to “Metrological compatibility-a key issue in further accreditation” by K. Heydorn

A discussion of proficiency testing (PT) topics started by Heydorn (Accred Qual Assur 15:643–645, 2010) is continued in the present paper. The role of PT in the accreditation of testing/analytical laboratories, the use of consensus values (average or weighted average, median, observed standard deviation, etc.) and a metrological background of PT schemes are discussed. It is shown that metrological traceability, comparability, and compatibility, as well as commutability of a reference material, are the key issues of any PT scheme that applies certified reference material as test items. Metrological compatibility of PT results in such schemes is a property demonstrating the closeness of the PT results to the certified value in comparison with the measurement uncertainty of their difference. The metrological background is especially important for the selection and use of PT schemes for a limited number of participants (fewer than 30) as detailed in IUPAC/CITAC Guide on the topic published in 2010 in Pure Appl Chem 82(5):1099–1135.     

Harmonisation of proficiency testing schemes

The harmonisation of proficiency testing (PT) schemes has been under debate for a long time. There are obvious reasons why harmonisation of the practices in PT would be beneficial. In many areas, there is still a belief that further harmonisation of practices in PT would improve the comparability of measurement data. In particular when two laboratories are to be compared that have not participated in a single PT, problems arise which allegedly can be overcome by further harmonisation of PT schemes. In practice, however, parties involved in PT are not always embracing the idea of harmonisation. With the results of two European projects in mind, a discussion is given on harmonisation aspects, and some considerations are given that may help to decide in practice whether harmonisation is likely to solve particular problems. The first project, the European Proficiency Testing Network (EPTN), is concerned with further harmonisation. The second European project (COEPT) aims at providing a basis to assess equivalence across proficiency tests, and explores the conditions under which such an assessment is feasible.

     

On the geometric modeling approach to empirical null distribution estimation for empirical Bayes modeling of multiple hypothesis testing

We study the geometric modeling approach to estimating the null distribution for the empirical Bayes modeling of multiple hypothesis testing. The commonly used method is a nonparametric approach based on the Poisson regression, which however could be unduly affected by the dependence among test statistics and perform very poorly under strong dependence. In this paper, we explore a finite mixture model based geometric modeling approach to empirical null distribution estimation and multiple hypothesis testing. Through simulations and applications to two public microarray data, we will illustrate its competitive performance.     

Evaluation of the use of consensus values in proficiency testing programmes

Proficiency testing (PT) is an essential tool used by laboratory accreditation bodies to assess the competency of laboratories. Because of limited resources of PT providers or for other reasons, the assigned reference value used in the calculation of z-score values has usually been derived from some sort of consensus value obtained by central tendency estimators such as the arithmetic mean or robust mean. However, if the assigned reference value deviates significantly from the ‘true value’ of the analyte in the test material, laboratories’ performance will be evaluated incorrectly. This paper evaluates the use of consensus values in proficiency testing programmes using the Monte Carlo simulation technique. The results indicated that the deviation of the assigned value from the true value could be as large as 40%, depending on the parameters of the proficiency testing programmes under investigation such as sample homogeneity, number of participant laboratories, concentration level, method precision and laboratory bias. To study how these parameters affect the degree of discrepancy between the consensus value and the true value, a fractional factorial design was also applied. The findings indicate that the number of participating laboratories and the distribution of laboratory bias were the prime two factors affecting the deviation of the consensus value from the true value.     

Interlinkages and recognition of proficiency-testing schemes

 Because proficiency testing (PT) is increasingly used for the accreditation of testing laboratories and as a tool for backing up existing multilateral recognition arrangements between accreditation bodies, the question of performance and comparability of the proficiency-test providers arises. In this paper different approaches to assess the equivalence of European PT schemes and the competence of their providers are presented. As a first step a workshop is proposed to agree on a pilot study. The final aim is to create confidence in the existing PT schemes in Europe and to use them as common European tools.     

Estimates of uncertainty of measurement from proficiency testing data: a case study

The results obtained by a laboratory over a number of proficiency testing/external quality assessment schemes (PT/EQAS) rounds can give information on the uncertainty of its measurements for a given test, provided that conditions such as full coverage of the routine analytical range, traceability, and small uncertainty of the assigned values (compared to the spread of the results) are met and provided that systematic deviations and any other sources of uncertainty are considered. As organisers of the Italian EQAS (ITEQAS) in occupational and environmental laboratory medicine, we tested this hypothesis using as model data from well-performing laboratories taking part in ITEQAS for lead in blood over the last 2 years. We also investigated how different PT/EQAS features (frequency of trials and number of samples) would affect a laboratory estimate of its uncertainty. Such information can be helpful in improving PT/EQAS organisation and define, for a given test: (a) the state of the art of the uncertainty of current measurement procedures, (b) identify needs for improvement of analytical methodologies and (c) set targets for acceptable uncertainty values.Presented at the Eurachem PT Workshop September 2005, Portorož, Slovenia.Papers published in this section do not necessarily reflect the opinion of the Editors, the Editorial Board and the Publisher.     

Quantitative Acoustic Microscopy for Determination of Properties of Hard Coatings

 For quantitative assessment of the properties of hard coatings there is an increasing demand for testing methods with high reliability of the test results, especially concerning the independence of the method and the comparability between different laboratories. This includes the knowledge about all the factors which influence the test procedure itself, determination of best testing conditions, testing of these conditions in round-robins to get a view of the comparability of results, and formulation of guidelines for standardization. In a European project several test methods for hard coatings on steel were investigated for this purpose and the elastic moduli of the coating and coating thickness were determined non-destructively by means of quantitative acoustic microscopy. This method and the instruments available had not yet been certified in the fields of coatings simply owing to the absence of standardised signal processing, followed by the determination of sound velocities and materials parameter extraction. For this purpose four laboratories carried out investigations and measurements on reference samples and on two types of hard coatings (titanium nitride and C-doped chromium) on M2 tool steel.     

Quality issues in proficiency testing

 Proficiency testing is a means of assessing the ability of laboratories to competently perform specific tests and/or measurements. It supplements a laboratory's own internal quality control procedure by providing an additional external audit of their testing capability and provides laboratories with a sound basis for continuous improvement. It is also a means towards achieving comparability of measurement between laboratories. Participation is one of the few ways in which a laboratory can compare its performance with that of other laboratories. Good performance in proficiency testing schemes provides independent evidence and hence reassurance to the laboratory and its clients that its procedures, test methods and other laboratory operations are under control. For test results to have any credibility, they must be traceable to a standard of measurement, preferably in terms of SI units, and must be accompanied by a statement of uncertainty. Analytical chemists are coming to realise that this is just as true in their field as it is for physical measurements, and applies equally to proficiency testing results and laboratory test reports. Recent approaches toward ensuring the quality and comparability of proficiency testing schemes and the means of evaluating proficiency test results are described. These have led to the drafting of guidelines and subsequently to the development of international requirements for the competence of scheme providers. Received: 2 January 1999 · Accepted: 7 April 1999     

Evaluation of sampling plans to detect Cry9C protein in corn flour and meal

StarLink is a genetically modified corn that produces an insecticidal protein, Cry9C. Studies were conducted to determine the variability and Cry9C distribution among sample test results when Cry9C protein was estimated in a bulk lot of corn flour and meal. Emphasis was placed on measuring sampling and analytical variances associated with each step of the test procedure used to measure Cry9C in corn flour and meal. Two commercially available enzyme-linked immunosorbent assay kits were used: one for the determination of Cry9C protein concentration and the other for % StarLink seed. The sampling and analytical variances associated with each step of the Cry9C test procedures were determined for flour and meal. Variances were found to be functions of Cry9C concentration, and regression equations were developed to describe the relationships. Because of the larger particle size, sampling variability associated with cornmeal was about double that for corn flour. For cornmeal, the sampling variance accounted for 92.6% of the total testing variability. The observed sampling and analytical distributions were compared with the Normal distribution. In almost all comparisons, the null hypothesis that the Cry9C protein values were sampled from a Normal distribution could not be rejected at 95% confidence limits. The Normal distribution and the variance estimates were used to evaluate the performance of several Cry9C protein sampling plans for corn flour and meal. Operating characteristic curves were developed and used to demonstrate the effect of increasing sample size on reducing false positives (seller's risk) and false negatives (buyer's risk).     

Distribution among sample test results when testing shelled corn lots for fumonisin.

The statistical distribution known as the compound gamma function was studied for suitability in describing the distribution of sample test results associated with testing lots of shelled corn for fumonisin. Thirty-two 1.1 kg test samples were taken from each of 16 contaminated lots of shelled corn. An observed distribution consisted of 32 sample fumonisin test results for each lot. The mean fumonisin concentration, c, and the variance, s2, among the 32 sample fumonisin test results along with the parameters for the compound gamma function were determined for each of the 16 observed distributions. The 16 observed distributions of sample fumonisin test results were compared with the compound gamma function using the Power Divergence test. The null hypothesis that the observed distribution could have resulted from sampling a family of compound gamma distributions was not rejected at the 5% significance level for 15 of the 16 lots studied. Parameters of the compound gamma distribution were calculated from the 32-fumonisin sample test results using the method of moments. Using regression analysis, equations were developed that related the parameters of the compound gamma distribution to fumonisin concentration and the variance associated with a fumonisin test procedure. An operating characteristic curve was developed for a fumonisin sampling plan to demonstrate the use of the compound gamma function.     

Proficiency testing program for the determination of total arsenic, cadmium, and lead in seawater shrimp

The proficiency testing (PT) program for 97 worldwide laboratories for determining total arsenic, cadmium, and lead in seawater shrimp under the auspices of the Asia-Pacific Laboratory Accreditation Cooperation (APLAC) is discussed. The program is one of the APLAC PT series whose primary purposes are to establish mutual agreement on the equivalence of the operation of APLAC member laboratories and to take corrective actions if testing deficiencies are identified. Pooled data for Cd and Pb were normally distributed with interlaboratory variations of 21.9 and 34.8%, respectively. The corresponding consensus mean values estimated by robust statistics were in good agreement with those obtained in the homogeneity tests. However, a bimodal distribution was observed from the determination of total As, in which 14 out of 74 participants reported much smaller values (0.482-6.4 mg/kg) as compared with the mean values of 60.9 mg/kg in the homogeneity test. The use of consensus mean is known to have significant deviation from the true value in bi- or multimodal distribution. Therefore, the mode value, a better estimate of central tendency, was chosen to assess participants' performance for total As. Estimates of the overall uncertainty from participants varied in this program, and some were recommended to acquire more comprehensive exposure toward important criteria as stipulated in ISO/IEC 17025.     

Role of proficiency testing in monitoring of standardization of hemoglobin A1c methods

After Hemoglobin A_1c (HbA_1c), therapeutic targets for monitoring diabetes therapy were recommended, first, National Glycohemoglobin Standardization Program (NGSP), then, the International Federation of Clinical Chemistry and Laboratory Medicine (IFCC) developed standardization initiatives. The aim of this article is to demonstrate the role of a proficiency testing (PT) programs in monitoring the long-term effect of these initiatives and the current status of HbA_1c measurement. Measurement precision as a coefficient of variation (CV), measurement bias (bias), and satisfactory HbA_1c result rates in proficiency testing (PT) surveys were evaluated using fresh single donor whole blood PT items and assigned values from a NGSP-certified secondary reference laboratory. Between 2000 and 2010, both CV and bias of the IC measurement method showed a decreasing trend. While the CV of the HPLC measurement method decreased, no significant change was observed in its bias. The rates of satisfactory HbA_1c results in PT surveys were higher in HPLC users than IC users. In 2010, the average CVs in HPLC and IC groups were 2.6 and 3.4?%, biases were 2.7 and 1.8?%, and corresponding total error (TE) estimates were 7.8 and 8.5?%, respectively. These TE values were higher than the maximum permissible measurement error of 7?%, developed based on clinical use of the test. The NGSP and the IFCC networks have promoted improvements in HbA_1c testing; however, tightening of NGSP method certification criteria seems to be necessary to achieve a maximum permissible measurement error of 7?%.     

Quality assurance and quality control in forest soil analyses: a comparison between European soil laboratories

In transnational monitoring programmes, a balance between international reference methods, which improve spatial comparability, and national analysis methods that favour temporal comparability, by their use and testing over many years, needs to be sought. Prior to the next Pan-European Forest Soil Survey, a third interlaboratory comparison of soil analysis methods was organised. All participating laboratories were requested to use the same reference methods. Fifty-two soil laboratories from 27 European countries analysed a total of 48 soil parameters on three soil samples which were typical for European forest soils. The results of the statistical analysis showed a high interlaboratory and intralaboratory variability, especially for the acid oxalate extractions, particle size distribution, exchangeable elements and total carbonates. The intercomparability of the test results did not improve compared to the previous ring test. As the exercise aimed primarily at comparing the performance of the laboratories, it was not powerful enough to find cause–effect relationships between the meta information provided by the laboratories and the variability of the test results.     

Quality assurance and quality control in forest soil analyses: a comparison between European soil laboratories

In transnational monitoring programmes, a balance between international reference methods, which improve spatial comparability, and national analysis methods that favour temporal comparability, by their use and testing over many years, needs to be sought. Prior to the next Pan-European Forest Soil Survey, a third interlaboratory comparison of soil analysis methods was organised. All participating laboratories were requested to use the same reference methods. Fifty-two soil laboratories from 27 European countries analysed a total of 48 soil parameters on three soil samples which were typical for European forest soils. The results of the statistical analysis showed a high interlaboratory and intralaboratory variability, especially for the acid oxalate extractions, particle size distribution, exchangeable elements and total carbonates. The intercomparability of the test results did not improve compared to the previous ring test. As the exercise aimed primarily at comparing the performance of the laboratories, it was not powerful enough to find cause–effect relationships between the meta information provided by the laboratories and the variability of the test results.