Methods validation: AOAC's three validation systems

Validated methods of analysis are needed for many purposes: enforcement of regulations, import/export control, in accredited laboratories, academia, institutions. The AOAC INTERNATIONAL Official Methods Program is designed to provide fully validated methods of analysis, based on interlaboratory testing by a minimum of eight laboratories. Another, lesser validation system is used for peer-verified methods of analysis where two or three laboratories participate. The system for performance testing of test kits is specially designed for a thorough testing of manufacturer claims, and can be obtained by submitting a kit to Performance Testing by the AOAC Research Institute.     

Statistical comparison of the slopes of two regression lines: A tutorial

Comparing the slopes of two regression lines is an almost daily task in analytical laboratories. The usual procedure is based on a Student’s t-test although literature differs in whether the standard errors of the slopes or the standard errors of the regressions should be employed to get a pooled standard error. In this work fundamental concepts on the use of the Student’s test were reviewed and Monte Carlo simulations were done to ascertain whether relevant differences arise when the two options are considered. It was concluded that for small sample sets (as it is usual in analytical laboratories) the Student’s t-test based on the standard error of regression models must be used and special attention must be paid on the equality of the models variances. Finally, alternative approaches were reviewed, with emphasis on a simple one based on the analysis of the covariance (ANCOVA).     

Using Bayes methods and mixture models in inter-laboratory studies with outliers

Inter-laboratory studies (especially so-called key comparisons) are conducted to evaluate both national and international equivalence of measurement. In these studies, a reference value of some measurand (the quantity intended to be measured) is developed and results for all laboratories are compared to this single value. How to determine the reference value is not completely obvious if there are observations and/or laboratories that could be considered outliers. Since ignoring results from one or more participating laboratories is untenable in practical terms, developing methods that are robust to the possibility that a small fraction of the laboratories produces observations unlike those from the others is critical. This paper outlines two Bayesian methods of analyzing inter-laboratory data that have been proposed in the literature and suggests three modifications of one that are more robust to outliers. A simulation study is conducted to compare the five methods.     

Review of European Union requirements: quality standards for food analysis laboratories

 The European Union has prescribed strict quality standards for official food laboratories and the methods of analysis to be used in laboratories when carrying out official food control work. These requirements, which are based on accreditation, participation in proficiency testing schemes and using validated methods of analysis, are described in detail. The similar approach being taken within the Codex Alimentarius Commission is also outlined. The procedures prescribed will ensure that official food control laboratories have in place the measures to ensure that consistently reliable data can be produced. Received: 29 November 1995 Accepted: 8 January 1996     

Quality and timeliness in medical laboratory testing

In terms of testing, modern laboratory medicine can be divided into centralized testing in central laboratories and point-of-care testing (POCT). Centralized laboratory medicine offers high-quality results, as guaranteed by the use of quality management programs and the excellence of the staff. POCT is performed by clinical staff, and so such testing has moved back closer to the patient. POCT has the advantage of shortening the turnaround time, which potentially benefits the patient. However, the clinical laboratory testing expertise of clinical staff is limited. Consequently, when deciding which components of laboratory testing must be conducted in central laboratories and which components as POCT (in relation to quality and timeliness), it will be medical necessity, medical utility, technological capabilities and costs that will have to be ascertained. Provided adequate quality can be guaranteed, POCT is preferable, considering its timeliness, when testing vital parameters. It is also preferred when the central laboratory cannot guarantee the delivery of results of short turn-around-time (STAT) markers within 60 or (even better) 30 min. POCT should not replace centralized medical laboratory testing in general, but it should be used in cases where positive effects on patient care have been clearly demonstrated.     

Quality issues in proficiency testing

 Proficiency testing is a means of assessing the ability of laboratories to competently perform specific tests and/or measurements. It supplements a laboratory's own internal quality control procedure by providing an additional external audit of their testing capability and provides laboratories with a sound basis for continuous improvement. It is also a means towards achieving comparability of measurement between laboratories. Participation is one of the few ways in which a laboratory can compare its performance with that of other laboratories. Good performance in proficiency testing schemes provides independent evidence and hence reassurance to the laboratory and its clients that its procedures, test methods and other laboratory operations are under control. For test results to have any credibility, they must be traceable to a standard of measurement, preferably in terms of SI units, and must be accompanied by a statement of uncertainty. Analytical chemists are coming to realise that this is just as true in their field as it is for physical measurements, and applies equally to proficiency testing results and laboratory test reports. Recent approaches toward ensuring the quality and comparability of proficiency testing schemes and the means of evaluating proficiency test results are described. These have led to the drafting of guidelines and subsequently to the development of international requirements for the competence of scheme providers. Received: 2 January 1999 · Accepted: 7 April 1999     

Development of melamine certified reference material in milk using two different isotope dilution mass spectrometry techniques

A new certified reference material (CRM) of melamine in milk GLHK-11-02 was developed aiming to address the great demand from the testing community after the melamine crises. The material was prepared by adding an appropriate quantity of melamine into the skimmed milk samples and the final product was in the form of fine lyophilized powder. Characterization of the material relied on two newly developed gravimetric isotope dilution mass spectrometry (IDMS) methods, one using liquid chromatography-tandem mass spectrometry (LC-MS/MS) and another gas chromatography-mass spectrometry (GC-MS). Experimental parameters with crucial effects on the performance of the two IDMS methods were thoroughly investigated. These included purity of standard used, equilibration time of isotopes, efficiency of extraction methods as well as possible interferences from the matrix and melamine analogues. Precision was found to be excellent with a coefficient of variation of 2.5% for the LC-IDMS/MS (n=46) and 1.9% for the GC-IDMS (n=30) respectively. Using one-tail Student's t-test at 95% confidence interval, analytical data sets generated from the two methods were found to exhibit no significant difference. Measurement accuracy of the methods was further verified through an Asia Pacific Metrology Program (APMP) pilot study. Analytical results of the present LC-IDMS/MS for the two milk test samples at the concentration level of about 0.45 and 3.5 mg kg(-1) were proven to be very good. There were excellent overlaps between our results and the assigned reference values, and the absolute deviation was less than 3.2%. Both the LC-IDMS/MS and GC-IDMS methods were shown to be sufficiently reliable and accurate for certification of the melamine CRM. Certified value of melamine in dry mass fraction in GLHK-11-02 was 1.14 mg kg(-1). Expanded uncertainty due to sample inhomogeneity, long term and short term stability and variability in the characterization procedure was at 7.1% or 0.08 mg kg(-1). The CRM is primarily used to provide a complete method validation for and to improve the technical competence of melamine analysis to food and chemical testing laboratories.     

The use of uncertainty estimates of test results in comparisons with acceptance limits

When a test is performed in order to qualify a material or a product for a certain use, the result is generally compared with an acceptance limit. The test result has an uncertainty which should be estimated and stated (e.g. in accordance with GUM). Very often this is not the case. Further, discussions often arise on the issue of how the uncertainty shall be considered in relationship to the acceptance limit. The intention of this note is to describe, in simple terms, the statistical background and to give some recommendations. In short, there are two clean-cut, extreme situations. The first case is when the uncertainty of the testing procedure is the dominating factor. Here it is found that the estimates of single laboratories cannot, generally, be used for comparisons with acceptance limits. One should have standardised, well-verified estimates based on comprehensive investigations of the method. It can also be concluded that comparisons between test results and acceptance limits have to be made with regard to the actual circumstances, as, e.g. how the acceptance limit is related to the risk. In the second case, the variation in the property of the material or product dominates and the uncertainty of the testing procedure is negligible. When the results are non-quantitative (go – no go), statistical methods can be used to estimate the risk taken with a certain sampling and acceptance strategy that a certain proportion of the batch to be delivered does not qualify. This should be considered more often in standardisation of product test methods. When the results are quantitative, a statistical analysis should be performed and the uncertainty should be compared with the acceptance limit as before, from the actual circumstances. When effects of testing uncertainty and product variation are comparable a sound treatment requires extensive experimental work. No short cuts can be made without loss of confidence! Received: 17 August 2001 Accepted: 21 March 2002     

Investigation of two computerized procedures for energy-dispersive X-ray fluorescence spectrometry by the fundamental parameter approach

Both the computerized procedures investigated for x-ray fluorescence spectrometry exploit the fundamental parameter approach. Procedures 1 and 2 are without and with absolute calibration of the spectrometer, respectively. The basic fundamental parameter equations are shown and the equivalence with the notation of influence coefficients is demonstrated. The advantages and limitations of the two procedures are discussed. If the sample contains no appreciable fraction of light elements, procedure 1 is well suited for standardless quantitative calculations. Procedure 2 is not limited by this restriction, and its use may indicate if the qualitative elemental analysis is complete. Even in the case of incomplete qualitative analysis, procedure 2 gives usable quantitative results. Calibration inaccuracies with procedure 2 are easily tested by procedure 1, thus the two procedures complement each other.     

Quality specifications for evaluation and comparison of performance among external quality assessment schemes in occupational and environmental laboratory medicine

Quality specifications (QS) are proposed for lead in blood and for aluminium, copper, selenium and zinc in serum as part of the aim to set standards of performance for laboratories so that results can be demonstrated to be fit for the purpose to which they are applied. The QS were established taking account of the analytical state-of-the-art, physiological variations in the concentrations of the analyte and the clinical purpose for which the assay is to be used. A procedure was devised that uses these QS to give equivalence of assessment among external quality assessment schemes (EQAS), thus avoiding conflicting information which has been demonstrated in the past. Advantages of this procedure are: to provide direct comparison of performance of laboratories taking part in different schemes, to provide equivalence of assessment of laboratory performance necessary to establish mutual recognition agreements, and to demonstrate the fitness for purpose of results from participants.Presented at the Eurachem PT Workshop September 2005, Portorož, Slovenia     

Is the assessment of interlaboratory comparison results for a small number of tests and limited number of participants reliable and rational?

Tests and/or test items can sometimes be expensive, unique, or only performed in a few laboratories. There can be cases where assigned values are unknown, there is no information, or only poor information on the probability density function attributed to the test result. Sometimes there are neither reference materials nor the ability to establish consensus values due to a lack of experts. It can be impossible to repeat a test on the same item because it is destroyed during the test itself, or the homogeneity of tested items is unknown and no criteria can be established. Specified technical requirements concerning proficiency testing and interlaboratory comparison schemes are generally not applicable in this situation. However, interlaboratory comparison could allow laboratories to have more confidence in their results. The present paper discusses three statistical methods of assessing interlaboratory comparison results obtained in such conditions. Two methods are based on an assigned value determined from participant results through robust analysis. The third is based on the compatibility of results assessed using the ζ parameter. This paper focuses on an interlaboratory comparison for two laboratories, each testing three samples. The use of statistical methods turns out to be high risk, particularly in terms of falsely accepting results. Additionally, is shown that methods dedicated to small samples are also not efficient in detecting discrepancies of test results.     

Bias analysis of proficiency testing programme results

A proficiency testing programme might involve a great number of participating laboratories coming from different countries or regions, and normally they analysed the same test materials using their own routine analytical methods. Hence, the results of a proficiency testing programme may contain valuable information which could serve purposes other than just performance evaluation. This study attempted to extract information from the results of a proficiency testing programme for the purposes of educating the participating laboratories as suggested by ISO/IEC 17043. The “bias analysis” approach introduced in this study was based on the statistical model of measurement and the nature of bias in chemical analysis. With this approach, the participating laboratories could estimate the bias associated with different settings of experimental conditions according to the statistics of subset distribution of the reported results from the participating laboratories. This would be useful for them to review the analytical procedures they used and modify their methods if needed. The approach was applied to the analysis of data obtained from a number of past proficiency testing programmes, and the findings were discussed in this paper.     

A pilot study (APMP.QM-P19) for the measurement of melamine in milk powder

A pilot study (APMP.QM-P19) of the Asia Pacific Metrology Program for the determination of melamine at two different levels in milk powder was organized in 2010. It was the first interlaboratory comparison programme in measuring melamine in food matrices for metrology institutes in the region. Melamine was chosen as the target analyte because of the recent crises that happened in 2007 and 2008 had caused substantial concern from consumers, food manufacturers, policy makers on food safety and the food testing communities. The objectives of APMP.QM-P19 were to evaluate the testing capability of participants on the quantification of trace level melamine in milk and milk products and to investigate the degree of equivalence with respect to the assigned values. In total, 19 laboratories (11 metrology institutes and eight food testing institutes) enrolled in the study. Analytical techniques reported by participants included liquid chromatography with ultra-violet and tandem mass spectrometry, gas chromatography with high-resolution mass spectrometry and enzyme-linked immunosorbent assay. Dispersion of analytical data in the two test materials was significant, and normal Gaussian distribution of data was not observed. The median was agreed to be used as the assigned values in the programme. Metrology institutes who employed isotope dilution mass spectrometry technique were found to give more consistent results and higher degree of equivalence than others. The performance of most of the food testing institutes was not satisfactory, and they were encouraged to have a thorough review of their validation procedure and estimation of uncertainty.     

Interlaboratory comparison of test results of an ordinal or nominal binary property: analysis of variation

Applications of a new statistical method Ordinal Analysis of Variance (ORDANOVA) for interlaboratory comparisons of measurement or test results of semi-quantitative (ordinal) and qualitative (binary) properties are discussed. ORDANOVA can be helpful for validation of measurement or test methods, proficiency testing of laboratories, development of reference materials with certified semi-quantitative and qualitative properties, that is, probably in every field where ANOVA is applied for quantitative properties. A statistics and criteria are proposed for performance assessment of laboratories active in semi-quantitative and qualitative testing and for other purposes of statistical analysis of such test results.     

An error model for evaluation and interpretation of proficiency testing

Data from proficiency testing can be used to increase our knowledge of the performance of populations of laboratories, individual laboratories and different measurement methods. To support the evaluation and interpretation of results from proficiency testing an error model containing different random and systematic components is presented. From a single round of a proficiency testing scheme the total variation in a population of laboratories can be estimated. With results from several rounds the random variation can be separated into a laboratory and time component and for individual laboratories it is then also possible to evaluate stability and bias in relation to the population mean. By comparing results from laboratories using different methods systematic differences between methods may be indicated. By using results from several rounds a systematic difference can be partitioned into two components: a common systematic difference, possibly depending on the level, and a sample-specific component. It is essential to distinguish between these two components as the former may be eliminated by a correction while the latter must be treated as a random component in the evaluation of uncertainty. Received: 20 November 2000 Accepted: 3 January 2001     

Interlinkages and recognition of proficiency-testing schemes

 Because proficiency testing (PT) is increasingly used for the accreditation of testing laboratories and as a tool for backing up existing multilateral recognition arrangements between accreditation bodies, the question of performance and comparability of the proficiency-test providers arises. In this paper different approaches to assess the equivalence of European PT schemes and the competence of their providers are presented. As a first step a workshop is proposed to agree on a pilot study. The final aim is to create confidence in the existing PT schemes in Europe and to use them as common European tools.     

Tunisian traceability system

The concept of metrology first appeared in Tunisia towards 1909. At the end of the 1990s, bodies for evaluating conformity of measurement at different levels have been instituted to meet calibration and testing needs of the national industry. These bodies were divided into three categories: Class A where we find mainly the Central Laboratory for Analysis and Testing LCAE and the National Defence Laboratory DEFNAT; these two laboratories are in charge of the technological upgrading of the other bodies of Class B which, in turn, would transfer their knowledge to the industries, the latter constituting Class C. Nowadays, the accreditation of Tunisian laboratories at the international level by recognized reference foreign bodies and participation in the European proficiency network enabled the national laboratories of Tunisia to establish the degree of equivalence between their measurement results and those of other foreign laboratories.     

A reliable two-dimensional gel electrophoresis procedure for separating neural proteins

A facile two-dimensional gel electrophoresis procedure has been developed for the analysis of neural tissue proteins which eliminates the serious problems associated with protein insolubility at the point of sample application onto polymerized first-dimension isoelectric focusing gels. This was accomplished by combining the methods of two previously published procedures. Our procedure provides an alternative method to the complex gel systems often employed for less soluble proteins, and yields very reproducible, high resolution separations. This procedure, which is in routine use in our laboratories for the analysis of total proteins extracted from retina and brain, produces protein patterns that are easily compared using both visual and computer-assisted image analysis techniques. Presented here are the results of a set of experiments designed to identify proteins unique to retina. This procedure should be useful to investigators studying protein changes resulting from genetic mutation, development, drug treatment or disease, in neural tissue as well as in virtually all other tissues.     

Laboratory effects models for interlaboratory comparisons

The statistical analysis of results from inter-laboratory comparisons (for example Key Comparisons, or Supplemental Comparisons) produces an estimate of the measurand (reference value) and statements of equivalence of the results from the participating laboratories. Methods to estimate the reference value have been proposed that rest on the idea of finding a so-called consistent subset of laboratories, that is, eliminating allegedly outlying participants. We propose an alternative statistical model that accommodates all participant data and incorporates the dispersion of the measurement values obtained by different laboratories into the total uncertainty of the various estimates. This model recognizes the fact that the dispersion of values between laboratories often is substantially larger than the measurement uncertainties provided by the participating laboratories. We illustrate the methods on data from key comparison CCQM–K25.     

Definition of assigned values for proficiency tests in water analysis

The definition of an assigned value is usually achieved by calculating mean values from the data (with different methods) or by designating reference laboratories. Neither method is completely satisfactory. In this paper a new method is presented for the definition of the assigned value for spiked samples with an unknown content of the analyte in the matrix. The method consists of two parts. The first is the estimation of the assigned values from the spiked amounts and the content in the matrix, based on the results of reference laboratories. The other is the designation of these reference laboratories by comparing their results with the assigned values. Because each of these parts requires the other, an iterative procedure is necessary. As an example, the results of a proficiency test for the analysis of copper in wastewater are used to compare the calculated values with those from other methods, e.g., the Huber estimation. Received: 25 September 2000 Accepted: 9 December 2000