Local comparability of proficiency testing results: determination of concrete slump and compressive strength values

A proficiency testing (PT) scheme is developed for comparability assessment of results of concrete slump and compressive strength determination. The scheme is based on preparing of a test portion/sample of a concrete in-house reference material (IHRM) at a reference laboratory (RL) in the same conditions for every PT participant. Therefore, in this scheme IHRM instability is not relevant as a source of measurement/test uncertainty, while intra- and between-samples inhomogeneity parameters are evaluated using the results of RL testing of the samples taken at the beginning, the middle and the end of the PT experiment. The IHRM assigned slump and compressive strength values are calculated as averaged RL results. Their uncertainties include the measurement/test uncertainty components and the components arising from the material inhomogeneity. The test results of 25 PT participants were compared with the IHRM assigned values taking into account both the uncertainties of the assigned values and the measurement/test uncertainties of the participants. Since traceability of the IHRM assigned values to the international measurement standards and SI units cannot be stated, local comparability of the results is assessed. It is shown, that comparability of the slump and compressive strength determination results is satisfactory, while uncertainty evaluation for slump results requires additional efforts.     

Proficiency testing in food microbiology: experience from implementation of ISO/IEC 17043 and ISO/TS 22117

The proficiency testing (PT) scheme ??AQUA?? for food microbiology was organised by the Istituto Zooprofilattico Sperimentale delle Venezie (IZSVe) according to ISO/IEC 17043 and ISO/TS 22117. This paper describes the IZSVe experience on the application of the above-mentioned standards for the PTs, with focus on the Enterobacteriaceae enumeration one. Freeze-dried food matrices contaminated with American Type Culture Collection bacterial strains were used as test samples for each microbiological PT organised by IZSVe. The sample homogeneity and stability were verified prior to distribution to participants and throughout the PT, respectively. The participating laboratories analysed samples using their routine methods, and results were transmitted to IZSVe. Data and methods used by each participating laboratory were analysed in order to evaluate the laboratory performance. With reference to the Enterobacteriaceae PT, the test samples were homogeneous and stable. In addition, most laboratory results were obtained using equivalent test methods. Statistical approaches applied to analyse data generated from all participating laboratories revealed similar outcomes as no significant outlying count and only 5?% of unacceptable results were observed. Finally, the z-score, with the standard deviation that does not vary from round toround, was applied to compare and to evaluate the performance of each laboratory over time highlighting possible persistent trends over several rounds.     

Preparation of in-house reference material of benzylpenicillin in milk and results of a Brazilian proficiency testing scheme

Two milk test materials containing benzylpenicillin and a benzylpenicillin-free material were prepared and used for the operation of a Brazilian proficiency testing (PT) scheme according to the requirements of ISO/IEC 17043. The PT scheme was designed to include laboratories in charge of confirmatory analysis as well as screening analysis. Sets of sample vials, benzylpenicillin-free and spiked items containing 0.2 g of lyophilized milk, were distributed to each participant. Of 20 participants who reported screening data, 18 accomplished satisfying results, providing evidence of the capability of Brazilian laboratories to produce reliable qualitative information. The assigned value (robust average calculated from 6 results) and its uncertainty were (8.28 ± 0.52) μg/L. Participants’ performance was evaluated using z-scores. A small number of participants were able to report quantitative benzylpenicillin results, and consequently care should be taken in the interpretation of the laboratory’s performance, considering the statistical nature of z-scores. Long-term stability testing on remaining PT items showed that a stable benzylpenicillin quality control material was produced and can be used for internal quality control or validation purposes.     

Comparability and compatibility of proficiency testing results in schemes with a limited number of participants

Comparability and compatibility of proficiency testing (PT) results are discussed for schemes with a limited number of participants (less than 20–30) based on the use of reference materials (RMs) as test items. Since PT results are a kind of measurement/analysis/test result, their comparability is a property conditioned by traceability to measurement standards applied in the measurement process. At the same time, metrological traceability of the certified value of the RM (sent to PT participants as test items) is also important, since the PT results are compared with the RM certified value. The RM position in the calibration hierarchy of measurement standards sets the degree of comparability for PT results, which can be assessed in the scheme. However, this assessment is influenced by commutability (adequacy or match) of the matrix RM used for PT and routine samples. Compatibility of PT results is a characteristic of the collective (group) performance of the laboratories participating in PT that can be expressed as closeness of the distribution of the PT results to the distribution of the RM data. Achieving quality-of-measurement/analysis/test results in the framework of the concept “tested once, accepted everywhere” requires both comparability and compatibility of the test results.     

Designs of experiment for proficiency testing with a limited number of participants

Metrological designs of experiment for proficiency testing (PT) with a limited number of participants are discussed. The designs are based on development of in-house reference materials (IHRMs) with traceable assigned values and fit-for-purpose uncertainties, used in a PT scheme as the measurement standards. When adequate certified reference materials (CRMs) are available, a comparative approach for IHRM development, using simultaneous analysis of IHRM and CRM test portions in pairs, is proposed for PT objectives. In the case where adequate CRMs are not available – for example, in the field of analysis of unstable aqueous systems – the function of the measurement standard for PT can be fulfilled by a synthetic IHRM prepared gravimetrically using non-adequate, non-aqueous CRMs or pure substances. The CRM or the pure substance is used as a spike for fortification of a natural water sample, while the natural water sample is used as a working IHRM for the spike determination. In this case the traceability chain is longer, since two IHRMs are added for one quantity determination, but it remains unbroken.Presented at the Second International Conference on Metrology –Trends and Applications in Calibration and Testing Laboratories, 4–6 November 2003, Eilat, Israel     

On the use of consensus means as assigned values

An assigned value can be derived by using either a consensus mean or a reference value. It is up to the proficiency testing provider to decide whether the consensus mean or the reference value might be used. If the consensus mean is used, it must be ensured that there really exists a consensus. This requirement is fulfilled if the participants?? results are not biased on average, and there is an agreement between the results with a precision which is fit for the intended use. The best way to avoid potentially ??biased?? assigned values is to use reference values thus ensuring that the assigned value is close to the ??true?? value.     

Evaluation of the use of consensus values in proficiency testing programmes

Proficiency testing (PT) is an essential tool used by laboratory accreditation bodies to assess the competency of laboratories. Because of limited resources of PT providers or for other reasons, the assigned reference value used in the calculation of z-score values has usually been derived from some sort of consensus value obtained by central tendency estimators such as the arithmetic mean or robust mean. However, if the assigned reference value deviates significantly from the ‘true value’ of the analyte in the test material, laboratories’ performance will be evaluated incorrectly. This paper evaluates the use of consensus values in proficiency testing programmes using the Monte Carlo simulation technique. The results indicated that the deviation of the assigned value from the true value could be as large as 40%, depending on the parameters of the proficiency testing programmes under investigation such as sample homogeneity, number of participant laboratories, concentration level, method precision and laboratory bias. To study how these parameters affect the degree of discrepancy between the consensus value and the true value, a fractional factorial design was also applied. The findings indicate that the number of participating laboratories and the distribution of laboratory bias were the prime two factors affecting the deviation of the consensus value from the true value.     

On the value of recent microscopic methods of identification used for organic substances

The value of recent methods of identification of organic substances by means of the micro melting point apparatus is investigated, using 3 groups of closely related substances (benzoic acid derivatives, barbituric acid derivatives, and sulphonamids).The experiments show that a combination of the melting point determination with an observation of sublimation and conversion during heating and, moreover, with the observation of the conversion of crystallised melting drops etc. very markedly increases the value of the determination.The preparation of derivatives from quantities below l mg can be performed more rapidly and more simply than the preparation of larger amounts. The melting point determination of the derivatives is of the greatest value for the identification of the respective substances.On the other hand, the melting point determination on eutectic mixtures of the substance in question with a test substance is found to be of much lower value as a method of identification, if there is a possibility for the occurrence of a number of closely related substances having the same melting point ; such substances often have the same eutectic melting point with the same test substance.The determination of the index of refraction of molten substances (i.e., the determination of the temperature interval within which the molten mass has the same index of refraction as a glass powder of a known index of refraction) is an excellent method of identification, since the scattering of the temperature intervals found for related substances is of the same magnitude, and for not related substances is considerably higher, than the scattering of their melting points.     

A review of proficiency testing scheme in Republic of Korea: in the field of water analysis

For regulatory purposes, it is often required to check the quality of environmental chemical analyses by routine proficiency testing (PT). In this study, the PT scheme of environmental laboratories in Republic of Korea has been reviewed. Especially, real water-based reference materials (RMs) were prepared and distributed for PT schemes. The consensus values from the PT were calculated by robust statistics with its standard uncertainties. The relative bias between the reference values and the consensus values was used as one of the indicators for the determination of the assigned values. The relationship between the performances of the laboratories, expressed by Z scores, and the concentration levels of the RMs were also revealed.     

Comparability of analytical results obtained in proficiency testing based on a metrological approach

A “yes–no” type of criterion is proposed for the assessment of comparability of proficiency testing (PT) results when the PT scheme is based on a metrological approach, i.e. on the use of a reference material as the test sample, etc. The criterion tests a null hypothesis concerning the insignificance of a bias of the mean of the results from a traceable value certified in the reference material used for the PT. Reliability of such assessment is determined by the probabilities of not rejecting the null hypothesis when it is true, and rejecting it when it is false (the alternative hypothesis is true). It is shown that a number of chemical, metrological and statistical reasons should be taken into account for careful formulation of the hypotheses, enabling the avoidance of an erroneous assessment of the comparability. The criterion can be helpful for PT providers and laboratory accreditation bodies in analysis of PT results.     

玩具油漆涂层中有机锡检测能力验证样品研制及应用分析

制备了用于可迁移有机锡检测能力验证的玩具油漆涂层样品,并开展了相关测定能力验证计划,采用迭代法、四分位法和敏感分析法3种不同统计方法对23家实验室数据进行统计分析,并对不满意结果的原因进行了分析。结果表明所研制样品的均匀性和稳定性良好,能满足能力验证要求。不同统计方法对实验室能力评定结果存在差异,但指定值差异很小,而能力评定标准差差异较大,运用卡方检验能选择出更合理的统计方法。     

Providing a traceable assigned value in a proficiency testing programme on the determination of benzoic acid in orange juice

A proficiency testing (PT) programme on the determination of benzoic acid in orange juice was organised by the Chemical Metrology Laboratory, Health Sciences Authority, Singapore, for the local food testing laboratories. The PT programme used a metrologically traceable assigned value determined by isotope dilution mass spectrometry to assess the performance of the participating laboratories. This paper discusses the reasons and approach to applying metrological principles to determine the assigned value of a PT programme. The procedure undertaken by the Laboratory to prepare a batch of PT sample, conduct homogeneity and stability testings, as well as value assignment, is presented. The discussions also include the estimation of the measurement uncertainties arising from the characterisation, testing for homogeneity and testing for stability of the PT sample. The Coordinator of the PT programme used the assigned value and its associated measurement uncertainty, together with those of the participating laboratories to assess their performances. Statistical tools applied in the performance evaluations are also presented.     

Preparation of reference material for organochlorine pesticides in a herbal matrix

The development of reference material for four organochlorine pesticides, namely hexachlorobenzene and three isomers of hexachlorocyclohexane (alpha-hexachlorocyclohexane, beta-hexachlorocyclohexane and gamma-hexachlorocyclohexane), in a ginseng root sample is presented. Raw materials (Panax ginseng) were purchased from a local market and confirmed to contain certain levels of incurred organochlorine pesticide residues by a validated gas chromatography-mass selective detection method. A total of more than 300 bottles each containing 25 g of samples were prepared after the materials had been freeze-dried, milled and thoroughly mixed. The homogeneity and stability of samples from randomly selected bottles were verified and the reference values were characterized using a highly precise isotope dilution gas chromatography-mass spectrometry (ID-GCMS) method that was recently developed by our laboratory. The purity of standard organochlorine chemicals was determined against certified reference materials to establish the accuracy of the ID-GCMS analysis. The concentrations (+/- expanded uncertainty) of hexachlorobenzene, alpha-hexachlorocyclohexane, beta-hexachlorocyclohexane and gamma-hexachlorocyclohexane in the reference material were 0.198 +/- 0.015, 0.450 +/- 0.022, 0.213 +/- 0.011 and 0.370 +/- 0.032 mg kg(-1), respectively. A portion (70 bottles) of the samples was also used in a proficiency testing (PT) scheme for assessing the testing capabilities of field laboratories. The consensus mean values of the PT obtained from the 70 participants were on the same order but deviated by -2.7 to -14.1% from those of the assigned reference values. Because of the wide spread of participants' data (relative standard deviation ranging from 44 to 56%), the PT results were not included in the calculation of the assigned values of the reference materials. The materials served as suitable reference materials to ascertain the quality control and validation processes for the determination of organochlorine pesticides in herbal matrices.     

Performance evaluation for proficiency testing with a limited number of participants

Proficiency testing (PT) is an essential tool for laboratories to assess their competency. Also, participation in PT has become one of the mandatory requirements for laboratory to seek accreditation according to ISO/IEC 17025. For this reason, the effectiveness of performance evaluation by PT scheme is of great concern for the participants and for accreditation bodies as well. In practice, owing to unavailability of other appropriate alternatives, PT scheme providers may have to choose using consensus values to evaluate the performance of participants. However, such consensus values approach was not recommended by relevant international guidelines for PT schemes with limited number of participants. With the use of Monte Carlo simulation technique, this study attempted to investigate the effectiveness of using consensus values for performance evaluation in PT schemes with limited number of participants. The simulation process was schemed according to the statistical model provided by ISO 5725-1 for laboratory measurement results, which covered components like method bias, laboratory bias, and measurement precision. The effectiveness of the consensus value approach was expressed as the percentage of participants in a simulation run could get the same evaluation result, either satisfactory or unsatisfactory, against the “true value.” The findings indicated that the number of participants, choice of consensus values, mass fraction of analyte, method bias, laboratory bias, and measurement repeatability of participating laboratories would all affect the effectiveness of the consensus value approach but at different extent. However, under certain circumstances, use of consensus value could still be considered as an acceptable approach for performance evaluation even the number of participants was limited. Some of the findings were further verified using real data from PT schemes where appropriate certified reference materials or reliable reference values were available.     

Development of a proficiency testing scheme for a limited number of participants in the field of natural water analysis

A proficiency testing (PT) scheme was developed for a limited number of analytical laboratories participating in the analysis of natural water in Israel. Three fit-for-intended-use reference materials (RMs) were prepared for a pilot PT from natural water: RM-blank and two in-house reference materials (IHRMs) with seven analytes fortified to achieve different levels of analyte concentrations. The measurands for the PT participants were the traceable spike values certified in the IHRMs, i.e., added mass concentrations of the analytes. The RM-blank and IHRMs were found to be homogeneous and stable over 11 days, the time necessary for the experimental part of the PT. The RMs were distributed to the PT participants as unknown test items similar to routine samples. The test results were evaluated using their deviations from the IHRM-certified values. Eight Israeli laboratories took part in the interlaboratory comparison. Individual laboratory performance and metrological compatibility of the PT results of the participants, as a local group of laboratories, were evaluated for every analyte.     

Proficiency test of pH,conductivity and dissolved oxygen concentration field measurements in river water

In 2013, Proftest SYKE organised the first proficiency test (PT) in Finland for field measurements of temperature, conductivity, dissolved oxygen concentration, oxygen saturation and pH value in river water. The aim was to pilot the organisation of an in situ proficiency test—particularly, how to select the test location—and how to assess the homogeneity and stability of the measurement site and the water to be tested. The focus was also to evaluate the suitability of the common field sensors used for water analysis, as well as the comparability of the results between the instruments under field conditions. The overall application of quality assurance procedures was also surveyed. This paper deals with the results, findings and recommendations for the measurement of pH, conductivity and dissolved oxygen concentration. In total, nine participants with 16 sensors took part in the proficiency test. For the evaluation of the performance of each participant, z scores were calculated allowing 3 % to 8 % deviation from the assigned value. The standard deviation of the participant’s results was lower than organiser expected, and 80 % of the results for pH, 79 % for conductivity and 69 % for dissolved oxygen concentration were regarded as satisfactory. According to the results, the most challenging measurement was for dissolved oxygen with a Clark cell-type measurement principle based on electrochemical reaction. All sensors tested in the PT were less than 5 years, old and they were calibrated according to the manufacturer’s instructions. None of the participants had estimated measurement uncertainty for their sensor measurements. In addition, internal and external quality assurance protocols were usually lacking.     

Establishment of an assigned value and its uncertainty for tumour markers in proficiency testing in China

Proficiency testing (PT) is an important aspect of clinical laboratory quality control for tumour markers. To evaluate the performance of an individual laboratory, the assigned value and the standard uncertainty of each tumour marker must be determined. However, no data about the standard uncertainty of the assigned value of any analyte in PT is available in China. The aim of this paper is to introduce a new approach for establishment of assigned value and its standard uncertainty for tumour markers. Five sets of control materials at different concentrations were assigned to each laboratory that participated in PT for tumour markers in 2012. We collected the data from each laboratory using the Clinet-EQA reporting system V1.5 and Clinet-EQA evaluation system V1.0 and developed a software program to calculate the assigned value and its uncertainty according to ISO 13528 (Statistical methods for use in proficiency testing by interlaboratory comparisons. ISO 13528. International Standards Organisation, Geneva, 2005). The assigned value and inter-laboratory CV calculated by the new approach was compared with that obtained by the current approach used in PT in our country. No significant difference between the two outcomes was found in our study. Most assigned values had a standard uncertainty that can meet the criterion: $ u_{X} \le \, 0.3\hat{\sigma } $ u X ≤ 0.3 σ ? , which meant the new approach for establishment of the assigned value was acceptable. It is expected that the new approach to determination of the assigned value and its standard uncertainty for PT for tumour markers in China will enable participants to evaluate their measurement uncertainty.     

The role of proficiency testing in the detection and resolution of calibration bias in the LeadCare® blood lead analyzer; limitations of peer-group assessment

Following implementation of the CLIA ‘88 laboratory regulations, the primary role of proficiency testing (external quality assessment, PT) in the U.S. has been widely viewed as one of assuring regulatory compliance. PT can also be an effective tool for detecting widespread analytical problems, subject to limitations based on the method of PT assigned value determination. A recent case study describes the role of two PT programs in detecting and resolving a calibration bias in the LeadCare blood lead analyzer, and illustrates the limitations of peer-group target determination in fulfilling that PT role.     

Application of the FAPAS protocol to a proficiency testing on Pb and Cd extracted from glazed ceramic

The proficiency-testing scheme (CNCA-07-06) concerning lead and cadmium determination extracted from glazed ceramic surfaces involved 43 Chinese laboratories as participants, and has been well implemented. All the test materials passed the statistical homogeneity tests, and were deemed suitable for distribution. Reference values were determined from the obtained results and the performance of participants was based on z-scores. As a result, the majority of participants obtained satisfactory z-scores and laboratories that need to revise their procedures were singled out. Overall, about 8.5% of all results submitted in this PT round were unsatisfactory in terms of z-scores (22 out of 258 total data sets submitted), while the percentage of doubtful z-scores was 4.6% (12 out of 258 total data sets). Besides, the factors that might be associated with unsatisfactory results were also evaluated and suggestions were made to help improve the performance of laboratories in this PT.     

Proficiency testing of pH and electrolytic conductivity measurements of groundwater: a case study of the difference between consensus and metrologically traceable values

A proficiency testing (PT) scheme was designed for measurements of pH and electrolytic conductivity (EC) of groundwater in Israel. A groundwater sample was used for preparation of test items fit-for-intended-use. The interlaboratory experiment was organized taking into account changes of the test item properties which occurred after sampling due to presence of CO₂ in the groundwater. A total of 34 PT participants reported pH measurements results, while 29 of them reported results of EC measurements. The pH results were normally distributed. However, a significant bias of the pH consensus value of the PT participants from the metrologically traceable certified value (obtained by the National Physical Laboratory of Israel) was detected. This bias was caused by the drift of the measurement response due to water stirring and CO₂ degassing from the sample during the measurement process. The bias of the EC consensus value from the metrologically traceable one was negligible, in spite of an abnormal distribution of the EC results of the PT participants.