An adaptive strategy for selecting representative calibration samples in the continuous wavelet domain for near-infrared spectral analysis

Sample selection is often used to improve the cost-effectiveness of near-infrared (NIR) spectral analysis. When raw NIR spectra are used, however, it is not easy to select appropriate samples, because of background interference and noise. In this paper, a novel adaptive strategy based on selection of representative NIR spectra in the continuous wavelet transform (CWT) domain is described. After pretreatment with the CWT, an extension of the Kennard–Stone (EKS) algorithm was used to adaptively select the most representative NIR spectra, which were then submitted to expensive chemical measurement and multivariate calibration. With the samples selected, a PLS model was finally built for prediction. It is of great interest to find that selection of representative samples in the CWT domain, rather than raw spectra, not only effectively eliminates background interference and noise but also further reduces the number of samples required for a good calibration, resulting in a high-quality regression model that is similar to the model obtained by use of all the samples. The results indicate that the proposed method can effectively enhance the cost-effectiveness of NIR spectral analysis. The strategy proposed here can also be applied to different analytical data for multivariate calibration.     

A new hybrid strategy for constructing a robust calibration model for near-infrared spectral analysis

A new hybrid algorithm is proposed for construction of a high-quality calibration model for near-infrared (NIR) spectra that is robust against both spectral interference (including background and noise) and multiple outliers. The algorithm is a combination of continuous wavelet transform (CWT) and a modified iterative reweighted PLS (mIRPLS) procedure. In the proposed algorithm the spectral interference is filtered by CWT at the first stage then mIRPLS is proposed to detect the multiple outliers in the CWT domain. Compared with the original IRPLS method, mIRPLS does not need to adjust variable parameters to achieve optimum calibration results, which makes it very convenient to perform in practice. The final PLS model is constructed robustly because both the spectral interference and multiple outliers are eliminated. In order to validate the effectiveness and universality of the algorithm, it was applied to two different sets of NIR spectra. The results indicate that the proposed strategy can greatly enhance the robustness and predictive ability of NIR spectral analysis.     

Removal of major interference sources in aqueous near-infrared spectroscopy techniques

This work describes a hybrid procedure for eliminating major interference sources in aqueous near-infrared (NIR) spectra, that include aqueous influence, noise, and systemic variations irrelevant to concentration. The scheme consists of two parts: extension of wavelet prism (WPe) and orthogonal signal correction (OSC). First, WPe is employed to remove variations due to aqueous absorbance and noise; then OSC is applied to remove systemic spectral variations irrelevant to concentration. Although water possesses strong absorption bands that overshadow and overlap the absorption bands of analytes, along with noise and systematic interference, successful calibration models can be generated by employing the method proposed here. We show that the elimination of major interference sources from the aqueous NIR spectra results in a substantial improvement in the precision of prediction, and reduces the required number of PLS components in the model. In addition, the strategy proposed here can be applied to various analytical data for quantitative purposes as well.     

Removal of interference signals due to water from in vivo near-infrared (NIR) spectra of blood glucose by region orthogonal signal correction (ROSC).

A novel chemometric method, region orthogonal signal correction (ROSC), is proposed and applied to pretreat near-infrared (NIR) spectra of blood glucose measured in vivo. Water is the most serious interference component in such kinds of noninvasive measurements, because it shows very high absorbance in the spectra. In the present study, the spectra of blood glucose in the range of 1212 - 1889 nm are used, in which the absorption of water around 1440 nm is very high. ROSC aims at removing the interference signal due to water from the spectra by selecting a set of spectra with a special region of 1404 - 1454 nm that mainly contain information about the variation of the interference component, water, and calculating the orthogonal components to the concentrations of glucose that will be removed. The difference between ROSC and orthogonal signal correction (OSC) is that ROSC uses a special region of spectra for the estimation of scores and loading weights of orthogonal components to pretreat the spectra in other regions, while OSC only uses one fixed region of spectra to calculate loadings, scores and weights of OSC components and removes the OSC components in the same region. A clear advantage of ROSC is that it is more interpretable than OSC, because one can select a spectral region to remove the variation of a special component such as water. Another chemometric method, moving window partial least squares (MWPLSR), is also used to select informative regions of glucose from the NIR spectra of blood glucose measured in vivo, leading to improved PLS models. Results of the application of ROSC demonstrate that ROSC-pretreated spectra including the whole spectral region of 1212 - 1889 nm or an informative region of 1600- 1730 nm selected by MWPLSR provide very good performance of the PLS models. Especially, the later region yields a model with RMSECV of 15.8911 mg/dL for four PLS components. ROSC is a potential chemometric technique in the pretreatment of various spectra.     

Influence of data pre-processing on the quantitative determination of the ash content and lipids in roasted coffee by near infrared spectroscopy

In near-infrared (NIR) measurements, some physical features of the sample can be responsible for effects like light scattering, which lead to systematic variations unrelated to the studied responses. These errors can disturb the robustness and reliability of multivariate calibration models. Several mathematical treatments are usually applied to remove systematic noise in data, being the most common derivation, standard normal variate (SNV) and multiplicative scatter correction (MSC). New mathematical treatments, such as orthogonal signal correction (OSC) and direct orthogonal signal correction (DOSC), have been developed to minimize the variability unrelated to the response in spectral data. In this work, these two new pre-processing methods were applied to a set of roasted coffee NIR spectra. A separate calibration model was developed to quantify the ash content and lipids in roasted coffee samples by PLS regression. The results provided by these correction methods were compared to those obtained with the original data and the data corrected by derivation, SNV and MSC. For both responses, OSC and DOSC treatments gave PLS calibration models with improved prediction abilities (4.9 and 3.3% RMSEP with corrected data versus 7.1 and 8.3% RMSEP with original data, respectively).     

A Background and noise elimination method for quantitative calibration of near infrared spectra

A new hybrid algorithm is proposed to eliminate the varying background and noise simultaneously for multivariate calibration of near infrared (NIR) spectral signals. The method is based on the use of multi-resolution, which is one of the main advantages provided by wavelet transform. The signals are firstly split into different frequency components, which keep the same data points of the original signals. In conjunction with a modified uninformative variable elimination (mUVE) criterion, the new method can be used to remove the low-frequency varying background and the high-frequency noise simultaneously. The method is successfully applied to simulated spectral data set and experimental NIR spectral data, resulting in more parsimonious multivariate models with higher precision. In addition, the proposed strategy can be applied to other spectral signals as well.     

Comparison of near-infrared and mid-infrared spectroscopy for the determination of distillation property of kerosene

Near-infrared (NIR) and mid-infrared (MIR) spectroscopy have been compared and evaluated for the determination of the distillation property of kerosene with the use of partial least squares (PLS) regression. Since kerosene is a complex mixture of similar hydrocarbons, both spectroscopic methods will be best evaluated with this complex sample matrix. PLS calibration models for each percent recovery temperature have been developed by using both NIR and MIR spectra without spectral pretreatment. Both methods have shown good correlation with the corresponding reference method, however NIR provided better calibration performance over MIR. To rationalize the improved calibration performance of NIR, spectra of the same kerosene sample were continuously collected and the corresponding spectral reproducibility was evaluated. The greater spectral reproducibility including signal-to-noise ratio of NIR led to the improved calibration performance, even though MIR spectroscopy provided more qualitative spectral information. The reproducibility of measurement, signal-to-noise ratio, and richness of qualitative information should be simultaneously considered for proper selection of a spectroscopic method for quantitative analysis.     

Wavelet unfolded partial least squares for near-infrared spectral quantitative analysis of blood and tobacco powder samples

Continuous wavelet transform (CWT) has been shown to be a high-performance signal processing technique in multivariate calibration. However, the signal processed by CWT with a specific wavelet may account for only a part of the information. To effectively utilize more abundant information contained in analytical signals, a method, named as wavelet unfolded partial least squares (WUPLS), was proposed. In the approach, the measured dataset is firstly extended by CWT with different wavelets, and then partial least squares (PLS) is employed to develop the quantitative model between the extended dataset and the target values. In order to select the representative wavelets, principal component analysis (PCA) is used to investigate the distribution of the signals obtained by CWT with different wavelets. The performance of the method was tested with blood and tobacco powder samples. Compared with the results obtained by PLS methods, the WUPLS method combined with signal processing techniques is proven to be a promising tool for improving the near-infrared (NIR) spectral analysis of complex samples.     

Simultaneous wavelength selection and outlier detection in multivariate regression of near-infrared spectra.

Near-infrared (NIR) spectrometry will present a more promising tool for quantitative measurement if the robustness and predictive ability of the partial least square (PLS) model are improved. In order to achieve the purpose, we present a new algorithm for simultaneous wavelength selection and outlier detection; at the same time, the problems of background and noise in multivariate calibration are also solved. The strategy is a combination of continuous wavelet transform (CWT) and modified iterative predictors and objects weighting PLS (mIPOW-PLS). CWT is performed as a pretreatment tool for eliminating background and noise synchronously; then, mIPOW-PLS is proposed to remove both the useless wavelengths and the multiple outliers in CWT domain. After pretreatment with CWT-mIPOW-PLS, a PLS model is built finally for prediction. The results indicate that the combination of CWT and mIPOW-PLS produces robust and parsimonious regression models with very few wavelengths.     

应用光谱技术无损检测油菜叶片中乙酰乳酸合成酶

应用可见/近红外光谱技术实现了油菜叶片中乙酰乳酸合成酶(ALS)的快速无损检测.对99个油菜样本进行光谱扫描,经过平滑、变量标准化、一阶求导等预处理后,应用偏最小二乘法(PLS)建立了ALS的预测模型.同时提取有效特征变量,作为反向传输人工神经网络(BPNN)和最小二乘-支持向量机(LS-SVM)的输入值,并建立相应的模型.用66个样本建模,33个样本验证.结果表明,LS-SVM模型能够获得最优的预测结果,预测集样本的相关系数(r)、预测标准差(RMSEP)和偏差(Bias)分别为0.998、 0.715和0.079,获得了满意的预测精度.结果表明,应用可见/近红外光谱技术结合LS-SVM检测油菜中乙酰乳酸合成酶是可行的,并能获得满意的预测精度,为进一步应用光谱技术进行油菜生长状况的大田监测奠定了基础.     

NIR analysis of pharmaceutical samples without reference data: improving the calibration

Using an appropriate set of samples to construct the calibration set is crucial with a view to ensuring accurate multivariate calibration of NIR spectroscopic data. In this work, we developed and optimized a new methodology for incorporating physical variability in pharmaceutical production based on the NIR spectrum for the process. Such a spectrum contains the spectral changes caused by each treatment applied to the component mixture during the production process. The proposed methodology involves adding a set of process spectra (viz. difference spectra between those for production tablets and a laboratory mixture of identical nominal composition) to the set of laboratory samples, which span the wanted concentration range, in order to construct a calibration set incorporating all physical changes undergone by the samples in each step of the production process. The best calibration model among those tested was selected by establishing the influence of spectral pretreatments used to obtain the process spectrum and construct the calibration models, and also by determining the multiplying factor m to be applied to the process spectra in order to ensure incorporation of all variability sources into the calibration model. The specific samples to be included in the calibration set were selected by principal component analysis (PCA). To this end, the new methodology for constructing calibration sets for determining the Active Principle Ingredients (API) and excipients was applied to Irbesartan tablets and validated by application to the API and excipients of paracetamol tablets. The proposed methodology provides simple, robust calibration models for determining the different components of a pharmaceutical formulation.     

Ensemble preprocessing of near-infrared (NIR) spectra for multivariate calibration

Preprocessing of raw near-infrared (NIR) spectral data is indispensable in multivariate calibration when the measured spectra are subject to significant noises, baselines and other undesirable factors. However, due to the lack of sufficient prior information and an incomplete knowledge of the raw data, NIR spectra preprocessing in multivariate calibration is still trial and error. How to select a proper method depends largely on both the nature of the data and the expertise and experience of the practitioners. This might limit the applications of multivariate calibration in many fields, where researchers are not very familiar with the characteristics of many preprocessing methods unique in chemometrics and have difficulties to select the most suitable methods. Another problem is many preprocessing methods, when used alone, might degrade the data in certain aspects or lose some useful information while improving certain qualities of the data. In order to tackle these problems, this paper proposes a new concept of data preprocessing, ensemble preprocessing method, where partial least squares (PLSs) models built on differently preprocessed data are combined by Monte Carlo cross validation (MCCV) stacked regression. Little or no prior information of the data and expertise are required. Moreover, fusion of complementary information obtained by different preprocessing methods often leads to a more stable and accurate calibration model. The investigation of two real data sets has demonstrated the advantages of the proposed method.     

A method for near-infrared spectral calibration of complex plant samples with wavelet transform and elimination of uninformative variables

An algorithm is proposed for extracting relevant information from near-infrared (NIR) spectra for multivariate calibration of routine components in complex plant samples. The algorithm is a combination of wavelet transform (WT) data compression and a procedure for uninformative variable elimination (UVE). After compression of the NIR spectra by WT, the UVE approach is used to eliminate the irrelevant wavelet coefficients. Finally, a calibration model is built from the retained wavelet coefficients to enable prediction. Because irrelevant information can be removed from the spectra used for multivariate calibration, the model based on the extracted relevant features is better than those obtained with full-spectrum data. Both prediction precision and calculation speed are improved.     

Determination of chlorogenic acid in plant samples by using near-infrared spectrum with wavelet transform preprocessing.

By theoretical analysis, it is found that wavelet transform (WT) with a wavelet function can be regarded as a smoothing and a differentiation process, and that the order of differentiation is determined by the vanishing moment, which is an important property of a wavelet function. Therefore, a method based on the continuous wavelet transform (CWT) for removing the background in the near-infrared (NIR) spectrum is proposed, and it is used in the determination of the chlorogenic acid in plant samples as a preprocessing tool for partial least square (PLS) modeling. It is shown that the benefit of the proposed method lies not only in its performance to improve the quality of PLS model and the prediction precision, but also in its simplicity and practicability. It may become a convenient and efficient tool for preprocessing NIR spectral data sets in multivariate calibration.     

基于近红外光谱技术与优化光谱预处理的陈皮产地鉴别研究

采集不同产地陈皮内侧和外侧的近红外光谱,采用不同光谱预处理方法进行预处理,筛选得到最佳光谱预处理方法,结合主成分分析法建立了陈皮产地的鉴别模型.实验发现,陈皮原始光谱中存在明显的基线漂移与背景干扰.使用单一光谱预处理可在一定程度上消除干扰的影响.经标准正态变量变换、多元散射校正、一阶导数、二阶导数与连续小波变换预处理后...     

Rapid Determination of Metabolites in Bio‐fluid Samples by Raman Spectroscopy and Optimum Combinations of Chemometric Methods

The application of Raman spectroscopic techniques combined with multivariate chemometrics signal processing promise new means for the rapid multidimensional analysis of metabolites non‐destructively, with little or no sample preparation and little sensitivity to water. However, Rayleigh scattering, fluorescence and uncontrolled variance present substantial challenges for the accurate quantitative analysis of metabolites at physiological levels in biologically varying samples. Effective strategies include the application of chemometrics pretreatments for reducing Raman spectral interference. However, the arbitrary application of individual or combined pretreatment procedures can significantly alter the outcome of a measurement, thereby complicating spectral analysis. This paper evaluates and compares six signal pretreatment methods for correcting the baseline variances, together with three variable selection methods for eliminating uninformative variables, all within the context of multivariate calibration models based on partial least squares (PLS) regression. Raman spectra of 90 artificial bio‐fluid samples with eight urine metabolites at near‐physiological concentrations were used to test these models. The combination of multiplicative scatter correction (MSC), continuous wavelet transform (CWT), randomization test (RT) and PLS modeling presented the best performance for all the metabolites. The correlation coefficient (R) between predicted and prepared concentration reached as high as 0.96.     

Orthogonal signal correction-partial least squares method for simultaneous spectrophotometric determination of cypermethrin and tetramethrin

The simultaneous determination of cypermethrin and tetramethrin mixtures by using spectrophotometric method is a difficult problem in analytical chemistry, due to spectral interferences. By multivariate calibration methods, such as partial least squares (PLS) regression, it is possible to obtain a model adjusted to the concentration values of the mixtures used in the calibration range. Orthogonal signal correction (OSC) is a preprocessing technique used for removing the information unrelated to the target variables based on constrained principal component analysis. OSC is a suitable preprocessing method for partial least squares calibration of mixtures without loss of prediction capacity using spectrophotometric method. In this study, the calibration model is based on absorption spectra in the 200-350 nm range for 25 different mixtures of cypermethrin and tetramethrin. Calibration matrices were containing 0.1-12.9 and 0.1-13.8 microg mL(-1) for cypermethrin and tetramethrin, respectively. The RMSEP for cypermethrin and tetramethrin with OSC and without OSC were 0.0884, 0.0614 and 0.2915, 0.2309, respectively. This procedure allows the simultaneous determination of cypermethrin and tetramethrin in synthetic and real samples good reliability of the determination was proved.     

Near-infrared calibration transfer based on spectral regression

A calibration transfer method for near-infrared (NIR) spectra based on spectral regression is proposed. Spectral regression method can reveal low dimensional manifold structure in high dimensional spectroscopic data and is suitable to transfer the NIR spectra of different instruments. A comparative study of the proposed method and piecewise direct standardization (PDS) for standardization on two benchmark NIR data sets is presented. Experimental results show that spectral regression method outperforms PDS and is quite competitive with PDS with background correction. When the standardization subset has sufficient samples, spectral regression method exhibits excellent performance.     

Maintaining the predictive abilities of multivariate calibration models by spectral space transformation

In quantitative on-line/in-line monitoring of chemical and bio-chemical processes using spectroscopic instruments, multivariate calibration models are indispensable for the extraction of chemical information from complex spectroscopic measurements. The development of reliable multivariate calibration models is generally time-consuming and costly. Therefore, once a reliable multivariate calibration model is established, it is expected to be used for an extended period. However, any change in the instrumental response or variations in the measurement conditions can render a multivariate calibration model invalid. In this contribution, a new method, spectral space transformation (SST), has been developed to maintain the predictive abilities of multivariate calibration models when the spectrometer or measurement conditions are altered. SST tries to eliminate the spectral differences induced by the changes in instruments or measurement conditions through the transformation between two spectral spaces spanned by the corresponding spectra of a subset of standardization samples measured on two instruments or under two sets of experimental conditions. The performance of the method has been tested on two data sets comprising NIR and MIR spectra. The experimental results show that SST can achieve satisfactory analyte predictions from spectroscopic measurements subject to spectrometer/probe alteration, when only a few standardization samples are used. Compared with the existing popular methods designed for the same purpose, i.e. global PLS, univariate slope and bias correction (SBC) and piecewise direct standardization (PDS), SST has the advantages of implementation simplicity, wider applicability and better performance in terms of predictive accuracy.     

Combining orthogonal signal correction and wavelet packet transform with partial least squares to analyze overlapping spectra of three kinds of metal ions

A novel method named OSC-WPT-PLS approach based on partial least squares (PLS) regression with orthogonal signal correction (OSC) and wavelet packet transform (WPT) as pre-processed tools was proposed for the simultaneous spectrophotometric determination of Al(III), Mn(II) and Co(II). This method combines the ideas of OSC and WPT with PLS regression for enhancing the ability of extracting characteristic information and the quality of regression. OSC is used to remove information in the response matrix D by subtracting the structured noise that is orthogonal to the concentration matrix C. Wavelet packet transform was applied to perform data compression, to extract relevant information, and to eliminate noise and collinearity. PLS was applied for multivariate calibration and noise reduction by eliminating the less important latent variables. In this case, using trials, the kind of wavelet function, the decomposition level, the number of OSC components and the number of PLS factors for the OSC-WPT-PLS method were selected as Daubechies 4, 3, 2 and 3, respectively. A program (POSCWPTPLS) was designed to perform the simultaneous spectrophotometric determination of Al(III), Mn(II) and Co(II). The relative standard errors of prediction (RSEP) obtained for total elements using OSC-WPT-PLS, WPT-PLS and PLS were compared. Experimental results demonstrated that the OSC-WPT-PLS method had the best performance among the three methods and was successful even when there was severe overlap of spectra.