Multivariate calibration of near infrared spectra by orthogonal WAVElet correction using a genetic algorithm

Orthogonal WAVElet correction (OWAVEC) is a pre-processing method aimed at simultaneously accomplishing two essential needs in multivariate calibration, signal correction and data compression, by combining the application of an orthogonal signal correction algorithm to remove information unrelated to a certain response with the great potential that wavelet analysis has shown for signal processing. In the previous version of the OWAVEC method, once the wavelet coefficients matrix had been computed from NIR spectra and deflated from irrelevant information in the orthogonalization step, effective data compression was achieved by selecting those largest correlation/variance wavelet coefficients serving as the basis for the development of a reliable regression model. This paper presents an evolution of the OWAVEC method, maintaining the first two stages in its application procedure (wavelet signal decomposition and direct orthogonalization) intact but incorporating genetic algorithms as a wavelet coefficients selection method to perform data compression and to improve the quality of the regression models developed later. Several specific applications dealing with diverse NIR regression problems are analyzed to evaluate the actual performance of the new OWAVEC method. Results provided by OWAVEC are also compared with those obtained with original data and with other orthogonal signal correction methods.     

A new hybrid strategy for constructing a robust calibration model for near-infrared spectral analysis

A new hybrid algorithm is proposed for construction of a high-quality calibration model for near-infrared (NIR) spectra that is robust against both spectral interference (including background and noise) and multiple outliers. The algorithm is a combination of continuous wavelet transform (CWT) and a modified iterative reweighted PLS (mIRPLS) procedure. In the proposed algorithm the spectral interference is filtered by CWT at the first stage then mIRPLS is proposed to detect the multiple outliers in the CWT domain. Compared with the original IRPLS method, mIRPLS does not need to adjust variable parameters to achieve optimum calibration results, which makes it very convenient to perform in practice. The final PLS model is constructed robustly because both the spectral interference and multiple outliers are eliminated. In order to validate the effectiveness and universality of the algorithm, it was applied to two different sets of NIR spectra. The results indicate that the proposed strategy can greatly enhance the robustness and predictive ability of NIR spectral analysis.     

An adaptive strategy for selecting representative calibration samples in the continuous wavelet domain for near-infrared spectral analysis

Sample selection is often used to improve the cost-effectiveness of near-infrared (NIR) spectral analysis. When raw NIR spectra are used, however, it is not easy to select appropriate samples, because of background interference and noise. In this paper, a novel adaptive strategy based on selection of representative NIR spectra in the continuous wavelet transform (CWT) domain is described. After pretreatment with the CWT, an extension of the Kennard–Stone (EKS) algorithm was used to adaptively select the most representative NIR spectra, which were then submitted to expensive chemical measurement and multivariate calibration. With the samples selected, a PLS model was finally built for prediction. It is of great interest to find that selection of representative samples in the CWT domain, rather than raw spectra, not only effectively eliminates background interference and noise but also further reduces the number of samples required for a good calibration, resulting in a high-quality regression model that is similar to the model obtained by use of all the samples. The results indicate that the proposed method can effectively enhance the cost-effectiveness of NIR spectral analysis. The strategy proposed here can also be applied to different analytical data for multivariate calibration.     

基于小波系数的近红外光谱局部建模方法与应用研究

局部建模方法使用与预测样本相似的样本建立模型,可解决光谱响应与浓度之间的非线性问题,扩大模型的适用范围,提高预测准确度。采用小波变换进行数据压缩并利用小波系数之间的欧氏距离作为光谱相似性的判据,实现了近红外光谱定量分析的局部建模方法,避免了样本之间的依赖性。将所建立的方法用于烟草样品中氯含量的测定,100次重复计算得到的预测集均方根误差(RMSEP)平均值为0.0665,标准偏差(σ)为0.0045,优于全局建模和基于主成分的局部建模方法。     

A method for near-infrared spectral calibration of complex plant samples with wavelet transform and elimination of uninformative variables

An algorithm is proposed for extracting relevant information from near-infrared (NIR) spectra for multivariate calibration of routine components in complex plant samples. The algorithm is a combination of wavelet transform (WT) data compression and a procedure for uninformative variable elimination (UVE). After compression of the NIR spectra by WT, the UVE approach is used to eliminate the irrelevant wavelet coefficients. Finally, a calibration model is built from the retained wavelet coefficients to enable prediction. Because irrelevant information can be removed from the spectra used for multivariate calibration, the model based on the extracted relevant features is better than those obtained with full-spectrum data. Both prediction precision and calculation speed are improved.     

离散小波变换-遗传算法-交互检验法用于近红外光谱数据的高倍压缩与变量筛选

用遗传算法(GA)与交互检验(CV)相结合建立了一种用于对近红外光谱(NIR)数据及其离散小波变换(DWT)系数进行变量筛选的方法，并应用于烟草样品中总挥发碱和总氮的同时测定。结果表明：NIR数据经DWT压缩为原始大小的3．3％时基本没有光谱信息的丢失；有效的变量筛选可以极大地减少模型中的变量个数，降低模型的复杂程度，改善预测的准确度。     

遗传算法用于偏最小二乘方法建模中的变量筛选

利用全局搜索方法－遗传算法（genetic algorithms,GA)对近红外光谱分析中的波长变量进行筛选,再用偏最小二乘方法（patrial least squares,PLS）建立分析校正模型。对两类样品的近红外光谱分析应用实例表明,这种选取变量进行校正的方法,不仅简化、优化了模型,而且增强了所建模型的预测能力,尤其适用于单纯PLS较以校正关联的体系。     

Determination of α-linolenic acid and linoleic acid in edible oils using near-infrared spectroscopy improved by wavelet transform and uninformative variable elimination

This paper proposes an analytical method for simultaneous near-infrared (NIR) spectrometric determination of α-linolenic and linoleic acid in eight types of edible vegetable oils and their blending. For this purpose, a combination of spectral wavelength selection by wavelet transform (WT) and elimination of uninformative variables (UVE) was proposed to obtain simple partial least square (PLS) models based on a small subset of wavelengths. WT was firstly utilized to compress full NIR spectra which contain 1413 redundant variables, and 42 wavelet approximate coefficients were obtained. UVE was then carried out to further select the informative variables. Finally, 27 and 19 wavelet approximate coefficients were selected by UVE for α-linolenic and linoleic acid, respectively. The selected variables were used as inputs of PLS model. Due to original spectra were compressed, and irrelevant variables were eliminated, more parsimonious and efficient model based on WT-UVE was obtained compared with the conventional PLS model with full spectra data. The coefficient of determination (r²) and root mean square error prediction set (RMSEP) for prediction set were 0.9345 and 0.0123 for α-linolenic acid prediction by WT-UVE-PLS model. The r² and RMSEP were 0.9054, 0.0437 for linoleic acid prediction. The good performance showed a potential application using WT-UVE to select NIR effective variables. WT-UVE can both speed up the calculation and improve the predicted results. The results indicated that it was feasible to fast determine α-linolenic acid and linoleic acid content in edible oils using NIR spectroscopy.     

Elimination of interference information by a new hybrid algorithm for quantitative calibration of near infrared spectra

A new hybrid algorithm is proposed to eliminate the interference information for multivariate calibration of near-infrared (NIR) spectra that includes noise, background and systemic spectral variation irrelevant to concentration. The method consists of two parts: approximate derivative based on continuous wavelet transform (CWT) and orthogonal signal correction (OSC). After the approximate derivative calculated by CWT, OSC was performed. It was successfully applied to real complex NIR spectral data to eliminate the interference information. Correction for the interference of NIR spectra resulted in a substantial improvement in the predicted precision, and a more concise calibration model was obtained. The proposed procedure also compared favourably with several pretreatment methods, and the new method appears to provide a high-performance pretreatment tool for multivariate calibration of NIR spectra. In addition, the strategy proposed here can be applied to various other spectral data for quantitative purposes as well.     

Determination of chlorogenic acid in plant samples by using near-infrared spectrum with wavelet transform preprocessing.

By theoretical analysis, it is found that wavelet transform (WT) with a wavelet function can be regarded as a smoothing and a differentiation process, and that the order of differentiation is determined by the vanishing moment, which is an important property of a wavelet function. Therefore, a method based on the continuous wavelet transform (CWT) for removing the background in the near-infrared (NIR) spectrum is proposed, and it is used in the determination of the chlorogenic acid in plant samples as a preprocessing tool for partial least square (PLS) modeling. It is shown that the benefit of the proposed method lies not only in its performance to improve the quality of PLS model and the prediction precision, but also in its simplicity and practicability. It may become a convenient and efficient tool for preprocessing NIR spectral data sets in multivariate calibration.     

Determination of Total Sugar in Tobacco by Near-Infrared Spectroscopy and Wavelet Transformation-Based Calibration

Based on wavelet transformation (WT) and mutual information (MI), a simple and effective procedure is proposed for multivariate calibration of near-infrared spectroscopy. In such a procedure, the original spectra of the training set are first transformed into a set of wavelet representations by wavelet prism transform. Then, the MI value between each wavelet coefficient variable and the dependent variable is calculated, resulting in a MI spectrum; by retaining a subset set of coefficients with higher MI, an update training set consisting of wavelet coefficients is obtained and reconstructed/converted back to the original domain. Based on this, a partial least square (PLS) model can be constructed and optimized. The optimal wavelet and decomposition level are determined by experiment. A NIR quantitative problem involving the determination of total sugar in tobacco is used to demonstrate the overall performance of the proposed procedure, named RPLS, meaning PLS in reconstructed original domain coupled with MI-induced variable selection in wavelet domain (RPLS). Three kinds of procedures, that is, conventional full-spectrum PLS in original domain (FPLS), PLS in original domain coupled with MI-induced variable selection (OPLS), and direct PLS in MI-based wavelet coefficients (WPLS), are used as reference. The result confirms that it can build more accurate and robust calibration models without increasing the complexity.     

Detecting the quality of glycerol monolaurate: A method for using Fourier transform infrared spectroscopy with wavelet transform and modified uninformative variable elimination

Glycerol monolaurate (GML) products contain many impurities, such as lauric acid and glucerol. The GML content is an important quality indicator for GML production. A hybrid variable selection algorithm, which is a combination of wavelet transform (WT) technology and modified uninformative variable eliminate (MUVE) method, was proposed to extract useful information from Fourier transform infrared (FT-IR) transmission spectroscopy for the determination of GML content. FT-IR spectra data were compressed by WT first; the irrelevant variables in the compressed wavelet coefficients were eliminated by MUVE. In the MUVE process, simulated annealing (SA) algorithm was employed to search the optimal cutoff threshold. After the WT-MUVE process, variables for the calibration model were reduced from 7366 to 163. Finally, the retained variables were employed as inputs of partial least squares (PLS) model to build the calibration model. For the prediction set, the correlation coefficient (r) of 0.9910 and root mean square error of prediction (RMSEP) of 4.8617 were obtained. The prediction result was better than the PLS model with full-spectra data. It was indicated that proposed WT-MUVE method could not only make the prediction more accurate, but also make the calibration model more parsimonious. Furthermore, the reconstructed spectra represented the projection of the selected wavelet coefficients into the original domain, affording the chemical interpretation of the predicted results. It is concluded that the FT-IR transmission spectroscopy technique with the proposed method is promising for the fast detection of GML content.     

Treating NIR data with orthogonal discrete wavelet transform: Predicting concentrations of a multi-component system through a small-scale calibration set

Through randomly arranging samples of a calibration set, treating their NIR spectra with orthogonal discrete wavelet transform, and selecting suitable variables in terms of correlation coefficient test (r-test), it is possible to extract features of each component in a multi-component system respectively and partial least squares (PLS) models based on these features are capable of predicting the concentration of every component. What is perhaps more important, with the proposed strategy, the predictive ability of the model is at least not impaired while the size of the calibration set can be obviously reduced. Therefore, it provides a more economical, rapid, as well as convenient approach of NIR quantitative analysis for multi-component system. In addition, all important factors and parameters related to the proposed strategy are discussed in detail.     

GA-ACE: Alternating conditional expectations regression with selection of significant predictors by genetic algorithms

The non-linear regression technique known as alternating conditional expectations (ACE) method is only applicable when the number of objects available for calibration is considerably greater than the number of considered predictors. Alternating conditional expectations regression with selection of significant predictors by genetic algorithms (GA-ACE), the non-linear regression technique presented here, is based on the ACE algorithm but introducing several modifications to resolve the applicability limitations of the original ACE method, thus facilitating the practical implementation of a very interesting calibration tool. In order to overcome the lack of reliability displayed by the original ACE algorithm when working on data sets characterized by a too large number of variables and prior to the development of the non-linear regression model, GA-ACE applies genetic algorithms as a variable selection technique to select a reduced subset of significant predictors able to accurately model and predict a considered variable response. Furthermore, GA-ACE actually provides two alternative application approaches, since it allows either the performance of prior data compression computing a number of principal components to be subsequently subjected to GA-selection, or working directly on original variables.In this study, GA-ACE was applied to two real calibration problems, with a very low observation/variable ratio (NIR data), and the results were compared with those obtained by several linear regression techniques usually employed. When using the GA-ACE non-linear method, notably improved regression models were developed for the two response variables modeled, with root mean square errors of the residuals in external prediction (RMSEP) equal to 11.51 and 6.03% for moisture and lipid contents of roasted coffee samples, respectively. The improvement achieved by applying the new non-linear method introduced is even more remarkable taking into account the results obtained with the best performance linear method (IPW-PLS) applied to predict the studied responses (14.61 and 7.74% RMSEP, respectively).     

Genetic algorithm optimisation combined with partial least squares regression and mutual information variable selection procedures in near-infrared quantitative analysis of cotton-viscose textiles

In this work, different approaches for variable selection are studied in the context of near-infrared (NIR) multivariate calibration of textile. First, a model-based regression method is proposed. It consists in genetic algorithm optimisation combined with partial least squares regression (GA-PLS). The second approach is a relevance measure of spectral variables based on mutual information (MI), which can be performed independently of any given regression model. As MI makes no assumption on the relationship between X and Y, non-linear methods such as feed-forward artificial neural network (ANN) are thus encouraged for modelling in a prediction context (MI-ANN). GA-PLS and MI-ANN models are developed for NIR quantitative prediction of cotton content in cotton-viscose textile samples. The results are compared to full-spectrum (480 variables) PLS model (FS-PLS). The model requires 11 latent variables and yielded a 3.74% RMS prediction error in the range 0-100%. GA-PLS provides more robust model based on 120 variables and slightly enhanced prediction performance (3.44% RMS error). Considering MI variable selection procedure, great improvement can be obtained as 12 variables only are retained. On the basis of these variables, a 12 inputs ANN model is trained and the corresponding prediction error is 3.43% RMS error.     

A variable differential consensus method for improving the quantitative near-infrared spectroscopic analysis

Consensus methods have presented promising tools for improving the reliability of quantitative models in near-infrared(NIR) spectroscopic analysis.A strategy for improving the performance of consensus methods in multivariate calibration of NIR spectra is proposed.In the approach,a subset of non-collinear variables is generated using successive projections algorithm(SPA) for each variable in the reduced spectra by uninformative variables elimination(UVE).Then sub-models are built using the variable subsets and the calibration subsets determined by Monte Carlo(MC) re-sampling,and the sub-model that produces minimal error in cross validation is selected as a member model.With repetition of the MC re-sampling,a series of member models are built and a consensus model is achieved by averaging all the member models.Since member models are built with the best variable subset and the randomly selected calibration subset,both the quality and the diversity of the member models are insured for the consensus model.Two NIR spectral datasets of tobacco lamina are used to investigate the proposed method.The superiority of the method in both accuracy and reliability is demonstrated.     

校正集样品特性对鼠药中毒鼠强近红外光谱分析的影响

根据市售鼠药样品成分各异且相对复杂,建立6种不同成分体系和9个不同样本容量的校正集,运用小波变换压缩鼠药的近红外透射光谱数据,结合BP反向神经网络算法对压缩的数据进行建模,考察校正集样品特性对模型预测能力的影响。试验结果表明:采用BP神经网络算法建立定量模型时,只要校正集样品中包含了与预测样品性质相似的样本,就能准确地对复杂样品进行近红外定量分析。当校正集容量分别为72和84时,模型预测结果趋于平稳。当校正集数量为96时,模型的最大相关系数为0.959 8,预测最小标准差和平均相对误差分别为1.893%和1.92%。     

A Background and noise elimination method for quantitative calibration of near infrared spectra

A new hybrid algorithm is proposed to eliminate the varying background and noise simultaneously for multivariate calibration of near infrared (NIR) spectral signals. The method is based on the use of multi-resolution, which is one of the main advantages provided by wavelet transform. The signals are firstly split into different frequency components, which keep the same data points of the original signals. In conjunction with a modified uninformative variable elimination (mUVE) criterion, the new method can be used to remove the low-frequency varying background and the high-frequency noise simultaneously. The method is successfully applied to simulated spectral data set and experimental NIR spectral data, resulting in more parsimonious multivariate models with higher precision. In addition, the proposed strategy can be applied to other spectral signals as well.     

Ensemble wavelet modelling for determination of wheat and gasoline properties by near and middle infrared spectroscopy

The wavelet transform has been shown to be a useful tool for multivariate calibration. However, the choice of wavelet transform settings (wavelet family, length and number of decomposition levels) for a given application is still an open problem. The present paper proposes an alternative approach, which consists of generating an ensemble model by combining individual models obtained with different wavelet transform settings. The advantages of the proposed method are demonstrated in two analytical problems, namely the determination of moisture and protein in wheat by near infrared spectroscopy and the determination of specific mass and three distillation temperatures (T10, T50, T90) in gasoline by middle infrared spectroscopy. In these problems, the results varied considerably among individual models, which underlines the risk associated to an inadequate choice of wavelet transform settings. In contrast, the ensemble model always provided adequate results in terms of prediction error and noise sensitivity. The proposed method can be seen as an advantageous alternative for multivariate calibration in the wavelet domain, as it frees the analyst from the need to choose a particular configuration for the wavelet transform.     

近红外光谱法测定端羟基环氧乙烷-四氢呋喃共聚醚的羟值

提出了用近红外光谱测定端羟基环氧乙烷-四氢呋喃共聚醚(PET)的羟值,结合主成分回归和偏最小二乘法建立了PET羟值与其近红外光谱之间的关联模型。结果表明,近红外光谱法与化学分析法的测定结果一致;近红外光谱法测定PET羟值的相对误差在5%以内;利用遗传算法选择部分波长建立校正可以降低模型的预测误差。