OWAVEC: a combination of wavelet analysis and an orthogonalization algorithm as a pre-processing step in multivariate calibration

In an spectroscopic context, when a calibration model based on partial least squares is developed to predict a response, it is often the case that a high percentage of variation in the data explained by the first latent variable is not accompanied by an equally high percentage of variation in the studied response. The addition of more components can slowly improve the calibration model, but with negative effects on the robustness and interpretability of the final model. To solve this problem, several pre-processing methods have been proposed to remove only a portion unrelated to the studied response from the spectral matrix.Moreover, the need for efficient compression methods is increasingly important due to the large size of the data currently collected. In this sense, discrete wavelet transform has proven that it can achieve good compression without losing relevant information when used on individual signals.This paper introduces a new pre-processing method, orthogonal wavelet correction (OWAVEC) that tries to lump together two important needs in multivariate calibration: signal correction and compression. The new method has been tested on a set of diesel fuels using viscosity as variable response, and its results have been compared not only with those obtained from original data but also with those provided by other correction methods. The first practical results are encouraging, as the method generates considerably better calibration models compared to the model developed from raw data and provides results as least so good as other orthogonal correction methods.     

小波变换方法的比较──红外光谱数据压缩

介绍了小波变换和多分辨分析的基本理论以及常用小波变换压缩数据的3种方法:(1)只保留模糊信号;(2)全部保留模糊信号及锐化信号中的较大值;(3)保留模糊信号及锐化信号中的较大值.将紧支集小波和正交三次B-样条小波压缩4-苯乙炔基-邻苯二甲酸酐的红外光谱数据进行了对比,计算表明正交三次B-样条小波变换方法效果较好,而在全部保留模糊信号及只保留锐化信号中数值较大的系数时,压缩比大而重建光谱数据与原始光谱数据间的均方差较小.     

Elimination of interference information by a new hybrid algorithm for quantitative calibration of near infrared spectra

A new hybrid algorithm is proposed to eliminate the interference information for multivariate calibration of near-infrared (NIR) spectra that includes noise, background and systemic spectral variation irrelevant to concentration. The method consists of two parts: approximate derivative based on continuous wavelet transform (CWT) and orthogonal signal correction (OSC). After the approximate derivative calculated by CWT, OSC was performed. It was successfully applied to real complex NIR spectral data to eliminate the interference information. Correction for the interference of NIR spectra resulted in a substantial improvement in the predicted precision, and a more concise calibration model was obtained. The proposed procedure also compared favourably with several pretreatment methods, and the new method appears to provide a high-performance pretreatment tool for multivariate calibration of NIR spectra. In addition, the strategy proposed here can be applied to various other spectral data for quantitative purposes as well.     

A method for near-infrared spectral calibration of complex plant samples with wavelet transform and elimination of uninformative variables

An algorithm is proposed for extracting relevant information from near-infrared (NIR) spectra for multivariate calibration of routine components in complex plant samples. The algorithm is a combination of wavelet transform (WT) data compression and a procedure for uninformative variable elimination (UVE). After compression of the NIR spectra by WT, the UVE approach is used to eliminate the irrelevant wavelet coefficients. Finally, a calibration model is built from the retained wavelet coefficients to enable prediction. Because irrelevant information can be removed from the spectra used for multivariate calibration, the model based on the extracted relevant features is better than those obtained with full-spectrum data. Both prediction precision and calculation speed are improved.     

Alternative common bases and signal compression for wavelets application in chemometrics

Representation or compression of data sets in the wavelet space is usually performed to retain the maximum variance of the original or pretreated data, like in the compression by means of principal components. In order to represent together a number of objects in the wavelet space, a common basis is required, and this common basis is usually obtained by means of the variance spectrum or of the variance wavelet tree. In this study, the use of alternative common bases is suggested, both for classification and regression problems. In the case of classification or class-modeling, the suggested common bases are based on the spectrum of the Fisher weights (a measure of the between-class to within-class variance ratio) or on the spectrum of the SIMCA discriminant weights. In the case of regression, the suggested common bases are obtained by the correlation spectrum (the correlation coefficients of the predictor variables with a response variable) or by the PLS (Partial Least Squares regression) importance of the predictors (the product between the absolute value of the regression coefficient of the predictor in the PLS model and its standard deviation). Other alternative strategies apply the Gram–Schmidt supervised orthogonalization to the wavelet coefficients. The results indicate that, both in classification and regression, the information retained after compression in the wavelets space can be more efficient than that retained with a common basis obtained by variance.     

Influence of data pre-processing on the quantitative determination of the ash content and lipids in roasted coffee by near infrared spectroscopy

In near-infrared (NIR) measurements, some physical features of the sample can be responsible for effects like light scattering, which lead to systematic variations unrelated to the studied responses. These errors can disturb the robustness and reliability of multivariate calibration models. Several mathematical treatments are usually applied to remove systematic noise in data, being the most common derivation, standard normal variate (SNV) and multiplicative scatter correction (MSC). New mathematical treatments, such as orthogonal signal correction (OSC) and direct orthogonal signal correction (DOSC), have been developed to minimize the variability unrelated to the response in spectral data. In this work, these two new pre-processing methods were applied to a set of roasted coffee NIR spectra. A separate calibration model was developed to quantify the ash content and lipids in roasted coffee samples by PLS regression. The results provided by these correction methods were compared to those obtained with the original data and the data corrected by derivation, SNV and MSC. For both responses, OSC and DOSC treatments gave PLS calibration models with improved prediction abilities (4.9 and 3.3% RMSEP with corrected data versus 7.1 and 8.3% RMSEP with original data, respectively).     

Combining orthogonal signal correction and wavelet packet transform with partial least squares to analyze overlapping spectra of three kinds of metal ions

A novel method named OSC-WPT-PLS approach based on partial least squares (PLS) regression with orthogonal signal correction (OSC) and wavelet packet transform (WPT) as pre-processed tools was proposed for the simultaneous spectrophotometric determination of Al(III), Mn(II) and Co(II). This method combines the ideas of OSC and WPT with PLS regression for enhancing the ability of extracting characteristic information and the quality of regression. OSC is used to remove information in the response matrix D by subtracting the structured noise that is orthogonal to the concentration matrix C. Wavelet packet transform was applied to perform data compression, to extract relevant information, and to eliminate noise and collinearity. PLS was applied for multivariate calibration and noise reduction by eliminating the less important latent variables. In this case, using trials, the kind of wavelet function, the decomposition level, the number of OSC components and the number of PLS factors for the OSC-WPT-PLS method were selected as Daubechies 4, 3, 2 and 3, respectively. A program (POSCWPTPLS) was designed to perform the simultaneous spectrophotometric determination of Al(III), Mn(II) and Co(II). The relative standard errors of prediction (RSEP) obtained for total elements using OSC-WPT-PLS, WPT-PLS and PLS were compared. Experimental results demonstrated that the OSC-WPT-PLS method had the best performance among the three methods and was successful even when there was severe overlap of spectra.     

近红外分析中光谱预处理及波长选择方法进展与应用

光谱预处理和波长选取方法在近红外光谱分析技术中相当重要。本文综述了常用的NIR预处理和波长选取方法及这一领域的最新进展,详细介绍正交信号校正(OSC)、净分析信号(NAS)和小波变换(WT)等新光谱预处理方法以及无信息变量消除(UVE)和遗传算法(GA)等波长选取方法,并给出了这些方法的具体算法和一些应用实例。     

Near-infrared determination of active substance content in intact low-dosage tablets

Near-infrared (NIR) spectroscopy can be applied to determine the active substance content of tablets. Its great advantage lies in the minimal sample preparation required, which helps to reduce the potential for error. The aim of this study is to show the feasibility of this method on low-dosage tablets. The influence of various spectral pretreatments [standard normal variate (SNV), multiplicative scatter correction (MSC), second derivative (D2), orthogonal signal correction (OSC), separately and combined] and regression methods on prediction error are compared. Partial least square (PLS) regression provided better prediction than principal component regression (PCR). SNV was applied to the first data set and SNV and a second derivative to the second set to maximise model accuracy for quantifying the active substance of intact pharmaceutical products using diffuse reflectance NIR. The models yielded standard errors of prediction (SEP) of 0.1768 and 0.0682 mg for the two products. The experiments were conducted with two low-dosage pharmaceutical forms and results of NIR predictions were comparable to currently approved methods. Diffuse reflectance NIR has the potential to become a reliable and robust quality control method for determining active tablet content.     

荧光光度法同时测定邻苯二酚、间苯二酚与对苯二酚

将一种直接信号校正(DOSC)-小波包变换(WPT)-偏最小二乘法(PLS)(DOSC-WPT-PLS)新方法用于解析荧光光谱严重重叠的邻苯二酚?间苯二酚和对苯二酚混合物,并对其进行测定。该法将DOSC、WPT及PLS 3种方法结合从而提高了获取特征信息的能力和回归质量。DOSC方法用于除去与浓度无关的结构噪音。利用WPT的时域和频域局部化的特点改进了除噪质量和数据压缩及信息提取能力。PLS方法用于多变量校准和噪音消除。处理该3种组分的荧光光谱数据,并实现了3种化合物的同时测定。设计了PDOSCWPTPLS程序执行相关计算,并对以上3种化学计量学方法进行了比较,其总体相对预测标准偏差分别为4.3%、7.7%、11.5%,结果表明DOSC-WPT-PLS法优于WPT-PLS法和PLS法。将该法用于测定自来水中邻苯二酚?间苯二酚和对苯二酚的含量,其回收率分别为99%~110%?95%~108%和98%~104%,结果满意。     

Quantitative Analysis of Berberine in Processed Coptis by Near-Infrared Diffuse Reflectance Spectroscopy

The near-infrared（NIR） diffuse reflectance spectroscopy was used to study the content of Berberine in the processed Coptis. The allocated proportions of Coptis to ginger, yellow liquor or Evodia rutaecarpa changed according to the results of orthogonal design as well as the temperature. For as withdrawing the full and effective information from the spectral data as possible, the spectral data was preprocessed through first derivative and multiplicative scatter correetion（MSC） according to the optimization results of different preprocessing methods. Firstly, the model was established by partial least squares（PLS）; the coefficient of determination（R2） of the prediction was 0.839, the root mean squared error of prediction（RMSEP） was 0.1422, and the mean relative error（RME） was 0.0276. Secondly, for reducing the dimension and removing noise, the spectral variables were highly effectively compressed via the wavelet transformation（WT） technology and the Haar wavelet was selected to decompose the spectral signals. After the wavelet coefficients from WT were input into the artificial neural network（ANN） instead of the spectra signal, the quantitative analysis model of Berberine in processed Coptis was established. The R^2 of the model was 0.9153, the RMSEP was 0.0444, and the RME was 0.0091. The values of appraisal index, namely R^2, RMSECV, and RME, indicate that the generalization ability and prediction precision of ANN are superior to those of PLS. The overall results show that NIR spectroscopy combined with ANN can be efficiently utilized for the rapid and accurate analysis of routine chemical compositions in Coptis. Accordingly, the result can provide technical support for the further analysis of Berberine and other components in processed Coptis. Simultaneously, the research can also offer the foundation of quantitative analysis of other NIR application.     

Filter design for molecular factor computing using wavelet functions

Molecular factor computing (MFC) is a new strategy that employs chemometric methods in an optical instrument to obtain analytical results directly using an appropriate filter without data processing. In the present contribution, a method for designing an MFC filter using wavelet functions was proposed for spectroscopic analysis. In this method, the MFC filter is designed as a linear combination of a set of wavelet functions. A multiple linear regression model relating the concentration to the wavelet coefficients is constructed, so that the wavelet coefficients are obtained by projecting the spectra onto the selected wavelet functions. These wavelet functions are selected by optimizing the model using a genetic algorithm (GA). Once the MFC filter is obtained, the concentration of a sample can be calculated directly by projecting the spectrum onto the filter. With three NIR datasets of corn, wheat and blood, it was shown that the performance of the designed filter is better than that of the optimized partial least squares models, and commonly used signal processing methods, such as background correction and variable selection, were not needed. More importantly, the designed filter can be used as an MFC filter in designing MFC-based instruments.     

An evaluation of orthogonal signal correction methods for the characterisation of arabica and robusta coffee varieties by NIRS

Two orthogonal signal correction methods (OSC and DOSC) were applied on a set of 83 roasted coffee NIR spectra from varied origins and varieties in order to remove information unrelated to a specific chemical response (caffeine), which was selected due to its high discriminant ability to differentiate between arabica and robusta coffee varieties. These corrected NIR spectra, as well as raw NIR spectra and three chemical quantities (caffeine, chlorogenic acids and total acidity), were used to develop separate classification models accordingly using the potential functions method as a class-modelling technique in order to evaluate their respective capacities to discriminate between coffee varieties and the influence of these pre-processing methods on the classification of the coffee samples into their corresponding variety class. The transformation of roasted coffee NIR spectra by means of an orthogonal signal correction method, taking into account in this correction a chemical response closely related to the sample origin, prompted a notable improvement in the specificity of the constructed classification models.     

Orthogonal signal correction, wavelet analysis, and multivariate calibration of complicated process fluorescence data

In this paper, multivariate calibration of complicated process fluorescence data is presented. Two data sets related to the production of white sugar are investigated. The first data set comprises 106 observations and 571 spectral variables, and the second data set 268 observations and 3997 spectral variables. In both applications, a single response, ash content, is modelled and predicted as a function of the spectral variables. Both data sets contain certain features making multivariate calibration efforts non-trivial. The objective is to show how principal component analysis (PCA) and partial least squares (PLS) regression can be used to overview the data sets and to establish predictively sound regression models. It is shown how a recently developed technique for signal filtering, orthogonal signal correction (OSC), can be applied in multivariate calibration to enhance predictive power. In addition, signal compression is tested on the larger data set using wavelet analysis. It is demonstrated that a compression down to 4% of the original matrix size — in the variable direction — is possible without loss of predictive power. It is concluded that the combination of OSC for pre-processing and wavelet analysis for compression of spectral data is promising for future use.     

Removal of major interference sources in aqueous near-infrared spectroscopy techniques

This work describes a hybrid procedure for eliminating major interference sources in aqueous near-infrared (NIR) spectra, that include aqueous influence, noise, and systemic variations irrelevant to concentration. The scheme consists of two parts: extension of wavelet prism (WPe) and orthogonal signal correction (OSC). First, WPe is employed to remove variations due to aqueous absorbance and noise; then OSC is applied to remove systemic spectral variations irrelevant to concentration. Although water possesses strong absorption bands that overshadow and overlap the absorption bands of analytes, along with noise and systematic interference, successful calibration models can be generated by employing the method proposed here. We show that the elimination of major interference sources from the aqueous NIR spectra results in a substantial improvement in the precision of prediction, and reduces the required number of PLS components in the model. In addition, the strategy proposed here can be applied to various analytical data for quantitative purposes as well.     

Selective removal of interference signals for near-infrared spectra of biomedical samples by using region orthogonal signal correction

New approach for chemometrics algorithm named region orthogonal signal correction (ROSC) has been introduced to improve the predictive ability of PLS models for biomedical components in blood serum developed from their NIR spectra in the 1280-1849 nm region. Firstly, a moving window partial least squares regression (MWPLSR) method was employed to locate the region due to water as a region of interference signals and to find the informative regions of glucose, albumin, cholesterol and triglyceride from NIR spectra of bovine serum samples. Next, a novel chemometrics method named searching combination moving window partial least squares (SCMWPLS) was used to optimize those informative regions. Then, the specific regions that contained the information of water, glucose, albumin, cholesterol and triglyceride were obtained. When an interested component in the bovine serum solution, such as glucose, albumin, cholesterol or triglyceride is being an analyte, the other three interests and water are considered as the interference factors. Thus, new approach for ROSC has employed for each specific region of interference signal to calculate the orthogonal components to the concentrations of analyte that were removed specifically from the NIR spectra of bovine serum in the region of 1280-1849 nm and the highest interference signal for model of analyte will be revealed. The comparison of PLS results for glucose, albumin, cholesterol and triglyceride built by using the whole region of original spectra and those developed by using the optimized regions suggested by SCMWPLS of original spectra, spectra treated OSC for orthogonal components of 1-3 and spectra treated ROSC using selected removing the highest interference signals from the spectra for orthogonal components of 1-3 are reported. It has been found that new approach of ROSC to remove the highest interference signal located by SCMWPLS improves of the performance of PLS modeling, yielding the lower RMSECV and smaller number of PLS factors.     

Orthogonal signal correction applied to the classification of wine and molasses vinegar samples by near-infrared spectroscopy. Feasibility study for the detection and quantification of adulterated vinegar samples

The most common fraudulent practice in the vinegar industry is the addition of alcohol of different origins to the base wine used to produce wine vinegar with the objective of reducing manufacturing costs. The mixture is then sold commercially as genuine wine vinegar, thus constituting a fraud to consumers and an unfair practice with respect to the rest of the vinegar sector. A method based on near-infrared spectroscopy has been developed to discriminate between white wine vinegar and alcohol or molasses vinegar. Orthogonal signal correction (OSC) was applied to a set of 96 vinegar NIR spectra from both original and artificial blends made in the laboratory, to remove information unrelated to a specific response. The specific response used to correct the spectra was the extent of adulteration of the vinegar samples. Both raw and corrected NIR spectra were used to develop separate classification models using the potential functions method as a class-modeling technique. The previous models were compared to evaluate the suitability of near-infrared spectroscopy as a rapid method for discrimination between vinegar origin. The transformation of vinegar NIR spectra by means of an orthogonal signal-correction method resulted in notable improvement of the specificity of the constructed classification models. The same orthogonal correction approach was also used to perform a calibration model able to detect and quantify the amount of exogenous alcohol added to the commercial product. This regression model can be used to quantify the extent of adulteration of new vinegar samples.     

有指导的正交投影技术结合斜率/截距校正法实现小试水分近红外定量模型向中试传递

针对小试制剂过程建立的近红外定量模型难以直接应用于中试或大生产过程中的问题,该文以小试和中试条件下多批次药用糊精流化床制粒过程为载体,在线采集其近红外光谱数据并测定水分含量,建立小试过程水分近红外定量模型,提出并应用有指导的正交投影技术结合斜率/截距校正的模型传递方法跨尺度预测中试样本,使中试两个测试集A和B的水分相对预测误差分别由51.04%和26.64%降至4.90%和3.99%,显著提高了模型预测的准确度。将该结果与无指导的正交投影技术结合斜率/截距校正法以及模型更新相比较,该方法能更加有效地去除待测样本光谱中的干扰信息,适用范围广,为小试建立的模型放大应用到中试甚至大生产过程提供了新方案。     

离散小波变换-遗传算法-交互检验法用于近红外光谱数据的高倍压缩与变量筛选

用遗传算法(GA)与交互检验(CV)相结合建立了一种用于对近红外光谱(NIR)数据及其离散小波变换(DWT)系数进行变量筛选的方法，并应用于烟草样品中总挥发碱和总氮的同时测定。结果表明：NIR数据经DWT压缩为原始大小的3．3％时基本没有光谱信息的丢失；有效的变量筛选可以极大地减少模型中的变量个数，降低模型的复杂程度，改善预测的准确度。     

Rapid and Nondestructive Analysis of Bacillus Calmette–Guerin Polysaccharide Nucleic Acid Injection by near-Infrared Spectroscopy with Chemometrics

A novel quantitative analytical method using near-infrared (NIR) spectroscopy combined with chemometrics has been developed to determine the polysaccharides and nucleic acids for routine quality analysis of Bacillus Calmette–Guerin polysaccharide and nucleic acid injections. A Monte-Carlo method was used to detect and discard outliers and to improve the predictive ability of the model. Various other spectral preprocessing methods such as smoothing, derivative, multiplicative scattering correction, standard normal variables, and orthogonal signal correction methods were used to remove noise and other irrelevant information from the spectra. Sample-set partitioning based on joint x–y distance method was utilized to divide the sample measurements into calibration and validation datasets. The optimal wavelength variables were determined by competitive adaptive weighted sampling. The model was established and cross-validated using partial least square regression. The root mean square errors of cross-validation for polysaccharides and nucleic acids were determined to be 0.0382 and 5.218, and the root mean square errors of prediction were 0.0229 and 6.282. The overall results show that NIR spectroscopy combined with chemometry is effective for the quantitative analysis of Bacillus Calmette–Guerin polysaccharide and nucleic acid injections.