An adaptive strategy for selecting representative calibration samples in the continuous wavelet domain for near-infrared spectral analysis

Sample selection is often used to improve the cost-effectiveness of near-infrared (NIR) spectral analysis. When raw NIR spectra are used, however, it is not easy to select appropriate samples, because of background interference and noise. In this paper, a novel adaptive strategy based on selection of representative NIR spectra in the continuous wavelet transform (CWT) domain is described. After pretreatment with the CWT, an extension of the Kennard–Stone (EKS) algorithm was used to adaptively select the most representative NIR spectra, which were then submitted to expensive chemical measurement and multivariate calibration. With the samples selected, a PLS model was finally built for prediction. It is of great interest to find that selection of representative samples in the CWT domain, rather than raw spectra, not only effectively eliminates background interference and noise but also further reduces the number of samples required for a good calibration, resulting in a high-quality regression model that is similar to the model obtained by use of all the samples. The results indicate that the proposed method can effectively enhance the cost-effectiveness of NIR spectral analysis. The strategy proposed here can also be applied to different analytical data for multivariate calibration.     

Elimination of interference information by a new hybrid algorithm for quantitative calibration of near infrared spectra

A new hybrid algorithm is proposed to eliminate the interference information for multivariate calibration of near-infrared (NIR) spectra that includes noise, background and systemic spectral variation irrelevant to concentration. The method consists of two parts: approximate derivative based on continuous wavelet transform (CWT) and orthogonal signal correction (OSC). After the approximate derivative calculated by CWT, OSC was performed. It was successfully applied to real complex NIR spectral data to eliminate the interference information. Correction for the interference of NIR spectra resulted in a substantial improvement in the predicted precision, and a more concise calibration model was obtained. The proposed procedure also compared favourably with several pretreatment methods, and the new method appears to provide a high-performance pretreatment tool for multivariate calibration of NIR spectra. In addition, the strategy proposed here can be applied to various other spectral data for quantitative purposes as well.     

Simultaneous wavelength selection and outlier detection in multivariate regression of near-infrared spectra.

Near-infrared (NIR) spectrometry will present a more promising tool for quantitative measurement if the robustness and predictive ability of the partial least square (PLS) model are improved. In order to achieve the purpose, we present a new algorithm for simultaneous wavelength selection and outlier detection; at the same time, the problems of background and noise in multivariate calibration are also solved. The strategy is a combination of continuous wavelet transform (CWT) and modified iterative predictors and objects weighting PLS (mIPOW-PLS). CWT is performed as a pretreatment tool for eliminating background and noise synchronously; then, mIPOW-PLS is proposed to remove both the useless wavelengths and the multiple outliers in CWT domain. After pretreatment with CWT-mIPOW-PLS, a PLS model is built finally for prediction. The results indicate that the combination of CWT and mIPOW-PLS produces robust and parsimonious regression models with very few wavelengths.     

偏最小二乘回归用于近红外光谱分析的稳健策略

偏最小二乘法（PLS）在近红外光谱（NIR）定量分析中占有重要地位,但预测结果往往容易受到样本分组和奇异样本等因素的影响,稳健性不强。多模型PLS (EPLS）方法在模型稳健性上得到提高,然而它无法识别样本中存在的奇异样本。为了同时提高模型的预测准确性和稳健性,本文提出了一种根据取样概率重新取样的多模型PLS方法,称为稳健共识PLS（RE-PLS）方法。该方法通过迭代赋权偏最小二乘法(IRPLS)计算样本回归残差得到每个校正集样本的取样概率,然后根据样本的取样概率来选择训练子集建立多个PLS模型,最后将所有PLS模型的预测结果平均作为最终预测结果。该方法用于两种不同植物样品的近红外光谱建模,并与传统的PLS及EPLS方法进行比较。结果表明该方法可以有效的避免校正集中奇异样本对模型的影响,同时可以提高预测精确度和稳健性。对于含有较多奇异样本的,复杂近红外光谱烟草实际样本,利用简单PLS或者EPLS方法建模预测效果不是很理想,而RE-PLS凭借其独特优势则有望在这种复杂光谱定量分析中得到广泛的应用。     

A strategy for enhancing the reliability of near-infrared spectral analysis

Near-infrared (NIR) spectroscopy will present a more promising tool for quantitative measurement if the reliability of the calibration model is further improved. To achieve this purpose, a new partial least squares (PLSs) technique based on Monte Carlo (MC) resampling is proposed, which is named as MCPLS. In this method, the outliers are firstly removed based on probability statistics. Then, the models without outliers are averaged and combined into a single prediction model as done in a consensus modeling, which can greatly enhance the reliability of PLS calibration. To validate the effectiveness and universality of the proposed method, it was applied to two different sets of NIR spectra. It was found that MCPLS could effectively avoid the swamping and masking effects caused by multiple outliers. The results show that the method is of value to enhance the reliability of PLS model involving complex NIR matrices with a small number of outliers.     

An ensemble method based on a self-organizing map for near-infrared spectral calibration of complex beverage samples

Based on a so-called ensemble strategy, an algorithm is proposed for near-infrared (NIR) spectral calibration of complex beverage samples. This algorithm is a combination of a novel training set/test set sample-selection procedure based on a Kohonen self-organizing map (SOM) with a simple procedure to calculate an average partial least-squares (PLS) calibration model, which is therefore named SOMEPLS. In order to verify the proposed SOMEPLS, two NIR beverage datasets involving the determination of sugar content are considered, and three kinds of reference algorithm, i.e., conventional PLS (CPLS), the Kennard-Stone (KS) algorithm in combination with PLS (KSPLS), and sample set partitioning based on the joint x-y distance (SPXY) algorithm in combination with PLS (SPXYPLS), are used. Of these, both KS and SPXY are well-known representative sample-selection algorithms. By comparison, it was found that when there is a training set of appropriate size, SOMEPLS can achieve better prediction accuracy than the three reference algorithms, but without increasing the complexity of the corresponding calibration model for the future application, indicating that SOMEPLS can serve as a promising tool for NIR spectral calibration.     

Variable selection by modified IPW (iterative predictor weighting)-PLS (partial least squares) in continuous wavelet regression models

Variable selection is often used to produce more robust and parsimonious regression models. But when they are applied directly to the raw near-infrared spectra, it is not easy to select appropriate variables because background and noise will often overshadow or overlap the absorption bands of analyte. In this work, a new hybrid algorithm based on the selection of the most informative variables in the continuous wavelet transform (CWT) domain is described. The strategy is a combination of CWT and a procedure of modified iterative predictor weighting-partial least square (mIPW-PLS). After elimination of the background and noise in NIR spectra by CWT, the mIPW-PLS approach is used to select the most informative CWT coefficients. With the selected CWT coefficients, a PLS model is built finally for prediction. It is indicated that the extraction of most important variables in the CWT domain can effectively avoid the interference of background and noise, and result in a high quality of regression model with a very small number of variables and fewer PLS components.     

Determination of chlorogenic acid in plant samples by using near-infrared spectrum with wavelet transform preprocessing.

By theoretical analysis, it is found that wavelet transform (WT) with a wavelet function can be regarded as a smoothing and a differentiation process, and that the order of differentiation is determined by the vanishing moment, which is an important property of a wavelet function. Therefore, a method based on the continuous wavelet transform (CWT) for removing the background in the near-infrared (NIR) spectrum is proposed, and it is used in the determination of the chlorogenic acid in plant samples as a preprocessing tool for partial least square (PLS) modeling. It is shown that the benefit of the proposed method lies not only in its performance to improve the quality of PLS model and the prediction precision, but also in its simplicity and practicability. It may become a convenient and efficient tool for preprocessing NIR spectral data sets in multivariate calibration.     

A consensus least squares support vector regression (LS-SVR) for analysis of near-infrared spectra of plant samples

Consensus modeling of combining the results of multiple independent models to produce a single prediction avoids the instability of single model. Based on the principle of consensus modeling, a consensus least squares support vector regression (LS-SVR) method for calibrating the near-infrared (NIR) spectra was proposed. In the proposed approach, NIR spectra of plant samples were firstly preprocessed using discrete wavelet transform (DWT) for filtering the spectral background and noise, then, consensus LS-SVR technique was used for building the calibration model. With an optimization of the parameters involved in the modeling, a satisfied model was achieved for predicting the content of reducing sugar in plant samples. The predicted results show that consensus LS-SVR model is more robust and reliable than the conventional partial least squares (PLS) and LS-SVR methods.     

Comparison of near-infrared and mid-infrared spectroscopy for the determination of distillation property of kerosene

Near-infrared (NIR) and mid-infrared (MIR) spectroscopy have been compared and evaluated for the determination of the distillation property of kerosene with the use of partial least squares (PLS) regression. Since kerosene is a complex mixture of similar hydrocarbons, both spectroscopic methods will be best evaluated with this complex sample matrix. PLS calibration models for each percent recovery temperature have been developed by using both NIR and MIR spectra without spectral pretreatment. Both methods have shown good correlation with the corresponding reference method, however NIR provided better calibration performance over MIR. To rationalize the improved calibration performance of NIR, spectra of the same kerosene sample were continuously collected and the corresponding spectral reproducibility was evaluated. The greater spectral reproducibility including signal-to-noise ratio of NIR led to the improved calibration performance, even though MIR spectroscopy provided more qualitative spectral information. The reproducibility of measurement, signal-to-noise ratio, and richness of qualitative information should be simultaneously considered for proper selection of a spectroscopic method for quantitative analysis.     

Wavelet unfolded partial least squares for near-infrared spectral quantitative analysis of blood and tobacco powder samples

Continuous wavelet transform (CWT) has been shown to be a high-performance signal processing technique in multivariate calibration. However, the signal processed by CWT with a specific wavelet may account for only a part of the information. To effectively utilize more abundant information contained in analytical signals, a method, named as wavelet unfolded partial least squares (WUPLS), was proposed. In the approach, the measured dataset is firstly extended by CWT with different wavelets, and then partial least squares (PLS) is employed to develop the quantitative model between the extended dataset and the target values. In order to select the representative wavelets, principal component analysis (PCA) is used to investigate the distribution of the signals obtained by CWT with different wavelets. The performance of the method was tested with blood and tobacco powder samples. Compared with the results obtained by PLS methods, the WUPLS method combined with signal processing techniques is proven to be a promising tool for improving the near-infrared (NIR) spectral analysis of complex samples.     

Standardization of near infrared spectra measured on multi-instrument

Calibration model transfer is essential for practical applications of near infrared (NIR) spectroscopy because the measurements of the spectra may be performed on different instruments and the difference between the instruments must be corrected. An approach for calibration transfer based on alternating trilinear decomposition (ATLD) algorithm is proposed in this work. From the three-way spectral matrix measured on different instruments, the relative intensity of concentration, spectrum and instrument is obtained using trilinear decomposition. Because the relative intensity of instrument is a reflection of the spectral difference between instruments, the spectra measured on different instruments can be standardized by a correction of the coefficients in the relative intensity. Two NIR datasets of corn and tobacco leaf samples measured with three instruments are used to test the performance of the method. The results show that, for both the datasets, the spectra measured on one instrument can be correctly predicted using the partial least squares (PLS) models built with the spectra measured on the other instruments.     

Removal of major interference sources in aqueous near-infrared spectroscopy techniques

This work describes a hybrid procedure for eliminating major interference sources in aqueous near-infrared (NIR) spectra, that include aqueous influence, noise, and systemic variations irrelevant to concentration. The scheme consists of two parts: extension of wavelet prism (WPe) and orthogonal signal correction (OSC). First, WPe is employed to remove variations due to aqueous absorbance and noise; then OSC is applied to remove systemic spectral variations irrelevant to concentration. Although water possesses strong absorption bands that overshadow and overlap the absorption bands of analytes, along with noise and systematic interference, successful calibration models can be generated by employing the method proposed here. We show that the elimination of major interference sources from the aqueous NIR spectra results in a substantial improvement in the precision of prediction, and reduces the required number of PLS components in the model. In addition, the strategy proposed here can be applied to various analytical data for quantitative purposes as well.     

Characterization and analysis of polymorphs by near-infrared spectrometry

The polymorphic purity of drug is of high pharmaceutical interest as it often dictates its bioavailability. In this work, we developed a rapid, efficient method for the characterization and determination of azithromycin polymorphs using near-infrared (NIR) spectrometry. The drug is characterized by comparison with a NIR spectral library that permits one to determine whether the amount of crystalline form contained in an amorphous azithromycin sample exceeds allowed levels. While the crystalline form is a hydrate, the amorphous form is anhydrous; however, the absorption of a small amount of moisture by the drug reduces the spectral differences between the two forms and hinders the establishment of an accurate calibration model. In this work, we determined the crystalline form by using a partial least-squares regression model (PLS1) for calibration and examined the influence of factors such as spectral treatment, wavelength range and moisture content on the results. The high correlation between the spectra for the two forms enabled the development of a PLS2 model for determining both species jointly. The proposed method was validated with a view to its subsequent use in the analytical control of azithromycin.     

复杂样品近红外光谱定量分析模型的构建方法

针对复杂样品近红外光谱分析中校正集的设计问题, 探讨了标准样品参与复杂样品建模的可行性. 通过标准样品和复杂基质样品共同构建的偏最小二乘(PLS)模型, 考察了波段筛选和建模参数对预测结果的影响. 结果表明, 采用PLS方法建立定量模型时, 校正集样品性质应该尽量与预测集样品相似, 当样品的性质相差较大时, 适当增加校正集样品的差异性可使模型具有更强的预测能力. 同时, 波段优选对提高预测结果的准确性具有重要的意义.     

段式正交信号校正方法及在小麦近红外光谱数据分析中的应用

针对光谱数据峰宽、局部效应显著、含有噪音、变量个数多及彼此间常存在严重的复共线性等问题,改进和设计一种光谱数据局部校正方法:基于窗口平滑的段式正交信号校正方法,并将之结合偏最小二乘回归,以实现光谱数据的预处理及定量分析。通过NIPALS算法初始化将滤去的正交成分,以近邻分段方式进行逐个波长点的正交信号校正。而后将去噪后的光谱矩阵作为新的自变量阵,通过偏最小二乘回归构建其与性质参变量间的校正模型。通过小麦近红外漫反射光谱数据的应用实验结果表明,本方法正交成分估计稳定,去噪明显,模型的预报性能优于其它方法,PLS成分数减少,模型更加简洁。     

Construction of global and robust near-infrared calibration models based on hybrid calibration sets using Partial Least Squares (PLS) regression

Near-infrared spectroscopy (NIR) models built on a particular instrument are often invalid on other instruments due to spectral inconsistencies between the instruments. In the present work, global and robust NIR calibration models were constructed by partial least square (PLS) regression based on hybrid calibration sets, which are composed of both primary and secondary spectra. Three datasets were used as case studies. The first consisted of 72 radix scutellaria samples measured on two NIR spectrometers with known baicalin content. The second was composed of 80 corn samples measured on two instruments with known moisture, oil, and protein concentrations. The third dataset included 279 primary samples of tobacco with known nicotine content and 78 secondary samples of tobacco with known nicotine concentrations. The effect of the number of secondary spectra in the hybrid calibration sets and the methods for selecting secondary spectra on the PLS model performance were investigated by comparing the results obtained from different calibration sets. This study shows that the global and robust calibration models accurately predicted both primary and secondary samples as long as the ratios of the number of primary spectra to the number of secondary spectra were less than 22. The models performance was not influenced by the selection method of the secondary spectra. The hybrid calibration sets included the primary spectral information and also the secondary spectra; information, rendering the constructed global and robust models applicable to both primary and secondary instruments.     

Novel Spectral Interval Selection Method Based on Synchronous Two-Dimensional Correlation Spectroscopy

Synchronous 2D correlation spectroscopy was first proposed to select informational spectral intervals in PLS calibration. The proposed method could extract the spectral intervals related to analyte. The results of its application to NIR/PLS determination of quercetin in extract of Ginkgo biloba leaves showed that the proposed method could find out an optimized region with which one could improve the performance of the corresponding PLS model, in terms of low prediction error, root mean square error of prediction (RMSEP), and comparing with the result obtained using whole spectra and interval PLS.     

A wavelength selection method based on random decision particle swarm optimization with attractor for near‐infrared spectral quantitative analysis

In this paper, we proposed a wavelength selection method based on random decision particle swarm optimization with attractor for near‐infrared (NIR) spectra quantitative analysis. The proposed method was incorporated with partial least square (PLS) to construct a prediction model. The proposed method chooses the current own optimal or the current global optimal to calculate the attractor. Then the particle updates its flight velocity by the attractor, and the particle state is updated by the random decision with the new velocity. Moreover, the root‐mean‐square error of cross‐validation is adopted as the fitness function for the proposed method. In order to demonstrate the usefulness of the proposed method, PLS with all wavelengths, uninformative variable elimination by PLS, elastic net, genetic algorithm combined with PLS, the discrete particle swarm optimization combined with PLS, the modified particle swarm optimization combined with PLS, the neighboring particle swarm optimization combined with PLS, and the proposed method are used for building the components quantitative analysis models of NIR spectral datasets, and the effectiveness of these models is compared. Two application studies are presented, which involve NIR data obtained from an experiment of meat content determination using NIR and a combustion procedure. Results verify that the proposed method has higher predictive ability for NIR spectral data and the number of selected wavelengths is less. The proposed method has faster convergence speed and could overcome the premature convergence problem. Furthermore, although improving the prediction precision may sacrifice the model complexity under a certain extent, the proposed method is overfitted slightly. Copyright © 2015 John Wiley & Sons, Ltd.     

A modification of the successive projections algorithm for spectral variable selection in the presence of unknown interferents

This work proposes a modification to the successive projections algorithm (SPA) aimed at selecting spectral variables for multiple linear regression (MLR) in the presence of unknown interferents not included in the calibration data set. The modified algorithm favours the selection of variables in which the effect of the interferent is less pronounced. The proposed procedure can be regarded as an adaptive modelling technique, because the spectral features of the samples to be analyzed are considered in the variable selection process. The advantages of this new approach are demonstrated in two analytical problems, namely (1) ultraviolet–visible spectrometric determination of tartrazine, allure red and sunset yellow in aqueous solutions under the interference of erythrosine, and (2) near-infrared spectrometric determination of ethanol in gasoline under the interference of toluene. In these case studies, the performance of conventional MLR-SPA models is substantially degraded by the presence of the interferent. This problem is circumvented by applying the proposed Adaptive MLR-SPA approach, which results in prediction errors smaller than those obtained by three other multivariate calibration techniques, namely stepwise regression, full-spectrum partial-least-squares (PLS) and PLS with variables selected by a genetic algorithm. An inspection of the variable selection results reveals that the Adaptive approach successfully avoids spectral regions in which the interference is more intense.