指数型寿命数据中异常值的检验

本文讨论了在指数型寿命数据中,对同时存在的异常大数据和异常小数据的检验方法,给出了一个明确的判别标准,并以一例说明其应用。     

傅里叶变换近红外光谱法测定大麦中蛋白质、淀粉和赖氨酸含量

采用傅里叶变换近红外光谱法测定大麦中蛋白质、淀粉、赖氨酸的含量，并用光谱影响值法(leverage)对异常值进行判断和处理。蛋白质、淀粉和赖氨酸含量近红外光谱分析模型的测定系数R。分别为0．985、0．973和0．978；检验集的化学值与模型预测值的相关系数r分别为0．9853、0．9644和0．9172，分析模型的预测相对标准偏差RSD分别为4．0％、2．4％和5．4％，该结果可替代经典分析方法，满足农产品快速分析的需要。     

最小二乘B样条函数在火箭发动机数据处理中的应用

在固体火箭发动机地面试验数据处理中,通过应用最小二乘三次B样条函数拟合的方法,进行野值辨识和修复。文中阐述了实际应用中的步骤及应注意的几点问题。为避免常规的控制序列中含有野值而影响数据辨识的问题,提出差别选取控制点循环过滤的方法,实现精度更高的野值辨识。进而采用门限判别方法,实现野值的修复。通过实测数据处理的应用表明,该算法对单个和成片野值的辨识度都很高,适用于火箭发动机地面试验数据处理。     

基于Bayes方法的诊断模型分析

文中证明了在Bayes框架下,当漂移参数7服从有信息先验时,在相当广泛的统计模型中,数据删除模型(CDM)和均值漂移模型(MSOM)的参数估计不相等,几个数值例子验证了相应的结论.     

稳健残差控制图的构建及在金融时序中的应用

对于呈现自相关和波动族聚性并存的受控过程,通常采用残差控制图对其进行监控。但异常点的存在会对自相关或波动族聚性模型的拟合产生重要影响,使得基于该模型的残差并非独立同分布导致常规残差控制图监控失效。为解决这类问题,本文提出稳健残差控制图。即建立稳健的ARMA模型解决自相关问题从而得到无自相关的残差序列,用稳健的GARCH模型来构建控制图的上下限。模拟和实证研究表明,本文提出的稳健残差控制图具有很好的抗异常点能力并能更好的对金融时间序列的异常现象进行监控。     

Some closed form robust moment‐based estimators for the MEM(1,1)

In this paper, we extend the closed form moment estimator (ordinary MCFE) for the autoregressive conditional duration model given by Lu et al (2016) and propose some closed form robust moment‐based estimators for the multiplicative error model to deal with the additive and innovational outliers. The robustification of the closed form estimator is done by replacing the sample mean and sample autocorrelation with some robust estimators. These estimators are more robust than the quasi‐maximum likelihood estimator (QMLE) often used to estimate this model, and they are easy to implement and do not require the use of any numerical optimization procedure and the choice of initial value. The performance of our proposal in estimating the parameters and forecasting conditional mean μ_t of the MEM(1,1) process is compared with the proposals existing in the literature via Monte Carlo experiments, and the results of these experiments show that our proposal outperforms the ordinary MCFE, QMLE, and least absolute deviation estimator in the presence of outliers in general. Finally, we fit the price durations of IBM stock with the robust closed form estimators and the benchmarks and analyze their performances in estimating model parameters and forecasting the irregularly spaced intraday Value at Risk.     

综合评价中异常值的识别及无量纲化处理方法

针对综合评价中的异常值现象,讨论了原始数据中是否存在异常值、若存在异常值该如何识别异常值以及对含有异常值的评价数据如何进行无量纲化处理三个问题。关于异常值的判断与识别,给出了以“中位数”为参考点,通过比较排序后两端数据偏离中位数的距离的处理思路。对含有异常值的评价数据的无量纲化处理问题,基于常用的“极值处理法”,通过分别指定异常值和非异常值无量纲化取值区间的方式,提出了一种分段的无量纲化处理方法。最后,通过与已有文献异常值识别及无量纲化处理结果的对比分析,验证了本文方法的有效性,发现本文给出的方法能够实现对异常值的适度筛选,且能够提升无量纲化数据分布均衡性。     

Simple outlier labeling based on quantile regression,with application to the steelmaking process

This paper introduces some methods for outlier identification in the regression setting, motivated by the analysis of steelmaking process data. The proposed methodology extends to the regression setting the boxplot rule, commonly used for outlier screening with univariate data. The focus here is on bivariate settings with a single covariate, but extensions are possible. The proposal is based on quantile regression, including an additional transformation parameter for selecting the best scale for linearity of the conditional quantiles. The resulting method is used to perform effective labeling of potential outliers, with a quite low computational complexity, allowing for simple implementation within statistical software as well as commonly used spreadsheets. Some simulation experiments have been carried out to study the swamping and masking properties of the proposal. The methodology is also illustrated by some real life examples, taking as the response variable the energy consumed in the melting process. Copyright © 2015 John Wiley & Sons, Ltd.     

Statistics-Based Outlier Detection and Correction Method for Amazon Customer Reviews

People nowadays use the internet to project their assessments, impressions, ideas, and observations about various subjects or products on numerous social networking sites. These sites serve as a great source to gather data for data analytics, sentiment analysis, natural language processing, etc. Conventionally, the true sentiment of a customer review matches its corresponding star rating. There are exceptions when the star rating of a review is opposite to its true nature. These are labeled as the outliers in a dataset in this work. The state-of-the-art methods for anomaly detection involve manual searching, predefined rules, or traditional machine learning techniques to detect such instances. This paper conducts a sentiment analysis and outlier detection case study for Amazon customer reviews, and it proposes a statistics-based outlier detection and correction method (SODCM), which helps identify such reviews and rectify their star ratings to enhance the performance of a sentiment analysis algorithm without any data loss. This paper focuses on performing SODCM in datasets containing customer reviews of various products, which are (a) scraped from Amazon.com and (b) publicly available. The paper also studies the dataset and concludes the effect of SODCM on the performance of a sentiment analysis algorithm. The results exhibit that SODCM achieves higher accuracy and recall percentage than other state-of-the-art anomaly detection algorithms.     

SKLOF:一种新的超新星候选范围约减算法

超新星是宇宙学中的“标准烛光”, 其在星系中爆发的概率很低, 是一种特殊、稀少的天体, 只有在大量观测的星系数据中才有机会遇到, 而正处于爆发期的超新星会照亮其整个星系从而在观测获得的星系光谱中具有较明显的特征。但是, 目前已发现的超新星数量相对于大量的天体而言又是非常稀少的, 搜寻它们所用的计算时间成为能否进行后续观测的关键, 因此需要寻找高效率的超新星搜寻方法。对超新星候选范围进行约减的LOF算法的时间复杂度较高, 计算量大, 不适用于大规模数据集。为此通过对LOF算法进行改进, 提出了一种在海量星系光谱中快速约减超新星候范围的新方法(SKLOF)。首先对光谱数据集中离中心点近的数据点进行数据剪枝, 剪掉那些肯定不是超新星候选体的光谱数据对象, 然后利用改进的LOF算法计算剩余的光谱数据的孤立性因子并降序排列进行离群搜索, 最后获得超新星候选体的较小的搜索范围以便进行后续的证认。实验结果表明, 该算法十分有效, 不仅在精确度上有所提高, 而且相比于LOF算法还进一步缩短了算法的运行时间, 提高了算法的执行效率。