ERM learning with unbounded sampling

The learning approach of empirical risk minimization (ERM) is taken for the regression problem in the least square framework. A standard assumption for the error analysis in the literature is the uniform boundedness of the output sampling process. In this paper we abandon this boundedness assumption and conduct error analysis for the ERM learning algorithm with unbounded sampling processes satisfying an increment condition for the moments of the output. The key novelty of our analysis is a covering number argument for estimating the sample error.     

Optimal learning rates for least squares regularized regression with unbounded sampling

A standard assumption in theoretical study of learning algorithms for regression is uniform boundedness of output sample values. This excludes the common case with Gaussian noise. In this paper we investigate the learning algorithm for regression generated by the least squares regularization scheme in reproducing kernel Hilbert spaces without the assumption of uniform boundedness for sampling. By imposing some incremental conditions on moments of the output variable, we derive learning rates in terms of regularity of the regression function and capacity of the hypothesis space. The novelty of our analysis is a new covering number argument for bounding the sample error.     

无界抽样情形下不定核的系数正则化回归

本文讨论样本依赖空间中无界抽样情形下最小二乘损失函数的系数正则化问题. 这里的学习准则与之前再生核Hilbert空间的准则有着本质差异: 核除了满足连续性和有界性之外, 不需要再满足对称性和正定性; 正则化子是函数关于样本展开系数的l²-范数; 样本输出是无界的. 上述差异给误差分析增加了额外难度. 本文的目的是在样本输出不满足一致有界的情形下, 通过l²-经验覆盖数给出误差的集中估计(concentration estimates). 通过引入一个恰当的Hilbert空间以及l²-经验覆盖数的技巧, 得到了与假设空间的容量以及与回归函数的正则性有关的较满意的学习速率.     

Benchmarking state-of-the-art classification algorithms for credit scoring

In this paper, we study the performance of various state-of-the-art classification algorithms applied to eight real-life credit scoring data sets. Some of the data sets originate from major Benelux and UK financial institutions. Different types of classifiers are evaluated and compared. Besides the well-known classification algorithms (eg logistic regression, discriminant analysis, k-nearest neighbour, neural networks and decision trees), this study also investigates the suitability and performance of some recently proposed, advanced kernel-based classification algorithms such as support vector machines and least-squares support vector machines (LS-SVMs). The performance is assessed using the classification accuracy and the area under the receiver operating characteristic curve. Statistically significant performance differences are identified using the appropriate test statistics. It is found that both the LS-SVM and neural network classifiers yield a very good performance, but also simple classifiers such as logistic regression and linear discriminant analysis perform very well for credit scoring.     

基于模糊Adaboost算法的支持向量回归机

针对单一支持向量回归机预测精度不十分良好的问题，结合Adaboost算法以及引入隶属函数，提出了一个基于模糊Aaboost算法的支持向量回归机模型。将该模型应用于金融时间序列预测问题的实验表明，预测精度有一定的提高，从而说明了该模型的有效性和可行性。     

Rates of convergence for partitioning and nearest neighbor regression estimates with unbounded data

Estimation of regression functions from independent and identically distributed data is considered. The L₂ error with integration with respect to the design measure is used as an error criterion. Usually in the analysis of the rate of convergence of estimates besides smoothness assumptions on the regression function and moment conditions on Y also boundedness assumptions on X are made. In this article we consider partitioning and nearest neighbor estimates and show that by replacing the boundedness assumption on X by a proper moment condition the same rate of convergence can be shown as for bounded data.     

SVM-Maj: a majorization approach to linear support vector machines with different hinge errors

Support vector machines (SVM) are becoming increasingly popular for the prediction of a binary dependent variable. SVMs perform very well with respect to competing techniques. Often, the solution of an SVM is obtained by switching to the dual. In this paper, we stick to the primal support vector machine problem, study its effective aspects, and propose varieties of convex loss functions such as the standard for SVM with the absolute hinge error as well as the quadratic hinge and the Huber hinge errors. We present an iterative majorization algorithm that minimizes each of the adaptations. In addition, we show that many of the features of an SVM are also obtained by an optimal scaling approach to regression. We illustrate this with an example from the literature and do a comparison of different methods on several empirical data sets.     

Testing covariates in high-dimensional regression

In a high-dimensional linear regression model, we propose a new procedure for testing statistical significance of a subset of regression coefficients. Specifically, we employ the partial covariances between the response variable and the tested covariates to obtain a test statistic. The resulting test is applicable even if the predictor dimension is much larger than the sample size. Under the null hypothesis, together with boundedness and moment conditions on the predictors, we show that the proposed test statistic is asymptotically standard normal, which is further supported by Monte Carlo experiments. A similar test can be extended to generalized linear models. The practical usefulness of the test is illustrated via an empirical example on paid search advertising.     

线性支持向量顺序回归机的原始问题的解集分析

本文主要对线性支持向量顺序回归机进行理论研究.对其相应原始问题解的存在性唯一性问题进行细致的分析,指明其解集的确切结构,并给出由对偶问题的解求出原始问题的解集的具体步骤.从而为建立理论上完备的线性支持向量顺序回归机提供了依据.     

天然气消费量的偏最小二乘支持向量机预测

结合偏最小二乘法和支持向量机的优缺点,提出基于偏最小二乘支持向量机的天然气消费量预测模型。首先,利用偏最小二乘法确定影响天然气消费量的新综合变量,建立以新综合变量为输入,天然气消费量为输出的支持向量机模型,对天然气消费量进行了预测;然后,与多元回归、偏最小二乘回归、普通支持向量机做误差检验比较,验证该方法的可行性与正确性。结果表明,此天然气消费量预测模型具有较高的精确度和应用价值。     

A review on consistency and robustness properties of support vector machines for heavy-tailed distributions

Support vector machines (SVMs) belong to the class of modern statistical machine learning techniques and can be described as M-estimators with a Hilbert norm regularization term for functions. SVMs are consistent and robust for classification and regression purposes if based on a Lipschitz continuous loss and a bounded continuous kernel with a dense reproducing kernel Hilbert space. For regression, one of the conditions used is that the output variable Y has a finite first absolute moment. This assumption, however, excludes heavy-tailed distributions. Recently, the applicability of SVMs was enlarged to these distributions by considering shifted loss functions. In this review paper, we briefly describe the approach of SVMs based on shifted loss functions and list some properties of such SVMs. Then, we prove that SVMs based on a bounded continuous kernel and on a convex and Lipschitz continuous, but not necessarily differentiable, shifted loss function have a bounded Bouligand influence function for all distributions, even for heavy-tailed distributions including extreme value distributions and Cauchy distributions. SVMs are thus robust in this sense. Our result covers the important loss functions ${\epsilon}$ -insensitive for regression and pinball for quantile regression, which were not covered by earlier results on the influence function. We demonstrate the usefulness of SVMs even for heavy-tailed distributions by applying SVMs to a simulated data set with Cauchy errors and to a data set of large fire insurance claims of Copenhagen Re.     

Multiclass Probability Estimation With Support Vector Machines

Multiclass classification and probability estimation have important applications in data analytics. Support vector machines (SVMs) have shown great success in various real-world problems due to their high classification accuracy. However, one main limitation of standard SVMs is that they do not provide class probability estimates, and thus fail to offer uncertainty measure about class prediction. In this article, we propose a simple yet effective framework to endow kernel SVMs with the feature of multiclass probability estimation. The new probability estimator does not rely on any parametric assumption on the data distribution, therefore, it is flexible and robust. Theoretically, we show that the proposed estimator is asymptotically consistent. Computationally, the new procedure can be conveniently implemented using standard SVM softwares. Our extensive numerical studies demonstrate competitive performance of the new estimator when compared with existing methods such as multiple logistic regression, linear discrimination analysis, tree-based methods, and random forest, under various classification settings. Supplementary materials for this article are available online.     

Parallel decomposition methods for linearly constrained problems subject to simple bound with application to the SVMs training

We consider the convex quadratic linearly constrained problem with bounded variables and with huge and dense Hessian matrix that arises in many applications such as the training problem of bias support vector machines. We propose a decomposition algorithmic scheme suitable to parallel implementations and we prove global convergence under suitable conditions. Focusing on support vector machines training, we outline how these assumptions can be satisfied in practice and we suggest various specific implementations. Extensions of the theoretical results to general linearly constrained problem are provided. We included numerical results on support vector machines with the aim of showing the viability and the effectiveness of the proposed scheme.     

自适应设计广义线性模型极大拟似然估计的渐近性

在对Fisher信息矩阵的最小特征根最一般的假定,响应变量的矩条件尽可能弱和其它正则条件下,证明了自适应设计广义线性模型中极大拟似然估计的强相合性与渐近正态性,同时给出了强收敛速度.     

Creating a quality map of a slate deposit using support vector machines

In this work, we create a quality map of a slate deposit, using the results of an investigation based on surface geology and continuous core borehole sampling. Once the quality of the slate and the location of the sampling points have been defined, different kinds of support vector machines (SVMs)—SVM classification (multiclass one-against-all), ordinal SVM and SVM regression—are used to draw up the quality map. The results are also compared with those for kriging.     

Two smooth support vector machines for $$\varepsilon $$-insensitive regression

In this paper, we propose two new smooth support vector machines for \(\varepsilon \)-insensitive regression. According to these two smooth support vector machines, we construct two systems of smooth equations based on two novel families of smoothing functions, from which we seek the solution to \(\varepsilon \)-support vector regression (\(\varepsilon \)-SVR). More specifically, using the proposed smoothing functions, we employ the smoothing Newton method to solve the systems of smooth equations. The algorithm is shown to be globally and quadratically convergent without any additional conditions. Numerical comparisons among different values of parameter are also reported.     

Efficiency of the OLS estimator in the vicinity of a spatial unit root

Previous results have indicated that the OLS estimator of the vector of regression coefficients can be nearly as efficient as the best linear unbiased estimator when the regression errors follow a spatial process with root in the vicinity of unity. Such results were derived under the assumption of a symmetric weights matrix, which simplifies the analysis considerably, but is very often not satisfied in applications. This paper provides nontrivial generalizations to the important case of nonsymmetric weights matrices.     

Machine learning techniques applied to the determination of osteoporosis incidence in post-menopausal women

Osteoporosis is a disease that mostly affects women in developed countries. It is characterised by reduced bone mineral density (BMD) and results in a higher incidence of fractured or broken bones. In this research we studied the relationship between BMD and diet and lifestyle habits for a sample of 305 post-menopausal women by constructing a non-linear model using the regression support vector machines technique. One aim of this model was to make an initial preliminary estimate of BMD in the studied women (on the basis of a questionnaire with questions mostly on dietary habits) so as to determine whether they needed densitometry testing. A second aim was to determine the factors with the greatest bearing on BMD with a view to proposing dietary and lifestyle improvements. These factors were determined using regression trees applied to the support vector machines predictions.     

主成分分析与支持向量机相结合的区域降水预测应用

将主成分分析和支持向量机回归相结合,以广西5、6月区域平均日降水量作为预报对象,进行区域日降水量预测研究.首先,整理分析大量的T213数值预报产品信息数据进行主成分分析,得到主成分数据序列;其次,根据主成分数据序列建立训练集训练支持向量机,并利用遗传算法优化参数;最后,输入支持向量机所需数据,得到主成分预测结果,建立广西日降水预报模型.实例计算结果表明,支持向量机回归模型比逐步回归模型有更好的预测能力.     

支持向量回归机和多小波变换在数字图像水印算法中的应用

给出一种多小波域上基于支持向量回归机的数字图像水印新算法,算法充分利用了多小波变换和支持向量回归机的优点.通过对新算法实验结果的分析和与其他算法的对比得到:算法对JPEG压缩、模糊处理、锐化等攻击具有很强的鲁棒性.