Testing covariates in high-dimensional regression

In a high-dimensional linear regression model, we propose a new procedure for testing statistical significance of a subset of regression coefficients. Specifically, we employ the partial covariances between the response variable and the tested covariates to obtain a test statistic. The resulting test is applicable even if the predictor dimension is much larger than the sample size. Under the null hypothesis, together with boundedness and moment conditions on the predictors, we show that the proposed test statistic is asymptotically standard normal, which is further supported by Monte Carlo experiments. A similar test can be extended to generalized linear models. The practical usefulness of the test is illustrated via an empirical example on paid search advertising.     

Least-squares regularized regression with dependent samples and q-penalty

Least-squares regularized learning algorithms for regression were well-studied in the literature when the sampling process is independent and the regularization term is the square of the norm in a reproducing kernel Hilbert space (RKHS). Some analysis has also been done for dependent sampling processes or regularizers being the qth power of the function norm (q-penalty) with 0?q?≤?2. The purpose of this article is to conduct error analysis of the least-squares regularized regression algorithm when the sampling sequence is weakly dependent satisfying an exponentially decaying α-mixing condition and when the regularizer takes the q-penalty with 0?q?≤?2. We use a covering number argument and derive learning rates in terms of the α-mixing decay, an approximation condition and the capacity of balls of the RKHS.     

A regularized method for solving constrained pseudoinverse problems

For a constrained pseudoinverse problem whose operators satisfy the complementarity condition we propose a one-parameter continuous regularization method of the second order. This method is based on stabilization of solutions to Cauchy problems for a linear differential equation of the second order in a Hilbert space which is obtained from the heavy ball method. We establish requirements to the parametric regularization function and perturbation levels that ensure the stability of the method in the class of all possible bounded perturbations.     

正则化回归算法的快速学习率

本文讨论了再生核Hilbert 空间上一类广泛的正则化回归算法的学习率问题. 在分析算法的样本误差时, 我们利用了一种复加权的经验过程, 保证了方差与惩罚泛函同时被阈值控制, 从而避免了繁琐的迭代过程. 本文得到了比之前文献结果更为快速的学习率.     

Low-rank elastic-net regularized multivariate Huber regression model

Heavy-tailed noise or strongly correlated predictors often go with the multivariate linear regression model. To tackle with these problems, this paper focuses on the matrix elastic-net regularized multivariate Huber regression model. This new model possesses the grouping effect property and the robustness to heavy-tailed noise. Meanwhile, it also has the ability of reducing the negative effect of outliers due to Huber loss. Furthermore, an accelerated proximal gradient algorithm is designed to solve the proposed model. Some numerical studies including a real data analysis are dedicated to show the efficiency of our method.     

Nonlinear logistic discrimination via regularized radial basis functions for classifying high-dimensional data

A flexible nonparametric method is proposed for classifying high- dimensional data with a complex structure. The proposed method can be regarded as an extended version of linear logistic discriminant procedures, in which the linear predictor is replaced by a radial-basis-expansion predictor. Radial basis functions with a hyperparameter are used to take the information on covariates and class labels into account; this was nearly impossible within the previously proposed hybrid learning framework. The penalized maximum likelihood estimation procedure is employed to obtain stable parameter estimates. A crucial issue in the model-construction process is the choice of a suitable model from candidates. This issue is examined from information-theoretic and Bayesian viewpoints and we employed Ando et al. (Japanese Journal of Applied Statistics, 31, 123–139, 2002)’s model evaluation criteria. The proposed method is available not only for the high-dimensional data but also for the variable selection problem. Real data analysis and Monte Carlo experiments show that our proposed method performs well in classifying future observations in practical situations. The simulation results also show that the use of the hyperparameter in the basis functions improves the prediction performance.     

Generalization errors of Laplacian regularized least squares regression

Semi-supervised learning is an emerging computational paradigm for machine learning,that aims to make better use of large amounts of inexpensive unlabeled data to improve the learning performance.While various methods have been proposed based on different intuitions,the crucial issue of generalization performance is still poorly understood.In this paper,we investigate the convergence property of the Laplacian regularized least squares regression,a semi-supervised learning algorithm based on manifold regularization.Moreover,the improvement of error bounds in terms of the number of labeled and unlabeled data is presented for the first time as far as we know.The convergence rate depends on the approximation property and the capacity of the reproducing kernel Hilbert space measured by covering numbers.Some new techniques are exploited for the analysis since an extra regularizer is introduced.     

Inequality constrained ridge regression estimator

We carry out the idea of inequality constrained least squares (ICLS) estimation of Liew (1976) to the inequality constrained ridge regression (ICRR) estimation. We propose ICRR estimator by reducing the primal–dual relation to the fundamental problem of Dantzig and Cottle, 1967, Cottle and Dantzig, 1974 with Lemke (1962) algorithm. Furthermore, we conduct a Monte Carlo experiment.     

Nonlinear regression modeling using regularized local likelihood method

We introduce a nonlinear regression modeling strategy, using a regularized local likelihood method. The local likelihood method is effective for analyzing data with complex structure. It might be, however, pointed out that the stability of the local likelihood estimator is not necessarily guaranteed in the case that the structure of system is quite complex. In order to overcome this difficulty, we propose a regularized local likelihood method with a polynomial function which unites local likelihood and regularization. A crucial issue in constructing nonlinear regression models is the choice of a smoothing parameter, the degree of polynomial and a regularization parameter. In order to evaluate models estimated by the regularized local likelihood method, we derive a model selection criterion from an information-theoretic point of view. Real data analysis and Monte Carlo experiments are conducted to examine the performance of our modeling strategy.     

Functional regression modeling via regularized Gaussian basis expansions

We consider the problem of constructing functional regression models for scalar responses and functional predictors, using Gaussian basis functions along with the technique of regularization. An advantage of our regularized Gaussian basis expansions to functional data analysis is that it creates a much more flexible instrument for transforming each individual’s observations into functional form. In constructing functional regression models there remains the problem of how to determine the number of basis functions and an appropriate value of a regularization parameter. We present model selection criteria for evaluating models estimated by the method of regularization in the context of functional regression models. The proposed functional regression models are applied to Canadian temperature data. Monte Carlo simulations are conducted to examine the efficiency of our modeling strategies. The simulation results show that the proposed procedure performs well especially in terms of flexibility and stable estimates.     

Consistent tuning parameter selection in high-dimensional group-penalized regression

Science China Mathematics - Various forms of penalized estimators with good statistical and computational properties have been proposed for variable selection respecting the grouping structure in...     

Robust high-dimensional regression for data with anomalous responses

Annals of the Institute of Statistical Mathematics - The accuracy of response variables is crucially important to train regression models. In some situations, including the high-dimensional case,...     

高维约束矩阵回归问题

高维约束矩阵回归是指高维情况下带非凸约束的多响应多预测统计回归问题,其数学模型是一个NP-难的矩阵优化,它在机器学习与人工智能、医学影像疾病诊疗、基因表达分析、脑神经网络、风险管理等领域有广泛应用.从高维约束矩阵回归的优化理论和算法两方面总结和评述这些新成果,同时,列出了相应的重要文献.     

Bayesian regularized regression based on composite quantile method

Acta Mathematicae Applicatae Sinica, English Series - Recently, variable selection based on penalized regression methods has received a great deal of attention, mostly through frequentist’s...     

Application of integral operator for regularized least-square regression

In this paper, we study the consistency of the regularized least-square regression in a general reproducing kernel Hilbert space. We characterize the compactness of the inclusion map from a reproducing kernel Hilbert space to the space of continuous functions and show that the capacity-based analysis by uniform covering numbers may fail in a very general setting. We prove the consistency and compute the learning rate by means of integral operator techniques. To this end, we study the properties of the integral operator. The analysis reveals that the essence of this approach is the isomorphism of the square root operator.     

Nonlinear regression modeling via regularized wavelets and smoothing parameter selection

We introduce regularized wavelet-based methods for nonlinear regression modeling when design points are not equally spaced. A crucial issue in the model building process is a choice of tuning parameters that control the smoothness of a fitted curve. We derive model selection criteria from an information-theoretic and also Bayesian approaches. Monte Carlo simulations are conducted to examine the performance of the proposed wavelet-based modeling technique.     

Learning rates of regularized regression on the unit sphere

This paper addresses the learning algorithm on the unit sphere. The main purpose is to present an error analysis for regression generated by regularized least square algorithms with spherical harmonics kernel. The excess error can be estimated by the sum of sample errors and regularization errors. Our study shows that by introducing a suitable spherical harmonics kernel, the regularization parameter can decrease arbitrarily fast with the sample size.     

Learning rates of least-square regularized regression with polynomial kernels

This paper presents learning rates for the least-square regularized regression algorithms with polynomial kernels. The target is the error analysis for the regression problem in learning theory. A regularization scheme is given, which yields sharp learning rates. The rates depend on the dimension of polynomial space and polynomial reproducing kernel Hilbert space measured by covering numbers. Meanwhile, we also establish the direct approximation theorem by Bernstein-Durrmeyer operators in with Borel probability measure.      

无界抽样情形下不定核的系数正则化回归

本文讨论样本依赖空间中无界抽样情形下最小二乘损失函数的系数正则化问题. 这里的学习准则与之前再生核Hilbert空间的准则有着本质差异: 核除了满足连续性和有界性之外, 不需要再满足对称性和正定性; 正则化子是函数关于样本展开系数的l²-范数; 样本输出是无界的. 上述差异给误差分析增加了额外难度. 本文的目的是在样本输出不满足一致有界的情形下, 通过l²-经验覆盖数给出误差的集中估计(concentration estimates). 通过引入一个恰当的Hilbert空间以及l²-经验覆盖数的技巧, 得到了与假设空间的容量以及与回归函数的正则性有关的较满意的学习速率.