Penalized Nonparametric Scalar-on-Function Regression via Principal Coordinates

A number of classical approaches to nonparametric regression have recently been extended to the case of functional predictors. This article introduces a new method of this type, which extends intermediate-rank penalized smoothing to scalar-on-function regression. In the proposed method, which we call principal coordinate ridge regression, one regresses the response on leading principal coordinates defined by a relevant distance among the functional predictors, while applying a ridge penalty. Our publicly available implementation, based on generalized additive modeling software, allows for fast optimal tuning parameter selection and for extensions to multiple functional predictors, exponential family-valued responses, and mixed-effects models. In an application to signature verification data, principal coordinate ridge regression, with dynamic time warping distance used to define the principal coordinates, is shown to outperform a functional generalized linear model. Supplementary materials for this article are available online.     

Spline Multiscale Smoothing to Control FDR for Exploring Features of Regression Curves

SiZer (significant zero crossing of the derivatives) is a multiscale smoothing method for exploring trends, maxima, and minima in data. In this article, a regression spline version of SiZer is proposed in a nonparametric regression setting by the fiducial method. The number of knots for spline interpolation is used as the scale parameter of the new SiZer, which controls the smoothness of estimate. In the construction of the new SiZer, multiple testing adjustment is made to control the row-wise false discovery rate (FDR) of SiZer. This adjustment is appealing for exploratory data analysis and has potential to increase the power. A special map is also produced on a continuous scale using p-values to assess the significance of features. Simulations and a real data application are carried out to investigate the performance of the proposed SiZer, in which several comparisons with other existing SiZers are presented. Supplementary materials for this article are available online.     

������ڵľֲ��ͷ������ع鷽��

??Inspired by intuitive meanings of truncated power basis's coefficients, the local penalization based on range's linear decreasing function is given in penalized spline regression model. This method gives less penalization to fitting curve where data is with more volatility, which makes fitted curve controls tradeoff between goodness-of-fit and smoothness better. Simulations show that regression models with local penalized spline obtain lower information rules' scores than global penalized spline when the data is with heteroskedasticity.     

Optimal testing for additivity in multiple nonparametric regression

We consider the problem of testing for additivity in the standard multiple nonparametric regression model. We derive optimal (in the minimax sense) non- adaptive and adaptive hypothesis testing procedures for additivity against the composite nonparametric alternative that the response function involves interactions of second or higher orders separated away from zero in L ²([0, 1] ^d )-norm and also possesses some smoothness properties. In order to shed some light on the theoretical results obtained, we carry out a wide simulation study to examine the finite sample performance of the proposed hypothesis testing procedures and compare them with a series of other tests for additivity available in the literature.     

Bootstrapping the nonparametric ARCH regression model

In this paper we introduce the nonparametric AR(1)–ARCH(1) model and show weak consistency of the Nadaraya–Watson estimators for the model. We propose a residual and a wild bootstrap method and prove weak consistency of the bootstrap estimators.     

The Fast RODEO for Local Polynomial Regression

An open challenge in nonparametric regression is finding fast, computationally efficient approaches to estimating local bandwidths for large datasets, in particular in two or more dimensions. In the work presented here, we introduce a novel local bandwidth estimation procedure for local polynomial regression, which combines the greedy search of the regularization of the derivative expectation operator (RODEO) algorithm with linear binning. The result is a fast, computationally efficient algorithm, which we refer to as the fast RODEO. We motivate the development of our algorithm by using a novel scale-space approach to derive the RODEO. We conclude with a toy example and a real-world example using data from the Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observation (CALIPSO) satellite validation study, where we show the fast RODEO’s improvement in accuracy and computational speed over two other standard approaches.     

Data Adaptive Ridging in Local Polynomial Regression

Abstract

When estimating a regression function or its derivatives, local polynomials are an attractive choice due to their flexibility and asymptotic performance. Seifert and Gasser proposed ridging of local polynomials to overcome problems with variance for random design while retaining their advantages. In this article we present a data-independent rule of thumb and a data-adaptive spatial choice of the ridge parameter in local linear regression. In a framework of penalized local least squares regression, the methods are generalized to higher order polynomials, to estimation of derivatives, and to multivariate designs. The main message is that ridging is a powerful tool for improving the performance of local polynomials. A rule of thumb offers drastic improvements; data-adaptive ridging brings further but modest gains in mean square error.     

LASSO Isotone for High-Dimensional Additive Isotonic Regression

Additive isotonic regression attempts to determine the relationship between a multidimensional observation variable and a response, under the constraint that the estimate is the additive sum of univariate component effects that are monotonically increasing. In this article, we present a new method for such regression called LASSO Isotone (LISO). LISO adapts ideas from sparse linear modeling to additive isotonic regression. Thus, it is viable in many situations with high-dimensional predictor variables, where selection of significant versus insignificant variables is required. We suggest an algorithm involving a modification of the backfitting algorithm CPAV. We give a numerical convergence result, and finally examine some of its properties through simulations. We also suggest some possible extensions that improve performance, and allow calculation to be carried out when the direction of the monotonicity is unknown. Supplemental materials are available online for this article.     

基于惩罚回归样条的积分近似计算与应用

提出一种利用惩罚回归样条拟合被积函数f(x),从而计算复杂积分∫baf(x)dx的新方法.在仅知f(x)带随机扰动的离散数据点集的情况下,利用基于截断幂形式的样条基函数,通过惩罚样条回归,给出函数的多项式拟合结果,再根据该多项式形式便捷计算出积分.模拟和实际应用结果显示该方法计算简单快捷,并具有较好的准确度.     

Penalized Estimation of Free-Knot Splines

Abstract

Polynomial splines are often used in statistical regression models for smooth response functions. When the number and location of the knots are optimized, the approximating power of the spline is improved and the model is nonparametric with locally determined smoothness. However, finding the optimal knot locations is an historically difficult problem. We present a new estimation approach that improves computational properties by penalizing coalescing knots. The resulting estimator is easier to compute than the unpenalized estimates of knot positions, eliminates unnecessary “corners” in the fitted curve, and in simulation studies, shows no increase in the loss. A number of GCV and AIC type criteria for choosing the number of knots are evaluated via simulation.     

A Parallel Algorithm for Large-Scale Nonconvex Penalized Quantile Regression

Penalized quantile regression (PQR) provides a useful tool for analyzing high-dimensional data with heterogeneity. However, its computation is challenging due to the nonsmoothness and (sometimes) the nonconvexity of the objective function. An iterative coordinate descent algorithm (QICD) was recently proposed to solve PQR with nonconvex penalty. The QICD significantly improves the computational speed but requires a double-loop. In this article, we propose an alternative algorithm based on the alternating direction method of multiplier (ADMM). By writing the PQR into a special ADMM form, we can solve the iterations exactly without using coordinate descent. This results in a new single-loop algorithm, which we refer to as the QPADM algorithm. The QPADM demonstrates favorable performance in both computational speed and statistical accuracy, particularly when the sample size n and/or the number of features p are large. Supplementary material for this article is available online.     

非参数回归函数估计的渐近正态性

本文研究了独立或相依样本时非参数回归函数的Ｎａｄａｒａｙａ-Ｗａｔｓｏｎ估计，在简洁合理的条件下，证明了估计量的渐近正态性．获得的结论可在时间序列分析中得到应用．     

Generalized Isotonic Regression

We present a new computational and statistical approach for fitting isotonic models under convex differentiable loss functions through recursive partitioning. Models along the partitioning path are also isotonic and can be viewed as regularized solutions to the problem. Our approach generalizes and subsumes the well-known work of Barlow and Brunk on fitting isotonic regressions subject to specially structured loss functions, and expands the range of loss functions that can be used (e.g., adding Huber loss for robust regression). This is accomplished through an algorithmic adjustment to a recursive partitioning approach recently developed for solving large-scale ?₂-loss isotonic regression problems. We prove that the new algorithm solves the generalized problem while maintaining the favorable computational and statistical properties of the l₂ algorithm. The results are demonstrated on both real and synthetic data in two settings: fitting count data using negative Poisson log-likelihood loss, and fitting robust isotonic regressions using Huber loss. Proofs of theorems and a MATLAB-based software package implementing our algorithm are available in the online supplementary materials.     

Automatic Smoothing Spline Projection Pursuit

Abstract

A highly flexible nonparametric regression model for predicting a response y given covariates {x_k}^d _k=1 is the projection pursuit regression (PPR) model ? = h(x) = β₀ + Σ_jβ_jf_j(α^T _jx) where the f_j , are general smooth functions with mean 0 and norm 1, and Σ^d _k=1α² _kj=1. The standard PPR algorithm of Friedman and Stuetzle (1981) estimates the smooth functions f_j using the supersmoother nonparametric scatterplot smoother. Friedman's algorithm constructs a model with M _max linear combinations, then prunes back to a simpler model of size M ≤ M _max, where M and M _max are specified by the user. This article discusses an alternative algorithm in which the smooth functions are estimated using smoothing splines. The direction coefficients α_j, the amount of smoothing in each direction, and the number of terms M and M _max are determined to optimize a single generalized cross-validation measure.     

Semismooth Newton Coordinate Descent Algorithm for Elastic-Net Penalized Huber Loss Regression and Quantile Regression

We propose an algorithm, semismooth Newton coordinate descent (SNCD), for the elastic-net penalized Huber loss regression and quantile regression in high dimensional settings. Unlike existing coordinate descent type algorithms, the SNCD updates a regression coefficient and its corresponding subgradient simultaneously in each iteration. It combines the strengths of the coordinate descent and the semismooth Newton algorithm, and effectively solves the computational challenges posed by dimensionality and nonsmoothness. We establish the convergence properties of the algorithm. In addition, we present an adaptive version of the “strong rule” for screening predictors to gain extra efficiency. Through numerical experiments, we demonstrate that the proposed algorithm is very efficient and scalable to ultrahigh dimensions. We illustrate the application via a real data example. Supplementary materials for this article are available online.     

半参数回归模型的两步估计——差分法和样条函数法

本文考虑半参数回归模型yi =XTiβ g(ti) ei(i =1 ,2 ,… ,n) .先利用差分的方法估计 ^β ,接着用样条函数的方法定义 ^g(T) ,最后讨论 ^g(t)的性质 .     

一类非参数回归函数估计的性质

设Z_(11),z_(12),…,Z_是在固定点(x_i,y_1),1≤≤n_1,1≤j≤n_2,的n_1n_2个观察值,适合模型 Z_(ij)=g(x_i,y_j)+ε_(ij),1≤i≤n_1,1≤j≤n_2。(1) 本文给出了g的一种估计并讨论了估计的性质。     

Improved Model Selection Method for a Regression Function with Dependent Noise

This paper is devoted to nonparametric estimation, through the -risk, of a regression function based on observations with spherically symmetric errors, which are dependent random variables (except in the normal case). We apply a model selection approach using improved estimates. In a nonasymptotic setting, an upper bound for the risk is obtained (oracle inequality). Moreover asymptotic properties are given, such as upper and lower bounds for the risk, which provide optimal rate of convergence for penalized estimators.     

函数型数据回归分析综述

随着计算机储存能力和在线观测技术的提高,当今数据越来越多的以曲线和图像的形式存在.曲线和图像数据两个最显著的特征是高维和相邻数据间高度相关.这些特征使得传统的多元统计分析方法不再适合,而函数型数据在处理曲线和图像数据中具有无可比拟的优势.近年来各种各样的函数型数据分析方法得以发展,其中包括数据的对齐、主成分分析、回归、分类、聚类等.本文主要介绍函数型数据回归分析研究的起源、发展及最新进展.具体地,本文首先介绍函数型数据的概念;其次介绍函数型主成分分析方法;再次着重介绍函数型回归模型的估计、变量选择和检验方法;最后将简要探讨函数型数据未来的可能发展方向.     

Robust Depth-Weighted Wavelet for Nonparametric Regression Models

In the nonparametric regression models, the original regression estimators including kernel estimator, Fourier series estimator and wavelet estimator are always constructed by the weighted sum of data, and the weights depend only on the distance between the design points and estimation points. As a result these estimators are not robust to the perturbations in data. In order to avoid this problem, a new nonparametric regression model, called the depth-weighted regression model, is introduced and then the depth-weighted wavelet estimation is defined. The new estimation is robust to the perturbations in data, which attains very high breakdown value close to 1/2. On the other hand, some asymptotic behaviours such as asymptotic normality are obtained. Some simulations illustrate that the proposed wavelet estimator is more robust than the original wavelet estimator and, as a price to pay for the robustness, the new method is slightly less efficient than the original method.