Optimal rates for spectral algorithms with least-squares regression over Hilbert spaces

In this paper, we study regression problems over a separable Hilbert space with the square loss, covering non-parametric regression over a reproducing kernel Hilbert space. We investigate a class of spectral/regularized algorithms, including ridge regression, principal component regression, and gradient methods. We prove optimal, high-probability convergence results in terms of variants of norms for the studied algorithms, considering a capacity assumption on the hypothesis space and a general source condition on the target function. Consequently, we obtain almost sure convergence results with optimal rates. Our results improve and generalize previous results, filling a theoretical gap for the non-attainable cases.     

A harmonic polynomial method with a regularization strategy for the boundary value problems of Laplace’s equation

This paper concerns a boundary value problem of Laplace’s equation, which is solved by determining the unknown coefficients in the expansion of harmonic polynomials. A regularization method is proposed to tackle the resulting ill-posed linear system. The stability and convergence results are provided and a validating numerical experiment is presented.     

An approximate solution for nonlinear backward parabolic equations

We consider the backward parabolic equation

     

A wavelet-Galerkin method for high order numerical differentiation

Numerical differentiation is a classical ill-posed problem. In this paper, we propose a wavelet-Galerkin method for high order numerical differentiation. By an appropriate choice of the regularization parameter an order optimal stability estimate of Hölder type is obtained. Some numerical examples show that the method is effective and stable.     

Theory of amorphous semiconductors

The basic concepts underlying the present theory of amorphous semiconductors are reviewed. Emphasis is put on simple band models.     

ConvexLAR: An Extension of Least Angle Regression

The least angle regression (LAR) was proposed by Efron, Hastie, Johnstone and Tibshirani in the year 2004 for continuous model selection in linear regression. It is motivated by a geometric argument and tracks a path along which the predictors enter successively and the active predictors always maintain the same absolute correlation (angle) with the residual vector. Although it gains popularity quickly, its extensions seem rare compared to the penalty methods. In this expository article, we show that the powerful geometric idea of LAR can be generalized in a fruitful way. We propose a ConvexLAR algorithm that works for any convex loss function and naturally extends to group selection and data adaptive variable selection. After simple modification, it also yields new exact path algorithms for certain penalty methods such as a convex loss function with lasso or group lasso penalty. Variable selection in recurrent event and panel count data analysis, Ada-Boost, and Gaussian graphical model is reconsidered from the ConvexLAR angle. Supplementary materials for this article are available online.     

Generalized Isotonic Regression

We present a new computational and statistical approach for fitting isotonic models under convex differentiable loss functions through recursive partitioning. Models along the partitioning path are also isotonic and can be viewed as regularized solutions to the problem. Our approach generalizes and subsumes the well-known work of Barlow and Brunk on fitting isotonic regressions subject to specially structured loss functions, and expands the range of loss functions that can be used (e.g., adding Huber loss for robust regression). This is accomplished through an algorithmic adjustment to a recursive partitioning approach recently developed for solving large-scale ?₂-loss isotonic regression problems. We prove that the new algorithm solves the generalized problem while maintaining the favorable computational and statistical properties of the l₂ algorithm. The results are demonstrated on both real and synthetic data in two settings: fitting count data using negative Poisson log-likelihood loss, and fitting robust isotonic regressions using Huber loss. Proofs of theorems and a MATLAB-based software package implementing our algorithm are available in the online supplementary materials.     

Automatic Feature Selection via Weighted Kernels and Regularization

Selecting important features in nonlinear kernel spaces is a difficult challenge in both classification and regression problems. This article proposes to achieve feature selection by optimizing a simple criterion: a feature-regularized loss function. Features within the kernel are weighted, and a lasso penalty is placed on these weights to encourage sparsity. This feature-regularized loss function is minimized by estimating the weights in conjunction with the coefficients of the original classification or regression problem, thereby automatically procuring a subset of important features. The algorithm, KerNel Iterative Feature Extraction (KNIFE), is applicable to a wide variety of kernels and high-dimensional kernel problems. In addition, a modification of KNIFE gives a computationally attractive method for graphically depicting nonlinear relationships between features by estimating their feature weights over a range of regularization parameters. The utility of KNIFE in selecting features through simulations and examples for both kernel regression and support vector machines is demonstrated. Feature path realizations also give graphical representations of important features and the nonlinear relationships among variables. Supplementary materials with computer code and an appendix on convergence analysis are available online.     

Automatic Smoothing With Wavelets for a Wide Class of Distributions

Wavelet-based denoising techniques are well suited to estimate spatially inhomogeneous signals. Waveshrink (Donoho and Johnstone) assumes independent Gaussian errors and equispaced sampling of the signal. Various articles have relaxed some of these assumptions, but a systematic generalization to distributions such as Poisson, binomial, or Bernoulli is missing. We consider a unifying l₁-penalized likelihood approach to regularize the maximum likelihood estimation by adding an l₁ penalty of the wavelet coefficients. Our approach works for all types of wavelets and for a range of noise distributions. We develop both an algorithm to solve the estimation problem and rules to select the smoothing parameter automatically. In particular, using results from Poisson processes, we give an explicit formula for the universal smoothing parameter to denoise Poisson measurements. Simulations show that the procedure is an improvement over other methods. An astronomy example is given.     

A Hessian Regularized Nonlinear Time Series Model

We introduce a nonparametric nonlinear time series model. The novel idea is to fit a model via penalization, where the penalty term is an unbiased estimator of the integrated Hessian of the underlying function. The underlying model assumption is very general: it has Hessian almost everywhere in its domain. Numerical experiments demonstrate that our model has better predictive power: if the underlying model complies with an existing parametric/semiparametric form (e.g., a threshold autoregressive model (TAR), an additive autoregressive model (AAR), or a functional coefficient autoregressive model (FAR)), our model performs comparably; if the underlying model does not comply with any preexisting form, our model outperforms in nearly all simulations. We name our model a Hessian regularized nonlinear model for time series (HRM). We conjecture on theoretical properties and use simulations to verify. Our method can be viewed as a way to generalize splines to high dimensions (when the number of variates is more than three), under which an analogous analytical derivation cannot work due to the curse of dimensionality. Supplemental materials are provided, and will help readers reproduce all results in the article.