Risk bounds for model selection via penalization

Performance bounds for criteria for model selection are developed using recent theory for sieves. The model selection criteria are based on an empirical loss or contrast function with an added penalty term motivated by empirical process theory and roughly proportional to the number of parameters needed to describe the model divided by the number of observations. Most of our examples involve density or regression estimation settings and we focus on the problem of estimating the unknown density or regression function. We show that the quadratic risk of the minimum penalized empirical contrast estimator is bounded by an index of the accuracy of the sieve. This accuracy index quantifies the trade-off among the candidate models between the approximation error and parameter dimension relative to sample size. If we choose a list of models which exhibit good approximation properties with respect to different classes of smoothness, the estimator can be simultaneously minimax rate optimal in each of those classes. This is what is usually called adaptation. The type of classes of smoothness in which one gets adaptation depends heavily on the list of models. If too many models are involved in order to get accurate approximation of many wide classes of functions simultaneously, it may happen that the estimator is only approximately adaptive (typically up to a slowly varying function of the sample size). We shall provide various illustrations of our method such as penalized maximum likelihood, projection or least squares estimation. The models will involve commonly used finite dimensional expansions such as piecewise polynomials with fixed or variable knots, trigonometric polynomials, wavelets, neural nets and related nonlinear expansions defined by superposition of ridge functions. Received: 7 July 1995 / Revised version: 1 November 1997     

Nonlinear wavelet estimation of regression function with random desigm

The nonlinear wavelet estimator of regression function with random design is constructed. The optimal uniform convergence rate of the estimator in a ball of Besov spaceB ³ _p,q is proved under quite general assumpations. The adaptive nonlinear wavelet estimator with near-optimal convergence rate in a wide range of smoothness function classes is also constructed. The properties of the nonlinear wavelet estimator given for random design regression and only with bounded third order moment of the error can be compared with those of nonlinear wavelet estimator given in literature for equal-spaced fixed design regression with i.i.d. Gauss error. Project supported by Doctoral Programme Foundation, the National Natural Science Foundation of China (Grant No. 19871003) and Natural Science Fundation of Heilongjiang Province, China.     

Wavelet Threshold Estimation of a Regression Function with Random Design

The wavelet threshold estimator of a regression function for the random design is constructed. The optimal uniform convergence rate of the estimator in a ball of Besov Space B^s_p, q is proved under general assumptions. The adaptive wavelet threshold estimator with near-optimal convergence rate in a wide range of Besov scale is also constructed.     

Upper bounds for the L 1-risk of the minimum L 1-distance regression estimator

A new estimator of a regression function is introduced via minimizing the L ₁-distance between some empirical function and its theoretical counterpart plus penalty for the roughness. The L ₁-risk of the estimator is bounded from above for every sample size no matter what the dependence structure of the observed random variables is. In the case of independent errors of measurement with a common variance the estimator is shown to achieve the optimal L ₁-rate of convergence within the class of m-times differentiable functions with bounded derivatives.     

Adaptive estimation in circular functional linear models

We consider the problem of estimating the slope parameter in circular functional linear regression, where scalar responses Y ₁, ..., Y _n are modeled in dependence of 1-periodic, second order stationary random functions X ₁, ...,X _n . We consider an orthogonal series estimator of the slope function β, by replacing the first m theoretical coefficients of its development in the trigonometric basis by adequate estimators. We propose a model selection procedure for m in a set of admissible values, by defining a contrast function minimized by our estimator and a theoretical penalty function; this first step assumes the degree of ill-posedness to be known. Then we generalize the procedure to a random set of admissible m’s and a random penalty function. The resulting estimator is completely data driven and reaches automatically what is known to be the optimal minimax rate of convergence, in terms of a general weighted L ²-risk. This means that we provide adaptive estimators of both β and its derivatives.     

Estimating a Distribution Function for Censored Time Series Data

Consider a long term study, where a series of dependent and possibly censored failure times is observed. Suppose that the failure times have a common marginal distribution function, but they exhibit a mode of time series structure such as α-mixing. The inference on the marginal distribution function is of interest to us. The main results of this article show that, under some regularity conditions, the Kaplan–Meier estimator enjoys uniform consistency with rates, and a stochastic process generated by the Kaplan–Meier estimator converges weakly to a certain Gaussian process with a specified covariance structure. Finally, an estimator of the limiting variance of the Kaplan–Meier estimator is proposed and its consistency is established.     

PA误差下回归函数小波估计的渐近性质

本文研究了回归函数小波估计的渐进性质的问题.利用概率不等式方法,获得了函数g(·)的小波估计量的r-阶矩相合,依概率收敛和强收敛以及渐进正态性的结果,所获的结果推广了其他混合相依下的相应结果.     

Estimating and Visualizing Conditional Densities

Abstract

We consider the kernel estimator of conditional density and derive its asymptotic bias, variance, and mean-square error. Optimal bandwidths (with respect to integrated mean-square error) are found and it is shown that the convergence rate of the density estimator is order n ^–2/3. We also note that the conditional mean function obtained from the estimator is equivalent to a kernel smoother. Given the undesirable bias properties of kernel smoothers, we seek a modified conditional density estimator that has mean equivalent to some other nonparametric regression smoother with better bias properties. It is also shown that our modified estimator has smaller mean square error than the standard estimator in some commonly occurring situations. Finally, three graphical methods for visualizing conditional density estimators are discussed and applied to a data set consisting of maximum daily temperatures in Melbourne, Australia.     

Estimation of intrinsic growth factors in a class of stochastic population model

This article discusses the problem of parameter estimation with nonlinear mean-reversion type stochastic differential equations (SDEs) driven by Brownian motion for population growth model. The estimator in the population model is the climate effects, population policy and environmental circumstances which affect the intrinsic rate of growth r. The consistency and asymptotic distribution of the estimator θ is studied in our general setting. In the calculation method, unlike previous study, since the nonlinear feature of the model, it is difficult to obtain an explicit formula for the estimator. To solve this, some criteria are used to derive an asymptotically consistent estimator. Furthermore Girsanov transformation is used to simplify the equations, which then gives rise to the corresponding convergence of the estimator being with respect to a family of probability measures indexed by the dispersion parameter, while in the literature the existing results have dealt with convergence with respect to a given probability measure.     

L 2 formulas for entire functions of exponential type

We express the integral of |f(x)|²,−∞<x<∞, as a summation involving different kind of nodes. Heref is an entire function of exponential type satisfying a certain growth condition. The method of proof uses interpolation formulas and orthogonality properties for some classes of entire functions of exponential type.     

On universal estimators in learning theory

This paper addresses the problem of constructing and analyzing estimators for the regression problem in supervised learning. Recently, there has been great interest in studying universal estimators. The term “universal” means that, on the one hand, the estimator does not depend on the a priori assumption that the regression function f _ρ belongs to some class F from a collection of classes F and, on the other hand, the estimation error for f _ρ is close to the optimal error for the class F. This paper is an illustration of how the general technique of constructing universal estimators, developed in the author’s previous paper, can be applied in concrete situations. The setting of the problem studied in the paper has been motivated by a recent paper by Smale and Zhou. The starting point for us is a kernel K(x, u) defined on X × Ω. On the base of this kernel, we build an estimator that is universal for classes defined in terms of nonlinear approximations with regard to the system {K(·, u)} _uεΩ. To construct an easily implementable estimator, we apply the relaxed greedy algorithm. Published in Russian in Trudy Matematicheskogo Instituta imeni V.A. Steklova, 2006, Vol. 255, pp. 256–272.     

Estimating the error distribution in nonparametric multiple regression with applications to model testing

In this paper we consider the estimation of the error distribution in a heteroscedastic nonparametric regression model with multivariate covariates. As estimator we consider the empirical distribution function of residuals, which are obtained from multivariate local polynomial fits of the regression and variance functions, respectively. Weak convergence of the empirical residual process to a Gaussian process is proved. We also consider various applications for testing model assumptions in nonparametric multiple regression. The model tests obtained are able to detect local alternatives that converge to zero at an n^−1/2-rate, independent of the covariate dimension. We consider in detail a test for additivity of the regression function.     

Consistency of an estimator of doubly nondecreasing regression functions

Consistency of a well known isotonic type estimator of doubly nondecreasing regression functions is obtained under hypotheses suggested by anr ^th order Kolmogorov condition, when finite moments of order greater than two are assumed, and under hypotheses that include uniformly bounded second moments. Observations are taken at points on a rectangular grid.     

Wavelet Estimator of Long-Range Dependent Processes

In this contribution, the statistical properties of the wavelet estimator of the long-range dependence parameter introduced in Abry et al. (1995) are discussed for a stationary Gaussian process. This contribution complements the heuristical discussion presented in Abry et al. (1999), by taking into account the correlation between the wavelet coefficients (which is discarded in the mentioned reference) and the bias due to the short-memory component. We derive expressions for the estimators asymptotic bias, variance and mean-squared error as functions of the scale used in the regression and some user-defined parameters. Consistency of the estimator is obtained as long as the scale index j _T goes to infinity and 2^j _T /T0, where T denotes the sample size. Under these and some additional conditions assumed in the paper, we also establish the asymptotic normality of this estimator.     

Edgeworth expansion in censored linear regression model

For the censored simple linear regression model, we establish a oneterm Edgeworth expansion for the Koul, Susarla and Van Ryzin type estimator of the regression coefficient. Our approach is to represent the estimator of the regression coefficient as an asymptoticU-statistic plus some ignorable terms and hence apply the known results on the Edgeworth expansions for asymptoticU-statistic. The counting process and martingale techniques are used to provide the proof of the main results.     

Log-det approximation based on uniformly distributed seeds and its application to Gaussian process regression

Maximum likelihood estimation (MLE) of hyperparameters in Gaussian process regression as well as other computational models usually and frequently requires the evaluation of the logarithm of the determinant of a positive-definite matrix (denoted by C hereafter). In general, the exact computation of is of O(N³) operations where N is the matrix dimension. The approximation of could be developed with O(N²) operations based on power-series expansion and randomized trace estimator. In this paper, the accuracy and effectiveness of using uniformly distributed seeds for approximation are investigated. The research shows that uniform-seed based approximation is an equally good alternative to Gaussian-seed based approximation, having slightly better approximation accuracy and smaller variance. Gaussian process regression examples also substantiate the effectiveness of such a uniform-seed based log-det approximation scheme.     

Estimation in the nonlinear errors-in-variables model

In the nonlinear structural errors-in-variables model we propose a consistent estimator of the unknown parameter, using a modified least squares criterion. Its rate of convergence strongly related to the regularity of the regression function, is generally slower than the parametric rate of convergence n^−1/2. Nevertheless, it is of order (log n)r/√n, r > 0, for some analytic regression functions.     

An algorithm for the estimation of a regression function by continuous piecewise linear functions

The problem of the estimation of a regression function by continuous piecewise linear functions is formulated as a nonconvex, nonsmooth optimization problem. Estimates are defined by minimization of the empirical L ₂ risk over a class of functions, which are defined as maxima of minima of linear functions. An algorithm for finding continuous piecewise linear functions is presented. We observe that the objective function in the optimization problem is semismooth, quasidifferentiable and piecewise partially separable. The use of these properties allow us to design an efficient algorithm for approximation of subgradients of the objective function and to apply the discrete gradient method for its minimization. We present computational results with some simulated data and compare the new estimator with a number of existing ones.     

Large deviation results for the nonparametric regression function estimator on functional data

This paper is devoted to the study of large deviation behavior in the setting of the estimation of the regression function on functional data. A large deviation principle is stated for a process Z _n , defined below, allowing to derive a pointwise large deviation principle for the Nadaraya- Watson-type l-indexed regression function estimator as a by-product. Moreover, a uniform over VC-classes Chernoff type large deviation result is stated for the deviation of the l-indexed regression estimator.     

Nonparametric Estimation for Semi-Markov Processes Based on its Hazard Rate Functions

The problem of estimating the Markov renewal matrix and the semi-Markov transition matrix based on a history of a finite semi-Markov process censored at time T (fixed) is addressed for the first time. Their asymptotic properties are studied. We begin by the definition of the transition rate of this process and propose a maximum likelihood estimator for the hazard rate functions and then we show that this estimator is uniformly strongly consistent and converges weakly to a normal random variable. We construct a new estimator for an absolute continous semi-Markov kernel and give detailed derivation of uniform strong consistency and weak convergence of this estimator as the censored time tends to infinity. This revised version was published online in June 2006 with corrections to the Cover Date.