Simple and efficient improvements of multivariate local linear regression

This paper studies improvements of multivariate local linear regression. Two intuitively appealing variance reduction techniques are proposed. They both yield estimators that retain the same asymptotic conditional bias as the multivariate local linear estimator and have smaller asymptotic conditional variances. The estimators are further examined in aspects of bandwidth selection, asymptotic relative efficiency and implementation. Their asymptotic relative efficiencies with respect to the multivariate local linear estimator are very attractive and increase exponentially as the number of covariates increases. Data-driven bandwidth selection procedures for the new estimators are straightforward given those for local linear regression. Since the proposed estimators each has a simple form, implementation is easy and requires much less or about the same amount of effort. In addition, boundary corrections are automatic as in the usual multivariate local linear regression.     

Nonparametric variance function estimation with missing data

In this paper, a fixed design regression model where the errors follow a strictly stationary process is considered. In this model the conditional mean function and the conditional variance function are unknown curves. Correlated errors when observations are missing in the response variable are assumed. Four nonparametric estimators of the conditional variance function based on local polynomial fitting are proposed. Expressions of the asymptotic bias and variance of these estimators are obtained. A simulation study illustrates the behavior of the proposed estimators.     

Functional-coefficient partially linear regression model

In this paper, the functional-coefficient partially linear regression (FCPLR) model is proposed by combining nonparametric and functional-coefficient regression (FCR) model. It includes the FCR model and the nonparametric regression (NPR) model as its special cases. It is also a generalization of the partially linear regression (PLR) model obtained by replacing the parameters in the PLR model with some functions of the covariates. The local linear technique and the integrated method are employed to give initial estimators of all functions in the FCPLR model. These initial estimators are asymptotically normal. The initial estimator of the constant part function shares the same bias as the local linear estimator of this function in the univariate nonparametric model, but the variance of the former is bigger than that of the latter. Similarly, initial estimators of every coefficient function share the same bias as the local linear estimates in the univariate FCR model, but the variance of the former is bigger than that of the latter. To decrease the variance of the initial estimates, a one-step back-fitting technique is used to obtain the improved estimators of all functions. The improved estimator of the constant part function has the same asymptotic normality property as the local linear nonparametric regression for univariate data. The improved estimators of the coefficient functions have the same asymptotic normality properties as the local linear estimates in FCR model. The bandwidths and the smoothing variables are selected by a data-driven method. Both simulated and real data examples related to nonlinear time series modeling are used to illustrate the applications of the FCPLR model.     

Multivariate Locally Weighted Polynomial Fitting and Partial Derivative Estimation

Nonparametric regression estimator based on locally weighted least squares fitting has been studied by Fan and Ruppert and Wand. The latter paper also studies, in the univariate case, nonparametric derivative estimators given by a locally weighted polynomial fitting. Compared with traditional kernel estimators, these estimators are often of simpler form and possess some better properties. In this paper, we develop current work on locally weighted regression and generalize locally weighted polynomial fitting to the estimation of partial derivatives in a multivariate regression context. Specifically, for both the regression and partial derivative estimators we prove joint asymptotic normality and derive explicit asymptotic expansions for their conditional bias and conditional convariance matrix (given observations of predictor variables) in each of the two important cases of local linear fit and local quadratic fit.     

Model checking in errors-in-variables regression

This paper discusses a class of minimum distance tests for fitting a parametric regression model to a class of regression functions in the errors-in-variables model. These tests are based on certain minimized distances between a nonparametric regression function estimator and a deconvolution kernel estimator of the conditional expectation of the parametric model being fitted. The paper establishes the asymptotic normality of the proposed test statistics under the null hypothesis and that of the corresponding minimum distance estimators. We also prove the consistency of the proposed tests against a fixed alternative and obtain the asymptotic distributions for general local alternatives. Simulation studies show that the testing procedures are quite satisfactory in the preservation of the finite sample level and in terms of a power comparison.     

Estimating the error distribution in nonparametric multiple regression with applications to model testing

In this paper we consider the estimation of the error distribution in a heteroscedastic nonparametric regression model with multivariate covariates. As estimator we consider the empirical distribution function of residuals, which are obtained from multivariate local polynomial fits of the regression and variance functions, respectively. Weak convergence of the empirical residual process to a Gaussian process is proved. We also consider various applications for testing model assumptions in nonparametric multiple regression. The model tests obtained are able to detect local alternatives that converge to zero at an n^−1/2-rate, independent of the covariate dimension. We consider in detail a test for additivity of the regression function.     

Asymptotic normality and Berry-Esseen results for conditional density estimator with censored and dependent data

In this paper we derive the asymptotic normality and a Berry-Esseen type bound for the kernel conditional density estimator proposed in Ould-Saïd and Cai (2005) [26] when the censored observations with multivariate covariates form a stationary α-mixing sequence.     

Testing independence in nonparametric regression

We propose a new test for independence of error and covariate in a nonparametric regression model. The test statistic is based on a kernel estimator for the L₂-distance between the conditional distribution and the unconditional distribution of the covariates. In contrast to tests so far available in literature, the test can be applied in the important case of multivariate covariates. It can also be adjusted for models with heteroscedastic variance. Asymptotic normality of the test statistic is shown. Simulation results and a real data example are presented.     

Local linear regression for functional predictor and scalar response

The aim of this work is to introduce a new nonparametric regression technique in the context of functional covariate and scalar response. We propose a local linear regression estimator and study its asymptotic behaviour. Its finite-sample performance is compared with a Nadayara-Watson type kernel regression estimator and with the linear regression estimator via a Monte Carlo study and the analysis of two real data sets. In all the scenarios considered, the local linear regression estimator performs better than the kernel one, in the sense that the mean squared prediction error is lower.     

Wavelet estimation of conditional density with truncated, censored and dependent data

In this paper we define a new nonlinear wavelet-based estimator of conditional density function for a random left truncation and right censoring model. We provide an asymptotic expression for the mean integrated squared error (MISE) of the estimator. It is assumed that the lifetime observations form a stationary α-mixing sequence. Unlike for kernel estimators, the MISE expression of the wavelet-based estimators is not affected by the presence of discontinuities in the curves. Also, asymptotic normality of the estimator is established.     

Cross-validated bagged learning

Many applications aim to learn a high dimensional parameter of a data generating distribution based on a sample of independent and identically distributed observations. For example, the goal might be to estimate the conditional mean of an outcome given a list of input variables. In this prediction context, bootstrap aggregating (bagging) has been introduced as a method to reduce the variance of a given estimator at little cost to bias. Bagging involves applying an estimator to multiple bootstrap samples and averaging the result across bootstrap samples. In order to address the curse of dimensionality, a common practice has been to apply bagging to estimators which themselves use cross-validation, thereby using cross-validation within a bootstrap sample to select fine-tuning parameters trading off bias and variance of the bootstrap sample-specific candidate estimators. In this article we point out that in order to achieve the correct bias variance trade-off for the parameter of interest, one should apply the cross-validation selector externally to candidate bagged estimators indexed by these fine-tuning parameters. We use three simulations to compare the new cross-validated bagging method with bagging of cross-validated estimators and bagging of non-cross-validated estimators.     

A new projection estimate for multivariate location with minimax bias

The maximum asymptotic bias of an estimator is a global robustness measure of its performance. The projection median estimator for multivariate location shows a remarkable behavior regarding asymptotic bias. In this paper we consider a modification of the projection median estimator which renders an estimate with better bias performance for point mass contaminations (the worst situation for the projection median estimator). Moreover, it achieves the lowest bound for an equivariant estimate for point mass contaminations.     

Limit theorems in the Fourier transform method for the estimation of multivariate volatility

In this paper, we prove some limit theorems for the Fourier estimator of multivariate volatility proposed by Malliavin and Mancino (2002, 2009) [14] and [15]. In a general framework of discrete time observations we establish the convergence of the estimator and some associated central limit theorems with explicit asymptotic variance. In particular, our results show that this estimator is consistent for synchronous data, but possibly biased for non-synchronous observations. Moreover, from our general central limit theorem, we deduce that the estimator can be efficient in the case of a synchronous regular sampling. In the non-synchronous sampling case, the expression of the asymptotic variance is in general less tractable. We study this case more precisely through the example of an alternate sampling.     

Local asymptotic normality in a stationary model for spatial extremes

De Haan and Pereira (2006) [6] provided models for spatial extremes in the case of stationarity, which depend on just one parameter β>0 measuring tail dependence, and they proposed different estimators for this parameter. We supplement this framework by establishing local asymptotic normality (LAN) of a corresponding point process of exceedances above a high multivariate threshold. Standard arguments from LAN theory then provide the asymptotic minimum variance within the class of regular estimators of β. It turns out that the relative frequency of exceedances is a regular estimator sequence with asymptotic minimum variance, if the underlying observations follow a multivariate extreme value distribution or a multivariate generalized Pareto distribution.     

Statistical inference in a panel data semiparametric regression model with serially correlated errors

We consider a panel data semiparametric partially linear regression model with an unknown vector β of regression coefficients, an unknown nonparametric function g(·) for nonlinear component, and unobservable serially correlated errors. The correlated errors are modeled by a vector autoregressive process which involves a constant intraclass correlation. Applying the pilot estimators of β and g(·), we construct estimators of the autoregressive coefficients, the intraclass correlation and the error variance, and investigate their asymptotic properties. Fitting the error structure results in a new semiparametric two-step estimator of β, which is shown to be asymptotically more efficient than the usual semiparametric least squares estimator in terms of asymptotic covariance matrix. Asymptotic normality of this new estimator is established, and a consistent estimator of its asymptotic covariance matrix is presented. Furthermore, a corresponding estimator of g(·) is also provided. These results can be used to make asymptotically efficient statistical inference. Some simulation studies are conducted to illustrate the finite sample performances of these proposed estimators.     

Nonparametric regression estimation with general parametric error covariance

The asymptotic distribution for the local linear estimator in nonparametric regression models is established under a general parametric error covariance with dependent and heterogeneously distributed regressors. A two-step estimation procedure that incorporates the parametric information in the error covariance matrix is proposed. Sufficient conditions for its asymptotic normality are given and its efficiency relative to the local linear estimator is established. We give examples of how our results are useful in some recently studied regression models. A Monte Carlo study confirms the asymptotic theory predictions and compares our estimator with some recently proposed alternative estimation procedures.     

The proportional hazards model for survey data from independent and clustered super-populations

Data from most complex surveys are subject to selection bias and clustering due to the sampling design. Results developed for a random sample from a super-population model may not apply. Ignoring the survey sampling weights may cause biased estimators and erroneous confidence intervals. In this paper, we use the design approach for fitting the proportional hazards (PH) model and prove formally the asymptotic normality of the sample maximum partial likelihood (SMPL) estimators under the PH model for both stochastically independent and clustered failure times. In the first case, we use the central limit theorem for martingales in the joint design-model space, and this enables us to obtain results for a general multistage sampling design under mild and easily verifiable conditions. In the case of clustered failure times, we require asymptotic normality in the sampling design space directly, and this holds for fewer sampling designs than in the first case. We also propose a variance estimator of the SMPL estimator. A key property of this variance estimator is that we do not have to specify the second-stage correlation model.     

Variance function estimation in multivariate nonparametric regression with fixed design

Variance function estimation in multivariate nonparametric regression is considered and the minimax rate of convergence is established in the iid Gaussian case. Our work uses the approach that generalizes the one used in [A. Munk, Bissantz, T. Wagner, G. Freitag, On difference based variance estimation in nonparametric regression when the covariate is high dimensional, J. R. Stat. Soc. B 67 (Part 1) (2005) 19-41] for the constant variance case. As is the case when the number of dimensions d=1, and very much contrary to standard thinking, it is often not desirable to base the estimator of the variance function on the residuals from an optimal estimator of the mean. Instead it is desirable to use estimators of the mean with minimal bias. Another important conclusion is that the first order difference based estimator that achieves minimax rate of convergence in the one-dimensional case does not do the same in the high dimensional case. Instead, the optimal order of differences depends on the number of dimensions.     

Asymptotic properties of conditional maximum likelihood estimator in a certain exponential model

The conditional maximum likelihood estimator is suggested as an alternative to the maximum likelihood estimator and is favorable for an estimator of a dispersion parameter in the normal distribution, the inverse-Gaussian distribution, and so on. However, it is not clear whether the conditional maximum likelihood estimator is asymptotically efficient in general. Consider the case where it is asymptotically efficient and its asymptotic covariance depends only on an objective parameter in an exponential model. This remand implies that the exponential model possesses a certain parallel foliation. In this situation, this paper investigates asymptotic properties of the conditional maximum likelihood estimator and compares the conditional maximum likelihood estimator with the maximum likelihood estimator. We see that the bias of the former is more robust than that of the latter and that two estimators are very close, especially in the sense of bias-corrected version. The mean Pythagorean relation is also discussed.