Sufficient dimension reduction for the conditional mean with a categorical predictor in multivariate regression

Recent sufficient dimension reduction methodologies in multivariate regression do not have direct application to a categorical predictor. For this, we define the multivariate central partial mean subspace and propose two methodologies to estimate it. The first method uses the ordinary least squares. Chi-squared distributed statistics for dimension tests are constructed, and an estimate of the target subspace is consistent and efficient. Moreover, the effects of continuous predictors can be tested without assuming any model. The second method extends Iterative Hessian Transformation to this context. For dimension estimation, permutation tests are used. Simulated and real data examples for illustrating various properties of the proposed methods are presented.     

Nonconcave penalized inverse regression in single-index models with high dimensional predictors

In this paper we aim to estimate the direction in general single-index models and to select important variables simultaneously when a diverging number of predictors are involved in regressions. Towards this end, we propose the nonconcave penalized inverse regression method. Specifically, the resulting estimation with the SCAD penalty enjoys an oracle property in semi-parametric models even when the dimension, p_n, of predictors goes to infinity. Under regularity conditions we also achieve the asymptotic normality when the dimension of predictor vector goes to infinity at the rate of p_n=o(n^1/3) where n is sample size, which enables us to construct confidence interval/region for the estimated index. The asymptotic results are augmented by simulations, and illustrated by analysis of an air pollution dataset.     

Dimension reduction in functional regression with categorical predictor

In the present paper, we consider dimension reduction methods for functional regression with a scalar response and the predictors including a random curve and a categorical random variable. To deal with the categorical random variable, we propose three potential dimension reduction methods: partial functional sliced inverse regression, marginal functional sliced inverse regression and conditional functional sliced inverse regression. Furthermore, we investigate the relationships among the three methods. In addition, a new modified BIC criterion for determining the dimension of the effective dimension reduction space is developed. Real and simulation data examples are then presented to show the effectiveness of the proposed methods.     

On a dimension reduction regression with covariate adjustment

In this paper, we consider a semiparametric modeling with multi-indices when neither the response nor the predictors can be directly observed and there are distortions from some multiplicative factors. In contrast to the existing methods in which the response distortion deteriorates estimation efficacy even for a simple linear model, the dimension reduction technique presented in this paper interestingly does not have to account for distortion of the response variable. The observed response can be used directly whether distortion is present or not. The resulting dimension reduction estimators are shown to be consistent and asymptotically normal. The results can be employed to test whether the central dimension reduction subspace has been estimated appropriately and whether the components in the basis directions in the space are significant. Thus, the method provides an alternative for determining the structural dimension of the subspace and for variable selection. A simulation study is carried out to assess the performance of the proposed method. The analysis of a real dataset demonstrates the potential usefulness of distortion removal.     

Successive direction extraction for estimating the central subspace in a multiple-index regression

In this paper we propose a dimension reduction method for estimating the directions in a multiple-index regression based on information extraction. This extends the recent work of Yin and Cook [X. Yin, R.D. Cook, Direction estimation in single-index regression, Biometrika 92 (2005) 371-384] who introduced the method and used it to estimate the direction in a single-index regression. While a formal extension seems conceptually straightforward, there is a fundamentally new aspect of our extension: We are able to show that, under the assumption of elliptical predictors, the estimation of multiple-index regressions can be decomposed into successive single-index estimation problems. This significantly reduces the computational complexity, because the nonparametric procedure involves only a one-dimensional search at each stage. In addition, we developed a permutation test to assist in estimating the dimension of a multiple-index regression.     

Efficient estimation in conditional single-index regression

Semiparametric single-index regression involves an unknown finite-dimensional parameter and an unknown (link) function. We consider estimation of the parameter via the pseudo-maximum likelihood method. For this purpose we estimate the conditional density of the response given a candidate index and maximize the obtained likelihood. We show that this technique of adaptation yields an asymptotically efficient estimator: it has minimal variance among all estimators.     

Structural test in regression on functional variables

Many papers deal with structural testing procedures in multivariate regression. More recently, various estimators have been proposed for regression models involving functional explanatory variables. Thanks to these new estimators, we propose a theoretical framework for structural testing procedures adapted to functional regression. The procedures introduced in this paper are innovative and make the link between former works on functional regression and others on structural testing procedures in multivariate regression. We prove asymptotic properties of the level and the power of our procedures under general assumptions that cover a large scope of possible applications: tests for no effect, linearity, dimension reduction, …     

Simple and efficient improvements of multivariate local linear regression

This paper studies improvements of multivariate local linear regression. Two intuitively appealing variance reduction techniques are proposed. They both yield estimators that retain the same asymptotic conditional bias as the multivariate local linear estimator and have smaller asymptotic conditional variances. The estimators are further examined in aspects of bandwidth selection, asymptotic relative efficiency and implementation. Their asymptotic relative efficiencies with respect to the multivariate local linear estimator are very attractive and increase exponentially as the number of covariates increases. Data-driven bandwidth selection procedures for the new estimators are straightforward given those for local linear regression. Since the proposed estimators each has a simple form, implementation is easy and requires much less or about the same amount of effort. In addition, boundary corrections are automatic as in the usual multivariate local linear regression.     

Thresholding projection estimators in functional linear models

We consider the problem of estimating the regression function in functional linear regression models by proposing a new type of projection estimators which combine dimension reduction and thresholding. The introduction of a threshold rule allows us to get consistency under broad assumptions as well as minimax rates of convergence under additional regularity hypotheses. We also consider the particular case of Sobolev spaces generated by the trigonometric basis which permits us to get easily mean squared error of prediction as well as estimators of the derivatives of the regression function. We prove that these estimators are minimax and rates of convergence are given for some particular cases.     

Estimating the error distribution in nonparametric multiple regression with applications to model testing

In this paper we consider the estimation of the error distribution in a heteroscedastic nonparametric regression model with multivariate covariates. As estimator we consider the empirical distribution function of residuals, which are obtained from multivariate local polynomial fits of the regression and variance functions, respectively. Weak convergence of the empirical residual process to a Gaussian process is proved. We also consider various applications for testing model assumptions in nonparametric multiple regression. The model tests obtained are able to detect local alternatives that converge to zero at an n^−1/2-rate, independent of the covariate dimension. We consider in detail a test for additivity of the regression function.     

Model checking in errors-in-variables regression

This paper discusses a class of minimum distance tests for fitting a parametric regression model to a class of regression functions in the errors-in-variables model. These tests are based on certain minimized distances between a nonparametric regression function estimator and a deconvolution kernel estimator of the conditional expectation of the parametric model being fitted. The paper establishes the asymptotic normality of the proposed test statistics under the null hypothesis and that of the corresponding minimum distance estimators. We also prove the consistency of the proposed tests against a fixed alternative and obtain the asymptotic distributions for general local alternatives. Simulation studies show that the testing procedures are quite satisfactory in the preservation of the finite sample level and in terms of a power comparison.     

An adaptive estimation of MAVE

Minimum average variance estimation (MAVE, Xia et al. (2002) [29]) is an effective dimension reduction method. It requires no strong probabilistic assumptions on the predictors, and can consistently estimate the central mean subspace. It is applicable to a wide range of models, including time series. However, the least squares criterion used in MAVE will lose its efficiency when the error is not normally distributed. In this article, we propose an adaptive MAVE which can be adaptive to different error distributions. We show that the proposed estimate has the same convergence rate as the original MAVE. An EM algorithm is proposed to implement the new adaptive MAVE. Using both simulation studies and a real data analysis, we demonstrate the superior finite sample performance of the proposed approach over the existing least squares based MAVE when the error distribution is non-normal and the comparable performance when the error is normal.     

A link-free method for testing the significance of predictors

One important step in regression analysis is to identify significant predictors from a pool of candidates so that a parsimonious model can be obtained using these significant predictors only. However, most of the existing methods assume linear relationships between response and predictors, which may be inappropriate in some applications. In this article, we discuss a link-free method that avoids specifying how the response depends on the predictors. Therefore, this method has no problem of model misspecification, and it is suitable for selecting significant predictors at the preliminary stage of data analysis. A test statistic is suggested and its asymptotic distribution is derived. Examples are used to demonstrate the proposed method.     

Dimension Reduction in Regressions With Exponential Family Predictors

We present first methodology for dimension reduction in regressions with predictors that, given the response, follow one-parameter exponential families. Our approach is based on modeling the conditional distribution of the predictors given the response, which allows us to derive and estimate a sufficient reduction of the predictors. We also propose a method of estimating the forward regression mean function without requiring an explicit forward regression model. Whereas nearly all existing estimators of the central subspace are limited to regressions with continuous predictors only, our proposed methodology extends estimation to regressions with all categorical or a mixture of categorical and continuous predictors. Supplementary materials including the proofs and the computer code are available from the JCGS website.     

Conditional information criteria for selecting variables in linear mixed models

In this paper, we consider the problem of selecting the variables of the fixed effects in the linear mixed models where the random effects are present and the observation vectors have been obtained from many clusters. As the variable selection procedure, here we use the Akaike Information Criterion, AIC. In the context of the mixed linear models, two kinds of AIC have been proposed: marginal AIC and conditional AIC. In this paper, we derive three versions of conditional AIC depending upon different estimators of the regression coefficients and the random effects. Through the simulation studies, it is shown that the proposed conditional AIC’s are superior to the marginal and conditional AIC’s proposed in the literature in the sense of selecting the true model. Finally, the results are extended to the case when the random effects in all the clusters are of the same dimension but have a common unknown covariance matrix.     

Empirical likelihood for a varying coefficient partially linear model with diverging number of parameters

The purpose of this paper is two-fold. First, for the estimation or inference about the parameters of interest in semiparametric models, the commonly used plug-in estimation for infinite-dimensional nuisance parameter creates non-negligible bias, and the least favorable curve or under-smoothing is popularly employed for bias reduction in the literature. To avoid such strong structure assumptions on the models and inconvenience of estimation implementation, for the diverging number of parameters in a varying coefficient partially linear model, we adopt a bias-corrected empirical likelihood (BCEL) in this paper. This method results in the distribution of the empirical likelihood ratio to be asymptotically tractable. It can then be directly applied to construct confidence region for the parameters of interest. Second, different from all existing methods that impose strong conditions to ensure consistency of estimation when diverging the number of the parameters goes to infinity as the sample size goes to infinity, we provide techniques to show that, other than the usual regularity conditions, the consistency holds under moment conditions alone on the covariates and error with a diverging rate being even faster than those in the literature. A simulation study is carried out to assess the performance of the proposed method and to compare it with the profile least squares method. A real dataset is analyzed for illustration.     

Flexible modeling based on copulas in nonparametric median regression

Consider the model Y=m(X)+ε, where m(⋅)=med(Y|⋅) is unknown but smooth. It is often assumed that ε and X are independent. However, in practice this assumption is violated in many cases. In this paper we propose modeling the dependence between ε and X by means of a copula model, i.e. (ε,X)∼C_θ(F_ε(⋅),F_X(⋅)), where C_θ is a copula function depending on an unknown parameter θ, and F_ε and F_X are the marginals of ε and X. Since many parametric copula families contain the independent copula as a special case, the so-obtained regression model is more flexible than the ‘classical’ regression model.We estimate the parameter θ via a pseudo-likelihood method and prove the asymptotic normality of the estimator, based on delicate empirical process theory. We also study the estimation of the conditional distribution of Y given X. The procedure is illustrated by means of a simulation study, and the method is applied to data on food expenditures in households.     

Statistical inference for panel data semiparametric partially linear regression models with heteroscedastic errors

We consider a panel data semiparametric partially linear regression model with an unknown parameter vector for the linear parametric component, an unknown nonparametric function for the nonlinear component, and a one-way error component structure which allows unequal error variances (referred to as heteroscedasticity). We develop procedures to detect heteroscedasticity and one-way error component structure, and propose a weighted semiparametric least squares estimator (WSLSE) of the parametric component in the presence of heteroscedasticity and/or one-way error component structure. This WSLSE is asymptotically more efficient than the usual semiparametric least squares estimator considered in the literature. The asymptotic properties of the WSLSE are derived. The nonparametric component of the model is estimated by the local polynomial method. Some simulations are conducted to demonstrate the finite sample performances of the proposed testing and estimation procedures. An example of application on a set of panel data of medical expenditures in Australia is also illustrated.     

Risk bounds for model selection via penalization

Performance bounds for criteria for model selection are developed using recent theory for sieves. The model selection criteria are based on an empirical loss or contrast function with an added penalty term motivated by empirical process theory and roughly proportional to the number of parameters needed to describe the model divided by the number of observations. Most of our examples involve density or regression estimation settings and we focus on the problem of estimating the unknown density or regression function. We show that the quadratic risk of the minimum penalized empirical contrast estimator is bounded by an index of the accuracy of the sieve. This accuracy index quantifies the trade-off among the candidate models between the approximation error and parameter dimension relative to sample size. If we choose a list of models which exhibit good approximation properties with respect to different classes of smoothness, the estimator can be simultaneously minimax rate optimal in each of those classes. This is what is usually called adaptation. The type of classes of smoothness in which one gets adaptation depends heavily on the list of models. If too many models are involved in order to get accurate approximation of many wide classes of functions simultaneously, it may happen that the estimator is only approximately adaptive (typically up to a slowly varying function of the sample size). We shall provide various illustrations of our method such as penalized maximum likelihood, projection or least squares estimation. The models will involve commonly used finite dimensional expansions such as piecewise polynomials with fixed or variable knots, trigonometric polynomials, wavelets, neural nets and related nonlinear expansions defined by superposition of ridge functions. Received: 7 July 1995 / Revised version: 1 November 1997     

Testing independence in nonparametric regression

We propose a new test for independence of error and covariate in a nonparametric regression model. The test statistic is based on a kernel estimator for the L₂-distance between the conditional distribution and the unconditional distribution of the covariates. In contrast to tests so far available in literature, the test can be applied in the important case of multivariate covariates. It can also be adjusted for models with heteroscedastic variance. Asymptotic normality of the test statistic is shown. Simulation results and a real data example are presented.