Spline Estimation of Discontinuous Regression Functions

Abstract

This article deals with regression function estimation when the regression function is smooth at all but a finite number of points. An important question is: How can one produce discontinuous output without knowledge of the location of discontinuity points? Unlike most commonly used smoothers that tend to blur discontinuity in the data, we need to find a smoother that can detect such discontinuity. In this article, linear splines are used to estimate discontinuous regression functions. A procedure of knot-merging is introduced for the estimation of regression functions near discontinuous points. The basic idea is to use multiple knots for spline estimates. We use an automatic procedure involving the least squares method, stepwise knot addition, stepwise basis deletion, knot-merging, and the Bayes information criterion to select the final model. The proposed method can produce discontinuous outputs. Numerical examples using both simulated and real data are given to illustrate the performance of the proposed method.     

Empirical likelihood-based subset selection for partially linear autoregressive models

Based on the empirical likelihood method, the subset selection and hypothesis test for parameters in a partially linear autoregressive model are investigated. We show that the empirical log-likelihood ratio at the true parameters converges to the standard chi-square distribution. We then present the definitions of the empirical likelihood-based Bayes information criteria （EBIC） and Akaike information criteria （EAIC）. The results show that EBIC is consistent at selecting subset variables while EAIC is not. Simulation studies demonstrate that the proposed empirical likelihood confidence regions have better coverage probabilities than the least square method, while EBIC has a higher chance to select the true model than EAIC.     

Learning mixtures of polynomials of multidimensional probability densities from data using B-spline interpolation

Non-parametric density estimation is an important technique in probabilistic modeling and reasoning with uncertainty. We present a method for learning mixtures of polynomials (MoPs) approximations of one-dimensional and multidimensional probability densities from data. The method is based on basis spline interpolation, where a density is approximated as a linear combination of basis splines. We compute maximum likelihood estimators of the mixing coefficients of the linear combination. The Bayesian information criterion is used as the score function to select the order of the polynomials and the number of pieces of the MoP. The method is evaluated in two ways. First, we test the approximation fitting. We sample artificial datasets from known one-dimensional and multidimensional densities and learn MoP approximations from the datasets. The quality of the approximations is analyzed according to different criteria, and the new proposal is compared with MoPs learned with Lagrange interpolation and mixtures of truncated basis functions. Second, the proposed method is used as a non-parametric density estimation technique in Bayesian classifiers. Two of the most widely studied Bayesian classifiers, i.e., the naive Bayes and tree-augmented naive Bayes classifiers, are implemented and compared. Results on real datasets show that the non-parametric Bayesian classifiers using MoPs are comparable to the kernel density-based Bayesian classifiers. We provide a free R package implementing the proposed methods.     

非线性回归模型中的约束拟似然

在非线性回归模型中,拟得分函数是一类线性无偏估计函数中的最优者(GodambeandHeyde(1987),朱仲义(1996)),而由拟得分函数得到的拟似然估计在由线性无偏估计函数得到的估计类中具有渐近最优性(林路(1999)).本文则研究非线性回归模型中的有偏估计函数理论,构造了参数的约束拟似然估计,得到了约束拟似然的局部最优性,局部改进了拟似然估计,从而扩充了线性模型中的有偏估计理论.     

Restructuring forward step of MARS algorithm using a new knot selection procedure based on a mapping approach

In high dimensional data modeling, Multivariate Adaptive Regression Splines (MARS) is a popular nonparametric regression technique used to define the nonlinear relationship between a response variable and the predictors with the help of splines. MARS uses piecewise linear functions for local fit and apply an adaptive procedure to select the number and location of breaking points (called knots). The function estimation is basically generated via a two-stepwise procedure: forward selection and backward elimination. In the first step, a large number of local fits is obtained by selecting large number of knots via a lack-of-fit criteria; and in the latter one, the least contributing local fits or knots are removed. In conventional adaptive spline procedure, knots are selected from a set of all distinct data points that makes the forward selection procedure computationally expensive and leads to high local variance. To avoid this drawback, it is possible to restrict the knot points to a subset of data points. In this context, a new method is proposed for knot selection which bases on a mapping approach like self organizing maps. By this method, less but more representative data points are become eligible to be used as knots for function estimation in forward step of MARS. The proposed method is applied to many simulated and real datasets, and the results show that it proposes a time efficient forward step for the knot selection and model estimation without degrading the model accuracy and prediction performance.     

Empirical likelihood-based inferences for the area under the ROC curve with covariates

In the receiver operating characteristic (ROC) analysis,the area under the ROC curve (AUC) is a popular summary index of discriminatory accuracy of a diagnostic test.Incorporating covariates into ROC analysis can improve the diagnostic accuracy of the test.Regression model for the AUC is a tool to evaluate the effects of the covariates on the diagnostic accuracy.In this paper,empirical likelihood (EL) method is proposed for the AUC regression model.For the regression parameter vector,it can be shown that the asymptotic distribution of its EL ratio statistic is a weighted sum of independent chi-square distributions.Confidence regions are constructed for the parameter vector based on the newly developed empirical likelihood theorem,as well as for the covariate-specific AUC.Simulation studies were conducted to compare the relative performance of the proposed EL-based methods with the existing method in AUC regression.Finally,the proposed methods are illustrated with a real data set.     

具有高斯过程误差的函数型线性模型的统计诊断

提出了具有高斯过程误差的函数型回归模型的几种诊断方法.在此模型中,首先,在样条基的基础上,推导了回归系数函数的估计.随后,证明了数据删失模型和均值漂移模型的等价性.然后,研究了三种诊断方法,即残差分析、Cook距离和似然距离来诊断异常和强影响数据.最后,通过一个模拟例子和一个实例来阐述方法的有效性.     

Variational Bayes With Intractable Likelihood

Variational Bayes (VB) is rapidly becoming a popular tool for Bayesian inference in statistical modeling. However, the existing VB algorithms are restricted to cases where the likelihood is tractable, which precludes their use in many interesting situations such as in state--space models and in approximate Bayesian computation (ABC), where application of VB methods was previously impossible. This article extends the scope of application of VB to cases where the likelihood is intractable, but can be estimated unbiasedly. The proposed VB method therefore makes it possible to carry out Bayesian inference in many statistical applications, including state--space models and ABC. The method is generic in the sense that it can be applied to almost all statistical models without requiring too much model-based derivation, which is a drawback of many existing VB algorithms. We also show how the proposed method can be used to obtain highly accurate VB approximations of marginal posterior distributions. Supplementary material for this article is available online.     

附加信息下的局部线性估计

在回归模型中，对一类因变量函数的条件期望方程的附加信息，我们提出了基于极大经验似然方法的局部线性点估计，在一定条件下证明了这些估计的相合性和渐近正态性，而且估计的方差小于通常不带附加信息核估计的方差．模拟结果也显示了估计的优良性．     

Empirical Likelihood Semiparametric Regression Analysis under Random Censorship

This paper considers large sample inference for the regression parameter in a partly linear model for right censored data. We introduce an estimated empirical likelihood for the regression parameter and show that its limiting distribution is a mixture of central chi-squared distributions. A Monte Carlo method is proposed to approximate the limiting distribution. This enables one to make empirical likelihood-based inference for the regression parameter. We also develop an adjusted empirical likelihood method which only appeals to standard chi-square tables. Finite sample performance of the proposed methods is illustrated in a simulation study.     

半参数变系数部分线性模型的统计推断

本文在多种复杂数据下, 研究一类半参数变系数部分线性模型的统计推断理论和方法. 首先在纵向数据和测量误差数据等复杂数据下, 研究半参数变系数部分线性模型的经验似然推断问题, 分别提出分组的和纠偏的经验似然方法. 该方法可以有效地处理纵向数据的组内相关性给构造经验似然比函数所带来的困难. 其次在测量误差数据和缺失数据等复杂数据下, 研究模型的变量选择问题, 分别提出一个“纠偏” 的和基于借补值的变量选择方法. 该变量选择方法可以同时选择参数分量及非参数分量中的重要变量, 并且变量选择与回归系数的估计同时进行. 通过选择适当的惩罚参数, 证明该变量选择方法可以相合地识别出真实模型, 并且所得的正则估计具有oracle 性质.     

Nonlinear logistic discrimination via regularized radial basis functions for classifying high-dimensional data

A flexible nonparametric method is proposed for classifying high- dimensional data with a complex structure. The proposed method can be regarded as an extended version of linear logistic discriminant procedures, in which the linear predictor is replaced by a radial-basis-expansion predictor. Radial basis functions with a hyperparameter are used to take the information on covariates and class labels into account; this was nearly impossible within the previously proposed hybrid learning framework. The penalized maximum likelihood estimation procedure is employed to obtain stable parameter estimates. A crucial issue in the model-construction process is the choice of a suitable model from candidates. This issue is examined from information-theoretic and Bayesian viewpoints and we employed Ando et al. (Japanese Journal of Applied Statistics, 31, 123–139, 2002)’s model evaluation criteria. The proposed method is available not only for the high-dimensional data but also for the variable selection problem. Real data analysis and Monte Carlo experiments show that our proposed method performs well in classifying future observations in practical situations. The simulation results also show that the use of the hyperparameter in the basis functions improves the prediction performance.     

�������ݾ�ֵ��Э����������Ч�Ƚ�����

In this article, we develop efficient robust method for estimation of mean and covariance simultaneously for longitudinal data in regression model. Based on Cholesky decomposition for the covariance matrix and rewriting the regression model, we propose a weighted least square estimator, in which the weights are estimated under generalized empirical likelihood framework. The proposed estimator obtains high efficiency from the close connection to empirical likelihood method, and achieves robustness by bounding the weighted sum of squared residuals. Simulation study shows that, compared to existing robust estimation methods for longitudinal data, the proposed estimator has relatively high efficiency and comparable robustness. In the end, the proposed method is used to analyse a real data set.     

Comparison of Parametric and Bootstrap Approaches to Obtaining Confidence Intervals for Logspline Density Estimation

In earlier articles, we developed an automated methodology for using cubic splines with tail linear constraints to model the logarithm of a univariate density function. This methodology was subsequently modified so that the knots were determined by stepwise addition-deletion and the remaining coefficients were determined by maximum likelihood estimation. An alternative approach, referred to as the free knot spline procedure, is to use the maximum likelihood method to estimate the knot locations as well as the remaining coefficients. This article compares various approaches to constructing confidence intervals for logspline density estimates, for both the stepwise procedure and the free knot procedure. It is concluded that a variation of the bootstrap, in which only a limited number of bootstrap simulations are used to estimate standard errors that are combined with standard normal quantiles, seems to perform the best, especially when coverages and computing time are both taken into account.     

��I�ͻ�Ͻ�β��ָ��——�������ֲ�����ʧЧģ�͵�ͳ�Ʒ���

In this paper, we investigate a competing risks model based on exponentiated Weibull distribution under Type-I progressively hybrid censoring scheme. To estimate the unknown parameters and reliability function, the maximum likelihood estimators and asymptotic confidence intervals are derived. Since Bayesian posterior density functions cannot be given in closed forms, we adopt Markov chain Monte Carlo method to calculate approximate Bayes estimators and highest posterior density credible intervals. To illustrate the estimation methods, a simulation study is carried out with numerical results. It is concluded that the maximum likelihood estimation and Bayesian estimation can be used for statistical inference in competing risks model under Type-I progressively hybrid censoring scheme.     

泊松回归模型中协变量含误差的经验似然推断

在模型的协变量含有测量误差的情况下,考虑一类泊松回归模型的统计推断问题.通过巧妙地构造辅助随机向量,提出一个工具变量类型的经验似然统计推断方法.证明构造的经验对数似然比函数渐近服从标准卡方分布,进而给出了回归系数的置信区间.所提出的估计方法可以有效地消除测量误差对估计精度的影响,并且具有较好的有限样本性质.     

Parameter estimation for a two-parameter bathtub-shaped lifetime distribution

A two-parameter distribution was revisited by Chen (2000) [7]. This distribution can have a bathtub-shaped or increasing failure rate function which enables it to fit real lifetime data sets. Maximum likelihood and Bayes estimates of the two unknown parameters are discussed in this paper. It is assumed in the Bayes case that the unknown parameters have gamma priors. Explicit forms of Bayes estimators cannot be obtained. Different approximations are used to establish point estimates and two sided Bayesian probability intervals for the parameters. Monte Carlo simulations are applied to the comparison between the maximum likelihood estimates and the approximate Bayes estimates obtained under non-informative prior assumptions. Analysis of a real data set is also been presented for illustrative purposes.     

基于因变量抽样设计下线性回归模型的假设检验问题

在许多大型队列研究中,采用节约成本并能提高效率的抽样机制至关重要,基于因变量的抽样设计正是这样一种有偏抽样机制.这种方法最大的优点在于:能够将资源集中在那些包含有更多的协变量与因变量关系信息的研究群体上.本文研究基于因变量抽样设计下的线性模型中回归方程显著性检验以及回归系数显著性检验问题.基于一种半参数经验轮廓似然的方法,我们分别为回归方程检验与回归系数检验提出了相应的检验统计量,获得了所提出检验统计量的渐近性质.通过模拟研究评估了所提出的检验方法在有限样本下的表现,并应用提出的方法分析了一个孕妇分娩的实际数据.     

Likelihood estimation of consumer preferences in choice-based conjoint analysis

In marketing research the measurement of individual preferences and assessment of utility functions have long traditions. Conjoint analysis, and particularly choice-based conjoint analysis (CBC), is frequently employed for such measurement. The world today appears increasingly customer or user oriented wherefore research intensity in conjoint analysis is rapidly increasing in various fields, OR/MS being no exception. Although several optimization based approaches have been suggested since the introduction of the Hierarchical Bayes (HB) method for estimating CBC utility functions, recent comparisons indicate that challenging HB is hard. Based on likelihood maximization we propose a method called LM and compare its performance with HB using twelve field data sets. Performance comparisons are based on holdout validation, i.e. predictive performance. Average performance of LM indicates an improvement over HB and the difference is statistically significant. We also use simulation based data sets to compare the performance for parameter recovery. In terms of both predictive performance and RMSE a smaller number of questions in CBC appears to favor LM over HB.     

Penalized empirical likelihood estimation of semiparametric models

We propose an empirical likelihood-based estimation method for conditional estimating equations containing unknown functions, which can be applied for various semiparametric models. The proposed method is based on the methods of conditional empirical likelihood and penalization. Thus, our estimator is called the penalized empirical likelihood (PEL) estimator. For the whole parameter including infinite-dimensional unknown functions, we derive the consistency and a convergence rate of the PEL estimator. Furthermore, for the finite-dimensional parametric component, we show the asymptotic normality and efficiency of the PEL estimator. We illustrate the theory by three examples. Simulation results show reasonable finite sample properties of our estimator.