Difference based ridge and Liu type estimators in semiparametric regression models

We consider a difference based ridge regression estimator and a Liu type estimator of the regression parameters in the partial linear semiparametric regression model, y=Xβ+f+ε. Both estimators are analyzed and compared in the sense of mean-squared error. We consider the case of independent errors with equal variance and give conditions under which the proposed estimators are superior to the unbiased difference based estimation technique. We extend the results to account for heteroscedasticity and autocovariance in the error terms. Finally, we illustrate the performance of these estimators with an application to the determinants of electricity consumption in Germany.     

An improved model averaging scheme for logistic regression

Recently, penalized regression methods have attracted much attention in the statistical literature. In this article, we argue that such methods can be improved for the purposes of prediction by utilizing model averaging ideas. We propose a new algorithm that combines penalized regression with model averaging for improved prediction. We also discuss the issue of model selection versus model averaging and propose a diagnostic based on the notion of generalized degrees of freedom. The proposed methods are studied using both simulated and real data.     

Bayesian networks with a logistic regression model for the conditional probabilities

Logistic regression techniques can be used to restrict the conditional probabilities of a Bayesian network for discrete variables. More specifically, each variable of the network can be modeled through a logistic regression model, in which the parents of the variable define the covariates. When all main effects and interactions between the parent variables are incorporated as covariates, the conditional probabilities are estimated without restrictions, as in a traditional Bayesian network. By incorporating interaction terms up to a specific order only, the number of parameters can be drastically reduced. Furthermore, ordered logistic regression can be used when the categories of a variable are ordered, resulting in even more parsimonious models. Parameters are estimated by a modified junction tree algorithm. The approach is illustrated with the Alarm network.     

Confidence sets and coverage probabilities based on preliminary estimators in logistic regression models

In this paper we present recentered confidence sets for the parameters of a logistic regression model based on preliminary minimum ?$?$-divergence estimators. Asymptotic coverage probabilities are given as well as a simulation study in order to analyze the coverage probabilities for small and moderate sample sizes.     

混合误差半参数回归模型估计的相合性

研究了误差为ψ混合和ψ混合序列的半参数回归模型,综合最小二乘法和非参数权函数估计方法,分别定义了待估参数β和未知函数夕的估计量βm,n和9m,n(χ).利用混合序列的矩不等式及凸函数的性质,在较弱的条件下证明了这些估计量的强相合性与矩相合性,这些结果推广了已有的相应的研究结果.     

Modified restricted Liu estimator in logistic regression model

?iray et al. (Commun Stat Simul Comput, 2014) proposed a restricted Liu estimator in logistic regression model with linear restrictions. However, this estimator did not satisfy the linear restrictions. In this paper, we introduce a modified restricted Liu estimator in logistic regression model with linear restrictions. Our results show that the new estimator satisfies the linear restrictions. We also discuss the properties of the new estimator under the matrix mean squared error criterion. Finally, a Monte Carlo study and a numerical example are given to show the performances of the new estimator.     

Generalized ridge regression,least squares with stochastic prior information,and Bayesian estimators

The ridge estimator of the usual linear model is generalized by the introduction of an a priori vector r and an associated positive semidefinite matrix S. It is then shown that the generalized ridge estimator can be justified in two ways: (a) by the minimization of the residual sum of squares subject to a constraint on the length, in the metric S, of the vector of differences between r and the estimated linear model coefficients, (b) by incorporating prior knowledge, r playing the role of the vector of means and S proportional to the precision matrix. Both a Bayesian and an Aitken generalized least squares frameworks are used for the latter. The properties of the new estimator are derived and compared to the ordinary least squares estimator. The new method is illustrated with different assumptions on the form of the S matrix.     

Extreme logistic regression

Kernel logistic regression (KLR) is a very powerful algorithm that has been shown to be very competitive with many state-of the art machine learning algorithms such as support vector machines (SVM). Unlike SVM, KLR can be easily extended to multi-class problems and produces class posterior probability estimates making it very useful for many real world applications. However, the training of KLR using gradient based methods or iterative re-weighted least squares can be unbearably slow for large datasets. Coupled with poor conditioning and parameter tuning, training KLR can quickly design matrix become infeasible for some real datasets. The goal of this paper is to present simple, fast, scalable, and efficient algorithms for learning KLR. First, based on a simple approximation of the logistic function, a least square algorithm for KLR is derived that avoids the iterative tuning of gradient based methods. Second, inspired by the extreme learning machine (ELM) theory, an explicit feature space is constructed through a generalized single hidden layer feedforward network and used for training iterative re-weighted least squares KLR (IRLS-KLR) and the newly proposed least squares KLR (LS-KLR). Finally, for large-scale and/or poorly conditioned problems, a robust and efficient preconditioned learning technique is proposed for learning the algorithms presented in the paper. Numerical results on a series of artificial and 12 real bench-mark datasets show first that LS-KLR compares favorable with SVM and traditional IRLS-KLR in terms of accuracy and learning speed. Second, the extension of ELM to KLR results in simple, scalable and very fast algorithms with comparable generalization performance to their original versions. Finally, the introduced preconditioned learning method can significantly increase the learning speed of IRLS-KLR.     

在线Logistic回归

针对连续数据流分类问题,基于在线学习理论,提出一种在线logistic回归算法.研究带有正则项的在线logistic回归,提出了在线logistic-l2回归模型,并给出了理论界估计.最终实验结果表明,随着在线迭代次数的增加,提出的模型与算法能够达到离线预测的分类结果.本文工作为处理海量流数据分类问题提供了一种新的有效方法.     

Random effects logistic regression model for ranking efficiency in data envelopment analysis

Ranking efficiency based on data envelopment analysis (DEA) results can be used for grouping decision-making units (DMUs). The resulting group membership can be partly related to the environmental characteristics of DMU, which are not used either as input or output. Utilizing the expert knowledge on super efficiency DEA results, we propose a multinomial Dirichlet regression model, which can be used for the purpose of selection of new projects. A case study is presented in the context of ranking analysis of new information technology commercialization projects. It is expected that our proposed approach can complement the DEA ranking results with environmental factors and at the same time it facilitates the prediction of efficiency of new DMUs with only given environmental characteristics.     

Multinomial logistic regression algorithm

The lower bound principle (introduced in Böhning and Lindsay 1988, Ann. Inst. Statist. Math., 40, 641–663), Böhning (1989, Biometrika, 76, 375–383) consists of replacing the second derivative matrix by a global lower bound in the Loewner ordering. This bound is used in the Newton-Raphson iteration instead of the Hessian matrix leading to a monotonically converging sequence of iterates. Here, we apply this principle to the multinomial logistic regression model, where it becomes specifically attractive.Supplement to Monotonicity of quadratic-approximation algorithms by Böhning and Lindsay (1988). Ann. Inst. Statist. Math., 40, 641–663.This research was supported by the German Research Foundation.     

James-Stein estimators for time series regression models

The least squares (LS) estimator seems the natural estimator of the coefficients of a Gaussian linear regression model. However, if the dimension of the vector of coefficients is greater than 2 and the residuals are independent and identically distributed, this conventional estimator is not admissible. James and Stein [Estimation with quadratic loss, Proceedings of the Fourth Berkely Symposium vol. 1, 1961, pp. 361-379] proposed a shrinkage estimator (James-Stein estimator) which improves the least squares estimator with respect to the mean squares error loss function. In this paper, we investigate the mean squares error of the James-Stein (JS) estimator for the regression coefficients when the residuals are generated from a Gaussian stationary process. Then, sufficient conditions for the JS to improve the LS are given. It is important to know the influence of the dependence on the JS. Also numerical studies illuminate some interesting features of the improvement. The results have potential applications to economics, engineering, and natural sciences.     

A family of estimators for multivariate kurtosis in a nonnormal linear regression model

In this paper, we propose a new estimator for a kurtosis in a multivariate nonnormal linear regression model. Usually, an estimator is constructed from an arithmetic mean of the second power of the squared sample Mahalanobis distances between observations and their estimated values. The estimator gives an underestimation and has a large bias, even if the sample size is not small. We replace this squared distance with a transformed squared norm of the Studentized residual using a monotonic increasing function. Our proposed estimator is defined by an arithmetic mean of the second power of these squared transformed squared norms with a correction term and a tuning parameter. The correction term adjusts our estimator to an unbiased estimator under normality, and the tuning parameter controls the sizes of the squared norms of the residuals. The family of our estimators includes estimators based on ordinary least squares and predicted residuals. We verify that the bias of our new estimator is smaller than usual by constructing numerical experiments.     

Random effects logistic regression model for default prediction of technology credit guarantee fund

Korean government has been funding the small and medium enterprises (SME) with superior technology based on scorecard. However high default rate of funded SMEs has been reported. In order to effectively manage such governmental fund, it is important to develop accurate scoring model for SMEs. In this paper, we provide a random effects logistic regression model to predict the default of funded SMEs based on both financial and non-financial factors. Advantage of such a random effects model lies in the ability of accommodating not only the individual characteristics of each SME but also the uncertainty that cannot be explained by such individual factors. It is expected that our study can contribute to effective management of government funds by proposing the prediction models for defaults of funded SMEs.     

Bias analysis for misclassification in a multicategorical exposure in a logistic regression model

We derive the expressions of asymptotic biases when ignoring the misclassification in a multicategory exposure. For a model with a misclassified exposure variable only, we provide a general conclusion on the direction of the biases under nondifferential misclassification assumption. To better understand the bias formulas, we use a numerical example.     

The breakdown behavior of the maximum likelihood estimator in the logistic regression model

In this note we discuss the breakdown behavior of the maximum likelihood (ML) estimator in the logistic regression model. We formally prove that the ML-estimator never explodes to infinity, but rather breaks down to zero when adding severe outliers to a data set. An example confirms this behavior.     

NA样本半参数回归模型估计的矩相合性

考虑了误差为NA序列的半参数回归模型,利用非参数估计方法给出了模型参数的最小二乘估计和加权最小二乘估计,并在适当条件下得到了它们的矩相合性.     

Random effects logistic regression model for data envelopment analysis with correlated decision making units

This study attempts to exploit information from environmental variables together with data envelopment analysis (DEA) efficiency scores for efficiency predictions of groups with more limited information. Based on DEA efficiency scores, decision-making units (DMUs) are sorted into two sets containing efficient and inefficient units, respectively. Then they are reshuffled into homogeneous groups with respect to environmental factors. We assume that the efficiency of DMUs in such a homogeneous group would be correlated. However, efficiency of different groups would vary. A beta binomial logistic model is proposed to fit such phenomena and is applied to predict the performance of a new group of commercialization projects for given environmental characteristics.     

Regularized fuzzy clusterwise ridge regression

Fuzzy clusterwise regression has been a useful method for investigating cluster-level heterogeneity of observations based on linear regression. This method integrates fuzzy clustering and ordinary least-squares regression, thereby enabling to estimate regression coefficients for each cluster and fuzzy cluster memberships of observations simultaneously. In practice, however, fuzzy clusterwise regression may suffer from multicollinearity as it builds on ordinary least-squares regression. To deal with this problem in fuzzy clusterwise regression, a new method, called regularized fuzzy clusterwise ridge regression, is proposed that combines ridge regression with regularized fuzzy clustering in a unified framework. In the proposed method, ridge regression is adopted to estimate clusterwise regression coefficients while handling potential multicollinearity among predictor variables. In addition, regularized fuzzy clustering based on maximizing entropy is utilized to systematically determine an optimal degree of fuzziness in memberships. A simulation study is conducted to evaluate parameter recovery of the proposed method as compared to the extant non-regularized counterpart. The usefulness of the proposed method is illustrated by an application concerning the relationship among the characteristics of used cars.