Multiple criteria linear regression

The unknown parameters in multiple linear regression models may be estimated using any one of a number of criteria such as the minimization of the sum of squared errors MSSE, the minimization of the sum of absolute errors MSAE, and the minimization of the maximum absolute error MMAE. At present, the MSSE or the least squares criterion continues to be the most popular. However, at times the choice of a criterion is not clear from statistical, practical or other considerations. Under such circumstances, it may be more appropriate to use multiple criteria rather than a single criterion to estimate the unknown parameters in a multiple linear regression model. We motivate the use of multiple criteria estimation in linear regression models with an example, propose a few models, and outline a solution procedure.     

Least absolute value and chebychev estimation utilizing least squares results

In exploratory data analysis and curve fitting in particular, it is often desirable to observe residual values obtained with different estimation criteria. The goal with most linear model curve-fitting procedures is to minimize, in some sense, the vector of residuals. Perhaps three of the most common estimation criteria require minimizing: the sum of the absolute residuals (least absolute value or L₁ norm); the sum of the squared residuals (least squares or L₂ norm); and the maximum residual (Chebychev or L norm). This paper demonstrates that utilizing the least squares residuals to provide an advanced start for the least absolute value and Chebychev procedures results in a significant reduction in computational effort. Computational results are provided.     

Some selection criteria for nested binary choice models: a comparative study

This paper deals with the problem of choosing the optimum criterion to select the best of a set of nested binary choice models. Special attention is given to the procedures which are derived in a decision-theoretic framework, called model selection criteria (MSC). We propose a new criterion, which we call C ₂, whose theoretical behaviour is compared with that of the AIC and SBIC criteria. The result of the theoretical study shows that the SBIC is the best criterion whatever the situation we consider, while the AIC and C ₂ are only adequate in some cases. The Monte Carlo experiment that is carried out corroborates the theoretical results and adds others: finite sample behaviour and robustness to changes in some aspects of the data generating process. The classical hypothesis testing procedures LR and LM are included and compared with the three criteria of the MSC category. The authors wish to thank the financial support provided by the Spanish Department of Education under project BEC 2003-01757.     

A Unified Approach to Second Order Optimality Criteria in Nonlinear Design of Experiments

In the nonlinear regression model we consider the optimal design problem with a second order design D-criterion. Our purpose is to present a general approach to this problem, which includes the asymptotic second order bias and variance criterion of the least squares estimator and criteria using the volume of confidence regions based on different statistics. Under assumptions of regularity for these statistics a second order approximation of the volume of these regions is derived which is proposed as a quadratic optimality criterion. These criteria include volumes of confidence regions based on the u _n - representable statistics. An important difference between the criteria presented in this paper and the second order criteria commonly employed in the recent literature is that the former criteria are independent of the vector of residuals. Moreover, a refined version of the commonly applied criteria is obtained, which also includes effects of nonlinearity caused by third derivatives of the response function.     

Positive shrinkage, improved pretest and absolute penalty estimators in partially linear models

Shrinkage estimators of a partially linear regression parameter vector are constructed by shrinking estimators in the direction of the estimate which is appropriate when the regression parameters are restricted to a linear subspace. We investigate the asymptotic properties of positive Stein-type and improved pretest semiparametric estimators under quadratic loss. Under an asymptotic distributional quadratic risk criterion, their relative dominance picture is explored analytically. It is shown that positive Stein-type semiparametric estimators perform better than the usual Stein-type and least square semiparametric estimators and that an improved pretest semiparametric estimator is superior to the usual pretest semiparametric estimator. We also consider an absolute penalty type estimator for partially linear models and give a Monte Carlo simulation comparisons of positive shrinkage, improved pretest and the absolute penalty type estimators. The comparison shows that the shrinkage method performs better than the absolute penalty type estimation method when the dimension of the parameter space is much larger than that of the linear subspace.     

Dealing with the multiplicity of solutions of the ℓ1 and ℓ∞ regression models

It is well known that the least absolute value (?_∞) and the least sum of absolute deviations (?₁) algorithms produce estimators that are not necessarily unique. In this paper it is shown how the set of all solutions of the ?₁ and ?_∞ regression problems for moderately large sample sizes can be obtained. In addition, if the multiplicity of solutions wants to be avoided, two new methods giving the same optimal ?₁ and ?_∞ values, but supplying unique solutions, are proposed. The idea consists of using two steps: in the first step the optimal values of the ?₁ and ?_∞ errors are calculated, and in the second step, in case of non-uniqueness of solutions, one of the multiple solutions is selected according to a different criterion. For the ?_∞ the procedure is used sequentially but removing, in each iteration, the data points with maximum absolute residual and adding the corresponding constraints for keeping these residuals, and this process is repeated until no change in the solution is obtained. In this way not only the maximum absolute residual values are minimized in the modified method, but also the maximum absolute residual values of the remaining points sequentially, until no further improvement is possible. In the ?₁ case a least squares criterion is used but restricted to the ?₁ residual condition. Thus, in the modified ?₁ method not only the ?₁ residual is minimized, but also the sum of squared residuals subject to the ?₁ residual. The methods are illustrated by their application to some well known examples and their performances are tested by some simulations, which show that the lack of uniqueness problem cannot be corrected for some experimental designs by increasing the sample size.     

A Kalman-type condition for stochastic approximate controllability

We are interested in the approximate controllability property for a linear stochastic differential equation. For deterministic control necessary and sufficient criterion exists and is called Kalman condition. In the stochastic framework criteria are already known either when the control fully acts on the noise coefficient or when there is no control acting on the noise. We propose a generalization of Kalman condition for the general case. To cite this article: D. Goreac, C. R. Acad. Sci. Paris, Ser. I 346 (2008).     

Robust errors-in-variables linear regression via Laplace distribution

Robust estimation procedures for linear and mixture linear errors-in-variables regression models are proposed based on the relationship between the least absolute deviation criterion and maximum likelihood estimation in a Laplace distribution. The finite sample performance of the proposed procedures is evaluated by simulation studies.     

多元广义岭估计及K值选取的 Q(c)准则

当自变量间存在复共线性时,最小二乘估计就表现出不稳定并可能导致错误的结果.本文采用广义岭估计β(K)来估计多元线性模型的回归系数β=vec(B),通过岭参数K值的选取,可使广义岭估计的均方误差MSE小于最小二乘估计的MSE.指出了广义岭估计中根据MSE准则选取K值存在的主要缺陷,采用了一种选取K值的新准则Q(c),它包含MSE准则和最小二乘LS准则作为特例,从理论上证明和讨论了Q(c)准则的优良性,阐明了c值的统计含义,并给出了确定c值的方法.     

线性规划在回归分析中的应用

线性回归中,针对最小二乘法的两个替代准则一绝对离差和最小准则以及最大绝对离差最小准则,利用线性规划技术建立回归预测模型。实用分析表明,线性规划模型具有较好的预测效果,有郊地消除了统计数据中异常值对回归方程的影响。     

L∞ norm estimation with linear restrictions on the parameters

Chebychev estimation, or L_∞ norm estimation, has as its criterion the minimization of the largest absolute residual. This paper presents a linear programming algorithm which allows linear restrictions on the parameters, and which utilizes a reduced basis and multiple pivots.     

A Comparative Investigation on Model Selection in Independent Factor Analysis

With uncorrelated Gaussian factors extended to mutually independent factors beyond Gaussian, the conventional factor analysis is extended to what is recently called independent factor analysis. Typically, it is called binary factor analysis (BFA) when the factors are binary and called non-Gaussian factor analysis (NFA) when the factors are from real non-Gaussian distributions. A crucial issue in both BFA and NFA is the determination of the number of factors. In the literature of statistics, there are a number of model selection criteria that can be used for this purpose. Also, the Bayesian Ying-Yang (BYY) harmony learning provides a new principle for this purpose. This paper further investigates BYY harmony learning in comparison with existing typical criteria, including Akaik’s information criterion (AIC), the consistent Akaike’s information criterion (CAIC), the Bayesian inference criterion (BIC), and the cross-validation (CV) criterion on selection of the number of factors. This comparative study is made via experiments on the data sets with different sample sizes, data space dimensions, noise variances, and hidden factors numbers. Experiments have shown that for both BFA and NFA, in most cases BIC outperforms AIC, CAIC, and CV while the BYY criterion is either comparable with or better than BIC. In consideration of the fact that the selection by these criteria has to be implemented at the second stage based on a set of candidate models which have to be obtained at the first stage of parameter learning, while BYY harmony learning can provide not only a new class of criteria implemented in a similar way but also a new family of algorithms that perform parameter learning at the first stage with automated model selection, BYY harmony learning is more preferred since computing costs can be saved significantly.     

A ranking model in uncertain, imprecise and multi-experts contexts: The application of evidence theory

We consider ranking problems where the actions are evaluated on a set of ordinal criteria. The evaluation of each alternative with respect to each criterion may be imperfect and is provided by one or several experts. We model each imperfect evaluation as a basic belief assignment (BBA). In order to rank the BBAs characterizing the performances of the actions according to each criterion, a new concept called RBBD and based on the comparison of these BBAs to ideal or nadir BBAs is proposed. This is performed using belief distances that measure the dissimilarity of each BBA to the ideal or nadir BBAs. A model inspired by Xu et al.’s method is also proposed and illustrated by a pedagogical example.     

Solving a bi-criterion cutting stock problem with open-ended demand: a case study

We consider a real-life cutting stock problem with two types of orders. All orders have to be cut from a given number of raws (also known as stock unit, master reel or jumbo). For each order the width of the final (also known as reels or units) and the number of finals is given. An order is called an exact order when the given number of finals must be produced exactly. An order is called an open order when at least the given number of finals must be produced. There is a given maximum on the number of finals that can be produced from a single raw which is determined by the number of knives on the machine. A pattern specifies the number of finals of a given width that will be produced from one raw. A solution consists of specifying a pattern for each raw such that in total the number of finals of exact orders is produced exactly and at least the number of finals of open orders is produced. There are two criteria defined for a solution. One criterion is the cutting loss: the total width of the raws minus the total width of the produced finals. The second criterion is the number of different patterns used in the solution. We describe a branch-and-bound algorithm that produces all Pareto-optimal solutions.     

A finite algorithm for the least two-norm solution of a linear program 1

By perturbing properly a linear program to a separable quadratic program it is possible to solve the latter in its dual variable space by iterative techniques such as sparsity-preserving SOR (successive overtaxation techniques). In this way large sparse linear programs can be handled.

In this paper we give a new computational criterion to check whether the solution of the perturbed quadratic program provides the least 2-norm solution of the original linear program. This criterion improves on the criterion proposed in an earlier paper.

We also describe an algorithm for solving linear programs which is based on the SOR methods. The main property of this algorithm is that, under mild assumptions, it finds the least 2-norm solution of a linear program in a finite number of iteration.s     

Alternative optimality criteria of portfolio selection based upon threshold stopping rule

This paper proposes two types of alternative criteria of optimality for the continuous time portfolio selection problem. The optimality criteria, the so–called Laplace–Stieltjes transform (LST) criteria, are based on the assumption that the financial agent has a target level for the wealth accumulation process. These criteria are closely related to the so–called threshold stopping investment rule. We analytically derive the LST criteria and numerically compare them with the well–known Kelly criterion. It is shown that the portfolio strategies suggested may overcome the problem that the growth portfolio is often overestimated in several investment situations.     

Two optimization problems in linear regression with interval data

We consider a linear regression model where neither regressors nor the dependent variable is observable; only intervals are available which are assumed to cover the unobservable data points. Our task is to compute tight bounds for the residual errors of minimum-norm estimators of regression parameters with various norms (corresponding to least absolute deviations (LAD), ordinary least squares (OLS), generalized least squares (GLS) and Chebyshev approximation). The computation of the error bounds can be formulated as a pair of max–min and min–min box-constrained optimization problems. We give a detailed complexity-theoretic analysis of them. First, we prove that they are NP-hard in general. Then, further analysis explains the sources of NP-hardness. We investigate three restrictions when the problem is solvable in polynomial time: the case when the parameter space is known apriori to be restricted into a particular orthant, the case when the regression model has a fixed number of regression parameters, and the case when only the dependent variable is observed with errors. We propose a method, called orthant decomposition of the parameter space, which is the main tool for obtaining polynomial-time computability results.     

Note on Multiple Objective Dynamic Programming

A technique for finding MINSUM and MINMAX solutions to multi-criteria decision problems, called Multi Objective Dynamic Programming, capable of handling a wide range of linear, nonlinear, deterministic and stochastic multi-criteria decision problems, is presented and illustrated. Multiple objectives are considered by defining an adjoint state space and solving a (N + 1) terminal optimisation problem. The method efficiently generates both individual (criterion) optima and multiple criteria solutions in a single pass. Sensitivity analysis on weights over the various objectives is easily performed.     

Quasistability of a Vector Trajectory Majority Optimization Problem

We consider a multicriteria combinatorial problem with majority optimality principle whose particular criteria are of the form MINSUM, MINMAX, and MINMIN. We obtain a lower attainable bound for the radius of quasistability of such a problem in the case of the Chebyshev norm on the space of perturbing parameters of the vector criterion. We give sufficient conditions for the quasistability of the problem; these are also necessary in the case of linear special criteria.     

On characterizingL 1-supports,natural and Chebyshev approximates

Characterizations of optimal linear supports and approximates in realn-dimensional space are derived. We first characterize anL ₁-support; that is, a support hyperplane for a continuous function, over a solid, compact set, which is optimal in theL ₁-sense. We then characterize best approximates; that is, affinely linear functions which are the best approximates to a continuous function on a compact set by some optimality criterion. The characterization is first derived under the Natural criterion and then expanded to the Chebyshev criterion, utilizing Geoffrion's Criterion Equivalence.