An additive utility mixed integer programming model for nonlinear discriminant analysis

Mathematical programming (MP) discriminant analysis models can be used to develop classification models for assigning observations of unknown class membership to one of a number of specified classes using values of a set of features associated with each observation. Since most MP discriminant analysis models generate linear discriminant functions, these MP models are generally used to develop linear classification models. Nonlinear classifiers may, however, have better classification performance than linear classifiers. In this paper, a mixed integer programming model is developed to generate nonlinear discriminant functions composed of monotone piecewise-linear marginal utility functions for each feature and the cut-off value for class membership. It is also shown that this model can be extended for feature selection. The performance of this new MP model for two-group discriminant analysis is compared with statistical discriminant analysis and other MP discriminant analysis models using a real problem and a number of simulated problem sets.     

Classification accuracy in discriminant analysis: a mixed integer programming approach

Classification models can be developed by statistical or mathematical programming discriminant analysis techniques. Variable selection extensions of these techniques allow the development of classification models with a limited number of variables. Although stepwise statistical variable selection methods are widely used, the performance of the resultant classification models may not be optimal because of the stepwise selection protocol and the nature of the group separation criterion. A mixed integer programming approach for selecting variables for maximum classification accuracy is developed in this paper and the performance of this approach, measured by the leave-one-out hit rate, is compared with the published results from a statistical approach in which all possible variable subsets were considered. Although this mixed integer programming approach can only be applied to problems with a relatively small number of observations, it may be of great value where classification decisions must be based on a limited number of observations.     

Examining the effect of second-order terms in mathematical programming approaches to the classification problem

Research on mathematical programming approaches to the classification problem has focused almost exclusively on linear discriminant functions with only first-order terms. While many of these first-order models have displayed excellent classificatory performance when compared to Fisher's linear discriminant method, they cannot compete with Smith's quadratic discriminant method on certain data sets. In this paper, we investigate the appropriateness of including second-order terms in mathematical programming models. Various issues are addressed, such as performance of models with small to moderate sample size, need for crossproduct terms, and loss of power by the mathematical programming models under conditions ideal for the parametric procedures. A simulation study is conducted to assess the relative performance of first-order and second-order mathematical programming models to the parametric procedures. The simulation study indicates that mathematical programming models using polynomial functions may be prone to overfitting on the training samples which in turn may cause rather poor fits on the validation samples. The simulation study also indicates that inclusion of cross-product terms may hurt a polynomial model's accuracy on the validation samples, although omission of them means that the model is not invariant to nonsingular transformations of the data.     

Heuristics for feature selection in mathematical programming discriminant analysis models

In developing a classification model for assigning observations of unknown class to one of a number of specified classes using the values of a set of features associated with each observation, it is often desirable to base the classifier on a limited number of features. Mathematical programming discriminant analysis methods for developing classification models can be extended for feature selection. Classification accuracy can be used as the feature selection criterion by using a mixed integer programming (MIP) model in which a binary variable is associated with each training sample observation, but the binary variable requirements limit the size of problems to which this approach can be applied. Heuristic feature selection methods for problems with large numbers of observations are developed in this paper. These heuristic procedures, which are based on the MIP model for maximizing classification accuracy, are then applied to three credit scoring data sets.     

Mathematical programming models for piecewise-linear discriminant analysis

Mathematical programming (MP) discriminant analysis models are widely used to generate linear discriminant functions that can be adopted as classification models. Nonlinear classification models may have better classification performance than linear classifiers, but although MP methods can be used to generate nonlinear discriminant functions, functions of specified form must be evaluated separately. Piecewise-linear functions can approximate nonlinear functions, and two new MP methods for generating piecewise-linear discriminant functions are developed in this paper. The first method uses maximization of classification accuracy (MCA) as the objective, while the second uses an approach based on minimization of the sum of deviations (MSD). The use of these new MP models is illustrated in an application to a test problem and the results are compared with those from standard MCA and MSD models.     

Evaluating the effect of gap size in a single function mathematical programming model for the three-group classification problem

This study examines the impact that the size of the classification gap can have on the classificatory performance of a mathematical programming based discriminant model. In mathematical programming based models that project the discriminant scores onto a line, the discriminant score of an observation may fall into the gap between adjacent group intervals; thus there is no clear cut way to determine the group in which the observation should be classified. We examine a procedure that we refer to as the split gap approach. The split gap approach is defined as a strategy of estimating the performance of a mathematical programming based model using a nonzero gap size to separate group intervals and then splitting the gap between adjacent group intervals to classify future observations. Studies that propose models with a classification gap generally do not assess the effect of the gap on the performance of the model. This paper investigates this effect. A theoretical assessment and a Monte Carlo simulation are used to determine the impact of different gap sizes on a mixed integer programming model using a single function classification model for the three-group case.     

Reducing the number of variables in integer quadratic programming problem

In this paper, a new variable reduction technique is presented for general integer quadratic programming problem (GP), under which some variables of (GP) can be fixed at zero without sacrificing optimality. A sufficient condition and a necessary condition for the identification of dominated terms are provided. By comparing the given data of the problem and the upper bound of the variables, if they meet certain conditions, some variables can be fixed at zero. We report a computational study to demonstrate the efficacy of the proposed technique in solving general integer quadratic programming problems. Furthermore, we discuss separable integer quadratic programming problems in a simpler and clearer form.     

Multiple objective linear programming models with interval coefficients – an illustrated overview

In most real-world situations, the coefficients of decision support models are not exactly known. In this context, it is convenient to consider an extension of traditional mathematical programming models incorporating their intrinsic uncertainty, without assuming the exactness of the model coefficients. Interval programming is one of the tools to tackle uncertainty in mathematical programming models. Moreover, most real-world problems inherently impose the need to consider multiple, conflicting and incommensurate objective functions. This paper provides an illustrated overview of the state of the art of Interval Programming in the context of multiple objective linear programming models.     

Distributionally robust mixed integer linear programs: Persistency models with applications

In this paper, we review recent advances in the distributional analysis of mixed integer linear programs with random objective coefficients. Suppose that the probability distribution of the objective coefficients is incompletely specified and characterized through partial moment information. Conic programming methods have been recently used to find distributionally robust bounds for the expected optimal value of mixed integer linear programs over the set of all distributions with the given moment information. These methods also provide additional information on the probability that a binary variable attains a value of 1 in the optimal solution for 0–1 integer linear programs. This probability is defined as the persistency of a binary variable. In this paper, we provide an overview of the complexity results for these models, conic programming formulations that are readily implementable with standard solvers and important applications of persistency models. The main message that we hope to convey through this review is that tools of conic programming provide important insights in the probabilistic analysis of discrete optimization problems. These tools lead to distributionally robust bounds with applications in activity networks, vertex packing, discrete choice models, random walks and sequencing problems, and newsvendor problems.     

A reduction dynamic programming algorithm for the bi-objective integer knapsack problem

This paper presents a backward state reduction dynamic programming algorithm for generating the exact Pareto frontier for the bi-objective integer knapsack problem. The algorithm is developed addressing a reduced problem built after applying variable fixing techniques based on the core concept. First, an approximate core is obtained by eliminating dominated items. Second, the items included in the approximate core are subject to the reduction of the upper bounds by applying a set of weighted-sum functions associated with the efficient extreme solutions of the linear relaxation of the multi-objective integer knapsack problem. Third, the items are classified according to the values of their upper bounds; items with zero upper bounds can be eliminated. Finally, the remaining items are used to form a mixed network with different upper bounds. The numerical results obtained from different types of bi-objective instances show the effectiveness of the mixed network and associated dynamic programming algorithm.     

A result in surrogate duality for certain integer programming problems

We consider linear programming problems with some equality constraints. For such problems, surrogate relaxation formulations relaxing equality constraints existwith zero primal-dual gap both when all variables are restricted to be integers and when no variable is required to be integer. However, for such surrogate formulations, when the variables are mixed-integer, the primal-dual gap may not be zero. We establish this latter result by a counterexample.     

Binary fuzzy goal programming

Goal programming is an important technique for solving many decision/management problems. Fuzzy goal programming involves applying the fuzzy set theory to goal programming, thus allowing the model to take into account the vague aspirations of a decision-maker. Using preference-based membership functions, we can define the fuzzy problem through natural language terms or vague phenomena. In fact, decision-making involves the achievement of fuzzy goals, some of them are met and some not because these goals are subject to the function of environment/resource constraints. Thus, binary fuzzy goal programming is employed where the problem cannot be solved by conventional goal programming approaches. This paper proposes a new idea of how to program the binary fuzzy goal programming model. The binary fuzzy goal programming model can then be solved using the integer programming method. Finally, an illustrative example is included to demonstrate the correctness and usefulness of the proposed model.     

Duality in mathematics and linear and integer programming

Linear programming (LP) duality is examined in the context of other dualities in mathematics. The mathematical and economic properties of LP duality are discussed and its uses are considered. These mathematical and economic properties are then examined in relation to possible integer programming (IP) dualities. A number of possible IP duals are considered in this light and shown to capture some but not all desirable properties. It is shown that inherent in IP models are inequality and congruence constraints, both of which give on their own well-defined duals. However, taken together, no totally satisfactory dual emerges. The superadditive dual based on the Gomory and Chvátal functions is then described, and its properties are contrasted with LP duals and other IP duals. Finally, possible practical uses of IP duals are considered.The author is indebted to Professor H. B. Griffiths for many stimulating conversations on this topic.     

A linear programming embedded genetic algorithm for an integrated cell formation and lot sizing considering product quality

Production lot sizing models are often used to decide the best lot size to minimize operation cost, inventory cost, and setup cost. Cellular manufacturing analyses mainly address how machines should be grouped and parts be produced. In this paper, a mathematical programming model is developed following an integrated approach for cell configuration and lot sizing in a dynamic manufacturing environment. The model development also considers the impact of lot sizes on product quality. Solution of the mathematical model is to minimize both production and quality related costs. The proposed model, with nonlinear terms and integer variables, cannot be solved for real size problems efficiently due to its NP-complexity. To solve the model for practical purposes, a linear programming embedded genetic algorithm was developed. The algorithm searches over the integer variables and for each integer solution visited the corresponding values of the continuous variables are determined by solving a linear programming subproblem using the simplex algorithm. Numerical examples showed that the proposed method is efficient and effective in searching for near optimal solutions.     

A separable integer programming problem equivalent to its continual version

The separable integer programming problem with so called nested constraints is shown to be equivalent to its continual version obtained by piecewise linear continuation of the cost functions. A new approach to solution of the latter based on its successive reduction in size is suggested. When applied to the problem with piecewise linear convex functions it leads to two algorithms for its solution applicable also to the similar integer problem. These algorithms turn out more efficient than those obtained by dynamic programming approach.     

An interactive fuzzy satisficing method for random fuzzy multiobjective integer programming problems through probability maximization with possibility

This paper considers multiobjective integer programming problems where each coefficient of the objective functions is expressed by a random fuzzy variable. A new decision making model is proposed by incorporating the concept of probability maximization into a possibilistic programming model. For solving transformed deterministic problems, genetic algorithms with double strings for nonlinear integer programming problems are introduced. An interactive fuzzy satisficing method is presented for deriving a satisficing solution to a decision maker by updating the reference probability levels. An illustrative numerical example is provided to clarify the proposed method.     

A comparative analysis of linear fitting for non-linear functions on optimization: A case study: Air pollution problems

A very frequent problem in advanced mathematical programming models is the linear approximation of convex and non-convex non-linear functions in either the constraints or the objective function of an otherwise linear programming problem. In this paper, based on a model that has been developed for the evaluation and selection of pollutant emission control policies and standards, we shall study several ways of representing non-linear functions of a single argument in mixed integer, separable and related programming terms. Thus we shall study the approximations based on piecewise constant, piecewise adjacent, piecewise non-adjacent additional and piecewise non-adjacent segmented functions. In each type of modelization we show the problem size and optimization results of using the following techniques: separable programming, mixed integer programming with Special Ordered Sets of type 1, linear programming with Special Ordered Sets of type 2 and mixed integer programming using strategies based on the quasi-integrality of the binary variables.     

Ranking in quadratic integer programming problems

The present paper develops an algorithm for ranking the integer feasible solutions of a quadratic integer programming (QIP) problem. A linear integer programming (LIP) problem is constructed which provides bounds on the values of the objective function of the quadratic problem. The integer feasible solutions of this related integer linear programming problem are systematically scanned to rank the integer feasible solutions of the quadratic problem in non-decreasing order of the objective function values. The ranking in the QIP problem is useful in solving a nonlinear integer programming problem in which some other complicated nonlinear restrictions are imposed which cannot be included in the simple linear constraints of QIP, the objective function being still quadratic.     

An outline of linear programming

Linear programming deals with the maximization or minimization of linear functions subject to linear inequality constraints. Definitions and practical examples are given in Section 2. Section 3 gives a geometric interpretation of the simplex algorithm. Section 4 develops the mathematical theory of duality and gives heuristic interpretations in terms of shadow prices. Section 5 studies those linear programs for which integer solutions are normally required and discusses those cases in which integer solutions arise naturally and those in which special techniques must be used.The preparation of this paper was supported in part by the Army Research Office under Contract No. DA-ARO-D-31-124-70-G42.     

Support vector machines maximizing geometric margins for multi-class classification

Machine learning is a very interesting and important branch of artificial intelligence. Among many learning models, the support vector machine is a popular model with high classification ability which can be trained by mathematical programming methods. Since the model was originally formulated for binary classification, various kinds of extensions have been investigated for multi-class classification. In this paper, we review some existing models, and introduce new models which we recently proposed. The models are derived from the viewpoint of multi-objective maximization of geometric margins for a discriminant function, and each model can be trained by solving a second-order cone programming problem. We show that discriminant functions with high generalization ability can be obtained by these models through some numerical experiments.