k-Plane Clustering

A finite new algorithm is proposed for clustering m given points in n-dimensional real space into k clusters by generating k planes that constitute a local solution to the nonconvex problem of minimizing the sum of squares of the 2-norm distances between each point and a nearest plane. The key to the algorithm lies in a formulation that generates a plane in n-dimensional space that minimizes the sum of the squares of the 2-norm distances to each of m₁ given points in the space. The plane is generated by an eigenvector corresponding to a smallest eigenvalue of an n × n simple matrix derived from the m₁ points. The algorithm was tested on the publicly available Wisconsin Breast Prognosis Cancer database to generate well separated patient survival curves. In contrast, the k-mean algorithm did not generate such well-separated survival curves.     

Fitting concentric circles to measurements

Measurements for fitting a given number of concentric circles are recorded. For each concentric circle several measurements are taken. The problem is to fit the given number of circles to the data such that all circles have a common center. This is a generalization of the problem of fitting a set of points to one circle. Three objectives, to be minimized, are considered: the least squares of distances from the circles, the maximum distance from the circles, and the sum of the distances from the circles. Very efficient optimal solution procedures are constructed. Problems based on a total of 10,000 measurements are solved in about 10 s with the least squares objective, $<$ 2 s with the maximum distance objective, and a little more than 1 min for the minisum objective.     

Clustering Methods in Microarrays

Summary DCT Given a finite set of points in an Euclidean space the \emph{spanning tree} is a tree of minimal length having the given points as vertices. The length of the tree is the sum of the distances of all connected point pairs of the tree. The clustering tree with a given length of a given finite set of points is the spanning tree of an appropriately chosen other set of points approximating the given set of points with minimal sum of square distances among all spanning trees with the given length. DCM A matrix of real numbers is said to be column monotone orderable if there exists an ordering of columns of the matrix such that all rows of the matrix become monotone after ordering. The {\emph{monotone sum of squares of a matrix}} is the minimum of sum of squares of differences of the elements of the matrix and a column monotone orderable matrix where the minimum is taken on the set of all column monotone orderable matrices. Decomposition clusters of monotone orderings of a matrix is a clustering ofthe rows of the matrix into given number of clusters such that thesum of monotone sum of squares of the matrices formed by the rowsof the same cluster is minimal.DCP A matrix of real numbers is said to be column partitionable if there exists a partition of the columns such that the elements belonging to the same subset of the partition are equal in each row. Given a partition of the columns of a matrix the partition sum of squares of the matrix is the minimum of the sum of square of differences of the elements of the matrix and a column partitionable matrix where the minimum is taken on the set of all column partitionable matrices. Decomposition of the rows of a matrix into clusters of partitions is the minimization of the corresponding partition sum of squares given the number of clusters and the sizes of the subsets of the partitions.     

On the solution of the errors in variables problem using thel 1 norm

A fundamental problem in data analysis is that of fitting a given model to observed data. It is commonly assumed that only the dependent variable values are in error, and the least squares criterion is often used to fit the model. When significant errors occur in all the variables, then an alternative approach which is frequently suggested for this errors in variables problem is to minimize the sum of squared orthogonal distances between each data point and the curve described by the model equation. It has long been recognized that the use of least squares is not always satisfactory, and thel ₁ criterion is often superior when estimating the true form of data which contain some very inaccurate observations. In this paper the measure of goodness of fit is taken to be thel ₁ norm of the errors. A Levenberg-Marquardt method is proposed, and the main objective is to take full advantage of the structure of the subproblems so that they can be solved efficiently.     

Locating a general minisum ��circle�� on the plane

We approximate a set of given points in the plane by the boundary of a convex and symmetric set which is the unit circle of some norm. This generalizes previous work on the subject which considers Euclidean circles only. More precisely, we examine the problem of locating and scaling the unit circle of some given norm k with respect to given points on the plane such that the sum of weighted distances (as measured by the same norm k) between the circumference of the circle and the points is minimized. We present general results and are able to identify a finite dominating set in the case that k is a polyhedral norm.     

Optimization of multiple covering of a bounded set with circles

Numerical algorithms for the optimization of multiple covering of a bounded set G in the plane P with equal circles are proposed. The variants in which G is a connected bounded set in P or a finite set in P are considered. The circles may be centered at arbitrary points of G or at points belonging to a given set. Minimization of the radius of the given number of circles and minimization of the number of circles of a given radius are considered. Models and solution algorithms are described, and estimates of the solutions provided by most variants are given. Numerical results are presented.     

Locating a minisum circle in the plane

We consider the problem of locating a circle with respect to existing facilities in the plane such that the sum of weighted distances between the circle and the facilities is minimized, i.e., we approximate a set of given points by a circle regarding the sum of weighted distances. If the radius of the circle is a variable we show that there always exists an optimal circle passing through two of the existing facilities. For the case of a fixed radius we provide characterizations of optimal circles in special cases. Solution procedures are suggested.     

On a lower and upper bound for the curvature of ellipses with more than two foci

Let a set of points in the Euclidean plane be given. We are going to investigate the levels of the function measuring the sum of distances from the elements of the pointset which are called foci. Levels with only one focus are circles. In case of two different points as foci they are ellipses in the usual sense. If the set of the foci consists of more than two points then we have the so-called polyellipses. In this paper we investigate them from the viewpoint of differential geometry. We give a lower and upper bound for the curvature involving explicit constants. They depend on the number of the foci, the rate of the level and the global minimum of the function measuring the sum of the distances. The minimizer will be characterized by a theorem due to E. Weiszfeld together with a new proof. Explicit examples will also be given. As an application we present a new proof for a theorem due to P. Erd?s and I. Vincze. The result states that the approximation of a regular triangle by circumscribed polyellipses has an absolute error in the sense that there is no way to exceed it even if the number of the foci are arbitrary large.     

Flow shop scheduling algorithms for minimizing the completion time variance and the sum of squares of completion time deviations from a common due date

We consider two problems of m-machine flow shop scheduling in this paper: one, with the objective of minimizing the variance of completion times of jobs, and the other with the objective of minimizing the sum of squares of deviations of job completion times from a common due date. Lower bounds on the sum of squares of deviations of job completion times from the mean completion time of jobs for a given partial sequence are first presented. Using these lower bounds, a branch and bound algorithm based on breadth-first search procedure for scheduling n jobs on m-machines with the objective of minimizing completion time variance (CTV) is developed to obtain the best permutation sequence. We also present two lower bounds and thereafter, a branch and bound algorithm with the objective of minimizing the sum of squares of deviations of job completion times from a given common due date (called the MSD problem). The computational experience with the working of the two proposed branch and bound algorithms is also reported. Two heuristics, one for each of the two problems, are developed. The computational experience on the evaluation of the heuristics is discussed.     

Orthogonal regression: a teaching perspective

A well-known approach to linear least squares regression is that which involves minimizing the sum of squared orthogonal projections of data points onto the best fit line. This form of regression is known as orthogonal regression, and the linear model that it yields is known as the major axis. A similar method, reduced major axis regression, is predicated on minimizing the total sum of triangular areas formed between data points and the best fit line. Either of these methods is appropriately applied when both x and y are measured, a typical case in the natural sciences. In comparison to classical linear regression, equation derivation for the slope of the major axis and reduced major axis lines is a nontrivial process. For this reason, derivations are presented herein drawing from previous literature with as few steps as possible to enable an easily accessible understanding. Application to eruption data for Old Faithful geyser, Yellowstone National Park, Wyoming and Montana, USA enables a teaching opportunity for choice of model.     

Locating least-distant lines in the plane

In this paper we deal with locating a line in a plane. Given a set of existing facilities, represented by points in the plane, our objective is to find a straight line l minimizing the sum of weighted distances to the existing facilities, or minimizing the maximum weighted distance to the existing facilities, respectively. We show that for all distance measures derived from norms, one of the lines minimizing the sum objective contains at least two of the existing facilities. For the center objective we always get an optimal line which is at maximum distance from at least three of the existing facilities. If all weights are equal, there is an optimal line which is parallel to one facet of the convex hull of the existing facilities.     

Linear separation and approximation by minimizing the sum of concave functions of distances

One recently proposed criterion to separate two data sets in Classification is to use a hyperplane that minimizes the sum of distances to it from all the misclassified data points, where misclassification means lying on the wrong side of the hyperplane, or rather in the wrong halfspace. In this paper we study an extension of this problem: we seek the hyperplane minimizing the sum of concave nondecreasing functions of the distances of misclassified points to it. It is shown that an optimal hyperplane exists containing at least $d$ affinely independent points. This extends the result known for the minimization of the sum of distances, and enables to use combinatorial local-search heuristics for this problem. As a corollary, the same result is obtained for the approximation problem in which a hyperplane minimizing the sum of concave nondecreasing functions of the distances from a set of data points is sought.     

Cutting ellipses from area-minimizing rectangles

A set of ellipses, with given semi-major and semi-minor axes, is to be cut from a rectangular design plate, while minimizing the area of the design rectangle. The design plate is subject to lower and upper bounds of its widths and lengths; the ellipses are free of any orientation restrictions. We present new mathematical programming formulations for this ellipse cutting problem. The key idea in the developed non-convex nonlinear programming models is to use separating hyperlines to ensure the ellipses do not overlap with each other. For small number of ellipses we compute feasible points which are globally optimal subject to the finite arithmetic of the global solvers at hand. However, for more than 14 ellipses none of the local or global NLP solvers available in GAMS can even compute a feasible point. Therefore, we develop polylithic approaches, in which the ellipses are added sequentially in a strip-packing fashion to the rectangle restricted in width, but unrestricted in length. The rectangle’s area is minimized in each step in a greedy fashion. The sequence in which we add the ellipses is random; this adds some GRASP flavor to our approach. The polylithic algorithms allow us to compute good, near optimal solutions for up to 100 ellipses.     

Incomplete total least squares

Fitting data points with some model function such that the sum of squared orthogonal distances is minimized is well-known as TLS, i.e. as total least squares, see Van Huffel (1997). We consider situations where the model is such that there might be no perpendiculars from certain data points onto the model function and where one has to replace certain orthogonal distances by shortest ones, e.g. to corner or border line points. We introduce this considering the (now incomplete) TLS fit by a finite piece of a straight line. Then we study general model functions with linear parameters and modify a well-known descent algorithm (see Seufer (1996), Seufer/Sp?th (1997), Sp?th (1996), Sp?th (1997a) and Sp?th (1997b)) for fitting with them. As applications (to be used in computational metrology) we discuss incomplete TLS fitting with a segment of a circle, the area of a circle in space, with a cylinder, and with a rectangle (see also Gander/Hrebicek (1993)). Numerical examples are given for each case. Received August 27, 1997 / Revised version received February 23, 1998     

An aggregate subgradient method for nonsmooth convex minimization

A class of implementable algorithms is described for minimizing any convex, not necessarily differentiable, functionf of several variables. The methods require only the calculation off and one subgradient off at designated points. They generalize Lemarechal's bundle method. More specifically, instead of using all previously computed subgradients in search direction finding subproblems that are quadratic programming problems, the methods use an aggregate subgradient which is recursively updated as the algorithms proceed. Each algorithm yields a minimizing sequence of points, and iff has any minimizers, then this sequence converges to a solution of the problem. Particular members of this algorithm class terminate whenf is piecewise linear. The methods are easy to implement and have flexible storage requirements and computational effort per iteration that can be controlled by a user. Research sponsored by the Institute of Automatic Control, Technical University of Warsaw, Poland, under Project R.I.21.     

A coordinate gradient descent method for <Emphasis Type="Italic">ℓ</Emphasis><Subscript>1</Subscript>-regularized convex minimization

In applications such as signal processing and statistics, many problems involve finding sparse solutions to under-determined linear systems of equations. These problems can be formulated as a structured nonsmooth optimization problems, i.e., the problem of minimizing ℓ ₁-regularized linear least squares problems. In this paper, we propose a block coordinate gradient descent method (abbreviated as CGD) to solve the more general ℓ ₁-regularized convex minimization problems, i.e., the problem of minimizing an ℓ ₁-regularized convex smooth function. We establish a Q-linear convergence rate for our method when the coordinate block is chosen by a Gauss-Southwell-type rule to ensure sufficient descent. We propose efficient implementations of the CGD method and report numerical results for solving large-scale ℓ ₁-regularized linear least squares problems arising in compressed sensing and image deconvolution as well as large-scale ℓ ₁-regularized logistic regression problems for feature selection in data classification. Comparison with several state-of-the-art algorithms specifically designed for solving large-scale ℓ ₁-regularized linear least squares or logistic regression problems suggests that an efficiently implemented CGD method may outperform these algorithms despite the fact that the CGD method is not specifically designed just to solve these special classes of problems.     

UOBYQA: unconstrained optimization by quadratic approximation

UOBYQA is a new algorithm for general unconstrained optimization calculations, that takes account of the curvature of the objective function, F say, by forming quadratic models by interpolation. Therefore, because no first derivatives are required, each model is defined by ?(n+1)(n+2) values of F, where n is the number of variables, and the interpolation points must have the property that no nonzero quadratic polynomial vanishes at all of them. A typical iteration of the algorithm generates a new vector of variables, _t say, either by minimizing the quadratic model subject to a trust region bound, or by a procedure that should improve the accuracy of the model. Then usually F( _t ) is obtained, and one of the interpolation points is replaced by _t . Therefore the paper addresses the initial positions of the interpolation points, the adjustment of trust region radii, the calculation of _t in the two cases that have been mentioned, and the selection of the point to be replaced. Further, UOBYQA works with the Lagrange functions of the interpolation equations explicitly, so their coefficients are updated when an interpolation point is moved. The Lagrange functions assist the procedure that improves the model, and also they provide an estimate of the error of the quadratic approximation to F, which allows the algorithm to achieve a fast rate of convergence. These features are discussed and a summary of the algorithm is given. Finally, a Fortran implementation of UOBYQA is applied to several choices of F, in order to investigate accuracy, robustness in the presence of rounding errors, the effects of first derivative discontinuities, and the amount of work. The numerical results are very promising for n≤20, but larger values are problematical, because the routine work of an iteration is of fourth order in the number of variables. Received: December 7, 2000 / Accepted: August 31, 2001?Published online April 12, 2002     

Algorithms for the computation of functionals defined on the solution of a discrete ill-posed problem

We study a linear, discrete ill-posed problem, by which we mean a very ill-conditioned linear least squares problem. In particular we consider the case when one is primarily interested in computing a functional defined on the solution rather than the solution itself. In order to alleviate the ill-conditioning we require the norm of the solution to be smaller than a given constant. Thus we are lead to minimizing a linear functional subject to two quadratic constraints. We study existence and uniqueness for this problem and show that it is essentially equivalent to a least squares problem with a linear and a quadratic constraint, which is easier to handle computationally. Efficient algorithms are suggested for this problem.     

Construction of two-step Runge-Kutta methods of high order for ordinary differential equations

The construction of two-step Runge-Kutta methods of order p and stage order q=p with stability polynomial given in advance is described. This polynomial is chosen to have a large interval of absolute stability for explicit methods and to be A-stable and L-stable for implicit methods. After satisfying the order and stage order conditions the remaining free parameters are computed by minimizing the sum of squares of the difference between the stability function of the method and a given polynomial at a sufficiently large number of points in the complex plane. This revised version was published online in August 2006 with corrections to the Cover Date.     

Book Reviews

This paper modifies the usual meaning of regression from ‘minimizing the sum of squared distances’ to ‘minimizing the sum of square perpendicular distances’. With this modified definition, the best‐fit plane, circle, and sphere may be meaningfully considered.

These regression problems are motivated by the current development of software to control robotized coordinate measuring machines (CMMs) used to perform quality assurance work. A brief outline of CMMs is given and the application of the modified regression definition to a plane, circle and sphere is illustrated. In §3 the equation of the best fitting plane is calculated for various sets of data points.