Gradient-based Regularization Parameter Selection for Problems With Nonsmooth Penalty Functions

In high-dimensional and/or nonparametric regression problems , regularization (or penalization) is used to control model complexity and induce desired structure. Each penalty has a weight parameter that indicates how strongly the structure corresponding to that penalty should be enforced. Typically, the parameters are chosen to minimize the error on a separate validation set using a simple grid search or a gradient-free optimization method. It is more efficient to tune parameters if the gradient can be determined, but this is often difficult for problems with nonsmooth penalty functions. Here, we show that for many penalized regression problems, the validation loss is actually smooth almost-everywhere with respect to the penalty parameters. We can, therefore, apply a modified gradient descent algorithm to tune parameters. Through simulation studies on example regression problems, we find that increasing the number of penalty parameters and tuning them using our method can decrease the generalization error.     

Optimal Parameter Tuning for Multiclass Support Vector Machines in Machinery Health State Estimation

The increasing demand for high reliability, safety and availability of technical systems calls for innovative maintenance strategies. The use of prognostic health management (PHM) approach where maintenance action is taken based on current and future health state of a component or system is rapidly gaining popularity in the maintenance industry. Multiclass support vector machines (MC-SVM) has been identified as a promising algorithm in PHM applications due to its high classification accuracy. However, it requires parameter tuning for each application, with the objective of minimizing the classification error. This is a single objective optimization problem which requires the use of optimization algorithms that are capable of exhaustively searching for the global optimum parameters. This work proposes the use of hybrid differential evolution (DE) and particle swarm optimization (PSO) in optimally tuning the MC-SVM parameters. DE identifies the search limit of the parameters while PSO finds the global optimum within the search limit. The feasibility of the approach is verified using bearing run-to-failure data and the results show that the proposed method significantly increases health state classification accuracy. (© 2014 Wiley-VCH Verlag GmbH & Co. KGaA, Weinheim)     

A very fast algorithm for matrix factorization

We present a very fast algorithm for general matrix factorization of a data matrix for use in the statistical analysis of high-dimensional data via latent factors. Such data are prevalent across many application areas and generate an ever-increasing demand for methods of dimension reduction in order to undertake the statistical analysis of interest. Our algorithm uses a gradient-based approach which can be used with an arbitrary loss function provided the latter is differentiable. The speed and effectiveness of our algorithm for dimension reduction is demonstrated in the context of supervised classification of some real high-dimensional data sets from the bioinformatics literature.     

Energy contracts management by stochastic programming techniques

We consider the problem of optimal management of energy contracts, with bounds on the local (time step) amounts and global (whole period) amounts to be traded, integer constraint on the decision variables and uncertainty on prices only. After building a finite state Markov chain by using vectorial quantization tree method, we rely on the stochastic dual dynamic programming (SDDP) method to solve the continuous relaxation of this stochastic optimization problem. An heuristic for computing sub optimal solutions to the integer optimization problem, based on the Bellman values of the continuous relaxation, is provided. Combining the previous techniques, we are able to deal with high-dimensional state variables problems. Numerical tests applied to realistic energy markets problems have been performed.     

Minimizing Sum of Truncated Convex Functions and Its Applications

In this article, we study a class of problems where the sum of truncated convex functions is minimized. In statistical applications, they are commonly encountered when ?₀-penalized models are fitted and usually lead to NP-Hard non-convex optimization problems. In this article, we propose a general algorithm for the global minimizer in low-dimensional settings. We also extend the algorithm to high-dimensional settings, where an approximate solution can be found efficiently. We introduce several applications where the sum of truncated convex functions is used, compare our proposed algorithm with other existing algorithms in simulation studies, and show its utility in edge-preserving image restoration on real data.     

Lease expiration management for a single lease term in the apartment industry

Lease expiration management (LEM) in the apartment industry aims to control the number of lease expirations and thus achieve maximal revenue growth. We examine rental rate strategies in the context of LEM for apartment buildings that offer a single lease term and face demand uncertainty. We show that the building may incur a significant revenue loss if it fails to account for LEM in the determination of the rental rate. We also show that the use of LEM is a compromise approach between a limited optimization, where no future demand information is available, and a global optimization, where complete future demand information is available. We show that the use of LEM can enhance the apartment building’s revenue by as much as 8% when the desired number of expirations and associated costs are appropriately estimated. Numerical examples are included to illustrate the major results derived from our models and the impact on the apartment’s revenue of sensitivity to the desired number of expirations and associated costs.     

A solution approach for optimizing long- and short-term production scheduling at LKAB’s Kiruna mine

We present a mixed-integer program to schedule long- and short-term production at LKAB’s Kiruna mine, an underground sublevel caving mine located in northern Sweden. The model minimizes deviations from monthly preplanned production quantities while adhering to operational constraints. Because of the mathematical structure of the model and its moderately large size, instances spanning a time horizon of more than a year or two tend to be intractable. We develop an optimization-based decomposition heuristic that, on average, obtains better solutions faster than solving the model directly. We show that for realistic data sets, we can generate solutions with deviations that comprise about 3-6% of total demand in about a third of an hour.     

Bounds on a trauma outcome function via optimization

One measure of the effectiveness of institutional trauma and burn management based on collected patient data involves the computation of a standard normal Z statistic. A potential weakness of the measure arises from incomplete patient data. In this paper, we apply methods of fractional programming and global optimization to efficiently calculate bounds on the computed effectiveness of an institution. The measure of effectiveness (i.e., the trauma outcome function) is briefly described, the optimization problems associated with its upper and lower bounds are defined and characterized, and appropriate solution procedures are developed. We solve an example problem to illustrate the method.     

A benders decomposition approach for an integrated airline schedule design and fleet assignment problem with flight retiming, schedule balance, and demand recapture

The airline’s ability to offer flight schedules that provide service to passengers at desired times in competitive markets, while matching demand with an aircraft fleet of suitable size and composition, can significantly impact its profits. In this spirit, optional flight legs can be considered to construct a profitable schedule by optimally selecting among such alternatives in concert with assigning the available aircraft fleet to all the scheduled legs. Examining itinerary-based demands as well as multiple fare-classes can effectively capture network effects and realistic demand patterns. In addition, allowing flexibility on the departure times of scheduled flight legs can increase connection opportunities for passengers, hence yielding robust schedules while saving fleet assignment costs within the framework of an integrated model. Airlines can also capture an adequate market share by balancing flight schedules throughout the day, and recapture considerations can contribute to more realistic accepted demand realizations. We therefore propose in this paper a model that integrates the schedule design and fleet assignment processes while considering flexible flight times, schedule balance, and recapture issues, along with optional legs, path/itinerary-based demands, and multiple fare-classes. A polyhedral analysis is conducted to generate several classes of valid inequalities, which are used along with suitable separation routines to tighten the model representation. Solution approaches are designed by applying Benders decomposition method to the resulting tightened model, and computational results are presented using real data obtained from United Airlines to demonstrate the efficacy of the proposed procedures.     

A Lagrangian relaxation approach to the mixed-product assembly line sequencing problem: A case study of a door-lock company in Taiwan

In mixed-product assembly line sequencing, the production resources required for the assembly lines should be scheduled to minimize the overall cost and meet customer demand. In this paper, we study an assembly line sequencing problem for the door-lock industry in Taiwan and develop an integer programming formulation with realistic constraints. The complex solution space makes the resulting program difficult to solve using commercial optimization packages. Therefore, a heuristic based on the Lagrangian relaxation principle is developed to solve this problem efficiently. We evaluate the efficiency of the developed Lagrangian relaxation heuristic by comparing its solutions with those obtained using a commercial optimization package: the computational results show that the developed heuristic solves the real-world problem faster than the optimization package by almost 15 times in CPU time at a comparable solution quality.     

Sensitivity analysis and calibration of the covariance matrix for stable portfolio selection

We recommend an implementation of the Markowitz problem to generate stable portfolios with respect to perturbations of the problem parameters. The stability is obtained proposing novel calibrations of the covariance matrix between the returns that can be cast as convex or quasiconvex optimization problems. A statistical study as well as a sensitivity analysis of the Markowitz problem allow us to justify these calibrations. Our approach can be used to do a global and explicit sensitivity analysis of a class of quadratic optimization problems. Numerical simulations finally show the benefits of the proposed calibrations using real data.     

Estimating the Number of Clusters via the GUD Statistic

Estimating the number of clusters is one of the most difficult problems in cluster analysis. Most previous approaches require knowing the data matrix and may not work when only a Euclidean distance matrix is available. Other approaches also suffer from the curse of dimensionality and work poorly in high dimension. In this article, we develop a new statistic, called the GUD statistic, based on the idea of the Gap method, but use the determinant of the pooled within-group scatter matrix instead of the within-cluster sum of squared distances. Some theory is developed to show this statistic can work well when only the Euclidean distance matrix is known. More generally, this statistic can even work for any dissimilarity matrix that satisfies some properties. We also propose a modification for high-dimensional datasets, called the R-GUD statistic, which can give a robust estimation in high-dimensional settings. The simulation shows our method needs less information but is generally found to be more accurate and robust than other methods considered in the study, especially in many difficult settings.     

Comparison of iterative improvement techniques for schedule optimization

Due to complexity reasons of realistic scheduling applications, often iterative improvement techniques that perform a kind of local search to improve a given schedule are proposed instead of enumeration techniques that guarantee optimal solutions. In this paper we describe an experimental comparison of four iterative improvement techniques for schedule optimization that differ in the local search methodology. These techniques are iterative deepening, random search, tabu search and genetic algorithms. To compare the performance of these techniques, we use the same evaluation function, knowledge representation and data from one application. The evaluation function is defined on the gradual satisfaction of explicitly represented domain constraints and optimization functions. The satisfactions of individual constraints are weighted and aggregated for the whole schedule. We have applied these techniques on data of a steel making plant in Linz (Austria). In contrast to other applications of iterative improvement techniques reported in the literature, our application is constrained by a greater variety of antagonistic criteria that are partly contradictory.     

基于偏最小二乘变换与Kriging模型的全局优化方法

针对传统Kriging模型在多变量(高维)输入全局优化中因超参数过多而引发收敛速度慢,精度低,建模效率不高问题,提出了基于偏最小二乘变换技术和Kriging模型的有效全局优化方法.首先,构造偏最小二乘高斯核函数;其次,借助差分进化算法寻找满足期望改进准则最大化条件的新样本点;然后,将不同核函数和期望改进准则组合,构建四种有效全局优化算法并进行比较;最后,数值算例结果表明,基于偏最小二乘变换的Kriging全局优化方法在解决高维全局优化问题方面相比于标准的全局优化算法在收敛精度及收敛速度方面更具优势.     

Armijo搜索下的记忆梯度法及其收敛性

本文提出一种新的无约束优化记忆梯度算法,在Armijo搜索下,该算法在每步迭代时利用了前面迭代点的信息,增加了参数选择的自由度,适于求解大规模无约束优化问题。分析了算法的全局收敛性。     

Encoding and decoding of analog signals with a population of neurons

In this paper, we discuss how an analog signal can be encoded using biophysically realistic neural networks. Using the activity curve of a single neuron, we argue that the activities can be pooled over a population so that the weighted sum of the activities approximate a given function. Since the activities of neurons are not available as a variable, we propose to generate them in real time by a suitable low-pass filter. Using the proposed scheme, we demonstrate how simple ordinary differential equations can be solved. In effect, the ordinary differential equations are solved by dynamically updating the activities of the neurons. In an actual biological neural network, the activities of the cells are not obtained by a low-pass filter. They are integrated in the network by a suitable synaptic input. A new optimization algorithm for finding a set of optimal synaptic weights has been proposed and successfully implemented using a software package GENESIS. The difference between biological neural networks and artificial neural networks is discussed in somewhat greater details. The important concepts are illustrated by implementing a memory and by solving a periodic ordinary differential equation, the Van der Pol oscillator.     

Chaotic harmony search algorithms

Harmony Search (HS) is one of the newest and the easiest to code music inspired heuristics for optimization problems. Like the use of chaos in adjusting note parameters such as pitch, dynamic, rhythm, duration, tempo, instrument selection, attack time, etc. in real music and in sound synthesis and timbre construction, this paper proposes new HS algorithms that use chaotic maps for parameter adaptation in order to improve the convergence characteristics and to prevent the HS to get stuck on local solutions. This has been done by using of chaotic number generators each time a random number is needed by the classical HS algorithm. Seven new chaotic HS algorithms have been proposed and different chaotic maps have been analyzed in the benchmark functions. It has been detected that coupling emergent results in different areas, like those of HS and complex dynamics, can improve the quality of results in some optimization problems. It has been also shown that, some of the proposed methods have somewhat increased the solution quality, that is in some cases they improved the global searching capability by escaping the local solutions.     

A model for the periodic optimization of bus dispatching times

We model the problem of dispatching time control in rolling horizons following a periodic optimization approach reactionary to travel time and passenger demand disturbances. This model provides more flexibility to transport planners allowing them to adjust the bus schedules during the daily operations. We prove that our periodic optimization model is a convex quadratic program, guaranteeing the global optimality of its solution. To reduce the computational burden, we introduce an iterative algorithm that uses gradient approximations to obtain an approximate dispatching solution. The proposed solution method is found to be significantly faster than exact optimization approaches for quadratic programming and maintains an (almost) negligible optimality gap in realistic bus operation scenarios. Finally, we show that our periodic optimization method outperforms myopic methods that adjust the dispatching time of each bus trip in isolation using operational data from bus line 302 in Singapore.     

Persistent Cohomology and Circular Coordinates

Nonlinear dimensionality reduction (NLDR) algorithms such as Isomap, LLE, and Laplacian Eigenmaps address the problem of representing high-dimensional nonlinear data in terms of low-dimensional coordinates which represent the intrinsic structure of the data. This paradigm incorporates the assumption that real-valued coordinates provide a rich enough class of functions to represent the data faithfully and efficiently. On the other hand, there are simple structures which challenge this assumption: the circle, for example, is one-dimensional, but its faithful representation requires two real coordinates. In this work, we present a strategy for constructing circle-valued functions on a statistical data set. We develop a machinery of persistent cohomology to identify candidates for significant circle-structures in the data, and we use harmonic smoothing and integration to obtain the circle-valued coordinate functions themselves. We suggest that this enriched class of coordinate functions permits a precise NLDR analysis of a broader range of realistic data sets.     

A global optimization method for the design of space trajectories

The problem of optimally designing a trajectory for a space mission is considered in this paper. Actual mission design is a complex, multi-disciplinary and multi-objective activity with relevant economic implications. In this paper we will consider some simplified models proposed by the European Space Agency as test problems for global optimization (GTOP database). We show that many trajectory optimization problems can be quite efficiently solved by means of relatively simple global optimization techniques relying on standard methods for local optimization. We show in this paper that our approach has been able to find trajectories which in many cases outperform those already known. We also conjecture that this problem displays a “funnel structure” similar, in some sense, to that of molecular optimization problems.