Global optimization of general constrained grey-box models: new method and its application to constrained PDEs for pressure swing adsorption

This paper introduces a novel methodology for the global optimization of general constrained grey-box problems. A grey-box problem may contain a combination of black-box constraints and constraints with a known functional form. The novel features of this work include (i) the selection of initial samples through a subset selection optimization problem from a large number of faster low-fidelity model samples (when a low-fidelity model is available), (ii) the exploration of a diverse set of interpolating and non-interpolating functional forms for representing the objective function and each of the constraints, (iii) the global optimization of the parameter estimation of surrogate functions and the global optimization of the constrained grey-box formulation, and (iv) the updating of variable bounds based on a clustering technique. The performance of the algorithm is presented for a set of case studies representing an expensive non-linear algebraic partial differential equation simulation of a pressure swing adsorption system for \(\hbox {CO}_{2}\). We address three significant sources of variability and their effects on the consistency and reliability of the algorithm: (i) the initial sampling variability, (ii) the type of surrogate function, and (iii) global versus local optimization of the surrogate function parameter estimation and overall surrogate constrained grey-box problem. It is shown that globally optimizing the parameters in the parameter estimation model, and globally optimizing the constrained grey-box formulation has a significant impact on the performance. The effect of sampling variability is mitigated by a two-stage sampling approach which exploits information from reduced-order models. Finally, the proposed global optimization approach is compared to existing constrained derivative-free optimization algorithms.     

改进二元分布估计算法求解置换流水车间调度问题

针对置换流水车间调度这类组合最优化问题的求解,提出了一种改进二元分布估计算法(Improved binary estimation distribution algorithm, I-EDA)。算法以二元分布估计算法为架构,使用NEH(Nawaz-Enscore-Ham)启发式算法生成初始解,提高了初始解的质量;通过对优势解的统计采样构建位置矩阵模型和链接矩阵模型,依照两个矩阵模型的合并概率组合链接区块产生子代。提出了NEH插入式重组策略和基于位置概率的交换策略和两种全新局部搜索机制替代原二元分布估计算法的相邻交换法,以进一步筛选优势解。最后通过对Reeves标准测试集的仿真实验和算法比较验证了所提出算法的有效性。     

Influence of ensemble surrogate models and sampling strategy on the solution quality of algorithms for computationally expensive black-box global optimization problems

This paper examines the influence of two major aspects on the solution quality of surrogate model algorithms for computationally expensive black-box global optimization problems, namely the surrogate model choice and the method of iteratively selecting sample points. A random sampling strategy (algorithm SO-M-c) and a strategy where the minimum point of the response surface is used as new sample point (algorithm SO-M-s) are compared in numerical experiments. Various surrogate models and their combinations have been used within the SO-M-c and SO-M-s sampling frameworks. The Dempster–Shafer Theory approach used in the algorithm by Müller and Piché (J Glob Optim 51:79–104, 2011) has been used for combining the surrogate models. The algorithms are numerically compared on 13 deterministic literature test problems with 2–30 dimensions, an application problem that deals with groundwater bioremediation, and an application that arises in energy generation using tethered kites. NOMAD and the particle swarm pattern search algorithm (PSWARM), which are derivative-free optimization methods, have been included in the comparison. The algorithms have also been compared to a kriging method that uses the expected improvement as sampling strategy (FEI), which is similar to the Efficient Global Optimization (EGO) algorithm. Data and performance profiles show that surrogate model combinations containing the cubic radial basis function (RBF) model work best regardless of the sampling strategy, whereas using only a polynomial regression model should be avoided. Kriging and combinations including kriging perform in general worse than when RBF models are used. NOMAD, PSWARM, and FEI perform for most problems worse than SO-M-s and SO-M-c. Within the scope of this study a Matlab toolbox has been developed that allows the user to choose, among others, between various sampling strategies and surrogate models and their combinations. The open source toolbox is available from the authors upon request.     

Path relinking and GRG for artificial neural networks

Artificial neural networks (ANN) have been widely used for both classification and prediction. This paper is focused on the prediction problem in which an unknown function is approximated. ANNs can be viewed as models of real systems, built by tuning parameters known as weights. In training the net, the problem is to find the weights that optimize its performance (i.e., to minimize the error over the training set). Although the most popular method for training these networks is back propagation, other optimization methods such as tabu search or scatter search have been successfully applied to solve this problem. In this paper we propose a path relinking implementation to solve the neural network training problem. Our method uses GRG, a gradient-based local NLP solver, as an improvement phase, while previous approaches used simpler local optimizers. The experimentation shows that the proposed procedure can compete with the best-known algorithms in terms of solution quality, consuming a reasonable computational effort.     

An artificial neural network meta-model for constrained simulation optimization

This paper presents artificial neural network (ANN) meta-models for expensive continuous simulation optimization (SO) with stochastic constraints. These meta-models are used within a sequential experimental design to approximate the objective function and the stochastic constraints. To capture the non-linear nature of the ANN, the SO problem is iteratively approximated via non-linear programming problems whose (near) optimal solutions obtain estimates of the global optima. Following the optimization step, a cutting plane-relaxation scheme is invoked to drop uninformative estimates of the global optima from the experimental design. This approximation is iterated until a terminating condition is met. To study the robustness and efficiency of the proposed algorithm, a realistic inventory model is used; the results are compared with those of the OptQuest optimization package. These numerical results indicate that the proposed meta-model-based algorithm performs quite competitively while requiring slightly fewer simulation observations.     

Modelling public transport trips by radial basis function neural networks

Artificial neural networks (ANNs) are one of the recently explored advanced technologies, which show promise in the area of transportation engineering. The presented study used two different ANN algorithms, feed forward back-propagation (FFBP) and radial basis function (RBF), for the purpose of daily trip flow forecasting. The ANN predictions were quite close to the observations as reflected in the selected performance criteria. The selected stochastic model performance was quite poor compared with ANN results. It was seen that the RBF neural network did not provide negative forecasts in contrast to FFBP applications. Besides, the local minima problem faced by some FFBP algorithms was not encountered in RBF networks.     

Analysis of natural convection from a column of cold horizontal cylinders using Artificial Neural Network

Artificial Neural Networks (ANNs) offer an alternative way to tackle complex problems. They can learn from the examples and once trained can perform predictions and generalizations at high speed. They are particularly useful in behavior or system identification. According to the above advantages of ANN in the present paper ANN is used to predict natural convection heat transfer and fluid flow from a column of cold horizontal circular cylinders having uniform surface temperature. Governing equations are solved in a few specified cases by finite volume method to generate the database for training the ANN in the range of Rayleigh numbers of 10⁵–10⁸ and a range of cylinder spacing of 0.5, 1.0, and 1.5 diameters, thereafter a Multi-Layer Perceptron (MLP) network is used to capture the behavior of flow and temperature fields and then generalized this behavior to predict the flow and temperature fields for any other Rayleigh numbers. Different training algorithms are used and it is found that the resilient back-propagation algorithm is the best algorithm regarding the faster training procedure. To validate the accuracy of the trained network, comparison is performed among the ANN and available CFD results. It is observed that ANN can be used more efficiently to determine cold plume and thermal field in lesser computational time. Based on the generalized results from the ANN new correlations are developed to estimate natural convection from a column of cold horizontal cylinders with respect to a single horizontal cylinder.     

A hybrid optimization approach for automated parameter estimation problems

We present a hybrid optimization approach for solving automated parameter estimation models. The hybrid approach is based on the coupling of the Simultaneous Perturbation Stochastic Approximation (SPSA) [1] and a Newton-Krylov Interior-Point method (NKIP) [2] via a surrogate model. The global method SPSA performs a stochastic search to find target regions with low function values. Next, we generate a surrogate model based on the points of regions on which the local method NKIP algorithm is applied for finding an optimal solution. We illustrate the behavior of the hybrid optimization algorithm on one testcase. (© 2008 WILEY-VCH Verlag GmbH & Co. KGaA, Weinheim)     

Comparison of non-linear optimization algorithms for yield curve estimation

The yield curve is a very important financial tool used in investment and policy decisions. Its estimation from market data is essentially a non-linear optimization problem. In this paper, we compare a diversity of non-linear optimization algorithms for estimating yield curves based on actual bond market data and conclude that certain classes of algorithms are more effective due to the nature of the problem.     

Scale economies and production function estimation for object-oriented software component and source code documentation size

In this study, we investigate the factors that influence the object-oriented (OO) component size and source code documentation. For multiple inputs and multiple outputs, we use data envelopment analysis to illustrate that non-linear variable returns to scale (VRS) economies exist for OO component size and source code documentation. The existence of non-linear variable returns to scale economies indicates that non-linear regression models will perform better than linear regression models. Using empirical data, we compare the performance of non-linear artificial neural network (ANN) forecasting model and linear regression model. Our results indicate that the ANN model performs well when VRS economies exist between multiple inputs and multiple outputs.     

Mixture surrogate models based on Dempster-Shafer theory for global optimization problems

Recent research in algorithms for solving global optimization problems using response surface methodology has shown that it is in general not possible to use one surrogate model for solving different kinds of problems. In this paper the approach of applying Dempster-Shafer theory to surrogate model selection and their combination is described. Various conflict redistribution rules have been examined with respect to their influence on the results. Furthermore, the implications of the surrogate model type, i.e. using combined, single or a hybrid of both, have been studied. The suggested algorithms were applied to several well-known global optimization test problems. The results indicate that the used approach leads for all problems to a thorough exploration of the variable domain, i.e. the vicinities of global optima could be detected, and that the global minima could in most cases be approximated with high accuracy.     

ARTIFICIAL NEURAL NETWORK MODELING FOR REFORESTATION DESIGN THROUGH THE DOMINANT TREES BOLE‐VOLUME ESTIMATION

Abstract In the management of restoration reforestations or recreational reforestations of trees, the density of the planted trees and the site conditions can influence the growth and bole volume of the dominant tree. The ability to influence growth of these trees in a reforestation contributes greatly to the formation of large dimension trees and thereby to the production of commercially valuable wood. The potential of two artificial neural network (ANN) architectures in modeling the dominant Pinus brutia tree bole volume in reforestation configuration at 12 years of age was investigated: (1) the multilayer perceptron architecture using a back‐propagation algorithm and (2) the cascade‐correlation architecture, utilizing (a) either the nonlinear Kalman's filter theory or (b) the adaptive gradient descent learning rule. The incentive for developing bole‐volume equations using ANN techniques was to demonstrate an alternative new methodology in the field of reforestation design, which would enable estimation and optimization of the bole volume of dominant trees in reforestations using easily measurable site and competition factors. The usage of the ANNs for the estimation of dominant tree bole volume through site and competition factors can be a very useful tool in forest management practice.     

Maximum a Posteriori Sequence Estimation Using Monte Carlo Particle Filters

We develop methods for performing maximum a posteriori (MAP) sequence estimation in non-linear non-Gaussian dynamic models. The methods rely on a particle cloud representation of the filtering distribution which evolves through time using importance sampling and resampling ideas. MAP sequence estimation is then performed using a classical dynamic programming technique applied to the discretised version of the state space. In contrast with standard approaches to the problem which essentially compare only the trajectories generated directly during the filtering stage, our method efficiently computes the optimal trajectory over all combinations of the filtered states. A particular strength of the method is that MAP sequence estimation is performed sequentially in one single forwards pass through the data without the requirement of an additional backward sweep. An application to estimation of a non-linear time series model and to spectral estimation for time-varying autoregressions is described.     

Derivative-free optimization and neural networks for robust regression

Large outliers break down linear and nonlinear regression models. Robust regression methods allow one to filter out the outliers when building a model. By replacing the traditional least squares criterion with the least trimmed squares (LTS) criterion, in which half of data is treated as potential outliers, one can fit accurate regression models to strongly contaminated data. High-breakdown methods have become very well established in linear regression, but have started being applied for non-linear regression only recently. In this work, we examine the problem of fitting artificial neural networks (ANNs) to contaminated data using LTS criterion. We introduce a penalized LTS criterion which prevents unnecessary removal of valid data. Training of ANNs leads to a challenging non-smooth global optimization problem. We compare the efficiency of several derivative-free optimization methods in solving it, and show that our approach identifies the outliers correctly when ANNs are used for nonlinear regression.     

Multi-step prediction of time series with random missing data

Multi-step prediction is still an open challenge in time series prediction. Moreover, practical observations are often incomplete because of sensor failure or outliers causing missing data. Therefore, it is very important to carry out research on multi-step prediction of time series with random missing data. Based on nonlinear filters and multilayer perceptron artificial neural networks (ANNs), one novel approach for multi-step prediction of time series with random missing data is proposed in the study. With the basis of original nonlinear filters which do not consider the missing data, first we obtain the generalized nonlinear filters by using a sequence of independent Bernoulli random variables to model random interruptions. Then the multi-step prediction model of time series with random missing data, which can be fit for the online training of generalized nonlinear filters, is established by using the ANN’s weights to present the state vector and the ANN’s outputs to present the observation equation. The performance between the original nonlinear filters based ANN model for multi-step prediction of time series with missing data and the generalized nonlinear filters based ANN model for multi-step prediction of time series with missing data is compared. Numerical results have demonstrated that the generalized nonlinear filters based ANN are proportionally superior to the original nonlinear filters based ANN for multi-step prediction of time series with missing data.     

A computational study on the performance of artificial neural networks under changing structural design and data distribution

This paper provides the results of our computational studies on artificial neural networks (ANNs) under various structural design and data distributions. A two-group classification problem is investigated. Simulated data with varying kurtosis and variance are used to examine how the ANN performs with respect to certain structural design (size and addition of input and weight noise) characteristics. The results of our study indicate that additive noise, size, and data distribution characteristics play an important role in learning, reliability and predictive ability of ANNs.     

Parameters optimization of selected casting processes using teaching–learning-based optimization algorithm

In the present work, mathematical models of three important casting processes are considered namely squeeze casting, continuous casting and die casting for the parameters optimization of respective processes. A recently developed advanced optimization algorithm named as teaching–learning-based optimization (TLBO) is used for the parameters optimization of these casting processes. Each process is described with a suitable example which involves respective process parameters. The mathematical model related to the squeeze casting is a multi-objective problem whereas the model related to the continuous casting is multi-objective multi-constrained problem and the problem related to the die casting is a single objective problem. The mathematical models which are considered in the present work were previously attempted by genetic algorithm and simulated annealing algorithms. However, attempt is made in the present work to minimize the computational efforts using the TLBO algorithm. Considerable improvements in results are obtained in all the cases and it is believed that a global optimum solution is achieved in the case of die casting process.     

Online surrogate multiobjective optimization algorithm for contaminated groundwater remediation designs

This paper proposes an online surrogate model-assisted multiobjective optimization framework to identify optimal remediation strategies for groundwater contaminated with dense non-aqueous phase liquids. The optimization involves three objectives: minimizing the remediation cost and duration and maximizing the contamination removal rate. The proposed framework adopts a multiobjective feasibility-enhanced particle swarm optimization algorithm to solve the optimization model and uses an online surrogate model as a substitute for the time-consuming multiphase flow model for calculating contamination removal rates during the optimization process. The resulting approach allows decision makers to find a balance among the remediation cost, remediation duration and contamination removal rate for remediating contaminated groundwater. The new algorithm is compared with the nondominated sorting genetic algorithm II, which is an extensively applied and well-known algorithm. The results show that the Pareto solutions obtained by the new algorithm have greater diversity and stability than those obtained by the nondominated sorting genetic algorithm II, indicating that the new algorithm is more applicable than the nondominated sorting genetic algorithm II for optimizing remediation strategies for contaminated groundwater. Additionally, the surrogate model and Pareto optimal set obtained by the proposed framework are compared with those of the offline surrogate model-assisted multiobjective optimization framework. The results indicate that the surrogate model accuracy and Pareto front achieved by the proposed framework outperform those of the offline surrogate model-assisted optimization framework. Thus, we conclude that the proposed framework can effectively enhance the surrogate model accuracy and further extend the comprehensive performance of Pareto solutions.     

An instantaneous replenishment model under the effect of a sampling policy for defective items

We propose to study a EOQ-type inventory model with unreliable supply, with each order containing a random proportion of defective items. Every time an order is received, an acceptance sampling plan is applied to the lot, according to which only a sample is inspected instead of the whole lot. If the sample conforms to the standards, i.e. if the number of imperfect items is below an “acceptance number”, no further screening is performed. Otherwise, the lot is subject to 100% screening. We formulate an integer non-linear mathematical program that integrates inventory and quality decisions into a unified profit model, to jointly determine the optimal lot size and optimal sampling plan, characterized by a sample size, and an acceptance number. The optimal decisions are determined in a way to achieve a certain average outgoing quality limit (AOQL), which is the highest proportion of defective items in the outgoing material sold to customers. We provide a counter-example demonstrating that the expected profit function, objective of the mathematical program, is not jointly concave in the lot and sample size. However, we show that for a given sampling plan, the expected profit function is concave in the lot size. A solution procedure is presented to compute the optimal solution. Numerical analysis is provided to gain managerial insights by analyzing the impact of changing various model parameters on the optimal solution. We also show numerically that the optimal profit determined using this model is significantly higher when compared to the optimal profit obtained using Salameh and Jaber (2000)’s [1] model, indicating much higher profits when acceptance sampling is used.     

An efficient numerical method to solve 2-D interval bi-modular problems via orthogonal polynomial expansion

An interval analysis of uncertain bi-modular problems is presented by addressing the efficiency of deterministic solution and reduction of computational cost on the non-linear FE iteration. Firstly, the singularity of 2-D conventional bi-modular constitutive matrix is pointed out via a concise mathematical illustration, and is removed via a complement of shear modulus consistent with the coaxial condition. A new FE model with a full rank constitutive matrix is developed to solve deterministic bi-modular problems, which is well performed in the numerical tests, particularly in term of convergence. Secondly, an orthogonal polynomial expansion based surrogate is constructed to alleviate the heavy computational burden caused by repeated non-linear FE solution in the optimization process for bounds estimation. Numerical examples are given to illustrate the accuracy and efficiency of proposed approach, and a good accordance can be observed between the results obtained by the proposed approach and reference solutions.