Optimization of manufacturing systems using a neural network metamodel with a new training approach

In this study, two manufacturing systems, a kanban-controlled system and a multi-stage, multi-server production line in a diamond tool production system, are optimized utilizing neural network metamodels (tst_NNM) trained via tabu search (TS) which was developed previously by the authors. The most widely used training algorithm for neural networks has been back propagation which is based on a gradient technique that requires significant computational effort. To deal with the major shortcomings of back propagation (BP) such as the tendency to converge to a local optimal and a slow convergence rate, the TS metaheuristic method is used for the training of artificial neural networks to improve the performance of the metamodelling approach. The metamodels are analysed based on their ability to predict simulation results versus traditional neural network metamodels that have been trained by BP algorithm (bp_NNM). Computational results show that tst_NNM is superior to bp_NNM for both of the manufacturing systems.     

Image compression with a dynamic autoassociative neural network

Image compression using neural networks has been attempted with some promise. Among the architectures, feedforward backpropagation networks (FFBPN) have been used in several attempts. Although it is demonstrated that using the mean quadratic error function is equivalent to applying the Karhunen-Loeve transformation method, promise still arises from directed learning possibilities, generalization abilities and performance of the network once trained. In this paper we propose an architecture and an improved training method to attempt to solve some of the shortcomings of traditional data compression systems based on feedforward neural networks trained with backpropagation—the dynamic autoassociation neural network (DANN).The successful application of neural networks to any task requires proper training of the network. In this research, this issue is taken as the main consideration in the design of DANN. We emphasize the convergence of the learning process proposed by DANN. This process provides an escape mechanism, by adding neurons in a random state, to avoid the local minima trapping seen in traditional PFBPN. Also, DANN's training algorithm constrains the error for every pattern to an allowed interval to balance the training for every pattern, thus improving the performance rates in recognition and generalization. The addition of these two mechanisms to DANN's training algorithm has the result of improving the final quality of the images processed by DANN.The results of several tasks presented to DANN-based compression are compared and contrasted with the performance of an FFBPN-based system applied to the same task. These results indicate that DANN is superior to FFBPN when applied to image compression.     

The use of random weights for the training of multilayer networks of neurons with Heaviside characteristics

Artificial neural networks have, in recent years, been very successfully applied in a wide range of areas. A major reason for this success has been the existence of a training algorithm called backpropagation. This algorithm relies upon the neural units in a network having input/output characteristics that are continuously differentiable. Such units are significantly less easy to implement in silicon than are neural units with Heaviside (step-function) characteristics. In this paper, we show how a training algorithm similar to backpropagation can be developed for 2-layer networks of Heaviside units by treating the network weights (i.e., interconnection strengths) as random variables. This is then used as a basis for the development of a training algorithm for networks with any number of layers by drawing upon the idea of internal representations. Some examples are given to illustrate the performance of these learning algorithms.     

Convergence of Online Gradient Method with Penalty for BP Neural Networks

Online gradient method has been widely used as a learning algorithm for training feedforward neural networks. Penalty is often introduced into the training procedure to improve the generalization performance and to decrease the magnitude of network weights. In this paper, some weight boundedness and deterministic con- vergence theorems are proved for the online gradient method with penalty for BP neural network with a hidden layer, assuming that the training samples are supplied with the network in a fixed order within each epoch. The monotonicity of the error function with penalty is also guaranteed in the training iteration. Simulation results for a 3-bits parity problem are presented to support our theoretical results.     

Neural network as a simulation metamodel in economic analysis of risky projects

An artificial neural network (ANN) model for economic analysis of risky projects is presented in this paper. Outputs of conventional simulation models are used as neural network training inputs. The neural network model is then used to predict the potential returns from an investment project having stochastic parameters. The nondeterministic aspects of the project include the initial investment, the magnitude of the rate of return, and the investment period. Backpropagation method is used in the neural network modeling. Sigmoid and hyperbolic tangent functions are used in the learning aspect of the system. Analysis of the outputs of the neural network model indicates that more predictive capability can be achieved by coupling conventional simulation with neural network approaches. The trained network was able to predict simulation output based on the input values with very good accuracy for conditions not in its training set. This allowed an analysis of the future performance of the investment project without having to run additional expensive and time-consuming simulation experiments.     

Approximating Unknown Mappings: An Experimental Evaluation

Different methodologies have been introduced in recent years with the aim of approximating unknown functions. Basically, these methodologies are general frameworks for representing non-linear mappings from several input variables to several output variables. Research into this problem occurs in applied mathematics (multivariate function approximation), statistics (nonparametric multiple regression) and computer science (neural networks). However, since these methodologies have been proposed in different fields, most of the previous papers treat them in isolation, ignoring contributions in the other areas. In this paper we consider five well known approaches for function approximation. Specifically we target polynomial approximation, general additive models (Gam), local regression (Loess), multivariate additive regression splines (Mars) and artificial neural networks (Ann).Neural networks can be viewed as models of real systems, built by tuning parameters known as weights. In training the net, the problem is to find the weights that optimize its performance (i.e. to minimize the error over the training set). Although the most popular method for Ann training is back propagation, other optimization methods based on metaheuristics have recently been adapted to this problem, outperforming classical approaches. In this paper we propose a short term memory tabu search method, coupled with path relinking and BFGS (a gradient-based local NLP solver) to provide high quality solutions to this problem. The experimentation with 15 functions previously reported shows that a feed-forward neural network with one hidden layer, trained with our procedure, can compete with the best-known approximating methods. The experimental results also show the effectiveness of a new mechanism to avoid overfitting in neural network training.     

Pattern memory analysis based on stability theory of cellular neural networks

In this paper, several sufficient conditions are obtained to guarantee that the n-dimensional cellular neural network can have even (?2ⁿ) memory patterns. In addition, the estimations of attractive domain of such stable memory patterns are obtained. These conditions, which can be directly derived from the parameters of the neural networks, are easily verified. A new design procedure for cellular neural networks is developed based on stability theory (rather than the well-known perceptron training algorithm), and the convergence in the new design procedure is guaranteed by the obtained local stability theorems. Finally, the validity and performance of the obtained results are illustrated by two examples.     

Simulation of nonlinear fractional dynamics arising in the modeling of cognitive decision making using a new fractional neural network

By the rapid growth of available data, providing data-driven solutions for nonlinear (fractional) dynamical systems becomes more important than before. In this paper, a new fractional neural network model that uses fractional order of Jacobi functions as its activation functions for one of the hidden layers is proposed to approximate the solution of fractional differential equations and fractional partial differential equations arising from mathematical modeling of cognitive-decision-making processes and several other scientific subjects. This neural network uses roots of Jacobi polynomials as the training dataset, and the Levenberg-Marquardt algorithm is chosen as the optimizer. The linear and nonlinear fractional dynamics are considered as test examples showing the effectiveness and applicability of the proposed neural network. The numerical results are compared with the obtained results of some other networks and numerical approaches such as meshless methods. Numerical experiments are presented confirming that the proposed model is accurate, fast, and feasible.     

变尺度快速二阶非线性优化算法在神经网络辨识中的应用

神经网络是非线性系统建模与辨识的重要方法 ,反向传播 (BP)算法常常用在神经网络的权值训练中 ,但是 BP算法的收敛速度慢 .本文提出一种变尺度二阶快速优化方法 ,在这种方法中用二阶插值法来优化搜索学习速率 ,然后将这一方法应用于神经网络的辨识中 ,仿真研究表明新算法有更快的收敛速度和更好的收敛精度 .     

用在线梯度法训练积单元神经网络的收敛性分析

<正>1引言仅由加和单元构成的传统前向神经网络已经广泛应用于模式识别及函数逼近等领域.但在处理比较复杂的问题时,这种网络往往需要补充大量的隐节点,这样就不可避免地增     

An improved determination approach to the structure and parameters of dynamic structure-based neural networks

Dynamic structure-based neural networks are being extensively applied in many fields of science and engineering. A novel dynamic structure-based neural network determination approach using orthogonal genetic algorithm with quantization is proposed in this paper. Both the parameter (the threshold of each neuron and the weight between neurons) and the transfer function (the transfer function of each layer and the network training function) of the dynamic structure-based neural network are optimized using this approach. In order to satisfy the dynamic transform of the neural network structure, the population adjustment operation was introduced into orthogonal genetic algorithm with quantization for dynamic modification of the population’s dimensionality. A mathematical example was applied to evaluate this approach. The experiment results suggested that this approach is feasible, correct and valid.     

On the Generalization Problem

The generalization problem considered in this paper assumes that a limited amount of input and output data from a system is available, and that from this information an estimate of the output produced by another input is required. The ideas arose in the study of neural networks, but apply equally to any approximation approach. The main result is that the type of neural network to be used for generalization should be determined by the prior knowledge about the nature of the output from the system. Without such information, either of two networks matching the training data is equally likely to be the better at estimating the output generated by the same system at a new input. Therefore, the search for an optimum generalization network for use on all problems is inappropriate.For both (0, 1) and accurate real outputs, it is shown that simple approximations exist that fit the data, so these will be equally likely to generalize better than more sophisticated networks, unless prior knowledge is available that excludes them. For noisy real outputs, it is shown that the standard least squares approach forces the neural network to approximate an incorrect process; an alternative approach is outlined, which again is much easier to learn and use.     

Path relinking and GRG for artificial neural networks

Artificial neural networks (ANN) have been widely used for both classification and prediction. This paper is focused on the prediction problem in which an unknown function is approximated. ANNs can be viewed as models of real systems, built by tuning parameters known as weights. In training the net, the problem is to find the weights that optimize its performance (i.e., to minimize the error over the training set). Although the most popular method for training these networks is back propagation, other optimization methods such as tabu search or scatter search have been successfully applied to solve this problem. In this paper we propose a path relinking implementation to solve the neural network training problem. Our method uses GRG, a gradient-based local NLP solver, as an improvement phase, while previous approaches used simpler local optimizers. The experimentation shows that the proposed procedure can compete with the best-known algorithms in terms of solution quality, consuming a reasonable computational effort.     

Online Gradient Methods with a Punishing Term for Neural Networks

1 IntroductionOnline gradient methods (OGM, for short) are widely used for training neuraJ networks (cf.Il,2,3,4]). Its iterative convergence for linear models is proved in e.g. I5,6,71. A nonlinearn1odel is considered in [8]. During the iterative training procedure, sometimes (see the nextsection of this paper)the weight of the network may become very laxge, causing difficultiesin the implementation of the network by electronic circuits. A revised error fUnction ispresented in [gl to prev…     

Linear unlearning for cross-validation

The leave-one-out cross-validation scheme for generalization assessment of neural network models is computationally expensive due to replicated training sessions. In this paper we suggest linear unlearning of examples as an approach to approximative cross-validation. Further, we discuss the possibility of exploiting the ensemble of networks offered by leave-one-out for performing ensemble predictions. We show that the generalization performance of the equally weighted ensemble predictor is identical to that of the network trained on the whole training set.Numerical experiments on the sunspot time series prediction benchmark demonstrate the potential of the linear unlearning technique.     

Classification by ensembles of neural networks

We introduce a new procedure for training of artificial neural networks by using the approximation of an objective function by arithmetic mean of an ensemble of selected randomly generated neural networks, and apply this procedure to the classification (or pattern recognition) problem. This approach differs from the standard one based on the optimization theory. In particular, any neural network from the mentioned ensemble may not be an approximation of the objective function.     

Non-convergence of stochastic gradient descent in the training of deep neural networks

Deep neural networks have successfully been trained in various application areas with stochastic gradient descent. However, there exists no rigorous mathematical explanation why this works so well. The training of neural networks with stochastic gradient descent has four different discretization parameters: (i) the network architecture; (ii) the amount of training data; (iii) the number of gradient steps; and (iv) the number of randomly initialized gradient trajectories. While it can be shown that the approximation error converges to zero if all four parameters are sent to infinity in the right order, we demonstrate in this paper that stochastic gradient descent fails to converge for ReLU networks if their depth is much larger than their width and the number of random initializations does not increase to infinity fast enough.     

A note on knowledge discovery using neural networks and its application to credit card screening

We address an important issue in knowledge discovery using neural networks that has been left out in a recent article “Knowledge discovery using a neural network simultaneous optimization algorithm on a real world classification problem” by Sexton et al. [R.S. Sexton, S. McMurtrey, D.J. Cleavenger, Knowledge discovery using a neural network simultaneous optimization algorithm on a real world classification problem, European Journal of Operational Research 168 (2006) 1009–1018]. This important issue is the generation of comprehensible rule sets from trained neural networks. In this note, we present our neural network rule extraction algorithm that is very effective in discovering knowledge embedded in a neural network. This algorithm is particularly appropriate in applications where comprehensibility as well as accuracy are required. For the same data sets used by Sexton et al. our algorithm produces accurate rule sets that are concise and comprehensible, and hence helps validate the claim that neural networks could be viable alternatives to other data mining tools for knowledge discovery.