A hybrid algorithm to optimize RBF network architecture and parameters for nonlinear time series prediction

This paper proposes a novel hybrid algorithm for automatic selection of the proper input variables, the number of hidden nodes of the radial basis function (RBF) network, and optimizing network parameters (weights, centers and widths) simultaneously. In the proposed algorithm, the inputs and the number of hidden nodes of the RBF network are represented by binary-coded strings and evolved by a genetic algorithm (GA). Simultaneously, for each chromosome with fixed inputs and number of hidden nodes, the corresponding parameters of the network are real-coded and optimized by a gradient-based fast-converging parameter estimation method. Performance of the presented hybrid approach is evaluated by several benchmark time series modeling and prediction problems. Experimental results show that the proposed approach produces parsimonious RBF networks, and obtains better modeling accuracy than some other algorithms.     

Improving Prediction of Neural Networks: A Study of Two Financial Prediction Tasks

Neural networks are excellent mapping tools for complex financial data. Their mapping capabilities however do not always result in good generalizability for financial prediction models. Increasing the number of nodes and hidden layers in a neural network model produces better mapping of the data since the number of parameters available to the model increases. This is detrimental to generalizability of the model since the model memorizes idiosyncratic patterns in the data. A neural network model can be expected to be more generalizable if the model architecture is made less complex by using fewer input nodes. In this study we simplify the neural network by eliminating input nodes that have the least contribution to the prediction of a desired outcome. We also provide a theoretical relationship of the sensitivity of output variables to the input variables under certain conditions. This research initiates an effort in identifying methods that would improve the generalizability of neural networks in financial prediction tasks by using mergers and bankruptcy models. The result indicates that incorporating more variables that appear relevant in a model does not necessarily improve prediction performance.     

采用组合参数的神经网络结构损伤检测分析研究

提出由结构前几阶固有频率变化率、频率变化比值和动柔度置信因子构成的组合参数作为神经网络的输入向量的方法进行结构损伤检测,全面分析了不同参数作为神经网络输入向量的损伤效果,利用数值模拟对悬臂梁、桁架结构进行分析,采用不同的输入参数进行比较·　分析结果表明,采用组合参数训练的神经网络,对结构损伤位置和程度识别较采用单一参数具有更好的识别效果·     

Backfitting neural networks

Summary  Regression and classification problems can be viewed as special cases of the problem of function estimation. It is rather well known that a two-layer perceptron with sigmoidal transformation functions can approximate any continuous function on the compact subsets ofRP if there are sufficient number of hidden nodes. In this paper, we present an algorithm for fitting perceptron models, which is quite different from the usual backpropagation or Levenberg-Marquardt algorithm. This new algorithm based on backfitting ensures a better convergence than backpropagation. We have also used resampling techniques to select an ideal number of hidden nodes automatically using the training data itself. This resampling technique helps to avoid the problem of overfitting that one faces for the usual perceptron learning algorithms without any model selection scheme. Case studies and simulation results are presented to illustrate the performance of this proposed algorithm.     

Neural networks for a class of bi-level variational inequalities

In this paper, a class of bi-level variational inequalities for describing some practical equilibrium problems, which especially arise from engineering, management and economics, is presented, and a neural network approach for solving the bi-level variational inequalities is proposed. The energy function and neural dynamics of the proposed neural network are defined in this paper, and then the existence of the solution and the asymptotic stability of the neural network are shown. The simulation algorithm is presented and the performance of the proposed neural network approach is demonstrated by some numerical examples.     

Selective responses by entropy minimization

In this paper, we propose a method of entropy minimization for increasing selectivity and obtaining simple network architectures. An entropy function is defined with respect to the state of hidden units. By minimizing this entropy, the selectivity of hidden units can significantly be increased. Since a unit tends to respond to specific input patterns, the meaning or the function of the hidden units can easily be understood. In addition, we have observed that by minimizing the entropy, some units are forced to be inactive, responding to no input patterns. Thus, these inactive units can be deleted, and we can construct smaller network architectures. We applied the entropy method to standard and recurrent back-propagation. Experimental results confirmed that the number of units selectively responding to a specific pattern increased gradually, while units with low selectivity responding to multiple patterns decreased as entropy decreased. In addition, the number of units responding to no input patterns increased in proportion to the decrease of the entropy. These results show that the entropy minimization method can be used to improve the selectivity, and therefore, the interpretability of the network's behaviors. Then, the method can be used to suppress unnecessary units and to produce simple internal representation or simple network architectures.     

Identification of MIMO systems using radial basis function networks with hybrid learning algorithm

When a radial basis function network (RBFN) is used for identification of a nonlinear multi-input multi-output (MIMO) system, the number of hidden layer nodes, the initial parameters of the kernel, and the initial weights of the network must be determined first. For this purpose, a systematic way that integrates the support vector regression (SVR) and the least squares regression (LSR) is proposed to construct the initial structure of the RBFN. The first step of the proposed method is to determine the number of hidden layer nodes and the initial parameters of the kernel by the SVR method. Then the weights of the RBFN are determined by solving a simple minimization problem based on the concept of LSR. After initialization, an annealing robust learning algorithm (ARLA) is then applied to train the RBFN. With the proposed initialization approach, one can find that the designed RBFN has few hidden layer nodes while maintaining good performance. To show the feasibility and superiority of the annealing robust radial basis function networks (ARRBFNs) for identification of MIMO systems, several illustrative examples are included.     

基于散乱阈值结点的周期神经网络的逼近阶

§1Introduction Inrecentyearstherehasbeengrowinginterestintheproblemofneuralnetworkand relatedapproximation,manyimportantresultsareobtained.Becauseofitsabilityof parallelcomputationinlargescaleandofperfectself-adaptingandapproximation,the neuralnetworkhasbeenwidelyapplied.Theapproximationabilityoftheneuralnetwork dependsonitstopologicalstructure.LetRsbeans-dimensionalEuclidSpaceand(x)isa realfunctiondefinedonRs.When(x)isanexcitationfunctionandx∈Rsisaninput vector,thesimpleneuralnetwork…     

Neuro-fuzzy technique for navigation of multiple mobile robots

In this paper, navigation techniques for several mobile robots are investigated in a totally unknown environment. In the beginning, Fuzzy logic controllers (FLC) using different membership functions are developed and used to navigate mobile robots. First a fuzzy controller has been used with four types of input members, two types of output members and three parameters each. Next two types of fuzzy controllers have been developed having same input members and output members with five parameters each. Each robot has an array of sensors for measuring the distances of obstacles around it and an image sensor for detecting the bearing of the target. It is found that the FLC having Gaussian membership function is best suitable for navigation of multiple mobile robots. Then a hybrid neuro-fuzzy technique has been designed for the same problem. The neuro-fuzzy technique being used here comprises a neural network, which is acting as a pre processor for a fuzzy controller. The neural network considered for neuro-fuzzy technique is a multi-layer perceptron, with two hidden layers. These techniques have been demonstrated in simulation mode, which depicts that the robots are able to avoid obstacles and reach the targets efficiently. Amongst the techniques developed neuro-fuzzy technique is found to be most efficient for mobile robots navigation. Experimental verifications have been done with the simulation results to prove the authenticity of the developed neuro-fuzzy technique.     

On the near optimality of the stochastic approximation of smooth functions by neural networks

We consider the problem of approximating the Sobolev class of functions by neural networks with a single hidden layer, establishing both upper and lower bounds. The upper bound uses a probabilistic approach, based on the Radon and wavelet transforms, and yields similar rates to those derived recently under more restrictive conditions on the activation function. Moreover, the construction using the Radon and wavelet transforms seems very natural to the problem. Additionally, geometrical arguments are used to establish lower bounds for two types of commonly used activation functions. The results demonstrate the tightness of the bounds, up to a factor logarithmic in the number of nodes of the neural network. This revised version was published online in June 2006 with corrections to the Cover Date.     

Rare Event Probability Estimation in the Presence of Epistemic Uncertainty on Input Probability Distribution Parameters

The accurate estimation of rare event probabilities is a crucial problem in engineering to characterize the reliability of complex systems. Several methods such as Importance Sampling or Importance Splitting have been proposed to perform the estimation of such events more accurately (i.e., with a lower variance) than crude Monte Carlo method. However, these methods assume that the probability distributions of the input variables are exactly defined (e.g., mean and covariance matrix perfectly known if the input variables are defined through Gaussian laws) and are not able to determine the impact of a change in the input distribution parameters on the probability of interest. The problem considered in this paper is the propagation of the input distribution parameter uncertainty defined by intervals to the rare event probability. This problem induces intricate optimization and numerous probability estimations in order to determine the upper and lower bounds of the probability estimate. The calculation of these bounds is often numerically intractable for rare event probability (say 10^?5), due to the high computational cost required. A new methodology is proposed to solve this problem with a reduced simulation budget, using the adaptive Importance Sampling. To this end, a method for estimating the Importance Sampling optimal auxiliary distribution is proposed, based on preceding Importance Sampling estimations. Furthermore, a Kriging-based adaptive Importance Sampling is used in order to minimize the number of evaluations of the computationally expensive simulation code. To determine the bounds of the probability estimate, an evolutionary algorithm is employed. This algorithm has been selected to deal with noisy problems since the Importance Sampling probability estimate is a random variable. The efficiency of the proposed approach, in terms of accuracy of the found results and computational cost, is assessed on academic and engineering test cases.     

The numerical solution of linear ordinary differential equations by feedforward neural networks

It is demonstrated, through theory and examples, how it is possible to construct directly and noniteratively a feedforward neural network to approximate arbitrary linear ordinary differential equations. The method, using the hard limit transfer function, is linear in storage and processing time, and the L₂ norm of the network approximation error decreases quadratically with the increasing number of hidden layer neurons. The construction requires imposing certain constraints on the values of the input, bias, and output weights, and the attribution of certain roles to each of these parameters.

All results presented used the hard limit transfer function. However, the noniterative approach should also be applicable to the use of hyperbolic tangents, sigmoids, and radial basis functions.     

单隐层神经网络与最佳多项式逼近

研究单隐层神经网络逼近问题．以最佳多项式逼近为度量,用构造性方法估计单隐层神经网络逼近连续函数的速度．所获结果表明:对定义在紧集上的任何连续函数,均可以构造一个单隐层神经网络逼近该函数,并且其逼近速度不超过该函数的最佳多项式逼近的二倍．     

快速自底向上构造神经网络的方法

介绍了一种构造神经网络的新方法 .常规的瀑流关联 (Cascade-Correlation)算法起始于最小网络(没有隐含神经元 ) ,然后逐一地往网络里增加新隐含神经元并训练 ,结束于期望性能的获得 .我们提出一种与构造算法 (Constructive Algorithm)相关的快速算法 ,这种算法从适当的初始网络结构开始 ,然后不断地往网络里增加新的神经元和相关权值 ,直到满意的结果获得为止 .实验证明 ,这种快速方法与以往的常规瀑流关联方法相比 ,有几方面优点 :更好的分类性能 ,更小的网络结构和更快的学习速度 .     

A new dynamic programming procedure for three-staged cutting patterns

Three-staged patterns are often used to solve the 2D cutting stock problem of rectangular items. They can be divided into items in three stages: Vertical cuts divide the plate into segments; then horizontal cuts divide the segments into strips, and finally vertical cuts divide the strips into items. An algorithm for unconstrained three-staged patterns is presented, where a set of rectangular item types are packed into the plate so as to maximize the pattern value, and there is no constraint on the frequencies of each item type. It can be used jointly with the linear programming approach to solve the cutting stock problem. The algorithm solves three large knapsack problems to obtain the optimal pattern: One for the item layout on the widest strip, one for the strip layout on the longest segment, and the third for the segment layout on the plate. The computational results indicate that the algorithm is efficient.     

Wind energy prediction using a two-hidden layer neural network

The power generated by wind turbines changes rapidly because of the continuous fluctuation of wind speed and air density. As a consequence, it can be important to predict the energy production, starting from some basic input parameters. The aim of this paper is to show that a two-hidden layer neural network can represent a useful tool to carefully predict the wind energy output. By using proper experimental data (collected from three wind farm in Southern Italy) in combination with a back propagation learning algorithm, a suitable neural architecture is found, characterized by the hyperbolic tangent transfer function in the first hidden layer and the logarithmic sigmoid transfer function in the second hidden layer. Simulation results are reported, showing that the estimated wind energy values (predicted by the proposed network) are in good agreement with the experimental measured values.     

Performance evaluation of artificial neural network approaches in forecasting reservoir inflow

This study investigates the potential of Time Lag Recurrent Neural Networks (TLRN) for modeling the daily inflow into Eleviyan reservoir, Iran. TLRN are extended with short term memory structures that have local recurrent connections, thus making them an appropriate model for processing temporal (time-varying) information. For this study, the daily inflow into Eleviyan reservoir between years 2004–2007 was considered. To compare the performance of TLRN, a back propagation neural network was used. The TLRN model with gamma memory structure, eight input layer nodes, two hidden layer and one output layer (8-2-1) was found performing best out of three different models used in forecasting daily inflow. A comparison of results with back propagation neural network suggest that neither TLRN nor back propagation approaches were good in forecasting high inflow but, both approaches perform well when used to forecast low inflow values. However, statistical test suggests that both TLRN and back propagation neural network models were able to reproduce similar basic statistics as that of the actual data.     

A hidden Markov regime-switching model for option valuation

We investigate two approaches, namely, the Esscher transform and the extended Girsanov’s principle, for option valuation in a discrete-time hidden Markov regime-switching Gaussian model. The model’s parameters including the interest rate, the appreciation rate and the volatility of a risky asset are governed by a discrete-time, finite-state, hidden Markov chain whose states represent the hidden states of an economy. We give a recursive filter for the hidden Markov chain and estimates of model parameters using a filter-based EM algorithm. We also derive predictors for the hidden Markov chain and some related quantities. These quantities are used to estimate the price of a standard European call option. Numerical examples based on real financial data are provided to illustrate the implementation of the proposed method.     

具有随机隐单元的渐增前馈神经网络的$L^2(R^d)$逼近能力

This paper studies approximation capability to L2(Rd) functions of incremental constructive feedforward neural networks(FNN) with random hidden units.Two kinds of therelayered feedforward neural networks are considered:radial basis function(RBF) neural networks and translation and dilation invariant(TDI) neural networks.In comparison with conventional methods that existence approach is mainly used in approximation theories for neural networks,we follow a constructive approach to prove that one may simply randomly choose parameters of hidden units and then adjust the weights between the hidden units and the output unit to make the neural network approximate any function in L2(Rd) to any accuracy.Our result shows given any non-zero activation function g :R+→R and g(x Rd) ∈ L2(Rd) for RBF hidden units,or any non-zero activation function g(x) ∈ L2(Rd) for TDI hidden units,the incremental network function fn with randomly generated hidden units converges to any target function in L2(Rd) with probability one as the number of hidden units n→∞,if one only properly adjusts the weights between the hidden units and output unit.     

Multi-step ahead nonlinear identification of Lorenz’s chaotic system using radial basis neural network with learning by clustering and particle swarm optimization

An important problem in engineering is the identification of nonlinear systems, among them radial basis function neural networks (RBF-NN) using Gaussian activation functions models, which have received particular attention due to their potential to approximate nonlinear behavior. Several design methods have been proposed for choosing the centers and spread of Gaussian functions and training the RBF-NN. The selection of RBF-NN parameters such as centers, spreads, and weights can be understood as a system identification problem. This paper presents a hybrid training approach based on clustering methods (k-means and c-means) to tune the centers of Gaussian functions used in the hidden layer of RBF-NNs. This design also uses particle swarm optimization (PSO) for centers (local clustering search method) and spread tuning, and the Penrose–Moore pseudoinverse for the adjustment of RBF-NN weight outputs. Simulations involving this RBF-NN design to identify Lorenz’s chaotic system indicate that the performance of the proposed method is superior to that of the conventional RBF-NN trained for k-means and the Penrose–Moore pseudoinverse for multi-step ahead forecasting.