Neural networks and seasonality: Some technical considerations

Debate continues regarding the capacity of feedforward neural networks (NNs) to deal with seasonality without pre-processing. The purpose of this paper is to provide, with examples, some theoretical perspective for the debate. In the first instance it considers possible specification errors arising through use of autoregressive forms. Secondly, it examines seasonal variation in the context of the so-called ‘universal approximation’ capabilities of NNs, finding that a short (bounded) sinusoidal series is easy for the network but that a series with many turning points becomes progressively more difficult. This follows from results contained in one of the seminal papers on NN approximation. It is confirmed in examples which also show that, to model seasonality with NNs, very large numbers of hidden nodes may be required.     

具有随机隐单元的渐增前馈神经网络的$L^2(R^d)$逼近能力

This paper studies approximation capability to L2(Rd) functions of incremental constructive feedforward neural networks(FNN) with random hidden units.Two kinds of therelayered feedforward neural networks are considered:radial basis function(RBF) neural networks and translation and dilation invariant(TDI) neural networks.In comparison with conventional methods that existence approach is mainly used in approximation theories for neural networks,we follow a constructive approach to prove that one may simply randomly choose parameters of hidden units and then adjust the weights between the hidden units and the output unit to make the neural network approximate any function in L2(Rd) to any accuracy.Our result shows given any non-zero activation function g :R+→R and g(x Rd) ∈ L2(Rd) for RBF hidden units,or any non-zero activation function g(x) ∈ L2(Rd) for TDI hidden units,the incremental network function fn with randomly generated hidden units converges to any target function in L2(Rd) with probability one as the number of hidden units n→∞,if one only properly adjusts the weights between the hidden units and output unit.     

An online gradient method with momentum for two-layer feedforward neural networks

An online gradient method with momentum for two-layer feedforward neural networks is considered. The momentum coefficient is chosen in an adaptive manner to accelerate and stabilize the learning procedure of the network weights. Corresponding convergence results are proved, that is, the weak convergence result is proved under the uniformly boundedness assumption of the activation function and its derivatives, moreover, if the number of elements of the stationary point set for the error function is finite, then strong convergence result holds.     

Neural networks,linear functions and neglected non-linearity

The multiplicity of approximation theorems for Neural Networks do not relate to approximation of linear functions per se. The problem for the network is to construct a linear function by superpositions of non-linear activation functions such as the sigmoid function. This issue is important for applications of NNs in statistical tests for neglected nonlinearity, where it is common practice to include a linear function through skip-layer connections. Our theoretical analysis and evidence point in a similar direction, suggesting that the network can in fact provide linear approximations without additional assistance. Our paper suggests that skip-layer connections are unnecessary, and if employed could lead to misleading results.Received: August 2002, Revised: March 2003, AMS Classification: 82c32The authors are grateful to Prof. Mick Silver and to GFK Marketing for help with the provision of data.     

Neural networks and finite-order approximations

Email: Curry{at}Cardiff.ac.uk This paper investigates the approximation properties of standardfeedforward neural networks (NNs) through the application ofmultivanate Thylor-series expansions. The capacity to approximatearbitrary functional forms is central to the NN philosophy,but is usually proved by allowing the number of hidden nodesto increase to infinity. The Thylor-series approach does notdepend on such limiting cases, lie paper shows how the seriesapproximation depends on individual network weights. The roleof the bias term is taken as an example. We are also able tocompare the sigmoid and hyperbolic-tangent activation functions,with particular emphasis on their impact on the bias term. Thepaper concludes by discussing the potential importance of ourresults for NN modelling: of particular importance is the trainingprocess.     

Parameter redundancy in neural networks: an application of Chebyshev polynomials

This paper deals with feedforward neural networks containing a single hidden layer and with sigmoid/logistic activation function. Training such a network is equivalent to implementing nonlinear regression using a flexible functional form, but the functional form in question is not easy to deal with. The Chebyshev polynomials are suggested as a way forward, providing an approximation to the network which is superior to Taylor series expansions. Application of these approximations suggests that the network is liable to a ‘naturally occurring’ parameter redundancy, which has implications for the training process as well as certain statistical implications. On the other hand, parameter redundancy does not appear to damage the fundamental property of universal approximation.      

Comparing performance of feedforward neural nets and K-means for cluster-based market segmentation

We compare the performance of a specifically designed feedforward artificial neural network with one layer of hidden units to the K-means clustering technique in solving the problem of cluster-based market segmentation. The data set analyzed consists of usages of brands (product category: household cleaners) in different usage situations. The proposed feedforward neural network model results in a two segment solution that is confirmed by appropriate tests. On the other hand, the K-means algorithm fails in discovering any somewhat stronger cluster structure. Classification of respondents on the basis of external criteria is better for the neural network solution. We also demonstrate the managerial interpretability of the network results.     

The numerical solution of linear ordinary differential equations by feedforward neural networks

It is demonstrated, through theory and examples, how it is possible to construct directly and noniteratively a feedforward neural network to approximate arbitrary linear ordinary differential equations. The method, using the hard limit transfer function, is linear in storage and processing time, and the L₂ norm of the network approximation error decreases quadratically with the increasing number of hidden layer neurons. The construction requires imposing certain constraints on the values of the input, bias, and output weights, and the attribution of certain roles to each of these parameters.

All results presented used the hard limit transfer function. However, the noniterative approach should also be applicable to the use of hyperbolic tangents, sigmoids, and radial basis functions.     

Modified optimization layer by layer algorithm for learning multilayer perceptrons

1. IntroductionThe feedforward Multilayer Perceptron (MLP) is one of the most widely used artificial neural networks among other network models. Its field of application includes patternrecognition, identification and control of dynamic systems, system modeling and nonlinearprediction of time series, etc. [1--41 founded on its nonlinear function approximation capability. Research of this type of networks has been stimulated since the discovery andpopularization of the Backpropagation learnin…     

Convergence of Online Gradient Method with Penalty for BP Neural Networks

Online gradient method has been widely used as a learning algorithm for training feedforward neural networks. Penalty is often introduced into the training procedure to improve the generalization performance and to decrease the magnitude of network weights. In this paper, some weight boundedness and deterministic con- vergence theorems are proved for the online gradient method with penalty for BP neural network with a hidden layer, assuming that the training samples are supplied with the network in a fixed order within each epoch. The monotonicity of the error function with penalty is also guaranteed in the training iteration. Simulation results for a 3-bits parity problem are presented to support our theoretical results.     

快速自底向上构造神经网络的方法

介绍了一种构造神经网络的新方法 .常规的瀑流关联 (Cascade-Correlation)算法起始于最小网络(没有隐含神经元 ) ,然后逐一地往网络里增加新隐含神经元并训练 ,结束于期望性能的获得 .我们提出一种与构造算法 (Constructive Algorithm)相关的快速算法 ,这种算法从适当的初始网络结构开始 ,然后不断地往网络里增加新的神经元和相关权值 ,直到满意的结果获得为止 .实验证明 ,这种快速方法与以往的常规瀑流关联方法相比 ,有几方面优点 :更好的分类性能 ,更小的网络结构和更快的学习速度 .     

神经网络的加权本质逼近阶

证明了具有单一隐层的神经网络在L_ω~q的逼近,获得了网络逼近的上界估计和下界估计.这一结果揭示了神经网络在加权逼近的意义下,网络的收敛阶与隐层单元个数之间的关系,为神经网络的应用提供了重要的理论基础.     

关于多层前馈神经网络学习能力的一个注记

给出一种通过适当地选择输入层与隐层间的连接权,来减少单隐层前馈型神经网络隐层节点的个数的方法.应用此方法,分析了具有两个隐层节点的标准单隐层网络的学习能力,并对二元和三元XOR问题中的权值的选择问题进行了详细的讨论.     

A cutting-plane algorithm for solving a weighted influence interdiction problem

We consider a two-stage defender-attacker game that takes place on a network, in which the attacker seeks to take control over (or “influence”) as many nodes as possible. The defender acts first in this game by protecting a subset of nodes that cannot be influenced by the attacker. With full knowledge of the defender’s action, the attacker can then influence an initial subset of unprotected nodes. The influence then spreads over a finite number of time stages, where an uninfluenced node becomes influenced at time t if a threshold number of its neighbors are influenced at time t?1. The attacker’s objective is to maximize the weighted number of nodes that are influenced over the time horizon, where the weights depend both on the node and on the time at which that is influenced. This defender-attacker game is especially difficult to optimize, because the attacker’s problem itself is NP-hard, which precludes a standard inner-dualization approach that is common in many interdiction studies. We provide three models for solving the attacker’s problem, and develop a tailored cutting-plane algorithm for solving the defender’s problem. We then demonstrate the computational efficacy of our proposed algorithms on a set of randomly generated instances.     

Models for network evolution

This paper describes mathematical models for network evolution when ties (edges) are directed and the node set is fixed. Each of these models implies a specific type of departure from the standard null binomial model. We provide statistical tests that, in keeping with these models, are sensitive to particular types of departures from the null. Each model (and associated test) discussed follows directly from one or more socio‐cognitive theories about how individuals alter the colleagues with whom they are likely to interact. The models include triad completion models, degree variance models, polarization and balkanization models, the Holland‐Leinhardt models, metric models, and the constructural model. We find that many of these models, in their basic form, tend asymptotically towards an equilibrium distribution centered at the completely connected network (i.e., all individuals are equally likely to interact with all other individuals); a fact that can inhibit the development of satisfactory tests.     

A novel hybrid algorithm combining hunting search with harmony search algorithm for training neural networks

Neural networks (NNs) are one of the most widely used techniques for pattern classification. Owing to the most common back-propagation training algorithm of NN being extremely computationally intensive and it having some drawbacks, such as converging into local minima, many meta-heuristic algorithms have been applied to training of NNs. This paper presents a novel hybrid algorithm which is the integration of Harmony Search (HS) and Hunting Search (HuS) algorithms, called h_HS-HuS, in order to train Feed-Forward Neural Networks (FFNNs) for pattern classification. HS and HuS algorithms are recently proposed meta-heuristic algorithms inspired from the improvisation process of musicians and hunting of animals, respectively. Harmony search builds up the main structure of the hybrid algorithm, and HuS forms the pitch adjustment phase of the HS algorithm. The performance proposed algorithm is compared to conventional and meta-heuristic algorithms. Empirical tests are carried out by training NNs on nine widely used classification benchmark problems. The experimental results show that the proposed hybrid harmony-hunting algorithm is highly capable of training NNs.     

Machine-part cell formation in group technology using a modified ART1 method

Group Technology (GT) is a useful way of increasing the productivity for manufacturing high quality products and improving the flexibility of manufacturing systems. Cell formation (CF) is a key step in GT. It is used in designing good cellular manufacturing systems using the similarities between parts in relation to the machines in their manufacture. It can identify part families and machine groups. Recently, neural networks (NNs) have been widely applied in GT due to their robust and adaptive nature. NNs are very suitable in CF with a wide variety of real applications. Although Dagli and Huggahalli adopted the ART1 network with an application in machine-part CF, there are still several drawbacks to this approach. To address these concerns, we propose a modified ART1 neural learning algorithm. In our modified ART1, the vigilance parameter can be simply estimated by the data so that it is more efficient and reliable than Dagli and Huggahalli’s method for selecting a vigilance value. We then apply the proposed algorithm to machine-part CF in GT. Several examples are presented to illustrate its efficiency. In comparison with Dagli and Huggahalli’s method based on the performance measure proposed by Chandrasekaran and Rajagopalan, our modified ART1 neural learning algorithm provides better results. Overall, the proposed algorithm is vigilance parameter-free and very efficient to use in CF with a wide variety of machine/part matrices.     

Inter-model comparison of CFD and neural network analysis of natural convection heat transfer in a partitioned enclosure

CFD analysis of heat and mass flow due to natural convection in partitioned enclosures has recently been the focus of many CFD researchers. In some cases, it was reported in the literature that different CFD solutions (due to different numerical stability characteristics) were obtained for different mesh quality, time step, and discretization order. The objective of this paper is to investigate the feasibility of using neural networks (NNs) as a means lending support to the authenticity of steady-state CFD solutions for such ill-posed problems through inter-model comparisons. Attention is focused on using NNs trained on a database generated by numerically-stable CFD analysis to predict flow variables for the aforementioned ill-posed cases, thereby giving confidence in steady-state CFD results for these cases. Three types of NNs were evaluated and parametric studies were performed to optimize network designs for best predictions of the flow variables.     

A local linear embedding module for evolutionary computation optimization

A Local Linear Embedding (LLE) module enhances the performance of two Evolutionary Computation (EC) algorithms employed as search tools in global optimization problems. The LLE employs the stochastic sampling of the data space inherent in Evolutionary Computation in order to reconstruct an approximate mapping from the data space back into the parameter space. This allows to map the target data vector directly into the parameter space in order to obtain a rough estimate of the global optimum, which is then added to the EC generation. This process is iterated and considerably improves the EC convergence. Thirteen standard test functions and two real-world optimization problems serve to benchmark the performance of the method. In most of our tests, optimization aided by the LLE mapping outperforms standard implementations of a genetic algorithm and a particle swarm optimization. The number and ranges of functions we tested suggest that the proposed algorithm can be considered as a valid alternative to traditional EC tools in more general applications. The performance improvement in the early stage of the convergence also suggests that this hybrid implementation could be successful as an initial global search to select candidates for subsequent local optimization.     

Corrections to the Central Limit Theorem for Heavy-tailed Probability Densities

Classical Edgeworth expansions provide asymptotic correction terms to the Central Limit Theorem (CLT) up to an order that depends on the number of moments available. In this paper, we provide subsequent correction terms beyond those given by a standard Edgeworth expansion in the general case of regularly varying distributions with diverging moments (beyond the second). The subsequent terms can be expressed in a simple closed form in terms of certain special functions (Dawson’s integral and parabolic cylinder functions), and there are qualitative differences depending on whether the number of moments available is even, odd, or not an integer, and whether the distributions are symmetric or not. If the increments have an even number of moments, then additional logarithmic corrections must also be incorporated in the expansion parameter. An interesting feature of our correction terms for the CLT is that they become dominant outside the central region and blend naturally with known large-deviation asymptotics when these are applied formally to the spatial scales of the CLT.