A scatter-search-based learning algorithm for neural network training

In this article, we propose a new scatter-search-based learning algorithm to train feed-forward neural networks. The algorithm also incorporates elements of tabu search. We describe the elements of the new approach and test the new learning algorithm on a series of classification problems. The test results demonstrate that the algorithm is significantly superior to several implementations of back-propagation.     

Evaluation of fuzzy regression models by fuzzy neural network

In this paper, a novel hybrid method based on fuzzy neural network for approximate fuzzy coefficients (parameters) of fuzzy linear and nonlinear regression models with fuzzy output and crisp inputs, is presented. Here a neural network is considered as a part of a large field called neural computing or soft computing. Moreover, in order to find the approximate parameters, a simple algorithm from the cost function of the fuzzy neural network is proposed. Finally, we illustrate our approach by some numerical examples.     

The estimate for approximation error of spherical neural networks

Compared with planar hyperplane, fitting data on the sphere has been an important and an active issue in geoscience, metrology, brain imaging, and so on. In this paper, with the help of the Jackson‐type theorem of polynomial approximation on the sphere, we construct spherical feed‐forward neural networks to approximate the continuous function defined on the sphere. As a metric, the modulus of smoothness of spherical function is used to measure the error of the approximation, and a Jackson‐type theorem on the approximation is established. Copyright © 2011 John Wiley & Sons, Ltd.     

Approximating smooth and sparse functions by deep neural networks: Optimal approximation rates and saturation

Constructing neural networks for function approximation is a classical and longstanding topic in approximation theory. In this paper, we aim at constructing deep neural networks with three hidden layers using a sigmoidal activation function to approximate smooth and sparse functions. Specifically, we prove that the constructed deep nets with controllable magnitude of free parameters can reach the optimal approximation rate in approximating both smooth and sparse functions. In particular, we prove that neural networks with three hidden layers can avoid the phenomenon of saturation, i.e., the phenomenon that for some neural network architectures, the approximation rate stops improving for functions of very high smoothness.     

The construction and approximation of feedforward neural network with hyp erb olic tangent function

In this paper, we discuss some analytic properties of hyperbolic tangent function and estimate some approximation errors of neural network operators with the hyperbolic tangent activation functionFirstly, an equation of partitions of unity for the hyperbolic tangent function is givenThen, two kinds of quasi-interpolation type neural network operators are constructed to approximate univariate and bivariate functions, respectivelyAlso, the errors of the approximation are estimated by means of the modulus of continuity of functionMoreover, for approximated functions with high order derivatives, the approximation errors of the constructed operators are estimated.     

Nonmonotone BFGS-trained recurrent neural networks for temporal sequence processing

In this paper we propose a nonmonotone approach to recurrent neural networks training for temporal sequence processing applications. This approach allows learning performance to deteriorate in some iterations, nevertheless the network’s performance is improved over time. A self-scaling BFGS is equipped with an adaptive nonmonotone technique that employs approximations of the Lipschitz constant and is tested on a set of sequence processing problems. Simulation results show that the proposed algorithm outperforms the BFGS as well as other methods previously applied to these sequences, providing an effective modification that is capable of training recurrent networks of various architectures.     

Non-convergence of stochastic gradient descent in the training of deep neural networks

Deep neural networks have successfully been trained in various application areas with stochastic gradient descent. However, there exists no rigorous mathematical explanation why this works so well. The training of neural networks with stochastic gradient descent has four different discretization parameters: (i) the network architecture; (ii) the amount of training data; (iii) the number of gradient steps; and (iv) the number of randomly initialized gradient trajectories. While it can be shown that the approximation error converges to zero if all four parameters are sent to infinity in the right order, we demonstrate in this paper that stochastic gradient descent fails to converge for ReLU networks if their depth is much larger than their width and the number of random initializations does not increase to infinity fast enough.     

Derivative-free optimization and neural networks for robust regression

Large outliers break down linear and nonlinear regression models. Robust regression methods allow one to filter out the outliers when building a model. By replacing the traditional least squares criterion with the least trimmed squares (LTS) criterion, in which half of data is treated as potential outliers, one can fit accurate regression models to strongly contaminated data. High-breakdown methods have become very well established in linear regression, but have started being applied for non-linear regression only recently. In this work, we examine the problem of fitting artificial neural networks (ANNs) to contaminated data using LTS criterion. We introduce a penalized LTS criterion which prevents unnecessary removal of valid data. Training of ANNs leads to a challenging non-smooth global optimization problem. We compare the efficiency of several derivative-free optimization methods in solving it, and show that our approach identifies the outliers correctly when ANNs are used for nonlinear regression.     

Approximation by radial bases and neural networks

In this paper, we study approximation by radial basis functions including Gaussian, multiquadric, and thin plate spline functions, and derive order of approximation under certain conditions. Moreover, neural networks are also constructed by wavelet recovery formula and wavelet frames.     

The capability of approximation for neural networks interpolant on the sphere

Compared with planar hyperplane, fitting data on the sphere has been an important and active issue in geoscience, metrology, brain imaging, and so on. In this paper, using a functional approach, we rigorously prove that for given distinct samples on the unit sphere there exists a feed‐forward neural network with single hidden layer which can interpolate the samples, and simultaneously near best approximate the target function in continuous function space. Also, by using the relation between spherical positive definite radial basis functions and the basis function on the Euclidean space ?^{d + 1}, a similar result in a spherical Sobolev space is established. Copyright © 2010 John Wiley & Sons, Ltd.     

A deep neural network-based method for solving obstacle problems

In this paper, we propose a method based on deep neural networks to solve obstacle problems. By introducing penalty terms, we reformulate the obstacle problem as a minimization optimization problem and utilize a deep neural network to approximate its solution. The convergence analysis is established by decomposing the error into three parts: approximation error, statistical error and optimization error. The approximate error is bounded by the depth and width of the network, the statistical error is estimated by the number of samples, and the optimization error is reflected in the empirical loss term. Due to its unsupervised and meshless advantages, the proposed method has wide applicability. Numerical experiments illustrate the effectiveness and robustness of the proposed method and verify the theoretical proof.     

Fuzzy regression analysis using neural networks

In this paper, we propose simple but powerful methods for fuzzy regression analysis using neural networks. Since neural networks have high capability as an approximator of nonlinear mappings, the proposed methods can be applied to more complex systems than the existing LP based methods. First we propose learning algorithms of neural networks for determining a nonlinear interval model from the given input-output patterns. A nonlinear interval model whose outputs approximately include all the given patterns can be determined by two neural networks. Next we show two methods for deriving nonlinear fuzzy models from the interval model determined by the proposed algorithms. Nonlinear fuzzy models whose h-level sets approximately include all the given patterns can be derived. Last we show an application of the proposed methods to a real problem.     

New LMI-based criteria for global robust stability of delayed neural networks

Some novel, linear matrix inequality based, criteria for the uniqueness and global robust stability of the equilibrium point of Hopfield-type neural networks with delay are presented. A comparison of the present criteria with the previous criteria is made.     

Fuzzified neural network based on fuzzy number operations

The fuzzified neural network based on fuzzy number operations is presented as a powerful modelling tool here. We systematically introduce ideas and concepts of a novel neural network based on fuzzy number operations. First we suggest how to compute the results of addition, subtraction, multiplication and division for two fuzzy numbers. Second we propose a learning algorithm, and present some ideas about the choice of fuzzy weights and fuzzy biases and a numerical scheme for the calculation of outputs of the fuzzified neural network. Finally, we show some results of computer simulations.     

Approximation by neural networks with sigmoidal functions

In this paper, we introduce a type of approximation operators of neural networks with sigmodal functions on compact intervals, and obtain the pointwise and uniform estimates of the ap- proximation. To improve the approximation rate, we further introduce a type of combinations of neurM networks. Moreover, we show that the derivatives of functions can also be simultaneously approximated by the derivatives of the combinations. We also apply our method to construct approximation operators of neural networks with sigmodal functions on infinite intervals.     

Augmented Lagrangian Active Set Methods for Obstacle Problems

Active set strategies for two-dimensional and three-dimensional, unilateral and bilateral obstacle problems are described. Emphasis is given to algorithms resulting from the augmented Lagrangian (i.e., primal-dual formulation of the discretized obstacle problems), for which convergence and rate of convergence are considered. For the bilateral case, modifications of the basic primal-dual algorithm are also introduced and analyzed. Finally, efficient computer realizations that are based on multigrid and multilevel methods are suggested and different aspects of the proposed techniques are investigated through numerical experiments.     

Encapsulation of air vessel design in a neural network

The trial and error process of calculating the characteristics of an air vessel suitable to protect a rising main against the effects of hydraulic transients has proved to be cumbersome for the design engineer. The own experience and sets of charts, which can be found in the literature, can provide some help. The aim of this paper is to present a neural network allowing instantaneous and direct calculation of air and vessel volumes from the system parameters. This neural network has been implemented in the hydraulic transient simulation package DYAGATS.     

Multigrid and multilevel methods for nonconforming elements

In this paper we study theoretical properties of multigrid algorithms and multilevel preconditioners for discretizations of second-order elliptic problems using nonconforming rotated finite elements in two space dimensions. In particular, for the case of square partitions and the Laplacian we derive properties of the associated intergrid transfer operators which allow us to prove convergence of the -cycle with any number of smoothing steps and close-to-optimal condition number estimates for -cycle preconditioners. This is in contrast to most of the other nonconforming finite element discretizations where only results for -cycles with a sufficiently large number of smoothing steps and variable -cycle multigrid preconditioners are available. Some numerical tests, including also a comparison with a preconditioner obtained by switching from the nonconforming rotated discretization to a discretization by conforming bilinear elements on the same partition, illustrate the theory.