A scatter-search-based learning algorithm for neural network training

In this article, we propose a new scatter-search-based learning algorithm to train feed-forward neural networks. The algorithm also incorporates elements of tabu search. We describe the elements of the new approach and test the new learning algorithm on a series of classification problems. The test results demonstrate that the algorithm is significantly superior to several implementations of back-propagation.     

Max—min神经网络的一种有效学习算法

本文在Ａ．Ｂｌａｎｃｏ等人的算法的基础上，提出了ｍａｘ－ｍｉｎ神经网络的一种改进了的反馈学习算法，严格证明了该算法的迭代收敛性，理论分析及实例计算结果均表明，本文算法具有算法简单，收敛速度快，输出误差小等显著特点。     

A shunting inhibitory cellular neural network with continuously distributed delays of neutral type

In this paper, a shunting inhibitory cellular neural network with continuously distributed delays of neutral type is considered. We establish some new results about the existence and exponential stability of the almost periodic solution for the shunting inhibitory cellular neural network.     

Complex-valued Zhang neural network for online complex-valued time-varying matrix inversion

In this paper, a new complex-valued recurrent neural network (CVRNN) called complex-valued Zhang neural network (CVZNN) is proposed and simulated to solve the complex-valued time-varying matrix-inversion problems. Such a CVZNN model is designed based on a matrix-valued error function in the complex domain, and utilizes the complex-valued first-order time-derivative information of the complex-valued time-varying matrix for online inversion. Superior to the conventional complex-valued gradient-based neural network (CVGNN) and its related methods, the state matrix of the resultant CVZNN model can globally exponentially converge to the theoretical inverse of the complex-valued time-varying matrix in an error-free manner. Moreover, by exploiting the design parameter γ>1, superior convergence can be achieved for the CVZNN model to solve such complex-valued time-varying matrix inversion problems, as compared with the situation without design parameter γ involved (i.e., the situation with γ=1). Computer-simulation results substantiate the theoretical analysis and further demonstrate the efficacy of such a CVZNN model for online complex-valued time-varying matrix inversion.     

TRUST REGION METHOD IN NEURAL NETWORK

1.IntroductionHopfieldandTank[5]presentedamodeltosolvetravellingsalesmanproblem,thusinitiatingtheapplicationofneuralnetwork(NN)inthefieldofoptimization.SincethenmanyNNmodelshavebeenproposedtosolvelinearprogramming(LP)problems(13,8,11,14,15])andquadraticprogramming(oP)problems([1,8,20]),asLPandoPhavefundamentalimportanceinthetheoryandpracticeofoptimization.Therewerealsoafewmodelsforgeneralnonlinearprogramming(NP)problem([2,6,9,18]).InthispaperwewillpresentaHopfield-typeneuralnetworkmodelw…     

Evaluation of fully fuzzy matrix equations by fuzzy neural network

In this paper, a new hybrid method based on fuzzy neural network for approximate solution of fully fuzzy matrix equations of the form AX=D$\mathit{AX}=D$, where A and D are two fuzzy number matrices and the unknown matrix X is a fuzzy number matrix, is presented. Then, we propose some definitions which are fuzzy zero number, fuzzy one number and fuzzy identity matrix. Based on these definitions, direct computation of fuzzy inverse matrix is done using fuzzy matrix equations and fuzzy neural network. It is noted that the uniqueness of the calculated fuzzy inverse matrix is not guaranteed. Here a neural network is considered as a part of a large field called neural computing or soft computing. Moreover, in order to find the approximate solution of fuzzy matrix equations that supposedly has a unique fuzzy solution, a simple algorithm from the cost function of the fuzzy neural network is proposed. To illustrate the easy application of the proposed method, numerical examples are given and the obtained results are discussed.     

基于类神经网络的城市交通信号区域优化算法

提出了一种可并行处理的交通信号配时区域优化模型和相应算法. 算法从局部枚举最优方案出发, 在枚举计算每个交叉口信号灯方案的罚分时, 在真实罚分的基础上叠加虚拟导向罚分. 虚拟导向罚分通过动态通行权重来计算. 将枚举法和虚拟导向罚分相结合, 使得算法具有空间和时间上的全局优化特性. 在道路处于饱和或过饱和状态时, 该算法相对于传统的单点定时或单点感应等交通信号配时方案具有明显的优化效果.     

New approach to applying neural network in nonlinear dynamic model

In this work, radial basis function neural network (RBF-NN) is applied to emulate an extended Kalman filter (EKF) in a data assimilation scenario. The dynamical model studied here is based on the one-dimensional shallow water equation DYNAMO-1D. This code is simple when compared with an operational primitive equation models for numerical weather prediction. Although simple, the DYNAMO-1D is rich for representing some atmospheric motions, such as Rossby and gravity waves. It has been shown in the literature that the ability of the EKF to track nonlinear models depends on the frequency and accuracy of the observations and model errors. In some cases, just fourth-order moment EKF works well, but will be unwieldy when applied to high-dimensional state space. Artificial Neural Network (ANN) is an alternative solution for this computational complexity problem, once the ANN is trained offline with a high order Kalman filter, even though this Kalman filter has high computational cost (which is not a problem during ANN training phase). The results achieved in this work encourage us to apply this technique on operational model. However, it is not yet possible to assure convergence in high dimensional problems.     

Analysis of pseudo-periodic chronological series with irregularly time-spaced data in view to their prediction. IV. Using a symbolic learning method

We first present briefly the CALM learning method, based upon the idea of belief. Then we state the multi-agents scheme in which such a method can be used to predict numerical values. The basic idea is to simulate the expert's reasoning in front of a graphical display of the numerical values representing the phenomenon he wants to study:

• (a) First, looking at local shapes in the curve
• (b) Secondly, using maxima, minima and/or zero-crossings to prevent long range errors in the prediction.

We present some results on the astronomy problem presented by M. O. Menessier about the prediction of brightness variation of Mira stars.     

Evaluation of fuzzy regression models by fuzzy neural network

In this paper, a novel hybrid method based on fuzzy neural network for approximate fuzzy coefficients (parameters) of fuzzy linear and nonlinear regression models with fuzzy output and crisp inputs, is presented. Here a neural network is considered as a part of a large field called neural computing or soft computing. Moreover, in order to find the approximate parameters, a simple algorithm from the cost function of the fuzzy neural network is proposed. Finally, we illustrate our approach by some numerical examples.     

On the robustness of regularized pairwise learning methods based on kernels

Regularized empirical risk minimization including support vector machines plays an important role in machine learning theory. In this paper regularized pairwise learning (RPL) methods based on kernels will be investigated. One example is regularized minimization of the error entropy loss which has recently attracted quite some interest from the viewpoint of consistency and learning rates. This paper shows that such RPL methods and also their empirical bootstrap have additionally good statistical robustness properties, if the loss function and the kernel are chosen appropriately. We treat two cases of particular interest: (i) a bounded and non-convex loss function and (ii) an unbounded convex loss function satisfying a certain Lipschitz type condition.     

Horseshoe in a model of artificial neural network

In this paper, we study a three-dimensional general model of artificial neural network (ANN). To confirm the chaotic behavior in this neural network demonstrated in numerical studies, we consider a cross-section properly chosen for the attractor obtained and study the dynamics of the corresponding Poincaré map, and rigorously verify the existence of horseshoe by computer-assisted verification arguments.     

Universality of deep convolutional neural networks

Deep learning has been widely applied and brought breakthroughs in speech recognition, computer vision, and many other domains. Deep neural network architectures and computational issues have been well studied in machine learning. But there lacks a theoretical foundation for understanding the approximation or generalization ability of deep learning methods generated by the network architectures such as deep convolutional neural networks. Here we show that a deep convolutional neural network (CNN) is universal, meaning that it can be used to approximate any continuous function to an arbitrary accuracy when the depth of the neural network is large enough. This answers an open question in learning theory. Our quantitative estimate, given tightly in terms of the number of free parameters to be computed, verifies the efficiency of deep CNNs in dealing with large dimensional data. Our study also demonstrates the role of convolutions in deep CNNs.     

A novel neural network based on NCP function for solving constrained nonconvex optimization problems

This article presents a novel neural network (NN) based on NCP function for solving nonconvex nonlinear optimization (NCNO) problem subject to nonlinear inequality constraints. We first apply the p‐power convexification of the Lagrangian function in the NCNO problem. The proposed NN is a gradient model which is constructed by an NCP function and an unconstrained minimization problem. The main feature of this NN is that its equilibrium point coincides with the optimal solution of the original problem. Under a proper assumption and utilizing a suitable Lyapunov function, it is shown that the proposed NN is Lyapunov stable and convergent to an exact optimal solution of the original problem. Finally, simulation results on two numerical examples and two practical examples are given to show the effectiveness and applicability of the proposed NN. © 2015 Wiley Periodicals, Inc. Complexity 21: 130–141, 2016     

Solving a classification task by spiking neural network with STDP based on rate and temporal input encoding

This paper develops local learning algorithms to solve a classification task with the help of biologically inspired mathematical models of spiking neural networks involving the mechanism of spike-timing-dependent plasticity (STDP). The advantages of the models are their simplicity and, hence, the potential ability to be hardware-implemented in low-energy-consuming biomorphic computing devices. The methods developed are based on two key effects observed in neurons with STDP: mean firing rate stabilization and memorizing repeating spike patterns. As the result, two algorithms to solve a classification task with a spiking neural network are proposed: the first based on rate encoding of the input data and the second based on temporal encoding. The accuracy of the algorithms is tested on the benchmark classification tasks of Fisher's Iris and Wisconsin breast cancer, with several combinations of input data normalization and preprocessing. The respective accuracies are 99% and 94% by F1-score.     

Design of fuzzy radial basis function-based polynomial neural networks

In this study, we introduce a new design methodology of fuzzy radial basis function-based polynomial neural networks. In many cases, these models do not come with capabilities to deal with granular information. With this regard, fuzzy sets offer several interesting and useful opportunities. This study presents the development of fuzzy radial basis function-based neural networks augmented with virtual input variables. The performance of the proposed category of models is quantified through a series of experiments, in which we use two machine learning data sets and two publicly available software development effort data.     

Numerical control of kohonen neural network for scattered data approximation

Surface reconstruction from scattered data using Kohonen neural network is presented in this paper. The network produces a topologically predefined grid from the unordered data which can be applied as a rough approximation of the input set or as a base surface for further process. The quality and computing time of the approximation can be controlled by numerical parameters. As a further application, ruled surface is produced from a set of unordered lines by the network. AMS subject classification 68U07, 65D17, 68T20     

Community detection-based deep neural network architectures: A fully automated framework based on Likert-scale data

Deep neural networks (DNNs) have emerged as a state-of-the-art tool in very different research fields due to its adaptive power to the decision space since they do not presuppose any linear relationship between data. Some of the main disadvantages of these trending models are that the choice of the network underlying architecture profoundly influences the performance of the model and that the architecture design requires prior knowledge of the field of study. The use of questionnaires is hugely extended in social/behavioral sciences. The main contribution of this work is to automate the process of a DNN architecture design by using an agglomerative hierarchical algorithm that mimics the conceptual structure of such surveys. Although the train had regression purposes, it is easily convertible to deal with classification tasks. Our proposed methodology will be tested with a database containing socio-demographic data and the responses to five psychometric Likert scales related to the prediction of happiness. These scales have been already used to design a DNN architecture based on the subdimension of the scales. We show that our new network configurations outperform the previous existing DNN architectures.     

Non-convergence of stochastic gradient descent in the training of deep neural networks

Deep neural networks have successfully been trained in various application areas with stochastic gradient descent. However, there exists no rigorous mathematical explanation why this works so well. The training of neural networks with stochastic gradient descent has four different discretization parameters: (i) the network architecture; (ii) the amount of training data; (iii) the number of gradient steps; and (iv) the number of randomly initialized gradient trajectories. While it can be shown that the approximation error converges to zero if all four parameters are sent to infinity in the right order, we demonstrate in this paper that stochastic gradient descent fails to converge for ReLU networks if their depth is much larger than their width and the number of random initializations does not increase to infinity fast enough.     

一种神经网络算子及其逼近阶估计

讨论了一种神经网络算子f_n(x)=sum from -n~2 to n~2 (f(k/n))/(n~α)b(n~(1-α)(x-k/n)),对f(x)的逼近误差|f_n(x)-f(x)|的上界在f(x)为连续和N阶连续可导两种情形下分别给出了该网络算子逼近的Jackson型估计.