Automatic structure and parameter training methods for modeling of mechanical systems by recurrent neural networks

Automatic nonlinear-system identification is very useful for various disciplines including, e.g., automatic control, mechanical diagnostics and financial market prediction. This paper describes a fully automatic structural and weight learning method for recurrent neural networks (RNN). The basic idea is training with residuals, i.e., a single hidden neuron RNN is trained to track the residuals of an existing network before it is augmented to the existing network to form a larger and, hopefully, better network. The network continues to grow until either a desired level of accuracy or a preset maximal number of neurons is reached. The method requires no guessing of initial weight values or the number of neurons in the hidden layer from users. This new structural and weight learning algorithm is used to find RNN models for a two-degree-of-freedom planar robot, a Van der Pol oscillator and a Mackey–Glass equation using their simulated responses to excitations. The algorithm is able to find good RNN models in all three cases.     

Recurrent neural dynamic models for equilibrium and eigenvalue problems

Neural networks (NN) have been used in a number of interesting applications. In this paper, two neural dynamic models which belong to the class of recurrent neural networks (RNN) have been formulated for the solution of equilibrium and eigenvalue problems. The RNN is comprised of two layers, namely, variable layer and constraint layer, which correspond to the number of design variables in the problem. In addition, the recurrent connections and feed forward connections are used to represent the incremental values in the design parameters. The stability of the neural dynamic model for the equilibrium problem has been guaranteed using Lyapunov's function. Illustrative examples and results of the computer simulation of the neural dynamic model have also been presented.     

Collective behavior of interacting locally synchronized oscillations in neuronal networks

Local circuits in the cortex and hippocampus are endowed with resonant, oscillatory firing properties which underlie oscillations in various frequency ranges (e.g. gamma range) frequently observed in the local field potentials, and in electroencephalography. Synchronized oscillations are thought to play important roles in information binding in the brain. This paper addresses the collective behavior of interacting locally synchronized oscillations in realistic neural networks. A network of five neurons is proposed in order to produce locally synchronized oscillations. The neuron models are Hindmarsh–Rose type with electrical and/or chemical couplings. We construct large-scale models using networks of such units which capture the essential features of the dynamics of cells and their connectivity patterns. The profile of the spike synchronization is then investigated considering different model parameters such as strength and ratio of excitatory/inhibitory connections. We also show that transmission time-delay might enhance the spike synchrony. The influence of spike-timing-dependence-plasticity is also studies on the spike synchronization.     

Interpolation by neural network operators activated by ramp functions

In this paper, a family of interpolation neural network operators are introduced. Here, ramp functions as well as sigmoidal functions generated by central B-splines are considered as activation functions. The interpolation properties of these operators are proved, together with a uniform approximation theorem with order, for continuous functions defined on bounded intervals. The relations with the theory of neural networks and with the theory of the generalized sampling operators are discussed.     

Cantorian Fractal Spacetime and Information in Neural Network of the Human Brain

The neural networks of the human brain act as very efficient parallel processing computers co-ordinating memory related responses to a multitude of input signals from sensory organs. Information storage, update and appropriate retrieval are controlled at the molecular level by the neuronal cytoskeleton which serves as the internal communication network within neurons. Information flow in the highly ordered parallel networks of the filamentous protein polymers which make up the cytoskeleton may be compared to atmospheric flows which exhibit long-range spatiotemporal correlations, i.e. long-term memory. Such long-range spatiotemporal correlations are ubiquitous to real world dynamical systems and is recently identified as signature of self-organized criticality or chaos. The signatures of self-organized criticality i.e. long-range temporal correlations have recently been identified in the electrical activity of the brain. The physics of self-organized criticality or chaos is not yet identified. A recently developed non-deterministic cell dynamical system model for atmospheric flows predicts the observed long-range spatiotemporal correlations as intrinsic to quantum-like mechanics governing flow dynamics. The model visualises large scale circulations to form as the result of spatial integration of enclosed small scale perturbations with intrinsic two-way ordered energy flow between the scales. Such a concept maybe applied for the collection and integration of a multitude of signals at the cytoskeletal level and manifested in activation of neurons in the macroscale. The cytoskeleton networks inside neurons may be the elementary units of a unified dynamic memory circulation network with intrinsic global response to local stimuli. A cell dynamical system model for human memory circulation network analogous to atmospheric circulations network is presented in this paper. The model like the analysis of Koruga et al. make use of certain connections to the concept of Cantorian-Fractal spacetime.     

具有不连续神经激励的循环神经网络的周期解存在性

本文研究了一类可以描述为右端不连续微分方程的循环神经网络模型.在并不要求激励函数连续、有界及单调非减的情况下,通过利用线性矩阵不等式和微分包含中的Cellina近似选择定理,得到了该神经网络模型存在周期解的充分条件.最后,给出了一个数值例子用以说明本文结果的有效性.     

基于神经网络的股票预测模型

针对股票市场的特征提取困难、预测精度较低等问题,本文基于深度学习算法,构建了一系列用于股票市场预测的神经网络模型,包括基于多层感知机(MLP)、卷积神经网络(CNN)、递归神经网络(RNN)、长短期记忆网络(LSTM)和门控神经单元(GRU)的模型。 针对RNN、LSTM和GRU无法充分利用所参考的时间维度的信息,引入注意力机制(Attention Mechanism) 给各时间维度的信息赋予不同权重,区分不同信息对预测的重要程度,从而提升递归网络模型的性能。上述模型均基于股票数据进行了优化,基于上证指数对各类模型进行了充分的对比实验,探索了模型中重要变量对性能的影响,旨在为基于神经网络的股票预测模型给出具体的优化方向。     

基于投影算子的回归神经网络模型及其在最优化问题中的应用

研究了一种基于投影算子的神经网络模型．与以前研究投影算子的值域一般是n维欧氏空间中的紧凸子集不同,而是n维欧氏空间中未必有界的闭凸子集,同时目标函数也是一般的连续可微函数,未必为凸函数．证明了所研究的神经网络模型具有整体解轨道,以及当目标函数满足某些条件时解轨道的整体收敛性．此外,还将所研究的模型应用于闭凸约束极小化问题以及非线性互补问题和隐互补问题中,并通过数值模拟说明了该神经网络方法的有效性．     

球面神经网络的构造与逼近

研究球面神经网络的构造与逼近问题.利用球面广义的de la Vallee Poussin平均、球面求积公式及改进的单变量Cardaliaguet-Euvrard神经网络算子,构造具logistic激活函数的单隐层前向网络,并给出了Jackson型误差估计.     

Convergence for a family of neural network operators in Orlicz spaces

In this paper, we develop the theory for a family of neural network (NN) operators of the Kantorovich type, in the general setting of Orlicz spaces. In particular, a modular convergence theorem is established. In this way, we study the above family of operators in many instances of useful spaces by a unique general approach. The above NN operators provide a constructive approximation process, in which the coefficients, the weights, and the thresholds of the networks needed in order to approximate a given function f , are known. At the end of the paper, several examples of Orlicz spaces, and of sigmoidal activation functions for which the present theory can be applied, are studied in details.     

����fMRI���ݵ�ȫ�Զ�̬�������������������Ե��о�

There're about 10^{11} neurons in the human brain.Through the synaptic junction, neurons have formed a highly complex network.And it is really important to figure out the information expressed in the network, which will contribute to the resolution of the prevention and diagnosis of cognitive disorder of human beings. This paper uses the schizophrenia and healthy controlled subjects' fMRI data to construct the brain network model, in order to explores abnormal topological properties of schizophrenics' brain network based on graph theory. When studying the human brain network information traditionally by the basement of graph theory, it's all assure that the human brain network model has invariance, so it takes the whole period of time series data in constructing human network model, which is a kind static network. However, it's hard to ensure this because of the nonstationarity of fMRI functional time series data. Thus, when constructing human brain network model, we should take its time-variation into consideration, then construct a dynamic brain network. We can explore the brain network information better. In this research, we segment the time series data, using time windows, to constructing dynamic brain network model, then analyze it combined with the knowledge of graph theory, thereby reducing effects that the nonstationarity of fMRI functional time series data will have. Comparing dynamic brain network of the schizophrenic patients with normal controls subjects' in different level, the results show that there are difference in single node property, group network property of schizophrenic patients and normal control subjects' whole brain dynamic functional connectivity network. The discovery of these difference in network topological properties has provide new clues for the further study on the pathological mechanism of schizophrenia.     

A genetic algorithm to obtain the optimal recurrent neural network

Selecting the optimal topology of a neural network for a particular application is a difficult task. In the case of recurrent neural networks, most methods only induce topologies in which their neurons are fully connected. In this paper, we present a genetic algorithm capable of obtaining not only the optimal topology of a recurrent neural network but also the least number of connections necessary. Finally, this genetic algorithm is applied to a problem of grammatical inference using neural networks, with very good results.     

Impact of Correlated Activity and STDP on Network Structure

Synaptic strengths between neurons are plastic and modified by spontaneous activity and information from the outside. There is increasing interest in the impact of correlated neuron activity and learning rules on global network structure. Here the networks of exponential integrate-and-fire neurons with spike timing-dependent plasticity (STDP) learning rules are considered, by providing the theoretical approximation of spiking cross-covariance between connected neurons and the theory for the evolution of synaptic weights. Background input mean and variance highly affect the spiking covariance, even for the fixed baseline firing rate and connection. Through analyzing the effects of covariance and STDP on vector fields for pairwise correlated neurons under fixed baseline firing rate, we show that the connections from a neuron with lower input mean to that with higher one will strengthen for balanced Hebbian STDP. However, this situation is reversed for Anti-Hebbian cases. Moreover, for potentiation dominated STDP, the synaptic weights for the networks of neurons with lower input mean are more likely to be enhanced. In addition, these properties found from coupled neurons also hold for large recurrent networks in both theories and simulations. This study provides a self-consistent theoretical method for understanding how correlated spiking activity and STDP shape the network structure and an approach for predicting structures of large networks through the analysis of simple neural circuits.     

基于忆阻的时滞神经网络的全局稳定性

忆阻是近年来新发现的一类非线性电子元件．与通常的电阻不同,忆阻的阻值会随着通过的电流量的大小和方向不同而改变．这个特性使得忆阻具有了记忆的功能,在很多方面有着广泛的应用．该文给出了简化的忆阻的数学模型,基于该模型构造了时滞神经网络,利用微分包含理论、Lyapunov方法和同胚映射原理研究了其全局渐近稳定性问题,确保模型平衡点存在性、唯一性和一致全局渐近稳定性的充分条件被获得．最后提供的具有仿真的例子验证了获得的理论结果．     

Some Joys and Trials of Mathematical Neuroscience

I describe the basic components of the nervous system—neurons and their connections via chemical synapses and electrical gap junctions—and review the model for the action potential produced by a single neuron, proposed by Hodgkin and Huxley (HH) over 60 years ago. I then review simplifications of the HH model and extensions that address bursting behavior typical of motoneurons, and describe some models of neural circuits found in pattern generators for locomotion. Such circuits can be studied and modeled in relative isolation from the central nervous system and brain, but the brain itself (and especially the human cortex) presents a much greater challenge due to the huge numbers of neurons and synapses involved. Nonetheless, simple stochastic accumulator models can reproduce both behavioral and electrophysiological data and offer explanations for human behavior in perceptual decisions. In the second part of the paper I introduce these models and describe their relation to an optimal strategy for identifying a signal obscured by noise, thus providing a norm against which behavior can be assessed and suggesting reasons for suboptimal performance. Accumulators describe average activities in brain areas associated with the stimuli and response modes used in the experiments, and they can be derived, albeit non-rigorously, from simplified HH models of excitatory and inhibitory neural populations. Finally, I note topics excluded due to space constraints and identify some open problems.     

Solubility and stability of recurrent neural networks with nonlinearity or time-varying delays

Because to apply a deterministic RNN to a noisy time series and the existence of a linear approximation are doubtful, we reconsider the solubility and stability of a recurrent neural network (RNN). Simpler methods are proposed to replace the complicated nonsingular M-matrix method when nonlinearities exist and to replace the complicated linear matrix inequality method when time-varying delays exist.     

The construction and approximation of feedforward neural network with hyp erb olic tangent function

In this paper, we discuss some analytic properties of hyperbolic tangent function and estimate some approximation errors of neural network operators with the hyperbolic tangent activation functionFirstly, an equation of partitions of unity for the hyperbolic tangent function is givenThen, two kinds of quasi-interpolation type neural network operators are constructed to approximate univariate and bivariate functions, respectivelyAlso, the errors of the approximation are estimated by means of the modulus of continuity of functionMoreover, for approximated functions with high order derivatives, the approximation errors of the constructed operators are estimated.     

Structure-reversibility and departure functions of queueing networks with batch movements and state dependent routing

We consider characterizations of departure functions in Markovian queueing networks with batch movements and state-dependent routing in discrete-time and in continuous-time. For this purpose, the notion of structure-reversibility is introduced, which means that the time-reversed dynamics of a queueing network corresponds with the same type of queueing network. The notion is useful to derive a traffic equation. We also introduce a multi-source model, which means that there are different types of outside sources, to capture a wider range of applications. Characterizations of the departure functions are obtained for any routing mechanism of customers satisfying a recurrent condition. These results give a unified view to queueing network models with linear traffic equations. Furthermore, they enable us to consider new examples as well as show limited usages of this kind of queueing networks. This revised version was published online in June 2006 with corrections to the Cover Date.     

Global output convergence of recurrent neural networks with distributed delays

This paper discusses the global output convergence of a class of recurrent neural networks with distributed delays. The inputs of the neural networks are required to be time varying and the activation functions to be globally continuous and monotone nondecreasing. By using the definiteness of matrix and the properties of M-matrix, several sufficient conditions are established to guarantee the global output convergence of this class of neural networks. Symmetry in the connection weight matrices and the boundedness of the activation functions are not required in this paper. The convergence results are useful in solving some optimization problems and in the design of recurrent neural networks with distributed delays.     

默认模式网络与任务正网络之间相互拮抗的神经动力学分析

任务正激活与任务负激活的工作机制是认知功能实现的基本要素.这一拮抗关系的失衡或者受损可能会引发一系列严重的退行性神经疾病，然而到目前为止，尚不清楚这种拮抗现象的神经机制.该文基于默认模式网络与任务正网络在突触层面上相互抑制的假设，并结合多种刺激条件下的工作记忆模型，进行了计算机数值模拟.研究结果表明: 1) 任务正网络与任务负网络之间在神经活动上呈现出拮抗关系； 2) 伴随着工作记忆刺激方向数目的增加，任务负网络神经活动的衰减程度会随之增大； 3) 当工作记忆相关的脑区其神经活动增加时，任务负网络的神经活动减少； 4) 并且随着工作记忆任务难度的增加，任务负网络的神经活动会迅速衰减.这些计算结果都与神经科学实验数据是匹配的.由于任务负激活是默认模式网络的主要特征，因此默认模式网络与任务正网络在突触层面上的相互抑制是这两种不同性质网络之间形成拮抗关系的根本原因.