Learning in Convolutional Neural Networks Accelerated by Transfer Entropy

Recently, there is a growing interest in applying Transfer Entropy (TE) in quantifying the effective connectivity between artificial neurons. In a feedforward network, the TE can be used to quantify the relationships between neuron output pairs located in different layers. Our focus is on how to include the TE in the learning mechanisms of a Convolutional Neural Network (CNN) architecture. We introduce a novel training mechanism for CNN architectures which integrates the TE feedback connections. Adding the TE feedback parameter accelerates the training process, as fewer epochs are needed. On the flip side, it adds computational overhead to each epoch. According to our experiments on CNN classifiers, to achieve a reasonable computational overhead–accuracy trade-off, it is efficient to consider only the inter-neural information transfer of the neuron pairs between the last two fully connected layers. The TE acts as a smoothing factor, generating stability and becoming active only periodically, not after processing each input sample. Therefore, we can consider the TE is in our model a slowly changing meta-parameter.     

Bistability Analysis of Excitatory-Inhibitory Neural Networks in Limited-Sustained-Activity Regime

Bistable behavior of neuronal complex networks is investigated in the limited-sustained-activity regime when the network is composed of excitatory and inhibitory neurons. The standard stability analysis is performed on the two metastable states separately. Both theoretical analysis and numerical simulations show consistently that the difference between time scales of excitatory and inhibitory populations can influence the dynamical behaviors of the neuronal networks dramatically, leading to the transition from bistable behaviors with memory effects to the collapse of bistable behaviors.
These results may suggest one possible neuronal information processing by only tuning time scales.     

On the Role of the Excitation/Inhibition Balance of Homeostatic Artificial Neural Networks

Homeostatic models of artificial neural networks have been developed to explain the self-organization of a stable dynamical connectivity between the neurons of the net. These models are typically two-population models, with excitatory and inhibitory cells. In these models, connectivity is a means to regulate cell activity, and in consequence, intracellular calcium levels towards a desired target level. The excitation/inhibition (E/I) balance is usually set to 80:20, a value characteristic for cortical cell distributions. We study the behavior of these homeostatic models outside of the physiological range of the E/I balance, and we find a pronounced bifurcation at about the physiological value of this balance. Lower inhibition values lead to sparsely connected networks. At a certain threshold value, the neurons develop a reasonably connected network that can fulfill the homeostasis criteria in a stable way. Beyond the threshold, the behavior of the artificial neural network changes drastically, with failing homeostasis and in consequence with an exploding number of connections. While the exact value of the balance at the bifurcation point is subject to the parameters of the model, the existence of this bifurcation might explain the stability of a certain E/I balance across a wide range of biological neural networks. Assuming that this class of models describes the self-organization of biological network connectivity reasonably realistically, the omnipresent physiological balance might represent a case of self-organized criticality in order to obtain a good connectivity while allowing for a stable intracellular calcium homeostasis.     

Existence and stability of persistent states in large neuronal networks

We study the existence and stability of persistent states in large networks of quadratic integrate-and-fire neurons. The networks consist of two populations, one excitatory and one inhibitory. The stability of the asynchronous state is studied analytically. Our study demonstrates the role of recurrent inhibition and inhibitory-inhibitory interactions in stable persistent activity in large neuronal networks.     

Eigenvalue spectra of random matrices for neural networks

The dynamics of neural networks is influenced strongly by the spectrum of eigenvalues of the matrix describing their synaptic connectivity. In large networks, elements of the synaptic connectivity matrix can be chosen randomly from appropriate distributions, making results from random matrix theory highly relevant. Unfortunately, classic results on the eigenvalue spectra of random matrices do not apply to synaptic connectivity matrices because of the constraint that individual neurons are either excitatory or inhibitory. Therefore, we compute eigenvalue spectra of large random matrices with excitatory and inhibitory columns drawn from distributions with different means and equal or different variances.     

Percolation in living neural networks

We study living neural networks by measuring the neurons' response to a global electrical stimulation. Neural connectivity is lowered by reducing the synaptic strength, chemically blocking neurotransmitter receptors. We use a graph-theoretic approach to show that the connectivity undergoes a percolation transition. This occurs as the giant component disintegrates, characterized by a power law with an exponent beta approximately or = 0.65. Beta is independent of the balance between excitatory and inhibitory neurons and indicates that the degree distribution is Gaussian rather than scale free.     

Bifurcation in neuronal networks with hub structure

We investigate bifurcations in neuronal networks with a hub structure. It is known that hubs play a leading role in characterizing the network dynamical behavior. However, the dynamics of hubs or star-coupled systems is not well understood. Here, we study rather subnetworks with a star-like configuration. This coupled system is an important motif in complex networks. Thus, our study is a basic step for understanding structure formation in large networks. We use the Morris-Lecar neuron with class I and class II excitabilities as a node. Homogeneous (coupling the same class neurons) and heterogeneous (coupling different class neurons) cases are considered for both excitatory and inhibitory coupling. For the homogeneous system class II neurons are suitable for achieving both complete and cluster synchronization in excitatory and inhibitory coupling, respectively. For the heterogeneous system with inhibitory coupling, the class I hub neuron has a wider parameter region of synchronous firings than the class II hub. Moreover, the class I hub neuron with the excitatory synapse gives rise to bifurcations of synchronized states and multi-stability (coexistence of a few different states) is observed.     

An analytical approach to neuronal connectivity

This paper describes how to analytically characterize the connectivity of neuromorphic networks taking into account the morphology of their elements. By assuming that all neurons have the same shape and are regularly distributed along a two-dimensional orthogonal lattice with parameter , we obtain the exact number of connections and cycles of any length by applying convolutions and the respective spectral density derived from the adjacency matrix. It is shown that neuronal shape plays an important role in defining the spatial distribution of synapses in neuronal networks. In addition, we observe that neuromorphic networks typically present an interesting property where the pattern of connections is progressively shifted along the spatial domain for increasing connection lengths. This arises from the fact that the axon reference point usually does not coincide with the cell center of mass of neurons. Morphological measurements for characterization of the spatial distribution of connections, including the adjacency matrix spectral density and the lacunarity of the connections, are suggested and illustrated. We also show that Hopfield networks with connectivity defined by different neuronal morphologies, which are quantified by the analytical approach proposed herein, lead to distinct performances for associative recall, as measured by the overlap index. The potential of our approach is illustrated for digital images of real neuronal cells.     

Synaptic Plasticity and Spike Synchronisation in Neuronal Networks

Brain plasticity, also known as neuroplasticity, is a fundamental mechanism of neuronal adaptation in response to changes in the environment or due to brain injury. In this review, we show our results about the effects of synaptic plasticity on neuronal networks composed by Hodgkin-Huxley neurons. We show that the final topology of the evolved network depends crucially on the ratio between the strengths of the inhibitory and excitatory synapses. Excitation of the same order of inhibition revels an evolved network that presents the rich-club phenomenon, well known to exist in the brain. For initial networks with considerably larger inhibitory strengths, we observe the emergence of a complex evolved topology, where neurons sparsely connected to other neurons, also a typical topology of the brain. The presence of noise enhances the strength of both types of synapses, but if the initial network has synapses of both natures with similar strengths. Finally, we show how the synchronous behaviour of the evolved network will reflect its evolved topology.     

Phase synchronization of bursting neural networks with electrical and delayed dynamic chemical couplings

Diffusive electrical connections in neuronal networks are instantaneous, while excitatoryor inhibitory couplings through chemical synapses contain a transmission time-delay.Moreover, chemical synapses are nonlinear dynamical systems whose behavior can bedescribed by nonlinear differential equations. In this work, neuronal networks withdiffusive electrical couplings and time-delayed dynamic chemical couplings are considered.We investigate the effects of distributed time delays on phase synchronization of burstingneurons. We observe that in both excitatory and Inhibitory chemical connections, the phasesynchronization might be enhanced when time-delay is taken into account. This distributedtime delay can induce a variety of phase-coherent dynamical behaviors. We also study thecollective dynamics of network of bursting neurons. The network model presents theso-called Small-World property, encompassing neurons whose dynamics have two time scales(fast and slow time scales). The neuron parameters in such Small-World network, aresupposed to be slightly different such that, there may be synchronization of the bursting(slow) activity if the coupling strengths are large enough. Bounds for the criticalcoupling strengths to obtain burst synchronization in terms of the network structure aregiven. Our studies show that the network synchronizability is improved, as itsheterogeneity is reduced. The roles of synaptic parameters, more precisely those of thecoupling strengths and the network size are also investigated.     

Mean Field Theory of Self-Organizing Memristive Connectomes

Biological neuronal networks are characterized by nonlinear interactions and complex connectivity. Given the growing impetus to build neuromorphic computers, understanding physical devices that exhibit structures and functionalities similar to biological neural networks is an important step toward this goal. Self-organizing circuits of nanodevices are at the forefront of the research in neuromorphic computing, as their behavior mimics synaptic plasticity features of biological neuronal circuits. However, an effective theory to describe their behavior is lacking. This study provides for the first time an effective mean field theory for the emergent voltage-induced polymorphism of circuits of a nanowire connectome, showing that the behavior of these circuits can be explained by a low-dimensional dynamical equation. The equation can be derived from the microscopic dynamics of a single memristive junction in analytical form. The effective model is tested on experiments of nanowire networks and show that it fits both the potentiation and depression of these synapse-mimicking circuits. It is shown that this theory applies beyond the case of nanowire networks by formulating a general mean-field theory of conductance transitions in self-organizing memristive connectomes.     

Coherent periodic activity in excitatory Erdös-Renyi neural networks: the role of network connectivity

In this article, we investigate the role of connectivity in promoting coherent activity in excitatory neural networks. In particular, we would like to understand if the onset of collective oscillations can be related to a minimal average connectivity and how this critical connectivity depends on the number of neurons in the networks. For these purposes, we consider an excitatory random network of leaky integrate-and-fire pulse coupled neurons. The neurons are connected as in a directed Erdo?s-Renyi graph with average connectivity scaling as a power law with the number of neurons in the network. The scaling is controlled by a parameter γ, which allows to pass from massively connected to sparse networks and therefore to modify the topology of the system. At a macroscopic level, we observe two distinct dynamical phases: an asynchronous state corresponding to a desynchronized dynamics of the neurons and a regime of partial synchronization (PS) associated with a coherent periodic activity of the network. At low connectivity, the system is in an asynchronous state, while PS emerges above a certain critical average connectivity (c). For sufficiently large networks, (c) saturates to a constant value suggesting that a minimal average connectivity is sufficient to observe coherent activity in systems of any size irrespectively of the kind of considered network: sparse or massively connected. However, this value depends on the nature of the synapses: reliable or unreliable. For unreliable synapses, the critical value required to observe the onset of macroscopic behaviors is noticeably smaller than for reliable synaptic transmission. Due to the disorder present in the system, for finite number of neurons we have inhomogeneities in the neuronal behaviors, inducing a weak form of chaos, which vanishes in the thermodynamic limit. In such a limit, the disordered systems exhibit regular (non chaotic) dynamics and their properties correspond to that of a homogeneous fully connected network for any γ-value. Apart for the peculiar exception of sparse networks, which remain intrinsically inhomogeneous at any system size.     

Engineered self-organization of neural networks using carbon nanotube clusters

A novel approach was developed to form engineered, electrically viable, neuronal networks, consisting of ganglion-like clusters of neurons. In the present method, the clusters are formed as the cells migrate on low affinity substrate towards high affinity, lithographically defined carbon nanotube templates on which they adhere and assemble. Subsequently, the gangliated neurons send neurites to form interconnected networks with pre-designed geometry and graph connectivity. This process is distinct from previously reported formation of clusterized neural networks in which a network of linked neurons collapses via neuronal migration along the inter-neuron links. The template preparation method is based on photo-lithography, micro-contact printing and carbon nanotube chemical vapor deposition techniques. The present work provides a new approach to form complex, engineered, interconnected neuronal network with pre-designed geometry via engineering the self-assembly process of neurons.     

Stochastic resonance and synchronization behaviors of excitatory-inhibitory small-world network subjected to electromagnetic induction

The phenomenon of stochastic resonance and synchronization on some complex neuronal networks have been investigated extensively.These studies are of great significance for us to understand the weak signal detection and information transmission in neural systems.Moreover,the complex electrical activities of a cell can induce time-varying electromagnetic fields,of which the internal fluctuation can change collective electrical activities of neuronal networks.However,in the past there have been a few corresponding research papers on the influence of the electromagnetic induction among neurons on the collective dynamics of the complex system.Therefore,modeling each node by imposing electromagnetic radiation on the networks and investigating stochastic resonance in a hybrid network can extend the interest of the work to the understanding of these network dynamics.In this paper,we construct a small-world network consisting of excitatory neurons and inhibitory neurons,in which the effect of electromagnetic induction that is considered by using magnetic flow and the modulation of magnetic flow on membrane potential is described by using memristor coupling.According to our proposed network model,we investigate the effect of induced electric field generated by magnetic stimulation on the transition of bursting phase synchronization of neuronal system under electromagnetic radiation.It is shown that the intensity and frequency of the electric field can induce the transition of the network bursting phase synchronization.Moreover,we also analyze the effect of magnetic flow on the detection of weak signals and stochastic resonance by introducing a subthreshold pacemaker into a single cell of the network and we find that there is an optimal electromagnetic radiation intensity,where the phenomenon of stochastic resonance occurs and the degree of response to the weak signal is maximized.Simulation results show that the extension of the subthreshold pacemaker in the network also depends greatly on coupling strength.The presented results may have important implications for the theoretical study of magnetic stimulation technology,thus promoting further development of transcranial magnetic stimulation(TMS) as an effective means of treating certain neurological diseases.     

Neuronal devices: understanding neuronal cultures through percolation helps prepare for the next step

Cultures of dissociated neurons are an invaluable experimental tool in studying neuronal networks at an intermediate scale in an in vitro controlled physico-chemical environment. Moreover, current micro-fabrication techniques allow the design of a custom connectivity between subpopulations, which could make it possible to carry out computations with devices involving living cells. The quorum percolation (QP) model has been designed in the context of neurobiology to describe bursts of activity occurring in neuronal cultures from the point of view of collective phenomena rather than from a dynamical synchronization approach. Such a model is well suited to describe triggered activity in neuronal devices, and its generic character points at the necessity of heavily structured devices to go beyond collective bursting. We derive a continuous extension of the QP model, seen as information propagation on a non-metric directed graph, and discuss how its critical behavior might give insight on the connectivity of neuronal networks. The link with metric graphs, embedded in a two-dimensional space, is tackled by the introduction of a geometrical model based upon a random walk, where axon growth is constrained by obstacles such as walls and channels. This provides a starting point for the construction of neuronal devices in vitro capable of more complex behaviors. Lastly, we show how simulations of bursts with a dynamical adaptive integrate-and-fire model can be interpreted in terms of QP, confirming the robustness of this synchronized behavior.     

Hierarchical features of large-scale cortical connectivity

The analysis of complex networks has revealed patterns of organization in a variety of natural and artificial systems, including neuronal networks of the brain at multiple scales. In this paper, we describe a novel analysis of the large-scale connectivity between regions of the mammalian cerebral cortex, utilizing a set of hierarchical measurements proposed recently. We examine previously identified functional clusters of brain regions in macaque visual cortex and cat cortex and find significant differences between such clusters in terms of several hierarchical measures, revealing differences in how these clusters are embedded in the overall cortical architecture. For example, the ventral cluster of visual cortex maintains structurally more segregated, less divergent connections than the dorsal cluster, which may point to functionally different roles of their constituent brain regions.     

Multistability of clustered states in a globally inhibitory network

We study a network of m identical excitatory cells projecting excitatory synaptic connections onto a single inhibitory interneuron, which is reciprocally coupled to all excitatory cells through inhibitory synapses possessing short-term synaptic depression. We find that such a network with global inhibition possesses multiple stable activity patterns with distinct periods, characterized by the clustering of the excitatory cells into synchronized sub-populations. We prove the existence and stability of n-cluster solutions in a m-cell network. Using methods of geometric singular perturbation theory, we show that any n-cluster solution must satisfy a set of consistency conditions that can be geometrically derived. We then characterize the basin of attraction of each solution. Although frequency dependent depression is not necessary for multistability, we discuss how it plays a key role in determining network behavior. We find a functional relationship between the level of synaptic depression, the number of clusters and the interspike interval between neurons. This relationship is much less pronounced in the absence of depression. Implications for temporal coding and memory storage are discussed.     

Reproducible sequence generation in random neural ensembles

Little is known about the conditions that neural circuits have to satisfy to generate reproducible sequences. Evidently, the genetic code cannot control all the details of the complex circuits in the brain. In this Letter, we give the conditions on the connectivity degree that lead to reproducible and robust sequences in a neural population of randomly coupled excitatory and inhibitory neurons. In contrast to the traditional theoretical view we show that the sequences do not need to be learned. In the framework proposed here just the averaged characteristics of the random circuits have to be under genetic control. We found that rhythmic sequences can be generated if random networks are in the vicinity of an excitatory-inhibitory synaptic balance. Reproducible transient sequences, on the other hand, are found far from a synaptic balance.     

Temporal decorrelation of collective oscillations in neural networks with local inhibition and long-range excitation

We consider two neuronal networks coupled by long-range excitatory interactions. Oscillations in the gamma frequency band are generated within each network by local inhibition. When long-range excitation is weak, these oscillations phase lock with a phase shift dependent on the strength of local inhibition. Increasing the strength of long-range excitation induces a transition to chaos via period doubling or quasiperiodic scenarios. In the chaotic regime, oscillatory activity undergoes fast temporal decorrelation. The generality of these dynamical properties is assessed in firing-rate models as well as in large networks of conductance-based neurons.     

Changing motif distributions in complex networks by manipulating rich-club connections

The role of rich-club connectivity is significant in the structural property and functional behavior of complex networks. In this study, we find whether or not a very small portion of rich nodes connected to each other can strongly affect the frequency of occurrence of basic building blocks (motifs) within a heterogeneous network. Conversely whether a homogeneous network has a rich-club or not generally has no significant effect on its structure. These findings open the possibility to optimize and control the structure of complex networks by manipulating rich-club connections. Furthermore, based on the subgraph ratio profile, we develop a more rigorous approach to judge whether a network has a rich-club or not. The new method does not calculate how many links there are among rich nodes but depends on how the links among rich nodes can affect the overall structure as well as the function of a given network.