A biologically constrained learning mechanism in networks of formal neurons

A new learning mechanism is proposed for networks of formal neurons analogous to Ising spin systems; it brings such models substantially closer to biological data in three respects: first, the learning procedure is applied initially to a network with random connections (which may be similar to a spin-glass system), instead of starting from a system void of any knowledge (as in the Hopfield model); second, the resultant couplings are not symmetrical; third, patterns can be stored without changing the sign of the coupling coefficients. It is shown that the storage capacity of such networks is similar to that of the Hopfield network, and that it is not significantly affected by the restriction of keeping the couplings' signs constant throughout the learning phase. Although this approach does not claim to model the central nervous system, it provides new insight on a frontier area between statistical physics, artificial intelligence, and neurobiology.     

Nonlinear neural networks. II. Information processing

Information processing in nonlinear neural networks with a finite numberq of stored patterns is studied. Each network is characterized completely by its synaptic kernelQ. At low temperatures, the nonlinearity typically results in 2^q–2–q metastable, pure states in addition to theq retrieval states that are associated with theq stored patterns. These spurious states start appearing at a temperature , which depends onq. We give sufficient conditions to guarantee that the retrieval states bifurcate first at a critical temperatureT _c and that /T _c 0 asq. Hence, there is a large temperature range whereonly the retrieval states and certain symmetric mixtures thereof exist. The latter are unstable, as they appear atT _c . For clipped synapses, the bifurcation and stability structure is analyzed in detail and shown to approach that of the (linear) Hopfield model asq. We also investigate memories that forget and indicate how forgetfulness can be explained in terms of the eigenvalue spectrum of the synaptic kernelQ.     

Finite-state neural networks. A step toward the simulation of very large systems

Neural networks composed of neurons withQ _N states and synapses withQ states are studied analytically and numerically. Analytically it is shown that these finite-state networks are much more efficient at information storage than networks with continuous synapses. In order to take the utmost advantage of networks with finite-state elements, a multineuron and multisynapse coding scheme is introduced which allows the simulation of networks having 1.0×10⁹ couplings at a speed of 7.1×10⁹ coupling evaluations per second on asingle processor of the Cray-YMP. A local learning algorithm is also introduced which allows for the efficient training of large networks with finite-state elements.     

Topology and dynamics of attractor neural networks: The role of loopiness

We derive an exact representation of the topological effect on the dynamics of sequence processing neural networks within signal-to-noise analysis. A new network structure parameter, loopiness coefficient, is introduced to quantitatively study the loop effect on network dynamics. A large loopiness coefficient means a high probability of finding loops in the networks. We develop recursive equations for the overlap parameters of neural networks in terms of their loopiness. It was found that a large loopiness increases the correlation among the network states at different times and eventually reduces the performance of neural networks. The theory is applied to several network topological structures, including fully-connected, densely-connected random, densely-connected regular and densely-connected small-world, where encouraging results are obtained.     

On training feedforward neural networks

A new algorithm that mapsn-dimensional binary vectors intom-dimensional binary vectors using 3-layered feedforward neural networks is described. The algorithm is based on a representation of the mapping in terms of the corners of then-dimensional signal cube. The weights to the hidden layer are found by a corner classification algorithm and the weights to the output layer are all equal to 1. Two corner classification algorithms are described. The first one is based on the perceptron algorithm and it performs generalization. The computing power of this algorithm may be gauged from the example that the exclusive-Or problem that requires several thousand iterative steps using the backpropagation algorithm was solved in 8 steps. Another corner classification algorithm presented in this paper does not require any computations to find the weights. However, in its basic form it does not perform generalization.     

Renormalized theory of the time-dependent pair distribution function. I. General formulation

A classical renormalized theory of a time-dependent pair-distribution function (TDPDF), previously introduced by Oppenheim and Bloom, is presented. An equation of motion for the TDPDF is derived in which the memory function of the system appears. This is then split into a part which contains only static correlation functions and a part which describes the dynamics. The mean field approximation is discussed in some detail and contact is made witn the theory of Oppenheim and Bloom.Work supported in part by a National Research Council of Canada operating grant.     

Decoherence effects in Bose–Einstein condensate interferometry I. General theory

The present paper outlines a basic theoretical treatment of decoherence and dephasing effects in interferometry based on single component Bose–Einstein condensates in double potential wells, where two condensate modes may be involved. Results for both two mode condensates and the simpler single mode condensate case are presented. The approach involves a hybrid phase space distribution functional method where the condensate modes are described via a truncated Wigner representation, whilst the basically unoccupied non-condensate modes are described via a positive P representation. The Hamiltonian for the system is described in terms of quantum field operators for the condensate and non-condensate modes. The functional Fokker–Planck equation for the double phase space distribution functional is derived. Equivalent Ito stochastic equations for the condensate and non-condensate fields that replace the field operators are obtained, and stochastic averages of products of these fields give the quantum correlation functions that can be used to interpret interferometry experiments. The stochastic field equations are the sum of a deterministic term obtained from the drift vector in the functional Fokker–Planck equation, and a noise field whose stochastic properties are determined from the diffusion matrix in the functional Fokker–Planck equation. The stochastic properties of the noise field terms are similar to those for Gaussian–Markov processes in that the stochastic averages of odd numbers of noise fields are zero and those for even numbers of noise field terms are the sums of products of stochastic averages associated with pairs of noise fields. However each pair is represented by an element of the diffusion matrix rather than products of the noise fields themselves, as in the case of Gaussian–Markov processes. The treatment starts from a generalised mean field theory for two condensate modes, where generalised coupled Gross–Pitaevskii equations are obtained for the modes and matrix mechanics equations are derived for the amplitudes describing possible fragmentations of the condensate between the two modes. These self-consistent sets of equations are derived via the Dirac–Frenkel variational principle. Numerical studies for interferometry experiments would involve using the solutions from the generalised mean field theory in calculations for the stochastic fields from the Ito stochastic field equations.     

Sizing of spheroidal and cylindrical particles in a binary mixture by measurement of scattered light intensity: application of neural networks

The problem on the retrieval of sizes of an individual optically soft particle taken from binary mixtures of either oblate and prolate spheroids or cylinders and oblate spheroids is considered. It is based on multiangle scattered light intensity data. The multilevel neural networks method with a linear activation function and the method of the discrimination functions are used. Neural networks to retrieve characteristics of cylinders, oblate and prolate spheroids are designed. The errors in retrieved particle characteristics are investigated for the radius of an equivolume sphere in the range of 0.3-, shape parameter of spheroidal and cylindrical particles from -0.5 to 0.5 and 0 to 0.5, respectively.     

Fluctuations in a fluid under a stationary heat flux. I. General theory

We present the basic formulas for a unified treatment of the correlation functions of the hydrodynamic variables in a fluid between two horizontal plates which is exposed to a stationary heat flux in the presence of a gravity field (Rayleigh-Bénard system). Our analysis is based on fluctuating hydrodynamics. In this paper (I) we show that in the nonequilibrium stationary state the hydrodynamic fluctuations evolve on slow and fast time scales that are widely separated. A time scale perturbation theory is used to diagonalize the hydrodynamic operator partially. This enables us to derive the eigenvalue equations for the nonequilibrium hydrodynamic modes. Therein we take into account the variation of the macroscopic quantities with position. The correlation functions are formally expressed in terms of the nonequilibrium modes. In paper II the slow hydrodynamic modes (viscous and viscoheat modes) will be determined explicitly for ideal heat-conducting plates with stick boundary conditions and used to compute the slow part of the correlation functions; in paper III the fast hydrodynamic modes (sound modes) will be explicitly determined for stick boundary conditions and used to compute the fast part of the correlation functions. In these papers we will also compute the shape and intensity of the lines measured in light scattering experiments.     

Firing activity of complex space-clamped FitzHugh-Nagumo neural networks

We investigate how firing activity of complex neural networks depends on the random long-range connections and coupling strength. Network elements are described by excitable space-clamped FitzHugh-Nagumo (SCFHN) neurons with the values of parameters at which no firing activity occurs. It is found that for a given appropriate coupling strength C, there exists a critical fraction of random connections (or randomness) p^*, such that if p > p^* the firing neurons, which are absent in the nearest-neighbor network, occur. The firing activity becomes more frequent as randomness p is further increased. On the other hand, when the p is smaller, there are no active neurons in network, no matter what the value of C is. For a given larger p, there exist optimal coupling strength levels, where firing activity reaches its maximum. To the best of our knowledge, this is a novel mechanism for the emergence of firing activity in neurons.     

Economic small-world behavior in weighted networks

The small-world phenomenon has been already the subject of a huge variety of papers, showing its appeareance in a variety of systems. However, some big holes still remain to be filled, as the commonly adopted mathematical formulation is valid only for topological networks. In this paper we propose a generalization of the theory of small worlds based on two leading concepts, efficiency and cost, and valid also for weighted networks. Efficiency measures how well information propagates over the network, and cost measures how expensive it is to build a network. The combination of these factors leads us to introduce the concept of economic small worlds, that formalizes the idea of networks that are “cheap” to build, and nevertheless efficient in propagating information, both at global and local scale. In this way we provide an adequate tool to quantitatively analyze the behaviour of complex networks in the real world. Various complex systems are studied, ranging from the realm of neural networks, to social sciences, to communication and transportation networks. In each case, economic small worlds are found. Moreover, using the economic small-world framework, the construction principles of these networks can be quantitatively analyzed and compared, giving good insights on how efficiency and economy principles combine up to shape all these systems. Received 6 November 2002 / Received in final form 24 January 2003 Published online 1st April 2003 RID="a" ID="a"e-mail: latora@ct.infn.it     

On learning and energy-entropy dependence in recurrent and nonrecurrent signed networks

Learning of patterns by neural networks obeying general rules of sensory transduction and of converting membrane potentials to spiking frequencies is considered. Any finite number of cellsA can sample a pattern playing on any finite number of cells without causing irrevocable sampling bias ifA = orA =. Total energy transfer from inputs ofA to outputs of depends on the entropy of the input distribution. Pattern completion on recall trials can occur without destroying perfect memory even ifA = by choosing the signal thresholds sufficiently large. The mathematical results are global limit and oscillation theorems for a class of nonlinear functional-differential systems.The preparation of this work was supported in part by the National Science Foundation (GP 9003), the Office of Naval Research (N00014-67-A-024-OQ16), and the A.P. Sloan Foundation.     

Statistical physics of inference: thresholds and algorithms

Many questions of fundamental interest in today's science can be formulated as inference problems: some partial, or noisy, observations are performed over a set of variables and the goal is to recover, or infer, the values of the variables based on the indirect information contained in the measurements. For such problems, the central scientific questions are: Under what conditions is the information contained in the measurements sufficient for a satisfactory inference to be possible? What are the most efficient algorithms for this task? A growing body of work has shown that often we can understand and locate these fundamental barriers by thinking of them as phase transitions in the sense of statistical physics. Moreover, it turned out that we can use the gained physical insight to develop new promising algorithms. The connection between inference and statistical physics is currently witnessing an impressive renaissance and we review here the current state-of-the-art, with a pedagogical focus on the Ising model which, formulated as an inference problem, we call the planted spin glass. In terms of applications we review two classes of problems: (i) inference of clusters on graphs and networks, with community detection as a special case and (ii) estimating a signal from its noisy linear measurements, with compressed sensing as a case of sparse estimation. Our goal is to provide a pedagogical review for researchers in physics and other fields interested in this fascinating topic.     

Efficient Hopfield pattern recognition on a scale-free neural network

Neural networks are supposed to recognise blurred images (or patterns) of N pixels (bits) each. Application of the network to an initial blurred version of one of P pre-assigned patterns should converge to the correct pattern. In the “standard" Hopfield model, the N “neurons” are connected to each other via N² bonds which contain the information on the stored patterns. Thus computer time and memory in general grow with N². The Hebb rule assigns synaptic coupling strengths proportional to the overlap of the stored patterns at the two coupled neurons. Here we simulate the Hopfield model on the Barabási-Albert scale-free network, in which each newly added neuron is connected to only m other neurons, and at the end the number of neurons with q neighbours decays as 1/q ³. Although the quality of retrieval decreases for small m, we find good associative memory for 1 ≪ m ≪ N. Hence, these networks gain a factor N/m ≫ 1 in the computer memory and time. Received 12 January 2003 Published online 11 April 2003 RID="a" ID="a"e-mail: stauffer@thp.uni-koeln.de     

Random quantum magnets with broad disorder distribution

We study the critical behavior of Ising quantum magnets with broadly distributed random couplings (J), such that P(ln J) ∼ | ln J|^{-1 - α}, α > 1, for large | ln J| (Lévy flight statistics). For sufficiently broad distributions, α < , the critical behavior is controlled by a line of fixed points, where the critical exponents vary with the Lévy index, α. In one dimension, with = 2, we obtained several exact results through a mapping to surviving Riemann walks. In two dimensions the varying critical exponents have been calculated by a numerical implementation of the Ma-Dasgupta-Hu renormalization group method leading to ≈ 4.5. Thus in the region 2 < α < , where the central limit theorem holds for | ln J| the broadness of the distribution is relevant for the 2d quantum Ising model. Received 6 December 2000 and Received in final form 22 January 2001     

Statistical physics and practical training of soft-committee machines

Equilibrium states of large layered neural networks with differentiable activation function and a single, linear output unit are investigated using the replica formalism. The quenched free energy of a student network with a very large number of hidden units learning a rule of perfectly matching complexity is calculated analytically. The system undergoes a first order phase transition from unspecialized to specialized student configurations at a critical size of the training set. Computer simulations of learning by stochastic gradient descent from a fixed training set demonstrate that the equilibrium results describe quantitatively the plateau states which occur in practical training procedures at sufficiently small but finite learning rates. Received 16 December 1998