Probabilistic Models with Deep Neural Networks

Recent advances in statistical inference have significantly expanded the toolbox of probabilistic modeling. Historically, probabilistic modeling has been constrained to very restricted model classes, where exact or approximate probabilistic inference is feasible. However, developments in variational inference, a general form of approximate probabilistic inference that originated in statistical physics, have enabled probabilistic modeling to overcome these limitations: (i) Approximate probabilistic inference is now possible over a broad class of probabilistic models containing a large number of parameters, and (ii) scalable inference methods based on stochastic gradient descent and distributed computing engines allow probabilistic modeling to be applied to massive data sets. One important practical consequence of these advances is the possibility to include deep neural networks within probabilistic models, thereby capturing complex non-linear stochastic relationships between the random variables. These advances, in conjunction with the release of novel probabilistic modeling toolboxes, have greatly expanded the scope of applications of probabilistic models, and allowed the models to take advantage of the recent strides made by the deep learning community. In this paper, we provide an overview of the main concepts, methods, and tools needed to use deep neural networks within a probabilistic modeling framework.     

Superpositional Quantum Network Topologies

We introduce superposition-based quantum networks composed of (i) the classical perceptron model of multilayered, feedforward neural networks and (ii) the algebraic model of evolving reticular quantum structures as described in quantum gravity. The main feature of this model is moving from particular neural topologies to a quantum metastructure which embodies many differing topological patterns. Using quantum parallelism, training is possible on superpositions of different network topologies. As a result, not only classical transition functions, but also topology becomes a subject of training. The main feature of our model is that particular neural networks, with different topologies, are quantum states. We consider high-dimensionaldissipative quantum structures as candidates for implementation of the model.     

Topology and dynamics of attractor neural networks: The role of loopiness

We derive an exact representation of the topological effect on the dynamics of sequence processing neural networks within signal-to-noise analysis. A new network structure parameter, loopiness coefficient, is introduced to quantitatively study the loop effect on network dynamics. A large loopiness coefficient means a high probability of finding loops in the networks. We develop recursive equations for the overlap parameters of neural networks in terms of their loopiness. It was found that a large loopiness increases the correlation among the network states at different times and eventually reduces the performance of neural networks. The theory is applied to several network topological structures, including fully-connected, densely-connected random, densely-connected regular and densely-connected small-world, where encouraging results are obtained.     

Winsorization for Robust Bayesian Neural Networks

With the advent of big data and the popularity of black-box deep learning methods, it is imperative to address the robustness of neural networks to noise and outliers. We propose the use of Winsorization to recover model performances when the data may have outliers and other aberrant observations. We provide a comparative analysis of several probabilistic artificial intelligence and machine learning techniques for supervised learning case studies. Broadly, Winsorization is a versatile technique for accounting for outliers in data. However, different probabilistic machine learning techniques have different levels of efficiency when used on outlier-prone data, with or without Winsorization. We notice that Gaussian processes are extremely vulnerable to outliers, while deep learning techniques in general are more robust.     

Neural networks: A biased overview

An overview of recent activity in the field of neural networks is presented. The long-range aim of this research is to understand how the brain works. First some of the problems are stated and terminology defined; then an attempt is made to explain why physicists are drawn to the field, and their main potential contribution. In particular, in recent years some interesting models have been introduced by physicists. A small subset of these models is described, with particular emphasis on those that are analytically soluble. Finally a brief review of the history and recent developments of single- and multilayer perceptrons is given, bringing the situation up to date regarding the central immediate problem of the field: search for a learning algorithm that has an associated convergence theorem.     

Neural networks for prediction of acoustical properties of polyurethane foams

This paper presents a study of neural networks for prediction of acoustical properties of polyurethane foams. The proposed neural network model of the foam uses easily measured parameters such as frequency, airflow resistivity and density to predict multiple acoustical properties including the sound absorption coefficient and the surface impedance. Such a model is quite robust in the sense that it can be used to develop models for many different classes of materials with different sets of input and output parameters. The current neural network model of the foam is empirical and provides a useful complement to the existing analytical and numerical approaches.     

Neural network modeling of emotion

This article reviews the history and development of computational neural network modeling of cognitive and behavioral processes that involve emotion. The exposition starts with models of classical conditioning dating from the early 1970s. Then it proceeds toward models of interactions between emotion and attention. Then models of emotional influences on decision making are reviewed, including some speculative (not and not yet simulated) models of the evolution of decision rules. Through the late 1980s, the neural networks developed to model emotional processes were mainly embodiments of significant functional principles motivated by psychological data. In the last two decades, network models of these processes have become much more detailed in their incorporation of known physiological properties of specific brain regions, while preserving many of the psychological principles from the earlier models.Most network models of emotional processes so far have dealt with positive and negative emotion in general, rather than specific emotions such as fear, joy, sadness, and anger. But a later section of this article reviews a few models relevant to specific emotions: one family of models of auditory fear conditioning in rats, and one model of induced pleasure enhancing creativity in humans. Then models of emotional disorders are reviewed. The article concludes with philosophical statements about the essential contributions of emotion to intelligent behavior and the importance of quantitative theories and models to the interdisciplinary enterprise of understanding the interactions of emotion, cognition, and behavior.     

Object reconstruction in multilayer neural network based profilometry using grating structure comprising two regions with different spatial periods

Sampling of the Fourier transforms of fingerprints is studied with neural networks to detect regions useful for their classification. Ring-wedge detector (RWD) is modified and simulated to sample such regions. The output of the detector is propagated through a three-layer feedforward-backpropagation neural network for checking the classification performance. Modified detector's performance is also compared with that of RWD. It has been found that fingerprints scanned at 500 dpi resolution and cropped to a size of 200×200 contain useful information for their classification in a band of width 20 pixels with inner radius approx. 60 pixels.     

Neural networks and bounded rationality

Traditionally the emphasis in neural network research has been on improving their performance as a means of pattern recognition. Here we take an alternative approach and explore the remarkable similarity between the under-performance of neural networks trained to behave optimally in economic situations and observed human performance in the laboratory under similar circumstances. In particular, we show that neural networks are consistent with observed laboratory play in two very important senses. Firstly, they select a rule for behavior which appears very similar to that used by laboratory subjects. Secondly, using this rule they perform optimally only approximately 60% of the time.     

Does money matter in inflation forecasting?

This paper provides the most fully comprehensive evidence to date on whether or not monetary aggregates are valuable for forecasting US inflation in the early to mid 2000s. We explore a wide range of different definitions of money, including different methods of aggregation and different collections of included monetary assets. In our forecasting experiment we use two nonlinear techniques, namely, recurrent neural networks and kernel recursive least squares regression—techniques that are new to macroeconomics. Recurrent neural networks operate with potentially unbounded input memory, while the kernel regression technique is a finite memory predictor. The two methodologies compete to find the best fitting US inflation forecasting models and are then compared to forecasts from a naïve random walk model. The best models were nonlinear autoregressive models based on kernel methods. Our findings do not provide much support for the usefulness of monetary aggregates in forecasting inflation. Beyond its economic findings, our study is in the tradition of physicists’ long-standing interest in the interconnections among statistical mechanics, neural networks, and related nonparametric statistical methods, and suggests potential avenues of extension for such studies.     

人工神经网络的光学实现

本文讨论了光学神经网络的优势及其所面临的主要问题,简述了它近十年来的发展．光互连是光学实现神经网络最具吸引力的技术,光电混合集成的灵巧象素器件又融合了电子器件灵活、可编程、易控制的特点．二者的结合是当前光学神经网络发展的主要趋势,它们为神经网络的进一步发展和应用,为超大规模神经网络的实现和应用提供了一种很有前途的方案．     

On the parallel dynamics of theQ-state Potts andQ-Ising neural networks

Using a probabilistic approach, the parallel dynamics of theQ-state Potts andQ-Ising neural networks are studied at zero and at nonzero temperatures. Evolution equations are derived for the first time step and arbitraryQ. These formulas constitute recursion relations for the exact parallel dynamics of the extremely diluted asymmetric versions of these networks. An explicit analysis, including dynamical capacity-temperature diagrams and the temperature dependence of the overlap, is carried out forQ=3. Both types of models are compared.On leave of absence from the Laboratory of Theoretical Physics, Joint Institute for Nuclear Research, Dubna 141980, Russia.     

基于高斯过程的混沌时间序列单步与多步预测

针对混沌时间序列单步和多步预测,提出基于复合协方差函数的高斯过程 (GP)模型方法.GP模型的确立由协方差函数决定,通过对训练数据集的学习,在证据最大化框架内,利用矩阵运算和优化算法自适应地确定协方差函数和均值函数中的超参数.GP模型与神经网络、模糊模型相比,其可调整参数很少.将不同复合协方差函数的GP模型应用在混沌时间序列单步及多步提前预测中,并与单一协方差函数的GP、支持向量机、最小二乘支持向量机、径向基函数神经网络等方法进行了比较.仿真结果表明,基于不同复合协方差函数的GP方法能精确地预测混沌时间序 关键词： 高斯过程 混沌时间序列 预测 模型比较     

Analysis on global exponential robust stability of reaction-diffusion neural networks with S-type distributed delays

To avoid the unstable phenomena caused by time delays and perturbations, we investigate the sufficient conditions to ensure the global exponential robust stability with a convergence rate for the reaction-diffusion neural networks with S-type distributed delays. Because S-type distributed delays lead to some difficulty, we also introduce a new generalized Halanay inequality and a novel method-system-approximation method into the qualitative research of neural networks. Moreover, the sufficient criteria provided here, which are rather accessible and feasible, have wider adaptive range.     

基于PCA和PNN的高甘油三脂血清荧光光谱识别

基于主成分分析和概率神经网络,提出了一种有效识别高甘油三脂血清荧光光谱的新方法.研究测量了正常和高甘油三脂血清在290 nm和350 nm激发光下产生的荧光光谱,并分别以3种采样间隔(1 nm、2 nm和5 nm)提取荧光强度作为样品的初始特征;利用主成分分析法对初始特征进行分析,以累积可信度大于95%的主成分作为样品特征;构建了4层概率神经网络,并分析了平滑系数和采样间隔对识别效果的影响.实验结果表明,当采样间隔采用5 nm,平滑系数位于0.26~0.92区间时,正常和高甘油三脂血清样品的识别率可达到95%和100%.     

State generators and complex neural memories

The mechanism of self-indexing for feedback neural networks that generates memories from short subsequences is generalized so that a single bit together with an appropriate update order suffices for each memory. This mechanism explains how stimulating an appropriate neuron can recall a memory. Although information is distributed in this model, yet our self-indexing mechanism makes it appear localized. Also a new complex valued neuron model is presented to generalize McCulloch-Pitts neurons.     

Learning and forgetting on asymmetric,diluted neural networks

It is possible to construct diluted asymmetric models of neural networks for which the dynamics can be calculated exactly. We test several learning schemes, in particular, models for which the values of the synapses remain bounded and depend on the history. Our analytical results on the relative efficiencies of the various learning schemes are qualitatively similar to the corresponding ones obtained numerically on fully connected symmetric networks.     

Artificial Neural Networks for Resonant Frequency Calculation of Rectangular Microstrip Antennas with Thin and Thick Substrates

Neural models based on multilayered perceptrons for computing the resonant frequency of rectangular microstrip antennas with thin and thick substrates are presented. Eleven learning algorithms, Levenberg-Marquardt, conjugate gradient of Fletcher-Reeves, conjugate gradient of Powell-Beale, bayesian regularization, scaled conjugate gradient, Broyden-Fletcher-Goldfarb-Shanno, resilient backpropagation, conjugate of Polak-Ribiére, backpropagation with adaptive learning rate, one-step secant, and backpropagation with momentum, are used to train the multilayered perceptrons. The resonant frequency results obtained by using neural models are in very good agreement with the experimental results available in the literature. When the performances of neural models are compared with each other, the best result is obtained from the multilayered perceptrons trained by Levenberg-Marquardt algorithm.     

Control of chaotic systems using targeting by extended control regions method

In this work, the previously proposed extended control regions (ECR) algorithm for targeting is improved by using individual neural networks for each activation region. The improved version, which exploits the short time predictability of the chaotic system more efficiently, gives better performance with respect to training time and average reaching time while maintaining the advantages of the previous method. Moreover, the simulation results revealed that the meaningful number of activation regions of the controller using improved ECR is nearly linearly related with the prediction horizon of the chaotic system to be targeted, which can be used as a criterion for choosing the number of activation region.     

基于概率神经网络的荧光光谱法识别高甘油三脂血清

针对因正常和高甘油三脂血清荧光光谱混叠致使其识别率不高的问题,首先测量了正常和高甘油三脂血清样品在260,370,580 nm激发光下产生的荧光光谱,并以荧光强度作为样品的初始特征;其次,采用主成分分析法对初始特征进行分析和提取,获得了样品的特征向量;最后,构建了4层概率神经网络,并对正常和高甘油三脂血清样品进行了识别。对采用不同荧光光谱进行血清样品识别的效果进行了对比,结果表明,采用260 nm和370 nm荧光光谱识别正常和高甘油三脂血清的正确率分别为100%和95%。实验验证了研究方案的可行性和效果,对发展荧光光谱技术在识别高甘油三脂血症中的应用具有重要的意义和价值。