Relationship between Age and Value of Information for a Noisy Ornstein–Uhlenbeck Process

The age of information (AoI) has been widely used to quantify the information freshness in real-time status update systems. As the AoI is independent of the inherent property of the source data and the context, we introduce a mutual information-based value of information (VoI) framework for hidden Markov models. In this paper, we investigate the VoI and its relationship to the AoI for a noisy Ornstein–Uhlenbeck (OU) process. We explore the effects of correlation and noise on their relationship, and find logarithmic, exponential and linear dependencies between the two in three different regimes. This gives the formal justification for the selection of non-linear AoI functions previously reported in other works. Moreover, we study the statistical properties of the VoI in the example of a queue model, deriving its distribution functions and moments. The lower and upper bounds of the average VoI are also analysed, which can be used for the design and optimisation of freshness-aware networks. Numerical results are presented and further show that, compared with the traditional linear age and some basic non-linear age functions, the proposed VoI framework is more general and suitable for various contexts.     

Neural Dynamics under Active Inference: Plausibility and Efficiency of Information Processing

Active inference is a normative framework for explaining behaviour under the free energy principle—a theory of self-organisation originating in neuroscience. It specifies neuronal dynamics for state-estimation in terms of a descent on (variational) free energy—a measure of the fit between an internal (generative) model and sensory observations. The free energy gradient is a prediction error—plausibly encoded in the average membrane potentials of neuronal populations. Conversely, the expected probability of a state can be expressed in terms of neuronal firing rates. We show that this is consistent with current models of neuronal dynamics and establish face validity by synthesising plausible electrophysiological responses. We then show that these neuronal dynamics approximate natural gradient descent, a well-known optimisation algorithm from information geometry that follows the steepest descent of the objective in information space. We compare the information length of belief updating in both schemes, a measure of the distance travelled in information space that has a direct interpretation in terms of metabolic cost. We show that neural dynamics under active inference are metabolically efficient and suggest that neural representations in biological agents may evolve by approximating steepest descent in information space towards the point of optimal inference.     

Neural Estimator of Information for Time-Series Data with Dependency

Novel approaches to estimate information measures using neural networks are well-celebrated in recent years both in the information theory and machine learning communities. These neural-based estimators are shown to converge to the true values when estimating mutual information and conditional mutual information using independent samples. However, if the samples in the dataset are not independent, the consistency of these estimators requires further investigation. This is of particular interest for a more complex measure such as the directed information, which is pivotal in characterizing causality and is meaningful over time-dependent variables. The extension of the convergence proof for such cases is not trivial and demands further assumptions on the data. In this paper, we show that our neural estimator for conditional mutual information is consistent when the dataset is generated with samples of a stationary and ergodic source. In other words, we show that our information estimator using neural networks converges asymptotically to the true value with probability one. Besides universal functional approximation of neural networks, a core lemma to show the convergence is Birkhoff’s ergodic theorem. Additionally, we use the technique to estimate directed information and demonstrate the effectiveness of our approach in simulations.     

Data-Driven Corrections of Partial Lotka–Volterra Models

In many applications of interacting systems, we are only interested in the dynamic behavior of a subset of all possible active species. For example, this is true in combustion models (many transient chemical species are not of interest in a given reaction) and in epidemiological models (only certain subpopulations are consequential). Thus, it is common to use greatly reduced or partial models in which only the interactions among the species of interest are known. In this work, we explore the use of an embedded, sparse, and data-driven discrepancy operator to augment these partial interaction models. Preliminary results show that the model error caused by severe reductions—e.g., elimination of hundreds of terms—can be captured with sparse operators, built with only a small fraction of that number. The operator is embedded within the differential equations of the model, which allows the action of the operator to be interpretable. Moreover, it is constrained by available physical information and calibrated over many scenarios. These qualities of the discrepancy model—interpretability, physical consistency, and robustness to different scenarios—are intended to support reliable predictions under extrapolative conditions.     

Do Co-Worker Networks Increase or Decrease Productivity Differences?

Do labor mobility and co-worker networks contribute to convergence or divergence between regions? Based on the previous literature, labor mobility contributes to knowledge transfer between firms. Therefore, mobility may contribute to decreasing productivity differences, while limited mobility sustains higher differences. The effect of co-worker networks, however, can be two-fold in this process; they transmit information about potential jobs, which may enhance the mobility of workers—even between regions—and this enhanced mobility may contribute to levelling of differences. However, if mobility between regions involves movement costs, co-worker networks may concentrate locally—possibly contributing to the persistence of regional differences. In this paper, we build an agent-based model of labor mobility across firms and regions with knowledge spillovers that reflects key empirical observations on labor markets. We analyze the impact of network information provided about potential employers in this model and find that it contributes to increasing inter-regional mobility, and subsequently, to decreasing regional differences. We also find that both the density of coworker networks, as well as their regional concentrations, decrease if network information is available.     

Augmenting Paraphrase Generation with Syntax Information Using Graph Convolutional Networks

Paraphrase generation is an important yet challenging task in natural language processing. Neural network-based approaches have achieved remarkable success in sequence-to-sequence learning. Previous paraphrase generation work generally ignores syntactic information regardless of its availability, with the assumption that neural nets could learn such linguistic knowledge implicitly. In this work, we make an endeavor to probe into the efficacy of explicit syntactic information for the task of paraphrase generation. Syntactic information can appear in the form of dependency trees, which could be easily acquired from off-the-shelf syntactic parsers. Such tree structures could be conveniently encoded via graph convolutional networks to obtain more meaningful sentence representations, which could improve generated paraphrases. Through extensive experiments on four paraphrase datasets with different sizes and genres, we demonstrate the utility of syntactic information in neural paraphrase generation under the framework of sequence-to-sequence modeling. Specifically, our graph convolutional network-enhanced models consistently outperform their syntax-agnostic counterparts using multiple evaluation metrics.     

Causal Geometry

Information geometry has offered a way to formally study the efficacy of scientific models by quantifying the impact of model parameters on the predicted effects. However, there has been little formal investigation of causation in this framework, despite causal models being a fundamental part of science and explanation. Here, we introduce causal geometry, which formalizes not only how outcomes are impacted by parameters, but also how the parameters of a model can be intervened upon. Therefore, we introduce a geometric version of “effective information”—a known measure of the informativeness of a causal relationship. We show that it is given by the matching between the space of effects and the space of interventions, in the form of their geometric congruence. Therefore, given a fixed intervention capability, an effective causal model is one that is well matched to those interventions. This is a consequence of “causal emergence,” wherein macroscopic causal relationships may carry more information than “fundamental” microscopic ones. We thus argue that a coarse-grained model may, paradoxically, be more informative than the microscopic one, especially when it better matches the scale of accessible interventions—as we illustrate on toy examples.     

Minimal Developmental Computation: A Causal Network Approach to Understand Morphogenetic Pattern Formation

What information-processing strategies and general principles are sufficient to enable self-organized morphogenesis in embryogenesis and regeneration? We designed and analyzed a minimal model of self-scaling axial patterning consisting of a cellular network that develops activity patterns within implicitly set bounds. The properties of the cells are determined by internal ‘genetic’ networks with an architecture shared across all cells. We used machine-learning to identify models that enable this virtual mini-embryo to pattern a typical axial gradient while simultaneously sensing the set boundaries within which to develop it from homogeneous conditions—a setting that captures the essence of early embryogenesis. Interestingly, the model revealed several features (such as planar polarity and regenerative re-scaling capacity) for which it was not directly selected, showing how these common biological design principles can emerge as a consequence of simple patterning modes. A novel “causal network” analysis of the best model furthermore revealed that the originally symmetric model dynamically integrates into intercellular causal networks characterized by broken-symmetry, long-range influence and modularity, offering an interpretable macroscale-circuit-based explanation for phenotypic patterning. This work shows how computation could occur in biological development and how machine learning approaches can generate hypotheses and deepen our understanding of how featureless tissues might develop sophisticated patterns—an essential step towards predictive control of morphogenesis in regenerative medicine or synthetic bioengineering contexts. The tools developed here also have the potential to benefit machine learning via new forms of backpropagation and by leveraging the novel distributed self-representation mechanisms to improve robustness and generalization.     

Perceptron-like computation based on biologically-inspired neurons with heterosynaptic mechanisms

Perceptrons are one of the fundamental paradigms in artificial neural networks and a keyprocessing scheme in supervised classification tasks. However, the algorithm they provideis given in terms of unrealistically simple processing units and connections andtherefore, its implementation in real neural networks is hard to be fulfilled. In thiswork, we present a neural circuit able to perform perceptron’s computation based onrealistic models of neurons and synapses. The model uses Wang-Buzsáki neurons withcoupling provided by axodendritic and axoaxonic synapses (heterosynapsis). The maincharacteristics of the feedforward perceptron operation are conserved, which allows tocombine both approaches: whereas the classical artificial system can be used to learn aparticular problem, its solution can be directly implemented in this neural circuit. As aresult, we propose a biologically-inspired system able to work appropriately in a widerange of frequencies and system parameters, while keeping robust to noise and error.     

New Interfaces and Approaches to Machine Learning When Classifying Gestures within Music

Interactive music uses wearable sensors (i.e., gestural interfaces—GIs) and biometric datasets to reinvent traditional human–computer interaction and enhance music composition. In recent years, machine learning (ML) has been important for the artform. This is because ML helps process complex biometric datasets from GIs when predicting musical actions (termed performance gestures). ML allows musicians to create novel interactions with digital media. Wekinator is a popular ML software amongst artists, allowing users to train models through demonstration. It is built on the Waikato Environment for Knowledge Analysis (WEKA) framework, which is used to build supervised predictive models. Previous research has used biometric data from GIs to train specific ML models. However, previous research does not inform optimum ML model choice, within music, or compare model performance. Wekinator offers several ML models. Thus, we used Wekinator and the Myo armband GI and study three performance gestures for piano practice to solve this problem. Using these, we trained all models in Wekinator and investigated their accuracy, how gesture representation affects model accuracy and if optimisation can arise. Results show that neural networks are the strongest continuous classifiers, mapping behaviour differs amongst continuous models, optimisation can occur and gesture representation disparately affects model mapping behaviour; impacting music practice.     

Stochastic Chaos and Markov Blankets

In this treatment of random dynamical systems, we consider the existence—and identification—of conditional independencies at nonequilibrium steady-state. These independencies underwrite a particular partition of states, in which internal states are statistically secluded from external states by blanket states. The existence of such partitions has interesting implications for the information geometry of internal states. In brief, this geometry can be read as a physics of sentience, where internal states look as if they are inferring external states. However, the existence of such partitions—and the functional form of the underlying densities—have yet to be established. Here, using the Lorenz system as the basis of stochastic chaos, we leverage the Helmholtz decomposition—and polynomial expansions—to parameterise the steady-state density in terms of surprisal or self-information. We then show how Markov blankets can be identified—using the accompanying Hessian—to characterise the coupling between internal and external states in terms of a generalised synchrony or synchronisation of chaos. We conclude by suggesting that this kind of synchronisation may provide a mathematical basis for an elemental form of (autonomous or active) sentience in biology.     

External Stimuli on Neural Networks: Analytical and Numerical Approaches

Based on the behavior of living beings, which react mostly to external stimuli, we introduce a neural-network model that uses external patterns as a fundamental tool for the process of recognition. In this proposal, external stimuli appear as an additional field, and basins of attraction, representing memories, arise in accordance with this new field. This is in contrast to the more-common attractor neural networks, where memories are attractors inside well-defined basins of attraction. We show that this procedure considerably increases the storage capabilities of the neural network; this property is illustrated by the standard Hopfield model, which reveals that the recognition capacity of our model may be enlarged, typically, by a factor 102. The primary challenge here consists in calibrating the influence of the external stimulus, in order to attenuate the noise generated by memories that are not correlated with the external pattern. The system is analyzed primarily through numerical simulations. However, since there is the possibility of performing analytical calculations for the Hopfield model, the agreement between these two approaches can be tested—matching results are indicated in some cases. We also show that the present proposal exhibits a crucial attribute of living beings, which concerns their ability to react promptly to changes in the external environment. Additionally, we illustrate that this new approach may significantly enlarge the recognition capacity of neural networks in various situations; with correlated and non-correlated memories, as well as diluted, symmetric, or asymmetric interactions (synapses). This demonstrates that it can be implemented easily on a wide diversity of models.     

Why Dilated Convolutional Neural Networks: A Proof of Their Optimality

One of the most effective image processing techniques is the use of convolutional neural networks that use convolutional layers. In each such layer, the value of the layer’s output signal at each point is a combination of the layer’s input signals corresponding to several neighboring points. To improve the accuracy, researchers have developed a version of this technique, in which only data from some of the neighboring points is processed. It turns out that the most efficient case—called dilated convolution—is when we select the neighboring points whose differences in both coordinates are divisible by some constant ℓ. In this paper, we explain this empirical efficiency by proving that for all reasonable optimality criteria, dilated convolution is indeed better than possible alternatives.     

Switching and Swapping of Quantum Information: Entropy and Entanglement Level

Information switching and swapping seem to be fundamental elements of quantum communication protocols. Another crucial issue is the presence of entanglement and its level in inspected quantum systems. In this article, a formal definition of the operation of the swapping local quantum information and its existence proof, together with some elementary properties analysed through the prism of the concept of the entropy, are presented. As an example of the local information swapping usage, we demonstrate a certain realisation of the quantum switch. Entanglement levels, during the work of the switch, are calculated with the Negativity measure and a separability criterion based on the von Neumann entropy, spectral decomposition and Schmidt decomposition. Results of numerical experiments, during which the entanglement levels are estimated for systems under consideration with and without distortions, are presented. The noise is generated by the Dzyaloshinskii-Moriya interaction and the intrinsic decoherence is modelled by the Milburn equation. This work contains a switch realisation in a circuit form—built out of elementary quantum gates, and a scheme of the circuit which estimates levels of entanglement during the switch’s operating.     

Brain-like large scale cognitive networks and dynamics

A new approach to the study of the brain and its functions known as Human Connectomics has been recently established. Starting from magnetic resonance images (MRI) of brain scans, it is possible to identify the fibers that link brain areas and to build an adjacency matrix that connects these areas, thus creating the brain connectome. The topology of these networks provides a lot of information about the organizational structure of the brain (both structural and functional). Nevertheless this knowledge is rarely used to investigate the possible emerging brain dynamics linked to cognitive functions. In this work, we implement finite state models on neural networks to display the outcoming brain dynamics, using different types of networks, which correspond to diverse segmentation methods and brain atlases. From the simulations, we observe that the behavior of these systems is completely different from random and/or artificially generated networks. The emergence of stable structures, which might correspond to brain cognitive circuits, has also been detected.     

Taming the Chaos in Neural Network Time Series Predictions

Machine learning methods, such as Long Short-Term Memory (LSTM) neural networks can predict real-life time series data. Here, we present a new approach to predict time series data combining interpolation techniques, randomly parameterized LSTM neural networks and measures of signal complexity, which we will refer to as complexity measures throughout this research. First, we interpolate the time series data under study. Next, we predict the time series data using an ensemble of randomly parameterized LSTM neural networks. Finally, we filter the ensemble prediction based on the original data complexity to improve the predictability, i.e., we keep only predictions with a complexity close to that of the training data. We test the proposed approach on five different univariate time series data. We use linear and fractal interpolation to increase the amount of data. We tested five different complexity measures for the ensemble filters for time series data, i.e., the Hurst exponent, Shannon’s entropy, Fisher’s information, SVD entropy, and the spectrum of Lyapunov exponents. Our results show that the interpolated predictions consistently outperformed the non-interpolated ones. The best ensemble predictions always beat a baseline prediction based on a neural network with only a single hidden LSTM, gated recurrent unit (GRU) or simple recurrent neural network (RNN) layer. The complexity filters can reduce the error of a random ensemble prediction by a factor of 10. Further, because we use randomly parameterized neural networks, no hyperparameter tuning is required. We prove this method useful for real-time time series prediction because the optimization of hyperparameters, which is usually very costly and time-intensive, can be circumvented with the presented approach.     

Discrimination of Patients with Varying Degrees of Coronary Artery Stenosis by ECG and PCG Signals Based on Entropy

Coronary heart disease (CHD) is the leading cause of cardiovascular death. This study aimed to propose an effective method for mining cardiac mechano-electric coupling information and to evaluate its ability to distinguish patients with varying degrees of coronary artery stenosis (VDCAS). Five minutes of electrocardiogram and phonocardiogram signals was collected synchronously from 191 VDCAS patients to construct heartbeat interval (RRI)–systolic time interval (STI), RRI–diastolic time interval (DTI), HR-corrected QT interval (QTcI)–STI, QTcI–DTI, Tpeak–Tend interval (TpeI)–STI, TpeI–DTI, Tpe/QT interval (Tpe/QTI)–STI, and Tpe/QTI–DTI series. Then, the cross sample entropy (XSampEn), cross fuzzy entropy (XFuzzyEn), joint distribution entropy (JDistEn), magnitude-squared coherence function, cross power spectral density, and mutual information were applied to evaluate the coupling of the series. Subsequently, support vector machine recursive feature elimination and XGBoost were utilized for feature selection and classification, respectively. Results showed that the joint analysis of XSampEn, XFuzzyEn, and JDistEn had the best ability to distinguish patients with VDCAS. The classification accuracy of severe CHD—mild-to-moderate CHD group, severe CHD—chest pain and normal coronary angiography (CPNCA) group, and mild-to-moderate CHD—CPNCA group were 0.8043, 0.7659, and 0.7500, respectively. The study indicates that the joint analysis of XSampEn, XFuzzyEn, and JDistEn can effectively capture the cardiac mechano-electric coupling information of patients with VDCAS, which can provide valuable information for clinicians to diagnose CHD.     

Redundant Information Neural Estimation

We introduce the Redundant Information Neural Estimator (RINE), a method that allows efficient estimation for the component of information about a target variable that is common to a set of sources, known as the “redundant information”. We show that existing definitions of the redundant information can be recast in terms of an optimization over a family of functions. In contrast to previous information decompositions, which can only be evaluated for discrete variables over small alphabets, we show that optimizing over functions enables the approximation of the redundant information for high-dimensional and continuous predictors. We demonstrate this on high-dimensional image classification and motor-neuroscience tasks.     

Novel Convolutional Neural Network with Variational Information Bottleneck for P300 Detection

In the area of brain-computer interfaces (BCI), the detection of P300 is a very important technique and has a lot of applications. Although this problem has been studied for decades, it is still a tough problem in electroencephalography (EEG) signal processing owing to its high dimension features and low signal-to-noise ratio (SNR). Recently, neural networks, like conventional neural networks (CNN), has shown excellent performance on many applications. However, standard convolutional neural networks suffer from performance degradation on dealing with noisy data or data with too many redundant information. In this paper, we proposed a novel convolutional neural network with variational information bottleneck for P300 detection. Wiht the CNN architecture and information bottleneck, the proposed network termed P300-VIB-Net could remove the redundant information in data effectively. The experimental results on BCI competition data sets show that P300-VIB-Net achieves cutting-edge character recognition performance. Furthermore, the proposed model is capable of restricting the flow of irrelevant information adaptively in the network from perspective of information theory. The experimental results show that P300-VIB-Net is a promising tool for P300 detection.     

基于Kendall改进的同步算法癫痫脑网络分析

提出了一种基于Kendall等级相关改进的同步算法IRC(inverse rank correlation).Kendall等级相关是非线性动力学分析的一般化算法,可有效地度量变量间的非线性相关性.复杂网络的研究已逐渐深入到社会科学的各个领域,脑网络的研究已经成为当今脑功能研究的热点.利用改进的IRC算法,基于脑电EEG(electroencephalogram)数据来构建大脑功能性网络.对构建的脑功能网络的度指标进行了分析,以调查癫痫脑功能网络是否异于正常人.结果显示:使用该改进的算法能够对癫痫和正常脑功能网络显著区分,且只需要记录很短的脑电数据.实验结果数据表明,该方法适用于区分癫痫和正常脑组织网络度指标,它可有助于进一步地加深对大脑的神经动力学行为的研究,并为临床诊断提供有效工具.