共查询到20条相似文献，搜索用时 15 毫秒
1.
It is often claimed that the entropy of a network’s degree distribution is a proxy for its robustness. Here, we clarify the link between degree distribution entropy and giant component robustness to node removal by showing that the former merely sets a lower bound to the latter for randomly configured networks when no other network characteristics are specified. Furthermore, we show that, for networks of fixed expected degree that follow degree distributions of the same form, the degree distribution entropy is not indicative of robustness. By contrast, we show that the remaining degree entropy and robustness have a positive monotonic relationship and give an analytic expression for the remaining degree entropy of the log-normal distribution. We also show that degree-degree correlations are not by themselves indicative of a network’s robustness for real networks. We propose an adjustment to how mutual information is measured which better encapsulates structural properties related to robustness. 相似文献
2.
3.
Xiyu Shi Varuna De-Silva Yusuf Aslan Erhan Ekmekcioglu Ahmet Kondoz 《Entropy (Basel, Switzerland)》2022,24(1)
Deep learning has proven to be an important element of modern data processing technology, which has found its application in many areas such as multimodal sensor data processing and understanding, data generation and anomaly detection. While the use of deep learning is booming in many real-world tasks, the internal processes of how it draws results is still uncertain. Understanding the data processing pathways within a deep neural network is important for transparency and better resource utilisation. In this paper, a method utilising information theoretic measures is used to reveal the typical learning patterns of convolutional neural networks, which are commonly used for image processing tasks. For this purpose, training samples, true labels and estimated labels are considered to be random variables. The mutual information and conditional entropy between these variables are then studied using information theoretical measures. This paper shows that more convolutional layers in the network improve its learning and unnecessarily higher numbers of convolutional layers do not improve the learning any further. The number of convolutional layers that need to be added to a neural network to gain the desired learning level can be determined with the help of theoretic information quantities including entropy, inequality and mutual information among the inputs to the network. The kernel size of convolutional layers only affects the learning speed of the network. This study also shows that where the dropout layer is applied to has no significant effects on the learning of networks with a lower dropout rate, and it is better placed immediately after the last convolutional layer with higher dropout rates. 相似文献
4.
Mateu Sbert Shuning Chen Miquel Feixas Marius Vila Amos Golan 《Entropy (Basel, Switzerland)》2020,22(12)
Information theory, and the concept of information channel, allows us to calculate the mutual information between the source (input) and the receiver (output), both represented by probability distributions over their possible states. In this paper, we use the theory behind the information channel to provide an enhanced interpretation to a Social Accounting Matrix (SAM), a square matrix whose columns and rows present the expenditure and receipt accounts of economic actors. Under our interpretation, the SAM’s coefficients, which, conceptually, can be viewed as a Markov chain, can be interpreted as an information channel, allowing us to optimize the desired level of aggregation within the SAM. In addition, the developed information measures can describe accurately the evolution of a SAM over time. Interpreting the SAM matrix as an ergodic chain could show the effect of a shock on the economy after several periods or economic cycles. Under our new framework, finding the power limit of the matrix allows one to check (and confirm) whether the matrix is well-constructed (irreducible and aperiodic), and obtain new optimization functions to balance the SAM matrix. In addition to the theory, we also provide two empirical examples that support our channel concept and help to understand the associated measures. 相似文献
5.
在有干扰(噪声)无记忆信道和有记忆信道两种情况下,运用协同学的方法研究如何分配输入概率获取二元对称离散信道最大的平均互信息. 研究结果表明：对于无记忆二元对称离散信道,最大平均互信息与信息论中相同输入概率使平均互信息最大化的结果是一致的;对于干扰(噪声)有记忆二元对称离散信道,考虑输入、输出符号满足不同程度的记忆度和干扰(噪声)因子情况下,得到符号最佳输入概率. 拓展了一种研究信息论的方法. 关键词：协同学二元对称离散信道互信息转移概率 相似文献
6.
Sina Molavipour Hamid Ghourchian Germn Bassi Mikael Skoglund 《Entropy (Basel, Switzerland)》2021,23(6)
Novel approaches to estimate information measures using neural networks are well-celebrated in recent years both in the information theory and machine learning communities. These neural-based estimators are shown to converge to the true values when estimating mutual information and conditional mutual information using independent samples. However, if the samples in the dataset are not independent, the consistency of these estimators requires further investigation. This is of particular interest for a more complex measure such as the directed information, which is pivotal in characterizing causality and is meaningful over time-dependent variables. The extension of the convergence proof for such cases is not trivial and demands further assumptions on the data. In this paper, we show that our neural estimator for conditional mutual information is consistent when the dataset is generated with samples of a stationary and ergodic source. In other words, we show that our information estimator using neural networks converges asymptotically to the true value with probability one. Besides universal functional approximation of neural networks, a core lemma to show the convergence is Birkhoff’s ergodic theorem. Additionally, we use the technique to estimate directed information and demonstrate the effectiveness of our approach in simulations. 相似文献
7.
一种基于文本互信息的金融复杂网络模型 总被引：1，自引：0，他引：1
复杂网络能够解决许多金融问题,能够发现金融市场的拓扑结构特征,反映不同金融主体之间的相互依赖关系.相关性度量在金融复杂网络构建中至关重要.通过将多元金融时间序列符号化,借鉴文本特征提取以及信息论的方法,定义了一种基于文本互信息的相关系数.为检验方法的有效性,分别构建了基于不同相关系数(Pearson和文本互信息)和不同网络缩减方法(阈值和最小生成树)的4个金融复杂网络模型.在阈值网络中提出了使用分位数来确定阈值的方法,将相关系数6等分,取第4部分的中点作为阈值,此时基于Pearson和文本互信息的阈值模型将会有相近的边数,有利于这两种模型的对比.数据使用了沪深两地证券市场地区指数收盘价,时间从2006年1月4日至2016年12月30日,共计2673个交易日.从网络节点相关性看,基于文本互信息的方法能够体现出大约20%的非线性相关关系;在网络整体拓扑指标上,本文计算了4种指标,结果显示能够使所保留的节点联系更为紧密,有效提高保留节点的重要性以及挖掘出更好的社区结构;最后,计算了阈值网络的动态指标,将数据按年分别构建网络,缩减方法只用了阈值方法,结果显示本文提出的方法在小世界动态和网络度中心性等指标上能够成功捕捉到样本区间内存在的两次异常波动.此外,本文构建的地区金融网络具有服从幂律分布、动态稳定性、一些经济欠发达地区在金融地区网络中占据重要地位等特性. 相似文献
8.
本文采用互信息方法对磁刺激内关穴过程中的脑电信 号进行了两两通道间非线性时域关联特性分析, 构建了不同频率刺激前、刺激中、刺激后的脑功能网络, 并基于复杂网络理论对脑功能网络的特征进行了深入研究. 结果表明, 磁刺激频率为3 Hz 时, 大脑功能网络的平均度、平均聚类系数和全局效率与刺激前相比均有显著升高, 平均路径长度显著降低, 并且相应脑功能网络的\"小世界\"属性有所增强, 信息在大脑各区域间的传递更加高效. 本研究首次开展了磁刺激穴位复杂脑功能网络的构建与分析, 为探索磁刺激穴位对大脑神经调节的作用和机理提供新思路和新方法.关键词：复杂网络磁刺激脑功能网络互信息 相似文献
9.
Modeling a causal association as arising from a communication process between cause and effect, simplifies the discovery of causal skeletons. The communication channels enabling these communication processes, are fully characterized by stochastic tensors, and therefore allow us to use linear algebra. This tensor-based approach reduces the dimensionality of the data needed to test for conditional independence, e.g., for systems comprising three variables, pair-wise determined tensors suffice to infer the causal skeleton. The only thing needed is a minor extension to information theory, namely the concept of path information. 相似文献
10.
11.
Steganography is one of the most crucial methods for information hiding, which embeds secret data on an ordinary file or a cover message for avoiding detection. We designed a novel rate-distortion-based large-capacity secure steganographic system, called rate-distortion-based Stego (RD-Stego), to effectively solve the above requirement. The considered effectiveness of our system design includes embedding capacity, adaptability to chosen cover attacks, and the stability of the trained model. The proposed stego scheme can hide multiple three-channel color images and QR codes within another three-channel color image with low visual distortion. Empirically, with a certain degree of robustness against the chosen cover attack, we state that the system offers up to 192+ bits-per-pixel (bpp) embedding of a payload and leaks no secret-related information. Moreover, to provide theoretical foundations for our cost function design, a mutual information-based explanation of the choices of regulation processes is herein included. Finally, we justify our system’s claimed advantages through a series of experiments with publicly available benchmark datasets. 相似文献
12.
We present a quantum information network in which quantum information density is used for performing quantum computing or
teleportation. The photons are entangled in quantum channels and play a role of flying ebit to transmit interaction among
the nodes. A particular quantum Gaussian channel is constructed; it permits photon-encoded information to transmit quantum
signals with certain quantum parallelism. The corresponding quantum dynamical mutual information is discussed, and the controlling
nodes connectivity by driving the network is studied. With regard to different driving functions, the connectivity distribution
of the network is complicated. They obey positive or negative power law, and also influence the assortativity coefficient
or the dynamical property of the network.
相似文献
13.
Jesús Malo 《Entropy (Basel, Switzerland)》2022,24(10)
Biological neural networks for color vision (also known as color appearance models) consist of a cascade of linear + nonlinear layers that modify the linear measurements at the retinal photo-receptors leading to an internal (nonlinear) representation of color that correlates with psychophysical experience. The basic layers of these networks include: (1) chromatic adaptation (normalization of the mean and covariance of the color manifold); (2) change to opponent color channels (PCA-like rotation in the color space); and (3) saturating nonlinearities to obtain perceptually Euclidean color representations (similar to dimension-wise equalization). The Efficient Coding Hypothesis argues that these transforms should emerge from information-theoretic goals. In case this hypothesis holds in color vision, the question is what is the coding gain due to the different layers of the color appearance networks? In this work, a representative family of color appearance models is analyzed in terms of how the redundancy among the chromatic components is modified along the network and how much information is transferred from the input data to the noisy response. The proposed analysis is performed using data and methods that were not available before: (1) new colorimetrically calibrated scenes in different CIE illuminations for the proper evaluation of chromatic adaptation; and (2) new statistical tools to estimate (multivariate) information-theoretic quantities between multidimensional sets based on Gaussianization. The results confirm that the efficient coding hypothesis holds for current color vision models, and identify the psychophysical mechanisms critically responsible for gains in information transference: opponent channels and their nonlinear nature are more important than chromatic adaptation at the retina. 相似文献
14.
Michele Monti Jonathan Fiorentino Edoardo Milanetti Giorgio Gosti Gian Gaetano Tartaglia 《Entropy (Basel, Switzerland)》2022,24(2)
Methods for time series prediction and classification of gene regulatory networks (GRNs) from gene expression data have been treated separately so far. The recent emergence of attention-based recurrent neural network (RNN) models boosted the interpretability of RNN parameters, making them appealing for the understanding of gene interactions. In this work, we generated synthetic time series gene expression data from a range of archetypal GRNs and we relied on a dual attention RNN to predict the gene temporal dynamics. We show that the prediction is extremely accurate for GRNs with different architectures. Next, we focused on the attention mechanism of the RNN and, using tools from graph theory, we found that its graph properties allow one to hierarchically distinguish different architectures of the GRN. We show that the GRN responded differently to the addition of noise in the prediction by the RNN and we related the noise response to the analysis of the attention mechanism. In conclusion, this work provides a way to understand and exploit the attention mechanism of RNNs and it paves the way to RNN-based methods for time series prediction and inference of GRNs from gene expression data. 相似文献
15.
Fitness landscapes are a powerful metaphor for understanding the evolution of biological systems. These landscapes describe how genotypes are connected to each other through mutation and related through fitness. Empirical studies of fitness landscapes have increasingly revealed conserved topographical features across diverse taxa, e.g., the accessibility of genotypes and “ruggedness”. As a result, theoretical studies are needed to investigate how evolution proceeds on fitness landscapes with such conserved features. Here, we develop and study a model of evolution on fitness landscapes using the lens of Gene Regulatory Networks (GRNs), where the regulatory products are computed from multiple genes and collectively treated as phenotypes. With the assumption that regulation is a binary process, we prove the existence of empirically observed, topographical features such as accessibility and connectivity. We further show that these results hold across arbitrary fitness functions and that a trade-off between accessibility and ruggedness need not exist. Then, using graph theory and a coarse-graining approach, we deduce a mesoscopic structure underlying GRN fitness landscapes where the information necessary to predict a population’s evolutionary trajectory is retained with minimal complexity. Using this coarse-graining, we develop a bottom-up algorithm to construct such mesoscopic backbones, which does not require computing the genotype network and is therefore far more efficient than brute-force approaches. Altogether, this work provides mathematical results of high-dimensional fitness landscapes and a path toward connecting theory to empirical studies. 相似文献
16.
The impact of observational noise on the analysis of scale-free networks is studied. Various noise sources are modeled as random link removal, random link exchange and random link addition. Emphasis is on the resulting modifications for the node-degree distribution and for a functional ranking based on betweenness centrality. The implications for estimated gene-expressed networks for childhood acute lymphoblastic leukemia are discussed. 相似文献
17.
Fitts studied the problem of information capacity and transfer in the speed–accuracy motor paradigm using a theoretical approach developed from Shannon and Weaver’s information theory. The information processing (bit/s) estimated in Fitts’ study is calculated from the movement time required to achieve the required task index of difficulty but is essentially different from Shannon’s information entropy. Thus, we estimated the information entropy of multiple human movement trajectories and the mutual information among trajectories for the continuous aiming task in Fitts’ paradigm. Further, we estimated the information processing moment by moment. Two methods were considered: (1) encoded values encompassing the coordinates of the three dimensions and (2) coordinate values associated with each direction in the three dimensions. Information entropy indicates the magnitude of variation at each time point, and the structure of this variation varies with the index of difficulty. The ratio of entropy to mutual information was examined, and it was found that information was processed from the first half of the trajectory in difficult tasks. In addition, since these values calculated from the encoded method were higher than those from the conventional method, this method may be able to estimate these values successfully. 相似文献
18.
Digital communication receivers extract information about the transmitted data from the received signal in subsequent processing steps, such as synchronization, demodulation and channel decoding. Technically, the receiver-side signal processing for conducting these tasks is complex and hence causes bottleneck situations in terms of power, delay and chip area. Typically, many bits per sample are required to represent and process the received signal in the digital receiver hardware accurately. In addition, demanding arithmetical operations are required in the signal processing algorithms. A popular recent trend is designing entire receiver chains or some of their crucial building blocks from an information theoretical perspective. Signal processing blocks with very simple mathematical operations can be designed to directly maximize the relevant information that flows through them. At the same time, a strong quantization reduces the number of bits processed in the receiver to further lower the complexity. The described system design approach follows the principle of the information bottleneck method. Different authors proposed various ideas to design and implement mutual information-maximizing signal processing units. The first important aim of this article is to explain the fundamental similarities between the information bottleneck method and the functionalities of communication receivers. Based on that, we present and investigate new results on an entire receiver chain that is designed following the information bottleneck design principle. Afterwards, we give an overview of different techniques following the information bottleneck design paradigm from the literature, mainly dealing with channel decoding applications. We analyze the similarities of the different approaches for information bottleneck signal processing. This comparison leads to a general view on information bottleneck signal processing which goes back to the learning of parameters of trainable functions that maximize the relevant mutual information under compression. 相似文献
19.