Information Bottleneck Signal Processing and Learning to Maximize Relevant Information for Communication Receivers

Digital communication receivers extract information about the transmitted data from the received signal in subsequent processing steps, such as synchronization, demodulation and channel decoding. Technically, the receiver-side signal processing for conducting these tasks is complex and hence causes bottleneck situations in terms of power, delay and chip area. Typically, many bits per sample are required to represent and process the received signal in the digital receiver hardware accurately. In addition, demanding arithmetical operations are required in the signal processing algorithms. A popular recent trend is designing entire receiver chains or some of their crucial building blocks from an information theoretical perspective. Signal processing blocks with very simple mathematical operations can be designed to directly maximize the relevant information that flows through them. At the same time, a strong quantization reduces the number of bits processed in the receiver to further lower the complexity. The described system design approach follows the principle of the information bottleneck method. Different authors proposed various ideas to design and implement mutual information-maximizing signal processing units. The first important aim of this article is to explain the fundamental similarities between the information bottleneck method and the functionalities of communication receivers. Based on that, we present and investigate new results on an entire receiver chain that is designed following the information bottleneck design principle. Afterwards, we give an overview of different techniques following the information bottleneck design paradigm from the literature, mainly dealing with channel decoding applications. We analyze the similarities of the different approaches for information bottleneck signal processing. This comparison leads to a general view on information bottleneck signal processing which goes back to the learning of parameters of trainable functions that maximize the relevant mutual information under compression.     

Estimating Information Processing of Human Fast Continuous Tapping from Trajectories

Fitts studied the problem of information capacity and transfer in the speed–accuracy motor paradigm using a theoretical approach developed from Shannon and Weaver’s information theory. The information processing (bit/s) estimated in Fitts’ study is calculated from the movement time required to achieve the required task index of difficulty but is essentially different from Shannon’s information entropy. Thus, we estimated the information entropy of multiple human movement trajectories and the mutual information among trajectories for the continuous aiming task in Fitts’ paradigm. Further, we estimated the information processing moment by moment. Two methods were considered: (1) encoded values encompassing the coordinates of the three dimensions and (2) coordinate values associated with each direction in the three dimensions. Information entropy indicates the magnitude of variation at each time point, and the structure of this variation varies with the index of difficulty. The ratio of entropy to mutual information was examined, and it was found that information was processed from the first half of the trajectory in difficult tasks. In addition, since these values calculated from the encoded method were higher than those from the conventional method, this method may be able to estimate these values successfully.     

麦克斯韦妖与信息处理的物理极限

文章系统地评述了麦克斯韦妖佯谬相关的热力学基本观念的发端、历史沿革以及当前正在发展的科学前沿问题。文章作者从以下两个方面详细地阐述了为什么信息处理过程本质上是一个与麦克斯韦妖观念相“纠缠”的物理过程：(1)信息认知和提取可以辅助物理系统更有效地做功;(2) 物理定律会对信息处理过程施加一个不可逾越的物理极限。这些分析与概念的澄清将有助于正确理解计算过程和热力学之间的关系。     

Interpreting Social Accounting Matrix (SAM) as an Information Channel

Information theory, and the concept of information channel, allows us to calculate the mutual information between the source (input) and the receiver (output), both represented by probability distributions over their possible states. In this paper, we use the theory behind the information channel to provide an enhanced interpretation to a Social Accounting Matrix (SAM), a square matrix whose columns and rows present the expenditure and receipt accounts of economic actors. Under our interpretation, the SAM’s coefficients, which, conceptually, can be viewed as a Markov chain, can be interpreted as an information channel, allowing us to optimize the desired level of aggregation within the SAM. In addition, the developed information measures can describe accurately the evolution of a SAM over time. Interpreting the SAM matrix as an ergodic chain could show the effect of a shock on the economy after several periods or economic cycles. Under our new framework, finding the power limit of the matrix allows one to check (and confirm) whether the matrix is well-constructed (irreducible and aperiodic), and obtain new optimization functions to balance the SAM matrix. In addition to the theory, we also provide two empirical examples that support our channel concept and help to understand the associated measures.     

Verification of Information Thermodynamics in a Trapped Ion System

Information thermodynamics has developed rapidly over past years, and the trapped ions, as a controllable quantum system, have demonstrated feasibility to experimentally verify the theoretical predictions in the information thermodynamics. Here, we address some representative theories of information thermodynamics, such as the quantum Landauer principle, information equality based on the two-point measurement, information-theoretical bound of irreversibility, and speed limit restrained by the entropy production of system, and review their experimental demonstration in the trapped ion system. In these schemes, the typical physical processes, such as the entropy flow, energy transfer, and information flow, build the connection between thermodynamic processes and information variation. We then elucidate the concrete quantum control strategies to simulate these processes by using quantum operators and the decay paths in the trapped-ion system. Based on them, some significantly dynamical processes in the trapped ion system to realize the newly proposed information-thermodynamic models is reviewed. Although only some latest experimental results of information thermodynamics with a single trapped-ion quantum system are reviewed here, we expect to find more exploration in the future with more ions involved in the experimental systems.     

Partial Information Decomposition and the Information Delta: A Geometric Unification Disentangling Non-Pairwise Information

Information theory provides robust measures of multivariable interdependence, but classically does little to characterize the multivariable relationships it detects. The Partial Information Decomposition (PID) characterizes the mutual information between variables by decomposing it into unique, redundant, and synergistic components. This has been usefully applied, particularly in neuroscience, but there is currently no generally accepted method for its computation. Independently, the Information Delta framework characterizes non-pairwise dependencies in genetic datasets. This framework has developed an intuitive geometric interpretation for how discrete functions encode information, but lacks some important generalizations. This paper shows that the PID and Delta frameworks are largely equivalent. We equate their key expressions, allowing for results in one framework to apply towards open questions in the other. For example, we find that the approach of Bertschinger et al. is useful for the open Information Delta question of how to deal with linkage disequilibrium. We also show how PID solutions can be mapped onto the space of delta measures. Using Bertschinger et al. as an example solution, we identify a specific plane in delta-space on which this approach’s optimization is constrained, and compute it for all possible three-variable discrete functions of a three-letter alphabet. This yields a clear geometric picture of how a given solution decomposes information.     

Genetic and entropy methods for laser process prediction

Laser material processing can be modelled via an analytical or numerical approach. This article formulates two unconventional analytical approaches for the prediction of laser processing parameters. An evolutionary algorithm is used for the prediction of the optimal conditions for pulsed laser cutting. Continuous wave laser cutting is characterised by assuming a likelihood bias function, from which an optimal velocity mapping is obtained utilising the MAXENT concept. The mapping function serves as a topological guide to the prediction of cutting rates. Both prediction schemes give good agreement with experimental results.     

Photonic quantum information processing

Information processing with light is ubiquitous, from communication, metrology and imaging to computing. When we consider light as a quantum mechanical object, new ways of information processing become possible. In this review I give an overview of how quantum information processing can be implemented with single photons, and what hurdles still need to be overcome to implement the various applications in practice. I will place special emphasis on the quantum mechanical properties of light that make it different from classical light, and how these properties relate to quantum information processing tasks.     

The Causal Interaction between Complex Subsystems

Information flow provides a natural measure for the causal interaction between dynamical events. This study extends our previous rigorous formalism of componentwise information flow to the bulk information flow between two complex subsystems of a large-dimensional parental system. Analytical formulas have been obtained in a closed form. Under a Gaussian assumption, their maximum likelihood estimators have also been obtained. These formulas have been validated using different subsystems with preset relations, and they yield causalities just as expected. On the contrary, the commonly used proxies for the characterization of subsystems, such as averages and principal components, generally do not work correctly. This study can help diagnose the emergence of patterns in complex systems and is expected to have applications in many real world problems in different disciplines such as climate science, fluid dynamics, neuroscience, financial economics, etc.     

量子相干

量子相干不仅是量子力学中的一个基本概念,同时也是重要的量子信息处理的物理资源.随着基于资源理论框架的量子相干度量方案的提出,量子相干度的量化研究成为近年来人们关注的一个热点问题.量子相干作为一种物理资源也十分脆弱,极容易受到环境噪声的影响而产生退相干,因此开放系统中的量子相干演化和保持也是人们广泛关注的课题.另外,量子相干在量子多体系统、量子热动力学、量子生物学等领域也有着潜在的应用价值.本文介绍量子相干度量的资源理论框架和基于该框架定义的相对熵相干性、l1范数相干性、基于量子纠缠的相干性、基于凸顶结构的相干性和相干鲁棒性等量子相干度量函数,概述开放系统中量子相干演化的动力学行为、典型信道的量子相干产生和破坏能力以及量子相干的冻结等现象,同时例举量子相干在Deutsch-Jozsa算法、Grover算法以及量子多体系统相变问题研究等方面的重要应用.量子相干研究仍处于快速发展之中,期望本综述能为该领域的发展带来启示.     

The pyramidal life cycle of economic structures

The pyramidal character of life cycles of economic structures is empirically confirmed on the basis of official statistical data of the years 2003 and 2004. The ergodic hypothesis is applied as one of methods of economic forecasting in evolutionary economics, econophysics and applied researches.     

Cognition as Morphological/Morphogenetic Embodied Computation In Vivo

Cognition, historically considered uniquely human capacity, has been recently found to be the ability of all living organisms, from single cells and up. This study approaches cognition from an info-computational stance, in which structures in nature are seen as information, and processes (information dynamics) are seen as computation, from the perspective of a cognizing agent. Cognition is understood as a network of concurrent morphological/morphogenetic computations unfolding as a result of self-assembly, self-organization, and autopoiesis of physical, chemical, and biological agents. The present-day human-centric view of cognition still prevailing in major encyclopedias has a variety of open problems. This article considers recent research about morphological computation, morphogenesis, agency, basal cognition, extended evolutionary synthesis, free energy principle, cognition as Bayesian learning, active inference, and related topics, offering new theoretical and practical perspectives on problems inherent to the old computationalist cognitive models which were based on abstract symbol processing, and unaware of actual physical constraints and affordances of the embodiment of cognizing agents. A better understanding of cognition is centrally important for future artificial intelligence, robotics, medicine, and related fields.     

Ups and downs of economics and econophysics — Facebook forecast

What is econophysics and its relationship with economics? What is the state of economics after the global economic crisis, and is there a future for the paradigm of market equilibrium, with imaginary perfect competition and rational agents? Can the next paradigm of economics adopt important assumptions derived from econophysics models: that markets are chaotic systems, striving to extremes as bubbles and crashes show, with psychologically motivated, statistically predictable individual behaviors? Is the future of econophysics, as predicted here, to disappear and become a part of economics? A good test of the current state of econophysics and its methods is the valuation of Facebook immediately after the initial public offering — this forecast indicates that Facebook is highly overvalued, and its IPO valuation of 104 billion dollars is mostly the new financial bubble based on the expectations of unlimited growth, although it’s easy to prove that Facebook is close to the upper limit of its users.     

Is econophysics a new discipline? The neopositivist argument

Econophysics is a new approach which applies various models and concepts associated with statistical physics to economic (and financial) phenomena. This field of research is a new step in the history and the evolution of Physics Sciences and the question about the disciplinary characteristics of this field must be asked. At first glance, it might appear that economics and econophysics share the same subject of research (that of analysis of economic reality). In this paper I will use neopositivism to show that econophysics is methodologically very different from economics and that it can be considered as a separate discipline. The neopositivist framework provides econophysics with some arguments for rejecting mainstream economics.     

Entropy: From Thermodynamics to Information Processing

Entropy is a concept that emerged in the 19th century. It used to be associated with heat harnessed by a thermal machine to perform work during the Industrial Revolution. However, there was an unprecedented scientific revolution in the 20th century due to one of its most essential innovations, i.e., the information theory, which also encompasses the concept of entropy. Therefore, the following question is naturally raised: “what is the difference, if any, between concepts of entropy in each field of knowledge?” There are misconceptions, as there have been multiple attempts to conciliate the entropy of thermodynamics with that of information theory. Entropy is most commonly defined as “disorder”, although it is not a good analogy since “order” is a subjective human concept, and “disorder” cannot always be obtained from entropy. Therefore, this paper presents a historical background on the evolution of the term “entropy”, and provides mathematical evidence and logical arguments regarding its interconnection in various scientific areas, with the objective of providing a theoretical review and reference material for a broad audience.     

Quantum Pseudo-Telepathy

Quantum information processing is at the crossroads of physics, mathematics and computer science. It is concerned with what we can and cannot do with quantum information that goes beyond the abilities of classical information processing devices. Communication complexity is an area of classical computer science that aims at quantifying the amount of communication necessary to solve distributed computational problems. Quantum communication complexity uses quantum mechanics to reduce the amount of communication that would be classically required. Pseudo-telepathy is a surprising application of quantum information processing to communication complexity. Thanks to entanglement, perhaps the most nonclassical manifestation of quantum mechanics, two or more quantum players can accomplish a distributed task with no need for communication whatsoever, which would be an impossible feat for classical players. After a detailed overview of the principle and purpose of pseudo-telepathy, we present a survey of recent and not-so-recent work on the subject. In particular, we describe and analyse all the pseudo-telepathy games currently known to the authors. Supported in Part by Canada’s Natural Sciences and Engineering Research Council (NSERC), the Canada Research Chair programme and the Canadian Institute for Advanced Research (CIAR). Supported in part by a scholarship from Canada’s NSERC. Supported in part by Canada’s NSERC Québec’s Fonds de recherche sur la nature et les technologies (FQRNT), the CIAR and the Mathematics of Information Technology and Complex Systems Network (MITACS).     

On Entropy,Information, and Conservation of Information

The term entropy is used in different meanings in different contexts, sometimes in contradictory ways, resulting in misunderstandings and confusion. The root cause of the problem is the close resemblance of the defining mathematical expressions of entropy in statistical thermodynamics and information in the communications field, also called entropy, differing only by a constant factor with the unit ‘J/K’ in thermodynamics and ‘bits’ in the information theory. The thermodynamic property entropy is closely associated with the physical quantities of thermal energy and temperature, while the entropy used in the communications field is a mathematical abstraction based on probabilities of messages. The terms information and entropy are often used interchangeably in several branches of sciences. This practice gives rise to the phrase conservation of entropy in the sense of conservation of information, which is in contradiction to the fundamental increase of entropy principle in thermodynamics as an expression of the second law. The aim of this paper is to clarify matters and eliminate confusion by putting things into their rightful places within their domains. The notion of conservation of information is also put into a proper perspective.     

Discovering Higher-Order Interactions Through Neural Information Decomposition

If regularity in data takes the form of higher-order functions among groups of variables, models which are biased towards lower-order functions may easily mistake the data for noise. To distinguish whether this is the case, one must be able to quantify the contribution of different orders of dependence to the total information. Recent work in information theory attempts to do this through measures of multivariate mutual information (MMI) and information decomposition (ID). Despite substantial theoretical progress, practical issues related to tractability and learnability of higher-order functions are still largely unaddressed. In this work, we introduce a new approach to information decomposition—termed Neural Information Decomposition (NID)—which is both theoretically grounded, and can be efficiently estimated in practice using neural networks. We show on synthetic data that NID can learn to distinguish higher-order functions from noise, while many unsupervised probability models cannot. Additionally, we demonstrate the usefulness of this framework as a tool for exploring biological and artificial neural networks.     

Second law of thermodynamics with discrete quantum feedback control

A new thermodynamic inequality is derived which leads to the maximum work that can be extracted from multi-heat-baths with the assistance of discrete quantum feedback control. The maximum work is determined by the free-energy difference and a generalized mutual information content between the thermodynamic system and the feedback controller. This maximum work can exceed that in conventional thermodynamics and, in the case of a heat cycle with two heat baths, the heat efficiency can be greater than that of the Carnot cycle. The consistency of our results with the second law of thermodynamics is ensured by the fact that work is needed for information processing of the feedback controller.