On Rényi Permutation Entropy

Among various modifications of the permutation entropy defined as the Shannon entropy of the ordinal pattern distribution underlying a system, a variant based on Rényi entropies was considered in a few papers. This paper discusses the relatively new concept of Rényi permutation entropies in dependence of non-negative real number q parameterizing the family of Rényi entropies and providing the Shannon entropy for q=1. Its relationship to Kolmogorov–Sinai entropy and, for q=2, to the recently introduced symbolic correlation integral are touched.     

On the Entropy of a One-Dimensional Gas with and without Mixing Using Sinai Billiard

A one-dimensional gas comprising N point particles undergoing elastic collisions within a finite space described by a Sinai billiard generating identical dynamical trajectories are calculated and analyzed with regard to strict extensivity of the entropy definitions of Boltzmann–Gibbs. Due to the collisions, trajectories of gas particles are strongly correlated and exhibit both chaotic and periodic properties. Probability distributions for the position of each particle in the one-dimensional gas can be obtained analytically, elucidating that the entropy in this special case is extensive at any given number N. Furthermore, the entropy obtained can be interpreted as a measure of the extent of interactions between molecules. The results obtained for the non-mixable one-dimensional system are generalized to mixable one- and two-dimensional systems, the latter by a simple example only providing similar findings.     

Entropy-Based Classification of Elementary Cellular Automata under Asynchronous Updating: An Experimental Study

Classification of asynchronous elementary cellular automata (AECAs) was explored in the first place by Fates et al. (Complex Systems, 2004) who employed the asymptotic density of cells as a key metric to measure their robustness to stochastic transitions. Unfortunately, the asymptotic density seems unable to distinguish the robustnesses of all AECAs. In this paper, we put forward a method that goes one step further via adopting a metric entropy (Martin, Complex Systems, 2000), with the aim of measuring the asymptotic mean entropy of local pattern distribution in the cell space of any AECA. Numerical experiments demonstrate that such an entropy-based measure can actually facilitate a complete classification of the robustnesses of all AECA models, even when all local patterns are restricted to length 1. To gain more insights into the complexity concerning the forward evolution of all AECAs, we consider another entropy defined in the form of Kolmogorov–Sinai entropy and conduct preliminary experiments on classifying their uncertainties measured in terms of the proposed entropy. The results reveal that AECAs with low uncertainty tend to converge remarkably faster than models with high uncertainty.     

Belavkin–Staszewski Relative Entropy,Conditional Entropy,and Mutual Information

Belavkin–Staszewski relative entropy can naturally characterize the effects of the possible noncommutativity of quantum states. In this paper, two new conditional entropy terms and four new mutual information terms are first defined by replacing quantum relative entropy with Belavkin–Staszewski relative entropy. Next, their basic properties are investigated, especially in classical-quantum settings. In particular, we show the weak concavity of the Belavkin–Staszewski conditional entropy and obtain the chain rule for the Belavkin–Staszewski mutual information. Finally, the subadditivity of the Belavkin–Staszewski relative entropy is established, i.e., the Belavkin–Staszewski relative entropy of a joint system is less than the sum of that of its corresponding subsystems with the help of some multiplicative and additive factors. Meanwhile, we also provide a certain subadditivity of the geometric Rényi relative entropy.     

Quantum dynamics, measurement and entropy

The aim of this paper is to present a line of ideas, centred around entropy production andquantum dynamics, emerging from von Neumann's work on foundations of quantum mechanics and leading to current research. The concepts of measurement, dynamical evolution and entropy were central in J. von Neumann's work. Further developments led to the introduction of generalized measurements in terms of positive operator-valued measures, closely connected to the theory of open systems. Fundamental properties of quantum entropy were derived and Kolmogorov and Sinai related the chaotic properties of classical dynamical systems with asymptotic entropy production. Finally, entropy production in quantum dynamical systems was linked with repeated measurement processes and a whole research area on nonequilibrium phenomena in quantum dynamical systems seems to emerge.     

Transfer Entropy of West Pacific Earthquakes to Inner Van Allen Belt Electron Bursts

Lithosphere-ionosphere non-linear interactions create a complex system where links between different phenomena can remain hidden. The statistical correlation between West Pacific strong earthquakes and high-energy electron bursts escaping trapped conditions was demonstrated in past works. Here, it is investigated from the point of view of information. Starting from the conditional probability statistical model, which was deduced from the correlation, the Shannon entropy, the joint entropy, and the conditional entropy are calculated. Time-delayed mutual information and transfer entropy have also been calculated analytically here for binary events: by including correlations between consecutive earthquake events, and between consecutive earthquakes and electron bursts. These quantities have been evaluated for the complex dynamical system of lithosphere-ionosphere; although the expressions calculated by probabilities resulted in being valid for each pair of binary events. Peaks occurred for the same time delay as in the correlations, Δt = 1.5–3.5 h, and as well as for a new time delay, Δt = −58.5–−56.5 h, for the transfer entropy; this last is linked to EQ self-correlations from the analysis. Even if the low number of self-correlated EQs makes this second peak insignificant in this case, it is of interest to separate the non-linear contribution of the transfer entropy of binary events in the study of a complex system.     

Novel Features for Binary Time Series Based on Branch Length Similarity Entropy

Branch length similarity (BLS) entropy is defined in a network consisting of a single node and branches. In this study, we mapped the binary time-series signal to the circumference of the time circle so that the BLS entropy can be calculated for the binary time-series. We obtained the BLS entropy values for “1” signals on the time circle. The set of values are the BLS entropy profile. We selected the local maximum (minimum) point, slope, and inflection point of the entropy profile as the characteristic features of the binary time-series and investigated and explored their significance. The local maximum (minimum) point indicates the time at which the rate of change in the signal density becomes zero. The slope and inflection points correspond to the degree of change in the signal density and the time at which the signal density changes occur, respectively. Moreover, we show that the characteristic features can be widely used in binary time-series analysis by characterizing the movement trajectory of Caenorhabditis elegans. We also mention the problems that need to be explored mathematically in relation to the features and propose candidates for additional features based on the BLS entropy profile.     

A Generalized Measure of Cumulative Residual Entropy

In this work, we introduce a generalized measure of cumulative residual entropy and study its properties. We show that several existing measures of entropy, such as cumulative residual entropy, weighted cumulative residual entropy and cumulative residual Tsallis entropy, are all special cases of this generalized cumulative residual entropy. We also propose a measure of generalized cumulative entropy, which includes cumulative entropy, weighted cumulative entropy and cumulative Tsallis entropy as special cases. We discuss a generating function approach, using which we derive different entropy measures. We provide residual and cumulative versions of Sharma–Taneja–Mittal entropy and obtain them as special cases this generalized measure of entropy. Finally, using the newly introduced entropy measures, we establish some relationships between entropy and extropy measures.     

Estimation for Entropy and Parameters of Generalized Bilal Distribution under Adaptive Type II Progressive Hybrid Censoring Scheme

Entropy measures the uncertainty associated with a random variable. It has important applications in cybernetics, probability theory, astrophysics, life sciences and other fields. Recently, many authors focused on the estimation of entropy with different life distributions. However, the estimation of entropy for the generalized Bilal (GB) distribution has not yet been involved. In this paper, we consider the estimation of the entropy and the parameters with GB distribution based on adaptive Type-II progressive hybrid censored data. Maximum likelihood estimation of the entropy and the parameters are obtained using the Newton–Raphson iteration method. Bayesian estimations under different loss functions are provided with the help of Lindley’s approximation. The approximate confidence interval and the Bayesian credible interval of the parameters and entropy are obtained by using the delta and Markov chain Monte Carlo (MCMC) methods, respectively. Monte Carlo simulation studies are carried out to observe the performances of the different point and interval estimations. Finally, a real data set has been analyzed for illustrative purposes.     

Analysis on the Development of Digital Economy in Guangdong Province Based on Improved Entropy Method and Multivariate Statistical Analysis

The lack of adequate indicators in the research of digital economy may lead to the shortage of data support on decision making for governments. To solve this problem, first we establish a digital economy indicator evaluation system by dividing the digital economy into four types: “basic type”, “technology type”, “integration type” and “service type” and select 5 indicators for each type. On this basis, the weight of each indicator is calculated to find the deficiencies in the development of some digital economic fields by the improved entropy method. By drawing on the empowerment idea of Analytic Hierarchy Process, the improved entropy method firstly compares the difference coefficient of indicators in pairs and maps the comparison results to the scales 1–9. Then, the judgment matrix is constructed based on the information entropy, which can solve as much as possible the problem that the difference among the weight of each indicator is too large in traditional entropy method. The results indicate that: the development of digital economy in Guangdong Province was relatively balanced from 2015 to 2018 and will be better in the future while the development of rural e-commerce in Guangdong Province is relatively backward, and there is an obvious digital gap between urban and rural areas. Next we extract two new variables respectively to replace the 20 indicators we select through principal component analysis and factor analysis methods in multivariate statistical analysis, which can retain the original information to the greatest extent and provide convenience for further research in the future. Finally, we and provide constructive comments of digital economy in Guangdong Province from 2015 to 2018.     

From Rényi Entropy Power to Information Scan of Quantum States

In this paper, we generalize the notion of Shannon’s entropy power to the Rényi-entropy setting. With this, we propose generalizations of the de Bruijn identity, isoperimetric inequality, or Stam inequality. This framework not only allows for finding new estimation inequalities, but it also provides a convenient technical framework for the derivation of a one-parameter family of Rényi-entropy-power-based quantum-mechanical uncertainty relations. To illustrate the usefulness of the Rényi entropy power obtained, we show how the information probability distribution associated with a quantum state can be reconstructed in a process that is akin to quantum-state tomography. We illustrate the inner workings of this with the so-called “cat states”, which are of fundamental interest and practical use in schemes such as quantum metrology. Salient issues, including the extension of the notion of entropy power to Tsallis entropy and ensuing implications in estimation theory, are also briefly discussed.     

The Systematic Bias of Entropy Calculation in the Multi-Scale Entropy Algorithm

Entropy indicates irregularity or randomness of a dynamic system. Over the decades, entropy calculated at different scales of the system through subsampling or coarse graining has been used as a surrogate measure of system complexity. One popular multi-scale entropy analysis is the multi-scale sample entropy (MSE), which calculates entropy through the sample entropy (SampEn) formula at each time scale. SampEn is defined by the “logarithmic likelihood” that a small section (within a window of a length m) of the data “matches” with other sections will still “match” the others if the section window length increases by one. “Match” is defined by a threshold of r times standard deviation of the entire time series. A problem of current MSE algorithm is that SampEn calculations at different scales are based on the same matching threshold defined by the original time series but data standard deviation actually changes with the subsampling scales. Using a fixed threshold will automatically introduce systematic bias to the calculation results. The purpose of this paper is to mathematically present this systematic bias and to provide methods for correcting it. Our work will help the large MSE user community avoiding introducing the bias to their multi-scale SampEn calculation results.     

Note on Chaos and Diffusion

Using standard definitions of chaos (as positive Kolmogorov–Sinai entropy) and diffusion (that multiple time distribution functions are Gaussian), we show numerically that both chaotic and nonchaotic systems exhibit diffusion, and hence that there is no direct logical connection between the two properties. This extends a previous result for two time distribution functions.     

Alternative Entropy Measures and Generalized Khinchin–Shannon Inequalities

The Khinchin–Shannon generalized inequalities for entropy measures in Information Theory, are a paradigm which can be used to test the Synergy of the distributions of probabilities of occurrence in physical systems. The rich algebraic structure associated with the introduction of escort probabilities seems to be essential for deriving these inequalities for the two-parameter Sharma–Mittal set of entropy measures. We also emphasize the derivation of these inequalities for the special cases of one-parameter Havrda–Charvat’s, Rényi’s and Landsberg–Vedral’s entropy measures.     

Time-Reversed Dynamical Entropy and Irreversibility in Markovian Random Processes

A concept of time-reversed entropy per unit time is introduced in analogy with the entropy per unit time by Shannon, Kolmogorov, and Sinai. This time-reversed entropy per unit time characterizes the dynamical randomness of a stochastic process backward in time, while the standard entropy per unit time characterizes the dynamical randomness forward in time. The difference between the time-reversed and standard entropies per unit time is shown to give the entropy production of Markovian processes in nonequilibrium steady states.     

Entropy: From Thermodynamics to Information Processing

Entropy is a concept that emerged in the 19th century. It used to be associated with heat harnessed by a thermal machine to perform work during the Industrial Revolution. However, there was an unprecedented scientific revolution in the 20th century due to one of its most essential innovations, i.e., the information theory, which also encompasses the concept of entropy. Therefore, the following question is naturally raised: “what is the difference, if any, between concepts of entropy in each field of knowledge?” There are misconceptions, as there have been multiple attempts to conciliate the entropy of thermodynamics with that of information theory. Entropy is most commonly defined as “disorder”, although it is not a good analogy since “order” is a subjective human concept, and “disorder” cannot always be obtained from entropy. Therefore, this paper presents a historical background on the evolution of the term “entropy”, and provides mathematical evidence and logical arguments regarding its interconnection in various scientific areas, with the objective of providing a theoretical review and reference material for a broad audience.     

Entropy Estimation Using a Linguistic Zipf–Mandelbrot–Li Model for Natural Sequences

Entropy estimation faces numerous challenges when applied to various real-world problems. Our interest is in divergence and entropy estimation algorithms which are capable of rapid estimation for natural sequence data such as human and synthetic languages. This typically requires a large amount of data; however, we propose a new approach which is based on a new rank-based analytic Zipf–Mandelbrot–Li probabilistic model. Unlike previous approaches, which do not consider the nature of the probability distribution in relation to language; here, we introduce a novel analytic Zipfian model which includes linguistic constraints. This provides more accurate distributions for natural sequences such as natural or synthetic emergent languages. Results are given which indicates the performance of the proposed ZML model. We derive an entropy estimation method which incorporates the linguistic constraint-based Zipf–Mandelbrot–Li into a new non-equiprobable coincidence counting algorithm which is shown to be effective for tasks such as entropy rate estimation with limited data.     

The Use of the Statistical Entropy in Some New Approaches for the Description of Biosystems

Some problems of describing biological systems with the use of entropy as a measure of the complexity of these systems are considered. Entropy is studied both for the organism as a whole and for its parts down to the molecular level. Correlation of actions of various parts of the whole organism, intercellular interactions and control, as well as cooperativity on the microlevel lead to a more complex structure and lower statistical entropy. For a multicellular organism, entropy is much lower than entropy for the same mass of a colony of unicellular organisms. Cooperativity always reduces the entropy of the system; a simple example of ligand binding to a macromolecule carrying two reaction centers shows how entropy is consistent with the ambiguity of the result in the Bernoulli test scheme. Particular attention is paid to the qualitative and quantitative relationship between the entropy of the system and the cooperativity of ligand binding to macromolecules. A kinetic model of metabolism. corresponding to Schrödinger’s concept of the maintenance biosystems by “negentropy feeding”, is proposed. This model allows calculating the nonequilibrium local entropy and comparing it with the local equilibrium entropy inherent in non-living matter.     

Asymptotic Normality for Plug-In Estimators of Generalized Shannon’s Entropy

Shannon’s entropy is one of the building blocks of information theory and an essential aspect of Machine Learning (ML) methods (e.g., Random Forests). Yet, it is only finitely defined for distributions with fast decaying tails on a countable alphabet. The unboundedness of Shannon’s entropy over the general class of all distributions on an alphabet prevents its potential utility from being fully realized. To fill the void in the foundation of information theory, Zhang (2020) proposed generalized Shannon’s entropy, which is finitely defined everywhere. The plug-in estimator, adopted in almost all entropy-based ML method packages, is one of the most popular approaches to estimating Shannon’s entropy. The asymptotic distribution for Shannon’s entropy’s plug-in estimator was well studied in the existing literature. This paper studies the asymptotic properties for the plug-in estimator of generalized Shannon’s entropy on countable alphabets. The developed asymptotic properties require no assumptions on the original distribution. The proposed asymptotic properties allow for interval estimation and statistical tests with generalized Shannon’s entropy.