Permutation complexity via duality between values and orderings

We study the permutation complexity of finite-state stationary stochastic processes based on a duality between values and orderings between values. First, we establish a duality between the set of all words of a fixed length and the set of all permutations of the same length. Second, on this basis, we give an elementary alternative proof of the equality between the permutation entropy rate and the entropy rate for a finite-state stationary stochastic processes first proved in [J.M. Amigó, M.B. Kennel, L. Kocarev, The permutation entropy rate equals the metric entropy rate for ergodic information sources and ergodic dynamical systems, Physica D 210 (2005) 77-95]. Third, we show that further information on the relationship between the structure of values and the structure of orderings for finite-state stationary stochastic processes beyond the entropy rate can be obtained from the established duality. In particular, we prove that the permutation excess entropy is equal to the excess entropy, which is a measure of global correlation present in a stationary stochastic process, for finite-state stationary ergodic Markov processes.     

Symbolic transfer entropy rate is equal to transfer entropy rate for bivariate finite-alphabet stationary ergodic Markov processes

Transfer entropy is a measure of the magnitude and the direction of information flow between jointly distributed stochastic processes. In recent years, its permutation analogues are considered in the literature to estimate the transfer entropy by counting the number of occurrences of orderings of values, not the values themselves. It has been suggested that the method of permutation is easy to implement, computationally low cost and robust to noise when applying to real world time series data. In this paper, we initiate a theoretical treatment of the corresponding rates. In particular, we consider the transfer entropy rate and its permutation analogue, the symbolic transfer entropy rate, and show that they are equal for any bivariate finite-alphabet stationary ergodic Markov process. This result is an illustration of the duality method introduced in [T. Haruna, K. Nakajima, Physica D 240, 1370 (2011)]. We also discuss the relationship among the transfer entropy rate, the time-delayed mutual information rate and their permutation analogues.     

A Review of Shannon and Differential Entropy Rate Estimation

In this paper, we present a review of Shannon and differential entropy rate estimation techniques. Entropy rate, which measures the average information gain from a stochastic process, is a measure of uncertainty and complexity of a stochastic process. We discuss the estimation of entropy rate from empirical data, and review both parametric and non-parametric techniques. We look at many different assumptions on properties of the processes for parametric processes, in particular focussing on Markov and Gaussian assumptions. Non-parametric estimation relies on limit theorems which involve the entropy rate from observations, and to discuss these, we introduce some theory and the practical implementations of estimators of this type.     

Sequential Gibbs Measures and Factor Maps

We define the notion of sequential Gibbs measures, inspired by on the classical notion of Gibbs measures and recent examples from the study of non-uniform hyperbolic dynamics. Extending previous results of Kempton and Pollicott (Factors of Gibbs measures for full shifts, entropy of hidden Markov processes and connections to dynamical systems, Cambridge University Press, Cambridge, 2011) and Chazottes and Ugalde (On the preservation of Gibbsianness under symbol amalgamation, entropy of hidden Markov processes and connections to dynamical systems, Cambridge University Press, Cambridge, 2011), we show that the images of one block factor maps of a sequential Gibbs measure are also a sequential Gibbs measure, with the same sequence of Gibbs times. We obtain some estimates on the regularity of the potential of the image measure at almost every point.     

Informational and Causal Architecture of Continuous-time Renewal Processes

We introduce the minimal maximally predictive models (\(\epsilon \text{-machines }\)) of processes generated by certain hidden semi-Markov models. Their causal states are either discrete, mixed, or continuous random variables and causal-state transitions are described by partial differential equations. As an application, we present a complete analysis of the \(\epsilon \text{-machines }\) of continuous-time renewal processes. This leads to closed-form expressions for their entropy rate, statistical complexity, excess entropy, and differential information anatomy rates.     

A New Method for Calculation of Single Seismic Phase of Cylindrically Multilayered Media Including Liquid Interlayer

A new method based on generalized reflection and transmission （R/T） coefficients method is proposed to calculate the single seismic phase （SSP） of cylindrically multilayered media including liquid interlayer. The use of normalization factors and normalized Lamé coefficients makes the algorithm stable numerically. Using the modified R/T matrices, we derive the iterative expressions of generalized R/T matrices, and by using the iterative relation we determine the SSP of each interface and the full waveforms. To show the superiority of this new approach for investigating of reflection and transmission properties of cylindrically multilayered media, we simulate the full waveforms and SSPs of cased hole model with annulus I （casing-cement interface） channelling （or, cross-flow）. The generalized reflection coefficient spectra and SSPs of interfaces obtained show the propagation mechanism of each component of full waveform clearly.     

How hidden are hidden processes? A primer on crypticity and entropy convergence

We investigate a stationary process's crypticity--a measure of the difference between its hidden state information and its observed information--using the causal states of computational mechanics. Here, we motivate crypticity and cryptic order as physically meaningful quantities that monitor how hidden a hidden process is. This is done by recasting previous results on the convergence of block entropy and block-state entropy in a geometric setting, one that is more intuitive and that leads to a number of new results. For example, we connect crypticity to how an observer synchronizes to a process. We show that the block-causal-state entropy is a convex function of block length. We give a complete analysis of spin chains. We present a classification scheme that surveys stationary processes in terms of their possible cryptic and Markov orders. We illustrate related entropy convergence behaviors using a new form of foliated information diagram. Finally, along the way, we provide a variety of interpretations of crypticity and cryptic order to establish their naturalness and pervasiveness. This is also a first step in developing applications in spatially extended and network dynamical systems.     

Multiresolution information measures applied to speech recognition

Considerable advances in automatic speech recognition have been made in the last decades, thanks specially to the use of hidden Markov models. In the field of speech signal analysis, different techniques have been developed. However, deterioration in the performance of the speech recognizers has been observed when they are trained with clean signal and tested with noisy signals. This is still an open problem in this field. Continuous multiresolution entropy has been shown to be robust to additive noise in applications to different physiological signals. In previous works we have included Shannon and Tsallis entropies, and their corresponding divergences, in different speech analysis and recognition systems. In this paper we present an extension of the continuous multiresolution entropy to different divergences and we propose them as new dimensions for the pre-processing stage of a speech recognition system. This approach takes into account information about changes in the dynamics of speech signal at different scales. The methods proposed here are tested with speech signals corrupted with babble and white noise. Their performance is compared with classical mel cepstral parametrization. The results suggest that these continuous multiresolution entropy related measures provide valuable information to the speech recognition system and that they could be considered to be included as an extra component in the pre-processing stage.     

An approach to comparing Kolmogorov-Sinai and permutation entropy

In this paper we discuss the relationship between permutation entropy and Kolmogorov-Sinai entropy in the one-dimensional case. For this, we consider partitions of the state space of a dynamical system using ordinal patterns of order (d + n? 1) on the one hand, and using n-letter words of ordinal patterns of order d on the other hand. The answer to the question of how different these partitions are provides an approach to comparing the entropies.     

Topological permutation entropy

Permutation entropy quantifies the diversity of possible ordering of the successively observed values a random or deterministic system can take, just as Shannon entropy quantifies the diversity of the values themselves. When the observable or state variable has a natural order relation, making permutation entropy possible to compute, then the asymptotic rate of growth in permutation entropy with word length forms an alternative means of describing the intrinsic entropy rate of a source. Herein, extending a previous result on metric entropy rate, we show that the topological permutation entropy rate for expansive maps equals the conventional topological entropy rate familiar from symbolic dynamics. This result is not limited to one-dimensional maps.     

A measure of statistical complexity based on predictive information with application to finite spin systems

We propose the binding information as an information theoretic measure of complexity between multiple random variables, such as those found in the Ising or Potts models of interacting spins, and compare it with several previously proposed measures of statistical complexity, including excess entropy, Bialek et al.?s predictive information, and the multi-information. We discuss and prove some of the properties of binding information, particularly in relation to multi-information and entropy, and show that, in the case of binary random variables, the processes which maximise binding information are the ‘parity’ processes. The computation of binding information is demonstrated on Ising models of finite spin systems, showing that various upper and lower bounds are respected and also that there is a strong relationship between the introduction of high-order interactions and an increase of binding-information. Finally we discuss some of the implications this has for the use of the binding information as a measure of complexity.     

A multifractal analysis of equilibrium measures for conformal expanding maps and Moran-like geometric constructions

In this paper we establish the complete multifractal formalism for equilibrium measures for Hölder continuous conformal expanding maps andexpanding Markov Moran-like geometric constructions. Examples include Markov maps of an interval, beta transformations of an interval, rational maps with hyperbolic Julia sets, and conformal toral endomorphisms. We also construct a Hölder continuous homeomorphism of a compact metric space with an ergodic invariant measure of positive entropy for which the dimension spectrum is not convex, and hence the multifractal formalism fails.     

Steady state speed distribution analysis for a combined cellular automaton traffic model

Cellular Automaton （CA） based traffic flow models have been extensively studied due to their effectiveness and simplicity in recent years. This paper develops a discrete time Markov chain （DTMC） analytical framework for a Nagel-Schreckenberg and Fukui Ishibashi combined CA model （W^2H traffic flow model） from microscopic point of view to capture the macroscopic steady state speed distributions. The inter-vehicle spacing Maxkov chain and the steady state speed Markov chain are proved to be irreducible and ergodic. The theoretical speed probability distributions depending on the traffic density and stochastic delay probability are in good accordance with numerical simulations. The derived fundamental diagram of the average speed from theoretical speed distributions is equivalent to the results in the previous work.     

Approximation for shell-model level densities: Ergodicity

Using the supersymmetry method, we show that the maximum entropy approach for the calculation of nuclear shell-model partial and total level densities, developed in a previous paper, is ergodic.On leave from the Charles University, Prague, Czech Republic     

Permutation entropy and detrend fluctuation analysis for the natural complexity of cardiac heart interbeat signals

We compute fractal dimension and permutation entropy for healthy and people who have experienced heart failure. Our result shows that permutation entropy is a suitable approach as well as detrend fluctuation analysis (DFA). The result of DFA shows that the fractal dimensions for healthy and heart failure are different as well as the permutation entropy result. The fluctuation value for permutation entropy for an individual who has experienced heart failure is bigger than for a healthy person. There is some specific change in the interbeat signal of a person who has experienced heart failure, but there is not previous trend for a healthy person.     

Fluctuation Theorems for Entropy Production and Heat Dissipation in Periodically Driven Markov Chains

Asymptotic fluctuation theorems are statements of a Gallavotti-Cohen symmetry in the rate function of either the time-averaged entropy production or heat dissipation of a process. Such theorems have been proved for various general classes of continuous-time deterministic and stochastic processes, but always under the assumption that the forces driving the system are time independent, and often relying on the existence of a limiting ergodic distribution. In this paper we extend the asymptotic fluctuation theorem for the first time to inhomogeneous continuous-time processes without a stationary distribution, considering specifically a finite state Markov chain driven by periodic transition rates. We find that for both entropy production and heat dissipation, the usual Gallavotti-Cohen symmetry of the rate function is generalized to an analogous relation between the rate functions of the original process and its corresponding backward process, in which the trajectory and the driving protocol have been time-reversed. The effect is that spontaneous positive fluctuations in the long time average of each quantity in the forward process are exponentially more likely than spontaneous negative fluctuations in the backward process, and vice-versa, revealing that the distributions of fluctuations in universes in which time moves forward and backward are related. As an additional result, the asymptotic time-averaged entropy production is obtained as the integral of a periodic entropy production rate that generalizes the constant rate pertaining to homogeneous dynamics.     

Mixing Bandt-Pompe and Lempel-Ziv approaches: another way to analyze the complexity of continuous-state sequences

In this paper, we propose to mix the approach underlying Bandt-Pompe permutation entropy with Lempel-Ziv complexity, to design what we call Lempel-Ziv permutation complexity. The principle consists of two steps: (i) transformation of a continuous-state series that is intrinsically multivariate or arises from embedding into a sequence of permutation vectors, where the components are the positions of the components of the initial vector when re-arranged; (ii) performing the Lempel-Ziv complexity for this series of ‘symbols’, as part of a discrete finite-size alphabet. On the one hand, the permutation entropy of Bandt-Pompe aims at the study of the entropy of such a sequence; i.e., the entropy of patterns in a sequence (e.g., local increases or decreases). On the other hand, the Lempel-Ziv complexity of a discrete-state sequence aims at the study of the temporal organization of the symbols (i.e., the rate of compressibility of the sequence). Thus, the Lempel-Ziv permutation complexity aims to take advantage of both of these methods. The potential from such a combined approach – of a permutation procedure and a complexity analysis – is evaluated through the illustration of some simulated data and some real data. In both cases, we compare the individual approaches and the combined approach.     

Exploration of Entropy Pair Functional Theory

Evaluation of the entropy from molecular dynamics (MD) simulation remains an outstanding challenge. The standard approach requires thermodynamic integration across a series of simulations. Recent work Nicholson et al. demonstrated the ability to construct a functional that returns excess entropy, based on the pair correlation function (PCF); it was capable of providing, with acceptable accuracy, the absolute excess entropy of iron simulated with a pair potential in both fluid and crystalline states. In this work, the general applicability of the Entropy Pair Functional Theory (EPFT) approach is explored by applying it to three many-body interaction potentials. These potentials are state of the art for large scale models for the three materials in this study: Fe modelled with a modified embedded atom method (MEAM) potential, Cu modelled with an MEAM and Si modelled with a Tersoff potential. We demonstrate the robust nature of EPFT in determining excess entropy for diverse systems with many-body interactions. These are steps toward a universal Entropy Pair Functional, EPF, that can be applied with confidence to determine the entropy associated with sophisticated optimized potentials and first principles simulations of liquids, crystals, engineered structures, and defects.