Estimating the Shannon entropy: recurrence plots versus symbolic dynamics

Recurrence plots were first introduced to quantify the recurrence properties of chaotic dynamics. A few years later, the recurrence quantification analysis was introduced to transform graphical representations into statistical analysis. Among the different measures introduced, a Shannon entropy was found to be correlated with the inverse of the largest Lyapunov exponent. The discrepancy between this and the usual interpretation of a Shannon entropy is solved here by using a new definition--still based on the recurrence plots--and it is verified that this new definition is correlated with the largest Lyapunov exponent, as expected from the Pesin conjecture. A comparison with a Shannon entropy computed from symbolic dynamics is also provided.     

Nonlinear signal analysis to understand the dynamics of the protein sequences

Recurrence plots are a useful tool to identify structure in a data set in a time resolved way qualitatively. Recurrence plots and its quantification has become an important research tool in the analysis of nonlinear dynamical systems. In the present work, we utilize the recurrence property to study the protein sequences. The sequences that we analyze belong to two distinct classes, viz., soluble proteins and proteins that form inclusion bodies when over expressed in Escherichia coli. We use Kyte-Doolittle hydrophobicity scale in the analysis. We study the underlying dynamics and extract the information which codes the essential class of a protein using simple statistical and global characteristics based features as well as some advanced features based on recurrence quantification. The extracted features are used in probability estimation using Gaussian Process Classification technique. The results give meaningful insights to the level of understanding the protein sequence dynamics.     

Recurrence quantification analysis and principal components in the detection of short complex signals

Recurrence plots were introduced to aid in the detection of signals in complicated data series. This effort was taken a step further by the quantification of recurrence plot elements. We now demonstrate the utility of combining recurrence quantification analysis with principal components analysis to allow for a probabilistic evaluation of the presence of deterministic signals in relatively short data lengths.     

Recurrence plot statistics and the effect of embedding

Recurrence plots provide a graphical representation of the recurrent patterns in a timeseries, the quantification of which is a relatively new field. Here we derive analytical expressions which relate the values of key statistics, notably determinism and entropy of line length distribution, to the correlation sum as a function of embedding dimension. These expressions are obtained by deriving the transformation which generates an embedded recurrence plot from an unembedded plot. A single unembedded recurrence plot thus provides the statistics of all possible embedded recurrence plots. If the correlation sum scales exponentially with embedding dimension, we show that these statistics are determined entirely by the exponent of the exponential. This explains the results of Iwanski and Bradley [J.S. Iwanski, E. Bradley, Recurrence plots of experimental data: to embed or not to embed? Chaos 8 (1998) 861–871] who found that certain recurrence plot statistics are apparently invariant to embedding dimension for certain low-dimensional systems. We also examine the relationship between the mutual information content of two timeseries and the common recurrent structure seen in their recurrence plots. This allows time-localized contributions to mutual information to be visualized. This technique is demonstrated using geomagnetic index data; we show that the AU and AL geomagnetic indices share half their information, and find the timescale on which mutual features appear.     

Measuring volatility in the Nordic spot electricity market using Recurrence Quantification Analysis

In this work, we have applied Recurrence Quantification Analysis (RQA)to data sets taken from the Nordic spot electricity market. Our main interest was in trying to correlate their volatility with variables obtained from the quantification of recurrence plots (RP). For this reason we have based our analysis on known historical events: the evolution of the Nord Pool market and climatic factors, i.e. dry and wet years, and we have compared several dispersion measures with RQA measures in correspondence of these events. The analysis suggests that two RQA measures: DET and LAM can be used as a measure of the inverse of the volatility. The main advantage of using DET and LAM is that these measures provide also information about the underlying dynamics. This fact is shown using shuffled and linear Gaussian surrogates of the real time series.     

两种群随机动力系统的信息熵和动力学研究

随机种群动力学模型是研究种群间以及种群与不确定性环境间相互作用的动力学行为的数学模型. 本文从概率密度以及信息熵流、熵产生的演化角度探讨了两种群生态系统的Itô (或Statonovich)意义下随机模型的动力学行为.利用Fokker-Planck方程及其边界条件 和信息熵定义导出信息熵流(平均散度)和熵产生的关系式,并通过数值路径积分法捕 捉到熵流的非线性变化趋势以及信息熵的极值点位置与概率密度的快速迁移和分岔的联系. 应用数值路径积分法计算结果表明Itô (或Statonovich)意义下两种随机模型的概率密度 和信息熵的极值点位置不同但演化趋势一致.     

On generalized entropy measures and pathways

Product probability property, known in the literature as statistical independence, is examined first. Then generalized entropies are introduced, all of which give generalizations to Shannon entropy. It is shown that the nature of the recursivity postulate automatically determines the logarithmic functional form for Shannon entropy. Due to the logarithmic nature, Shannon entropy naturally gives rise to additivity, when applied to situations having product probability property. It is argued that the natural process is non-additivity, important, for example, in statistical mechanics [C. Tsallis, Possible generalization of Boltzmann-Gibbs statistics, J. Stat. Phys. 52 (1988) 479-487; E.G.D. Cohen, Boltzmann and Einstein: statistics and dynamics—an unsolved problem, Pramana 64 (2005) 635-643.], even in product probability property situations and additivity can hold due to the involvement of a recursivity postulate leading to a logarithmic function. Generalized entropies are introduced and some of their properties are examined. Situations are examined where a generalized entropy of order α leads to pathway models, exponential and power law behavior and related differential equations. Connection of this entropy to Kerridge's measure of “inaccuracy” is also explored.     

Randomizing nonlinear maps via symbolic dynamics

Pseudo Random Number Generators (PRNG) have attracted intense attention due to their obvious importance for many branches of science and technology. A randomizing technique is a procedure designed to improve the PRNG randomness degree according the specific requirements. It is obviously important to quantify its effectiveness. In order to classify randomizing techniques based on a symbolic dynamics’ approach, we advance a novel, physically motivated representation based on the statistical properties of chaotic systems. Recourse is made to a plane that has as coordinates (i) the Shannon entropy and (ii) a form of the statistical complexity measure. Each statistical quantifier incorporates a different probability distribution function, generating thus a representation that (i) sheds insight into just how each randomizing technique operates and also (ii) quantifies its effectiveness. Using the Logistic Map and the Three Way Bernoulli Map as typical examples of chaotic dynamics it is shown that our methodology allows for choosing the more convenient randomizing technique in each instance. Comparison with measures of complexity based on diagonal lines on the recurrence plots [N. Marwan, M.C. Romano, M. Thiel, J. Kurths, Phys. Rep. 438 (2007) 237] support the main conclusions of this paper.     

Shannon Entropy in LS-Coupled Configuration Space for Ni-like Isoelectronic Sequence

The Shannon entropy in an LS-coupled configuration space has been calculated through a transformation from that in a jj-coupled configuration space for a Ni-like isoelectronic sequence. The sudden change of Shannon entropy, information exchange, eigenlevel anticrossing, and strong configuration interaction have been presented for adjacent levels. It is shown that eigenlevel anticrossing is a sufficient and necessary condition for the sudden change of Shannon entropy, and both of them are a sufficient condition for information exchange, which is the same as the case of the jj-coupled configuration space. It is found that the structure of sudden change from jj-coupled into LS-coupled configuration spaces through the LS-jj transformation is invariant for Shannon entropy along the isoelectronic sequence. What is more, in an LS-coupled configuration space, there are a large number of information exchanges between energy levels whether with or without strong configuration interaction, and most of the ground and single excited states of Ni-like ions are more suitable to be described by a jj-coupled or other configuration basis set instead of an LS-coupled configuration basis set according to the configuration mixing coefficients and their Shannon entropy. In this sense, Shannon entropy can also be used to measure the applicability of a configuration basis set or the purity of atomic state functions in different coupling schemes.     

From Chaos to Ordering: New Studies in the Shannon Entropy of 2D Patterns

Properties of the Voronoi tessellations arising from random 2D distribution points are reported. We applied an iterative procedure to the Voronoi diagrams generated by a set of points randomly placed on the plane. The procedure implied dividing the edges of Voronoi cells into equal or random parts. The dividing points were then used to construct the following Voronoi diagram. Repeating this procedure led to a surprising effect of the positional ordering of Voronoi cells, reminiscent of the formation of lamellae and spherulites in linear semi-crystalline polymers and metallic glasses. Thus, we can conclude that by applying even a simple set of rules to a random set of seeds, we can introduce order into an initially disordered system. At the same time, the Shannon (Voronoi) entropy showed a tendency to attain values that are typical for completely random patterns; thus, the Shannon (Voronoi) entropy does not distinguish the short-range ordering. The Shannon entropy and the continuous measure of symmetry of the patterns demonstrated the distinct asymptotic behavior, while approaching the close saturation values with the increase in the number of iteration steps. The Shannon entropy grew with the number of iterations, whereas the continuous measure of symmetry of the same patterns demonstrated the opposite asymptotic behavior. The Shannon (Voronoi) entropy is not an unambiguous measure of order in the 2D patterns. The more symmetrical patterns may demonstrate the higher values of the Shannon entropy.     

Stochastic Properties of Fractional Generalized Cumulative Residual Entropy and Its Extensions

The fractional generalized cumulative residual entropy (FGCRE) has been introduced recently as a novel uncertainty measure which can be compared with the fractional Shannon entropy. Various properties of the FGCRE have been studied in the literature. In this paper, further results for this measure are obtained. The results include new representations of the FGCRE and a derivation of some bounds for it. We conduct a number of stochastic comparisons using this measure and detect the connections it has with some well-known stochastic orders and other reliability measures. We also show that the FGCRE is the Bayesian risk of a mean residual lifetime (MRL) under a suitable prior distribution function. A normalized version of the FGCRE is considered and its properties and connections with the Lorenz curve ordering are studied. The dynamic version of the measure is considered in the context of the residual lifetime and appropriate aging paths.     

Stochastic analysis of recurrence plots with applications to the detection of deterministic signals

Recurrence plots have been widely used for a variety of purposes such as analyzing dynamical systems, denoising, as well as detection of deterministic signals embedded in noise. Though it has been postulated previously that recurrence plots contain time correlation information here we make the relationship between unthresholded recurrence plots and the covariance of a random process more precise. Computations using examples from harmonic processes, autoregressive models, and outputs from nonlinear systems are shown to illustrate this relationship. Finally, the use of recurrence plots for detection of deterministic signals in the presence of noise is investigated and compared to traditional signal detection methods based on the likelihood ratio test. Results using simulated data show that detectors based on certain statistics derived from recurrence plots are sub-optimal when compared to well-known detectors based on the likelihood ratio.     

The Fisher information measure and Shannon entropy for particulate matter measurements

We investigated the dynamics of particulate matter data, recorded in Tito, a small industrial area of southern Italy. The analysis of these signals was performed using the Fisher information measure (FIM), which is a powerful tool for investigating complex and nonstationary signals, and the Shannon entropy, which is a well-known tool for investigating the degree of disorder in dynamical systems. Our results point to an increase of disorder and complexity from fine to coarse particulates.     

The Odyssey of Entropy: Cryptography

After being introduced by Shannon as a measure of disorder and unavailable information, the notion of entropy has found its applications in a broad range of scientific disciplines. In this paper, we present a systematic review on the applications of entropy and related information-theoretical concepts in the design, implementation and evaluation of cryptographic schemes, algorithms, devices and systems. Moreover, we study existing trends, and establish a roadmap for future research in these areas.     

A Two-Parameter Fractional Tsallis Decision Tree

Decision trees are decision support data mining tools that create, as the name suggests, a tree-like model. The classical C4.5 decision tree, based on the Shannon entropy, is a simple algorithm to calculate the gain ratio and then split the attributes based on this entropy measure. Tsallis and Renyi entropies (instead of Shannon) can be employed to generate a decision tree with better results. In practice, the entropic index parameter of these entropies is tuned to outperform the classical decision trees. However, this process is carried out by testing a range of values for a given database, which is time-consuming and unfeasible for massive data. This paper introduces a decision tree based on a two-parameter fractional Tsallis entropy. We propose a constructionist approach to the representation of databases as complex networks that enable us an efficient computation of the parameters of this entropy using the box-covering algorithm and renormalization of the complex network. The experimental results support the conclusion that the two-parameter fractional Tsallis entropy is a more sensitive measure than parametric Renyi, Tsallis, and Gini index precedents for a decision tree classifier.     

A Quantitative Comparison between Shannon and Tsallis–Havrda–Charvat Entropies Applied to Cancer Outcome Prediction

In this paper, we propose to quantitatively compare loss functions based on parameterized Tsallis–Havrda–Charvat entropy and classical Shannon entropy for the training of a deep network in the case of small datasets which are usually encountered in medical applications. Shannon cross-entropy is widely used as a loss function for most neural networks applied to the segmentation, classification and detection of images. Shannon entropy is a particular case of Tsallis–Havrda–Charvat entropy. In this work, we compare these two entropies through a medical application for predicting recurrence in patients with head–neck and lung cancers after treatment. Based on both CT images and patient information, a multitask deep neural network is proposed to perform a recurrence prediction task using cross-entropy as a loss function and an image reconstruction task. Tsallis–Havrda–Charvat cross-entropy is a parameterized cross-entropy with the parameter α . Shannon entropy is a particular case of Tsallis–Havrda–Charvat entropy for α=1 . The influence of this parameter on the final prediction results is studied. In this paper, the experiments are conducted on two datasets including in total 580 patients, of whom 434 suffered from head–neck cancers and 146 from lung cancers. The results show that Tsallis–Havrda–Charvat entropy can achieve better performance in terms of prediction accuracy with some values of α .     

A comparison of the Shannon and Kullback information measures

Two widely used information measures are compared. It is shown that the Kullback measure, unlike the Shannon measure, provides the basis for a consistent theory of information which extends to continuous sample spaces and to nonconstant prior distributions. It is shown that the Kullback measure is a generalization of the Shannon measure, and that the Kullback measure has more reasonable additivity properties than does the Shannon measure. The results lend support to Jaynes's entropy maximization procedure.     

Informational analysis of seismic sequences by applying the Fisher Information Measure and the Shannon entropy: An application to the 2004–2010 seismicity of Aswan area (Egypt)

The interevent-time (IET) and interevent-distance (IED) series of seismic events occurred at Aswan area (Egypt) from 2004 to 2010 were investigated by means of the Fisher Information Measure and the Shannon entropy. The analysis was performed varying the depth and the magnitude thresholds. The results point out to an increase of level of organization and order with the decrease of magnitude threshold and the increase of depth threshold for the IET series, while the IED series are characterized by a level of uncertainty approximately constant with the threshold magnitude. The complexity measure, calculated as the product of the Fisher Information Measure and the Shannon entropy power, presents very similar pattern for both the types of seismic series, indicating an increasing complexity with the decrease of the threshold magnitude and the increase of the threshold depth.     

Application of permutation entropy method in the analysis of chaotic,noisy, and chaotic noisy series

We have considered a permutation entropy method for analyzing chaotic, noisy, and chaotic noisy series. We have introduced the concept of permutation entropy from a survey of some features of information entropy (Shannon entropy), described the algorithm for its calculation, and indicated the advantages of this approach in the analysis of time series; the application of this method in the analysis of various model systems and experimental data has also been demonstrated.     

Editorial

Symbolic dynamics is a powerful tool in the study of dynamical systems. The purpose of symbolic dynamics is to provide a simplified picture of complicated dynamics, that gives some insight into its complexity. To this end, the state space of the system is partitioned in a finite number of pieces, and the exact trajectories of individual points are traded off by the trajectory relative to that partition. These so-called coarse-grained trajectories turn out to be realisations of a stationary random process with a finite alphabet. In particular, the entropy of a dynamical system can be approximated by the Shannon entropy of any of its symbolic dynamics (the finer the partition, the better the approximation). Today, symbolic dynamics is an independent field of theoretical physics and applied mathematics with applications to such important disciplines as cryptology, time series analysis, and data-compression.