Rényi and Tsallis entropies for incomplete or overcomplete systems of events

In this article, Shannon, Rényi and Tsallis entropies are considered for a system of events characterized by an arbitrary probability distribution P that can be incomplete, complete or overcomplete. After a suitable transformation that leads to the escort probabilities of P, these can be written as the canonical probability distribution for a set of pseudo-energies (Hartley information, E_n=−lnP_n) and a dimensionless parameter q that plays the role of thermodynamics β. Several relations between the entropies are presented, including the analysis of compound systems. The method is illustrated with an example.     

On the way towards a generalized entropy maximization procedure

We propose a generalized entropy maximization procedure, which takes into account the generalized averaging procedures and information gain definitions underlying the generalized entropies. This novel generalized procedure is then applied to Rényi and Tsallis entropies. The generalized entropy maximization procedure for Rényi entropies results in the exponential stationary distribution asymptotically for q∈(0,1] in contrast to the stationary distribution of the inverse power law obtained through the ordinary entropy maximization procedure. Another result of the generalized entropy maximization procedure is that one can naturally obtain all the possible stationary distributions associated with the Tsallis entropies by employing either ordinary or q-generalized Fourier transforms in the averaging procedure.     

Escort entropies and divergences and related canonical distribution

We discuss two families of two-parameter entropies and divergences, derived from the standard Rényi and Tsallis entropies and divergences. These divergences and entropies are found as divergences or entropies of escort distributions. Exploiting the nonnegativity of the divergences, we derive the expression of the canonical distribution associated to the new entropies and a observable given as an escort-mean value. We show that this canonical distribution extends, and smoothly connects, the results obtained in nonextensive thermodynamics for the standard and generalized mean value constraints.     

Tsallis entropy measure of noise-aided information transmission in a binary channel

Noise-aided information transmission via stochastic resonance is shown and analyzed in a binary channel by means of information measures based on the Tsallis entropy. The analysis extends the classic reference of binary information transmission based on the Shannon entropy, and also parallels a recent study based on the Rényi entropy. The conditions for a maximally pronounced stochastic resonance identify optimal Tsallis measures. The study involves a correspondence between Tsallis and Rényi information measures, specially relevant to the characterization of stochastic resonance, and establishing that for such effects identical properties are shared in common by both Tsallis and Rényi measures.     

Pathway model,superstatistics, Tsallis statistics,and a generalized measure of entropy

The pathway model of Mathai [A pathway to matrix-variate gamma and normal densities, Linear Algebra Appl. 396 (2005) 317–328] is shown to be inferable from the maximization of a certain generalized entropy measure. This entropy is a variant of the generalized entropy of order α$\alpha$, considered in Mathai and Rathie [Basic Concepts in Information Theory and Statistics: Axiomatic Foundations and Applications, Wiley Halsted, New York and Wiley Eastern, New Delhi, 1975], and it is also associated with Shannon, Boltzmann–Gibbs, Rényi, Tsallis, and Havrda–Charvát entropies. The generalized entropy measure introduced here is also shown to have interesting statistical properties and it can be given probabilistic interpretations in terms of inaccuracy measure, expected value, and information content in a scheme. Particular cases of the pathway model are shown to be Tsallis statistics [C. Tsallis, Possible generalization of Boltzmann-Gibbs statistics, J. Stat. Phys. 52 (1988) 479–487] and superstatistics introduced by Beck and Cohen [Superstatistics, Physica A 322 (2003) 267–275]. The pathway model's connection to fractional calculus is illustrated by considering a fractional reaction equation.     

On Rényi Permutation Entropy

Among various modifications of the permutation entropy defined as the Shannon entropy of the ordinal pattern distribution underlying a system, a variant based on Rényi entropies was considered in a few papers. This paper discusses the relatively new concept of Rényi permutation entropies in dependence of non-negative real number q parameterizing the family of Rényi entropies and providing the Shannon entropy for q=1. Its relationship to Kolmogorov–Sinai entropy and, for q=2, to the recently introduced symbolic correlation integral are touched.     

Critique of multinomial coefficients method for evaluating Tsallis and Rényi entropies

Oikonomou [Th. Oikonomou, Physica A 386 (2007) 119] has published a calculation which purports to show that the Tsallis and Rényi entropies can be obtained from the generalized multinomial coefficients. In this paper, we prove that the method of generalized multinomial coefficients failed to determine the Tsallis entropy at equilibrium. Moreover, it is shown that Oikonomou’s analysis contains mistakes which led to misleading statements related to the Jaynes principle of maximum entropy, the Tsallis and the Rényi statistics.     

Source coding with escort distributions and Rényi entropy bounds

We discuss the interest of escort distributions and Rényi entropy in the context of source coding. We first recall a source coding theorem by Campbell relating a generalized measure of length to the Rényi-Tsallis entropy. We show that the associated optimal codes can be obtained using considerations on escort-distributions. We propose a new family of measure of length involving escort-distributions and we show that these generalized lengths are also bounded below by the Rényi entropy. Furthermore, we obtain that the standard Shannon codes lengths are optimum for the new generalized lengths measures, whatever the entropic index. Finally, we show that there exists in this setting an interplay between standard and escort distributions.     

Rényi entropy and quantum phase transition in the Dicke model

The Rényi entropies of the Dicke model are presented. This quantum-optical model describes a single-mode bosonic field interacting with an ensemble of N two-level atoms. There is a quantum phase transition in the N→∞ limit. It is shown that there is an abrupt change in the Rényi entropy of order β at the transition point. Around the critical value of the coupling strength λc the Rényi entropy is proportional to the logarithm of the characteristic length and diverges as ln|λc−λ| for any order β. The pseudocapacity defined here in analogy with the heat capacity exhibits the phase transition. The critical exponent for the Dicke model is found to be 1 for any value of the parameter β.     

Nonextensive random-matrix theory based on Kaniadakis entropy

The joint eigenvalue distributions of random-matrix ensembles are derived by applying the principle maximum entropy to the Rényi, Abe and Kaniadakis entropies. While the Rényi entropy produces essentially the same matrix-element distributions as the previously obtained expression by using the Tsallis entropy, and the Abe entropy does not lead to a closed form expression, the Kaniadakis entropy leads to a new generalized form of the Wigner surmise that describes a transition of the spacing distribution from chaos to order. This expression is compared with the corresponding expression obtained by assuming Tsallis' entropy as well as the results of a previous numerical experiment.     

Tsallis, Rényi and nonextensive Gaussian entropy derived from the respective multinomial coefficients

We explore the generalization of the ordinary multinomial coefficient, based on the deformed q-multiplication and q-division. Aim of this study is to construct the appropriate multinomial coefficients, from which one can obtain the Tsallis, Rényi and nonextensive Gaussian entropy, respectively. We show that for all three above entropies there are two possible ways to define the generalized multinomial coefficient. Its consequence is discussed.     

Entropy and information characteristics of qubit states

Shannon entropy, Rényi entropy, and Tsallis entropy are discussed for the tomographic probability distributions of qubit states. Relative entropy and its properties are considered for the tomographic probability distribution describing the states of multi-spin systems. New inequalities for Hermite polynomials are obtained.     

Instabilities of Rényi entropies

We show that for systems with a large number of microstates Rényi entropies do not represent experimentally observable quantities except the Rényi entropy that coincides with the Shannon entropy.Work supported by the DFG (1978); author is recipient of a Feodor-Lynen grant from the Alexander von Humboldt Stiftung.     

Comment on “Critique of multinomial coefficient method for evaluating Tsallis and Rényi entropies” by A.S. Parvan

Parvan [A.S. Parvan, Critique of multinomial coefficients method for evaluating Tsallis and Rényi entropies, Physica A 389 (2010) 5645] has recently presented some calculations in order to demonstrate the incorrectness of the results obtained from the generalized multinomial coefficients (GMC) presented in Oikonomou (2007) [1]. According to Parvan, the aforementioned approach of studying maximum entropy probability distributions is erroneous. In this comment I demonstrate that Parvan’s arguments do not hold true and that the obtained results from GMC do not present either mathematical or physical discrepancies.     

Tsallis distribution as a standard maximum entropy solution with ‘tail’ constraint

We show that Tsallis' distributions can be derived from the standard (Shannon) maximum entropy setting, by incorporating a constraint on the divergence between the distribution and another distribution imagined as its tail. In this setting, we find an underlying entropy which is the Rényi entropy. Furthermore, escort distributions and generalized means appear as a direct consequence of the construction. Finally, the “maximum entropy tail distribution” is identified as a Generalized Pareto Distribution.     

Acoustic classification of Australian anurans based on hybrid spectral-entropy approach

A new hybrid method for automated frog sound identification, using spectral centroid, Shannon entropy and Rényi entropy is proposed. The advantage of using entropy based information theoretic approach for analyzing complexity of bioacoustics signals in animal vocalization is discussed. Sound samples from nine species of Microhylidae frogs are first segmented into syllables. Fourier spectral centroid, Shannon entropy and Rényi entropy of the syllables are then determined. Finally, nonparametric k-th nearest neighbour (k-NN) classifier is used to recognize the frog species based on these three extracted features. Result shows that the k-NN classifier based on these selected features is capable to identify the species of the frogs with an average accuracy of 98%. It is found that the accuracy reduces significantly only when the noise levels higher than −20 dB.     

A Two-Parameter Fractional Tsallis Decision Tree

Decision trees are decision support data mining tools that create, as the name suggests, a tree-like model. The classical C4.5 decision tree, based on the Shannon entropy, is a simple algorithm to calculate the gain ratio and then split the attributes based on this entropy measure. Tsallis and Renyi entropies (instead of Shannon) can be employed to generate a decision tree with better results. In practice, the entropic index parameter of these entropies is tuned to outperform the classical decision trees. However, this process is carried out by testing a range of values for a given database, which is time-consuming and unfeasible for massive data. This paper introduces a decision tree based on a two-parameter fractional Tsallis entropy. We propose a constructionist approach to the representation of databases as complex networks that enable us an efficient computation of the parameters of this entropy using the box-covering algorithm and renormalization of the complex network. The experimental results support the conclusion that the two-parameter fractional Tsallis entropy is a more sensitive measure than parametric Renyi, Tsallis, and Gini index precedents for a decision tree classifier.     

Alternative Entropy Measures and Generalized Khinchin–Shannon Inequalities

The Khinchin–Shannon generalized inequalities for entropy measures in Information Theory, are a paradigm which can be used to test the Synergy of the distributions of probabilities of occurrence in physical systems. The rich algebraic structure associated with the introduction of escort probabilities seems to be essential for deriving these inequalities for the two-parameter Sharma–Mittal set of entropy measures. We also emphasize the derivation of these inequalities for the special cases of one-parameter Havrda–Charvat’s, Rényi’s and Landsberg–Vedral’s entropy measures.     

The superdiffusion entropy production paradox in the space-fractional case for extended entropies

Contrary to intuition, entropy production rates grow as reversible, wave-like behavior is approached. This paradox was discovered in time-fractional diffusion equations. It was found to persist for extended entropies and for space-fractional diffusion as well. This paper completes the possibilities by showing that the paradox persists for Tsallis and Rényi entropies in the space-fractional case. Complications arising due to the heavy tail solutions of space-fractional diffusion equations are discussed in detail.     

Rényi’s information transfer between financial time series

In this paper, we quantify the statistical coherence between financial time series by means of the Rényi entropy. With the help of Campbell’s coding theorem, we show that the Rényi entropy selectively emphasizes only certain sectors of the underlying empirical distribution while strongly suppressing others. This accentuation is controlled with Rényi’s parameter q$q$. To tackle the issue of the information flow between time series, we formulate the concept of Rényi’s transfer entropy as a measure of information that is transferred only between certain parts of underlying distributions. This is particularly pertinent in financial time series, where the knowledge of marginal events such as spikes or sudden jumps is of a crucial importance. We apply the Rényian information flow to stock market time series from 11 world stock indices as sampled at a daily rate in the time period 02.01.1990–31.12.2009. Corresponding heat maps and net information flows are represented graphically. A detailed discussion of the transfer entropy between the DAX and S&P500 indices based on minute tick data gathered in the period 02.04.2008–11.09.2009 is also provided. Our analysis shows that the bivariate information flow between world markets is strongly asymmetric with a distinct information surplus flowing from the Asia–Pacific region to both European and US markets. An important yet less dramatic excess of information also flows from Europe to the US. This is particularly clearly seen from a careful analysis of Rényi information flow between the DAX and S&P500 indices.