Entropy: From Thermodynamics to Information Processing

Entropy is a concept that emerged in the 19th century. It used to be associated with heat harnessed by a thermal machine to perform work during the Industrial Revolution. However, there was an unprecedented scientific revolution in the 20th century due to one of its most essential innovations, i.e., the information theory, which also encompasses the concept of entropy. Therefore, the following question is naturally raised: “what is the difference, if any, between concepts of entropy in each field of knowledge?” There are misconceptions, as there have been multiple attempts to conciliate the entropy of thermodynamics with that of information theory. Entropy is most commonly defined as “disorder”, although it is not a good analogy since “order” is a subjective human concept, and “disorder” cannot always be obtained from entropy. Therefore, this paper presents a historical background on the evolution of the term “entropy”, and provides mathematical evidence and logical arguments regarding its interconnection in various scientific areas, with the objective of providing a theoretical review and reference material for a broad audience.     

Provably Secure Symmetric Private Information Retrieval with Quantum Cryptography

Private information retrieval (PIR) is a database query protocol that provides user privacy in that the user can learn a particular entry of the database of his interest but his query would be hidden from the data centre. Symmetric private information retrieval (SPIR) takes PIR further by additionally offering database privacy, where the user cannot learn any additional entries of the database. Unconditionally secure SPIR solutions with multiple databases are known classically, but are unrealistic because they require long shared secret keys between the parties for secure communication and shared randomness in the protocol. Here, we propose using quantum key distribution (QKD) instead for a practical implementation, which can realise both the secure communication and shared randomness requirements. We prove that QKD maintains the security of the SPIR protocol and that it is also secure against any external eavesdropper. We also show how such a classical-quantum system could be implemented practically, using the example of a two-database SPIR protocol with keys generated by measurement device-independent QKD. Through key rate calculations, we show that such an implementation is feasible at the metropolitan level with current QKD technology.     

Geometric Partition Entropy: Coarse-Graining a Continuous State Space

Entropy is re-examined as a quantification of ignorance in the predictability of a one dimensional continuous phenomenon. Although traditional estimators for entropy have been widely utilized in this context, we show that both the thermodynamic and Shannon’s theory of entropy are fundamentally discrete, and that the limiting process used to define differential entropy suffers from similar problems to those encountered in thermodynamics. In contrast, we consider a sampled data set to be observations of microstates (unmeasurable in thermodynamics and nonexistent in Shannon’s discrete theory), meaning, in this context, it is the macrostates of the underlying phenomenon that are unknown. To obtain a particular coarse-grained model we define macrostates using quantiles of the sample and define an ignorance density distribution based on the distances between quantiles. The geometric partition entropy is then just the Shannon entropy of this finite distribution. Our measure is more consistent and informative than histogram-binning, especially when applied to complex distributions and those with extreme outliers or under limited sampling. Its computational efficiency and avoidance of negative values can also make it preferable to geometric estimators such as k-nearest neighbors. We suggest applications that are unique to this estimator and illustrate its general utility through an application to time series in the approximation of an ergodic symbolic dynamics from limited observations.     

Entropy Sources Based on Silicon Chips: True Random Number Generator and Physical Unclonable Function

Entropy is a measure of uncertainty or randomness. It is the foundation for almost all cryptographic systems. True random number generators (TRNGs) and physical unclonable functions (PUFs) are the silicon primitives to respectively harvest dynamic and static entropy to generate random bit streams. In this survey paper, we present a systematic and comprehensive review of different state-of-the-art methods to harvest entropy from silicon-based devices, including the implementations, applications, and the security of the designs. Furthermore, we conclude the trends of the entropy source design to point out the current spots of entropy harvesting.     

A Three-Stage Quantum Cryptography Protocol

We present a three-stage quantum cryptographic protocol based on public key cryptography in which each party uses its own secret key. Unlike the BB84 protocol, where the qubits are transmitted in only one direction and classical information exchanged thereafter, the communication in the proposed protocol remains quantum in each stage. A related system of key distribution is also described.     

Vibrational Entropy of Crystalline Solids from Covariance of Atomic Displacements

The vibrational entropy of a solid at finite temperature is investigated from the perspective of information theory. Ab initio molecular dynamics (AIMD) simulations generate ensembles of atomic configurations at finite temperature from which we obtain the N-body distribution of atomic displacements, ρN. We calculate the information-theoretic entropy from the expectation value of lnρN. At a first level of approximation, treating individual atomic displacements independently, our method may be applied using Debye–Waller B-factors, allowing diffraction experiments to obtain an upper bound on the thermodynamic entropy. At the next level of approximation we correct the overestimation through inclusion of displacement covariances. We apply this approach to elemental body-centered cubic sodium and face-centered cubic aluminum, showing good agreement with experimental values above the Debye temperatures of the metals. Below the Debye temperatures, we extract an effective vibrational density of states from eigenvalues of the covariance matrix, and then evaluate the entropy quantum mechanically, again yielding good agreement with experiment down to low temperatures. Our method readily generalizes to complex solids, as we demonstrate for a high entropy alloy. Further, our method applies in cases where the quasiharmonic approximation fails, as we demonstrate by calculating the HCP/BCC transition in Ti.     

Colored Texture Analysis Fuzzy Entropy Methods with a Dermoscopic Application

Texture analysis is a subject of intensive focus in research due to its significant role in the field of image processing. However, few studies focus on colored texture analysis and even fewer use information theory concepts. Entropy measures have been proven competent for gray scale images. However, to the best of our knowledge, there are no well-established entropy methods that deal with colored images yet. Therefore, we propose the recent colored bidimensional fuzzy entropy measure, FuzEnC2D, and introduce its new multi-channel approaches, FuzEnV2D and FuzEnM2D, for the analysis of colored images. We investigate their sensitivity to parameters and ability to identify images with different irregularity degrees, and therefore different textures. Moreover, we study their behavior with colored Brodatz images in different color spaces. After verifying the results with test images, we employ the three methods for analyzing dermoscopic images of malignant melanoma and benign melanocytic nevi. FuzEnC2D, FuzEnV2D, and FuzEnM2D illustrate a good differentiation ability between the two—similar in appearance—pigmented skin lesions. The results outperform those of a well-known texture analysis measure. Our work provides the first entropy measure studying colored images using both single and multi-channel approaches.     

Selecting an Effective Entropy Estimator for Short Sequences of Bits and Bytes with Maximum Entropy

Entropy makes it possible to measure the uncertainty about an information source from the distribution of its output symbols. It is known that the maximum Shannon’s entropy of a discrete source of information is reached when its symbols follow a Uniform distribution. In cryptography, these sources have great applications since they allow for the highest security standards to be reached. In this work, the most effective estimator is selected to estimate entropy in short samples of bytes and bits with maximum entropy. For this, 18 estimators were compared. Results concerning the comparisons published in the literature between these estimators are discussed. The most suitable estimator is determined experimentally, based on its bias, the mean square error short samples of bytes and bits.     

Area Entropy and Quantized Mass of Black Holes from Information Theory

In this paper, we present a derivation of the black hole area entropy with the relationship between entropy and information. The curved space of a black hole allows objects to be imaged in the same way as camera lenses. The maximal information that a black hole can gain is limited by both the Compton wavelength of the object and the diameter of the black hole. When an object falls into a black hole, its information disappears due to the no-hair theorem, and the entropy of the black hole increases correspondingly. The area entropy of a black hole can thus be obtained, which indicates that the Bekenstein–Hawking entropy is information entropy rather than thermodynamic entropy. The quantum corrections of black hole entropy are also obtained according to the limit of Compton wavelength of the captured particles, which makes the mass of a black hole naturally quantized. Our work provides an information-theoretic perspective for understanding the nature of black hole entropy.     

Information Theory for Biological Sequence Classification: A Novel Feature Extraction Technique Based on Tsallis Entropy

In recent years, there has been an exponential growth in sequencing projects due to accelerated technological advances, leading to a significant increase in the amount of data and resulting in new challenges for biological sequence analysis. Consequently, the use of techniques capable of analyzing large amounts of data has been explored, such as machine learning (ML) algorithms. ML algorithms are being used to analyze and classify biological sequences, despite the intrinsic difficulty in extracting and finding representative biological sequence methods suitable for them. Thereby, extracting numerical features to represent sequences makes it statistically feasible to use universal concepts from Information Theory, such as Tsallis and Shannon entropy. In this study, we propose a novel Tsallis entropy-based feature extractor to provide useful information to classify biological sequences. To assess its relevance, we prepared five case studies: (1) an analysis of the entropic index q; (2) performance testing of the best entropic indices on new datasets; (3) a comparison made with Shannon entropy and (4) generalized entropies; (5) an investigation of the Tsallis entropy in the context of dimensionality reduction. As a result, our proposal proved to be effective, being superior to Shannon entropy and robust in terms of generalization, and also potentially representative for collecting information in fewer dimensions compared with methods such as Singular Value Decomposition and Uniform Manifold Approximation and Projection.     

Entropy Method for Decision-Making: Uncertainty Cycles in Tourism Demand

A new methodology is presented for measuring, classifying and predicting the cycles of uncertainty that occur in temporary decision-making in the tourist accommodation market (apartments and hotels). Special attention is paid to the role of entropy and cycles in the process under the Adaptive Markets Hypothesis. The work scheme analyses random cycles from time to time, and in the frequency domain, the linear and nonlinear causality relationships between variables are studied. The period analysed is from January 2005 to December 2018; the following empirical results stand out: (1) On longer scales, the periodicity of the uncertainty of decision-making is between 6 and 12 months, respectively, for all the nationalities described. (2) The elasticity of demand for tourist apartments is approximately 1% due to changes in demand for tourist hotels. (3) The elasticity of the uncertainty factor is highly correlated with the country of origin of tourists visiting Spain. For example, it has been empirically shown that increases of 1% in uncertainty cause increases in the demand for apartments of 2.12% (worldwide), 3.05% (UK), 1.91% (Germany), 1.78% (France), 7.21% (Ireland), 3.61% (The Netherlands) respectively. This modelling has an explanatory capacity of 99% in all the models analysed.     

The Concavity of Entropy and Extremum Principles in Thermodynamics

We revisit the concavity property of the thermodynamic entropy in order to formulate a general proof of the minimum energy principle as well as of other equivalent extremum principles that are valid for thermodynamic potentials and corresponding Massieu functions under different constraints. The current derivation aims at providing a coherent formal framework for such principles which may be also pedagogically useful as it fully exploits and highlights the equivalence between different schemes. We also elucidate the consequences of the extremum principles for the general shape of thermodynamic potentials in relation to first-order phase transitions.     

相空间中脑电近似熵和信息熵的计算

提出一种基于相空间重构脑电信号来计算脑电近似熵和信息熵的新方法．实验计算结果表明,癫痫患者脑电和正常人脑电的近似熵和信息熵随相空间嵌入维数的变化有明显的不同．     

Time Dependence of Entropy Flux and Entropy Production of a Dissipative Dynamical System Driven by Non-Gaussian Noise

A stochastic dissipative dynamical system driven by non-Gaussian noise is investigated. A general approximate Fokker-Planck equation of the system is derived through a path-integral approach. Based on the definition of Shannon＇s information entropy, the exact time dependence of entropy flux and entropy production of the system is calculated both in the absence and in the presence of non-equilibrium constraint. The present calculation can be used to interpret the interplay of the dissipative constant and non-Gaussian noise on the entropy flux and entropy production.     

Logical Entropy of Information Sources

In this paper, we present the concept of the logical entropy of order m, logical mutual information, and the logical entropy for information sources. We found upper and lower bounds for the logical entropy of a random variable by using convex functions. We show that the logical entropy of the joint distributions X1 and X2 is always less than the sum of the logical entropy of the variables X1 and X2. We define the logical Shannon entropy and logical metric permutation entropy to an information system and examine the properties of this kind of entropy. Finally, we examine the amount of the logical metric entropy and permutation logical entropy for maps.     

Entropy and Random Vectors

Barron⁽¹⁾ produced a proof of the Central Limit Theorem for real-valued IID random variables, in the sense of convergence in relative entropy. Here, we establish a similar result for independent real-valued random vectors, not necessarily identically distributed. The main developments required are a generalisation of De Bruijn's identity, and various inequalities proposed in ref. 2.     

Information Entropy Squeezing of a Two-Level Atom Interacting with Two-Mode Coherent Fields

From a quantum information point of view we investigate the entropy squeezing properties for a two-level atom interacting with the two-mode coherent fields via the two-photon transition. We discuss the influences of the initial state of the system on the atomic information entropy squeezing. Our results show that the squeezed component number,squeezed direction, and time of the information entropy squeezing can be controlled by choosing atomic distribution angle,the relative phase between the atom and the two-mode field, and the difference of the average photon number of the two field modes, respectively. Quantum information entropy is a remarkable precision measure for the atomic squeezing.     

Quantum Information and Entropy

Thermodynamic entropy is not an entirely satisfactory measure of information of a quantum state. This entropy for an unknown pure state is zero, although repeated measurements on copies of such a pure state do communicate information. In view of this, we propose a new measure for the informational entropy of a quantum state that includes information in the pure states and the thermodynamic entropy. The origin of information is explained in terms of an interplay between unitary and non-unitary evolution. Such complementarity is also at the basis of the so-called interaction-free measurement.     

Information Entropy Squeezing of a Two-Level Atom Interacting with Two-Mode Coherent Fields

From a quantum information point of view we investigate the entropy squeezing properties for a two-level atom interacting with the two-mode coherent fields via the two-photon transition. We discuss the influences of the initial state of the system on the atomic information entropy squeezing. Our results show that the squeezed component number, squeezed direction, and time of the information entropy squeezing can be controlled by choosing atomic distribution angle, the relative phase between the atom and the two-mode field, and the difference of the average photon number of the two field modes, respectively. Quantum information entropy is a remarkable precision measure for the atomic squeezing.     

The Systematic Bias of Entropy Calculation in the Multi-Scale Entropy Algorithm

Entropy indicates irregularity or randomness of a dynamic system. Over the decades, entropy calculated at different scales of the system through subsampling or coarse graining has been used as a surrogate measure of system complexity. One popular multi-scale entropy analysis is the multi-scale sample entropy (MSE), which calculates entropy through the sample entropy (SampEn) formula at each time scale. SampEn is defined by the “logarithmic likelihood” that a small section (within a window of a length m) of the data “matches” with other sections will still “match” the others if the section window length increases by one. “Match” is defined by a threshold of r times standard deviation of the entire time series. A problem of current MSE algorithm is that SampEn calculations at different scales are based on the same matching threshold defined by the original time series but data standard deviation actually changes with the subsampling scales. Using a fixed threshold will automatically introduce systematic bias to the calculation results. The purpose of this paper is to mathematically present this systematic bias and to provide methods for correcting it. Our work will help the large MSE user community avoiding introducing the bias to their multi-scale SampEn calculation results.