Entropy: From Thermodynamics to Information Processing

Entropy is a concept that emerged in the 19th century. It used to be associated with heat harnessed by a thermal machine to perform work during the Industrial Revolution. However, there was an unprecedented scientific revolution in the 20th century due to one of its most essential innovations, i.e., the information theory, which also encompasses the concept of entropy. Therefore, the following question is naturally raised: “what is the difference, if any, between concepts of entropy in each field of knowledge?” There are misconceptions, as there have been multiple attempts to conciliate the entropy of thermodynamics with that of information theory. Entropy is most commonly defined as “disorder”, although it is not a good analogy since “order” is a subjective human concept, and “disorder” cannot always be obtained from entropy. Therefore, this paper presents a historical background on the evolution of the term “entropy”, and provides mathematical evidence and logical arguments regarding its interconnection in various scientific areas, with the objective of providing a theoretical review and reference material for a broad audience.     

The Systematic Bias of Entropy Calculation in the Multi-Scale Entropy Algorithm

Entropy indicates irregularity or randomness of a dynamic system. Over the decades, entropy calculated at different scales of the system through subsampling or coarse graining has been used as a surrogate measure of system complexity. One popular multi-scale entropy analysis is the multi-scale sample entropy (MSE), which calculates entropy through the sample entropy (SampEn) formula at each time scale. SampEn is defined by the “logarithmic likelihood” that a small section (within a window of a length m) of the data “matches” with other sections will still “match” the others if the section window length increases by one. “Match” is defined by a threshold of r times standard deviation of the entire time series. A problem of current MSE algorithm is that SampEn calculations at different scales are based on the same matching threshold defined by the original time series but data standard deviation actually changes with the subsampling scales. Using a fixed threshold will automatically introduce systematic bias to the calculation results. The purpose of this paper is to mathematically present this systematic bias and to provide methods for correcting it. Our work will help the large MSE user community avoiding introducing the bias to their multi-scale SampEn calculation results.     

Rényi Entropy and Free Energy

The Rényi entropy is a generalization of the usual concept of entropy which depends on a parameter q. In fact, Rényi entropy is closely related to free energy. Suppose we start with a system in thermal equilibrium and then suddenly divide the temperature by q. Then the maximum amount of work the system can perform as it moves to equilibrium at the new temperature divided by the change in temperature equals the system’s Rényi entropy in its original state. This result applies to both classical and quantum systems. Mathematically, we can express this result as follows: the Rényi entropy of a system in thermal equilibrium is without the ‘q−1-derivative’ of its free energy with respect to the temperature. This shows that Rényi entropy is a q-deformation of the usual concept of entropy.     

From Rényi Entropy Power to Information Scan of Quantum States

In this paper, we generalize the notion of Shannon’s entropy power to the Rényi-entropy setting. With this, we propose generalizations of the de Bruijn identity, isoperimetric inequality, or Stam inequality. This framework not only allows for finding new estimation inequalities, but it also provides a convenient technical framework for the derivation of a one-parameter family of Rényi-entropy-power-based quantum-mechanical uncertainty relations. To illustrate the usefulness of the Rényi entropy power obtained, we show how the information probability distribution associated with a quantum state can be reconstructed in a process that is akin to quantum-state tomography. We illustrate the inner workings of this with the so-called “cat states”, which are of fundamental interest and practical use in schemes such as quantum metrology. Salient issues, including the extension of the notion of entropy power to Tsallis entropy and ensuing implications in estimation theory, are also briefly discussed.     

Novel Features for Binary Time Series Based on Branch Length Similarity Entropy

Branch length similarity (BLS) entropy is defined in a network consisting of a single node and branches. In this study, we mapped the binary time-series signal to the circumference of the time circle so that the BLS entropy can be calculated for the binary time-series. We obtained the BLS entropy values for “1” signals on the time circle. The set of values are the BLS entropy profile. We selected the local maximum (minimum) point, slope, and inflection point of the entropy profile as the characteristic features of the binary time-series and investigated and explored their significance. The local maximum (minimum) point indicates the time at which the rate of change in the signal density becomes zero. The slope and inflection points correspond to the degree of change in the signal density and the time at which the signal density changes occur, respectively. Moreover, we show that the characteristic features can be widely used in binary time-series analysis by characterizing the movement trajectory of Caenorhabditis elegans. We also mention the problems that need to be explored mathematically in relation to the features and propose candidates for additional features based on the BLS entropy profile.     

Thermodynamic Properties of the First-Generation Hybrid Dendrimer with “Carbosilane Core/Phenylene Shell” Structure

The molar heat capacity of the first-generation hybrid dendrimer with a “carbosilane core/phenylene shell” structure was measured for the first time in the temperature range T = 6–600 K using a precise adiabatic vacuum calorimeter and DSC. In the above temperature interval, the glass transition of the studied compound was observed, and its thermodynamic characteristics were determined. The standard thermodynamic functions (the enthalpy, the entropy, and the Gibbs energy) of the hybrid dendrimer were calculated over the range from T = 0 to 600 K using the experimentally determined heat capacity. The standard entropy of formation of the investigated dendrimer was evaluated at T = 298.15 K. The obtained thermodynamic properties of the studied hybrid dendrimer were compared and discussed with the literature data for some of the first-generation organosilicon and pyridylphenylene dendrimers.     

Which Physical Quantity Deserves the Name “Quantity of Heat”?

“What is heat?” was the title of a 1954 article by Freeman J. Dyson, published in Scientific American. Apparently, it was appropriate to ask this question at that time. The answer is given in the very first sentence of the article: heat is disordered energy. We will ask the same question again, but with a different expectation for its answer. Let us imagine that all the thermodynamic knowledge is already available: both the theory of phenomenological thermodynamics and that of statistical thermodynamics, including quantum statistics, but that the term “heat” has not yet been attributed to any of the variables of the theory. With the question “What is heat?” we now mean: which of the physical quantities deserves this name? There are several candidates: the quantities Q, H, Etherm and S. We can then formulate a desideratum, or a profile: What properties should such a measure of the quantity or amount of heat ideally have? Then, we evaluate all the candidates for their suitability. It turns out that the winner is the quantity S, which we know by the name of entropy. In the second part of the paper, we examine why entropy has not succeeded in establishing itself as a measure for the amount of heat, and we show that there is a real chance today to make up for what was missed.     

Partitioning Entropy with Action Mechanics: Predicting Chemical Reaction Rates and Gaseous Equilibria of Reactions of Hydrogen from Molecular Properties

Our intention is to provide easy methods for estimating entropy and chemical potentials for gas phase reactions. Clausius’ virial theorem set a basis for relating kinetic energy in a body of independent material particles to its potential energy, pointing to their complementary role with respect to the second law of maximum entropy. Based on this partitioning of thermal energy as sensible heat and also as a latent heat or field potential energy, in action mechanics we express the entropy of ideal gases as a capacity factor for enthalpy plus the configurational work to sustain the relative translational, rotational, and vibrational action. This yields algorithms for estimating chemical reaction rates and positions of equilibrium. All properties of state including entropy, work potential as Helmholtz and Gibbs energies, and activated transition state reaction rates can be estimated, using easily accessible molecular properties, such as atomic weights, bond lengths, moments of inertia, and vibrational frequencies. We conclude that the large molecular size of many enzymes may catalyze reaction rates because of their large radial inertia as colloidal particles, maximising action states by impulsive collisions. Understanding how Clausius’ virial theorem justifies partitioning between thermal and statistical properties of entropy, yielding a more complete view of the second law’s evolutionary nature and the principle of maximum entropy. The ease of performing these operations is illustrated with three important chemical gas phase reactions: the reversible dissociation of hydrogen molecules, lysis of water to hydrogen and oxygen, and the reversible formation of ammonia from nitrogen and hydrogen. Employing the ergal also introduced by Clausius to define the reversible internal work overcoming molecular interactions plus the configurational work of change in Gibbs energy, often neglected; this may provide a practical guide for managing industrial processes and risk in climate change at the global scale. The concepts developed should also have value as novel methods for the instruction of senior students.     

Information and Self-Organization II: Steady State and Phase Transition

This paper starts from Schrödinger’s famous question “what is life” and elucidates answers that invoke, in particular, Friston’s free energy principle and its relation to the method of Bayesian inference and to Synergetics 2nd foundation that utilizes Jaynes’ maximum entropy principle. Our presentation reflects the shift from the emphasis on physical principles to principles of information theory and Synergetics. In view of the expected general audience of this issue, we have chosen a somewhat tutorial style that does not require special knowledge on physics but familiarizes the reader with concepts rooted in information theory and Synergetics.     

Entropy Profiling: A Reduced—Parametric Measure of Kolmogorov—Sinai Entropy from Short-Term HRV Signal

Entropy profiling is a recently introduced approach that reduces parametric dependence in traditional Kolmogorov-Sinai (KS) entropy measurement algorithms. The choice of the threshold parameter r of vector distances in traditional entropy computations is crucial in deciding the accuracy of signal irregularity information retrieved by these methods. In addition to making parametric choices completely data-driven, entropy profiling generates a complete profile of entropy information as against a single entropy estimate (seen in traditional algorithms). The benefits of using “profiling” instead of “estimation” are: (a) precursory methods such as approximate and sample entropy that have had the limitation of handling short-term signals (less than 1000 samples) are now made capable of the same; (b) the entropy measure can capture complexity information from short and long-term signals without multi-scaling; and (c) this new approach facilitates enhanced information retrieval from short-term HRV signals. The novel concept of entropy profiling has greatly equipped traditional algorithms to overcome existing limitations and broaden applicability in the field of short-term signal analysis. In this work, we present a review of KS-entropy methods and their limitations in the context of short-term heart rate variability analysis and elucidate the benefits of using entropy profiling as an alternative for the same.     

Analysis on the Development of Digital Economy in Guangdong Province Based on Improved Entropy Method and Multivariate Statistical Analysis

The lack of adequate indicators in the research of digital economy may lead to the shortage of data support on decision making for governments. To solve this problem, first we establish a digital economy indicator evaluation system by dividing the digital economy into four types: “basic type”, “technology type”, “integration type” and “service type” and select 5 indicators for each type. On this basis, the weight of each indicator is calculated to find the deficiencies in the development of some digital economic fields by the improved entropy method. By drawing on the empowerment idea of Analytic Hierarchy Process, the improved entropy method firstly compares the difference coefficient of indicators in pairs and maps the comparison results to the scales 1–9. Then, the judgment matrix is constructed based on the information entropy, which can solve as much as possible the problem that the difference among the weight of each indicator is too large in traditional entropy method. The results indicate that: the development of digital economy in Guangdong Province was relatively balanced from 2015 to 2018 and will be better in the future while the development of rural e-commerce in Guangdong Province is relatively backward, and there is an obvious digital gap between urban and rural areas. Next we extract two new variables respectively to replace the 20 indicators we select through principal component analysis and factor analysis methods in multivariate statistical analysis, which can retain the original information to the greatest extent and provide convenience for further research in the future. Finally, we and provide constructive comments of digital economy in Guangdong Province from 2015 to 2018.     

Predictions of Conjugate Heat Transfer in Turbulent Channel Flow Using Advanced Wall-Modeled Large Eddy Simulation Techniques

In this paper, advanced wall-modeled large eddy simulation (LES) techniques are used to predict conjugate heat transfer processes in turbulent channel flow. Thereby, the thermal energy transfer process involves an interaction of conduction within a solid body and convection from the solid surface by fluid motion. The approaches comprise a two-layer RANS–LES approach (zonal LES), a hybrid RANS–LES representative, the so-called improved delayed detached eddy simulation method (IDDES) and a non-equilibrium wall function model (WFLES), respectively. The results obtained are evaluated in comparison with direct numerical simulation (DNS) data and wall-resolved LES including thermal cases of large Reynolds numbers where DNS data are not available in the literature. It turns out that zonal LES, IDDES and WFLES are able to predict heat and fluid flow statistics along with wall shear stresses and Nusselt numbers accurately and that are physically consistent. Furthermore, it is found that IDDES, WFLES and zonal LES exhibit significantly lower computational costs than wall-resolved LES. Since IDDES and especially zonal LES require considerable extra work to generate numerical grids, this study indicates in particular that WFLES offers a promising near-wall modeling strategy for LES of conjugated heat transfer problems. Finally, an entropy generation analysis using the various models showed that the viscous entropy production is zero inside the solid region, peaks at the solid–fluid interface and decreases rapidly with increasing wall distance within the fluid region. Except inside the solid region, where steep temperature gradients lead to high (thermal) entropy generation rates, a similar behavior is monitored for the entropy generation by heat transfer process.     

热力学平衡态的涨落判据

任何热力学量的涨落在平衡态下都不能太大.特别地,要求熵和能量的涨落不大,就可以给出低温下热力学平衡态的普遍适用的判据.一些算例表明,低温下即使1023个粒子也不一定能达到热力学平衡态.     

Structural Entropy of the Stochastic Block Models

With the rapid expansion of graphs and networks and the growing magnitude of data from all areas of science, effective treatment and compression schemes of context-dependent data is extremely desirable. A particularly interesting direction is to compress the data while keeping the “structural information” only and ignoring the concrete labelings. Under this direction, Choi and Szpankowski introduced the structures (unlabeled graphs) which allowed them to compute the structural entropy of the Erdős–Rényi random graph model. Moreover, they also provided an asymptotically optimal compression algorithm that (asymptotically) achieves this entropy limit and runs in expectation in linear time. In this paper, we consider the stochastic block models with an arbitrary number of parts. Indeed, we define a partitioned structural entropy for stochastic block models, which generalizes the structural entropy for unlabeled graphs and encodes the partition information as well. We then compute the partitioned structural entropy of the stochastic block models, and provide a compression scheme that asymptotically achieves this entropy limit.     

Entropy-Based Behavioural Efficiency of the Financial Market

The most known and used abstract model of the financial market is based on the concept of the informational efficiency (EMH) of that market. The paper proposes an alternative which could be named the behavioural efficiency of the financial market, which is based on the behavioural entropy instead of the informational entropy. More specifically, the paper supports the idea that, in the financial market, the only measure (if any) of the entropy is the available behaviours indicated by the implicit information. Therefore, the behavioural entropy is linked to the concept of behavioural efficiency. The paper argues that, in fact, in the financial markets, there is not a (real) informational efficiency, but there exists a behavioural efficiency instead. The proposal is based both on a new typology of information in the financial market (which provides the concept of implicit information—that is, that information ”translated” by the economic agents from observing the actual behaviours) and on a non-linear (more exactly, a logistic) curve linking the behavioural entropy to the behavioural efficiency of the financial markets. Finally, the paper proposes a synergic overcoming of both EMH and AMH based on the new concept of behavioural entropy in the financial market.     

Canonical Entropy of Reissner-Nordstrom Black Hole

Recently, Hawking radiation of the black hole has been studied using the tunnel effect method. It is found the radiation spectrum of the black hole is not a strictly pure thermal spectrum. How the departure from pure thermal spectrum affects the entropy? This is a very interesting problem. In this paper, we calculate the partition function by energy spectrum obtained by tunnel effect. Using the relation between the partition function and entropy, we derive the expression of entropy the general charged black hole. In our calculation, we not only consider the correction to the black hole entropy due to fluctuation of energy but also consider the effect of the change of the black hole charges on entropy. We discuss Reissner-Nordstrom black hole and obtain that Reissner-Nordstrom black hole cannot approach the extreme black hole by changing its charges.     

Associations between Neurocardiovascular Signal Entropy and Physical Frailty

In this cross-sectional study, the relationship between noninvasively measured neurocardiovascular signal entropy and physical frailty was explored in a sample of community-dwelling older adults from The Irish Longitudinal Study on Ageing (TILDA). The hypothesis under investigation was that dysfunction in the neurovascular and cardiovascular systems, as quantified by short-length signal complexity during a lying-to-stand test (active stand), could provide a marker for frailty. Frailty status (i.e., “non-frail”, “pre-frail”, and “frail”) was based on Fried’s criteria (i.e., exhaustion, unexplained weight loss, weakness, slowness, and low physical activity). Approximate entropy (ApEn) and sample entropy (SampEn) were calculated during resting (lying down), active standing, and recovery phases. There was continuously measured blood pressure/heart rate data from 2645 individuals (53.0% female) and frontal lobe tissue oxygenation data from 2225 participants (52.3% female); both samples had a mean (SD) age of 64.3 (7.7) years. Results revealed statistically significant associations between neurocardiovascular signal entropy and frailty status. Entropy differences between non-frail and pre-frail/frail were greater during resting state compared with standing and recovery phases. Compared with ApEn, SampEn seemed to have better discriminating power between non-frail and pre-frail/frail individuals. The quantification of entropy in short length neurocardiovascular signals could provide a clinically useful marker of the multiple physiological dysregulations that underlie physical frailty.     

H-Theorem in an Isolated Quantum Harmonic Oscillator

We consider the H-theorem in an isolated quantum harmonic oscillator through the time-dependent Schrödinger equation. The effect of potential in producing entropy is investigated in detail, and we found that including a barrier potential into a harmonic trap would lead to the thermalization of the system, while a harmonic trap alone would not thermalize the system. During thermalization, Shannon entropy increases, which shows that a microscopic quantum system still obeys the macroscopic thermodynamics law. Meanwhile, initial coherent mechanical energy transforms to incoherent thermal energy during thermalization, which exhibiting the decoherence of an oscillating wave packet featured by a large decreasing of autocorrelation length. When reaching thermal equilibrium, the wave packet comes to a halt, with the density distributions both in position and momentum spaces well-fitted by a microcanonical ensemble of statistical mechanics.