Inferring a Property of a Large System from a Small Number of Samples

Inferring the value of a property of a large stochastic system is a difficult task when the number of samples is insufficient to reliably estimate the probability distribution. The Bayesian estimator of the property of interest requires the knowledge of the prior distribution, and in many situations, it is not clear which prior should be used. Several estimators have been developed so far in which the proposed prior us individually tailored for each property of interest; such is the case, for example, for the entropy, the amount of mutual information, or the correlation between pairs of variables. In this paper, we propose a general framework to select priors that is valid for arbitrary properties. We first demonstrate that only certain aspects of the prior distribution actually affect the inference process. We then expand the sought prior as a linear combination of a one-dimensional family of indexed priors, each of which is obtained through a maximum entropy approach with constrained mean values of the property under study. In many cases of interest, only one or very few components of the expansion turn out to contribute to the Bayesian estimator, so it is often valid to only keep a single component. The relevant component is selected by the data, so no handcrafted priors are required. We test the performance of this approximation with a few paradigmatic examples and show that it performs well in comparison to the ad-hoc methods previously proposed in the literature. Our method highlights the connection between Bayesian inference and equilibrium statistical mechanics, since the most relevant component of the expansion can be argued to be that with the right temperature.     

A Generalized Measure of Cumulative Residual Entropy

In this work, we introduce a generalized measure of cumulative residual entropy and study its properties. We show that several existing measures of entropy, such as cumulative residual entropy, weighted cumulative residual entropy and cumulative residual Tsallis entropy, are all special cases of this generalized cumulative residual entropy. We also propose a measure of generalized cumulative entropy, which includes cumulative entropy, weighted cumulative entropy and cumulative Tsallis entropy as special cases. We discuss a generating function approach, using which we derive different entropy measures. We provide residual and cumulative versions of Sharma–Taneja–Mittal entropy and obtain them as special cases this generalized measure of entropy. Finally, using the newly introduced entropy measures, we establish some relationships between entropy and extropy measures.     

Benchmarking Analysis of the Accuracy of Classification Methods Related to Entropy

In the machine learning literature we can find numerous methods to solve classification problems. We propose two new performance measures to analyze such methods. These measures are defined by using the concept of proportional reduction of classification error with respect to three benchmark classifiers, the random and two intuitive classifiers which are based on how a non-expert person could realize classification simply by applying a frequentist approach. We show that these three simple methods are closely related to different aspects of the entropy of the dataset. Therefore, these measures account somewhat for entropy in the dataset when evaluating the performance of classifiers. This allows us to measure the improvement in the classification results compared to simple methods, and at the same time how entropy affects classification capacity. To illustrate how these new performance measures can be used to analyze classifiers taking into account the entropy of the dataset, we carry out an intensive experiment in which we use the well-known J48 algorithm, and a UCI repository dataset on which we have previously selected a subset of the most relevant attributes. Then we carry out an extensive experiment in which we consider four heuristic classifiers, and 11 datasets.     

基于复杂度的自适应门限弱小目标检测方法

针对红外弱小目标检测问题,提出了一种基于图像复杂度的自适应门限目标检测方法.讨论了天空中四类不同区域的图像信息熵.图像信息熵虽然较好地表达了图像的平均信息量,但对图像的突变点不敏感.将它改进得到图像方差加权信息熵,其较好地反映了图像的复杂度特征.将图像方差加权信息熵作为图像复杂度的定量描述,用两种特定的分析模板对图像复杂度进行分析.在目标区域中,两种分析模板得到的复杂度差异较大,而非目标区域的两种复杂度则基本没有差异.算法获取两种分析模板下的复杂度图像,再对两种复杂度图像做差,得到复杂度差值图像.对差值图像建立指数模型得到自适应分割门限完成目标检测.实验结果表明,该方法对低信杂比的红外云层背景弱小目标图像具有良好的检测效果.     

Composite Multiscale Cross-Sample Entropy Analysis for Long-Term Structural Health Monitoring of Residential Buildings

This study proposesd a novel, entropy-based structural health monitoring (SHM) system for measuring microvibration signals generated by actual buildings. A structural health diagnosis interface was established for demonstration purposes. To enhance the reliability and accuracy of entropy evaluation at various scales, composite multiscale cross-sample entropy (CMSCE) was adopted to increase the number of coarse-grained time series. The degree of similarity and asynchrony between ambient vibration signals measured on adjacent floors was used as an in-dicator for structural health assessment. A residential building that has been monitored since 1994 was selected for long-term monitoring. The accumulated database, including both the earthquake and ambient vibrations in each seismic event, provided the possibility to evaluate the practicability of the CMSCE-based method. Entropy curves obtained for each of the years, as well as the stable trend of the corresponding damage index (DI) graphs, demonstrated the relia-bility of the proposed SHM system. Moreover, two large earthquake events that occurred near the monitoring site were analyzed. The results revealed that the entropy values may have been slightly increased after the earthquakes. Positive DI values were obtained for higher floors, which could provide an early warning of structural instability. The proposed SHM system is highly stable and practical.     

Spectrum Sensing Implemented with Improved Fluctuation-Based Dispersion Entropy and Machine Learning

Spectrum sensing is an important function in radio frequency spectrum management and cognitive radio networks. Spectrum sensing is used by one wireless system (e.g., a secondary user) to detect the presence of a wireless service with higher priority (e.g., a primary user) with which it has to coexist in the radio frequency spectrum. If the wireless signal is detected, the second user system releases the given frequency to maintain the principle of not interfering. This paper proposes a machine learning implementation of spectrum sensing using the entropy measure as a feature vector. In the training phase, the information about the activity of the wireless service with higher priority is gathered, and the model is formed. In the classification phase, the wireless system compares the current sensing report to the created model to calculate the posterior probability and classify the sensing report into either the presence or absence of wireless service with higher priority. This paper proposes the novel application of the Fluctuation Dispersion Entropy (FDE) measure recently introduced in the research community as a feature vector to build the model and implement the classification. An improved implementation of the FDE (IFDE) is used to enhance the robustness to noise. IFDE is further enhanced with an adaptive method (AIFDE) to automatically select the hyper-parameter introduced in IFDE. Then, this paper combines the machine learning approach with the entropy measure approach, which are both recent developments in spectrum sensing research. The approach is compared to similar approaches in literature and the classical energy detection method using a generated radar signal data set with different conditions of SNR(dB) and fading conditions. The results show that the proposed approach is able to outperform the approaches from literature based on other entropy measures or the Energy Detector (ED) in a consistent way across different levels of SNR and fading conditions.     

一种基于单元信息熵的目标匹配改进算法

为解决图像信息熵无法有效进行图像匹配的问题,将图像单元信息熵和投影特征相结合,定义了图像单元信息熵,并提出了一种基于单元投影信息熵的图像匹配方法.在单元信息熵的基础上,在各个单元格内进行单元信息熵投影计算,然后按照一定的测度进行计算,从而实现图像的匹配.采用网格分层的搜索算法,加快搜索速度,提高其工程实用性.实验证明:该算法具有良好的抗几何失真能力和抗辐射失真的能力,以及很好的抗噪声干扰的能力,可以准确的进行目标匹配.     

New Pythagorean Entropy Measure with Application in Multi-Criteria Decision Analysis

The purpose of this paper is to propose a new Pythagorean fuzzy entropy for Pythagorean fuzzy sets, which is a continuation of the Pythagorean fuzzy entropy of intuitionistic sets. The Pythagorean fuzzy set continues the intuitionistic fuzzy set with the additional advantage that it is well equipped to overcome its imperfections. Its entropy determines the quantity of information in the Pythagorean fuzzy set. Thus, the proposed entropy provides a new flexible tool that is particularly useful in complex multi-criteria problems where uncertain data and inaccurate information are considered. The performance of the introduced method is illustrated in a real-life case study, including a multi-criteria company selection problem. In this example, we provide a numerical illustration to distinguish the entropy measure proposed from some existing entropies used for Pythagorean fuzzy sets and intuitionistic fuzzy sets. Statistical illustrations show that the proposed entropy measures are reliable for demonstrating the degree of fuzziness of both Pythagorean fuzzy set (PFS) and intuitionistic fuzzy sets (IFS). In addition, a multi-criteria decision-making method complex proportional assessment (COPRAS) was also proposed with weights calculated based on the proposed new entropy measure. Finally, to validate the reliability of the results obtained using the proposed entropy, a comparative analysis was performed with a set of carefully selected reference methods containing other generally used entropy measurement methods. The illustrated numerical example proves that the calculation results of the proposed new method are similar to those of several other up-to-date methods.     

Damage Detection of Regular Civil Buildings Using Modified Multi-Scale Symbolic Dynamic Entropy

Based on the examination of the fundamental characteristics of structures, structural health monitoring (SHM) has received increased attention in recent years. Studies have shown that the SHM method using entropy analysis can precisely identify the damaged location of the structure, which is very helpful for the daily inspection or maintenance of civil structures. Although entropy analysis has shown excellent accuracy, it still consumes too much time and too many resources in terms of data processing. To improve the dilemma, in this study, modified multi-scale symbolic dynamic entropy (MMSDE) is adopted to identify the damaged location of the civil structure. A damage index (DI) based on the entropy diagram is also proposed to clearly indicate the damage location. A seven-story numerical model was created to verify the efficiency of the proposed SHM system. The results of the analysis of each case of damage show that the MMSDE curve for the damaged floor is lower than that for the healthy floor, and the structural damage can be correctly diagnosed by the damage index. Subsequently, a scaled-down steel benchmark experiment, including 15 damage cases, was conducted to verify the practical performance of the SHM system. The confusion matrix was used to further evaluate the SHM system. The results demonstrated that the MMSD-based system can quickly diagnose structural safety with reliability and accuracy. It can be used in the field of long-term structural health monitoring in the near future.     

A Bayesian Entropy Approach to Sectoral Systemic Risk Modeling

We investigate the dynamics of systemic risk of European companies using an approach that merges paradigmatic risk measures such as Marginal Expected Shortfall, CoVaR, and Delta CoVaR, with a Bayesian entropy estimation method. Our purpose is to bring to light potential spillover effects of the entropy indicator for the systemic risk measures computed on the 24 sectors that compose the STOXX 600 index. Our results show that several sectors have a high proclivity for generating spillovers. In general, the largest influences are delivered by Capital Goods, Banks, Diversified Financials, Insurance, and Real Estate. We also bring detailed evidence on the sectors that are the most pregnable to spillovers and on those that represent the main contributors of spillovers.     

熵的一种通俗教法

熵如力、能量和动量一样是物理学中一个重要概念,若能用一种通俗易懂的方法设计熵的教学,对文科物理的教学有重要意义.为此本文提出了一种通俗的熵的教法,这一教法不需要学生学习热力学第二定律也可以建立熵的概念.具体教学设计如下:通过日常生活例子引入熵的概念(也就是玻尔兹曼熵),设计两个例子让学生会计算熵,通过具体问题的讨论让学生充分理解熵的意义,通过一个实例由玻尔兹曼熵引入克劳修斯熵公式,设计一个演示实验强化教学效果,将熵与环境保护联系起来融入人文情怀,最后还强调了熵计算的不同层次.教学设计完全采用基于问题学习(PBL)的教学模式.     

Selecting an Effective Entropy Estimator for Short Sequences of Bits and Bytes with Maximum Entropy

Entropy makes it possible to measure the uncertainty about an information source from the distribution of its output symbols. It is known that the maximum Shannon’s entropy of a discrete source of information is reached when its symbols follow a Uniform distribution. In cryptography, these sources have great applications since they allow for the highest security standards to be reached. In this work, the most effective estimator is selected to estimate entropy in short samples of bytes and bits with maximum entropy. For this, 18 estimators were compared. Results concerning the comparisons published in the literature between these estimators are discussed. The most suitable estimator is determined experimentally, based on its bias, the mean square error short samples of bytes and bits.     

基于改进功率谱熵的抑郁症脑电信号活跃性研究

采用非线性动力学方法研究脑精神疾病是近年来国内外学者研究的热点和趋势.针对脑精神疾病的研究和诊断中缺少客观有效的量化参数和量化指标的状况,提出了一种根据对时间序列功率谱划分而定义的谱熵,然后用其计算和分析脑电信号谱熵的方法.通过数据仿真试验证明该谱熵和信号活跃性之间存在正相关关系.基于这种相关性,应用该方法对抑郁症患者和正常对照组的脑电信号功率谱熵进行了数值计算,然后进行了分析对比和统计检验.实验结果表明:抑郁症患者脑电信号的功率谱熵在部分脑区显著弱于正常健康人.证明该谱熵能够表征大脑电生理活动状况,提供反映其活动性强弱的信息,可以作为度量大脑电生理活动性的一个参数.这对于能否将该功率谱熵作为诊断脑精神疾病的物理参数具有积极意义.     

等概率符号化样本熵应用于脑电分析

样本熵(或近似熵)以信息增长率刻画时间序列的复杂性,能应用于短时序列,因而在生理信号分析中被广泛采用.然而,一方面由于传统样本熵采用与标准差线性相关的容限,使得熵值易受非平稳突变干扰的影响,另一方面传统样本熵还受序列概率分布的影响,从而导致其并非单纯反映序列的信息增长率.针对上述两个问题,将符号动力学与样本熵结合,提出等概率符号化样本熵方法,并对其物理意义、数学推导及参数选取都做了详细阐述.通过对噪声数据的仿真计算,验证了该方法的正确性及其区分不同强度时间相关的有效性.此方法应用于脑电信号分析的结果表明,在不对信号做人工伪迹去除的前提下,只需要1.25 s的脑电信号即可有效地区分出注意力集中和注意力发散两种状态.这进一步证明了该方法可很好地抵御非平稳突变干扰,能快速获得短时序列的潜在动力学特性,对脑电生物反馈技术具有很大的应用价值.     

Complexity-entropy causality plane: A useful approach for distinguishing songs

Nowadays we are often faced with huge databases resulting from the rapid growth of data storage technologies. This is particularly true when dealing with music databases. In this context, it is essential to have techniques and tools able to discriminate properties from these massive sets. In this work, we report on a statistical analysis of more than ten thousand songs aiming to obtain a complexity hierarchy. Our approach is based on the estimation of the permutation entropy combined with an intensive complexity measure, building up the complexity-entropy causality plane. The results obtained indicate that this representation space is very promising to discriminate songs as well as to allow a relative quantitative comparison among songs. Additionally, we believe that the here-reported method may be applied in practical situations since it is simple, robust and has a fast numerical implementation.     

Asymptotic Normality for Plug-In Estimators of Generalized Shannon’s Entropy

Shannon’s entropy is one of the building blocks of information theory and an essential aspect of Machine Learning (ML) methods (e.g., Random Forests). Yet, it is only finitely defined for distributions with fast decaying tails on a countable alphabet. The unboundedness of Shannon’s entropy over the general class of all distributions on an alphabet prevents its potential utility from being fully realized. To fill the void in the foundation of information theory, Zhang (2020) proposed generalized Shannon’s entropy, which is finitely defined everywhere. The plug-in estimator, adopted in almost all entropy-based ML method packages, is one of the most popular approaches to estimating Shannon’s entropy. The asymptotic distribution for Shannon’s entropy’s plug-in estimator was well studied in the existing literature. This paper studies the asymptotic properties for the plug-in estimator of generalized Shannon’s entropy on countable alphabets. The developed asymptotic properties require no assumptions on the original distribution. The proposed asymptotic properties allow for interval estimation and statistical tests with generalized Shannon’s entropy.     

Entropy: From Thermodynamics to Information Processing

Entropy is a concept that emerged in the 19th century. It used to be associated with heat harnessed by a thermal machine to perform work during the Industrial Revolution. However, there was an unprecedented scientific revolution in the 20th century due to one of its most essential innovations, i.e., the information theory, which also encompasses the concept of entropy. Therefore, the following question is naturally raised: “what is the difference, if any, between concepts of entropy in each field of knowledge?” There are misconceptions, as there have been multiple attempts to conciliate the entropy of thermodynamics with that of information theory. Entropy is most commonly defined as “disorder”, although it is not a good analogy since “order” is a subjective human concept, and “disorder” cannot always be obtained from entropy. Therefore, this paper presents a historical background on the evolution of the term “entropy”, and provides mathematical evidence and logical arguments regarding its interconnection in various scientific areas, with the objective of providing a theoretical review and reference material for a broad audience.     

H函数在分子内振动能量再分配中的代数方法:水和氰化氘的共振研究

运用经典哈密顿代数方法 ,结合陪集表象构造了类比于热力学中熵的H函数。研究表明 ,H函数随时间增加而增大的特性可以用来表示分子内振动能量传递的方向和限度。同时 ,H函数的速率能够很好地表明水和氰化氘体系的共振特性