Physical entropy, information entropy and their evolution equations

Inspired by the evolution equation of nonequilibrium statistical physics entropy and the concise statistical formula of the entropy production rate, we develop a theory of the dynamic information entropy and build a nonlinear evolution equation of the information entropy density changing in time and state variable space. Its mathematical form and physical meaning are similar to the evolution equation of the physical entropy: The time rate of change of information entropy density originates together from drift, diffusion and production. The concise statistical formula of information entropy production rate is similar to that of physical entropy also. Furthermore, we study the similarity and difference between physical entropy and information entropy and the possible unification of the two statistical entropies, and discuss the relationship among the principle of entropy increase, the principle of equilibrium maximum entropy and the principle of maximum information entropy as well as the connection between them and the entropy evolution equation.     

Moment information and entropy evaluation for probability densities

How much information does the sequence of integer moments carry about the corresponding unknown absolutely continuous distribution? We prove that a reliable evaluation of the corresponding Shannon entropy can be done by exploiting some known theoretical results on the entropy convergence, uniquely involving exact moments without solving the underlying moment problem. All the procedure essentially rests on the solution of linear systems, with nearly singular matrices, and hence it requires both calculations in high precision and a pre-conditioning technique. Numerical examples are provided to support the theoretical results.     

Extension of quantum information by using entropy of deterministic functions

The concept of entropy of random variables first defined by Shannon has been generalized later in various ways by mathematicians who so obtained new measures of uncertainty, again for random variables. Recently, the author suggested another extension which provides a meaningful definition for the entropy of deterministic functions, both in the sense of Shannon and of Renyi. These measures of uncertainty are different from those which are utilized by physicists in the study of chaotic dynamics, like the Kolmogorov entropy for instance.

The aim of this paper is to go a step further, and to derive measures of uncertainty for operators, by using exactly the same rationale. After a short background on the entropies of deterministic functions, one obtains successively the entropy of a constant square matrix operator, the entropy of a varying square matrix operator, the entropy of the kernel of an integral transformation, and the entropy of differential operators defined by square matrices.

Then one carefully exhibits the relation which exists between these results and the quantum mechanical entropy first introduced by Von Neumann, and one so obtains a new generalized quantum mechanical entropy which applies to matrics which are not necessarily density matrices. Finally, some illustrative examples for future applications are outlined.     

Measuring agility of networked organizational structures via network entropy and mutual information

While the agility of networked organizational structures is important for organizational performance, studies on how to evaluate it remain scant, probably because the difficulty in measuring network evolution. In this conceptual paper, we propose two measures - network entropy and mutual information - to characterize the agility of networked organizational structure. Rooted in graph theory and information theory, these two measures capture network evolution in a comprehensive and parsimonious way. They indicate the uncertainty (or disorder) at the network level as well as the degree distribution at the individual level. We also propose an algorithm for applying them in the scenario of adding links to a network while holding the number of nodes fixed. Both simulated and real networks are used for demonstration. Implications and areas for future research are discussed in the end.     

Application of quorum response and information entropy to animal collective motion modeling

“Quorum response” is a type of social interaction in which an individual's chance of choosing an option is a nonlinear function of the number of other individuals already committing to it. This interaction has been widely used to characterize collective decision‐making in animal groups. Here, we first implement it in 1D and 2D models of collective animal movement, and find that the resulting group motion shows the characteristic behaviors which were observed in previous experimental and modeling studies. Further, the analytic form of quorum response renders us an opportunity to propose a mean field theory in 1D with globally interacting particles, so we can estimate the average time period between changes in the group direction (mean switching time). We find that the theoretical results provide an upper bound to the simulation results when the interaction radius grows from local to global. Information entropy, a concept widely used to quantify the uncertainty of a random variable, is introduced here as a new order parameter to study the evolution of systems of two cases in 2D models. The explicitly formulated probability of a particle's dynamic state in the framework of quorum response makes information entropy directly computable. We find that, besides the global order, information entropy can also capture the structural features of local order of the system which previous order parameters such as alignment cannot. © 2016 Wiley Periodicals, Inc. Complexity 21: 584–592, 2016     

Effective transfer entropy approach to information flow between exchange rates and stock markets

We investigate the strength and direction of information flow between exchange rates and stock prices in several emerging countries by the novel concept of effective transfer entropy (an alternative non-linear causality measure) with symbolic encoding methodology. Analysis shows that before the 2008 crisis, only low level interaction exists between these two variables and exchange rates dominate stock prices in general. During crisis, strong bidirectional interaction arises. In the post-crisis period, the strong interaction continues to exist and in general stock prices dominate exchange rates.     

Contrast enhancement for image by WNN and GA combining PSNR with information entropy

A new contrast enhancement algorithm for image is proposed combining genetic algorithm (GA) with wavelet neural network (WNN). In-complete Beta transform (IBT) is used to obtain non-linear gray transform curve so as to enhance global contrast for an image. GA determines optimal gray transform parameters. In order to avoid the expensive time for traditional contrast enhancement algorithms, which search optimal gray transform parameters in the whole parameters space, based on gray distribution of an image, a classification criterion is proposed. Contrast type for original image is determined by the new criterion. Parameters space is, respectively, determined according to different contrast types, which greatly shrink parameters space. Thus searching direction of GA is guided by the new parameter space. Considering the drawback of traditional histogram equalization that it reduces the information and enlarges noise and background blur in the processed image, a synthetic objective function is used as fitness function of GA combining peak signal-noise-ratio (PSNR) and information entropy. In order to calculate IBT in the whole image, WNN is used to approximate the IBT. In order to enhance the local contrast for image, discrete stationary wavelet transform (DSWT) is used to enhance detail in an image. Having implemented DSWT to an image, detail is enhanced by a non-linear operator in three high frequency sub-bands. The coefficients in the low frequency sub-bands are set as zero. Final enhanced image is obtained by adding the global enhanced image with the local enhanced image. Experimental results show that the new algorithm is able to well enhance the global and local contrast for image while keeping the noise and background blur from being greatly enlarged.     

Maximum entropy reconstruction using derivative information part 2: computational results

Summary. Maximum entropy density estimation, a technique for reconstructing an unknown density function on the basis of certain measurements, has applications in various areas of applied physical sciences and engineering. Here we present numerical results for the maximum entropy inversion program based on a new class of information measures which are designed to control derivative values of the unknown densities. Received January 3, 1994 / Revised version received May 25, 1994     

A maximum entropy approach to the newsvendor problem with partial information

In this paper, we consider the newsvendor model under partial information, i.e., where the demand distribution D is partly unknown. We focus on the classical case where the retailer only knows the expectation and variance of D. The standard approach is then to determine the order quantity using conservative rules such as minimax regret or Scarf’s rule. We compute instead the most likely demand distribution in the sense of maximum entropy. We then compare the performance of the maximum entropy approach with minimax regret and Scarf’s rule on large samples of randomly drawn demand distributions. We show that the average performance of the maximum entropy approach is considerably better than either alternative, and more surprisingly, that it is in most cases a better hedge against bad results.     

Utilizing the information theory of entropy to solve an off-line inspection problem

This study presents an off-line inspection problem for a batch produced from a process subject to random failures and exhibiting manufacturing variations. The objective of this paper is to develop an inspection policy in which units should be inspected in a particular order to find the transition unit in the batch under a required confidence level. This study develops an algorithm to compute the expected number of inspections. This approach uses the information theory of entropy to select an un-inspected unit to be inspected, and effectively minimizes the uncertainty of the transition unit in the production batch. A numerical example illustrates the proposed off-line inspection policy, and the effects of model parameters on the expected inspection number are investigated. The numerical example in this study indicates that full inspection is required when the required confidence level is one or the process has larger manufacturing variations.     

Estimating market shares in each market segment using the information entropy concept

Sales data of a certain product for the various competitors are usually available at the aggregate level. However these data give no clue to the heterogeneities in the sales pattern across different market segments. Heterogeneities are caused by different purchasing behavior in each market segment; as a purchaser in a segment will be attracted to the attributes of the product most important to that segment. This concept can be formalized via a simple attraction model that utilizes an elasticity measure for each quality or price attribute [G.S. Carpenter, L.G. Cooper, D.M. Hanssens, D.F. Midgley, Modeling asymmetric competition, Marketing Science 7 (4) (1998) 393–412]. Assessment of these elasticities is not difficult since customer response – in each market segment – to perception of quality and price is tracked by most firms [J. Ross, D. Georgoff, A survey of productive and quality issues in manufacturing. The state of the industry, Industrial Management 3 (5) (1991) 22–25]. This paper attempts to formulate a generic framework based on the information entropy concept that utilizes such an attraction model to estimate competitors’ sales in each market segment.     

基于熵值法的区域旅游业经济效益比较分析

为了克服区域旅游业经济效益比较分析中采用主观赋权法所带来的局限性,于是借助于信息工程学中的“熵”概念,利用熵的大小度量各指标在不同区域旅游业之间的差异程度,较为客观地揭示出各指标的重要性,同时也给出了基于熵值法的比较分析实例。     

Nonsymmetric entropy and maximum nonsymmetric entropy principle

Under the frame of a statistical model, the concept of nonsymmetric entropy which generalizes the concepts of Boltzmann’s entropy and Shannon’s entropy, is defined. Maximum nonsymmetric entropy principle is proved. Some important distribution laws such as power law, can be derived from this principle naturally. Especially, nonsymmetric entropy is more convenient than other entropy such as Tsallis’s entropy in deriving power laws.     

基于熵权的投资评价模型在风险投资中的应用

本着“实用性和现实操作性”原则,本文根据风险投资评价的实际操作,在引入粗糙集信息熵理论,导出基于多指标评价的熵权投资模型的基础上,通过问卷调查的实证研究方法,确定评价指标和权重,并例举实际(经适当简化)案例演算具体运算过程,以验证在实际风险投资中的可操作性。从而试图克服目前相关领域研究文献基本停留在方法研究阶段、所给的证例过于简单、没有实际运用价值的缺陷,也尝试探索粗糙集理论在风险投资管理中的应用。     

The role of duality in optimization problems involving entropy functionals with applications to information theory

We consider infinite-dimensional optimization problems involving entropy-type functionals in the objective function as well as as in the constraints. A duality theory is developed for such problems and applied to the reliability rate function problem in information theory.This research was supported by ONR Contracts N00014-81-C-0236 and N00014-82-K-0295 with the Center for Cybernetics Studies, University of Texas, Austin, Texas. The first author was partly supported by NSF.