The principle of the maximum entropy method

Abstract

The theoretical background of the maximum entropy method (MEM) when it is applied to restore the electron or nuclear densities from diffraction data is described. In MEM, the concept of “entropy” is introduced to deal with any incompleteness in an observation in a proper way. An incompleteness causes some ambiguities in the results to some extent. The essence of the method is to find a solution which necessarily agrees with the observation, leaving the measure of ambiguities (entropy) maximum. A few results for simple structures with typical types of chemical bonding are also presented.     

The maximum entropy principle as a consequence of the principle of Laplace

The maximum entropy principle states that the probability distribution which best represents our information is the one which maximizes the entropy with the given evidence as constraints. We prove that this principle is implied from the Laplace principle of equiprobabilities applied to the setS of allN-term sequences of results which are compatible with the given evidence. We generalize to the information gain method of Kullback.     

Some interesting consequences of the maximum entropy production principle

Two nonequilibrium phase transitions (morphological and hydrodynamic) are analyzed by applying the maximum entropy production principle. Quantitative analysis is for the first time compared with experiment. Nonequilibrium crystallization of ice and laminar-turbulent flow transition in a circular pipe are examined as examples of morphological and hydrodynamic transitions, respectively. For the latter transition, a minimum critical Reynolds number of 1200 is predicted. A discussion of this important and interesting result is presented.     

Application of the maximum information entropy principle to selforganizing systems

Self-organizing systems are systems which can acquire macroscopic spatial, temporal, or spatio-temporal structures by means of internal processes. Hitherto the distribution functions of the order parameters governing the macroscopic structures could be calculated by microscopic theories only. In the present paper we derive them from macroscopic quantities, where we demonstrate the procedure explicitly by means of the single and multimode laser close to the lasing threshold.     

The principle of maximum entropy and Boltzmann type kinetic equations

The problem of the paper is the possibility of a dynamical justification of the principle of maximum entropy in the sense of a dynamical semigroup of open systems. It has been shown that, under the assumption of a convex dynamical semigroup defined on discrete and finite probability distributions (a finite sample space), this principle cannot be realized. This is possible, however, for non-linear dynamical semigroups for some random variables called p-collision-type variables in analogy to the Boltzmann 2-collision problem.     

On the axiomatic approach to the maximum entropy principle of inference

Recent axiomatic derivations of the maximum entropy principle from consistency conditions are critically examined. We show that proper application of consistency conditions alone allows a wider class of functionals, essentially of the form ∝ dx p(x)[p(x)/g(x)] ^s , for some real numbers, to be used for inductive inference and the commonly used form − ∝ dx p(x)ln[p(x)/g(x)] is only a particular case. The role of the prior densityg(x) is clarified. It is possible to regard it as a geometric factor, describing the coordinate system used and it does not represent information of the same kind as obtained by measurements on the system in the form of expectation values.     

From dendrites and S-shaped growth curves to the maximum entropy production principle

S-shaped kinetic curves are very frequent in nature. Based on our own experimental evidence on the growth of single dendrites and analysis of literature data, we have demonstrated that such curves may result from the maximum entropy production principle. The proposed approach also explains other prevalent laws of relaxation in nonequilibrium systems (exponential, Kohlrausch, etc.).     

A practical computational framework for the multidimensional moment-constrained maximum entropy principle

The maximum entropy principle is a versatile tool for evaluating smooth approximations of probability density functions with a least bias beyond given constraints. In particular, the moment-based constraints are often a common prior information about a statistical state in various areas of science, including that of a forecast ensemble or a climate in atmospheric science. With that in mind, here we present a unified computational framework for an arbitrary number of phase space dimensions and moment constraints for both Shannon and relative entropies, together with a practical usable convex optimization algorithm based on the Newton method with an additional preconditioning and robust numerical integration routine. This optimization algorithm has already been used in three studies of predictability, and so far was found to be capable of producing reliable results in one- and two-dimensional phase spaces with moment constraints of up to order 4. The current work extensively references those earlier studies as practical examples of the applicability of the algorithm developed below.     

Information flow and maximum entropy measures for 1-D maps

The invariant measures of maximal metric entropy are constructed explicitly for some maps of the interval, by iterating the maps backward. The construction illustrates in a particularly clear way the information flow in simple systems, as well as recently conjectured relationships between dimensions of invariant measures, Lyapunov exponents, and entropies. maps, it is conjectured that the natural measure is the invariant measure with strongest mixing.     

Geodesic acoustic eigenmode for tokamak equilibrium with maximum of local GAM frequency

The geodesic acoustic eigenmode for tokamak equilibrium with the maximum of local GAM frequency is found analytically in the frame of MHD model. The analysis is based on the asymptotic matching technique.     

Free energy and entropy production in planar Couette flow far from equilibrium

The macroscopic thermodynamic stability of a system of 108 diatomic molecules undergoing planar Couette flow far from equilibrium is reported. The system is perturbed from the steady state using a nondissipative variable colour field which induces a polarization in the system. It is found that the steady state for the system corresponds to an extremum in the generalized free energy and entropy production. However, while the free energy is always a minimum, the entropy production may be either a minimum or a maximum depending upon the direction of the colour field. These results, for a molecular system, are fundamentally different from those for an equivalent atomic system.     

Application of the principle of maximum entropy production to the analysis of the morphological stability of a growing crystal

The morphological stability of spherical and cylindrical crystals and an infinite plane growing from a supersaturated solution is studied using the principle of maximum entropy production in the Mullins and Sekerka approximation. In contrast to the first two geometries, the computational results for a plane agree completely with the results obtained on the basis of the classical linear perturbation theory. The concept of the binodal of a morphological transition is introduced in order to interpret the results for the sphere and cylinder. The boundaries of the metastable region are investigated. Morphological phase diagrams of stable-unstable growth are presented in terms of the variables surface energy and supersaturation as well as the variables crystal size and supersaturation. The physical nature of the appearance of metastability in this system is discussed.     

Principle of maximum entropy and irreversible processes

More than twenty years have passed since E.T. Jaynes pioneered the information-theoretic approach to the foundations of statistical mechanics. In the intervening period numerous applications have been worked out and new insights developed into traditional problems. Although the natural extension of these ideas to nonequilibrium statistical mechanics and irreversible processes has also been formulated, and both theory and applications have been considerably developed, much of this work has not appeared in the open literature. Yet the basic ideas and consequent theory seem to have an extraordinary unifying and illuminating effect on the foundations of the many-body problem, as well as providing a transparent conceptual basis from which to view various groups of problems not previously formulated in a particularly sharp way. The aim of this review is to summarize these developments in the context of the principle of maximum entropy.     

Carnot engines and the principle of increase of entropy

On the basis of an axiomatization of classical thermodynamics given in a previous paper, the existence of Carnot engines is established, and used to prove rigorously the principle of increase of entropy and Clausius' inequality for compound systems.     

Distribution of local entropy in the Hilbert space of bi-partite quantum systems: origin of Jaynes' principle

For a closed bi-partite quantum system partitioned into system proper and environment we interpret the microcanonical and the canonical condition as constraints for the interaction between those two subsystems. In both cases the possible pure-state trajectories are confined to certain regions in Hilbert space. We show that in a properly defined thermodynamical limit almost all states within those accessible regions represent states of some maximum local entropy. For the microcanonical condition this dominant state still depends on the initial state; for the canonical condition it coincides with that defined by Jaynes' principle. It is these states which thermodynamical systems should generically evolve into. Received 13 June 2002 / Received in final form 14 November 2002 Published online 4 February 2003 RID="a" ID="a"e-mail: jochen@theol.physik.uni-stuttgart.de