Entropy of Hidden Markov Processes via Cycle Expansion

Hidden Markov Processes (HMP) is one of the basic tools of the modern probabilistic modeling. The characterization of their entropy remains however an open problem. Here the entropy of HMP is calculated via the cycle expansion of the zeta-function, a method adopted from the theory of dynamical systems. For a class of HMP this method produces exact results both for the entropy and the moment-generating function. The latter allows to estimate, via the Chernoff bound, the probabilities of large deviations for the HMP. More generally, the method offers a representation of the moment-generating function and of the entropy via convergent series.     

Entropy Preservation Under Markov Coding

Using the notion of topological entropy for non-compact sets we prove that for a C ¹⁺ -map with a finite Markov partition the corresponding coding map preserves topological entropy of subsets. We also provide an example of a piecewise linear conformal repeller with a Markov coding decreasing topological entropy. These results are generalized to the notions of u-dimensions.     

基于Contourlet域隐马尔可夫树模型的图像融合算法

本文提出一种基于Contourlet域隐马尔可夫树(HMT)模型的图像融合算法。由于Contourlet变换能克服小波变换在处理高维信号时的不足,它比小波变换具有更好的方向性、较高的逼近精度和更好的稀疏表达性能。而HMT模型能有效捕获尺度间、尺度内的contourlet系数特性。因此将Contourlet域HMT模型应用于图像融合领域,能充分挖掘数据之间的相关性,更好的提取图像边缘特征,为融合提取更多的特征信息。实验结果表明本文的算法获得的融合图像视觉效果良好,是一种有效且可行的融合算法。     

Hidden Markov models revealing the stress field underlying the earthquake generation

The application of the hidden Markov models (HMMs) is attempted for revealing key features for the earthquake generation which are not accessible to direct observation. Considering that the states of the HMM correspond to levels of the stress field, our objective is to identify these states. The observations are considered after grouping earthquake magnitudes and the cases of different number of states are examined. The problems of HMMs theory are solved and the ensuing HMMs are compared on the basis of Akaike and Bayesian information criteria. A new insight on the evaluation of future seismic hazard is given by calculating the mean number of steps for the first visit to a particular state, along with the respective variance. We further calculate an estimator of the mean number of steps for the first visit to a particular state and we construct its confidence interval. Additionally, a second approach to the problem is followed by assuming a different determination of observations. The HMMs applied to both approaches, contribute significantly to seismic hazard assessment via revealing the number of the stress levels as well as the way in which these levels are associated with certain earthquake occurrence.     

Distribution of Avalanche Sizes in the Hysteretic Response of the Random-Field Ising Model on a Bethe Lattice at Zero Temperature

We consider the zero-temperature single-spin-flip dynamics of the random-field Ising model on a Bethe lattice in the presence of an external field h. We derive the exact self-consistent equations to determine the distribution Prob(s) of avalanche sizes s as the external field increases from – to . We solve these equations explicitly for a rectangular distribution of the random fields for a linear chain and the Bethe lattice of coordination number z=3, and show that in these cases, Prob(s) decreases exponentially with s for large s for all h on the hysteresis loop. We find that for z4 and for small disorder, the magnetization shows a first-order discontinuity for several continuous and unimodal distributions of the random fields. The avalanche distribution Prob(s) varies as s ^–3/2 for large s near the discontinuity.     

The Markov property method applied to Ising model calculations

An efficient method of computation for models possessing the Markov property is set out. We apply this method to the two-dimensional ising model and report exact computations for up to 10 by 10 models with periodic boundary conditions. We find that critical-point, finite-size rounding is quite large in the renormalized coupling constant, which is not divergent at the critical point, in contrast to the energy, which is also not divergent and has no rounding there. The difference is traced to the continuity of the energy and the discontinuity of the renormalized coupling constant at the critical point.     

A Transformer-Based Hierarchical Variational AutoEncoder Combined Hidden Markov Model for Long Text Generation

The Variational AutoEncoder (VAE) has made significant progress in text generation, but it focused on short text (always a sentence). Long texts consist of multiple sentences. There is a particular relationship between each sentence, especially between the latent variables that control the generation of the sentences. The relationships between these latent variables help in generating continuous and logically connected long texts. There exist very few studies on the relationships between these latent variables. We proposed a method for combining the Transformer-Based Hierarchical Variational AutoEncoder and Hidden Markov Model (HT-HVAE) to learn multiple hierarchical latent variables and their relationships. This application improves long text generation. We use a hierarchical Transformer encoder to encode the long texts in order to obtain better hierarchical information of the long text. HT-HVAE’s generation network uses HMM to learn the relationship between latent variables. We also proposed a method for calculating the perplexity for the multiple hierarchical latent variable structure. The experimental results show that our model is more effective in the dataset with strong logic, alleviates the notorious posterior collapse problem, and generates more continuous and logically connected long text.     

Variational Beta Process Hidden Markov Models with Shared Hidden States for Trajectory Recognition

Hidden Markov model (HMM) is a vital model for trajectory recognition. As the number of hidden states in HMM is important and hard to be determined, many nonparametric methods like hierarchical Dirichlet process HMMs and Beta process HMMs (BP-HMMs) have been proposed to determine it automatically. Among these methods, the sampled BP-HMM models the shared information among different classes, which has been proved to be effective in several trajectory recognition scenes. However, the existing BP-HMM maintains a state transition probability matrix for each trajectory, which is inconvenient for classification. Furthermore, the approximate inference of the BP-HMM is based on sampling methods, which usually takes a long time to converge. To develop an efficient nonparametric sequential model that can capture cross-class shared information for trajectory recognition, we propose a novel variational BP-HMM model, in which the hidden states can be shared among different classes and each class chooses its own hidden states and maintains a unified transition probability matrix. In addition, we derive a variational inference method for the proposed model, which is more efficient than sampling-based methods. Experimental results on a synthetic dataset and two real-world datasets show that compared with the sampled BP-HMM and other related models, the variational BP-HMM has better performance in trajectory recognition.     

Statistical Entropy of a Lattice-Gas Model: Multiparticle Correlation Expansion

A formula expressing the statistical entropy of a lattice-gas model as a multiparticle correlation expansion is derived in the grand-canonical and in the canonical ensembles. The differences from the analogous expansion in the continuum case are elucidated. The Ising model in one dimension is discussed as a case study.     

基于隐马尔可夫链模型的软件维护性评估方法研究

针对当前软件可维护性评估主观性强,可操作性弱等问题,提出了定量描述维护性的维护时间统计概率描述方法,引入隐马尔可夫链(HMC)模型对维护性状态变迁过程进行描述,以可度量的维护性内部属性影响因素集量化值为观测序列,以维护时间统计概率为状态序列,构造了反映可维护性状态转移的HMC模型。收集配置管理库中软件模块历史维护时间从而确定完成维护任务频率来估计软件维护性初始状态,利用复杂网络特性计算软件维护性影响因素集的量化值,理论上即可评估出当前软件所处的维护性状态,最后运用实例对模型进行了训练与评估。结果表明,利用模型评估出的概率与实际维护任务统计出的可维护性概率基本一致,说明该方法可行且可重复,具有一定实践意义和研究前景。     

Stability of hierarchical interfaces in a random field model

We study a hierarchical model for interfaces in a random-field ferromagnet. We prove that in dimensionD>3, at low temperatures and for weak disorder, such interfaces are rigid. Our proof uses renormalization group transformations for stochastic sequences.     

Dynamical Entropy of Generalized Quantum Markov Chains on Gauge-invariant C *-algebras

We prove that the mean entropy and the dynamical entropy are equal for generalized quantum Markov chains on gauge-invariant C ^*-algebras.     

Renormalization group for Markov chains and application to metastability

In this paper we introduce a new renormalization group method for the study of the long-time behavior of Markov chains with finite state space and with transition probabilities exponentially small in an external parameter. A general approach of metastability problems emerges from this analysis and is discussed in detail in the case of a two-dimensional Ising system at low temperature,     

Characterization of fish schooling behavior with different numbers of Medaka (Oryzias latipes) and goldfish (Carassius auratus) using a Hidden Markov Model

Fish that swim in schools benefit from increased vigilance, and improved predator recognition and assessment. Fish school size varies according to species and environmental conditions. In this study, we present a Hidden Markov Model (HMM) that we use to characterize fish schooling behavior in different sized schools, and explore how school size affects schooling behavior. We recorded the schooling behavior of Medaka (Oryzias latipes) and goldfish (Carassius auratus  ) using different numbers of individual fish (10–40), in a circular aquarium. Eight to ten 3 s video clips were extracted from the recordings for each group size. Schooling behavior was characterized by three variables: linear speed, angular speed, and Pearson coefficient. The values of the variables were categorized into two events each for linear and angular speed (high and low), and three events for the Pearson coefficient (high, medium, and low). Schooling behavior was then described as a sequence of 12 events (2×2×3$2\times 2\times 3$), which was input to an HMM as data for training the model. Comparisons of model output with observations of actual schooling behavior demonstrated that the HMM was successful in characterizing fish schooling behavior. We briefly discuss possible applications of the HMM for recognition of fish species in a school, and for developing bio-monitoring systems to determine water quality.     

Calculating Topological Entropy

The attempt to find effective algorithms for calculating the topological entropy of piecewise monotone maps of the interval having more than three monotone pieces has proved to be a difficult problem. The algorithm introduced here is motivated by the fact that if f: [0, 1] → [0, 1] is a piecewise monotone map of the unit interval into itself, thenh(f)=lim_n→∞ (1/n) log Var(f ⁿ), where h(f) is the topological entropy off, and Var(f ⁿ) is the total variation off ⁿ. We show that it is not feasible to use this formula directly to calculate numerically the topological entropy of a piecewise monotone function, because of the slow convergence. However, a close examination of the reasons for this failure leads ultimately to the modified algorithm which is presented in this paper. We prove that this algorithm is equivalent to the standard power method for finding eigenvalues of matrices (with shift of origin) in those cases for which the function is Markov, and present encouraging experimental evidence for the usefulness of the algorithm in general by applying it to several one-parameter families of test functions.     

Entropy Treatment of Evolution Algebras

In this paper, by introducing an entropy of Markov evolution algebras, we treat the isomorphism of S-evolution algebras. A family of Markov evolution algebras is defined through the Hadamard product of structural matrices of non-negative real S-evolution algebras, and their isomorphism is studied by means of their entropy. Furthermore, the isomorphism of S-evolution algebras is treated using the concept of relative entropy.     

Entropy of spin clusters with frustrated geometry

Geometrically frustrated clusters of Ising spins of different shapes on a triangular lattice are studied by exact enumeration. The focus is laid on the ground-state energy and residual entropy behaviors as functions of the cluster shape and size, as well as the spin value. Depending on the cluster shape, the residual entropy density in approach to the thermodynamic limit can either vanish or remain finite and the dependence can be decreasing, increasing or non-monotonic. Nevertheless, the relative entropies normalized by the respective thermodynamic limit values turn out to be little sensitive to the spin value. Attention is drawn to magnetocaloric properties of systems of selected cluster shapes.     

基于互信息熵-近红外光谱的过程模式故障检测

近红外光谱分析在工业过程故障检测方面具有独特的优势,是一种准确且高效的方法。结合互信息熵和传统的主成分分析,对近红外光谱特征信息进行提取,通过构建过程的模式来刻画工业过程的运行状态。利用近红外光谱数据,从有机分子含氢基团振动信息中获取工业系统的过程模式,从微观分子层面探索提高工业过程故障检测准确率的有效方法,结合贝叶斯统计学习技术,提出了基于近红外光谱数据的工业过程故障检测技术。针对近红外光谱信息量丰富,谱带较宽,特征性不强的特点,首先对工业过程不同运行状态下的近红外光谱吸光度数据进行一阶导数预处理,采用主成分分析法（principal component analysis,PCA）压缩光谱数据量,扩大不同运行状态下光谱特征信息的差异性,提取光谱的内部特征信息。然后采用互信息熵（mutual information entropy,MIE）作为光谱特征信息相关性度量函数,基于最小冗余最大相关算法进一步减少光谱特征信息间的冗余并最大化光谱特征信息与类别的相关性,弥补了PCA无监督特征波长选择的不足,提出一种基于PCA-MIE的过程模式构建方法,获得的过程模式子集更紧凑更具类别表现力。再利用贝叶斯统计学习算法,根据后验概率对构建的模式子集进行决策,判别生产过程的正常状态和故障状态。由于过程模式子集结合了PCA浓聚方差的优势和互信息熵相关性测度的特征信息选择方法,蕴含了更多的近红外光谱的本质信息与内在规律,从而更能刻画工业过程的运行状态。接着,设置测试准确率TA作为评估标准,用以评价故障检测方法的性能效果。最后利用某化工厂提供的原油脱盐脱水过程近红外光谱数据对所提方法进行验证,并与传统近红外光谱特征信息提取方法PCA和MIE方法性能进行对比分析,结果表明基于PCA-MIE的过程模式故障检测方法几乎在所有维数子集上性能都优于其他两种方法,在特征维数为18维时获得最高的准确率94. 6%,证明了方法的优越性。     

Entropy description of a cooperation-competition system

Understanding the cooperation-competition dynamics is a long-standing challenge in studying complex systems.Inspired by the idea of Shannon entropy,we define competition information entropy and propose an entropy evolution model.The analytic results of the model of the relation between competition gain distribution parameters and entropy,as well as the relation between entropy and time are compared with empirical results obtained in 14 real world systems.They are found to be in good agreement with each other.     

Optical Entropy Network Mapped from Binary Feature Tree

The concept of a binary feature tree (BFT) and the principle of its formation are described. A pattern is divided into sub-parts by comparing its similarity with other patterns. The BFT is established by sub-parts of a group of patterns and mapped into a three layered neural network which Sethi called an entropy network. The interconnection pattern between the first and hidden layers is formed according to the “AND“ relationship of node feature patterns determined by BFT. The interconnection pattern between the hidden and last layers is obtained by training. The advantage of the proposed network is that the scale is small because a feature neuron is adopted and the interconnection is local instead of full; therefore, it is easily implemented by either hardware or software. Two simulation examples show the success of the entropy network for pattern recognition. A feature extraction by an optical inner product method is also described.Presented at 1996 International Topical Meeting on Optical Computing (OC ‘96), April 21-25, Sendai, Japan.