Filtering of continuous-time Markov chains

This paper discusses finite-dimensional optimal filters for partially observed Markov chains. A model for a system containing a finite number of components where each component behaves like an independent finite state continuous-time Markov chain is considered. Using measure change techniques various estimators are derived.     

Ergodic and adaptive control of hidden Markov models

A partially observed stochastic system is described by a discrete time pair of Markov processes. The observed state process has a transition probability that is controlled and depends on a hidden Markov process that also can be controlled. The hidden Markov process is completely observed in a closed set, which in particular can be the empty set and only observed through the other process in the complement of this closed set. An ergodic control problem is solved by a vanishing discount approach. In the case when the transition operators for the observed state process and the hidden Markov process depend on a parameter and the closed set, where the hidden Markov process is completely observed, is nonempty and recurrent an adaptive control is constructed based on this family of estimates that is almost optimal.     

On approximation of smoothing probabilities for hidden Markov models

We consider the smoothing probabilities of hidden Markov model (HMM). We show that under fairly general conditions for HMM, the exponential forgetting still holds, and the smoothing probabilities can be well approximated with the ones of double-sided HMM. This makes it possible to use ergodic theorems. As an application we consider the pointwise maximum a posteriori segmentation, and show that the corresponding risks converge.     

A correction on approximation of smoothing probabilities for hidden Markov models

In this note, we correct a mistake concerning Theorem 2.1 in Lember (2011a).     

Bayesian Monte Carlo estimation for profile hidden Markov models

Hidden Markov models are used as tools for pattern recognition in a number of areas, ranging from speech processing to biological sequence analysis. Profile hidden Markov models represent a class of so-called “left–right” models that have an architecture that is specifically relevant to classification of proteins into structural families based on their amino acid sequences. Standard learning methods for such models employ a variety of heuristics applied to the expectation-maximization implementation of the maximum likelihood estimation procedure in order to find the global maximum of the likelihood function. Here, we compare maximum likelihood estimation to fully Bayesian estimation of parameters for profile hidden Markov models with a small number of parameters. We find that, relative to maximum likelihood methods, Bayesian methods assign higher scores to data sequences that are distantly related to the pattern consensus, show better performance in classifying these sequences correctly, and continue to perform robustly with regard to misspecification of the number of model parameters. Though our study is limited in scope, we expect our results to remain relevant for models with a large number of parameters and other types of left–right hidden Markov models.     

Leroux's method for general hidden Markov models

The method introduced by Leroux [Maximum likelihood estimation for hidden Markov models, Stochastic Process Appl. 40 (1992) 127–143] to study the exact likelihood of hidden Markov models is extended to the case where the state variable evolves in an open interval of the real line. Under rather minimal assumptions, we obtain the convergence of the normalized log-likelihood function to a limit that we identify at the true value of the parameter. The method is illustrated in full details on the Kalman filter model.     

Adaptive control of three continuous-time portfolio and consumption models

An economic application of adaptive control is presented using three continuous time portfolio and consumption models that are natural generalizations of a model of Merton. In these models of the wealth of an individual investor, it is assumed that the various parameters are deterministic functions of time or stochastic processes. An adaptive control problem arises for each of these models when it is assumed that the average return rate of the risky asset, which is either a deterministic function or a stochastic process, is not observed. For these models, a recursive family of estimators of the average return rate of the risky asset is given based on the observations of the wealth. These estimates are used in the control of the wealth equation.This research was partially supported by NSF Grant No. ECS-84-03286-A01 and by University of Kansas General Research Allocation No. 3806-XO-0038.     

Optimal filters for a hidden Markov random field model

A Markov random field (MRF) is a useful technical tool for modeling dynamics systems exhibiting some type of spatio-temporal variability. In this paper, we propose optimal filters for the states of a partially observed temporal Markov random field. We also discuss parameters estimation. This generalizes an earlier work by Elliott and Aggoun [1].     

改进的带驻留时间隐Markov模型的前向-后向算法

对隐Maxkov模型(hidden Markov model:HMM)的状态驻留时间的概率进行了修订,给出了改进的带驻留时间隐Markov模型的结构,并在传统的隐Markov模型(traditional hidden Markov model:THMM)的基础上讨论了新模型的前向.后向变量,导出了新模型的前向-后向算法的迭代公式,同时也给出了新模型各个参数的重估公式.     

可列非齐次隐Markov模型的强大数定律

隐马尔科夫模型被广泛的应用于弱相依随机变量的建模,是研究神经生理学、发音过程和生物遗传等问题的有力工具。研究了可列非齐次隐 Markov 模型的若干性质,得到了这类模型的强大数定律,推广了有限非齐次马氏链的一类强大数定律。     

隐非齐次马尔可夫模型的强大数定律

在状态集都有限的情况下,给出了隐马尔可夫模型的一些性质定理．利用马氏链的强极限定理,得到了隐非齐次马尔可夫模型的强大数定律．     

A survey of Markov decision models for control of networks of queues

We review models for the optimal control of networks of queues. Our main emphasis is on models based on Markov decision theory and the characterization of the structure of optimal control policies.This research was partially supported by the National Science Foundation under Grant No. DDM-8719825. The Government has certain rights in this material. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. The research was also partially supported by the C.I.E.S. (France), while the author was on leave at INRIA, Sophia-Antipolis, 1991–92.     

Exponential bounds for convergence of entropy rate approximations in hidden Markov models satisfying a path-mergeability condition

A hidden Markov model (HMM) is said to have path-mergeable states   if for any two states i,j$i,j$ there exist a word w$w$ and state k$k$ such that it is possible to transition from both i$i$ and j$j$ to k$k$ while emitting w$w$. We show that for a finite HMM with path-mergeable states the block estimates of the entropy rate converge exponentially fast. We also show that the path-mergeability property is asymptotically typical in the space of HMM topologies and easily testable.     

Maximum likelihood estimator for hidden Markov models in continuous time

The paper studies large sample asymptotic properties of the Maximum Likelihood Estimator (MLE) for the parameter of a continuous time Markov chain, observed in white noise. Using the method of weak convergence of likelihoods due to Ibragimov and Khasminskii (Statistical estimation, vol 16 of Applications of mathematics. Springer-Verlag, New York), consistency, asymptotic normality and convergence of moments are established for MLE under certain strong ergodicity assumptions on the chain. This article has been written during the author’s visit at Laboratoire de Statistique et Processus, Universite du Maine, France, supported by the Chateaubriand fellowship.     

Selecting hidden Markov model state number with cross-validated likelihood

The problem of estimating the number of hidden states in a hidden Markov model is considered. Emphasis is placed on cross-validated likelihood criteria. Using cross-validation to assess the number of hidden states allows to circumvent the well-documented technical difficulties of the order identification problem in mixture models. Moreover, in a predictive perspective, it does not require that the sampling distribution belongs to one of the models in competition. However, computing cross-validated likelihood for hidden Markov models for which only one training sample is available, involves difficulties since the data are not independent. Two approaches are proposed to compute cross-validated likelihood for a hidden Markov model. The first one consists of using a deterministic half-sampling procedure, and the second one consists of an adaptation of the EM algorithm for hidden Markov models, to take into account randomly missing values induced by cross-validation. Numerical experiments on both simulated and real data sets compare different versions of cross-validated likelihood criterion and penalised likelihood criteria, including BIC and a penalised marginal likelihood criterion. Those numerical experiments highlight a promising behaviour of the deterministic half-sampling criterion.     

Universally consistent conditionalU-statistics for absolutely regular processes and its applications for hidden Markov models

A general class of conditionalU-statistics was introduced by W. Stute as a generalization of the Nadaraya-Watson estimates of a regression function. It was shown that such statistics are universally consistent. Also, universal consistentcies of the window andk _n -nearest neighbor estimators (as two special cases of the conditionalU-statistics) were proved. In this paper, we extend these results from the independent case to dependent case. The result is applied to verify the Bayes risk consistency of the corresponding discrimination rules. Research supported by the Office of Naval Research Contract N00014-91-J-1020.     

Interactive hidden Markov models and their applications

^** Email: wching{at}hkusua.hku.hk In this paper, we propose an Interactive hidden Markov model(IHMM). In a traditional HMM, the observable states are affecteddirectly by the hidden states, but not vice versa. In the proposedIHMM, the transitions of hidden states depend on the observablestates. We also develop an efficient estimation method for themodel parameters. Numerical examples on the sales demand dataand economic data are given to demonstrate the applicabilityof the model.     

Adaptive control of a continuous-time portfolio and consumption model

In this paper, an adaptive control problem is formulated and solved using Merton's stochastic differential equation for the wealth in a portfolio selection and consumption model. Since the asset prices are assumed to satisfy a log normal distribution, it suffices to consider two assets. It is assumed that the drift parameter for the price of the risky asset is unknown. A recursive family of estimators for this unknown parameter is defined and is shown to converge almost surely to the true value of the parameter. The controls in the equation for the wealth are obtained from the optimal controls where the estimates of the unknown parameter are substituted for the unknown parameter.This research was partially supported by NSF Grant No. ECS-84-03286-A01.The authors wish to thank P. Varaiya for some useful comments on this paper.     

Optimal control for linear discrete-time systems with Markov perturbations in Hilbert spaces

Email: vio{at}utgjiu.ro Received on September 12, 2007; Accepted on December 26, 2008 In this article, we discuss a quadratic control problem forlinear discrete-time systems with Markov perturbations in Hilbertspaces, which is linked to a discrete-time Riccati equationdefined on certain infinite-dimensional ordered Banach space.We prove that under stabilizability and stochastic uniform observabilityconditions, the Riccati equation has a unique, uniformly positive,bounded on N and stabilizing solution. Based on this result,we solve the proposed optimal control problem. An example illustratesthe theory.