Risk-sensitive and risk-neutral control for continuous-time hidden Markov models

In this paper the optimal control of a continuous-time hidden Markov model is discussed. The risk-sensitive problem involves a cost function which has an exponential form and a risk parameter, and is solved by defining an appropriate information state and dynamic programming. As the risk parameter tends to zero, the classical risk-neutral optimal control problem is recovered. The limits are proved using viscosity solution methods.The first author wishes to acknowledge the funding of the activities of the Cooperative Research Centre for Robust and Adaptive Systems by the Australian Commonwealth Government under the Cooperative Research Centers Program. The support of NSERC Grant A7964 is acknowledged by the second author, as is the hospitality of the Department of Systems Engineering and the Cooperative Research Centre for Robust and Adaptive Systems, Australian National University, in July 1993.     

隐非齐次马尔可夫模型的强大数定律

在状态集都有限的情况下,给出了隐马尔可夫模型的一些性质定理．利用马氏链的强极限定理,得到了隐非齐次马尔可夫模型的强大数定律．     

可列非齐次隐Markov模型的强大数定律

隐马尔科夫模型被广泛的应用于弱相依随机变量的建模,是研究神经生理学、发音过程和生物遗传等问题的有力工具。研究了可列非齐次隐 Markov 模型的若干性质,得到了这类模型的强大数定律,推广了有限非齐次马氏链的一类强大数定律。     

Stochastic adaptive control for continuous-time linear systems with quadratic cost

An adaptive control problem is formulated and solved for a completely observed, continuous-time, linear stochastic system with an ergodic quadratic cost criterion. The linear transformationsA of the state,B of the control, andC of the noise are assumed to be unknown. Assuming only thatA is stable and that the pair (A, C) is controllable and using a diminishing excitation control that is asymptotically negligible for an ergodic, quadratic cost criterion it is shown that a family of least-squares estimates is strongly consistent. Furthermore, an adaptive control is given using switchings that is self-optimizing for an ergodic, quadratic cost criterion.This research was partially supported b y NSF Grants ECS-9102714, ECS-9113029, and DMS-9305936.     

Interactive hidden Markov models and their applications

^** Email: wching{at}hkusua.hku.hk In this paper, we propose an Interactive hidden Markov model(IHMM). In a traditional HMM, the observable states are affecteddirectly by the hidden states, but not vice versa. In the proposedIHMM, the transitions of hidden states depend on the observablestates. We also develop an efficient estimation method for themodel parameters. Numerical examples on the sales demand dataand economic data are given to demonstrate the applicabilityof the model.     

Maximum likelihood estimator for hidden Markov models in continuous time

The paper studies large sample asymptotic properties of the Maximum Likelihood Estimator (MLE) for the parameter of a continuous time Markov chain, observed in white noise. Using the method of weak convergence of likelihoods due to Ibragimov and Khasminskii (Statistical estimation, vol 16 of Applications of mathematics. Springer-Verlag, New York), consistency, asymptotic normality and convergence of moments are established for MLE under certain strong ergodicity assumptions on the chain. This article has been written during the author’s visit at Laboratoire de Statistique et Processus, Universite du Maine, France, supported by the Chateaubriand fellowship.     

On approximation of smoothing probabilities for hidden Markov models

We consider the smoothing probabilities of hidden Markov model (HMM). We show that under fairly general conditions for HMM, the exponential forgetting still holds, and the smoothing probabilities can be well approximated with the ones of double-sided HMM. This makes it possible to use ergodic theorems. As an application we consider the pointwise maximum a posteriori segmentation, and show that the corresponding risks converge.     

A survey of Markov decision models for control of networks of queues

We review models for the optimal control of networks of queues. Our main emphasis is on models based on Markov decision theory and the characterization of the structure of optimal control policies.This research was partially supported by the National Science Foundation under Grant No. DDM-8719825. The Government has certain rights in this material. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. The research was also partially supported by the C.I.E.S. (France), while the author was on leave at INRIA, Sophia-Antipolis, 1991–92.     

Bayesian Monte Carlo estimation for profile hidden Markov models

Hidden Markov models are used as tools for pattern recognition in a number of areas, ranging from speech processing to biological sequence analysis. Profile hidden Markov models represent a class of so-called “left–right” models that have an architecture that is specifically relevant to classification of proteins into structural families based on their amino acid sequences. Standard learning methods for such models employ a variety of heuristics applied to the expectation-maximization implementation of the maximum likelihood estimation procedure in order to find the global maximum of the likelihood function. Here, we compare maximum likelihood estimation to fully Bayesian estimation of parameters for profile hidden Markov models with a small number of parameters. We find that, relative to maximum likelihood methods, Bayesian methods assign higher scores to data sequences that are distantly related to the pattern consensus, show better performance in classifying these sequences correctly, and continue to perform robustly with regard to misspecification of the number of model parameters. Though our study is limited in scope, we expect our results to remain relevant for models with a large number of parameters and other types of left–right hidden Markov models.     

Leroux's method for general hidden Markov models

The method introduced by Leroux [Maximum likelihood estimation for hidden Markov models, Stochastic Process Appl. 40 (1992) 127–143] to study the exact likelihood of hidden Markov models is extended to the case where the state variable evolves in an open interval of the real line. Under rather minimal assumptions, we obtain the convergence of the normalized log-likelihood function to a limit that we identify at the true value of the parameter. The method is illustrated in full details on the Kalman filter model.     

On the Milito-Cruz adaptive control scheme for Markov chains

Milito and Cruz have introduced a novel adaptive control scheme for finite Markov chains when a finite parametrized family of possible transition matrices is available. The scheme involves the minimization of a composite functional of the observed history of the process incorporating both control and estimation aspects. We prove the a.s. optimality of a similar scheme when the state space is countable and the parameter space a compact subset ofR ^d .     

A correction on approximation of smoothing probabilities for hidden Markov models

In this note, we correct a mistake concerning Theorem 2.1 in Lember (2011a).     

二阶离散隐马尔科夫模型的严格定义及等价性质

隐马氏模型作为一种具有双重随机过程的统计模型,具有可靠的概率统计理论基础和强有力的数学结构,已被广泛应用于语音识别、生物序列分析、金融数据分析等领域.由于传统的一阶隐马氏模型无法表示更远状态距离间的依赖关系,就可能会忽略很多有用的统计特征,故有人提出二阶隐马氏模型的概念,但此概念并不严格.本文给出二阶离散隐马尔科夫模型的严格定义,并研究了二阶离散隐马尔科夫模型的两个等价性质.     

Ergodic control of reflected diffusions with jumps

We discuss ergodicity properties of a controlled jumps diffusion process reflected from the boundary of a bounded domain. The control parameters act on the drift term and on a first-order-type jump density. The controlled process is generated via a Girsanov change of probability, and a long-run average criterion is optimized. An optimal stationary feedback is constructed by means of the Hamilton-Jacobi-Bellman equation.     

Computational issues in parameter estimation for stationary hidden Markov models

The parameters of a hidden Markov model (HMM) can be estimated by numerical maximization of the log-likelihood function or, more popularly, using the expectation–maximization (EM) algorithm. In its standard implementation the latter is unsuitable for fitting stationary hidden Markov models (HMMs). We show how it can be modified to achieve this. We propose a hybrid algorithm that is designed to combine the advantageous features of the two algorithms and compare the performance of the three algorithms using simulated data from a designed experiment, and a real data set. The properties investigated are speed of convergence, stability, dependence on initial values, different parameterizations. We also describe the results of an experiment to assess the true coverage probability of bootstrap-based confidence intervals for the parameters.     

Asymptotic behavior of Bayes estimators for hidden Markov models with application to ion channels

In this paper we study the asymptotic behavior of Bayes estimators for hidden Markov models as the number of observations goes to infinity. The theorem that we prove is similar to the Bernstein—von Mises theorem on the asymptotic behavior of the posterior distribution for the case of independent observations. We show that our theorem is applicable to a wide class of hidden Markov models. We also discuss the implication of the theorem’s assumptions for several models that are used in practical applications such as ion channel kinetics.      

Adaptive boundary control of stochastic linear distributed parameter systems described by analytic semigroups

A stochastic adaptive control problem is formulated and solved for some unknown linear, stochastic distributed parameter systems that are described by analytic semigroups. The control occurs on the boundary. The highest-order operator is assumed to be known but the lower-order operators contain unknown parameters. Furthermore, the linear operators of the state and the control on the boundary contain unknown parameters. The noise in the system is a cylindrical white Gaussian noise. The performance measure is an ergodic, quadratic cost functional. For the identification of the unknown parameters a diminishing excitation is used that has no effect on the ergodic cost functional but ensures sufficient excitation for strong consistency. The adaptive control is the certainty equivalence control for the ergodic, quadratic cost functional with switchings to the zero control.This research was partially supported by NSF Grants ECS-9102714, ECS-9113029, and DMS-9305936.     

On the ergodic and the adaptive control of stochastic differential delay systems

Some problems of ergodic control and adaptive control are formulated and solved for stochastic differential delay systems. The existence and the uniqueness of invariant measures that are solutions of the stochastic functional differential equations for these systems are verified. For an ergodic cost criterion, almost optimal controls are constructed. For an unknown system, the invariant measures and the optimal ergodic costs are shown to be continuous functions of the unknown parameters. Almost self-optimizing adaptive controls are feasibly constructed by an approximate certainty equivalence principle.This research was partially supported by NSF Grants ECS-91-02714 and ECS91-13029.