State estimation for partially observed Markov chains

Recursive equations are derived for the conditional distribution of the state of a Markov chain, given observations of a function of the state. Mainly continuous time chains are considered. The equations for the conditional distribution are given in matrix form and in differential equation form. The conditional distribution itself forms a Markov process. Special cases considered are doubly stochastic Poisson processes with a Markovian intensity, Markov chains with a random time, and Markovian approximations of semi-Markov processes. Further the results are used to compute the Radon-Nikodym derivative for two probability measures for a Markov chain, when a function of the state is observed.     

Hierarchic Markov processes and their applications in replacement models

In this paper a new notion of a hierarchic Markov process is introduced. It is a series of Markov decision processes called subprocesses built together in one Markov decision process called the main process. The hierarchic structure is specially designed to fit replacement models which in the traditional formulation as ordinary Markov decision processes are usually very large. The basic theory of hierarchic Markov processes is described and examples are given of applications in replacement models. The theory can be extended to fit a situation where the replacement decision depends on the quality of the new asset available for replacement.     

Birthing Markov processes at random rates

Summary This paper studies processes constructed by birthing the trajectories of a given Markov process along time according to random probabilities. Getoor has considered the case where the random probabilities are determined by comultiplicative functionals and proved for right processes that the post-birth process has the Markov property. Here randomizations of comultiplicative functionals are described which give rise to conditionally Markov processes. The main argument is developed for general Markov processes and the transition probabilities of the new process, including those from the pre-birth state, are explicited.     

Strong Markov Properties for Markov Random Fields

Markov properties and strong Markov properties for random fields are defined and discussed. Special attention is given to those defined by I. V. Evstigneev. The strong Markov nature of Markov random fields with respect to random domains such as [0, L], where L is a multidimensional extension of a stopping time, is explored. A special case of this extension is shown to generalize a result of Merzbach and Nualart for point processes. As an additional example, Evstigneev's Markov and strong Markov properties are considered for independent increment jump processes.     

Construction and regularity of measure-valued markov branching processes

A construction is given for a general class of measure-valued Markov branching processes. The underlying spatial motion process is an arbitrary Borel right Markov process, and state-dependent offspring laws are allowed. It is shown that such processes are Hunt processes in the Ray weak* topology, and have continuous paths if and only if the total mass process is continuous. The entrance spaces of such processes are described explicitly. Research supported in part by NSF Grant DMS 87-21237.     

Rainflow analysis in Coastal Engineering using switching second order Markov models

The paper deals with the use of Markov and switching Markov chain models of turning points to reproduce random sets of sea states. The advantages of these models are emphasized and compared with existing models based on wave height records, indicating that long and short range and period cycles are included, while the wave height records ignore this important information from the point of view of damage accumulation. Existing models for first order Markov processes are extended to the case of second order processes and closed formulas are given to derive the rainflow matrices of these processes. Finally, one illustrative example of application is given.     

On the rate of convergence of one inhomogeneous Markov algorithm of search for extremum

The paper is concerned with theoretical study of the rate of convergence of one inhomogeneous Markov random algorithm of search for extremum. Methods of random search have value in solving involved optimization problems. However, there are only few theoretical results concerning the rate of convergence of these algorithms.     

Cycle generating equations

Equations generating directed cycles in at most countable sets are considered. Such equations are the balance relations for recurrent continuous parameter Markov processes that in term of algebraic topology generate the 1-cycles. Connections of cycle generating equations with the recurrent behavior of Markov processes are given     

Markov链在教育测量中的应用

将“问题解决”的分析、建模、求解和检验四个阶段作为随机过程的状态得到一Markov链.利用有关随机过程的知识对Markov链进行分类和求解.由此,对学生解决问题的能力进行测量,得到了一些合理结果.     

Nonlinear separation functions and constrained extremum problems

In this paper, the image space analysis is employed to investigate constrained extremum problems. A new nonlinear separation function for the constrained extremum problems is given. Some optimality conditions and a strong duality theorem for the constrained extremum problem are obtained. These results extend and improve the corresponding ones in the literature.     

Filtering with Discrete State Observations

The problem of estimating a finite state Markov chain observed via a process on the same state space is discussed. Optimal solutions are given for both the ``weak' and ``strong' formulations of the problem. The ``weak' formulation proceeds using a reference probability and a measure change for the Markov chain. The ``strong' formulation considers an observation process related to perturbations of the counting processes associated with the Markov chain. In this case the ``small noise' convergence is investigated. Accepted 7 April 1998     

Markov decision processes and strongly excessive functions

Strongly excessive functions play an important role in the theory of Markov decision processes and Markov games. In this paper the following question is investigated: What are the properties of Markov decision processes which possess a strongly excessive function? A probabilistic characterization is presented in the form of a random drift through a partitioned state space. For strongly excessive functions which have a positive lower bound a characterization is given in terms of the lifetime distribution of the process.Finally we give a characterization in terms of the spectral radius.     

Kolmogorov equations in fractional derivatives for the transition probabilities of continuous-time Markov processes

A family of one-dimensional continuous-time Markov processes is considered, for which the author has earlier determined the transition probabilities by directly solving the Kolmogorov–Chapman equation; these probabilities have the form of single integrals. Analogues of the first and second Kolmogorov equations for the family of processes under consideration are obtained by using a procedure for obtaining integro-differential equations describing Markov processes with discontinuous trajectories. These equations turn out to be equations in fractional derivatives. The results are based on an asymptotic analysis of the transition probability as the start and end times of the transition approach each other. This analysis implies that the trajectories of a given Markov process are divided into two classes, depending on the interval in which they start. Some of the trajectories decay during a short time interval with a certain probability, and others are generated with a certain probability.     

Hitting distributions domination and subordinate resolvents; an analytic approach

We give an analytic version of the well known Shih's theorem concerning the Markov processes whose hitting distributions are dominated by those of a given process. The treatment is purely analytic, completely different from Shih's arguments and improves essentially his result (in the case when the given processes are transient     

On conservativeness and recurrence criteria for Markov processes

In the present paper, we give general criteria of conservativeness and recurrence for Markov processes associated with, not necessarily symmetric, Dirichlet spaces. The conservativeness criterion is applied to discuss a comparison theorem of conservativeness for diffusion processes. Also some sufficient conditions of conservativeness and recurrence for diffusion processes.are given.     

Markov processes generated by linear stochastic evolution equations †

A class of Hilbert space-valued Markov processes which can be expressed as the mild solution of a linear abstract evolution equation is studied. Sufficient conditions for the generator of the Markov process to be well-defined are given and Kolmogorov's equation and an equation for the characteristic function of the process are derived. The theory is illustrated by examples of parabolic, hyperbolic and delay stochastic differential equations.     

Multifractional Markov Processes in Heterogeneous Domains

In this article, we introduce a class of Markov processes whose transition probability densities are defined by multifractional pseudodifferential evolution equations on compact domains with variable local dimension. The infinitesimal generators of these Markov processes are given by the trace of strongly elliptic pseudodifferential operators of variable order on such domains. The results derived provide a pseudomultifractal version of some existing special classes of multifractional Markov processes. In particular, pseudostable processes are defined on domains with variable local dimension in this framework. In the case where the local dimension of the domain and the local Hölder exponents of the transition probability densities are constant, the existing results on fractal versions of Lévy processes are recovered.     

Classifying convex extremum problems over linear topologies having separation properties

It is shown that any convex or concave extremum problem possesses a subsidiary extremum problem which has certain homogeneous properties. Analogous to the given problem, the “homogenized” extremum problem seeks the minimum of a convex function or the maximum of a concave function over a convex domain. By using homogenized extremum problems, new relationships are developed between any given convex extremum problem (P) and a concave extremum problem (P¹) (also having a convex domain), called the “dual” problem of (P). This is achieved by combining all possibilities in tabular form of (1) the values of the extremum functions and (2) the nature of the convex domains including perturbations of all problems (P), (P¹), and each of their respective homogenized extremum problems.This detailed and refined classification is contrasted to the relationships obtainable by combining only the possible values of the extremum functions of the problems (P) and (P¹) and the possible limiting values of these functions stemming from perturbations of the convex constraint domains of (P) and (P¹), respectively.The extremum problems in this paper and classification results are set forth in real topologically paired vector spaces having the Hahn-Banach separation property.     

Optimal control of stationary Markov processes

Sufficient conditions are given for the optimal control of Markov processes when the control policy is stationary and the process possesses a stationary distribution. The costs are unbounded and additive, and may or may not be discounted. Applications to Semi-Markov processes are included, and the results for random walks are related to the author's previous papers on diffusion processes.     

The L��vy�CKhintchine type operators with variable Lipschitz continuous coefficients generate linear or nonlinear Markov processes and semigroups

Ito??s construction of Markovian solutions to stochastic equations driven by a Lévy noise is extended to nonlinear distribution dependent integrands aiming at the effective construction of linear and nonlinear Markov semigroups and the corresponding processes with a given pseudo-differential generator. It is shown that a conditionally positive integro-differential operator (of the Lévy?CKhintchine type) with variable coefficients (diffusion, drift and Lévy measure) depending Lipschitz continuously on its parameters (position and/or its distribution) generates a linear or nonlinear Markov semigroup, where the measures are metricized by the Wasserstein?CKantorovich metrics. This is a non-trivial but natural extension to general Markov processes of a long known fact for ordinary diffusions.