Continuous time shock markov decision processes with discounted criterion

This paper presents a new concept of Markov decision processes: continuous time shock Markov decision processes, which model Markovian controlled systems sequentially shocked by its environment. Between two adjacent shocks, the system can be modeled by continuous time Markov decision processes. But according to each shock, the system's parameters are changed and an instantaneous state transition occurs. After presenting the model, we prove that the optimality equation, which consists of countable equations, has a unique solution in some function space Ω     

A time-continuous markov chain interest model with applications to insurance

The force of interest is modelled by a homogeneous time-continuous Markov chain with finite state space. Ordinary differential equations are obtained for expected values of various functionals of this process, in particular for moments of present values of payment streams that may be deterministic or, possibly, also stochastic and driven by a time-continuous Markov chain. The homogeneity of the interest process gives rise to explicit formulae for expected values of some stationary functionals, e.g. moments of a perpetuity. Applications are made to some standard forms of insurance.     

Markov: A methodology for the solution of infinite time horizon markov decision processes

Algorithms are described for determining optimal policies for finite state, finite action, infinite discrete time horizon Markov decision processes. Both value-improvement and policy-improvement techniques are used in the algorithms. Computing procedures are also described. The algorithms are appropriate for processes that are either finite or infinite, deterministic or stochastic, discounted or undiscounted, in any meaningful combination of these features. Computing procedures are described in terms of initial data processing, bound improvements, process reduction, and testing and solution. Application of the methodology is illustrated with an example involving natural resource management. Management implications of certain hypothesized relationships between mallard survival and harvest rates are addressed by applying the optimality procedures to mallard population models.     

Ergodic behaviour of stochastic parabolic equations

The ergodic behaviour of homogeneous strong Feller irreducible Markov processes in Banach spaces is studied; in particular, existence and uniqueness of finite and -finite invariant measures are considered. The results obtained are applied to solutions of stochastic parabolic equations.     

Imprecise stochastic processes in discrete time: global models,imprecise Markov chains,and ergodic theorems

We justify and discuss expressions for joint lower and upper expectations in imprecise probability trees, in terms of the sub- and supermartingales that can be associated with such trees. These imprecise probability trees can be seen as discrete-time stochastic processes with finite state sets and transition probabilities that are imprecise, in the sense that they are only known to belong to some convex closed set of probability measures. We derive various properties for their joint lower and upper expectations, and in particular a law of iterated expectations. We then focus on the special case of imprecise Markov chains, investigate their Markov and stationarity properties, and use these, by way of an example, to derive a system of non-linear equations for lower and upper expected transition and return times. Most importantly, we prove a game-theoretic version of the strong law of large numbers for submartingale differences in imprecise probability trees, and use this to derive point-wise ergodic theorems for imprecise Markov chains.     

Almost self-optimizing strategies for the adaptive control of diffusion processes

The ergodic control of a multidimensional diffusion process described by a stochastic differential equation that has some unknown parameters appearing in the drift is investigated. The invariant measure of the diffusion process is shown to be a continuous function of the unknown parameters. For the optimal ergodic cost for the known system, an almost optimal adaptive control is constructed for the unknown system.This research was partially supported by NSF Grants ECS-87-18026, ECS-91-02714, and ECS-91-13029.     

Approximation of continuous time stochastic processes by a local linearization method

This paper investigates the rate of convergence of an alternative approximation method for stochastic differential equations. The rates of convergence of the one-step and multi-step approximation errors are proved to be and in the sense respectively, where is discrete time interval. The rate of convergence of the one-step approximation error is improved as compared with methods assuming the value of Brownian motion to be known only at discrete time. Through numerical experiments, the rate of convergence of the multi-step approximation error is seen to be much faster than in the conventional method.

     

Discrete Approximations of Controlled Stochastic Systems with Memory: A Survey

This survey article considers discrete approximations of an optimal control problem in which the controlled state equation is described by a general class of stochastic functional differential equations with a bounded memory. Specifically, three different approximation methods, namely (i) semidiscretization scheme; (ii) Markov chain approximation; and (iii) finite difference approximation, are investigated. The convergence results as well as error estimates are established for each of the approximation methods.     

Ergodic Theorems and Ergodic Decomposition for Markov Chains

This paper considers Markov chains on a locally compact separable metricspace, which have an invariant probability measure but with no otherassumption on the transition kernel. Within this context, the limit providedby several ergodic theorems is explicitly identified in terms of the limitof the expected occupation measures. We also extend Yosidasergodic decomposition for Feller-like kernels to arbitrarykernels, and present ergodic results for empirical occupation measures, aswell as for additive-noise systems.     

Linear programming in tector criterion markov and semi-Markov decision processes

Optimality problems in infinite horizon, discrete time, vector criterion Markov and semi-Markov decision processes are expressed as standard problems of multiobjective linear programming. Processes with discounting, absorbing processes and completely ergodie processes without discounting are investigated. The common properties and special structure of derived multiobjective linear programming problems are overviewed. Computational simplicities associated with these problems in comparison with general multiobjective linear programming problems are discussed. Methods for solving these problems are overviewed and simple numerical examples are given.     

On the approximation of continuous time threshold ARMA processes

Threshold autoregressive (AR) and autoregressive moving average (ARMA) processes with continuous time parameter have been discussed in several recent papers by Brockwellet al. (1991,Statist. Sinica,1, 401–410), Tong and Yeung (1991,Statist. Sinica,1, 411–430), Brockwell and Hyndman (1992,International Journal Forecasting,8, 157–173) and Brockwell (1994,J. Statist. Plann. Inference,39, 291–304). A threshold ARMA process with boundary width 2>0 is easy to define in terms of the unique strong solution of a stochastic differential equation whose coefficients are piecewise linear and Lipschitz. The positive boundary-width is a convenient mathematical device to smooth out the coefficient changes at the boundary and hence to ensure the existence and uniqueness of the strong solution of the stochastic differential equation from which the process is derived. In this paper we give a direct definition of a threshold ARMA processes with =0 in the important case when only the autoregressive coefficients change with the level of the process. (This of course includes all threshold AR processes with constant scale parameter.) The idea is to express the distributions of the process in terms of the weak solution of a certain stochastic differential equation. It is shown that the joint distributions of this solution with =0 are the weak limits as 0 of the distributions of the solution with >0. The sense in which the approximating sequence of processes used by Brockwell and Hyndman (1992,International Journal Forecasting,8, 157–173) converges to this weak solution is also investigated. Some numerical examples illustrate the value of the latter approximation in comparison with the more direct representation of the process obtained from the Cameron-Martin-Girsanov formula. It is used in particular to fit continuous-time threshold models to the sunspot and Canadian lynx series.Research partially supported by National Science Foundation Research Grants DMS 9105745 and 9243648.     

Approximation for Semilinear Stochastic Evolution Equations

We investigate the approximation by space and time discretization of quasi linear evolution equations driven by nuclear or space time white noise. An error bound for the implicit Euler, the explicit Euler, and the Crank–Nicholson scheme is given and the stability of the schemes are considered. Lastly we give some examples of different space approximation, i.e., we consider approximation by eigenfunction, finite differences and wavelets.     

A generalized Girsanov transformation of finite state stochastic processes in discrete time

Cohen and Elliott (2010) introduced the backward stochastic difference equations (BSDEs) on spaces related to discrete time, finite state processes. Motivated by obtaining the explicit solution of a linear BSDE under their framework, we develop a new type of Girsanov transformation in this paper.     

Density averaging for Markov processes and the invariant measures problem

A necessary and sufficient condition is given for the existence of a finite invariant measure equivalent to a given reference measure for a discrete time, general state Markov process. The condition is an extension of one given by D. Maharam in the deterministic case and involves an averaging method (called by Maraham ‘density averaging’) applied to the Radon-Nikodym derivatives with respect to the reference measure of the usual sequence of measures induced by the Markov process acting on the fixed reference     

Numerical solution of the Dirichlet problem for nonlinear parabolic equations by a probabilistic approach

A number of new layer methods for solving the Dirichlet problemfor semilinear parabolic equations are constructed by usingprobabilistic representations of their solutions. The methodsexploit the ideas of weak sense numerical integration of stochasticdifferential equations in a bounded domain. Despite their probabilisticnature these methods are nevertheless deterministic. Some convergencetheorems are proved. Numerical tests are presented.     

Constrained markov decision processes with compact state and action spaces: the average case

Constrained Markov decision processes with compact state and action spaces are studied under long-run average reward or cost criteria. By introducing a corresponding Lagrange function, a saddle-point theorem is given, by which the existence of a constrained optimal pair of initial state distribution and policy is shown. Also, under the hypothesis of Doeblin, the functional characterization of a constrained optimal policy is obtained     

Invariant Representation for Stochastic Differential Operator by Bsdes with Uniformly Continuous Coefficients and its Applications

In this paper, we prove that a kind of second order stochastic differential operator can be represented by the limit of solutions of BSDEs with uniformly continuous coefficients. This result is a generalization of the representation for the uniformly continuous generator. With the help of this representation, we obtain the corresponding converse comparison theorem for the BSDEs with uniformly continuous coefficients, and get some equivalent relationships between the properties of the generator g and the associated solutions of BSDEs. Moreover, we give a new proof about g-convexity.     

On a class of stochastic differential equations arising from the stochastic approximation theory

The paper dealt with generalized stochastic approximation procedures of Robbins-Monro type. We consider these procedures as strong solutions of some stochastic differential equations with respect to semimartingales and investigate their almost sure convergence and mean square convergence     

Approximating solutions of neutral stochastic evolution equations with jumps

In this paper, we establish existence and uniqueness of the mild solutions to a class of neutral stochastic evolution equations driven by Poisson random measures in some Hilbert space. Moreover, we adopt the Faedo-Galerkin scheme to approximate the solutions. This work was supported by the LPMC at Nankai University and National Natural Science Foundation of China (Grant No. 10671036)