Algorithmic aspects of mean–variance optimization in Markov decision processes

We consider finite horizon Markov decision processes under performance measures that involve both the mean and the variance of the cumulative reward. We show that either randomized or history-based policies can improve performance. We prove that the complexity of computing a policy that maximizes the mean reward under a variance constraint is NP-hard for some cases, and strongly NP-hard for others. We finally offer pseudopolynomial exact and approximation algorithms.     

A mean–variance optimization problem for discounted Markov decision processes

In this paper, we consider a mean–variance optimization problem for Markov decision processes (MDPs) over the set of (deterministic stationary) policies. Different from the usual formulation in MDPs, we aim to obtain the mean–variance optimal policy that minimizes the variance over a set of all policies with a given expected reward. For continuous-time MDPs with the discounted criterion and finite-state and action spaces, we prove that the mean–variance optimization problem can be transformed to an equivalent discounted optimization problem using the conditional expectation and Markov properties. Then, we show that a mean–variance optimal policy and the efficient frontier can be obtained by policy iteration methods with a finite number of iterations. We also address related issues such as a mutual fund theorem and illustrate our results with an example.     

Open-loop equilibrium reinsurance-investment strategy under mean–variance criterion with stochastic volatility

This paper investigates the open-loop equilibrium reinsurance-investment (RI) strategy under general stochastic volatility (SV) models. We resolve difficulties arising from the unbounded volatility process and the non-negativity constraint on the reinsurance strategy. The resolution enables us to derive the existence and uniqueness result for the time-consistent mean variance RI policy under both situations of constant and state-dependent risk aversions. We apply the general framework to popular SV models including the Heston, the 3/2 and the Hull–White models. Closed-form solutions are obtained for the aforementioned models under constant risk aversion, and the non-leveraged models under state-dependent risk aversion.     

Embedding a state space model into a Markov decision process

In agriculture Markov decision processes (MDPs) with finite state and action space are often used to model sequential decision making over time. For instance, states in the process represent possible levels of traits of the animal and transition probabilities are based on biological models estimated from data collected from the animal or herd.     

The central limit theorem for stationary Markov processes with normal generator—with applications to hypergroups

We extend the central limit theorem for additive functionals of a stationary, ergodic Markov chain with normal transition operator due to Gordin and Lif?ic, 1981 [A remark about a Markov process with normal transition operator, In: Third Vilnius Conference on Probability and Statistics 1, pp. 147–48] to continuous-time Markov processes with normal generators. As examples, we discuss random walks on compact commutative hypergroups as well as certain random walks on non-commutative, compact groups.     

Existence,uniqueness and ergodicity of Markov branching processes with immigration and instantaneous resurrection

We consider a modified Markov branching process incorporating with both state-independent immigration and instantaneous resurrection.The existence criterion of the process is firstly considered.We prove that if the sum of the resurrection rates is finite,then there does not exist any process.An existence criterion is then established when the sum of the resurrection rates is infinite.Some equivalent criteria,possessing the advantage of being easily checked,are obtained for the latter case.The uniqueness criterion for such process is also investigated.We prove that although there exist infinitely many of them,there always exists a unique honest process for a given q-matrix.This unique honest process is then constructed.The ergodicity property of this honest process is analysed in detail.We prove that this honest process is always ergodic and the explicit expression for the equilibrium distribution is established.     

Special solutions of the Chapman–Kolmogorov equation for multidimensional-state Markov processes with continuous time

The bilinear Chapman–Kolmogorov equation determines the dynamical behavior of Markov processes. The task to solve it directly (i.e., without linearizations) was posed by Bernstein in 1932 and was partially solved by Sarmanov in 1961 (solutions are represented by bilinear series). In 2007–2010, the author found several special solutions (represented both by Sarmanov-type series and by integrals) under the assumption that the state space of the Markov process is one-dimensional. In the presented paper, three special solutions have been found (in the integral form) for the multidimensional- state Markov process. Results have been illustrated using five examples, including an example that shows that the original equation has solutions without a probabilistic interpretation.     

The Perron–Frobenius theorem – a proof with the use of Markov chains

The Perron–Frobenius theorem for an irreducible nonnegative matrix is proved using the matrix graph and the ergodic theorem of the theory of Markov chains. Bibliography: 7 titles. Translated from Zapiski Nauchnykh Seminarov POMI, Vol. 359, 2008, pp. 5–16.     

Spectral methods for bivariate Markov processes with diffusion and discrete components and a variant of the Wright–Fisher model

The aim of this paper is to study differential and spectral properties of the infinitesimal operator of two dimensional Markov processes with diffusion and discrete components. The infinitesimal operator is now a second-order differential operator with matrix-valued coefficients, from which we can derive backward and forward equations, a spectral representation of the probability density, study recurrence of the process and the corresponding invariant distribution. All these results are applied to an example coming from group representation theory which can be viewed as a variant of the Wright–Fisher model involving only mutation effects.     

The L��vy�CKhintchine type operators with variable Lipschitz continuous coefficients generate linear or nonlinear Markov processes and semigroups

Ito??s construction of Markovian solutions to stochastic equations driven by a Lévy noise is extended to nonlinear distribution dependent integrands aiming at the effective construction of linear and nonlinear Markov semigroups and the corresponding processes with a given pseudo-differential generator. It is shown that a conditionally positive integro-differential operator (of the Lévy?CKhintchine type) with variable coefficients (diffusion, drift and Lévy measure) depending Lipschitz continuously on its parameters (position and/or its distribution) generates a linear or nonlinear Markov semigroup, where the measures are metricized by the Wasserstein?CKantorovich metrics. This is a non-trivial but natural extension to general Markov processes of a long known fact for ordinary diffusions.