Optimal stopping in infinite horizon: An eigenfunction expansion approach

We develop an eigenfunction expansion based value iteration algorithm to solve discrete time infinite horizon optimal stopping problems for a rich class of Markov processes that are important in applications. We provide convergence analysis for the value function and the exercise boundary, and derive easily computable error bounds for value iterations. As an application we develop a fast and accurate algorithm for pricing callable perpetual bonds under the CIR short rate model.     

Optimal stopping of Markov switching Lévy processes

We consider a finite time horizon optimal stopping of a regime-switching Lévy process. We prove that the value function of the optimal stopping problem can be characterized as the unique viscosity solution of the associated Hamilton–Jacobi–Bellman variational inequalities.     

Risk-sensitive control of continuous time Markov chains

We study risk-sensitive control of continuous time Markov chains taking values in discrete state space. We study both finite and infinite horizon problems. In the finite horizon problem we characterize the value function via Hamilton Jacobi Bellman equation and obtain an optimal Markov control. We do the same for infinite horizon discounted cost case. In the infinite horizon average cost case we establish the existence of an optimal stationary control under certain Lyapunov condition. We also develop a policy iteration algorithm for finding an optimal control.     

Optimal stopping with irregular reward functions

We consider optimal stopping problems with finite horizon for one-dimensional diffusions. We assume that the reward function is bounded and Borel-measurable, and we prove that the value function is continuous and can be characterized as the unique solution of a variational inequality in the sense of distributions.     

Conditions for the discovery of solution horizons

We present necessary and sufficient conditions for discrete infinite horizon optimization problems with unique solutions to be solvable. These problems can be equivalently viewed as the task of finding a shortest path in an infinite directed network. We provide general forward algorithms with stopping rules for their solution. The key condition required is that of weak reachability, which roughly requires that for any sequence of nodes or states, it must be possible from optimal states to reach states close in cost to states along this sequence. Moreover the costs to reach these states must converge to zero. Applications are considered in optimal search, undiscounted Markov decision processes, and deterministic infinite horizon optimization.This work was supported in part by NSF Grant ECS-8700836 to The University of Michigan.     

A stochastic programming approach for planning horizons of infinite horizon capacity planning problems

Planning horizon is a key issue in production planning. Different from previous approaches based on Markov Decision Processes, we study the planning horizon of capacity planning problems within the framework of stochastic programming. We first consider an infinite horizon stochastic capacity planning model involving a single resource, linear cost structure, and discrete distributions for general stochastic cost and demand data (non-Markovian and non-stationary). We give sufficient conditions for the existence of an optimal solution. Furthermore, we study the monotonicity property of the finite horizon approximation of the original problem. We show that, the optimal objective value and solution of the finite horizon approximation problem will converge to the optimal objective value and solution of the infinite horizon problem, when the time horizon goes to infinity. These convergence results, together with the integrality of decision variables, imply the existence of a planning horizon. We also develop a useful formula to calculate an upper bound on the planning horizon. Then by decomposition, we show the existence of a planning horizon for a class of very general stochastic capacity planning problems, which have complicated decision structure.     

Optimal policy for minimizing risk models in Markov decision processes

We consider the minimizing risk problems in discounted Markov decisions processes with countable state space and bounded general rewards. We characterize optimal values for finite and infinite horizon cases and give two sufficient conditions for the existence of an optimal policy in an infinite horizon case. These conditions are closely connected with Lemma 3 in White (1993), which is not correct as Wu and Lin (1999) point out. We obtain a condition for the lemma to be true, under which we show that there is an optimal policy. Under another condition we show that an optimal value is a unique solution to some optimality equation and there is an optimal policy on a transient set.     

Optimal Exercise Strategies for Operational Risk Insurance via Multiple Stopping Times

In this paper we demonstrate how to develop analytic closed form solutions to optimal multiple stopping time problems arising in the setting in which the value function acts on a compound process that is modified by the actions taken at the stopping times. This class of problem is particularly relevant in insurance and risk management settings and we demonstrate this on an important application domain based on insurance strategies in Operational Risk management for financial institutions. In this area of risk management the most prevalent class of loss process models is the Loss Distribution Approach (LDA) framework which involves modelling annual losses via a compound process. Given an LDA model framework, we consider Operational Risk insurance products that mitigate the risk for such loss processes and may reduce capital requirements. In particular, we consider insurance products that grant the policy holder the right to insure k of its annual Operational losses in a horizon of T years. We consider two insurance product structures and two general model settings, the first are families of relevant LDA loss models that we can obtain closed form optimal stopping rules for under each generic insurance mitigation structure and then secondly classes of LDA models for which we can develop closed form approximations of the optimal stopping rules. In particular, for losses following a compound Poisson process with jump size given by an Inverse-Gaussian distribution and two generic types of insurance mitigation, we are able to derive analytic expressions for the loss process modified by the insurance application, as well as closed form solutions for the optimal multiple stopping rules in discrete time (annually). When the combination of insurance mitigation and jump size distribution does not lead to tractable stopping rules we develop a principled class of closed form approximations to the optimal decision rule. These approximations are developed based on a class of orthogonal Askey polynomial series basis expansion representations of the annual loss compound process distribution and functions of this annual loss.     

A general method for finding the optimal threshold in discrete time

We develop an approach for solving one-sided optimal stopping problems in discrete time for general underlying Markov processes on the real line. The main idea is to transform the problem into an auxiliary problem for the ladder height variables. In case that the original problem has a one-sided solution and the auxiliary problem has a monotone structure, the corresponding myopic stopping time is optimal for the original problem as well. This elementary line of argument directly leads to a characterization of the optimal boundary in the original problem. The optimal threshold is given by the threshold of the myopic stopping time in the auxiliary problem. Supplying also a sufficient condition for our approach to work, we obtain solutions for many prominent examples in the literature, among others the problems of Novikov-Shiryaev, Shepp-Shiryaev, and the American put in option pricing under general conditions. As a further application we show that for underlying random walks (and Lévy processes in continuous time), general monotone and log-concave reward functions g lead to one-sided stopping problems.     

Existence and explicit determination of optimal stopping times

We consider large classes of continuous time optimal stopping problems for which we establish the existence and form of the optimal stopping times. These optimal times are then used to find approximate optimal solutions for a class of discrete time problems.     

Optimal Stopping Time on Semi-Markov Processes with Finite Horizon

Journal of Optimization Theory and Applications - In this paper, we consider the optimal stopping problems on semi-Markov processes (sMPs) with finite horizon and aim to establish the existence and...     

Limit of the solutions for the finite horizon problems as the optimal solution to the infinite horizon optimization problems

In this paper, we consider how to construct the optimal solutions for the undiscounted discrete time infinite horizon optimization problems. We present the conditions under which the limit of the solutions for the finite horizon problems is optimal among all attainable paths for the infinite horizon problem under two modified overtaking criteria, as well as the conditions under which it is the unique optimum under the sum-of-utilities criterion. The results are applied to a parametric example of a simple one-sector growth model to examine the impacts of discounting on the optimal path.     

Regularity of the value function and viscosity solutions in optimal stopping problems for general Markov processes

We consider optimal stopping problems for Markov processes with a semicontinuous reward function g , and we show that under suitable conditions the value function w = w [ g ] is itself semicontinuous and is a viscosity solution of the associated variational inequality.     

An application of Lemke's method to a class of Markov decision problems

This paper presents an application of Lemke's method to a class of Markov decision problems, appearing in the optimal stopping problems, and other well-known optimization problems. We consider a special case of the Markov decision problems with finitely many states, where the agent can choose one of the alternatives; getting a fixed reward immediately or paying the penalty for one term. We show that the problem can be reduced to a linear complementarity problem that can be solved by Lemke's method with the number of iterations less than the number of states. The reduced linear complementarity problem does not necessarily satisfy the copositive-plus condition. Nevertheless we show that the Lemke's method succeeds in solving the problem by proving that the problem satisfies a necessary and sufficient condition for the extended Lemke's method to compute a solution in the piecewise linear complementarity problem.     

Infinite horizon stopping problems with (nearly) total reward criteria

We study an infinite horizon optimal stopping Markov problem which is either undiscounted (total reward) or with a general Markovian discount rate. Using ergodic properties of the underlying Markov process, we establish the feasibility of the stopping problem and prove the existence of optimal and ε$\epsilon$-optimal stopping times. We show the continuity of the value function and its variational characterisation (in the viscosity sense) under different sets of assumptions satisfied by large classes of diffusion and jump–diffusion processes. In the case of a general discounted problem we relax a classical assumption that the discount rate is uniformly separated from zero.     

Optimal models with maximizing probability of first achieving target value in the preceding stages

Decision makers often face the need of performance guarantee with some sufficiently high probability. Such problems can be modelled using a discrete time Markov decision process (MDP) with a probability criterion for the first achieving target value. The objective is to find a policy that maximizes the probability of the total discounted reward exceeding a target value in the preceding stages. We show that our formulation cannot be described by former models with standard criteria. We provide the properties of the objective functions, optimal value functions and optimal policies. An algorithm for computing the optimal policies for the finite horizon case is given. In this stochastic stopping model, we prove that there exists an optimal deterministic and stationary policy and the optimality equation has a unique solution. Using perturbation analysis, we approximate general models and prove the existence of e-optimal policy for finite state space. We give an example for the reliability of the satellite sy     

Optimal stopping with continuous control of piecewise deterministic Markov processes

In this paper we consider the problem of optimal stopping and continuous control on some local parameters of a piecewise-deterministic Markov processes (PDP's). Optimality equations are obtained in terms of a set of variational inequalities as well as on the first jump time operator of the PDP. It is shown that if the final cost function is absolutely continuous along trajectories then so is the value function of the optimal stopping problem with continuous control. These results unify and generalize previous ones in the current literature.     

Optimal stopping for non-linear expectations—Part I

We develop a theory for solving continuous time optimal stopping problems for non-linear expectations. Our motivation is to consider problems in which the stopper uses risk measures to evaluate future rewards. Our development is presented in two parts. In the first part, we will develop the stochastic analysis tools that will be essential in solving the optimal stopping problems, which will be presented in Bayraktar and Yao (2011) [1].     

Optimal Synthesis of the Zermelo–Markov–Dubins Problem in a Constant Drift Field

We consider the optimal synthesis of the Zermelo–Markov–Dubins problem, that is, the problem of steering a vehicle with the kinematics of the Isaacs–Dubins car in minimum time in the presence of a drift field. By using standard optimal control tools, we characterize the family of control sequences that are sufficient for complete controllability and necessary for optimality for the special case of a constant field. Furthermore, we present a semianalytic scheme for the characterization of an optimal synthesis of the minimum-time problem. Finally, we establish a direct correspondence between the optimal syntheses of the Markov–Dubins and the Zermelo–Markov–Dubins problems by means of a discontinuous mapping.