Optimal stopping with continuous control of piecewise deterministic Markov processes

In this paper we consider the problem of optimal stopping and continuous control on some local parameters of a piecewise-deterministic Markov processes (PDP's). Optimality equations are obtained in terms of a set of variational inequalities as well as on the first jump time operator of the PDP. It is shown that if the final cost function is absolutely continuous along trajectories then so is the value function of the optimal stopping problem with continuous control. These results unify and generalize previous ones in the current literature.     

Switching control of piecewise-deterministic processes

A finite collection of piecewise-deterministic processes are controlled in order to minimize the expected value of a performance functional with continuous operating cost and discrete switching control costs. The solution of the associated dynamic programming equation is obtained by an iterative approximation using optimal stopping time problems.This research was supported in part by NSF Grant No. DMS-8508651 and by University of Tennessee Science Alliance Research Incentive Award.     

Impulse and continuous control of piecewise deterministic Markov processes

In this paper we consider the problem of impulse and continuous control on the jump rate and post jump location parameters of piecewise-deterministic Markov processes (PDP's). In a companion paper we studied the optimal stopping with continuous control problem of PDP's assuming only absolutely continuity along trajectories hypothesis on the final cost function. In this paper we apply these results to obtain optimality equations for the impulse and continuous control problem of PDP's in terms of a set of quasi-variational inequalities as well as on the first jump time operator of the process. No continuity or differential assumptions on the whole state space, neither stability assumptions on the parameters of the problem are required. It is shown that if the post intervention operator satisfies some locally lipschitz continuity along trajectories properties then so will the value function of the impulse and continuous control problem.     

Optimal stopping for partially observed piecewise-deterministic Markov processes

This paper deals with the optimal stopping problem under partial observation for piecewise-deterministic Markov processes. We first obtain a recursive formulation of the optimal filter process and derive the dynamic programming equation of the partially observed optimal stopping problem. Then, we propose a numerical method, based on the quantization of the discrete-time filter process and the inter-jump times, to approximate the value function and to compute an ?$?$-optimal stopping time. We prove the convergence of the algorithms and bound the rates of convergence.     

Piecewise Deterministic Markov Control Processes with Feedback Controls and Unbounded Costs

The control of piecewise-deterministic processes is studied where only local boundedness of the data is assumed. Moreover the discount rate may be zero. The value function is shown to be solution to the Bellman equation in a weak sense; however the solution concept is strong enough to generate optimal policies. Continuity and compactness conditions are given for the existence of nonrelaxed optimal feedback controls.     

On first-order quasi-variational inequalities with integral terms

This paper deals with first-order quasi-variational inequalities with integral terms associated with impulsive and switching control of piecewise-deterministic processes. Two formulations of quasi-variational inequalities are studied, characteristic and viscosity, and the relations between them are discussed. As a tool we apply convex analysis methods.     

Lower semicontinuity property of multiparameter optimal stopping value and its application to multiparameter prophet inequalities

This paper is concerned with the optimal stopping problem for discrete time multiparameter stochastic processes with the index set Nd. The optimal stopping value of a discrete time multiparameter integrable stochastic process whose negative part is uniformly integrable, is lower semicontinuous for the topology of convergence in distribution. The multiparameter version of prophet inequality for the one-parameter optimal stopping problem is formulated and the lower semicontinuity property of the optimal stopping value is applied to the multiparameter prophet inequality.     

Consumption-Investment Problem with Subsistence Consumption,Bankruptcy, and Random Market Coefficients

We consider a general continuous-time finite-horizon single-agent consumption and portfolio decision problem with subsistence consumption and value of bankruptcy. Our analysis allows for random market coefficients and general continuously differentiable concave utility functions. We study the time of bankruptcy as a problem of optimal stopping, and succeed in obtaining explicit formulas for the optimal consumption and wealth processes in terms of the optimal bankruptcy time. This paper extends the results of Karatzas, Lehoczky, and Shreve (Ref. 1) on the maximization of expected utility from consumption in a financial market with random coefficients by incorporating subsistence consumption and bankruptcy. It also addresses the random coefficients and finite-horizon version of the problem treated by Sethi, Taksar, and Presman (Ref. 2). The mathematical tools used in our analysis are optimal stopping, stochastic control, martingale theory, and Girsanov change of measure.     

On ergodic control problems for singularly perturbed Markov processes

In this paper ergodic control problems (optimal stopping, impulsive control, and stochastic control for singularly perturbed Feller Markov processes) are studied. As the main result the so-called limit control principle is shown to hold in each case. The results obtained depend on the averaging properties of the perturbed system which follow from the fact that the perturbing process does not depend on either the perturbed process or on the control.     

线性脉冲控制系统的能控性与最优控制

首先讨论了一类线性随机脉冲控制系统的精确能控性质,给出了该类控制系统的脉冲精确能控的等价的代数判据.然后提出了一个确定性的二维线性脉冲控制系统的时间-脉冲强度最优控制问题;利用动态规划原理,给出了脉冲最优控制的反馈形式和值函数的显式表达式;说明了值函数在整个平面上是连续的,在左右两个半平面的内部还是连续可微的.     

Optimal stopping time on discounted semi-Markov processes

This paper attempts to study the optimal stopping time for semi- Markov processes (SMPs) under the discount optimization criteria with unbounded cost rates. In our work, we introduce an explicit construction of the equivalent semi-Markov decision processes (SMDPs). The equivalence is embodied in the expected discounted cost functions of SMPs and SMDPs, that is, every stopping time of SMPs can induce a policy of SMDPs such that the value functions are equal, and vice versa. The existence of the optimal stopping time of SMPs is proved by this equivalence relation. Next, we give the optimality equation of the value function and develop an effective iterative algorithm for computing it. Moreover, we show that the optimal and ε-optimal stopping time can be characterized by the hitting time of the special sets. Finally, to illustrate the validity of our results, an example of a maintenance system is presented in the end.     

Continuity Properties of Optimal Multiple Stopping Value

This article is concerned with the optimal multiple stopping problem for discrete time finite stage stochastic processes. We study lower semicontinuity and continuity properties of optimal stopping values with respect to the topology of convergence in distribution. Also, we formulate the multiple stopping version of the prophet inequality for the optimal stopping problem and apply the lower semicontinuity property of optimal stopping values to the prophet inequality for the optimal multiple stopping problem.     

Additive comparisons of stopping values and supremum values for finite stage multiparameter stochastic processes

This paper concerns the optimal stopping problem for discrete time multiparameter stochastic processes with the index set Nd. In the classical optimal stopping problems, the comparisons between the expected reward of a player with complete foresight and the expected reward of a player using nonanticipating stop rules, known as prophet inequalities, have been studied by many authors. Ratio comparisons between these values in the case of multiparameter optimal stopping problems are studied by Krengel and Sucheston (1981) [9] and Tanaka (2007, 2006) [14] and [15]. In this paper an additive comparison in the case of finite stage multiparameter optimal stopping problems is given.     

Uniqueness of unbounded viscosity solutions for impulse control problem

We study here the impulse control problem in infinite as well as finite horizon. We allow the cost functionals and dynamics to be unbounded and hence the value function can possibly be unbounded. We prove that the value function is the unique viscosity solution in a suitable subclass of continuous functions, of the associated quasivariational inequality. Our uniqueness proof for the infinite horizon problem uses stopping time problem and for the finite horizon problem, comparison method. However, we assume proper growth conditions on the cost functionals and the dynamics.     

Optimal control of Markovian jump processes with partial information and applications to a parallel queueing model

We consider a stochastic control problem over an infinite horizon where the state process is influenced by an unobservable environment process. In particular, the Hidden-Markov-model and the Bayesian model are included. This model under partial information is transformed into an equivalent one with complete information by using the well-known filter technique. In particular, the optimal controls and the value functions of the original and the transformed problem are the same. An explicit representation of the filter process which is a piecewise-deterministic process, is also given. Then we propose two solution techniques for the transformed model. First, a generalized verification technique (with a generalized Hamilton–Jacobi–Bellman equation) is formulated where the strict differentiability of the value function is weaken to local Lipschitz continuity. Second, we present a discrete-time Markovian decision model by which we are able to compute an optimal control of our given problem. In this context we are also able to state a general existence result for optimal controls. The power of both solution techniques is finally demonstrated for a parallel queueing model with unknown service rates. In particular, the filter process is discussed in detail, the value function is explicitly computed and the optimal control is completely characterized in the symmetric case.     

Mayer and optimal stopping stochastic control problems with discontinuous cost

We study two classes of stochastic control problems with semicontinuous cost: the Mayer problem and optimal stopping for controlled diffusions. The value functions are introduced via linear optimization problems on appropriate sets of probability measures. These sets of constraints are described deterministically with respect to the coefficient functions. Both the lower and upper semicontinuous cases are considered. The value function is shown to be a generalized viscosity solution of the associated HJB system, respectively, of some variational inequality. Dual formulations are given, as well as the relations between the primal and dual value functions. Under classical convexity assumptions, we prove the equivalence between the linearized Mayer problem and the standard weak control formulation. Counter-examples are given for the general framework.     

Impulsive control in management: Prospects and applications

An outline of impulsive control and its applications in management is reviewed. Conditions for optimal impulsive controls when the dynamic process is given by a stochastic differential equation with Wiener and jump processes are given. Applications to inventory control, capacity expansion, vehicle dispatching, maintenance-replacement-inspection, and pricing problems are formulated, and optimality conditions found. Although there are many other applications that can be outlined, these provide some motivation for further study in this emerging field.This work was supported by the European Institute for Advanced Studies in Management, Brussels, Belgium.     

Optimal Synthesis for the Minimum Time Control Problems of Fed-Batch Bioprocesses for Growth Functions with Two Maxima

We address the problem of finding an optimal feedback control for feeding a fed-batch bioreactor with one species and one substrate from a given initial condition to a given target value in a minimal amount of time. Recently, the optimal synthesis (optimal feeding strategy) has been obtained in systems in which the microorganisms involved are represented by increasing growth functions or growth functions with one maxima, with either Monod or Haldane functions, respectively (widely used in bioprocesses modeling). In the present work, we allow impulsive controls corresponding to instantaneous dilutions, and we assume that the growth function of the microorganism present in the process has exactly two local maxima. This problem has been tackled from a numerical point of view without impulsive controls. In this article, we introduce two singular arc feeding strategies, and we define explicit regions of initial conditions in which the optimal strategy is either the first singular arc strategy or the second strategy.     

有限时区上的最优转换和停止问题

讨论了有限时区上的最优转换和停止问题,它是一类同时具备脉冲控制和最优停止特征的最优控制问题.问题的最优值以及最优转换和停止决策可以由具有混合障碍的多维反射倒向随机微分方程的解来刻画.接着考虑了形式更一般的反射倒向随机微分方程并证明了方程解的存在唯一性.