Markovian decision approach in housing policy

This article includes an empirical study of the housing market using the statistical method of Markov Process. The first phase of the study is devoted to measuring the filtering process in a selected neighborhood by estimating probabilities of transition from one income group to another, over the period 1949–1969 using four-year intervals. The estimated transition probabilities are then used to forecast occupancy structure for different periods and the suitability of applying the Markov Process for long term policy analysis in housing is examined. The final phase of the study includes an examination of steady state occupancy structure by various income categories of household. The study indicates a fruitful application of the Markov Process in long term housing policy analysis.     

Optimal cost and policy for a Markovian replacement problem

We consider the computation of the optimal cost and policy associated with a two-dimensional Markov replacement problem with partial observations, for two special cases of observation quality. Relying on structural results available for the optimal policy associated with these two particular models, we show that, in both cases, the infinitehorizon, optimal discounted cost function is piecewise linear, and provide formulas for computing the cost and the policy. Several examples illustrate the usefulness of the results.This research was supported by the Air Force Office of Scientific Research Grant AFOSR-86-0029, by the National Science Foundation Grant ECS-86-17860, by the Advanced Technology Program of the State of Texas, and by the Air Force Office of Scientific Research (AFSC) Contract F49620-89-C-0044.     

Optimal exit and valuation under demand uncertainty: A real options approach

This paper considers both the optimal exit strategy and the valuation of stochastic cash flows of a firm facing demand uncertainty and potential excess supply. By relying on the standard theory of linear diffusions and ordinary nonlinear programming, we derive the value of the rationally managed firm, and state the necessary condition for optimal exit. In contrast to the standard approaches in the real options literature, our analysis is completely independent of both dynamic programming and the smooth-fit principle. I demonstrate that irreversible exit is optimal only when the value of the future productive opportunities becomes smaller than the value of irreversibly exercising the option to exit and in this way avoid further cumulative losses. I also present the comparative static properties of the optimal exit threshold and demonstrate that increased uncertainty may increase or decrease the optimal exit threshold depending on the sign of the net convenience yield.     

Existence of optimal policy for time non-homogeneous discounted Markovian decision programming

In this paper we discuss the discrete time non-homogeneous discounted Markovian decision programming, where the state space and all action sets are countable. Suppose that the optimum value function is finite. We give the necessary and sufficient conditions for the existence of an optimal policy. Suppose that the absolute mean of rewards is relatively bounded. We also give the necessary and sufficient conditions for the existence of an optimal policy.     

The variational calculus and approximation in policy space for Markovian decision processes

The functional equations of Markovian decision processes yield the state values (and gain rate in the undiscounted case). Variational expressions are exhibited here for these state values (and gain rate); these expressions are stationary when evaluated at the correct values. When guesses for the values (and gain rate) are inserted into these variational expressions, a superior guess is usually obtained. Repetition of this procedure is shown to be equivalent to the method of successive approximations in policy space. Two other unusual features of this procedure are these: when the linear equations determining the Lagrange multipliers are non-singular, the variational expressions for the state variables are precisely one Newton-Raphson iteration; when applied to a linear objective function and piecewise-linear constraints, which arises for the functional equations of Markovian decision processes, the variational test quantity is piecewise constant, i.e., its first variation and higher variations all vanish. The latter explains its good performance (one-step convergence) if good estimates are available.     

Optimal systems for equipment maintenance and replacement under Markovian deterioration

Deterioration of equipment is modeled as a multistate discrete time controlled Markov process. The states are classified according to the degree of deterioration. The problem of design of optimal systems for equipment maintenance and replacement is considered when the decision-maker may take, in each stage, one of many available maintenance actions, classified according to their “stochastic effectiveness”; no action and replacement are included as alternatives. It is assumed that the transition probabilities satisfy two conditions which effectively describe a trend for monotonically increasing expected deterioration and rate of deterioration. Under these assumptions it is proved in the paper that the optimal (cost minimizing) decision system in an infinite horizon is of the control limit rule type, rapidly obtained by policy improvement algorithms. A numerical example is presented for a specific practical application; detailed data are available from the authors on request.     

Optimal decision under ambiguity for diffusion processes

In this paper we consider stochastic optimization problems for an ambiguity averse decision maker who is uncertain about the parameters of the underlying process. In a first part we consider problems of optimal stopping under drift ambiguity for one-dimensional diffusion processes. Analogously to the case of ordinary optimal stopping problems for one-dimensional Brownian motions we reduce the problem to the geometric problem of finding the smallest majorant of the reward function in a two-parameter function space. In a second part we solve optimal stopping problems when the underlying process may crash down. These problems are reduced to one optimal stopping problem and one Dynkin game. Examples are discussed.     

Optimal policy for minimizing risk models in Markov decision processes

We consider the minimizing risk problems in discounted Markov decisions processes with countable state space and bounded general rewards. We characterize optimal values for finite and infinite horizon cases and give two sufficient conditions for the existence of an optimal policy in an infinite horizon case. These conditions are closely connected with Lemma 3 in White (1993), which is not correct as Wu and Lin (1999) point out. We obtain a condition for the lemma to be true, under which we show that there is an optimal policy. Under another condition we show that an optimal value is a unique solution to some optimality equation and there is an optimal policy on a transient set.     

Optimal harvesting policy for general stochastic Logistic population model

We consider some optimal harvesting policies for a general stochastic Logistic population model. For two management objectives, that are maximum sustainable yield and the maximum retained profits, the optimal harvesting policies are obtained. Meanwhile, the optimal harvest effort, the maximum of expectation of sustainable yield (or retained profits) and the corresponding variance are given.     

Optimal consumption and portfolio under inflation and Markovian switching

In this paper, a financial market with Markovian switching and inflation is described, and the problem of maximizing expected utility of consumption discounted by inflation is given. Then, by a generalized Itô formula for Markov-modulated processes, the verification theorems of optimal policies are derived. Finally, the optimal consumption and portfolio policies for the constant relative risk aversion utility are given explicitly, and an economic analysis is made by numerical examples.     

Optimal replacement policy for obsolete components with general failure rates

Identical components are considered, which become obsolete once new‐type ones are available, more reliable and less energy consuming. We envision different possible replacement strategies for the old‐type components by the new‐type ones: either purely preventive, where all old‐type components are replaced as soon as the new‐type ones are available; either purely corrective, where the old‐type ones are replaced by new‐type ones only at failure; or a mixture of both strategies, where the old‐type ones are first replaced at failure by new‐type ones and next simultaneously preventively replaced after a fixed number of failed old‐type components. To evaluate the respective value of each possible strategy, a cost function is considered, which represents the mean total cost on some finite time interval [0, t]. This function takes into account replacement costs, with economical dependence between simultaneous replacements, and also some energy consumption (and/or production) cost, with a constant rate per unit time. A full analytical expression is provided for the cost function induced by each possible replacement strategy. The optimal strategy is derived in long‐time run. Numerical experiments conclude the paper. Copyright © 2008 John Wiley & Sons, Ltd.     

An optimality principle for Markovian decision processes

The following optimality principle is established for finite undiscounted or discounted Markov decision processes: If a policy is (gain, bias, or discounted) optimal in one state, it is also optimal for all states reachable from this state using this policy. The optimality principle is used constructively to demonstrate the existence of a policy that is optimal in every state, and then to derive the coupled functional equations satisfied by the optimal return vectors. This reverses the usual sequence, where one first establishes (via policy iteration or linear programming) the solvability of the coupled functional equations, and then shows that the solution is indeed the optimal return vector and that the maximizing policy for the functional equations is optimal for every state.     

Optimal static-dynamic hedges for exotic options under convex risk measures

We study the problem of optimally hedging exotic derivatives positions using a combination of dynamic trading strategies in underlying stocks and static positions in vanilla options when the performance is quantified by a convex risk measure. We establish conditions for the existence of an optimal static position for general convex risk measures, and then analyze in detail the case of shortfall risk with a power loss function. Here we find conditions for uniqueness of the static hedge. We illustrate the computational challenge of computing the market-adjusted risk measure in a simple diffusion model for an option on a non-traded asset.     

Optimal policy for brownian inventory models with general convex inventory cost

We study an inventory system in which products are ordered from outside to meet demands, and the cumulative demand is governed by a Brownian motion. Excessive demand is backlogged. We suppose that the shortage and holding costs associated with the inventory are given by a general convex function. The product ordering from outside incurs a linear ordering cost and a setup fee. There is a constant leadtime when placing an order. The optimal policy is established so as to minimize the discounted cost including the inventory cost and ordering cost.     

Optimal policy for a general repair replacement model: average reward case

For a general repair replacement model, we study two types ofreplacement policy.Replacement policy T replaces the systemat time T since the installation or last replacement, whilereplacement policy N replaces the system at the time of Nthfailure. Let T^* and N^* be the optimal among all policies T andN respectively. Under the expected average reward criterion,then we show that the optimal policy N^* is at least as goodas the optimal policy T^*. Furthermore, for a monotone processmodel, we determine the optimal policy N^* explicitly throughtwo different approaches.     

Optimal ordering policy for stock-dependent demand under progressive payment scheme

In this article, a mathematical model is developed to formulate optimal ordering policies for retailer when demand is partially constant and partially dependent on the stock, and the supplier offers progressive credit periods to settle the account. The notion of progressive credit period is as follows:     

Optimal resource acquisition policy for a newsvendor under supply risk

We consider a single-period manufacturing problem involving uncertainty in the availability of a production resource. The resource is stochastically available at the regular cost, but by paying a premium it is possible to reserve and hence guarantee any desired level of the resource in advance. Given the resource consumption rates for a number of products, the manufacturer needs to determine the optimal forward purchase quantity of the resource such that expected profit from selling the products is maximized. The problem is formulated as an extension of the traditional multi-item newsvendor problem. A computational optimization procedure is developed for solving the problem. We find that depending on the profit margins associated with the products, the optimal reservation amount of the resource may increase or decrease as the supply variability increases. The demand volatilities of products are observed to influence the forward purchase quantity of the resource in a similar manner.     

Optimal periodic replacement policy for repairable products under free-repair warranty

This paper investigates the effects of a free-repair warranty on the periodic replacement policy for a repairable product. Cost models are developed for both a warranted and a non-warranted product, and the corresponding optimal periodic replacement policies are derived such that the long-run expected cost rate is minimized. For a product with an increasing failure rate function, structural properties of these optimal policies are obtained. By comparing these optimal policies, we show that the optimal replacement period for a warranted product should be adjusted toward the end of the warranty period. Finally, examples are given to numerically illustrate the impact of a product warranty on the optimal periodic replacement policy.     

具有马尔可夫切换的不确定离散多智能体系统的鲁棒最优一致性

针对不确定二阶离散多智能体系统,研究了其在马尔可夫切换拓扑结构下的鲁棒最优一致性问题.基于智能体的邻居信息设计了控制协议,使得多智能体系统在满足保代价性能指标下最终趋于一致.利用线性矩阵不等式理论以及Lyapunov方法,得到了系统实现均方一致所需要的条件,并且证明了所有智能体的状态最终收敛到其初始状态平均值.进一步,设计了一个保代价性能指标,研究了系统在满足该性能指标下的一致性问题,得到了系统实现均方一致的条件.最后,通过数值仿真实例验证了所得结论的有效性.