A mean–variance optimization problem for discounted Markov decision processes

In this paper, we consider a mean–variance optimization problem for Markov decision processes (MDPs) over the set of (deterministic stationary) policies. Different from the usual formulation in MDPs, we aim to obtain the mean–variance optimal policy that minimizes the variance over a set of all policies with a given expected reward. For continuous-time MDPs with the discounted criterion and finite-state and action spaces, we prove that the mean–variance optimization problem can be transformed to an equivalent discounted optimization problem using the conditional expectation and Markov properties. Then, we show that a mean–variance optimal policy and the efficient frontier can be obtained by policy iteration methods with a finite number of iterations. We also address related issues such as a mutual fund theorem and illustrate our results with an example.     

Reduction of total-cost and average-cost MDPs with weakly continuous transition probabilities to discounted MDPs

This note describes sufficient conditions under which total-cost and average-cost Markov decision processes (MDPs) with general state and action spaces, and with weakly continuous transition probabilities, can be reduced to discounted MDPs. For undiscounted problems, these reductions imply the validity of optimality equations and the existence of stationary optimal policies. The reductions also provide methods for computing optimal policies. The results are applied to a capacitated inventory control problem with fixed costs and lost sales.     

Competing Markov decision processes

A class of discounted Markov decision processes (MDPs) is formed by bringing together individual MDPs sharing the same discount rate. These are in competition in the sense that at each decision epoch a single action is chosen from the union of the action sets of the individual MDPs. Such families of competing MDPs have been used to model a variety of problems in stochastic resource allocation and in the sequential design of experiments. Suppose thatS is a stationary strategy for such a family, thatS* is an optimal strategy and thatR(S),R(S*) denote the respective rewards earned. The paper extends (and explains) existing theory based on the Gittins index to give bounds onR(S*)-R(S) for this important class of processes. The procedures are illustrated by examples taken from the fields of stochastic scheduling and research planning.     

Differentiability of the mappings of Carnot-Carathéodory spaces in the Sobolev and <Emphasis Type="Italic">BV</Emphasis>-topologies

We prove differentiability of the mappings of the Sobolev classes and BV-mappings of Carnot-Carathéodory spaces in the topology of these classes. We infer from these results a generalization of the Calderón-Zygmund theorems for mappings of the Carnot-Carathéodory spaces and other facts.     

Existence of minimizers and necessary conditions in set-valued optimization with equilibrium constraints

In this paper we study set-valued optimization problems with equilibrium constraints (SOPECs) described by parametric generalized equations in the form 0 ∈ G(x) + Q(x), where both G and Q are set-valued mappings between infinite-dimensional spaces. Such models particularly arise from certain optimization-related problems governed by set-valued variational inequalities and first-order optimality conditions in nondifferentiable programming. We establish general results on the existence of optimal solutions under appropriate assumptions of the Palais-Smale type and then derive necessary conditions for optimality in the models under consideration by using advanced tools of variational analysis and generalized differentiation. Dedicated to Jiří V. Outrata on the occasion of his 60th birthday. This research was partly supported by the National Science Foundation under grants DMS-0304989 and DMS-0603846 and by the Australian Research Council under grant DP-0451168.     

The Morse–Sard Theorem and Luzin N-Property: A New Synthesis for Smooth and Sobolev Mappings

Considering regular mappings of Euclidean spaces, we study the distortion of the Hausdorff dimension of a given set under restrictions on the rank of the gradient on the set. This problem was solved for the classical cases of k-smooth and Hölder mappings by Dubovitskii, Bates, and Moreira. We solve the problem for Sobolev and fractional Sobolev classes as well. Here we study the Sobolev case under minimal integrability assumptions that guarantee in general only the continuity of a mapping (rather than differentiability everywhere). Some new facts are found out in the classical smooth case. The proofs are mostly based on our previous joint papers with Bourgain and Kristensen (2013, 2015).

     

On changing the spaces in Lagrange multiplier rules for the optimal control of non-linear operator equations

In this paper it is shown, how Lagrange multiplier rules for nonlinear optimal control problems in Banach spaces can be transferred by a simple device from the initial space to a more useful Banach space, in order to avoid unhandy dual spaces. The method is applied to state-equations of the type x-K(x,u)= 0, where the Fréchet-derivative of K has a certain smoothing property which is typical for integral operators.     

On differentiability of ruin functions under Markov-modulated models

This paper analyzes the continuity and differentiability of several classes of ruin functions under Markov-modulated insurance risk models with a barrier and threshold dividend strategy, respectively. Many ruin related functions in the literature, such as the expectation and the Laplace transform of the Gerber–Shiu discounted penalty function at ruin, of the total discounted dividends until ruin, and of the time-integrated discounted penalty and/or reward function of the risk process, etc, are special cases of the functions considered in this paper. Continuity and differentiability of these functions in the corresponding dual models are also studied.     

Continuous time control of Markov processes on an arbitrary state space: Average return criterion

The paper deals with continuous time Markov decision processes on a fairly general state space. The economic criterion is the long-run average return. A set of conditions is shown to be sufficient for a constant g to be optimal average return and a stationary policy π¹ to be optimal. This condition is shown to be satisfied under appropriate assumptions on the optimal discounted return function. A policy improvement algorithm is proposed and its convergence to an optimal policy is proved.     

Elliptic equations on convex domains with nonhomogeneous Dirichlet boundary conditions

In this paper, we will study the differentiability on the boundary of solutions of elliptic non-divergence differential equations on convex domains. The results are divided into two cases: (i) at the boundary points where the blow-up of the domain is not the half-space, if the boundary function is differentiable then the solution is differentiable; (ii) at the boundary points where the blow-up of the domain is the half-space, the differentiability of the solution needs an extra Dini condition for the boundary function. Counterexample is given to show that our results are optimal.     

Fréchet Embeddings of Negative Type Metrics

We show that every n-point metric of negative type (in particular, every n-point subset of L ₁) admits a Fréchet embedding into Euclidean space with distortion , a result which is tight up to the O(log log n) factor, even for Euclidean metrics. This strengthens our recent work on the Euclidean distortion of metrics of negative into Euclidean space. S. Arora supported by David and Lucile Packard Fellowship and NSF grant CCR-0205594. J.R. Lee supported by NSF grant CCR-0121555, NSF 0514993, NSF 0528414 and an NSF Graduate Research Fellowship.     

Differentiability of Lipschitz Maps from Metric Measure Spaces to Banach Spaces with the Radon–Nikodym Property

We prove the differentiability of Lipschitz maps X → V, where X denotes a PI space, i.e. a complete metric measure space satisfying a doubling condition and a Poincaré inequality, and V denotes a Banach space with the Radon–Nikodym Property (RNP). As a consequence, we obtain a bi-Lipschitz nonembedding theorem for RNP targets. The differentiation theorem depends on a new specification of the differentiable structure for PI spaces involving directional derivatives in the direction of velocity vectors to rectifiable curves. We give two different proofs of this, the second of which relies on a new characterization of the minimal upper gradient. There are strong implications for the infinitesimal structure of PI spaces which will be discussed elsewhere.     

Differentiability in optimization theory 1

This paper deals with necessary conditions for optimization problems with infinitely many inequality constraints assuming various differentiability conditions. By introducing a second topology N on a topological vector space we define generalized versions of differentiability and tangential cones. Different choices of N lead to Gâteaux-, Hadamaed- and weak differentiability with corresponding tangential cones. The general concept is used to derive necessary conditions for local optimal points in form of inequalities and generalized multiplier rules, Special versions of these theorems are obtained for different differentiability assumptions by choosing properly. An application to approximation theory is given.     

Differentiability of convex measures

In this paper we consider convex measures on finite-dimensional spaces. We prove the differentiability of convex measures in the Skorokhod sense (and under some natural conditions, in the Fomin sense also). Simultaneously we give some additional results on differentiability of convex measures.Translated fromMatematicheskie Zametki, Vol. 58, No. 6, pp. 862–871, December, 1995.This research was partially supported by the Russian Foundation for Basic Research under grant No. 94-01-01556, by the Ministry of Science under grant No. M38300 and by the International Science Foundation under grant No. M38000.     

Optimal intensity control of a multi-class queue

We study a mixed problem of optimal scheduling and input and output control of a single server queue with multi-classes of customers. The model extends the classical optimal scheduling problem by allowing the general point processes as the arrival and departure processes and the control of the arrival and departure intensities. The objective of our scheduling and control problem is to minimize the expected discounted inventory cost over an infinite horizon, and the problem is formulated as an intensity control. We find the well-knownc is the optimal solution to our problem.Supported in part by NSF under grant ECS-8658157, by ONR under contract N00014-84-K-0465, and by a grant from AT&T Bell Laboratories.The work was done while the author was a postdoctoral fellow in the Division of Applied Sciences, Harvard University, Cambridge, Massachusetts 02138.     

A survey of recent results on continuous-time Markov decision processes

This paper is a survey of recent results on continuous-time Markov decision processes (MDPs) withunbounded transition rates, and reward rates that may beunbounded from above and from below. These results pertain to discounted and average reward optimality criteria, which are the most commonly used criteria, and also to more selective concepts, such as bias optimality and sensitive discount criteria. For concreteness, we consider only MDPs with a countable state space, but we indicate how the results can be extended to more general MDPs or to Markov games. Research partially supported by grants NSFC, DRFP and NCET. Research partially supported by CONACyT (Mexico) Grant 45693-F.     

Optimal dividend strategies in discrete risk model with capital injections

In this paper we consider a doubly discrete model used in Dickson and Waters (biASTIN Bulletin 1991; 21 :199–221) to approximate the Cramér–Lundberg model. The company controls the amount of dividends paid out to the shareholders as well as the capital injections which make the company never ruin in order to maximize the cumulative expected discounted dividends minus the penalized discounted capital injections. We show that the optimal value function is the unique solution of a discrete Hamilton–Jacobi–Bellman equation by contraction mapping principle. Moreover, with capital injection, we reduce the optimal dividend strategy from band strategy in the discrete classical risk model without external capital injection into barrier strategy , which is consistent with the result in continuous time. We also give the equivalent condition when the optimal dividend barrier is equal to 0. Although there is no explicit solution to the value function and the optimal dividend barrier, we obtain the optimal dividend barrier and the approximating solution of the value function by Bellman's recursive algorithm. From the numerical calculations, we obtain some relevant economical insights. Copyright © 2010 John Wiley & Sons, Ltd.     

Mean integrated squared error of kernel estimators when the density and its derivative are not necessarily continuous

Summary Asymptotic properties of the mean integrated squared error (MISE) of kernel estimators of a density function, based on a sampleX ₁, …,X _n, were obtained by Rosenblatt [4] and Epanechnikov [1] for the case when the densityf and its derivativef′ are continuous. They found, under certain additional regularity conditions, that the optimal choiceh _n0 for the scale factorh _n=Kn^−α is given byh _n0=K₀n^−1/5 withK ₀ depending onf and the kernel; they also showed that MISE(h _n0)=O(n^−4/5) and Epanechnikov [1] found the optimal kernel. In this paper we investigate the robustness of these results to departures from the assumptions concerning the smoothness of the density function. In particular it is shown, under certain regularity conditions, that whenf is continuous but its derivativef′ is not, the optimal value of α in the scale factor becomes 1/4 and MISE(h _n0)=O(n^−3/4); for the case whenf is not continuous the optimal value of α becomes 1/2 and MISE(h _n0)=O(n^−1/2). For this last case the optimal kernel is shown to be the double exponential density. Supported by the Natural Sciences and Engineering Research Council of Canada under Grant Nr. A 3114 and by the Gouvernement du Québec, Programme de formation de chercheurs et d'action concertée.     

A characterization of Lévy probability distribution functions on Euclidean spaces

In 1937, Paul Lévy proved two theorems that characterize one-dimensional distribution functions of class L. In 1972, Urbanik generalized Lévy's first theorem. In this note, we generalize Lévy's second theorem and obtain a new characterization of Lévy probability distribution functions on Euclidean spaces. This result is used to obtain a new characterization of operator stable distribution functions on Euclidean spaces and to show that symmetric Lévy distribution functions on Euclidean spaces need not be symmetric unimodal.