Optimal infinite-horizon feedback laws for a general class of constrained discrete-time systems: Stability and moving-horizon approximations

Stability results are given for a class of feedback systems arising from the regulation of time-varying discrete-time systems using optimal infinite-horizon and moving-horizon feedback laws. The class is characterized by joint constraints on the state and the control, a general nonlinear cost function and nonlinear equations of motion possessing two special properties. It is shown that weak conditions on the cost function and the constraints are sufficient to guarantee uniform asymptotic stability of both the optimal infinite-horizon and moving-horizon feedback systems. The infinite-horizon cost associated with the moving-horizon feedback law approaches the optimal infinite-horizon cost as the moving horizon is extended.     

A risk model with renewal shot-noise Cox process

In this paper we generalise the risk models beyond the ordinary framework of affine processes or Markov processes and study a risk process where the claim arrivals are driven by a Cox process with renewal shot-noise intensity. The upper bounds of the finite-horizon and infinite-horizon ruin probabilities are investigated and an efficient and exact Monte Carlo simulation algorithm for this new process is developed. A more efficient estimation method for the infinite-horizon ruin probability based on importance sampling via a suitable change of probability measure is also provided; illustrative numerical examples are also provided.     

Relaxation of Minimax Optimal Control Problems with Infinite Horizon

A minimax optimal control problem with infinite horizon is studied. We analyze a relaxation of the controls, which allows us to consider a generalization of the original problem that not only has existence of an optimal control but also enables us to approximate the infinite-horizon problem with a sequence of finite-horizon problems. We give a set of conditions that are sufficient to solve directly, without relaxation, the infinite-horizon problem as the limit of finite-horizon problems.     

The multiparameter eigenvalue problem: Jordan vector semilattices

In this paper, multiparameter eigenvalue (MPE) problems for matrices are considered. The notion of Jordan vector semilattices as a generalization of the notion of Jordan vector chains is introduced for a multiple spectrum point of disconnected MPE problems. The notion of generating vector is introduced. For the linear case, a special form of equations determining Jordan vector semilattices is presented. The above notions are extended to the case of connected MPE problems, including linear ones. The relationship between the Jordan vector semilattices of a connected linear MPE problem and the Jordan vector chains of the corresponding simultaneous spectral problems for matrices is established. Bibliography: 5 titles. Translated fromZapiski Nauchnykh Seminarov POMI, Vol. 219, 1994, pp. 213–220. This paper was supported by the Russian Foundation of Fundamental Research (grant 94-01-00919). Translated by V. B. Khazanov.     

Stochastic Multiproduct Inventory Models with Limited Storage

This paper studies multiproduct inventory models with stochastic demands and a warehousing constraint. Finite horizon as well as stationary and nonstationary discounted-cost infinite-horizon problems are addressed. Existence of optimal feedback policies is established under fairly general assumptions. Furthermore, the structure of the optimal policies is analyzed when the ordering cost is linear and the inventory/backlog cost is convex. The optimal policies generalize the base-stock policies in the single-product case. Finally, in the stationary infinite-horizon case, a myopic policy is proved to be optimal if the product demands are independent and the cost functions are separable.     

First-order necessary conditions for infinite-horizon variational problems

We establish rigorously several pointwise or asymptotic firstorder necessary conditions for infinite-horizon variational problems in general form, in the framework of continuous time. We obtain several new results, and we extend to general differentiable Lagrangians some results known only in special cases. To realize this aim, we justify two different ways to associate a family of finite-horizon problems to an infinite-horizon problem.The authors thank an anonymous referee for providing important historical references     

Infinite-horizon deterministic dynamic programming in discrete time: a monotone convergence principle and a penalty method

We consider infinite-horizon deterministic dynamic programming problems in discrete time. We show that the value function of such a problem is always a fixed point of a modified version of the Bellman operator. We also show that value iteration converges increasingly to the value function if the initial function is dominated by the value function, is mapped upward by the modified Bellman operator and satisfies a transversality-like condition. These results require no assumption except for the general framework of infinite-horizon deterministic dynamic programming. As an application, we show that the value function can be approximated by computing the value function of an unconstrained version of the problem with the constraint replaced by a penalty function.     

A Carathéodory-Hamilton-Jacobi theory for infinite-horizon optimal control problems

In this paper, we extend Carathéodory's concept of equivalent variational problems to infinite-horizon optimal control problems. In such a setting, the usual concept of a minimum need not exist, and we therefore consider a weaker definition of optimality, known as catching up optimality. The extension presented here leads us to a Hamilton-Jacobi theory for infinite-horizon optimal control problems that closely parallels the classical work of Carathéodory as well as providing sufficient conditions for optimality. Finally, we show that the results given here subsume several previously known results as a special case.This research forms part of the author's doctoral dissertation, written at the University of Delaware, Newark, Delaware, under the supervision of Professor T. S. Angell.     

An on-line procedure in discounted infinite-horizon stochastic optimal control

We consider the discrete-time, infinite-horizon optimal control problem with discounted cost. We propose a test to detect forecast/planning horizons, and derive an on-line procedure for solving the original problem.     

Pontryagin principles for bounded discrete-time processes

Abstract

We establish necessary conditions and sufficient conditions of optimality in the form of Pontryagin principles for infinite-horizon discrete-time optimal control problems governed by a difference inequation.     

Some Extensions to a Direct Optimization Method

A direct method for the global extremization of a class of integrals, introduced in Refs. 1–3, is generalized to allow for constraints in the form of differential conditions and by considering the so-called infinite-horizon case.     

Missing Point Estimation for steady aerodynamic applications

Model reduction in aerodynamic applications usually makes use of Proper Orthogonal Decomposition (POD). In this work a POD-based method, called Missing Point Estimation (MPE), will be applied to steady state flows with variation of the flow parameter angle of attack. The basic idea of MPE is to select a subset of the computational grid points (control volumes) and limit the governing equations to these. Subsequently, the remaining equations are projected onto the POD subspace. This approach has the advantage that the nonlinear right hand side of the governing equations has to be evaluated only for some selected points (control volumes) which makes the MPE faster than classical Galerkin projection. (© 2011 Wiley-VCH Verlag GmbH & Co. KGaA, Weinheim)     

Mean,variance, and probabilistic criteria in finite Markov decision processes: A review

This paper is a survey of papers which make use of nonstandard Markov decision process criteria (i.e., those which do not seek simply to optimize expected returns per unit time or expected discounted return). It covers infinite-horizon nondiscounted formulations, infinite-horizon discounted formulations, and finite-horizon formulations. For problem formulations in terms solely of the probabilities of being in each state and taking each action, policy equivalence results are given which allow policies to be restricted to the class of Markov policies or to the randomizations of deterministic Markov policies. For problems which cannot be stated in such terms, in terms of the primitive state setI, formulations involving a redefinition of the states are examined.The author would like to thank two referees for a very thorough and helpful referceing of the original article and for the extra references (Refs. 47–52) now added to the original reference list.     

A Vector Minimum Superharmonic Approach to Solving Infinite-Horizon Discounted Markov Decision Processes

In this paper a new algorithm is provided for obtaining approximately optimal policies for infinite-horizon discounted Markov decision processes. In addition, some of the properties of the algorithm are established. The algorithm is based upon the fact that the optimal value function is the unique vector minimum function within the superharmonic set.     

On the existence of catching-up optimal solutions for Lagrange problems defined on unbounded intervals

In this paper, we are concerned with the question of the existence of optimal solutions for infinite-horizon optimal control problems of Lagrange type. In such problems, the objective or cost functional is described by an improper integral. As dictated by applications arising in mathematical economics, we do nota priori assume that this improper integral converges. This leads us to consider a weaker type of optimality, known as catching-up optimality. The results presented here utilize the classical convexity and seminormality conditions typically imposed in the existence theory for the case of finite intervals. These conditions are significantly weaker than those imposed by other authors; as a consequence, their existence results are contained as special cases of the results presented here. The method of proof utilizes the Carathéodory-Hamilton-Jacobi theory previously developed by the author for infinite-horizon optimal control problems.This research forms part of the author's doctoral dissertation written at the University of Delaware, Newark, Delaware under the supervision of Professor T. S. Angell.     

A method of acceleration for a class of multiplicative programming problems with exponent

Multiplicative programming problems with exponent (MPE) have many practical applications in various fields. In this paper, a method for accelerating global optimization is proposed for a class of multiplicative programming problems with exponent under multiplicative constraints using a suitable deleting technique. This technique offers the possibility of cutting away a large part of the currently investigated region in which the globally optimal solution of the MPE does not exist. The deleting technique can accelerate the convergence of the proposed global optimization algorithm. Two numerical examples are given to illustrate the feasibility of the deleting technique.     

Direct trajectory optimization and costate estimation of finite-horizon and infinite-horizon optimal control problems using a Radau pseudospectral method

A method is presented for direct trajectory optimization and costate estimation of finite-horizon and infinite-horizon optimal control problems using global collocation at Legendre-Gauss-Radau (LGR) points. A key feature of the method is that it provides an accurate way to map the KKT multipliers of the nonlinear programming problem to the costates of the optimal control problem. More precisely, it is shown that the dual multipliers for the discrete scheme correspond to a pseudospectral approximation of the adjoint equation using polynomials one degree smaller than that used for the state equation. The relationship between the coefficients of the pseudospectral scheme for the state equation and for the adjoint equation is established. Also, it is shown that the inverse of the pseudospectral LGR differentiation matrix is precisely the matrix associated with an implicit LGR integration scheme. Hence, the method presented in this paper can be thought of as either a global implicit integration method or a pseudospectral method. Numerical results show that the use of LGR collocation as described in this paper leads to the ability to determine accurate primal and dual solutions for both finite and infinite-horizon optimal control problems.     

Optimal harvesting in age-structured populations

In this paper, we discuss the optimal harvesting in age-structured populations. We characterize the optimal controls for finite-horizon problems and describe a scheme to approximate them. We also give an analysis of the infinite-horizon problem as a function of a bifurcation parameter.This research was supported by NSF Grant R11-89-05084.     

Approximate receding horizon approach for Markov decision processes: average reward case

We consider an approximation scheme for solving Markov decision processes (MDPs) with countable state space, finite action space, and bounded rewards that uses an approximate solution of a fixed finite-horizon sub-MDP of a given infinite-horizon MDP to create a stationary policy, which we call “approximate receding horizon control.” We first analyze the performance of the approximate receding horizon control for infinite-horizon average reward under an ergodicity assumption, which also generalizes the result obtained by White (J. Oper. Res. Soc. 33 (1982) 253-259). We then study two examples of the approximate receding horizon control via lower bounds to the exact solution to the sub-MDP. The first control policy is based on a finite-horizon approximation of Howard's policy improvement of a single policy and the second policy is based on a generalization of the single policy improvement for multiple policies. Along the study, we also provide a simple alternative proof on the policy improvement for countable state space. We finally discuss practical implementations of these schemes via simulation.     

Asset bubbles and efficiency in a generalized two-sector model

We consider a multi-sector infinite-horizon general equilibrium model. The issues of equilibrium existence, efficiency, and bubble emergence are addressed. We show how different assets give rise to different rational bubbles. We also point out that efficient bubbly equilibria may exist.