Relaxation of Minimax Optimal Control Problems with Infinite Horizon

A minimax optimal control problem with infinite horizon is studied. We analyze a relaxation of the controls, which allows us to consider a generalization of the original problem that not only has existence of an optimal control but also enables us to approximate the infinite-horizon problem with a sequence of finite-horizon problems. We give a set of conditions that are sufficient to solve directly, without relaxation, the infinite-horizon problem as the limit of finite-horizon problems.     

非时齐部分可观察Markov决策规划的最优策略问题

本文讨论了一类非时齐部分可观察Markov决策模型．在不改变状态空间可列性的条件下，把该模型转化为[5]中的一般化折扣模型，从而解决了其最优策略问题，并且得到了该模型的有限阶段逼近算法，其中该算法涉及的状态是可列的．     

Gradient-flow approach for computing a nonlinear-quadratic optimal-output feedback gain matrix

In this paper, an approach is proposed for solving a nonlinear-quadratic optimal regulator problem with linear static state feedback and infinite planning horizon. For such a problem, approximate problems are introduced and considered, which are obtained by combining a finite-horizon problem with an infinite-horizon linear problem in a certain way. A gradient-flow based algorithm is derived for these approximate problems. It is shown that an optimal solution to the original problem can be found as the limit of a sequence of solutions to the approximate problems. Several important properties are obtained. For illustration, two numerical examples are presented.This project was partially supported by a research grant from the Australian Research Council.     

Markov Perfect Equilibrium Existence for a Class of Undiscounted Infinite-Horizon Dynamic Games

We prove the existence of Markov perfect equilibria (MPE) for nonstationary undiscounted infinite-horizon dynamic games with alternating moves. A suitable finite-horizon equilibrium relaxation, the ending state constrained MPE, captures the relevant features of an infinite-horizon MPE for a long enough horizon, under a uniformly bounded reachability assumption.     

On the optimality of a full-service policy for a queueing system with discounted costs

We provide weak sufficient conditions for a full-service policy to be optimal in a queueing control problem in which the service rate is a dynamic decision variable. In our model there are service costs and holding costs and the objective is to minimize the expected total discounted cost over an infinite horizon. We begin with a semi-Markov decision model for a single-server queue with exponentially distributed inter-arrival and service times. Then we present a general model with weak probabilistic assumptions and demonstrate that the full-service policy minimizes both finite-horizon and infinite-horizon total discounted cost on each sample path.     

Approximation solution and suboptimality for discounted semi-markov decision problems with countable state space

In this article we consider the approximate solution for semi-Markov decision problems with infinite horizon, countable state space, discounted cost function and finite action space. We present converging sequences of lower and upper bounds for the value function and, moreover, we derive a method for exclusion of suboptimal actions.     

Multirate sampling and input-to-state stable receding horizon control for nonlinear systems

In this paper, a robust receding horizon control for multirate sampled-data nonlinear systems with bounded disturbances is presented. The proposed receding horizon control is based on the solution of Bolza-type optimal control problems for the approximate discrete-time model of the nominal system. “Low measurement rate” is assumed. It is shown that the multistep receding horizon controller that stabilizes the nominal approximate discrete-time model also practically input-to-state stabilizes the exact discrete-time system with disturbances.     

Optimal policy for minimizing risk models in Markov decision processes

We consider the minimizing risk problems in discounted Markov decisions processes with countable state space and bounded general rewards. We characterize optimal values for finite and infinite horizon cases and give two sufficient conditions for the existence of an optimal policy in an infinite horizon case. These conditions are closely connected with Lemma 3 in White (1993), which is not correct as Wu and Lin (1999) point out. We obtain a condition for the lemma to be true, under which we show that there is an optimal policy. Under another condition we show that an optimal value is a unique solution to some optimality equation and there is an optimal policy on a transient set.     

Suboptimal Explicit Receding Horizon Control via Approximate Multiparametric Quadratic Programming

Algorithms for solving multiparametric quadratic programming (MPQP) were recently proposed in Refs. 1–2 for computing explicit receding horizon control (RHC) laws for linear systems subject to linear constraints on input and state variables. The reason for this interest is that the solution to MPQP is a piecewise affine function of the state vector and thus it is easily implementable online. The main drawback of solving MPQP exactly is that, whenever the number of linear constraints involved in the optimization problem increases, the number of polyhedral cells in the piecewise affine partition of the parameter space may increase exponentially. In this paper, we address the problem of finding approximate solutions to MPQP, where the degree of approximation is arbitrary and allows to tradeoff between optimality and a smaller number of cells in the piecewise affine solution. We provide analytic formulas for bounding the errors on the optimal value and the optimizer, and for guaranteeing that the resulting suboptimal RHC law provides closed-loop stability and constraint fulfillment.     

Nonstationary Continuous-Time Markov Control Processes with Discounted Costs on Infinite Horizon

This paper concerns nonstationary continuous-time Markov control processes on Polish spaces, with the infinite-horizon discounted cost criterion. Necessary and sufficient conditions are given for a control policy to be optimal and asymptotically optimal. In addition, under suitable hypotheses, it is shown that the successive approximation procedure converges in the sense that the sequence of finite-horizon optimal cost functions and the corresponding optimal control policies both converge.     

Finite approximation for finite-horizon continuous-time Markov decision processes

In this paper we study the continuous-time Markov decision processes with a denumerable state space, a Borel action space, and unbounded transition and cost rates. The optimality criterion to be considered is the finite-horizon expected total cost criterion. Under the suitable conditions, we propose a finite approximation for the approximate computations of an optimal policy and the value function, and obtain the corresponding error estimations. Furthermore, our main results are illustrated with a controlled birth and death system.     

Boundaries of the receding horizon control for interconnected systems

In recent years, the finite-horizon quadratic minimization problem has become popular in process control, where the horizon is constantly rolled back. In this paper, this type of control, which is also called the receding horizon control, is considered for interconnected systems. First, the receding horizon control equations are formulated; then, some stability conditions depending on the interconnection norms and the horizon lengths are presented. For -coupled systems, stability results similar to centralized systems are obtained. For interconnected systems which are not -coupled, the existence of a horizon length and a corresponding stabilizing receding horizon control are derived. Finally, the performance of a locally computed receding horizon control for time-invariant and time-varying systems with different updating intervals is examined in an example.     

Optimal Policies for Production/Inventory Systems with Finite Capacity and Markov-Modulated Demand and Supply Processes

In many production/inventory systems, not only is the production/inventory capacity finite, but the systems are also subject to random production yields that are influenced by factors such as breakdowns, repairs, maintenance, learning, and the introduction of new technologies. In this paper, we consider a single-item, single-location, periodic-review model with finite capacity and Markov modulated demand and supply processes. When demand and supply processes are driven by two independent, discrete-time, finite-state, time-homogeneous Markov chains, we show that a modified, state-dependent, inflated base-stock policy is optimal for both the finite and infinite horizon planning problems. We also show that the finite-horizon solution converges to the infinite-horizon solution.     

A new iterative scheme for a countable family of relatively nonexpansive mappings and an equilibrium problem in Banach spaces

In this paper, we construct a new iterative scheme and prove strong convergence theorem for approximation of a common fixed point of a countable family of relatively nonexpansive mappings, which is also a solution to an equilibrium problem in a uniformly convex and uniformly smooth real Banach space. We apply our results to approximate fixed point of a nonexpansive mapping, which is also solution to an equilibrium problem in a real Hilbert space and prove strong convergence of general H-monotone mappings in a uniformly convex and uniformly smooth real Banach space. Our results extend many known recent results in the literature.     

Optimal harvesting in age-structured populations

In this paper, we discuss the optimal harvesting in age-structured populations. We characterize the optimal controls for finite-horizon problems and describe a scheme to approximate them. We also give an analysis of the infinite-horizon problem as a function of a bifurcation parameter.This research was supported by NSF Grant R11-89-05084.     

New iterative scheme with nonexpansive mappings for equilibrium problems and variational inequality problems in Hilbert spaces

Recently, Ceng, Guu and Yao introduced an iterative scheme by viscosity-like approximation method to approximate the fixed point of nonexpansive mappings and solve some variational inequalities in Hilbert space (see Ceng et al. (2009) [9]). Takahashi and Takahashi proposed an iteration scheme to solve an equilibrium problem and approximate the fixed point of nonexpansive mapping by viscosity approximation method in Hilbert space (see Takahashi and Takahashi (2007) [12]). In this paper, we introduce an iterative scheme by viscosity approximation method for finding a common element of the set of a countable family of nonexpansive mappings and the set of an equilibrium problem in a Hilbert space. We prove the strong convergence of the proposed iteration to the unique solution of a variational inequality.     

Direct trajectory optimization and costate estimation of finite-horizon and infinite-horizon optimal control problems using a Radau pseudospectral method

A method is presented for direct trajectory optimization and costate estimation of finite-horizon and infinite-horizon optimal control problems using global collocation at Legendre-Gauss-Radau (LGR) points. A key feature of the method is that it provides an accurate way to map the KKT multipliers of the nonlinear programming problem to the costates of the optimal control problem. More precisely, it is shown that the dual multipliers for the discrete scheme correspond to a pseudospectral approximation of the adjoint equation using polynomials one degree smaller than that used for the state equation. The relationship between the coefficients of the pseudospectral scheme for the state equation and for the adjoint equation is established. Also, it is shown that the inverse of the pseudospectral LGR differentiation matrix is precisely the matrix associated with an implicit LGR integration scheme. Hence, the method presented in this paper can be thought of as either a global implicit integration method or a pseudospectral method. Numerical results show that the use of LGR collocation as described in this paper leads to the ability to determine accurate primal and dual solutions for both finite and infinite-horizon optimal control problems.     

Optimality of monotonic policies for two-action Markovian decision processes,with applications to control of queues with delayed information

We consider a discrete-time Markov decision process with a partially ordered state space and two feasible control actions in each state. Our goal is to find general conditions, which are satisfied in a broad class of applications to control of queues, under which an optimal control policy is monotonic. An advantage of our approach is that it easily extends to problems with both information and action delays, which are common in applications to high-speed communication networks, among others. The transition probabilities are stochastically monotone and the one-stage reward submodular. We further assume that transitions from different states are coupled, in the sense that the state after a transition is distributed as a deterministic function of the current state and two random variables, one of which is controllable and the other uncontrollable. Finally, we make a monotonicity assumption about the sample-path effect of a pairwise switch of the actions in consecutive stages. Using induction on the horizon length, we demonstrate that optimal policies for the finite- and infinite-horizon discounted problems are monotonic. We apply these results to a single queueing facility with control of arrivals and/or services, under very general conditions. In this case, our results imply that an optimal control policy has threshold form. Finally, we show how monotonicity of an optimal policy extends in a natural way to problems with information and/or action delay, including delays of more than one time unit. Specifically, we show that, if a problem without delay satisfies our sufficient conditions for monotonicity of an optimal policy, then the same problem with information and/or action delay also has monotonic (e.g., threshold) optimal policies.     

Impulsively-controlled systems and reverse dwell time: A linear programming approach

We present a receding horizon algorithm that converges to the exact solution in polynomial time for a class of optimal impulse control problems with uniformly distributed impulse instants and governed by so-called reverse dwell time conditions. The cost has two separate terms, one depending on time and the second monotonically decreasing on the state norm. The obtained results have both theoretical and practical relevance. From a theoretical perspective we prove certain geometrical properties of the discrete set of feasible solutions. From a practical standpoint, such properties reduce the computational burden and speed up the search for the optimum thus making the algorithm suitable for the on-line implementation in real-time problems. Our approach consists in approximating the optimal impulse control problem via a binary linear programming problem with a totally unimodular constraint matrix. Hence, solving the binary linear programming problem is equivalent to solving its linear relaxation. Then, given the feasible solution from the linear relaxation, we find the optimal solution via receding horizon and local search. Numerical illustrations of a queueing system are performed.     

Asymptotic properties of constrained Markov Decision Processes

We present in this paper several asymptotic properties of constrained Markov Decision Processes (MDPs) with a countable state space. We treat both the discounted and the expected average cost, with unbounded cost. We are interested in (1) the convergence of finite horizon MDPs to the infinite horizon MDP, (2) convergence of MDPs with a truncated state space to the problem with infinite state space, (3) convergence of MDPs as the discount factor goes to a limit. In all these cases we establish the convergence of optimal values and policies. Moreover, based on the optimal policy for the limiting problem, we construct policies which are almost optimal for the other (approximating) problems. Based on the convergence of MDPs with a truncated state space to the problem with infinite state space, we show that an optimal stationary policy exists such that the number of randomisations it uses is less or equal to the number of constraints plus one. We finally apply the results to a dynamic scheduling problem.This work was partially supported by the Chateaubriand fellowship from the French embassy in Israel and by the European Grant BRA-QMIPS of CEC DG XIII