Approximation of average cost optimal policies for general Markov decision processes with unbounded costs

The aim of the paper is to show that Lyapunov-like ergodicity conditions on Markov decision processes with Borel state space and possibly unbounded cost provide the approximation of an average cost optimal policy by solvingn-stage optimization problems (n = 1, 2, ...). The used approach ensures the exponential rate of convergence. The approximation of this type would be useful to find adaptive procedures of control and to estimate stability of an optimal control under disturbances of the transition probability.Research supported in part by Consejo Nacional de Ciencia y Tecnologia (CONACYT) under grant 0635P-E9506.Research supported by Fondo del Sistema de Investigatión del Mar de Cortés under Grant SIMAC/94/CT-005.     

Optimal control of semilinear elliptic equations with pointwise constraints on the gradient of the state

In this paper we are concerned with optimal control problems governed by an elliptic semilinear equation, the control being distributed in . The existence of constraints on the control as well as pointwise constraints on the gradient of the state is assumed. A convenient choice of the control space permits us to derive the optimality conditions and study the adjoint state equation, which has derivatives of measures as data. In order to carry out this study, we prove a trace theorem and state Green's formula by using the transposition method.This research was partially supported by Dirección General de Investigation Cientifica y Técnica (Madrid).     

Uniformly convergent finite difference methods for singularly perturbed problems with turning points

In this work one-dimensional singular perturbation problems with turning points are considered. To resolve these problems numerically we consider a family of finite difference schemes, which includes classical methods in literature, such as the upwind method, the Samarskii method and exponential fitting type methods. Once the uniform convergence of the upwind method on irregular meshes has been established, the same property is easily shown on all the elements of the family.Work supported by a grant of the Diputación General de Aragón.     

Error estimates for the approximation of multibang control problems

In the present work we apply an augmented Lagrange method to solve pointwise state constrained elliptic optimal control problems. We prove strong convergence of the primal variables as well as weak convergence of the adjoint states and weak-* convergence of the multipliers associated to the state constraint. In addition, we show that the sequence of generated penalty parameters is bounded only in exceptional situations, which is different from classical results in finite-dimensional optimization. In addition, numerical results are presented.     

Stability of semilinear elliptic optimal control problems with pointwise state constraints

A semi-linear elliptic control problems with distributed control and pointwise inequality constraints on the control and the state is considered. The general optimization problem is perturbed by a certain class of perturbations, and we establish convergence of local solutions of the perturbed problems to a local solution of the unperturbed optimal control problem. This class of perturbations include finite element discretization as well as data perturbation such that the theory implies convergence of finite element approximation and stability w.r.t.?noisy data.     

Inexact Restoration for Euler Discretization of Box-Constrained Optimal Control Problems

The Inexact Restoration method for Euler discretization of state and control constrained optimal control problems is studied. Convergence of the discretized (finite-dimensional optimization) problem to an approximate solution using the Inexact Restoration method and convergence of the approximate solution to a continuous-time solution of the original problem are established. It is proved that a sufficient condition for convergence of the Inexact Restoration method is guaranteed to hold for the constrained optimal control problem. Numerical experiments employing the modelling language AMPL and optimization software Ipopt are carried out to illustrate the robustness of the Inexact Restoration method by means of two computationally challenging optimal control problems, one involving a container crane and the other a free-flying robot. The experiments interestingly demonstrate that one might be better-off using Ipopt as part of the Inexact Restoration method (in its subproblems) rather than using Ipopt directly on its own.     

On the approximation of infinite optimization problems with an application to optimal control problems

This paper is concerned with approximations to infinite optimization problems in Banach spaces. Under the assumption of a first order necessary and a second order sufficient optimality condition we derive convergence results for the optimal solutions and the optimal values of the approximating problems. An application to finite difference approximations of nonlinear optimal control problems with state constraints is given.     

Approximations of optimal control problems for semilinear elliptic equations with discontinuous coefficients and states and with controls in the coefficients multiplying the highest derivatives

Mathematical formulations of nonlinear optimal control problems for semilinear elliptic equations with discontinuous coefficients and solutions and with controls in the coefficients multiplying the highest derivatives are studied. Finite difference approximations of optimization problems are constructed, and the approximation error is estimated with respect to the state and the cost functional. Weak convergence of the approximations with respect to the control is proved. The approximations are regularized in the sense of Tikhonov.     

Boundary control problems for quasi-linear elliptic equations:

In this paper we prove some optimality conditions, in the form of a Pontryagin's principle, for boundary control problems governed by quasi-linear elliptic equations. Because of the presence of state constraints, we distinguish the cases of qualified and nonqualified conditions for optimality. Both cases are treated in the paper. Neither convexity of the control set nor differentiability of the functions involved in the control problem are assumed.This research was partially supported by Dirección General de Investigation Científica y Técnica (Madrid).     

Extended convergence results for the method of multipliers for nonstrictly binding inequality constraints

In this paper, we extend the classical convergence and rate of convergence results for the method of multipliers for equality constrained problems to general inequality constrained problems, without assuming the strict complementarity hypothesis at the local optimal solution. Instead, we consider an alternative second-order sufficient condition for a strict local minimum, which coincides with the standard one in the case of strict complementary slackness. As a consequence, new stopping rules are derived in order to guarantee a local linear rate of convergence for the method, even if the current Lagrangian is only asymptotically minimized in this more general setting. These extended results allow us to broaden the scope of applicability of the method of multipliers, in order to cover all those problems admitting loosely binding constraints at some optimal solution. This fact is not meaningless, since in practice this kind of problem seems to be more the rule rather than the exception.In proving the different results, we follow the classical primaldual approach to the method of multipliers, considering the approximate minimizers for the original augmented Lagrangian as the exact solutions for some adequate approximate augmented Lagrangian. In particular, we prove a general uniform continuity property concerning both their primal and their dual optimal solution set maps, a property that could be useful beyond the scope of this paper. This approach leads to very simple proofs of the preliminary results and to a straight-forward proof of the main results.The author gratefully acknowledges the referees for their helpful comments and remarks. This research was supported by FONDECYT (Fondo Nacional de Desarrollo Científico y Technológico de Chile).     

The Lagrange-Newton method for nonlinear optimal control problems

We investigate local convergence of the Lagrange-Newton method for nonlinear optimal control problems subject to control constraints including the situation where the terminal state is fixed. Sufficient conditions for local quadratic convergence of the method based on stability results for the solutions of nonlinear control problems are discussed.     

Application of variational iteration method for Hamilton–Jacobi–Bellman equations

In this paper, we use the variational iteration method (VIM) for optimal control problems. First, optimal control problems are transferred to Hamilton–Jacobi–Bellman (HJB) equation as a nonlinear first order hyperbolic partial differential equation. Then, the basic VIM is applied to construct a nonlinear optimal feedback control law. By this method, the control and state variables can be approximated as a function of time. Also, the numerical value of the performance index is obtained readily. In view of the convergence of the method, some illustrative examples are presented to show the efficiency and reliability of the presented method.     

Connections between the covector mapping theorem and convergence of pseudospectral methods for optimal control

In recent years, many practical nonlinear optimal control problems have been solved by pseudospectral (PS) methods. In particular, the Legendre PS method offers a Covector Mapping Theorem that blurs the distinction between traditional direct and indirect methods for optimal control. In an effort to better understand the PS approach for solving control problems, we present consistency results for nonlinear optimal control problems with mixed state and control constraints. A set of sufficient conditions is proved under which a solution of the discretized optimal control problem converges to the continuous solution. Convergence of the primal variables does not necessarily imply the convergence of the duals. This leads to a clarification of the Covector Mapping Theorem in its relationship to the convergence properties of PS methods and its connections to constraint qualifications. Conditions for the convergence of the duals are described and illustrated. An application of the ideas to the optimal attitude control of NPSAT1, a highly nonlinear spacecraft, shows that the method performs well for real-world problems. The research was supported in part by NPS, the Secretary of the Air Force, and AFOSR under grant number, F1ATA0-60-6-2G002.     

A convergence result for a sequence of distributed-parameter optimal control problems

In this paper, we consider a sequence of abstract optimal control problems by allowing the cost integrand, the partial differential operator, and the control constraint set all to vary simultaneously. Using the notions of -convergence of functions,G-convergence of operators, and Kuratowski-Mosco convergence of sets, we show that the values of the approximating problems converge to that of the limit problem. Also we show that a convergent sequence of optimal pairs for the approximating problems has a limit which is optimal for the limit problem. A concrete example of parabolic optimal control problems is worked out in detail.This research was supported by NSF Grant No. DMS-88-02688.     

Differential sensitivity of solutions of convex constrained optimal control problems for discrete systems

A family of optimal control problems for discrete systems that depend on a real parameter is considered. The problems are strongly convex and subject to state and control constraints. Some regularity conditions are imposed on the constraints.The control problems are reformulated as mathematical programming problems. It is shown that both the primal and dual optimal variables for these problems are right-differentiable functions of a parameter. The right-derivatives are characterized as solutions to auxiliary quadratic control problems. Conditions of continuous differentiability are discussed, and some estimates of the rate of convergence of the difference quotients to the respective derivatives are given.     

Sensitivity analysis of distributed-parameter optimal control problems for nonlinear parabolic equations

A sequence of optimal control problems for systems described by nonlinear parabolic equations is considered. It is proved that, under the -convergence of objective functionals, the parabolicG-convergence of operators in the state equations, and the Kuratowski convergence of control constraint sets, a convergent sequence of optimal pairs has a limit which is an optimal pair for the limit control problem. The convergence of minimal values is also obtained.This research was supported in part by the Istituto Nazionale di Alta Matematica F. Severi, Rome, Italy. Part of this research was carried out while the author was visiting the Scuola Normale Superiore, Pisa, Italy.     

A mixed finite element method with Lagrange multipliers for nonlinear exterior transmission problems

Summary. We apply a mixed finite element method to numerically solve a class of nonlinear exterior transmission problems in R ² with inhomogeneous interface conditions. Besides the usual unknowns required for the dual-mixed method, which include the gradient of the temperature in this nonlinear case, our approach makes use of the trace of the outer solution on the transmission boundary as a suitable Lagrange multiplier. In addition, we use a boundary integral operator to reduce the original transmission problem on the unbounded region into a nonlocal one on a bounded domain. In this way, we are lead to a two-fold saddle point operator equation as the resulting variational formulation. We prove that the continuous formulation and the associated Galerkin scheme defined with Raviart-Thomas spaces are well posed, and derive the a-priori estimates and the corresponding rate of convergence. Then, we introduce suitable local problems and deduce first an implicit reliable and quasi-efficient a-posteriori error estimate, and then a fully explicit reliable one. Finally, several numerical results illustrate the effectivity of the explicit estimate for the adaptive computation of the discrete solutions. Mathematics Subject Classification (2000): 65N30, 65N38, 65N22, 65F10This research was partially supported by CONICYT-Chile through the FONDAP Program in Applied Mathematics, and by the Dirección de Investigación of the Universidad de Concepción through the Advanced Research Groups Program.     

The Lagrange-Newton method for state constrained optimal control problems

Local convergence of the Lagrange-Newton method for optimization problems with two-norm discrepancy in abstract Banach spaces is investigated. Based on stability analysis of optimization problems with two-norm discrepancy, sufficient conditions for local superlinear convergence are derived. The abstract results are applied to optimal control problems for nonlinear ordinary differential equations subject to control and state constraints.This research was completed while the second author was a visitor at the University of Bayreuth, Germany, supported by grant No. CIPA3510CT920789 from the Commission of the European Communities.     

document

This work develops asymptotically optimal controls for discrete-time singularly perturbed Markov decision processes (MDPs) having weak and strong interactions. The focus is on finite-state-space-MDP problems. The state space of the underlying Markov chain can be decomposed into a number of recurrent classes or a number of recurrent classes and a group of transient states. Using a hierarchical control approach, continuous-time limit problems that are much simpler to handle than the original ones are derived. Based on the optimal solutions for the limit problems, nearly optimal decisions for the original problems are obtained. The asymptotic optimality of such controls is proved and the rate of convergence is provided. Infinite horizon problems are considered; both discounted costs and long-run average costs are examined.