Mayer and optimal stopping stochastic control problems with discontinuous cost

We study two classes of stochastic control problems with semicontinuous cost: the Mayer problem and optimal stopping for controlled diffusions. The value functions are introduced via linear optimization problems on appropriate sets of probability measures. These sets of constraints are described deterministically with respect to the coefficient functions. Both the lower and upper semicontinuous cases are considered. The value function is shown to be a generalized viscosity solution of the associated HJB system, respectively, of some variational inequality. Dual formulations are given, as well as the relations between the primal and dual value functions. Under classical convexity assumptions, we prove the equivalence between the linearized Mayer problem and the standard weak control formulation. Counter-examples are given for the general framework.     

Compactness of the space of non-randomized policies in countable-state sequential decision processes

For sequential decision processes with countable state spaces, we prove compactness of the set of strategic measures corresponding to nonrandomized policies. For the Borel state case, this set may not be compact (Piunovskiy, Optimal control of random sequences in problems with constraints. Kluwer, Boston, p. 170, 1997) in spite of compactness of the set of strategic measures corresponding to all policies (Schäl, On dynamic programming: compactness of the space of policies. Stoch Processes Appl 3(4):345–364, 1975b; Balder, On compactness of the space of policies in stochastic dynamic programming. Stoch Processes Appl 32(1):141–150, 1989). We use the compactness result from this paper to show the existence of optimal policies for countable-state constrained optimization of expected discounted and nonpositive rewards, when the optimality is considered within the class of nonrandomized policies. This paper also studies the convergence of a value-iteration algorithm for such constrained problems.     

Optimal control problems on manifolds: a dynamic programming approach

Dynamic programming identifies the value function of continuous time optimal control with a solution to the Hamilton-Jacobi equation, appropriately defined. This relationship in turn leads to sufficient conditions of global optimality, which have been widely used to confirm the optimality of putative minimisers. In continuous time optimal control, the dynamic programming methodology has been used for problems with state space a vector space. However there are many problems of interest in which it is necessary to regard the state space as a manifold. This paper extends dynamic programming to cover problems in which the state space is a general finite-dimensional C^∞ manifold. It shows that, also in a manifold setting, we can characterise the value function of a free time optimal control problem as a unique lower semicontinuous, lower bounded, generalised solution of the Hamilton-Jacobi equation. The application of these results is illustrated by the investigation of minimum time controllers for a rigid pendulum.     

Stability analysis of stochastic programs with second order dominance constraints

In this paper we present a stability analysis of a stochastic optimization problem with stochastic second order dominance constraints. We consider a perturbation of the underlying probability measure in the space of regular measures equipped with pseudometric discrepancy distance (Römisch in Stochastic Programming. Elsevier, Amsterdam, pp 483–554, 2003). By exploiting a result on error bounds in semi-infinite programming due to Gugat (Math Program Ser B 88:255–275, 2000), we show under the Slater constraint qualification that the optimal value function is Lipschitz continuous and the optimal solution set mapping is upper semicontinuous with respect to the perturbation of the probability measure. In particular, we consider the case when the probability measure is approximated by an empirical probability measure and show an exponential rate of convergence of the sequence of optimal solutions obtained from solving the approximation problem. The analysis is extended to the stationary points.     

Linearization Techniques for Controlled Piecewise Deterministic Markov Processes; Application to Zubov’s Method

We aim at characterizing domains of attraction for controlled piecewise deterministic processes using an occupational measure formulation and Zubov??s approach. Firstly, we provide linear programming (primal and dual) formulations of discounted, infinite horizon control problems for PDMPs. These formulations involve an infinite-dimensional set of probability measures and are obtained using viscosity solutions theory. Secondly, these tools allow to construct stabilizing measures and to avoid the assumption of stability under concatenation for controls. The domain of controllability is then characterized as some level set of a convenient solution of the associated Hamilton-Jacobi integral-differential equation. The theoretical results are applied to PDMPs associated to stochastic gene networks. Explicit computations are given for Cook??s model for gene expression.     

Generation of Degenerate Linear Programming Problems

We present a method for constructing linear programming problems with randomly generated data. Besides the number of variables and constraints, the dimensions of the primal and dual faces are given. We show that, for problems in which the constraint matrix is carelessly constructed with random entries, with probability one only one between primal degeneracy and dual degeneracy appears.     

Viability with Probabilistic Knowledge of Initial Condition, Application to Optimal Control

In this paper we provide an extension of the Viability and Invariance Theorems in the Wasserstein metric space of probability measures with finite quadratic moments in ? ^d for controlled systems of which the dynamic is bounded and Lipschitz. Then we characterize the viability and invariance kernels as the largest viability (resp. invariance) domains. As application of our result we consider an optimal control problem of Mayer type with lower semicontinuous cost function for the same controlled system with uncertainty on the initial state modeled by a probability measure. Following Frankowska, we prove using the epigraphical viability approach that the value function is the unique lower semicontinuous proximal episolution of a suitable Hamilton Jacobi equation.     

Canonical Duality Theory and Solutions to Constrained Nonconvex Quadratic Programming

This paper presents a perfect duality theory and a complete set of solutions to nonconvex quadratic programming problems subjected to inequality constraints. By use of the canonical dual transformation developed recently, a canonical dual problem is formulated, which is perfectly dual to the primal problem in the sense that they have the same set of KKT points. It is proved that the KKT points depend on the index of the Hessian matrix of the total cost function. The global and local extrema of the nonconvex quadratic function can be identified by the triality theory [11]. Results show that if the global extrema of the nonconvex quadratic function are located on the boundary of the primal feasible space, the dual solutions should be interior points of the dual feasible set, which can be solved by deterministic methods. Certain nonconvex quadratic programming problems in {\open {R}}^{n} can be converted into a dual problem with only one variable. It turns out that a complete set of solutions for quadratic programming over a sphere is obtained as a by-product. Several examples are illustrated.     

Lower bounds and duality in the optimal control of distributed systems

We develop a duality theory for problems of minimization of a convex functional on a convex set and apply this theory to optimal control problems with non-differentiable functional and state as well as control constraints. The dual problem is sometimes found to be simpler than the primal problem. The equivalence of the primal and the dual problem is established for problems in which the functional is strongly convex. A posteriori error are also given for such problems.     

Duality Theory in Fuzzy Linear Programming Problems with Fuzzy Coefficients

The concept of fuzzy scalar (inner) product that will be used in the fuzzy objective and inequality constraints of the fuzzy primal and dual linear programming problems with fuzzy coefficients is proposed in this paper. We also introduce a solution concept that is essentially similar to the notion of Pareto optimal solution in the multiobjective programming problems by imposing a partial ordering on the set of all fuzzy numbers. We then prove the weak and strong duality theorems for fuzzy linear programming problems with fuzzy coefficients.     

Differential sensitivity of solutions of convex constrained optimal control problems for discrete systems

A family of optimal control problems for discrete systems that depend on a real parameter is considered. The problems are strongly convex and subject to state and control constraints. Some regularity conditions are imposed on the constraints.The control problems are reformulated as mathematical programming problems. It is shown that both the primal and dual optimal variables for these problems are right-differentiable functions of a parameter. The right-derivatives are characterized as solutions to auxiliary quadratic control problems. Conditions of continuous differentiability are discussed, and some estimates of the rate of convergence of the difference quotients to the respective derivatives are given.     

Preconditioning and globalizing conjugate gradients in dual space for quadratically penalized nonlinear-least squares problems

When solving nonlinear least-squares problems, it is often useful to regularize the problem using a quadratic term, a practice which is especially common in applications arising in inverse calculations. A solution method derived from a trust-region Gauss-Newton algorithm is analyzed for such applications, where, contrary to the standard algorithm, the least-squares subproblem solved at each iteration of the method is rewritten as a quadratic minimization subject to linear equality constraints. This allows the exploitation of duality properties of the associated linearized problems. This paper considers a recent conjugate-gradient-like method which performs the quadratic minimization in the dual space and produces, in exact arithmetic, the same iterates as those produced by a standard conjugate-gradients method in the primal space. This dual algorithm is computationally interesting whenever the dimension of the dual space is significantly smaller than that of the primal space, yielding gains in terms of both memory usage and computational cost. The relation between this dual space solver and PSAS (Physical-space Statistical Analysis System), another well-known dual space technique used in data assimilation problems, is explained. The use of an effective preconditioning technique is proposed and refined convergence bounds derived, which results in a practical solution method. Finally, stopping rules adequate for a trust-region solver are proposed in the dual space, providing iterates that are equivalent to those obtained with a Steihaug-Toint truncated conjugate-gradient method in the primal space.     

Primal-dual proximal point algorithm for linearly constrained convex programming problems

A primal-dual version of the proximal point algorithm is developed for linearly constrained convex programming problems. The algorithm is an iterative method to find a saddle point of the Lagrangian of the problem. At each iteration of the algorithm, we compute an approximate saddle point of the Lagrangian function augmented by quadratic proximal terms of both primal and dual variables. Specifically, we first minimize the function with respect to the primal variables and then approximately maximize the resulting function of the dual variables. The merit of this approach exists in the fact that the latter function is differentiable and the maximization of this function is subject to no constraints. We discuss convergence properties of the algorithm and report some numerical results for network flow problems with separable quadratic costs.     

Using Efficient Anchoring Points for Generating Search Directions in Interior Multiobjective Linear Programming

This paper modifies the affine-scaling primal algorithm to multiobjective linear programming (MOLP) problems. The modification is based on generating search directions in the form of projected gradients augmented by search directions pointing toward what we refer to as anchoring points. These anchoring points are located on the boundary of the feasible region and, together with the current, interior, iterate, define a cone in which we make the next step towards a solution of the MOLP problem. These anchoring points can be generated in more than one way. In this paper we present an approach that generates efficient anchoring points where the choice of termination solution available to the decision maker at each iteration consists of a set of efficient solutions. This set of efficient solutions is being updated during the iterative process so that only the most preferred solutions are retained for future considerations. Current MOLP algorithms are simplex-based and make their progress toward the optimal solution by following an exterior trajectory along the vertices of the constraints polytope. Since the proposed algorithm makes its progress through the interior of the constraints polytope, there is no need for vertex information and, therefore, the search for an acceptable solution may prove less sensitive to problem size. We refer to the resulting class of MOLP algorithms that are based on the affine-scaling primal algorithm as affine-scaling interior multiobjective linear programming (ASIMOLP) algorithms.     

Dynamic Lagrangian dual and reduced RLT constructs for solving 0–1 mixed-integer programs

In this paper, we consider a dynamic Lagrangian dual optimization procedure for solving mixed-integer 0–1 linear programming problems. Similarly to delayed relax-and-cut approaches, the procedure dynamically appends valid inequalities to the linear programming relaxation as induced by the Reformulation-Linearization Technique (RLT). A Lagrangian dual algorithm that is augmented with a primal solution recovery scheme is applied implicitly to a full or partial first-level RLT relaxation, where RLT constraints that are currently being violated by the primal estimate are dynamically generated within the Lagrangian dual problem, thus controlling the size of the dual space while effectively capturing the strength of the RLT-enhanced relaxation. We present a preliminary computational study to demonstrate the efficacy of this approach.     

State aggregation in dynamic programming — An application to scheduling of independent jobs on parallel processors

Exact dynamic programming formulations of capacity loading problems will, in general, involve a prohibitively large state space. This paper offers a methodology for aggregation of the state space when dealing with such problems. Instead of future costs with respect to a known state, we are considering the costs in the worst (or best) case with respect to the given aggregate information. We apply the technique to scheduling of independent jobs on parallel processors.     

Optimality and duality in nonsmooth multiobjective fractional programming problem with constraints

In this paper, we establish some quotient calculus rules in terms of contingent derivatives for the two extended-real-valued functions defined on a Banach space and study a nonsmooth multiobjective fractional programming problem with set, generalized inequality and equality constraints. We define a new parametric problem associated with these problem and introduce some concepts for the (local) weak minimizers to such problems. Some primal and dual necessary optimality conditions in terms of contingent derivatives for the local weak minimizers are provided. Under suitable assumptions, sufficient optimality conditions for the local weak minimizers which are very close to necessary optimality conditions are obtained. An application of the result for establishing three parametric, Mond–Weir and Wolfe dual problems and several various duality theorems for the same is presented. Some examples are also given for our findings.

     

First passage Markov decision processes with constraints and varying discount factors

This paper focuses on the constrained optimality problem (COP) of first passage discrete-time Markov decision processes (DTMDPs) in denumerable state and compact Borel action spaces with multi-constraints, state-dependent discount factors, and possibly unbounded costs. By means of the properties of a so-called occupation measure of a policy, we show that the constrained optimality problem is equivalent to an (infinite-dimensional) linear programming on the set of occupation measures with some constraints, and thus prove the existence of an optimal policy under suitable conditions. Furthermore, using the equivalence between the constrained optimality problem and the linear programming, we obtain an exact form of an optimal policy for the case of finite states and actions. Finally, as an example, a controlled queueing system is given to illustrate our results.     

Existence of Neighboring Feasible Trajectories: Applications to Dynamic Programming for State-Constrained Optimal Control Problems

In this paper, the value function for an optimal control problem with endpoint and state constraints is characterized as the unique lower semicontinuous generalized solution of the Hamilton-Jacobi equation. This is achieved under a constraint qualification (CQ) concerning the interaction of the state and dynamic constraints. The novelty of the results reported here is partly the nature of (CQ) and partly the proof techniques employed, which are based on new estimates of the distance of the set of state trajectories satisfying a state constraint from a given trajectory which violates the constraint.     

Generalized Lagrangian duality for nonconvex polynomial programs with polynomial multipliers

In this paper, under the existence of a certificate of nonnegativity of the objective function over the given constraint set, we present saddle-point global optimality conditions and a generalized Lagrangian duality theorem for (not necessarily convex) polynomial optimization problems, where the Lagrange multipliers are polynomials. We show that the nonnegativity certificate together with the archimedean condition guarantees that the values of the Lasserre hierarchy of semidefinite programming (SDP) relaxations of the primal polynomial problem converge asymptotically to the common primal–dual value. We then show that the known regularity conditions that guarantee finite convergence of the Lasserre hierarchy also ensure that the nonnegativity certificate holds and the values of the SDP relaxations converge finitely to the common primal–dual value. Finally, we provide classes of nonconvex polynomial optimization problems for which the Slater condition guarantees the required nonnegativity certificate and the common primal–dual value with constant multipliers and the dual problems can be reformulated as semidefinite programs. These classes include some separable polynomial programs and quadratic optimization problems with quadratic constraints that admit certain hidden convexity. We also give several numerical examples that illustrate our results.