On optimal probabilities in stochastic coordinate descent methods

We propose and analyze a new parallel coordinate descent method—NSync—in which at each iteration a random subset of coordinates is updated, in parallel, allowing for the subsets to be chosen using an arbitrary probability law. This is the first method of this type. We derive convergence rates under a strong convexity assumption, and comment on how to assign probabilities to the sets to optimize the bound. The complexity and practical performance of the method can outperform its uniform variant by an order of magnitude. Surprisingly, the strategy of updating a single randomly selected coordinate per iteration—with optimal probabilities—may require less iterations, both in theory and practice, than the strategy of updating all coordinates at every iteration.     

The variational principle and stochastic optimal control

Using a non-convex minimization result of Ekeland it is shown that an “almost optimal” control almost surely minimizes the conditional expectation of the Hamiltonian of the stochastic system, when the expectation is taken with respect to the observed ff-field     

Methods of uniform optimal stochastic control

Translated from Upravlenie Nelineinymi Sistemami, Vsesoyuznyi Nauchno-Issledovatel'skii Institut Sistemnykh Issledovanii, Sbornik Trudov, No. 4, pp. 107–116, 1991.     

Upper bounds for ruin probabilities under stochastic interest rate and optimal investment strategies

In this paper, we study the upper bounds for ruin probabilities of an insurance company which invests its wealth in a stock and a bond. We assume that the interest rate of the bond is stochastic and it is described by a Cox-Ingersoll-Ross (CIR) model. For the stock price process, we consider both the case of constant volatility (driven by an O-U process) and the case of stochastic volatility (driven by a CIR model). In each case, under certain conditions, we obtain the minimal upper bound for ruin probability as well as the corresponding optimal investment strategy by a pure probabilistic method.     

Dynamic consistency for stochastic optimal control problems

For a sequence of dynamic optimization problems, we aim at discussing a notion of consistency over time. This notion can be informally introduced as follows. At the very first time step?t ₀, the decision maker formulates an optimization problem that yields optimal decision rules for all the forthcoming time steps?t ₀,t ₁,??,T; at the next time step?t ₁, he is able to formulate a new optimization problem starting at time?t ₁ that yields a new sequence of optimal decision rules. This process can be continued until the final time?T is reached. A?family of optimization problems formulated in this way is said to be dynamically consistent if the optimal strategies obtained when solving the original problem remain optimal for all subsequent problems. The notion of dynamic consistency, well-known in the field of economics, has been recently introduced in the context of risk measures, notably by Artzner et al. (Ann. Oper. Res. 152(1):5?C22, 2007) and studied in the stochastic programming framework by Shapiro (Oper. Res. Lett. 37(3):143?C147, 2009) and for Markov Decision Processes (MDP) by Ruszczynski (Math. Program. 125(2):235?C261, 2010). We here link this notion with the concept of ??state variable?? in MDP, and show that a significant class of dynamic optimization problems are dynamically consistent, provided that an adequate state variable is chosen.     

Robust optimal control of stochastic hyperelastic materials

Soft robots are highly nonlinear systems made of deformable materials such as elastomers, fluids and other soft matter, that often exhibit intrinsic uncertainty in their elastic responses under large strains due to microstructural inhomogeneity. These sources of uncertainty might cause a change in the dynamics of the system leading to a significant degree of complexity in its controllability. This issue poses theoretical and numerical challenges in the emerging field of optimal control of stochastic hyperelasticity. This paper states and solves the robust averaged control in stochastic hyperelasticity where the underlying state system corresponds to the minimization of a stochastic polyconvex strain energy function. Two bio-inspired optimal control problems under material uncertainty are addressed. The expected value of the L²-norm to a given target configuration is minimized to reduce the sensitivity of the spatial configuration to variations in the material parameters. The existence of optimal solutions for the robust averaged control problem is proved. Then the problem is solved numerically by using a gradient-based method. Two numerical experiments illustrate both the performance of the proposed method to ensure the robustness of the system and the significant differences that may occur when uncertainty is incorporated in this type of control problems.     

Particle methods for stochastic optimal control problems

When dealing with numerical solution of stochastic optimal control problems, stochastic dynamic programming is the natural framework. In order to try to overcome the so-called curse of dimensionality, the stochastic programming school promoted another approach based on scenario trees which can be seen as the combination of Monte Carlo sampling ideas on the one hand, and of a heuristic technique to handle causality (or nonanticipativeness) constraints on the other hand. However, if one considers that the solution of a stochastic optimal control problem is a feedback law which relates control to state variables, the numerical resolution of the optimization problem over a scenario tree should be completed by a feedback synthesis stage in which, at each time step of the scenario tree, control values at nodes are plotted against corresponding state values to provide a first discrete shape of this feedback law from which a continuous function can be finally inferred. From this point of view, the scenario tree approach faces an important difficulty: at the first time stages (close to the tree root), there are a few nodes (or Monte-Carlo particles), and therefore a relatively scarce amount of information to guess a feedback law, but this information is generally of a good quality (that is, viewed as a set of control value estimates for some particular state values, it has a small variance because the future of those nodes is rich enough); on the contrary, at the final time stages (near the tree leaves), the number of nodes increases but the variance gets large because the future of each node gets poor (and sometimes even deterministic). After this dilemma has been confirmed by numerical experiments, we have tried to derive new variational approaches. First of all, two different formulations of the essential constraint of nonanticipativeness are considered: one is called algebraic and the other one is called functional. Next, in both settings, we obtain optimality conditions for the corresponding optimal control problem. For the numerical resolution of those optimality conditions, an adaptive mesh discretization method is used in the state space in order to provide information for feedback synthesis. This mesh is naturally derived from a bunch of sample noise trajectories which need not to be put into the form of a tree prior to numerical resolution. In particular, an important consequence of this discrepancy with the scenario tree approach is that the same number of nodes (or points) are available from the beginning to the end of the time horizon. And this will be obtained without sacrifying the quality of the results (that is, the variance of the estimates). Results of experiments with a hydro-electric dam production management problem will be presented and will demonstrate the claimed improvements. A more realistic problem will also be presented in order to demonstrate the effectiveness of the method for high dimensional problems.     

Resource allocation with stochastic optimal control approach

A control-theoretic decision making system is proposed for an agent (decision maker) to “optimally” allocate and deploy his/her resources over time among a dynamically changing list of opportunities (e.g., financial assets), in an uncertain market environment. The solution is a sequence of actions with the objective of optimizing total reward function. This control-theoretic approach is unique in a sense that it solves the problem at distinct time epochs over a finite time horizon and strategies are discovered directly. Rather than basing a decision making system on forecasts or training via a reinforcement learning algorithm using current state data, we train our system via a Q-learning algorithm using Geometric Brownian Motion as an asset price function. While the above problem is quite general, we focus solely on the problem of dynamic financial portfolio management with the objective of maximizing the expected utility for a given risk level. The performance functions that we consider for our system are realized mean return, drawdown and standard deviation. We find that our model achieves a better return and drawdown compared to a known market index as a benchmark.     

Backward stochastic differential equations and applications to optimal control

We study the existence and uniqueness of the following kind of backward stochastic differential equation, $$x(t) + \int_t^T {f(x(s),y(s),s)ds + \int_t^T {y(s)dW(s) = X,} }$$ under local Lipschitz condition, where (Ω, ?,P, W(·), ?_t) is a standard Wiener process, for any given (x, y),f(x, y, ·) is an ?_t-adapted process, andX is ?_t-measurable. The problem is to look for an adapted pair (x(·),y(·)) that solves the above equation. A generalized matrix Riccati equation of that type is also investigated. A new form of stochastic maximum principle is obtained.     

Stability analysis and optimal control of stochastic singular systems

In this paper, problems of stability and optimal control for a class of stochastic singular systems are studied. Firstly, under some appropriate assumptions, some new results about mean-square admissibility are developed and the corresponding LMI sufficient condition is given. Secondly, finite-time horizon and infinite-time horizon linear quadratic (LQ) control problems for the stochastic singular system are investigated, in which the coefficients are allowed to be random in control input and quadratic criterion. Some results involving new stochastic generalized Riccati equation are discussed as well. Finally, the proposed LQ control model for stochastic singular systems provides an appropriate and effective framework to study the portfolio selection problem in light of the recent development on general stochastic LQ problems.     

Probabilistic inequality constraints in stochastic optimal control theory

There are many optimal control problems in which it is necessary or desirable to constrain the range of values of state variables. When stochastic inputs are involved, these inequality constraint problems are particularly difficult. Sometimes the constraints must be modeled as hard constraints which can never be violated, and other times it is more natural to prescribe a probability that the constraints will not be violated. This paper treats general problems of the latter type, in which probabilistic inequality constraints are imposed on the state variables or on combinations of state and control variables. A related class of problems in which the state is required to reach a target set with a prescribed probability is handled by the same methods. It is shown that the solutions to these problems can be obtained by solving a comparatively simple bilinear deterministic control problem.     

Duality theorem for the stochastic optimal control problem

We prove a duality theorem for the stochastic optimal control problem with a convex cost function and show that the minimizer satisfies a class of forward–backward stochastic differential equations. As an application, we give an approach, from the duality theorem, to h$h$-path processes for diffusion processes.     

An inverse optimal problem in discrete-time stochastic control

In this paper, we study an inverse optimal problem in discrete-time stochastic control. We give necessary and sufficient conditions for a solution to a system of stochastic difference equations to be the solution of a certain optimal control problem. Our results extend to the stochastic case the work of Dechert. In particular, we present a stochastic version of an important principle in welfare economics.     

Application of quantum stochastic calculus to optimal control

Given a plant whose output is described by a Hudson-Parthasarathy quantum stochastic differential equation [1–3] driven by standard quantum Brownian motion, we compute explicitly the control process that rapidly makes the size of the plant's output small, and keeps the energy used at a minimum. The solution to the quantum stochastic analogue of the linear regulator problem of classical stochastic control theory ([4], [5]) follows as a special case.Published in Matematicheskie Zametki, Vol. 53, No. 5, pp. 48–56, May, 1993.     

Necessary conditions for optimal singular stochastic control problems

In this paper, necessary conditions of optimality, in the form of a maximum principle, are obtained for singular stochastic control problems. This maximum principle is derived for a state process satisfying a general stochastic differential equation where the coefficient associated to the control process can be dependent on the state, extending earlier results of the literature.     

Indefinite stochastic linear-quadratic optimal control problems with random jumps and related stochastic Riccati equations

We discuss the stochastic linear-quadratic (LQ) optimal control problem with Poisson processes under the indefinite case. Based on the wellposedness of the LQ problem, the main idea is expressed by the definition of relax compensator that extends the stochastic Hamiltonian system and stochastic Riccati equation with Poisson processes (SREP) from the positive definite case to the indefinite case. We mainly study the existence and uniqueness of the solution for the stochastic Hamiltonian system and obtain the optimal control with open-loop form. Then, we further investigate the existence and uniqueness of the solution for SREP in some special case and obtain the optimal control in close-loop form.     

On fuzzy goals and maximizing decisions in stochastic optimal control

Fuzzy set theory has developed significantly in a mathematical direction during the past several years but few applications have emerged. This paper investigates the role of fuzzy set theory in certain optimal control formulations. In particular, it is shown that the well-known quadratic performance criterion in deterministic optimal control is equivalent to the exponential membership function of a certain fuzzy decision (set). In a stochastic setting, similar equivalences establish new definitions for “confluence of goals” and “maximizing decision” in fuzzy set theory. These and other definitions could lead to the development of a more applicable theory of fuzzy sets.     

Mayer and optimal stopping stochastic control problems with discontinuous cost

We study two classes of stochastic control problems with semicontinuous cost: the Mayer problem and optimal stopping for controlled diffusions. The value functions are introduced via linear optimization problems on appropriate sets of probability measures. These sets of constraints are described deterministically with respect to the coefficient functions. Both the lower and upper semicontinuous cases are considered. The value function is shown to be a generalized viscosity solution of the associated HJB system, respectively, of some variational inequality. Dual formulations are given, as well as the relations between the primal and dual value functions. Under classical convexity assumptions, we prove the equivalence between the linearized Mayer problem and the standard weak control formulation. Counter-examples are given for the general framework.