On the discrete adjoints of adaptive time stepping algorithms

We investigate the behavior of adaptive time stepping numerical algorithms under the reverse mode of automatic differentiation (AD). By differentiating the time step controller and the error estimator of the original algorithm, reverse mode AD generates spurious adjoint derivatives of the time steps. The resulting discrete adjoint models become inconsistent with the adjoint ODE, and yield incorrect derivatives. To regain consistency, one has to cancel out the contributions of the non-physical derivatives in the discrete adjoint model. We demonstrate that the discrete adjoint models of one-step, explicit adaptive algorithms, such as the Runge–Kutta schemes, can be made consistent with their continuous counterparts using simple code modifications. Furthermore, we extend the analysis to cover second order adjoint models derived through an extra forward mode differentiation of the discrete adjoint code. Several numerical examples support the mathematical derivations.     

Approximation of weak adjoints by reverse automatic differentiation of BDF methods

We shed light on the relation between the discrete adjoints of multistep backward differentiation formula (BDF) methods and the solution of the adjoint differential equation. To this end, we develop a functional-analytic framework based on a constrained variational problem and introduce the notion of weak adjoint solutions of ordinary differential equations. We devise a Petrov-Galerkin finite element (FE) interpretation of the BDF method and its discrete adjoint scheme obtained by reverse internal numerical differentiation. We show how the FE approximation of the weak adjoint is computed by the discrete adjoint scheme and prove its convergence in the space of normalized functions of bounded variation. We also show convergence of the discrete adjoints to the classical adjoints on the inner time interval. Finally, we give numerical results for non-adaptive and fully adaptive BDF schemes. The presented framework opens the way to carry over techniques on global error estimation from FE methods to BDF methods.     

NUMERICAL RECONSTRUCTION OF INTERNAL STATES BY BOUNDARY MEASUREMENTS FOR THE LAPLACE EQUATION

The purpose of the paper is to present a unified numerical method for problems consisting of the conventional boundary value problem. Cauchy problem, under-determined problem, and over-determined problem. The method is based on the direct variational approach, which paraphrases the problems into the primary and adjoint boundary value problems that can be tackled by commonly used computer programs for the numerical solution of the Laplace equation.     

Adjoint concepts for the optimal control of Burgers equation

Adjoint techniques are a common tool in the numerical treatment of optimal control problems. They are used for efficient evaluations of the gradient of the objective in gradient-based optimization algorithms. Different adjoint techniques for the optimal control of Burgers equation with Neumann boundary control are studied. The methods differ in the point in the numerical algorithm at which the adjoints are incorporated. Discretization methods for the continuous adjoint are discussed and compared with methods applying the application of the discrete adjoint. At the example of the implicit Euler method and the Crank Nicolson method it is shown that a discretization for the adjoint problem that is adjoint to the discretized optimal control problem avoids additional errors in gradient-based optimization algorithms. The approach of discrete adjoints coincides with that of automatic differentiation tools (AD) which provide exact gradient calculations on the discrete level.     

Determination of space‐dependent coefficients from temperature measurements using the conjugate gradient method

In this article, we consider coefficient identification problems in heat transfer concerned with the determination of the space‐dependent perfusion coefficient and/or thermal conductivity from interior temperature measurements using the conjugate gradient method (CGM). We establish the direct, sensitivity and adjoint problems and the iterative CGM algorithm which has to be stopped according to the discrepancy principle in order to reconstruct a stable solution for the inverse problem. The Sobolev gradient concept is introduced in the CGM iterative algorithm in order to improve the reconstructions. The numerical results illustrated for both exact and noisy data, in one‐ and two‐dimensions for single or double coefficient identifications show that the CGM is an efficient and stable method of inversion.     

Variational methods of constructing monotone approximations for atmospheric chemistry models

A new method of constructing efficient monotone numerical schemes for solving direct, adjoint, and inverse atmospheric chemistry problems is presented. It is a synthesis of variational principles combined with splitting and decomposition methods and a constructive implementation of Euler integrating multipliers (EIM) bymeans of a local adjoint problem technique. To increase the efficiency of calculations, a method of decomposing the multicomponent substance transformation operators in terms of the mechanisms of reactions is also proposed. With analytical EIMs, the systems of stiff ODEs are decomposed and reduced to equivalent systems of integral equations solved by noniterative multistage algorithms of a given order of accuracy. An unconventional variational method of constructing mutually consistent algorithms for direct and adjoint problems and sensitivity studies for complex models with constraints is described.     

Evaluating Gradients in Optimal Control: Continuous Adjoints Versus Automatic Differentiation

This paper deals with the numerical solution of optimal control problems for ODEs. The methods considered here rely on some standard optimization code to solve a discretized version of the control problem under consideration. We aim to make available to the optimization software not only the discrete objective functional, but also its gradient. The objective gradient can be computed either from forward (sensitivity) information or backward (adjoint) information. The purpose of this paper is to discuss various ways of adjoint computation. It will be shown both theoretically and numerically that methods based on the continuous adjoint equation require a careful choice of both the integrator and gradient assembly formulas in order to obtain a gradient consistent with the discretized control problem. Particular attention is given to automatic differentiation techniques which generate automatically a suitable integrator.     

Optimization problems for nonstationary wave processes

We give a general formulation of the optimization problem for nonstationary hyperbolic systems. Gradient algorithms are used for a directed numerical search. The adjoint problem is obtained in general form in order to compute the gradient. We prove that the types and characteristics of the direct and adjoint problems are the same. We recommend the use of identical total count difference schemes to solve both problems. Translated fromTeoreticheskaya i Prikladnaya Mekhanika, No. 24, 1993, pp. 89–93.     

Approximate Gradient Projection Method with Runge-Kutta Schemes for Optimal Control Problems

We consider an optimal control problem for systems governed by ordinary differential equations with control constraints. The state equation is discretized by the explicit fourth order Runge-Kutta scheme and the controls are approximated by discontinuous piecewise affine ones. We then propose an approximate gradient projection method that generates sequences of discrete controls and progressively refines the discretization during the iterations. Instead of using the exact discrete directional derivative, which is difficult to calculate, we use an approximate derivative of the cost functional defined by discretizing the continuous adjoint equation by the same Runge-Kutta scheme and the integral involved by Simpson's integration rule, both involving intermediate approximations. The main result is that accumulation points, if they exist, of sequences constructed by this method satisfy the weak necessary conditions for optimality for the continuous problem. Finally, numerical examples are given.     

Two-dimensional pursuit-evasion game with penalty on turning rates

An approach, both analytical and numerical, is used to solve a two-dimensional pursuit-evasion game characterized by a difficulty level intermediate between that of thesimple motion game (with freely and instantaneously oriented velocities) and that of thegame of two cars (with lower bounds on curvature radii). Each player's velocity has a constant modulus. The maneuvers are penalized by introducing, in the performance index, an integral term for the squared velocity turning rate.The local problem solution is relatively easy to find: the equations of motion and the adjoint equations can be integrated by means of elliptic functions and integrals. The global problem is more delicate to solve, because of the existence of a dispersal singular surface requiring an important numerical search to be determined. Thesynthesis problem (how to express the optimal strategies as functions of state) is not explicitly solvable, but a numerical approach using successive approximations can be developed. Illustrative interception trajectories are given.The authors are grateful to Mr. J. P. Peltier, Head, Guidance Group, Aerospace Mechanics Division, Systems Department, ONERA, Châtillon, France, for his suggestions and his efficient assistance in the numerical aspect of this study.     

The differentiation-enabled NAGWare Fortran compiler

We present a research prototype of the differentiation-enabled NAGWare Fortran compiler. The compiler provides built-in automatic differentiation capabilities for the generation of code for computing first and second derivatives of numerical simulation codes written in Fortran. Tangent-linear, adjoint, and second-order adjoint code is obtained by a semantic transformation of the compiler's intermediate representation. Three successful reference applications are discussed. (© 2008 WILEY-VCH Verlag GmbH & Co. KGaA, Weinheim)     

The Adjoint Newton Algorithm for Large-Scale Unconstrained Optimization in Meteorology Applications

A new algorithm is presented for carrying out large-scale unconstrained optimization required in variational data assimilation using the Newton method. The algorithm is referred to as the adjoint Newton algorithm. The adjoint Newton algorithm is based on the first- and second-order adjoint techniques allowing us to obtain the Newton line search direction by integrating a tangent linear equations model backwards in time (starting from a final condition with negative time steps). The error present in approximating the Hessian (the matrix of second-order derivatives) of the cost function with respect to the control variables in the quasi-Newton type algorithm is thus completely eliminated, while the storage problem related to the Hessian no longer exists since the explicit Hessian is not required in this algorithm. The adjoint Newton algorithm is applied to three one-dimensional models and to a two-dimensional limited-area shallow water equations model with both model generated and First Global Geophysical Experiment data. We compare the performance of the adjoint Newton algorithm with that of truncated Newton, adjoint truncated Newton, and LBFGS methods. Our numerical tests indicate that the adjoint Newton algorithm is very efficient and could find the minima within three or four iterations for problems tested here. In the case of the two-dimensional shallow water equations model, the adjoint Newton algorithm improves upon the efficiencies of the truncated Newton and LBFGS methods by a factor of at least 14 in terms of the CPU time required to satisfy the same convergence criterion.The Newton, truncated Newton and LBFGS methods are general purpose unconstrained minimization methods. The adjoint Newton algorithm is only useful for optimal control problems where the model equations serve as strong constraints and their corresponding tangent linear model may be integrated backwards in time. When the backwards integration of the tangent linear model is ill-posed in the sense of Hadamard, the adjoint Newton algorithm may not work. Thus, the adjoint Newton algorithm must be used with some caution. A possible solution to avoid the current weakness of the adjoint Newton algorithm is proposed.     

A truncated Newton optimization algorithm in meteorology applications with analytic Hessian/vector products

A modified version of the truncated-Newton algorithm of Nash ([24], [25], [29]) is presented differing from it only in the use of an exact Hessian vector product for carrying out the large-scale unconstrained optimization required in variational data assimilation. The exact Hessian vector products is obtained by solving an optimal control problem of distributed parameters. (i.e. the system under study occupies a certain spatial and temporal domain and is modeled by partial differential equations) The algorithm is referred to as the adjoint truncated-Newton algorithm. The adjoint truncated-Newton algorithm is based on the first and the second order adjoint techniques allowing to obtain a better approximation to the Newton line search direction for the problem tested here. The adjoint truncated-Newton algorithm is applied here to a limited-area shallow water equations model with model generated data where the initial conditions serve as control variables. We compare the performance of the adjoint truncated-Newton algorithm with that of the original truncated-Newton method [29] and the LBFGS (Limited Memory BFGS) method of Liu and Nocedal [23]. Our numerical tests yield results which are twice as fast as these obtained by the truncated-Newton algorithm and are faster than the LBFGS method both in terms of number of iterations as well as in terms of CPU time.     

Implicit semi-direct methods based on root-free sparse factorization procedures

In this paper new extendable sparse symmetric factorisation procedures are presented for the solution of self adjoint elliptic partial differential equations. The derived iterative methods are shown to be both competitive and computationally efficient in comparison with existing schemes. The application of the methods to a linear and non-linear elliptic boundary value problem in 2 dimensions is discussed and numerical results given.     

Hybrid evolutionary algorithm with Hermite radial basis function interpolants for computationally expensive adjoint solvers

In this paper, we present an evolutionary algorithm hybridized with a gradient-based optimization technique in the spirit of Lamarckian learning for efficient design optimization. In order to expedite gradient search, we employ local surrogate models that approximate the outputs of a computationally expensive Euler solver. Our focus is on the case when an adjoint Euler solver is available for efficiently computing the sensitivities of the outputs with respect to the design variables. We propose the idea of using Hermite interpolation to construct gradient-enhanced radial basis function networks that incorporate sensitivity data provided by the adjoint Euler solver. Further, we conduct local search using a trust-region framework that interleaves gradient-enhanced surrogate models with the computationally expensive adjoint Euler solver. This ensures that the present hybrid evolutionary algorithm inherits the convergence properties of the classical trust-region approach. We present numerical results for airfoil aerodynamic design optimization problems to show that the proposed algorithm converges to good designs on a limited computational budget.     

Optimal control of constrained time lag systems: Necessary conditions and numerical treatment

We consider retarded optimal control problems with constant delays in state and control variables under mixed controlstate inequality constraints. First order necessary optimality conditions in the form of Pontryagin's minimum principle are presented and discussed as well as numerical methods based upon discretization techniques and nonlinear programming. The minimum principle for the considered problem class leads to a boundary value problem which is retarded in the state dynamics and advanced in the costate dynamics. It can be shown that the Lagrange multipliers associated with the programming problem provide a consistent discretization of the advanced adjoint equation for the delayed control problem. (© 2008 WILEY-VCH Verlag GmbH & Co. KGaA, Weinheim)     

Optimal boundary control of glass cooling processes

In this paper, an optimal control problem for glass cooling processes is studied. We model glass cooling using the SP1 approximations to the radiative heat transfer equations. The control variable is the temperature at the boundary of the domain. This results in a boundary control problem for a parabolic/elliptic system which is treated by a constrained optimization approach. We consider several cost functionals of tracking‐type and formally derive the first‐order optimality system. Several numerical methods based on the adjoint variables are investigated. We present results of numerical simulations illustrating the feasibility and performance of the different approaches. Copyright © 2004 John Wiley & Sons, Ltd.     

Type I operators and the approximation of singular two-point boundary value problems

A class of self adjoint operators associated with second order singular ordinary differential expressions, arises naturally when the problem is weakly formulated and integration by parts is performed. We call this class Type I operators. It turns out that this class can be successfully used to tackle numerical approximations of singular two-point boundary value problems. They can also be approximated by regular differential operators in a straightforward manner without having to bring the delicate structure of singular differential operators to the forefront of the investigation.     

Consistent Numerical Schemes for Solving Nonlinear Inverse Source Problems with Gradient-Type Algorithms and Newton–Kantorovich Methods

In this paper, algorithms of solving an inverse source problem for systems of production–destruction equations are considered. Numerical schemes that are consistent to satisfy Lagrange’s identity for solving direct and adjoint problems are constructed. With the help of adjoint equations, a sensitivity operator with a discrete analog is constructed. It links perturbations of the measured values with those of the sought-for model parameters. This operator transforms the inverse problem to a quasilinear system of equations and allows applying Newton–Kantorovich methods to it. A numerical comparison of gradient algorithms based on consistent and inconsistent numerical schemes and a Newton–Kantorovich algorithm applied to solving an inverse source problem for a nonlinear Lorenz model is done.     

An expanded mixed covolume method for elliptic problems

We consider the mixed covolume method combining with the expanded mixed element for a system of first‐order partial differential equations resulting from the mixed formulation of a general self‐adjoint elliptic problem with a full diffusion tensor. The system can be used to model the transport of a contaminant carried by a flow in porous media. We use the lowest order Raviart‐Thomas mixed element space. We show the first‐order error estimate for the approximate solution in L² norm. We show the superconvergence both for pressure and velocity in certain discrete norms. We also get a finite difference scheme by using proper approximate integration formulas. Finally we give some numerical examples. © 2004 Wiley Periodicals, Inc. Numer Methods Partial Differential Eq, 2005