Fast derivatives of likelihood functionals for ODE based models using adjoint-state method

We consider time series data modeled by ordinary differential equations (ODEs), widespread models in physics, chemistry, biology and science in general. The sensitivity analysis of such dynamical systems usually requires calculation of various derivatives with respect to the model parameters. We employ the adjoint state method (ASM) for efficient computation of the first and the second derivatives of likelihood functionals constrained by ODEs with respect to the parameters of the underlying ODE model. Essentially, the gradient can be computed with a cost (measured by model evaluations) that is independent of the number of the ODE model parameters and the Hessian with a linear cost in the number of the parameters instead of the quadratic one. The sensitivity analysis becomes feasible even if the parametric space is high-dimensional. The main contributions are derivation and rigorous analysis of the ASM in the statistical context, when the discrete data are coupled with the continuous ODE model. Further, we present a highly optimized implementation of the results and its benchmarks on a number of problems. The results are directly applicable in (e.g.) maximum-likelihood estimation or Bayesian sampling of ODE based statistical models, allowing for faster, more stable estimation of parameters of the underlying ODE model.     

Particle Approximations of the Score and Observed Information Matrix for Parameter Estimation in State–Space Models With Linear Computational Cost

Poyiadjis, Doucet, and Singh showed how particle methods can be used to estimate both the score and the observed information matrix for state–space models. These methods either suffer from a computational cost that is quadratic in the number of particles, or produce estimates whose variance increases quadratically with the amount of data. This article introduces an alternative approach for estimating these terms at a computational cost that is linear in the number of particles. The method is derived using a combination of kernel density estimation, to avoid the particle degeneracy that causes the quadratically increasing variance, and Rao–Blackwellization. Crucially, we show the method is robust to the choice of bandwidth within the kernel density estimation, as it has good asymptotic properties regardless of this choice. Our estimates of the score and observed information matrix can be used within both online and batch procedures for estimating parameters for state–space models. Empirical results show improved parameter estimates compared to existing methods at a significantly reduced computational cost. Supplementary materials including code are available.     

Bootstrap Algorithms for Risk Models with Auxiliary Variable and Complex Samples

Resampling methods are often invoked in risk modelling when the stability of estimators of model parameters has to be assessed. The accuracy of variance estimates is crucial since the operational risk management affects strategies, decisions and policies. However, auxiliary variables and the complexity of the sampling design are seldom taken into proper account in variance estimation. In this paper bootstrap algorithms for finite population sampling are proposed in presence of an auxiliary variable and of complex samples. Results from a simulation study exploring the empirical performance of some bootstrap algorithms are presented.      

New interpolants for asymptotically correct defect control of BVODEs

The defect of a continuous approximate solution to an ODE is the amount by which that approximation fails to satisfy the ODE. A number of studies have explored the use of asymptotically correct defect estimates in the numerical solution of initial value ODEs (IVODEs). By employing an appropriately constructed interpolant to an approximate discrete solution to the ODE, various researchers have shown that it is possible to obtain estimates of the local error and/or the maximum defect that are asymptotically correct on each step, as the stepsize h →0. In this paper, we investigate the usefulness of asymptotically correct defect estimates for defect control in boundary value ODE (BVODE) codes. In the BVODE context, for a sequence of meshes which partition the problem interval, one computes a discrete numerical solution, constructs an interpolant, and estimates the maximum defect. The estimates (typically obtained by sampling the defect at a small number of points on each subinterval of the mesh) are used in a redistribution process to determine the next mesh and thus the availability of these more reliable maximum defect estimates can lead to improved meshes. As well, when such estimates are available, the code can terminate with more confidence that the defect is bounded throughout the problem domain by the user-prescribed tolerance. In this paper we employ a boot-strapping approach to derive interpolants that allow asymptotically correct defect estimates. Numerical results are included to demonstrate the validity of this approach.     

Comparative study on order-reduced methods for linear third-order ordinary differential equations

The linear third-order ordinary differential equation (ODE) can be transformed into a system of two second-order ODEs by introducing a variable replacement, which is different from the common order-reduced approach. We choose the functions p(x) and q(x) in the variable replacement to get different cases of the special order-reduced system for the linear third-order ODE. We analyze the numerical behavior and algebraic properties of the systems of linear equations resulting from the sinc discretizations of these special second-order ODE systems. Then the block-diagonal preconditioner is used to accelerate the convergence of the Krylov subspace iteration methods for solving the discretized system of linear equation. Numerical results show that these order-reduced methods are effective for solving the linear third-order ODEs.     

Sparse regression Chebyshev polynomial interval method for nonlinear dynamic systems under uncertainty

This paper proposes a new higher-efficiency interval method for the response bound estimation of nonlinear dynamic systems, whose uncertain parameters are bounded. This proposed method uses sparse regression and Chebyshev polynomials to help the interval analysis applied on the estimation. It is also a non-intrusive method which needs much fewer evaluations of original nonlinear dynamic systems than the other Chebyshev polynomials based interval methods. By using the proposed method, the response bound estimation of nonlinear dynamic systems can be performed more easily, even if the numerical simulation in nonlinear dynamic systems is costly or the number of uncertain parameters is higher than usual. In our approach, the sparse regression method “elastic net” is adopted to improve the sampling efficiency, but with sufficient accuracy. It alleviates the sample size required in coefficient calculation of the Chebyshev inclusion function in the sampling based methods. Moreover, some mature technologies are adopted to further reduce the sample size and to guarantee the accuracy of the estimation. So that the number of sampling, which solves the certain ordinary differential equations (ODEs), can be reduced significantly in the Chebyshev interval method. Three numerical examples are presented to illustrate the efficiency of proposed interval method. In particular, the last two examples are high dimension uncertain problems, which can further exhibit the ability to reduce the computational cost.     

Quadratic Störmer‐type methods for the solution of the Boussinesq equation by the method of lines

We study the numerical treatment of Boussinesq PDE equation using the method of lines. For the space discretization, we choose either classical finite differences or Fourier pseudospectral methods. Both cases result in a system of second‐order ordinary differential equations (ODEs) that is quadratic. In order to take advantage of this special feature, we choose to solve the ODE system using a new type of hybrid Numerov method specially constructed for such problems. Other efficient ODE solvers taken from the literature are used to solve the system of ODEs as well. By taking all the combinations of space discretization methods and ODE solvers, we discuss the stability and accuracy features revealed from the numerical tests. © 2008 Wiley Periodicals, Inc. Numer Methods Partial Differential Eq 2008     

Taylor series method for dynamical systems with control: Convergence and error estimates

To optimize a complicated function constructed from a solution of a system of ordinary differential equations (ODEs), it is very important to be able to approximate a solution of a system of ODEs very precisely. The precision delivered by the standard Runge-Kutta methods often is insufficient, resulting in a “noisy function” to optimize. We consider an initial-value problem for a system of ordinary differential equations having polynomial right-hand sides with respect to all dependent variables. First we show how to reduce a wide class of ODEs to such polynomial systems. Using the estimates for the Taylor series method, we construct a new “aggregative” Taylor series method and derive guaranteed a priori step-size and error estimates for Runge-Kutta methods of order r. Then we compare the 8,13-Prince-Dormand’s, Taylor series, and aggregative Taylor series methods using seven benchmark systems of equations, including van der Pol’s equations, the “brusselator,” equations of Jacobi’s elliptic functions, and linear and nonlinear stiff systems of equations. The numerical experiments show that the Taylor series method achieves the best precision, while the aggregative Taylor series method achieves the best computational time. The final section of this paper is devoted to a comparative study of the above numerical integration methods for systems of ODEs describing the optimal flight of a spacecraft from the Earth to the Moon. __________ Translated from Sovremennaya Matematika i Ee Prilozheniya (Contemporary Mathematics and Its Applications), Vol. 24, Dynamical Systems and Optimization, 2005.     

An optimization approach for the numerical approximation of differential equations

An alternative approach for the numerical approximation of ODEs is presented in this article. It is based on a variational framework recently introduced in S. Amat and P. Pedregal [A variational approach to implicit ODEs and differential inclusions, ESAIM: COCV 15 (2009), 149–172] where the solution is sought as the minimizer of an error functional tailored after the ODE in a rather straightforward way. A suitable discretization of this error functional is pursued, and it is performed using Hermite's interpolation and quadrature formulae. Notice that only Hermite's interpolation is necessary when polynomial systems of ODEs are considered (many models in practice use these types of equations). A comparison with implicit Runge–Kutta methods is analysed. With this variational strategy not only some classical collocation methods, but also new schemes that seem to have better numerical behaviour can be recovered. Although the driving idea is very simple, the strategy turns out to be very general and flexible. At the same time, it can be implemented efficiently.     

A preprocessing method for parameter estimation in ordinary differential equations

Parameter estimation for nonlinear differential equations is notoriously difficult because of poor or even no convergence of the nonlinear fit algorithm due to the lack of appropriate initial parameter values. This paper presents a method to gather such initial values by a simple estimation procedure. The method first determines the tangent slope and coordinates for a given solution of the ordinary differential equation (ODE) at randomly selected points in time. With these values the ODE is transformed into a system of equations, which is linear for linear appearance of the parameters in the ODE. For numerically generated data of the Lorenz attractor good estimates are obtained even at large noise levels. The method can be generalized to nonlinear parameter dependency. This case is illustrated using numerical data for a biological example. The typical problems of the method as well as their possible mitigation are discussed. Since a rigorous failure criterion of the method is missing, its results must be checked with a nonlinear fit algorithm. Therefore the method may serve as a preprocessing algorithm for nonlinear parameter fit algorithms. It can improve the convergence of the fit by providing initial parameter estimates close to optimal ones.     

Optimization based on quasi-Monte Carlo sampling to design state estimators for non-linear systems

State estimation for a class of non-linear, continuous-time dynamic systems affected by disturbances is investigated. The estimator is assigned a given structure that depends on an innovation function taking on the form of a ridge computational model, with some parameters to be optimized. The behaviour of the estimation error is analysed by using input-to-state stability. The design of the estimator is reduced to the determination of the parameters in such a way as to guarantee the regional exponential stability of the estimation error in a disturbance-free setting and to minimize a cost function that measures the effectiveness of the estimation when the system is affected by disturbances. Stability is achieved by constraining the derivative of a Lyapunov function to be negative definite on a grid of points, via the penalization of the constraints that are not satisfied. Low-discrepancy sampling techniques, typical of quasi-Monte Carlo methods, are exploited in order to reduce the computational burden in finding the optimal parameters of the innovation function. Simulation results are presented to investigate the performance of the estimator in comparison with the extended Kalman filter and in dependence of the complexity of the computational model and the sampling coarseness.     

线性常微分方程的全局误差估计和优化求解方法

线性常微分方程初值问题求解在许多应用中起着重要作用.目前,已存在很多的数值方法和求解器用于计算离散网格点上的近似解,但很少有对全局误差(global error)进行估计和优化的方法.本文首先通过将离散数值解插值成为可微函数用来定义方程的残差;再给出残差与近似解的关系定理并推导出全局误差的上界;然后以最小化残差的二范数为目标将方程求解问题转化为优化求解问题;最后通过分析导出矩阵的结构,提出利用共轭梯度法对其进行求解.之后将该方法应用于滤波电路和汽车悬架系统等实际问题.实验分析表明,本文估计方法对线性常微分方程的初值问题的全局误差具有比较好的估计效果,优化求解方法能够在不增加网格点的情形下求解出线性常微分方程在插值解空间中的全局最优解.     

An efficient time-step-based self-adaptive algorithm for predictor-corrector methods of Runge-Kutta type

Finding an efficient implementation variant for the numerical solution of problems from computational science and engineering involves many implementation decisions that are strongly influenced by the specific hardware architecture. The complexity of these architectures makes it difficult to find the best implementation variant by manual tuning. For numerical solution methods from linear algebra, auto-tuning techniques based on a global search engine as they are used for ATLAS or FFTW can be used successfully. These techniques generate different implementation variants at installation time and select one of these implementation variants either at installation time or at runtime, before the computation starts. For some numerical methods, auto-tuning at installation time cannot be applied directly, since the best implementation variant may strongly depend on the specific numerical problem to be solved. An example is solution methods for initial value problems (IVPs) of ordinary differential equations (ODEs), where the coupling structure of the ODE system to be solved has a large influence on the efficient use of the memory hierarchy of the hardware architecture. In this context, it is important to use auto-tuning techniques at runtime, which is possible because of the time-stepping nature of ODE solvers.In this article, we present a sequential self-adaptive ODE solver that selects the best implementation variant from a candidate pool at runtime during the first time steps, i.e., the auto-tuning phase already contributes to the progress of the computation. The implementation variants differ in the loop structure and the data structures used to realize the numerical algorithm, a predictor-corrector (PC) iteration scheme with Runge-Kutta (RK) corrector considered here as an example. For those implementation variants in the candidate pool that use loop tiling to exploit the memory hierarchy of a given hardware platform we investigate the selection of tile sizes. The self-adaptive ODE solver combines empirical search with a model-based approach in order to reduce the search space of possible tile sizes. Runtime experiments demonstrate the efficiency of the self-adaptive solver for different IVPs across a range of problem sizes and on different hardware architectures.     

Exact and analytical solutions for a nonlinear sigma model

We consider wave solutions to nonlinear sigma models in n dimensions. First, we reduce the system of governing PDEs into a system of ODEs through a traveling wave assumption. Under a new transform, we then reduce this system into a single nonlinear ODE. Making use of the method of homotopy analysis, we are able to construct approximate analytical solutions to this nonlinear ODE. We apply two distinct auxiliary linear operators and show that one of these permits solutions with lower residual error than the other. This demonstrates the effectiveness of properly selecting the auxiliary linear operator when performing homotopy analysis of a nonlinear problem. From here, we then obtain residual error‐minimizing values of the convergence control parameter. We find that properly selecting the convergence control parameter makes a drastic difference in the magnitude of the residual error. Together, appropriate selection of the auxiliary linear operator and of the convergence control parameter is shown to allow approximate solutions that quickly converge to the true solution, which means that few terms are needed in the construction of such solution. This, in turn, greatly improves computational efficiency. Copyright © 2013 John Wiley & Sons, Ltd.     

A Family of Estimators of Finite-Population Distribution Function Using Auxiliary Information

This paper considers the problem of estimating the finite-population distribution function and quantiles with the use of auxiliary information at the estimation stage of a survey. We propose the families of estimators of the distribution function of the study variate y using the knowledge of the distribution function of the auxiliary variate x. In addition to ratio, product and difference type estimators, many other estimators are identified as members of the proposed families. For these families the approximate variances are derived, and in addition, the optimum estimator is identified along with its approximate variance. Estimators based on the estimated optimum values of the unknown parameters used to minimize the variance are also given with their properties. Further, the family of estimators of a finite-population distribution function using two-phase sampling is given, and its properties are investigated.      

A generalized partially linear framework for variance functions

When model the heteroscedasticity in a broad class of partially linear models, we allow the variance function to be a partial linear model as well and the parameters in the variance function to be different from those in the mean function. We develop a two-step estimation procedure, where in the first step some initial estimates of the parameters in both the mean and variance functions are obtained and then in the second step the estimates are updated using the weights calculated based on the initial estimates. The resulting weighted estimators of the linear coefficients in both the mean and variance functions are shown to be asymptotically normal, more efficient than the initial un-weighted estimators, and most efficient in the sense of semiparametric efficiency for some special cases. Simulation experiments are conducted to examine the numerical performance of the proposed procedure, which is also applied to data from an air pollution study in Mexico City.     

Computational bounds on polynomial differential equations

In this paper we study from a computational perspective some properties of the solutions of polynomial ordinary differential equations.We consider elementary (in the sense of Analysis) discrete-time dynamical systems satisfying certain criteria of robustness. We show that those systems can be simulated with elementary and robust continuous-time dynamical systems which can be expanded into fully polynomial ordinary differential equations in Q[π]. This sets a computational lower bound on polynomial ODEs since the former class is large enough to include the dynamics of arbitrary Turing machines.We also apply the previous methods to show that the problem of determining whether the maximal interval of definition of an initial-value problem defined with polynomial ODEs is bounded or not is in general undecidable, even if the parameters of the system are computable and comparable and if the degree of the corresponding polynomial is at most 56.Combined with earlier results on the computability of solutions of polynomial ODEs, one can conclude that there is from a computational point of view a close connection between these systems and Turing machines.     

Classical and Bayesian inference in two parameter exponential distribution with randomly censored data

This paper deal with the classical and Bayesian estimation for two parameter exponential distribution having scale and location parameters with randomly censored data. The censoring time is also assumed to follow a two parameter exponential distribution with different scale but same location parameter. The main stress is on the location parameter in this paper. This parameter has not yet been studied with random censoring in literature. Fitting and using exponential distribution on the range \((0, \infty )\), specially when the minimum observation in the data set is significantly large, will give estimates far from accurate. First we obtain the maximum likelihood estimates of the unknown parameters with their variances and asymptotic confidence intervals. Some other classical methods of estimation such as method of moment, L-moments and least squares are also employed. Next, we discuss the Bayesian estimation of the unknown parameters using Gibbs sampling procedures under generalized entropy loss function with inverted gamma priors and Highest Posterior Density credible intervals. We also consider some reliability and experimental characteristics and their estimates. A Monte Carlo simulation study is performed to compare the proposed estimates. Two real data examples are given to illustrate the importance of the location parameter.     

Certain estimation problems for multivariate hypergeometric models

Certain estimation problems associated with the multivariate hypergeometric models: the property of completeness, maximum likelihood estimates of the parameters of multivariate negative hypergeometric, multivariate negative inverse hypergeometric, Bayesian estimation of the parameters of multivariate hypergeometric and multivariate inverse hypergeometrics are discussed in this paper. A two stage approach for generating the prior distribution, first by setting up a parametric super population and then choosing a prior distribution is followed. Posterior expectations and variances of certain functions of the parameters of the finite population are provided in cases of direct and inverse sampling procedures. It is shown that under extreme diffuseness of prior knowledge the posterior distribution of the finite population mean has an approximate mean and variance (N-n)S ²/Nn, providing a Bayesian interpretation for the classical unbiased estimates in traditional sample survey theory.