Parameter Estimation of Nonlinear Stochastic Differential Equations: Simulated Maximum Likelihood versus Extended Kalman Filter and Itô-Taylor Expansion

This article compares several estimation methods for nonlinear stochastic differential equations with discrete time measurements. The likelihood function is computed by Monte Carlo simulations of the transition probability (simulated maximum likelihood SML) using kernel density estimators and functional integrals and by using the extended Kalman filter (EKF and second-order nonlinear filter SNF). The relation with a local linearization method is discussed. A simulation study for a diffusion process in a double well potential (Ginzburg–Landau equation) shows that, for large sampling intervals, the SML methods lead to better estimation results than the likelihood approach via EKF and SNF. A second study using a nonlinear diffusion coefficient (generalized Cox–Ingersoll–Ross model) demonstrates that the EKF type estimators may serve as efficient alternatives to simple maximum quasilikelihood approaches and Monte Carlo methods.     

The extended Kalman Filter and the unscented Kalman Filter for Material Parameter Identification with Application in Tunneling

We present in this work the use of the extended Kalman filter (EKF) and unscented Kalman filter (UKF) for identification of constitutive material parameters with application in mechanized tunneling. Although both filters are based on the principle of recursive least squares estimation, one differs from another in terms of where approximation is made. Whereas in the EKF first-order Taylor series expansion is used to approximate the nonlinear modeling equation, in the UKF approximation of the probability density of the state is made using a small number of well defined points. To validate the methods, we performed parameter identification of the Hardening Soil constitutive model used for describing the soil behavior in an tunnel excavation model. Both methods showed fast and stable convergence of the considered soil parameters - the four parameters of the Hardening Soil model. Although the EKF requires less number of forward calculations of the numerical model, the UKF is favored since it does not require calculation of the derivatives of the observables with respect to the identifying parameters. (© 2013 Wiley-VCH Verlag GmbH & Co. KGaA, Weinheim)     

Time-Varying Parameters Prediction

This paper develops a method of adaptive modeling that may be applied to forecast non-stationary time series. The starting point are time-varying coefficients models introduced in statistics, econometrics and engineering. The basic step of modeling is represented by the implementation of adaptive recursive estimators for tracking parameters. This is achieved by unifying basic algorithms—such as recursive least squares (RLS) and extended Kalman filter (EKF)—into a general scheme and next by selecting its coefficients with the minimization of the sum of squared prediction errors. This defines a non-linear estimation problem that may be analyzed in the context of the conditional least squares (CLS) theory. A numerical application on the IBM stock price series of Box-Jenkins illustrates the method and shows its good forecasting ability.     

Explicit solution of a Kolmogorov equation

Ever since the technique of the Kalman-Bucy filter was popularized, there has been an intense interest in finding new classes of finite-dimensional recursive filters. In the late seventies, the concept of the estimation algebra of a filtering system was introduced. It has been the major tool in studying the Duncan-Mortensen-Zakai equation. Recently the second author has constructed general finite-dimensional filters which contain both Kalman-Bucy filters and Benes filter as special cases. In this paper we consider a filtering system with arbitrary nonlinear driftf(x) which satisfies some regularity assumption at infinity. This is a natural assumption in view of Theorem 10 of [DTWY] in a special case. Under the assumption on the observation h(x)=constant, we propose writing down the solution of the Duncan-Mortensen-Zakai equation explicitly.This research was supported by Army Grant DAAH-04-93G-0006.     

Continuity of the Filter with Unbounded Observation Coefficients

In this article, we study the continuity with respect to the trajectories of the observation process for the filter associated with nonlinear filtering problems when the coefficients depend on both the signal and the observation and the observation coefficient is unbounded.

To achieve this task we define a formal unnormalized filter and prove by limiting arguments that it is related to the original filter through a generalized Bayes formula, and is locally Lipschitz continuous with respect to the uniform norm.     

Estimating Functions for Nonlinear Time Series Models

This paper discusses the problem of estimation for two classes of nonlinear models, namely random coefficient autoregressive (RCA) and autoregressive conditional heteroskedasticity (ARCH) models. For the RCA model, first assuming that the nuisance parameters are known we construct an estimator for parameters of interest based on Godambe's asymptotically optimal estimating function. Then, using the conditional least squares (CLS) estimator given by Tjøstheim (1986, Stochastic Process. Appl., 21, 251–273) and classical moment estimators for the nuisance parameters, we propose an estimated version of this estimator. These results are extended to the case of vector parameter. Next, we turn to discuss the problem of estimating the ARCH model with unknown parameter vector. We construct an estimator for parameters of interest based on Godambe's optimal estimator allowing that a part of the estimator depends on unknown parameters. Then, substituting the CLS estimators for the unknown parameters, the estimated version is proposed. Comparisons between the CLS and estimated optimal estimator of the RCA model and between the CLS and estimated version of the ARCH model are given via simulation studies.     

Nonlinear continuous-discrete filtering using kernel density estimatesand functional integrals

We develop filter algorithms for nonlinear stochastic differential equations with discrete time measurements (continuous-discrete state space model). The apriori density (time update) is computed by Monte Carlo simulations of the Fokker-Planck equation using kernel density estimators and measurement updates are obtained by using the extended Kalman filter (EKF) updates. For small sampling intervals, a discretized continuous sampling approach (DCS) is used. A third algorithm utilizes a functional (path) integral representation of the transition density (functional integral filter FIF). The kernel density filter (KDF), DCS, and FIF are compared with the EKF and the Gaussian sum filter by using a Ginzburg-Landau-equation and a stochastic volatility model.     

Extended and Unscented Kalman filtering based feedforward neural networks for time series prediction

With the ability to deal with high non-linearity, artificial neural networks (ANNs) and support vector machines (SVMs) have been widely studied and successfully applied to time series prediction. However, good fitting results of ANNs and SVMs to nonlinear models do not guarantee an equally good prediction performance. One main reason is that their dynamics and properties are changing with time, and another key problem is the inherent noise of the fitting data. Nonlinear filtering methods have some advantages such as handling additive noises and following the movement of a system when the underlying model is evolving through time. The present paper investigates time series prediction algorithms by using a combination of nonlinear filtering approaches and the feedforward neural network (FNN). The nonlinear filtering model is established by using the FNN’s weights to present state equation and the FNN’s output to present the observation equation, and the input vector to the FNN is composed of the predicted signal with given length, then the extended Kalman filtering (EKF) and Unscented Kalman filtering (UKF) are used to online train the FNN. Time series prediction results are presented by the predicted observation value of nonlinear filtering approaches. To evaluate the proposed methods, the developed techniques are applied to the predictions of one simulated Mackey-Glass chaotic time series and one real monthly mean water levels time series. Generally, the prediction accuracy of the UKF-based FNN is better than the EKF-based FNN when the model is highly nonlinear. However, comparing from prediction accuracy and computational effort based on the prediction model proposed in our study, we draw the conclusion that the EKF-based FNN is superior to the UKF-based FNN for the theoretical Mackey-Glass time series prediction and the real monthly mean water levels time series prediction.     

Online state of charge estimation of Li-ion battery based on an improved unscented Kalman filter approach

An improved unscented Kalman filter approach is proposed to enhance online state of charge estimation in terms of both accuracy and robustness. The goal is to address the drawback associated with the unscented Kalman filter in terms of its requirement for an accurate model and a priori noise statistics. Firstly, Li-ion battery modelling and offline parameter identification is performed. Secondly, a sensitivity analysis experiment is designed to verify which model parameter has the greatest influence on state of charge estimation accuracy, in order to provide an appropriate parameter for the model adaptive algorithm. Thirdly, an improved unscented Kalman filter approach, composed of a model adaptive algorithm and a noise adaptive algorithm, is introduced. Finally, the results are discussed, which reveal that the proposed approach’s estimation error is less than 1.79% with acceptable robustness and time complexity.     

New approach to applying neural network in nonlinear dynamic model

In this work, radial basis function neural network (RBF-NN) is applied to emulate an extended Kalman filter (EKF) in a data assimilation scenario. The dynamical model studied here is based on the one-dimensional shallow water equation DYNAMO-1D. This code is simple when compared with an operational primitive equation models for numerical weather prediction. Although simple, the DYNAMO-1D is rich for representing some atmospheric motions, such as Rossby and gravity waves. It has been shown in the literature that the ability of the EKF to track nonlinear models depends on the frequency and accuracy of the observations and model errors. In some cases, just fourth-order moment EKF works well, but will be unwieldy when applied to high-dimensional state space. Artificial Neural Network (ANN) is an alternative solution for this computational complexity problem, once the ANN is trained offline with a high order Kalman filter, even though this Kalman filter has high computational cost (which is not a problem during ANN training phase). The results achieved in this work encourage us to apply this technique on operational model. However, it is not yet possible to assure convergence in high dimensional problems.     

A Combinatorial Approach to Piecewise Linear Time Series Analysis

Over recent years, several nonlinear time series models have been proposed in the literature. One model that has found a large number of successful applications is the threshold autoregressive model (TAR). The TAR model is a piecewise linear process whose central idea is to change the parameters of a linear autoregressive model according to the value of an observable variable, called the threshold variable. If this variable is a lagged value of the time series, the model is called a self-exciting threshold autoregressive (SETAR) model. In this article, we propose a heuristic to estimate a more general SETAR model, where the thresholds are multivariate. We formulate the task of finding multivariate thresholds as a combinatorial optimization problem. We develop an algorithm based on a greedy randomized adaptive search procedure (GRASP) to solve the problem. GRASP is an iterative randomized sampling technique that has been shown to quickly produce good quality solutions for a wide variety of optimization problems. The proposed model performs well on both simulated and real data.     

LPV identification of a turbocharged diesel engine

This work proposes a methodology of identifying linear parameter varying (LPV) models for nonlinear systems. First, linear local models in some operating points, by applying standard identifications procedures for linear systems in time domain, are obtained. Next, a LPV model with linear fractional dependence (LFR) with respect to measured variables is fitted with the condition of containing all the linear models identified in previous step (differential inclusion). The fit is carried out using nonlinear least squares algorithms. Finally, this identification methodology will then be applied to a nonlinear turbocharged diesel engine.     

On Multivariate Adaptive Approximation

Recently, A. Cohen, R. A. DeVore, P. Petrushev, and H. Xu investigated nonlinear approximation in the space BV (R ² ). They modified the classical adaptive algorithm to solve related extremal problems. In this paper, we further study the modified adaptive approximation and obtain results on some extremal problems related to the spaces V _σ,p ^r (R ^d ) of functions of ``Bounded Variation" and Besov spaces B ^α (R ^d ). November 23, 1998. Date revised: June 25, 1999. Date accepted: September 13, 1999.     

New results in Sridhar filtering theory: The discrete case

A nonlinear time-varying adaptive filter is introduced, and its derivation using optimal control concepts is given in detail. The filter, which is called the discrete Pontryagin filter, is basically an extension to Sridhar filtering theory. The proposed approach can easily replace the conventional methods of autoregressive (AR) and autoregressive moving average (ARMA) models in their many applications. Instead of using a large number of time-invariant parameters to describe the signal or the time series, a single time-varying function is enough. This function is estimated using optimization techniques. Many features are gained using this approach, such as simpler and compact filter equations and better overall accuracy. The statistical properties of the filter are given, and it is shown that the signal estimate will converge in thepth mean to the true value.     

Image Restoration Using Modifications of Simulated Annealing

Abstract

This article uses a modified version of the simulated annealing algorithm to restore degraded spatial patterns. Standard simulated annealing is used to find an image that is a posterior mode when the number of images under consideration precludes sequential search for a maximum. I incorporate jumping probabilities of the annealing algorithm without randomization. The convergence of our algorithm is proven under a practical annealing schedule. The same idea is also implemented to improve the performance of other modifications of simulated annealing. These include forcing proportions of labels in an image, using posterior marginals, and incorporating an edge process. This article also studies nonlinear presmoothing of the observations.     

Adaptive finite element methods for the identification of distributed parameters in elliptic equation

In this paper, adaptive finite element method is developed for the estimation of distributed parameter in elliptic equation. Both upper and lower error bound are derived and used to improve the accuracy by appropriate mesh refinement. An efficient preconditioned project gradient algorithm is employed to solve the nonlinear least-squares problem arising in the context of parameter identification problem. The efficiency of our error estimators is demonstrated by some numerical experiments.      

Asymptotic stability, ergodicity and other asymptotic properties of the nonlinear filter

In this work we study connections between various asymptotic properties of the nonlinear filter. It is assumed that the signal has a unique invariant probability measure. The key property of interest is expressed in terms of a relationship between the observation σ field and the tail σ field of the signal, in the stationary filtering problem. This property can be viewed as the permissibility of the interchange of the order of the operations of maximum and countable intersection for certain σ-fields. Under suitable conditions, it is shown that the above property is equivalent to various desirable properties of the filter such as

(a) uniqueness of invariant measure for the signal,

(b) uniqueness of invariant measure for the pair (signal, filter),

(c) a finite memory property of the filter,

(d) a property of finite time dependence between the signal and observation σ fields and

(e) asymptotic stability of the filter.

Previous works on the asymptotic stability of the filter for a variety of filtering models then identify a rich class of filtering problems for which the above equivalent properties hold.     

On Detecting Solutions of Polynomial Nonlinear Difference Equations

In this paper a method for discovering solutions of nonlinear polynomial difference equations is presented. It is based on the concepts of i -operator and star-product. These notions create a proper algebraic background by means of which we can find linear equations "included" into the original nonlinear one and to seek for solutions among them. A corresponding algorithm and some examples are also provided.     

Toward the Calculation of Higher-Dimensional Stable Manifolds and Stable Sets for Noninvertible and Piecewise-Smooth Maps

This paper presents a new numerical method for computing global stable manifolds and global stable sets of nonlinear discrete dynamical systems. For a given map f:ℝ ^d →ℝ ^d , the proposed method is capable of yielding large parts of stable manifolds and sets within a certain compact region M⊂ℝ ^d . The algorithm divides the region M in sets and uses an adaptive subdivision technique to approximate an outer covering of the manifolds. In contrast to similar approaches, the method requires neither the system’s inverse nor its Jacobian. Hence, it can also be applied to noninvertible and piecewise-smooth maps. The successful application of the method is illustrated by computation of one- and two-dimensional stable manifolds and global stable sets.