Recursive computational formulas of the least squares criterion functions for scalar system identification

The paper discusses recursive computation problems of the criterion functions of several least squares type parameter estimation methods for linear regression models, including the well-known recursive least squares (RLS) algorithm, the weighted RLS algorithm, the forgetting factor RLS algorithm and the finite-data-window RLS algorithm without or with a forgetting factor. The recursive computation formulas of the criterion functions are derived by using the recursive parameter estimation equations. The proposed recursive computation formulas can be extended to the estimation algorithms of the pseudo-linear regression models for equation error systems and output error systems. Finally, the simulation example is provided.     

Design of RLS Wiener fixed-lag smoother using covariance information in linear discrete stochastic systems

In this paper, we propose a new design for the recursive least-squares (RLS) Wiener fixed-lag smoother and filter in linear discrete-time wide-sense stationary stochastic systems. It is assumed that the signal is observed with additive white observation noise. The signal is uncorrelated with the observation noise. The estimators require knowledge of the system matrix, the observation matrix and the variance of the state vector. These quantities can be obtained from the auto-covariance function of the signal. In the estimation algorithms, moreover, the variance of the observation noise is assumed to be known, as a priori information.     

An innovative fractional order LMS algorithm for power signal parameter estimation

Parameter estimation is an important issue for the quality monitoring and reliability assessment of power systems. In this study, an innovative fractional order least mean square (I-FOLMS) adaptive algorithm is presented for an effective parameter estimation. The I-FOLMS algorithm exploits the fractional gradient in its recursive parameter update mechanism, because its performance can be tuned by means of the fractional order. High values of the fractional order are good for fast convergence, but lead to steady state mis-adjustments. While, low values provide a smooth steady state behavior, but require a compromise in the convergence rate. The effective performance of I-FOLMS is verified and validated through two numerical examples of power signals estimation for different levels of noise variance and values of the fractional orders.     

A preprocessing method for parameter estimation in ordinary differential equations

Parameter estimation for nonlinear differential equations is notoriously difficult because of poor or even no convergence of the nonlinear fit algorithm due to the lack of appropriate initial parameter values. This paper presents a method to gather such initial values by a simple estimation procedure. The method first determines the tangent slope and coordinates for a given solution of the ordinary differential equation (ODE) at randomly selected points in time. With these values the ODE is transformed into a system of equations, which is linear for linear appearance of the parameters in the ODE. For numerically generated data of the Lorenz attractor good estimates are obtained even at large noise levels. The method can be generalized to nonlinear parameter dependency. This case is illustrated using numerical data for a biological example. The typical problems of the method as well as their possible mitigation are discussed. Since a rigorous failure criterion of the method is missing, its results must be checked with a nonlinear fit algorithm. Therefore the method may serve as a preprocessing algorithm for nonlinear parameter fit algorithms. It can improve the convergence of the fit by providing initial parameter estimates close to optimal ones.     

Variational Approximation for Mixtures of Linear Mixed Models

Mixtures of linear mixed models (MLMMs) are useful for clustering grouped data and can be estimated by likelihood maximization through the Expectation–Maximization algorithm. A suitable number of components is then determined conventionally by comparing different mixture models using penalized log-likelihood criteria such as Bayesian information criterion. We propose fitting MLMMs with variational methods, which can perform parameter estimation and model selection simultaneously. We describe a variational approximation for MLMMs where the variational lower bound is in closed form, allowing for fast evaluation and develop a novel variational greedy algorithm for model selection and learning of the mixture components. This approach handles algorithm initialization and returns a plausible number of mixture components automatically. In cases of weak identifiability of certain model parameters, we use hierarchical centering to reparameterize the model and show empirically that there is a gain in efficiency in variational algorithms similar to that in Markov chain Monte Carlo (MCMC) algorithms. Related to this, we prove that the approximate rate of convergence of variational algorithms by Gaussian approximation is equal to that of the corresponding Gibbs sampler, which suggests that reparameterizations can lead to improved convergence in variational algorithms just as in MCMC algorithms. Supplementary materials for the article are available online.     

受控AR模型中参数估计误差的一致有界性分析

本针对系统为受控AR模型，其参数估计采用随机梯度算法时，用鞅收敛定理的推广形式分析了它的收敛性，得到了参数估计误差一致有界的结果．     

Estimation of spatial ar models

In this paper, some results in [1] on parameter estimation of spatial AR models are improved. Moreover, using some recent results on convergence rate of sample autocorrelations, we give strongly consistent order estimates for spatial AR models and iterated logarithmic convergence rate for AR parameter estimates, which improve the result in [2] of weakly consistent order estimation and strongly consistent parameter estimation.This project is supported by the Doctoral Programme Foundation of Institute of Higher Education and by the National Natural Science Foundation of China.     

混合模拟退火-进化策略在非线性参数估计中的应用

提出了一种混合模拟退火-进化策略算法应用在非线性参数估计中,方法克服了传统优化方法估计参数精度不高且容易陷入局部极小值等缺点,并且将模拟退火算法和进化策略算法相结合,充分发挥各自算法优点.最后通过给出非线性参数估计算例,结果表明,算法具有参数估计精度较高,收敛速度快,自适应性强,在实际工程中有较大的应用价值.     

General convergence results for stochastic approximations via weak convergence theory

The paper treats general convergence conditions for a class of algorithms for finding the minima of a function f(x) when f(x) is of unknown (or partly unknown) form, and when only noise corrupted observations can be taken. Such problems occur frequently in adaptive processes, and in many applications to statistics and estimation. The algorithms are of the stochastic approximation type. Several forms are dealt with—for estimation in either discrete or continuous time, with and without side constraints, and with or without periodic search renewal. The algorithms can be considered as sequential Monte Carlo methods for systems optimization. The innovations partly concern the method of proof. However, an interesting “constrained” and “renewed” algorithm is also considered. By using ideas from the theory of weak convergence of probability measures, we can get relatively short proofs, under much weaker conditions than heretofore required. For example, the noise can be correlated, and there are fewer restrictions on the step size. Furthermore, the nature of the method permits generalizations to more abstract cases (which occur for example, if we are optimizing a distributed parameter system). The results can be extended in many directions and variations of the technique can be used to get bounds on rates of convergence. Special forms of the method can be applied to many well-known “adaptive” procedures.     

Semimartingale stochastic approximation procedure and recursive estimation

The semimartingale stochastic approximation procedure, precisely, the Robbins-Monro type SDE, is introduced, which naturally includes both generalized stochastic approximation algorithms with martingale noises and recursive parameter estimation procedures for statistical models associated with semimartingales. General results concerning the asymptotic behavior of the solution are presented. In particular, the conditions ensuring the convergence, the rate of convergence, and the asymptotic expansion are established. The results concerning the Polyak weighted averaging procedure are also presented. __________ Translated from Sovremennaya Matematika i Ee Prilozheniya (Contemporary Mathematics and Its Applications), Vol. 45, Martingale Theory and Its Application, 2007.     

Particle-based online estimation of tangent filters with application to parameter estimation in nonlinear state-space models

This paper presents a novel algorithm for efficient online estimation of the filter derivatives in general hidden Markov models. The algorithm, which has a linear computational complexity and very limited memory requirements, is furnished with a number of convergence results, including a central limit theorem with an asymptotic variance that can be shown to be uniformly bounded in time. Using the proposed filter derivative estimator, we design a recursive maximum likelihood algorithm updating the parameters according the gradient of the one-step predictor log-likelihood. The efficiency of this online parameter estimation scheme is illustrated in a simulation study.

     

双边定时截尾样本下广义逆指数分布形状参数的估计

先给出了广义逆指数分布在双边定时截尾样本下形状参数的最大似然估计,并不能得到估计的显式表达式,但证明了参数在(0,+∞)上最大似然估计是唯一存在的.其次提出用EM算法求出形状参数的估计且该估计具有良好的收敛性,还给出了形状参数的EM估计的渐近方差和近似置信区间;最后通过数值模拟,对形状参数的最大似然估计和EM估计的效果进行了比较,说明了用EM算法求形状参数的估计是可行的,并且模拟效果相对比较好.     

Parameter and state estimation algorithm for single-input single-output linear systems using the canonical state space models

This paper presents a new parameter and state estimation algorithm for single-input single-output systems based on canonical state space models from the given input–output data. Difficulties of identification for state space models lie in that there exist unknown noise terms in the formation vector and unknown state variables. By means of the hierarchical identification principle, those noise terms in the information vector are replaced with the estimated residuals and a new least squares algorithm is proposed for parameter estimation and the system states are computed by using the estimated parameters. Finally, an example is provided.     

Approximate Bayesian Approach to Non-Gaussian Estimation in a Linear Model with Dependent State and Noise Vectors

This paper extends the results of Masreliez [8] on the design of non-Gaussian estimators for a more general class of the parameter estimation problem when the system state and the observation noise may be dependent and non-Gaussian simultaneously. It is shown that the proposed non-Gaussian algorithms can approximate with high precision the minimum mean square estimator. Application of the approach to the design of different optimal (and stable) estimation algorithms is illustrated. The efficiency of the proposed algorithms is tested in some simulation experiments. Accepted 5 September 2000. Online publication 26 February 2001.     

An exact minimum variance filter for a class of discrete time systems with random parameter perturbations

An exact, closed-form minimum variance filter is designed for a class of discrete time uncertain systems which allows for both multiplicative and additive noise sources. The multiplicative noise model includes a popular class of models (Cox-Ingersoll-Ross type models) in econometrics. The parameters of the system under consideration which describe the state transition are assumed to be subject to stochastic uncertainties. The problem addressed is the design of a filter that minimizes the trace of the estimation error variance. Sensitivity of the new filter to the size of parameter uncertainty, in terms of the variance of parameter perturbations, is also considered. We refer to the new filter as the ‘perturbed Kalman filter’ (PKF) since it reduces to the traditional (or unperturbed) Kalman filter as the size of stochastic perturbation approaches zero. We also consider a related approximate filtering heuristic for univariate time series and we refer to filter based on this heuristic as approximate perturbed Kalman filter (APKF). We test the performance of our new filters on three simulated numerical examples and compare the results with unperturbed Kalman filter that ignores the uncertainty in the transition equation. Through numerical examples, PKF and APKF are shown to outperform the traditional (or unperturbed) Kalman filter in terms of the size of the estimation error when stochastic uncertainties are present, even when the size of stochastic uncertainty is inaccurately identified.     

Design of recursive least-squares fixed-lag smoother using covariance information in linear continuous stochastic systems

This paper newly designs the recursive least-squares (RLS) fixed-lag smoother and filter using the covariance information in linear continuous-time stochastic systems. It is assumed that the signal is observed with additive white observation noise and the signal is uncorrelated with the observation noise. The estimators require the covariance information of the signal and the variance of the observation noise. The auto-covariance function of the signal is expressed in the semi-degenerate kernel form.     

Approximation of an Analog Diffusion Network with Applications to Image Estimation

This work is concerned with a numerical procedure for approximating an analog diffusion network. The key idea is to take advantage of the separable feature of the noise for the diffusion machine and use a parallel processing method to develop recursive algorithms. The asymptotic properties are studied. The main result of this paper is to establish the convergence of a continuous-time interpolation of the discrete-time algorithm to that of the analog diffusion network via weak convergence methods. The parallel processing feature of the network makes it attractive for solving large-scale optimization problems. Applications to image estimation are considered. Not only is this algorithm useful for the image estimation problems, but it is widely applicable to many related optimization problems.     

An efficient ECM algorithm for maximum likelihood estimation in mixtures of t-factor analyzers

Mixture of t factor analyzers (MtFA) have been shown to be a sound model-based tool for robust clustering of high-dimensional data. This approach, which is deemed to be one of natural parametric extensions with respect to normal-theory models, allows for accommodation of potential noise components, atypical observations or data with longer-than-normal tails. In this paper, we propose an efficient expectation conditional maximization (ECM) algorithm for fast maximum likelihood estimation of MtFA. The proposed algorithm inherits all appealing properties of the ordinary EM algorithm such as its stability and monotonicity, but has a faster convergence rate since its CM steps are governed by a much smaller fraction of missing information. Numerical experiments based on simulated and real data show that the new procedure outperforms the commonly used EM and AECM algorithms substantially in most of the situations, regardless of how the convergence speed is assessed by the computing time or number of iterations.     

Convergence properties of the least squares estimation algorithm for multivariable systems

This paper focuses on the convergence properties of the least squares parameter estimation algorithm for multivariable systems that can be parameterized into a class of multivariate linear regression models. The performance analysis of the algorithm by using the stochastic process theory and the martingale convergence theorem indicates that the parameter estimation errors converge to zero under weak conditions. The simulation results validate the proposed theorem.     

非线性模型误差方差估计的Bootstrap逼近和分布的收敛速度

本文对非线性模型误差方差的估计基于Jackknife虚拟值的Bootstrap方法建立了Bootstrap逼近,证明了逼近的相合性定理,得到了逼近的速度是o(n~(-1/2))。进一步,本文证明了误差方差估计的分布以理想的最佳速度o(n~(-1/2))收敛于正态分布的结论。