Acceleration of the EM algorithm using the vector epsilon algorithm

The expectation–maximization (EM) algorithm is a very general and popular iterative computational algorithm to find maximum likelihood estimates from incomplete data and broadly used to statistical analysis with missing data, because of its stability, flexibility and simplicity. However, it is often criticized that the convergence of the EM algorithm is slow. The various algorithms to accelerate the convergence of the EM algorithm have been proposed. The vector ε algorithm of Wynn (Math Comp 16:301–322, 1962) is used to accelerate the convergence of the EM algorithm in Kuroda and Sakakihara (Comput Stat Data Anal 51:1549–1561, 2006). In this paper, we provide the theoretical evaluation of the convergence of the ε-accelerated EM algorithm. The ε-accelerated EM algorithm does not use the information matrix but only uses the sequence of estimates obtained from iterations of the EM algorithm, and thus it keeps the flexibility and simplicity of the EM algorithm.     

Learning from incomplete data in Bayesian networks with qualitative influences

Domain experts can often quite reliably specify the sign of influences between variables in a Bayesian network. If we exploit this prior knowledge in estimating the probabilities of the network, it is more likely to be accepted by its users and may in fact be better calibrated with reality. We present two algorithms that exploit prior knowledge of qualitative influences in learning the parameters of a Bayesian network from incomplete data. The isotonic regression EM, or irEM, algorithm adds an isotonic regression step to standard EM in each iteration, to obtain parameter estimates that satisfy the given qualitative influences. In an attempt to reduce the computational burden involved, we further define the qirEM algorithm that enforces the constraints imposed by the qualitative influences only once, after convergence of standard EM. We evaluate the performance of both algorithms through experiments. Our results demonstrate that exploitation of the qualitative influences improves the parameter estimates over standard EM, and more so if the proportion of missing data is relatively large. The results also show that the qirEM algorithm performs just as well as its computationally more expensive counterpart irEM.     

Stochastic EM algorithms for parametric and semiparametric mixture models for right-censored lifetime data

Mixture models in reliability bring a useful compromise between parametric and nonparametric models, when several failure modes are suspected. The classical methods for estimation in mixture models rarely handle the additional difficulty coming from the fact that lifetime data are often censored, in a deterministic or random way. We present in this paper several iterative methods based on EM and Stochastic EM methodologies, that allow us to estimate parametric or semiparametric mixture models for randomly right censored lifetime data, provided they are identifiable. We consider different levels of completion for the (incomplete) observed data, and provide genuine or EM-like algorithms for several situations. In particular, we show that simulating the missing data coming from the mixture allows to plug a standard R package for survival data analysis in an EM algorithm’s M-step. Moreover, in censored semiparametric situations, a stochastic step is the only practical solution allowing computation of nonparametric estimates of the unknown survival function. The effectiveness of the new proposed algorithms are demonstrated in simulation studies and an actual dataset example from aeronautic industry.     

基于EM算法的混合模型中子总体个数的研究

混合模型已成为数据分析中最流行的技术之一,由于拥有数学模型,它通常比聚类分析中的传统的方法产生的结果更精确,而关键因素是混合模型中子总体个数,它决定了数据分析的最终结果。期望最大化(EM)算法常用在混合模型的参数估计,以及机器学习和聚类领域中的参数估计中,是一种从不完全数据或者是有缺失值的数据中求解参数极大似然估计的迭代算法。学者们往往采用AIC和BIC的方法来确定子总体的个数,而这两种方法在实际的应用中的效果并不稳定,甚至可能会产生错误的结果。针对此问题,本文提出了一种利用似然函数的碎石图来确定混合模型中子总体的个数的新方法。实验结果表明,本文方法确定的子总体的个数在大部分理想的情况下可以得到与AIC、BIC方法确定的聚类个数相同的结果,而在一般的实际数据中或条件不理想的状态下,碎石图方法也可以得到更可靠的结果。随后,本文将新方法在选取的黄石公园喷泉数据的参数估计中进行了实际的应用。     

Monte Carlo EM加速算法

EM算法是近年来常用的求后验众数的估计的一种数据增广算法, 但由于求出其E步中积分的显示表达式有时很困难, 甚至不可能, 限制了其应用的广泛性. 而Monte Carlo EM算法很好地解决了这个问题, 将EM算法中E步的积分用Monte Carlo模拟来有效实现, 使其适用性大大增强. 但无论是EM算法, 还是Monte Carlo EM算法, 其收敛速度都是线性的, 被缺损信息的倒数所控制, 当缺损数据的比例很高时, 收敛速度就非常缓慢. 而Newton-Raphson算法在后验众数的附近具有二次收敛速率. 本文提出Monte Carlo EM加速算法, 将Monte Carlo EM算法与Newton-Raphson算法结合, 既使得EM算法中的E步用Monte Carlo模拟得以实现, 又证明了该算法在后验众数附近具有二次收敛速度. 从而使其保留了Monte Carlo EM算法的优点, 并改进了Monte Carlo EM算法的收敛速度. 本文通过数值例子, 将Monte Carlo EM加速算法的结果与EM算法、Monte Carlo EM算法的结果进行比较, 进一步说明了Monte Carlo EM加速算法的优良性.     

Supervised learning of multivariate skew normal mixture models with missing information

We establish computationally flexible tools for the analysis of multivariate skew normal mixtures when missing values occur in data. To facilitate the computation and simplify the theoretical derivation, two auxiliary permutation matrices are incorporated into the model for the determination of observed and missing components of each observation and are manifestly effective in reducing the computational complexity. We present an analytically feasible EM algorithm for the supervised learning of parameters as well as missing observations. The proposed mixture analyzer, including the most commonly used Gaussian mixtures as a special case, allows practitioners to handle incomplete multivariate data sets in a wide range of considerations. The methodology is illustrated through a real data set with varying proportions of synthetic missing values generated by MCAR and MAR mechanisms and shown to perform well on classification tasks.     

Computational Strategies for Multivariate Linear Mixed-Effects Models With Missing Values

This article presents new computational techniques for multivariate longitudinal or clustered data with missing values. Current methodology for linear mixed-effects models can accommodate imbalance or missing data in a single response variable, but it cannot handle missing values in multiple responses or additional covariates. Applying a multivariate extension of a popular linear mixed-effects model, we create multiple imputations of missing values for subsequent analyses by a straightforward and effective Markov chain Monte Carlo procedure. We also derive and implement a new EM algorithm for parameter estimation which converges more rapidly than traditional EM algorithms because it does not treat the random effects as “missing data,” but integrates them out of the likelihood function analytically. These techniques are illustrated on models for adolescent alcohol use in a large school-based prevention trial.     

Computationally Efficient Estimation for the Generalized Odds Rate Mixture Cure Model With Interval-Censored Data

For semiparametric survival models with interval-censored data and a cure fraction, it is often difficult to derive nonparametric maximum likelihood estimation due to the challenge in maximizing the complex likelihood function. In this article, we propose a computationally efficient EM algorithm, facilitated by a gamma-Poisson data augmentation, for maximum likelihood estimation in a class of generalized odds rate mixture cure (GORMC) models with interval-censored data. The gamma-Poisson data augmentation greatly simplifies the EM estimation and enhances the convergence speed of the EM algorithm. The empirical properties of the proposed method are examined through extensive simulation studies and compared with numerical maximum likelihood estimates. An R package “GORCure” is developed to implement the proposed method and its use is illustrated by an application to the Aerobic Center Longitudinal Study dataset. Supplementary material for this article is available online.     

An efficient ECM algorithm for maximum likelihood estimation in mixtures of t-factor analyzers

Mixture of t factor analyzers (MtFA) have been shown to be a sound model-based tool for robust clustering of high-dimensional data. This approach, which is deemed to be one of natural parametric extensions with respect to normal-theory models, allows for accommodation of potential noise components, atypical observations or data with longer-than-normal tails. In this paper, we propose an efficient expectation conditional maximization (ECM) algorithm for fast maximum likelihood estimation of MtFA. The proposed algorithm inherits all appealing properties of the ordinary EM algorithm such as its stability and monotonicity, but has a faster convergence rate since its CM steps are governed by a much smaller fraction of missing information. Numerical experiments based on simulated and real data show that the new procedure outperforms the commonly used EM and AECM algorithms substantially in most of the situations, regardless of how the convergence speed is assessed by the computing time or number of iterations.     

Common fixed points of a nonexpansive semigroup and a convergence theorem for Mann iterations in geodesic metric spaces

First, we consider a strongly continuous semigroup of nonexpansive mappings defined on a closed convex subset of a complete CAT(0) space and prove a convergence of a Mann iteration to a common fixed point of the mappings. This result is motivated by a result of Kirk (2002) and of Suzuki (2002). Second, we obtain a result on limits of subsequences of Mann iterations of multivalued nonexpansive mappings on metric spaces of hyperbolic type, which leads to a convergence theorem for nonexpansive mappings on these spaces.     

Theory and inference for regression models with missing responses and covariates

In this paper, we carry out an in-depth theoretical investigation for inference with missing response and covariate data for general regression models. We assume that the missing data are missing at random (MAR) or missing completely at random (MCAR) throughout. Previous theoretical investigations in the literature have focused only on missing covariates or missing responses, but not both. Here, we consider theoretical properties of the estimates under three different estimation settings: complete case (CC) analysis, a complete response (CR) analysis that involves an analysis of those subjects with only completely observed responses, and the all case (AC) analysis, which is an analysis based on all of the cases. Under each scenario, we derive general expressions for the likelihood and devise estimation schemes based on the EM algorithm. We carry out a theoretical investigation of the three estimation methods in the normal linear model and analytically characterize the loss of information for each method, as well as derive and compare the asymptotic variances for each method assuming the missing data are MAR or MCAR. In addition, a theoretical investigation of bias for the CC method is also carried out. A simulation study and real dataset are given to illustrate the methodology.     

An iterative algorithm for a Cauchy problem in eddy-current modelling

We consider a linear steady-state eddy-current problem for a magnetic field in a bounded domain. The boundary consists of two parts: reachable with prescribed Cauchy data and unreachable with no data on it. We design an iterative (Landweber type) algorithm for solution of this problem. At each iteration step two auxiliary mixed well-posed boundary value problems are solved. The analysis of temporary problems is performed in suitable function spaces. This creates the basis for the convergence argument. The theoretical results are supported with numerical experiments.     

Comparison of the EM algorithm and alternatives

The Expectation-Maximization (EM) algorithm is widely used also in industry for parameter estimation within a Maximum Likelihood (ML) framework in case of missing data. It is well-known that EM shows good convergence in several cases of practical interest. To the best of our knowledge, results showing under which conditions EM converges fast are only available for specific cases. In this paper, we analyze the connection of the EM algorithm to other ascent methods as well as the convergence rates of the EM algorithm in general including also nonlinear models and apply this to the PMHT model. We compare the EM with other known iterative schemes such as gradient and Newton-type methods. It is shown that EM reaches Newton-convergence in case of well-separated objects and a Newton-EM combination turns out to be robust and efficient even in cases of closely-spaced targets.     

An Adaptive Partial Linearization Method for Optimization Problems on Product Sets

We consider a general class of composite optimization problems where the goal function is the sum of a smooth function and a non-necessary smooth convex separable function associated with some space partition, whereas the feasible set is a Cartesian product concordant to this partition. We suggest an adaptive version of the partial linearization method, which makes selective component-wise steps satisfying some descent condition and utilizes a sequence of control parameters. This technique is destined to reduce the computational expenses per iteration and maintain the basic convergence properties. We also establish its convergence rates and describe some examples of applications. Preliminary results of computations illustrate usefulness of the new method.     

Hilbert空间中强半压缩算子的Ishikawa迭代的误差估计和数据依赖

本文我们考虑了实Hilbert空间中强半压缩(SDC)算子的Ishikawa迭代的误差估计(不需要Lipschitz条件),同时得到了Ishikawa迭代的一些收敛性定理.此外,我们在三种情况下给出了SDC算子的数据依赖性结果.一些数值算例验证了我们的结果.     

Efficient ML Estimation of the Multivariate Normal Distribution from Incomplete Data

It is well known that the maximum likelihood estimates (MLEs) of a multivariate normal distribution from incomplete data with a monotone pattern have closed-form expressions and that the MLEs from incomplete data with a general missing-data pattern can be obtained using the Expectation-Maximization (EM) algorithm. This article gives closed-form expressions, analogous to the extension of the Bartlett decomposition, for both the MLEs of the parameters and the associated Fisher information matrix from incomplete data with a monotone missing-data pattern. For MLEs of the parameters from incomplete data with a general missing-data pattern, we implement EM and Expectation-Constrained-Maximization-Either (ECME), by augmenting the observed data into a complete monotone sample. We also provide a numerical example, which shows that the monotone EM (MEM) and monotone ECME (MECME) algorithms converge much faster than the EM algorithm.     

On spectral radii of matrices governing some numerical estimating algorithms with incomplete data

Many estimating procedures are carried out with incomplete data by means of different types of EM algorithms. They allow us to obtain maximum likelihood parameter estimates in classical inference and also estimates based on the posterior mode in Bayesian inference. This paper analyzes in detail the spectral radii of the Jacobian matrices algorithm as a possible way to evaluate convergence rates. The eigenvalues of such matrices are explicitly obtained in some cases and, in all of them, a geometric convergence rate is, at least, guaranteed near the optimum. Finally, a comparison between the leading eigenvalues of EM and direct and approximate EM-Bayes algorithms may suggest the efficiency of each case.     

基于删失数据的指数威布尔分布最大似然估计的新算法

本文讨论了指数威布尔分布当观测数据是删失数据情形时参数的最大似然估计问题.因为删失数据是一种不完全数据,我们利用EM算法来计算参数的近似最大似然估计.由于EM算法计算的复杂性,计算效率也不理想.为了克服牛顿-拉普森算法和EM算法的局限性,我们提出了一种新的方法.这种方法联合了指数威布尔分布到指数分布的变换和等效寿命数据的技巧,比牛顿-拉普森算法和EM算法更具有操作性.数据模拟讨论了这一方法的可行性.为了演示本文的方法,我们还提供了一个真实寿命数据分析的例子.     

On spectral radii of matrices governing some numerical estimating algorithms with incomplete data

Summary Many estimating procedures are carried out with incomplete data by means of different types of EM algorithms. They allow us to obtain maximum likelihood parameter estimates in classical inference and also estimates based on the posterior mode in Bayesian inference. This paper analyzes in detail the spectral radii of the Jacobian matrices algorithm as a possible way to evaluate convergence rates. The eigenvalues of such matrices are explicitly obtained in some cases and, in all of them, a geometric convergence rate is, at least, guaranteed near the optimum. Finally, a comparison between the leading eigenvalues of EM and direct and approximate EM-Bayes algorithms may suggest the efficiency of each case.