Sparse Variational Analysis of Linear Mixed Models for Large Data Sets

It is increasingly common to be faced with longitudinal or multi-level data sets that have large numbers of predictors and/or a large sample size. Current methods of fitting and inference for mixed effects models tend to perform poorly in such settings. When there are many variables, it is appealing to allow uncertainty in subset selection and to obtain a sparse characterization of the data. Bayesian methods are available to address these goals using Markov chain Monte Carlo (MCMC), but MCMC is very computationally expensive and can be infeasible in large p and/or large n problems. As a fast approximate Bayes solution, we recommend a novel approximation to the posterior relying on variational methods. Variational methods are used to approximate the posterior of the parameters in a decomposition of the variance components, with priors chosen to obtain a sparse solution that allows selection of random effects. The method is evaluated through a simulation study, and applied to an epidemiological application.     

Parallel Variational Bayes for Large Datasets With an Application to Generalized Linear Mixed Models

The article develops a hybrid variational Bayes (VB) algorithm that combines the mean-field and stochastic linear regression fixed-form VB methods. The new estimation algorithm can be used to approximate any posterior without relying on conjugate priors. We propose a divide and recombine strategy for the analysis of large datasets, which partitions a large dataset into smaller subsets and then combines the variational distributions that have been learned in parallel on each separate subset using the hybrid VB algorithm. We also describe an efficient model selection strategy using cross-validation, which is straightforward to implement as a by-product of the parallel run. The proposed method is applied to fitting generalized linear mixed models. The computational efficiency of the parallel and hybrid VB algorithm is demonstrated on several simulated and real datasets. Supplementary material for this article is available online.     

Regression Density Estimation With Variational Methods and Stochastic Approximation

Regression density estimation is the problem of flexibly estimating a response distribution as a function of covariates. An important approach to regression density estimation uses finite mixture models and our article considers flexible mixtures of heteroscedastic regression (MHR) models where the response distribution is a normal mixture, with the component means, variances, and mixture weights all varying as a function of covariates. Our article develops fast variational approximation (VA) methods for inference. Our motivation is that alternative computationally intensive Markov chain Monte Carlo (MCMC) methods for fitting mixture models are difficult to apply when it is desired to fit models repeatedly in exploratory analysis and model choice. Our article makes three contributions. First, a VA for MHR models is described where the variational lower bound is in closed form. Second, the basic approximation can be improved by using stochastic approximation (SA) methods to perturb the initial solution to attain higher accuracy. Third, the advantages of our approach for model choice and evaluation compared with MCMC-based approaches are illustrated. These advantages are particularly compelling for time series data where repeated refitting for one-step-ahead prediction in model choice and diagnostics and in rolling-window computations is very common. Supplementary materials for the article are available online.     

Maximum Likelihood for Generalized Linear Models with Nested Random Effects via High-Order,Multivariate Laplace Approximation

Abstract

Nested random effects models are often used to represent similar processes occurring in each of many clusters. Suppose that, given cluster-specific random effects b, the data y are distributed according to f(y|b, Θ), while b follows a density p(b|Θ). Likelihood inference requires maximization of ∫ f(y|b, Θ)p(b|Θdb with respect to Θ. Evaluation of this integral often proves difficult, making likelihood inference difficult to obtain. We propose a multivariate Taylor series approximation of the log of the integrand that can be made as accurate as desired if the integrand and all its partial derivatives with respect to b are continuous in the neighborhood of the posterior mode of b|Θ,y. We then apply a Laplace approximation to the integral and maximize the approximate integrated likelihood via Fisher scoring. We develop computational formulas that implement this approach for two-level generalized linear models with canonical link and multivariate normal random effects. A comparison with approximations based on penalized quasi-likelihood, Gauss—Hermite quadrature, and adaptive Gauss-Hermite quadrature reveals that, for the hierarchical logistic regression model under the simulated conditions, the sixth-order Laplace approach is remarkably accurate and computationally fast.     

Variational Approximations for Generalized Linear Latent Variable Models

Generalized linear latent variable models (GLLVMs) are a powerful class of models for understanding the relationships among multiple, correlated responses. Estimation, however, presents a major challenge, as the marginal likelihood does not possess a closed form for nonnormal responses. We propose a variational approximation (VA) method for estimating GLLVMs. For the common cases of binary, ordinal, and overdispersed count data, we derive fully closed-form approximations to the marginal log-likelihood function in each case. Compared to other methods such as the expectation-maximization algorithm, estimation using VA is fast and straightforward to implement. Predictions of the latent variables and associated uncertainty estimates are also obtained as part of the estimation process. Simulations show that VA estimation performs similar to or better than some currently available methods, both at predicting the latent variables and estimating their corresponding coefficients. They also show that VA estimation offers dramatic reductions in computation time particularly if the number of correlated responses is large relative to the number of observational units. We apply the variational approach to two datasets, estimating GLLVMs to understanding the patterns of variation in youth gratitude and for constructing ordination plots in bird abundance data. R code for performing VA estimation of GLLVMs is available online. Supplementary materials for this article are available online.     

A Sequential Algorithm for Fast Fitting of Dirichlet Process Mixture Models

In this article, we propose an improvement on the sequential updating and greedy search (SUGS) algorithm for fast fitting of Dirichlet process mixture models. The SUGS algorithm provides a means for very fast approximate Bayesian inference for mixture data which is particularly of use when datasets are so large that many standard Markov chain Monte Carlo (MCMC) algorithms cannot be applied efficiently, or take a prohibitively long time to converge. In particular, these ideas are used to initially interrogate the data, and to refine models such that one can potentially apply exact data analysis later on. SUGS relies upon sequentially allocating data to clusters and proceeding with an update of the posterior on the subsequent allocations and parameters which assumes this allocation is correct. Our modification softens this approach, by providing a probability distribution over allocations, with a similar computational cost; this approach has an interpretation as a variational Bayes procedure and hence we term it variational SUGS (VSUGS). It is shown in simulated examples that VSUGS can outperform, in terms of density estimation and classification, a version of the SUGS algorithm in many scenarios. In addition, we present a data analysis for flow cytometry data, and SNP data via a three-class Dirichlet process mixture model, illustrating the apparent improvement over the original SUGS algorithm.     

模糊映射的广义混合型强变分不等式

研究了新的一类模糊映射的广义混合型强变分不等式问题。证明了这类问题解的存在定理和收敛定理,给出解的带误差的Ishikawa型迭代算法。     

Existence and Uniqueness for a Linear Mixed Variational Inequality Arising in Electrical Circuits with Transistors

The main object of this paper is to present an existence and uniqueness result for a class of variational inequalities which is of particular interest to study electrical circuits involving devices like transistors.     

Proximal Methods for Mixed Variational Inequalities

A proximal point method for solving mixed variational inequalities is suggested and analyzed by using the auxiliary principle technique. It is shown that the convergence of the proposed method requires only the pseudomonotonicity of the operator, which is a weaker condition than monotonicity. As special cases, we obtain various known and new results for solving variational inequalities and related problems. Our proof of convergence is very simple as compared with other methods.     

模糊映射的完全广义混合型强变分不等式

研究关于模糊映射的一类新的变分不等式－模糊映射的完全广义混合型强变分不等式,得到此类变分不等式解的存在定理分解的一个逼近算法,推广了文[4]和文[8]的主要结果。     

模糊映射的完全广义混合型强变分包含

研究一类新的关于模糊映射的完全广义混合型强变分包含问题，给出解的逼近算法，证明这类问题解的一个存在定理和序列收敛定理。     

求解变分不等式的一种外逼近法的若干收敛性结果

本文讨论由文［１］提出的一种求解变分不等式问题的外逼近法,并在较弱条件下证明了该算法的收敛性     

Bayesian Computation of Software Reliability

Abstract

Bayesian methods for the Jelinski and Moranda and the Littlewood and Verrall models in software reliability are studied. A Gibbs sampling approach is employed to compute the Bayes estimates. In addition, prediction of future failure times and future reliabilities is examined. Model selection based on the mean squared prediction error and the prequential likelihood of the conditional predictive ordinates is developed.     

Likelihood Ratio Statistic for Exponential Mixtures

Let f ₀(x) be the exponential density and f (x) the translation model. Let (X _i) _i=1,n be i.i.d. random variables, with density g. We test that g is f ₀ against g is a simple mixture, using the LRT statistic. We prove that the LRT diverges to infinity with probability 1/2 and it is equal to 0 with probability 1/2. Therefore, the classical likelihood limiting theory does not hold.     

Splitting Algorithms for General Pseudomonotone Mixed Variational Inequalities

In this paper, we suggest and analyze a number of resolvent-splitting algorithms for solving general mixed variational inequalities by using the updating technique of the solution. The convergence of these new methods requires either monotonicity or pseudomonotonicity of the operator. Proof of convergence is very simple. Our new methods differ from the existing splitting methods for solving variational inequalities and complementarity problems. The new results are versatile and are easy to implement.     

半参数广义线性混合效应模型的影响分析

本文把随机效应当作是缺失数据并利用P-样条拟合非参数部分,从而得到了半参数广义线性混合效应模型(GPLMM)的MCNR估计算法;同时利用Q-函数,我们得到了模型的参数部分的广义Cook距离以及非参数部分的广义DFIT,此外,本文还研究了四种不同扰动情形的PLMM的局部影响分析,得到了相应的影响矩阵,最后,我们通过—个实际例子验证了所提出的诊断统计量的有效性。     

Online Variational Bayes Inference for High-Dimensional Correlated Data

High-dimensional data with hundreds of thousands of observations are becoming commonplace in many disciplines. The analysis of such data poses many computational challenges, especially when the observations are correlated over time and/or across space. In this article, we propose flexible hierarchical regression models for analyzing such data that accommodate serial and/or spatial correlation. We address the computational challenges involved in fitting these models by adopting an approximate inference framework. We develop an online variational Bayes algorithm that works by incrementally reading the data into memory one portion at a time. The performance of the method is assessed through simulation studies. The methodology is applied to analyze signal intensity in MRI images of subjects with knee osteoarthritis, using data from the Osteoarthritis Initiative. Supplementary materials for this article are available online.     

Prior influence in linear regression when the number of covariates increases to infinity

It is becoming more typical in regression problems today to have the situation where “p>n”, that is, where the number of covariates is greater than the number of observations. Approaches to this problem include such strategies as model selection and dimension reduction, and, of course, a Bayesian approach. However, the discrepancy between p and n can be so large, especially in genomic data, that examining the limiting case where p→∞ can be a relevant calculation. Here we look at the effect of a prior distribution on the coefficients, and in particular characterize the conditions under which, as p→∞, the prior does not overwhelm the data. Specifically, we find that the prior variance on the growing number of covariates must approach zero at rate 1/p, otherwise the prior will overwhelm the data and the posterior distribution of the regression coefficient will equal the prior distribution.     

Some New Iterative Methods for General Mixed Variational Inequalities

In this paper, we use the auxiliary principle technique to suggest a class of predictorcorrector methods for solving general mixed variational inequalities. The convergence of the proposed methods only requires the partially relaxed strongly monotonicity of the operator, which is weaker than co-coercivity. From special cases, we obtain various known and new results for solving various classes of variational inequalities and related problems.AMS Subject Classification (1991): 49J40, 90C33.