Bayesian Variable Selection via Particle Stochastic Search

We focus on Bayesian variable selection in regression models. One challenge is to search the huge model space adequately, while identifying high posterior probability regions. In the past decades, the main focus has been on the use of Markov chain Monte Carlo (MCMC) algorithms for these purposes. In this article, we propose a new computational approach based on sequential Monte Carlo (SMC), which we refer to as particle stochastic search (PSS). We illustrate PSS through applications to linear regression and probit models.     

Bayesian Synthetic Likelihood

Having the ability to work with complex models can be highly beneficial. However, complex models often have intractable likelihoods, so methods that involve evaluation of the likelihood function are infeasible. In these situations, the benefits of working with likelihood-free methods become apparent. Likelihood-free methods, such as parametric Bayesian indirect likelihood that uses the likelihood of an alternative parametric auxiliary model, have been explored throughout the literature as a viable alternative when the model of interest is complex. One of these methods is called the synthetic likelihood (SL), which uses a multivariate normal approximation of the distribution of a set of summary statistics. This article explores the accuracy and computational efficiency of the Bayesian version of the synthetic likelihood (BSL) approach in comparison to a competitor known as approximate Bayesian computation (ABC) and its sensitivity to its tuning parameters and assumptions. We relate BSL to pseudo-marginal methods and propose to use an alternative SL that uses an unbiased estimator of the SL, when the summary statistics have a multivariate normal distribution. Several applications of varying complexity are considered to illustrate the findings of this article. Supplemental materials are available online. Computer code for implementing the methods on all examples is available at https://github.com/cdrovandi/Bayesian-Synthetic-Likelihood.     

Model combination for credit risk assessment: A stacked generalization approach

The development of credit risk assessment models is often considered within a classification context. Recent studies on the development of classification models have shown that a combination of methods often provides improved classification results compared to a single-method approach. Within this context, this study explores the combination of different classification methods in developing efficient models for credit risk assessment. A variety of methods are considered in the combination, including machine learning approaches and statistical techniques. The results illustrate that combined models can outperform individual models for credit risk analysis. The analysis also covers important issues such as the impact of using different parameters for the combined models, the effect of attribute selection, as well as the effects of combining strong or weak models.     

Two-Stage Bayesian Approach for GWAS With Known Genealogy

Genome-wide association studies (GWAS) aim to assess relationships between single nucleotide polymorphisms (SNPs) and diseases. They are one of the most popular problems in genetics, and have some peculiarities given the large number of SNPs compared to the number of subjects in the study. Individuals might not be independent, especially in animal breeding studies or genetic diseases in isolated populations with highly inbred individuals. We propose a family-based GWAS model in a two-stage approach comprising a dimension reduction and a subsequent model selection. The first stage, in which the genetic relatedness between the subjects is taken into account, selects the promising SNPs. The second stage uses Bayes factors for comparison among all candidate models and a random search strategy for exploring the space of all the regression models in a fully Bayesian approach. A simulation study shows that our approach is superior to Bayesian lasso for model selection in this setting. We also illustrate its performance in a study on Beta-thalassemia disorder in an isolated population from Sardinia. Supplementary Material describing the implementation of the method proposed in this article is available online.     

Variational Approximation for Mixtures of Linear Mixed Models

Mixtures of linear mixed models (MLMMs) are useful for clustering grouped data and can be estimated by likelihood maximization through the Expectation–Maximization algorithm. A suitable number of components is then determined conventionally by comparing different mixture models using penalized log-likelihood criteria such as Bayesian information criterion. We propose fitting MLMMs with variational methods, which can perform parameter estimation and model selection simultaneously. We describe a variational approximation for MLMMs where the variational lower bound is in closed form, allowing for fast evaluation and develop a novel variational greedy algorithm for model selection and learning of the mixture components. This approach handles algorithm initialization and returns a plausible number of mixture components automatically. In cases of weak identifiability of certain model parameters, we use hierarchical centering to reparameterize the model and show empirically that there is a gain in efficiency in variational algorithms similar to that in Markov chain Monte Carlo (MCMC) algorithms. Related to this, we prove that the approximate rate of convergence of variational algorithms by Gaussian approximation is equal to that of the corresponding Gibbs sampler, which suggests that reparameterizations can lead to improved convergence in variational algorithms just as in MCMC algorithms. Supplementary materials for the article are available online.     

Bayesian Ensemble Trees (BET) for Clustering and Prediction in Heterogeneous Data

We propose a novel “tree-averaging” model that uses the ensemble of classification and regression trees (CART). Each constituent tree is estimated with a subset of similar data. We treat this grouping of subsets as Bayesian ensemble trees (BET) and model them as a Dirichlet process. We show that BET determines the optimal number of trees by adapting to the data heterogeneity. Compared with the other ensemble methods, BET requires much fewer trees and shows equivalent prediction accuracy using weighted averaging. Moreover, each tree in BET provides variable selection criterion and interpretation for each subset. We developed an efficient estimating procedure with improved estimation strategies in both CART and mixture models. We demonstrate these advantages of BET with simulations and illustrate the approach with a real-world data example involving regression of lung function measurements obtained from patients with cystic fibrosis. Supplementary materials for this article are available online.     

Adaptive Bayesian Nonstationary Modeling for Large Spatial Datasets Using Covariance Approximations

Gaussian process models have been widely used in spatial statistics but face tremendous modeling and computational challenges for very large nonstationary spatial datasets. To address these challenges, we develop a Bayesian modeling approach using a nonstationary covariance function constructed based on adaptively selected partitions. The partitioned nonstationary class allows one to knit together local covariance parameters into a valid global nonstationary covariance for prediction, where the local covariance parameters are allowed to be estimated within each partition to reduce computational cost. To further facilitate the computations in local covariance estimation and global prediction, we use the full-scale covariance approximation (FSA) approach for the Bayesian inference of our model. One of our contributions is to model the partitions stochastically by embedding a modified treed partitioning process into the hierarchical models that leads to automated partitioning and substantial computational benefits. We illustrate the utility of our method with simulation studies and the global Total Ozone Matrix Spectrometer (TOMS) data. Supplementary materials for this article are available online.     

Multivariate Adaptive Splines for Analysis of Longitudinal Data

Abstract

An essential feature of longitudinal data is the existence of autocorrelation among the observations from the same unit or subject. Two-stage random-effects linear models are commonly used to analyze longitudinal data. These models are not flexible enough, however, for exploring the underlying data structures and, especially, for describing time trends. Semi-parametric models have been proposed recently to accommodate general time trends. But these semi-parametric models do not provide a convenient way to explore interactions among time and other covariates although such interactions exist in many applications. Moreover, semi-parametric models require specifying the design matrix of the covariates (time excluded). We propose nonparametric models to resolve these issues. To fit nonparametric models, we use the novel technique of the multivariate adaptive regression splines for the estimation of mean curve and then apply an EM-like iterative procedure for covariance estimation. After giving a general algorithm of model building, we show how to design a fast algorithm. We use both simulated and published data to illustrate the use of our proposed method.     

Logarithmic Pooling of Priors Linked by a Deterministic Simulation Model

Abstract

We consider Bayesian inference when priors and likelihoods are both available for inputs and outputs of a deterministic simulation model. This problem is fundamentally related to the issue of aggregating (i.e., pooling) expert opinion. We survey alternative strategies for aggregation, then describe computational approaches for implementing pooled inference for simulation models. Our approach (1) numerically transforms all priors to the same space; (2) uses log pooling to combine priors; and (3) then draws standard Bayesian inference. We use importance sampling methods, including an iterative, adaptive approach that is more flexible and has less bias in some instances than a simpler alternative. Our exploratory examples are the first steps toward extension of the approach for highly complex and even noninvertible models.     

Learning the Structure of Mixed Graphical Models

We consider the problem of learning the structure of a pairwise graphical model over continuous and discrete variables. We present a new pairwise model for graphical models with both continuous and discrete variables that is amenable to structure learning. In previous work, authors have considered structure learning of Gaussian graphical models and structure learning of discrete models. Our approach is a natural generalization of these two lines of work to the mixed case. The penalization scheme involves a novel symmetric use of the group-lasso norm and follows naturally from a particular parameterization of the model. Supplementary materials for this article are available online.     

Bayesian Variable and Transformation Selection in Linear Regression

This article suggests a method for variable and transformation selection based on posterior probabilities. Our approach allows for consideration of all possible combinations of untransformed and transformed predictors along with transformed and untransformed versions of the response. To transform the predictors in the model, we use a change-point model, or “change-point transformation,” which can yield more interpretable models and transformations than the standard Box–Tidwell approach. We also address the problem of model uncertainty in the selection of models. By averaging over models, we account for the uncertainty inherent in inference based on a single model chosen from the set of models under consideration. We use a Markov chain Monte Carlo model composition (MC³) method which allows us to average over linear regression models when the space of models under consideration is very large. This considers the selection of variables and transformations at the same time. In an example, we show that model averaging improves predictive performance as compared with any single model that might reasonably be selected, both in terms of overall predictive score and of the coverage of prediction intervals. Software to apply the proposed methodology is available via StatLib.     

A Computationally Efficient Projection-Based Approach for Spatial Generalized Linear Mixed Models

Inference for spatial generalized linear mixed models (SGLMMs) for high-dimensional non-Gaussian spatial data is computationally intensive. The computational challenge is due to the high-dimensional random effects and because Markov chain Monte Carlo (MCMC) algorithms for these models tend to be slow mixing. Moreover, spatial confounding inflates the variance of fixed effect (regression coefficient) estimates. Our approach addresses both the computational and confounding issues by replacing the high-dimensional spatial random effects with a reduced-dimensional representation based on random projections. Standard MCMC algorithms mix well and the reduced-dimensional setting speeds up computations per iteration. We show, via simulated examples, that Bayesian inference for this reduced-dimensional approach works well both in terms of inference as well as prediction; our methods also compare favorably to existing “reduced-rank” approaches. We also apply our methods to two real world data examples, one on bird count data and the other classifying rock types. Supplementary material for this article is available online.     

Gaussian Variational Approximation With a Factor Covariance Structure

Variational approximations have the potential to scale Bayesian computations to large datasets and highly parameterized models. Gaussian approximations are popular, but can be computationally burdensome when an unrestricted covariance matrix is employed and the dimension of the model parameter is high. To circumvent this problem, we consider a factor covariance structure as a parsimonious representation. General stochastic gradient ascent methods are described for efficient implementation, with gradient estimates obtained using the so-called “reparameterization trick.” The end result is a flexible and efficient approach to high-dimensional Gaussian variational approximation. We illustrate using robust P-spline regression and logistic regression models. For the latter, we consider eight real datasets, including datasets with many more covariates than observations, and another with mixed effects. In all cases, our variational method provides fast and accurate estimates. Supplementary material for this article is available online.     

Structured additive regression for overdispersed and zero‐inflated count data

In count data regression there can be several problems that prevent the use of the standard Poisson log‐linear model: overdispersion, caused by unobserved heterogeneity or correlation, excess of zeros, non‐linear effects of continuous covariates or of time scales, and spatial effects. We develop Bayesian count data models that can deal with these issues simultaneously and within a unified inferential approach. Models for overdispersed or zero‐inflated data are combined with semiparametrically structured additive predictors, resulting in a rich class of count data regression models. Inference is fully Bayesian and is carried out by computationally efficient MCMC techniques. Simulation studies investigate performance, in particular how well different model components can be identified. Applications to patent data and to data from a car insurance illustrate the potential and, to some extent, limitations of our approach. Copyright © 2006 John Wiley & Sons, Ltd.     

A systematic approach for examining the impact of calibration uncertainty in disease modeling

In model-based analysis for comparative evaluation of strategies for disease treatment and management, the model of the disease is arguably the most critical element. A fundamental challenge in identifying model parameters arises from the limitations of available data, which challenges the ability to uniquely link model parameters to calibration targets. Consequently, the calibration of disease models leads to the discovery of multiple models that are similarly consistent with available data. This phenomenon is known as calibration uncertainty and its effect is transferred to the results of the analysis. Insufficient examination of the breadth of potential model parameters can create a false sense of confidence in the model recommendation, and ultimately cast doubt on the value of the analysis. This paper introduces a systematic approach to the examination of calibration uncertainty and its impact. We begin with a model of the calibration process as a constrained optimization problem and introduce the notion of plausible models which define the uncertainty region for model parameters. We illustrate the approach using a fictitious disease, and explore various methods for interpreting the outputs obtained.     

线性回归诊断的若干问题

本文对于线性回归诊断提出了几种新的模型和方法。我们首次研究了方差加权和均值漂移的混合模型,得到了相应的诊断统计量。本文还引入了罚函数方法,并以此为工具,讨论了若干有偏估计的影响度量,最后,本文提出了基于重心的诊断统计量,对于识别异常点有较好的效果。     

基于B样条和聚类分析的非参数轮廓监控方法研究

在轮廓监控中,产品或过程的质量特征可以由一种特定的函数关系表示。如果轮廓的函数形式是已知的,则可以使用参数化方法来监控轮廓。然而,当轮廓形态复杂时,继续使用参数方法则可能导致由于模型设定不准确而无法正确识别异常轮廓的问题。因此本文提出了一种基于非参数回归的新方法以解决制造过程中常见的复杂轮廓监控问题。所提方法将基于非参数回归的B样条与迭代的聚类分析过程相结合,在应用过程中不需要对轮廓的形式进行限制性假设。仿真研究评估了该监控方法在不同变异情况下的性能,并且通过与现有方法的比较分析,验证了该方法的有效性和优越性。最后通过轮廓监控领域的一个经典案例说明了新方法的实际应用效果。     

How Many Communities Are There?

Stochastic blockmodels and variants thereof are among the most widely used approaches to community detection for social networks and relational data. A stochastic blockmodel partitions the nodes of a network into disjoint sets, called communities. The approach is inherently related to clustering with mixture models; and raises a similar model selection problem for the number of communities. The Bayesian information criterion (BIC) is a popular solution, however, for stochastic blockmodels, the conditional independence assumption given the communities of the endpoints among different edges is usually violated in practice. In this regard, we propose composite likelihood BIC (CL-BIC) to select the number of communities, and we show it is robust against possible misspecifications in the underlying stochastic blockmodel assumptions. We derive the requisite methodology and illustrate the approach using both simulated and real data. Supplementary materials containing the relevant computer code are available online.     

Minimum classification error training in example based speech and pattern recognition using sparse weight matrices

The Minimum Classification Error (MCE) criterion is a well-known criterion in pattern classification systems. The aim of MCE training is to minimize the resulting classification error when trying to classify a new data set. Usually, these classification systems use some form of statistical model to describe the data. These systems usually do not work very well when this underlying model is incorrect. Speech recognition systems traditionally use Hidden Markov Models (HMM) with Gaussian (or Gaussian mixture) probability density functions as their basic model. It is well known that these models make some assumptions that are not correct. In example based approaches, these statistical models are absent and are replaced by the pure data. The absence of statistical models has created the need for parameters to model the data space accurately. For this work, we use the MCE criterion to create a system that is able to work together with this example based approach. Moreover, we extend the locally scaled distance measure with sparse, block diagonal weight matrices resulting in a better model for the data space and avoiding the computational load caused by using full matrices. We illustrate the approach with some example experiments on databases from pattern recognition and with speech recognition.     

Some remarks on Bayesian inference for one-way ANOVA models

We consider the standard one-way ANOVA model; it is well-known that classical statistical procedures are based on a scalar non-centrality parameter. In this paper we explore both marginal likelihood and integrated likelihood functions for this parameter and we show that they exactly lead to the same answer. On the other hand, we prove that a fully Bayesian testing procedure may provide different conclusions, depending on what is considered to be the real quantity of interest in the model or, said differently, which are the competing hypotheses. We illustrate these issues via a real data example.