Sampling Schemes for Bayesian Variable Selection in Generalized Linear Models

Bayesian approaches to prediction and the assessment of predictive uncertainty in generalized linear models are often based on averaging predictions over different models, and this requires methods for accounting for model uncertainty. When there are linear dependencies among potential predictor variables in a generalized linear model, existing Markov chain Monte Carlo algorithms for sampling from the posterior distribution on the model and parameter space in Bayesian variable selection problems may not work well. This article describes a sampling algorithm based on the Swendsen-Wang algorithm for the Ising model, and which works well when the predictors are far from orthogonality. In problems of variable selection for generalized linear models we can index different models by a binary parameter vector, where each binary variable indicates whether or not a given predictor variable is included in the model. The posterior distribution on the model is a distribution on this collection of binary strings, and by thinking of this posterior distribution as a binary spatial field we apply a sampling scheme inspired by the Swendsen-Wang algorithm for the Ising model in order to sample from the model posterior distribution. The algorithm we describe extends a similar algorithm for variable selection problems in linear models. The benefits of the algorithm are demonstrated for both real and simulated data.     

An efficient sampling scheme for dynamic generalized models

A multimove sampling scheme for the state parameters of non-Gaussian and nonlinear dynamic models for univariate time series is proposed. This procedure follows the Bayesian framework, within a Gibbs sampling algorithm with steps of the Metropolis–Hastings algorithm. This sampling scheme combines the conjugate updating approach for generalized dynamic linear models, with the backward sampling of the state parameters used in normal dynamic linear models. A quite extensive Monte Carlo study is conducted in order to compare the results obtained using our proposed method, conjugate updating backward sampling (CUBS), with those obtained using some algorithms previously proposed in the Bayesian literature. We compare the performance of CUBS with other sampling schemes using two real datasets. Then we apply our algorithm in a stochastic volatility model. CUBS significantly reduces the computing time needed to attain convergence of the chains, and is relatively simple to implement.     

Prediction of soil water retention curve using Bayesian updating from limited measurement data

A soil water retention curve is one of the fundamental elements used to describe unsaturated soil. The accurate determination of soil water retention curve requires sufficient available information. However, the amount of measurement data is generally limited due to the restriction of time or test apparatus. As a result, it is a challenge to determine the soil water retention curve from limited measurement data. To address this problem, a Bayesian framework is proposed. In the Bayesian framework, Bayesian updating can be employed using the posterior distribution that is obtained by the Markov chain Monte Carlo sampling method with the Delayed Rejection Adaptive Metropolis algorithm. The parameters of soil water retention curve model are represented by the sample statistics of updating posterior distribution. A new updating algorithm based on Bayesian framework is proposed to predict the soil water retention curve using the ideal data and the limited measurement data of the granite residual soil and sand. The results show that the proposed prediction algorithm exhibits an excellent capability for more accurately determining the soil water retention curve with limited measured data. The uncertainty of updating parameters and the influence of the prior knowledge can be reduced. The converged results can be derived using the proposed prediction algorithm even if the prior knowledge is incomplete.     

Bayesian analysis of Birnbaum–Saunders distribution via the generalized ratio-of-uniforms method

This paper deals with the Bayesian inference for the parameters of the Birnbaum–Saunders distribution. We adopt the inverse-gamma priors for the shape and scale parameters because the continuous conjugate joint prior distribution does not exist and the reference prior (or independent Jeffreys’ prior) results in an improper posterior distribution. We propose an efficient sampling algorithm via the generalized ratio-of-uniforms method to compute the Bayesian estimates and the credible intervals. One appealing advantage of the proposed procedure over other sampling techniques is that it efficiently generates independent samples from the required posterior distribution. Simulation studies are conducted to investigate the behavior of the proposed method, and two real-data applications are analyzed for illustrative purposes.     

A data assimilation process for linear ill‐posed problems

In this paper, an iteration process is considered to solve linear ill‐posed problems. Based on the randomness of the involved variables, this kind of problems is regarded as simulation problems of the posterior distribution of the unknown variable given the noise data. We construct a new ensemble Kalman filter‐based method to seek the posterior target distribution. Despite the ensemble Kalman filter method having widespread applications, there has been little analysis of its theoretical properties, especially in the field of inverse problems. This paper analyzes the propagation of the error with the iteration step for the proposed algorithm. The theoretical analysis shows that the proposed algorithm is convergence. We compare the numerical effect with the Bayesian inversion approach by two numerical examples: backward heat conduction problem and the first kind of integral equation. The numerical tests show that the proposed algorithm is effective and competitive with the Bayesian method. Copyright © 2017 John Wiley & Sons, Ltd.     

A Bayesian finite element model updating with combined normal and lognormal probability distributions using modal measurements

The present work is associated with Bayesian finite element (FE) model updating using modal measurements based on maximizing the posterior probability instead of any sampling based approach. Such Bayesian updating framework usually employs normal distribution in updating of parameters, although normal distribution has usual statistical issues while using non-negative parameters. These issues are proposed to be dealt with incorporating lognormal distribution for non-negative parameters. Detailed formulations are carried out for model updating, uncertainty-estimation and probabilistic detection of changes/damages of structural parameters using combined normal-lognormal probability distribution in this Bayesian framework. Normal and lognormal distributions are considered for eigen-system equation and structural (mass and stiffness) parameters respectively, while these two distributions are jointly considered for likelihood function. Important advantages in FE model updating (e.g. utilization of incomplete measured modal data, non-requirement of mode-matching) are also retained in this combined normal-lognormal distribution based proposed FE model updating approach. For demonstrating the efficiency of this proposed approach, a two dimensional truss structure is considered with multiple damage cases. Satisfactory performances are observed in model updating and subsequent probabilistic estimations, however level of performances are found to be weakened with increasing levels in damage scenario (as usual). Moreover, performances of this proposed FE model updating approach are compared with the typical normal distribution based updating approach for those damage cases demonstrating quite similar level of performances. The proposed approach also demonstrates better computational efficiency (achieving higher accuracy in lesser computation time) in comparison with two prominent Markov Chain Monte Carlo (MCMC) techniques (viz. Metropolis-Hastings algorithm and Gibbs sampling).     

Bayesian analysis for step‐stress accelerated life testing under progressive interval censoring

A step‐stress accelerated life testing model is considered for progressive type‐I censored experiments when the tested items are not monitored continuously but inspected at prespecified time points, producing thus grouped data. The underlying lifetime distributions belong to a general scale family of distributions. The points of stress‐level change are simultaneously inspection points as well while there is the option of assigning additional inspection points in between the stress‐level change points. In a Bayesian framework, the posterior distributions of the parameters of the model are derived for characteristic choices of prior distributions, as conjugate‐like and normal priors; vague or noninformative. The developed approach is illustrated on a simulated example and on a real data set, both known from the literature. The results are compared to previous analyses; frequentist or Bayes.     

Modelling non-stationary dynamic gene regulatory processes with the BGM model

Recently, a Bayesian network model for inferring non-stationary regulatory processes from gene expression time series has been proposed. The Bayesian Gaussian Mixture (BGM) Bayesian network model divides the data into disjunct compartments (data subsets) by a free allocation model, and infers network structures, which are kept fixed for all compartments. Fixing the network structure allows for some information sharing among compartments, and each compartment is modelled separately and independently with the Gaussian BGe scoring metric for Bayesian networks. The BGM model can equally be applied to both static (steady-state) and dynamic (time series) gene expression data. However, it is this flexibility that renders its application to time series data suboptimal. To improve the performance of the BGM model on time series data we propose a revised approach in which the free allocation of data points is replaced by a changepoint process so as to take the temporal structure into account. The practical inference follows the Bayesian paradigm and approximately samples the network, the number of compartments and the changepoint locations from the posterior distribution with Markov chain Monte Carlo (MCMC). Our empirical results show that the proposed modification leads to a more efficient inference tool for analysing gene expression time series.     

Posterior Simulation in the Generalized Linear Mixed Model With Semiparametric Random Effects

Generalized linear mixed models with semiparametric random effects are useful in a wide variety of Bayesian applications. When the random effects arise from a mixture of Dirichlet process (MDP) model with normal base measure, Gibbs samplingalgorithms based on the Pólya urn scheme are often used to simulate posterior draws in conjugate models (essentially, linear regression models and models for binary outcomes). In the nonconjugate case, some common problems associated with existing simulation algorithms include convergence and mixing difficulties.

This article proposes an algorithm for MDP models with exponential family likelihoods and normal base measures. The algorithm proceeds by making a Laplace approximation to the likelihood function, thereby matching the proposal with that of the Gibbs sampler. The proposal is accepted or rejected via a Metropolis-Hastings step. For conjugate MDP models, the algorithm is identical to the Gibbs sampler. The performance of the technique is investigated using a Poisson regression model with semi-parametric random effects. The algorithm performs efficiently and reliably, even in problems where large-sample results do not guarantee the success of the Laplace approximation. This is demonstrated by a simulation study where most of the count data consist of small numbers. The technique is associated with substantial benefits relative to existing methods, both in terms of convergence properties and computational cost.     

A simulation-based multivariate Bayesian control chart for real time condition-based maintenance of complex systems

When complex systems are monitored, multi-observations from several sensors or sources may be available. These observations can be fused through Bayesian theory to give a posterior probabilistic estimate of the underlying state which is often not directly observable. This forms the basis of a Bayesian control chart where the estimated posterior probability of the state can be compared with a preset threshold level to assess whether a full inspection is needed or not. Maintenance can then be carried out if indicated as necessary by the inspection. This paper considers the design of such multivariate Bayesian control chart where both the transition between states and the relationship between observed information and the state are not Markovian. Since analytical or numerical solutions are difficult for the case considered in this paper, Monte Carlo simulation is used to obtain the optimal control chart parameters, which are the monitoring interval and the upper control limit. A two-stage failure process characterised by the delay time concept is used to describe the underlying state transition process and Bayesian theory is used to compute the posterior probability of the underlying state, which is embedded in the simulation algorithm. Extensive examples are shown to demonstrate the modelling idea.     

Adaptive Sequential Monte Carlo for Multiple Changepoint Analysis

Process monitoring and control requires the detection of structural changes in a data stream in real time. This article introduces an efficient sequential Monte Carlo algorithm designed for learning unknown changepoints in continuous time. The method is intuitively simple: new changepoints for the latest window of data are proposed by conditioning only on data observed since the most recent estimated changepoint, as these observations carry most of the information about the current state of the process. The proposed method shows improved performance over the current state of the art. Another advantage of the proposed algorithm is that it can be made adaptive, varying the number of particles according to the apparent local complexity of the target changepoint probability distribution. This saves valuable computing time when changes in the changepoint distribution are negligible, and enables rebalancing of the importance weights of existing particles when a significant change in the target distribution is encountered. The plain and adaptive versions of the method are illustrated using the canonical continuous time changepoint problem of inferring the intensity of an inhomogeneous Poisson process, although the method is generally applicable to any changepoint problem. Performance is demonstrated using both conjugate and nonconjugate Bayesian models for the intensity. Appendices to the article are available online, illustrating the method on other models and applications.     

可变抽样区间几何EWMA控制图的经济设计

为了降低生产过程周期成本,本文对单位缺陷数服从几何分布时,可变抽样区间的指数加权移动平均(EWMA)控制图进行经济设计。首先建立可变抽样区间几何EWMA控制图的经济模型,使单位时间期望费用最小来确定参数的最优值;其次用遗传算法来寻找经济模型的最优解;最后对可变抽样区间几何EWMA控制图的经济模型进行灵敏度分析和最优性分析。研究结果表明单位时间期望费用分别随着异常原因发生的频率、过程失控时单位时间的质量费用、发现异常原因的时间期望值和纠正过程的时间期望值的增大而增大。     

基于预防维修和质量损失函数的VSI EWMA控制图经济设计

为了提高过程监控效率的同时降低过程控制成本,研究可变抽样区间(VSI)指数加权移动平均(EWMA)控制图的经济设计问题。首先建立基于预防维修和质量损失函数的VSI EWMA控制图联合经济模型;使单位时间的损失成本函数最小来确定参数的最优值;其次用遗传算法来寻找联合经济模型的最优解,并给出一个算例。最后对VSI EWMA控制图联合经济模型进行灵敏度分析,得出控制图模型参数对设计参数的影响关系。     

Bayesian variable selection in generalized linear models using a combination of stochastic optimization methods

In this paper the usage of a stochastic optimization algorithm as a model search tool is proposed for the Bayesian variable selection problem in generalized linear models. Combining aspects of three well known stochastic optimization algorithms, namely, simulated annealing, genetic algorithm and tabu search, a powerful model search algorithm is produced. After choosing suitable priors, the posterior model probability is used as a criterion function for the algorithm; in cases when it is not analytically tractable Laplace approximation is used. The proposed algorithm is illustrated on normal linear and logistic regression models, for simulated and real-life examples, and it is shown that, with a very low computational cost, it achieves improved performance when compared with popular MCMC algorithms, such as the MCMC model composition, as well as with “vanilla” versions of simulated annealing, genetic algorithm and tabu search.     

Hierarchical Bayesian autoregressive models for large space–time data with applications to ozone concentration modelling

Increasingly large volumes of space–time data are collected everywhere by mobile computing applications, and in many of these cases, temporal data are obtained by registering events, for example, telecommunication or Web traffic data. Having both the spatial and temporal dimensions adds substantial complexity to data analysis and inference tasks. The computational complexity increases rapidly for fitting Bayesian hierarchical models, as such a task involves repeated inversion of large matrices. The primary focus of this paper is on developing space–time autoregressive models under the hierarchical Bayesian setup. To handle large data sets, a recently developed Gaussian predictive process approximation method is extended to include autoregressive terms of latent space–time processes. Specifically, a space–time autoregressive process, supported on a set of a smaller number of knot locations, is spatially interpolated to approximate the original space–time process. The resulting model is specified within a hierarchical Bayesian framework, and Markov chain Monte Carlo techniques are used to make inference. The proposed model is applied for analysing the daily maximum 8‐h average ground level ozone concentration data from 1997 to 2006 from a large study region in the Eastern United States. The developed methods allow accurate spatial prediction of a temporally aggregated ozone summary, known as the primary ozone standard, along with its uncertainty, at any unmonitored location during the study period. Trends in spatial patterns of many features of the posterior predictive distribution of the primary standard, such as the probability of noncompliance with respect to the standard, are obtained and illustrated. Copyright © 2012 John Wiley & Sons, Ltd.     

An adaptive stochastic global optimization algorithm for one-dimensional functions

In this paper a new algorithm is proposed, based upon the idea of modeling the objective function of a global optimization problem as a sample path from a Wiener process. Unlike previous work in this field, in the proposed model the parameter of the Wiener process is considered as a random variable whose conditional (posterior) distribution function is updated on-line. Stopping criteria for Bayesian algorithms are discussed and detailed proofs on finite-time stopping are provided.This research has been partially supported by Progetto MURST 40% Metodi di Ottimizzazione per le Decisioni.     

Bayesian demand updating in the lost sales newsvendor problem: A two-moment approximation

We consider Bayesian updating of demand in a lost sales newsvendor model with censored observations. In a lost sales environment, where the arrival process is not recorded, the exact demand is not observed if it exceeds the beginning stock level, resulting in censored observations. Adopting a Bayesian approach for updating the demand distribution, we develop expressions for the exact posteriors starting with conjugate priors, for negative binomial, gamma, Poisson and normal distributions. Having shown that non-informative priors result in degenerate predictive densities except for negative binomial demand, we propose an approximation within the conjugate family by matching the first two moments of the posterior distribution. The conjugacy property of the priors also ensure analytical tractability and ease of computation in successive updates. In our numerical study, we show that the posteriors and the predictive demand distributions obtained exactly and with the approximation are very close to each other, and that the approximation works very well from both probabilistic and operational perspectives in a sequential updating setting as well.     

The Bayesian Change Point and Variable Selection Algorithm: Application to the δ18O Proxy Record of the Plio-Pleistocene

In this article, we introduce the Bayesian change point and variable selection algorithm that uses dynamic programming recursions to draw direct samples from a very high-dimensional space in a computationally efficient manner, and apply this algorithm to a geoscience problem that concerns the Earth's history of glaciation. Strong evidence exists for at least two changes in the behavior of the Earth's glaciers over the last five million years. Around 2.7 Ma, the extent of glacial cover on the Earth increased, but the frequency of glacial melting events remained constant at 41 kyr. A more dramatic change occurred around 1 Ma. For over three decades, the “Mid-Pleistocene Transition” has been described in the geoscience literature not only by a further increase in the magnitude of glacial cover, but also as the dividing point between the 41 kyr and the 100 kyr glacial worlds. Given such striking changes in the glacial record, it is clear that a model whose parameters can change through time is essential for the analysis of these data. The Bayesian change point algorithm provides a probabilistic solution to a data segmentation problem, while the exact Bayesian inference in regression procedure performs variable selection within each regime delineated by the change points. Together, they can model a time series in which the predictor variables as well as the parameters of the model are allowed to change with time. Our algorithm allows one to simultaneously perform variable selection and change point analysis in a computationally efficient manner. Supplementary materials including MATLAB code for the Bayesian change point and variable selection algorithm and the datasets described in this article are available online or by contacting the first author.     

Merging experts’ opinions: A Bayesian hierarchical model with mixture of prior distributions

In this paper, a general approach is proposed to address a full Bayesian analysis for the class of quadratic natural exponential families in the presence of several expert sources of prior information. By expressing the opinion of each expert as a conjugate prior distribution, a mixture model is used by the decision maker to arrive at a consensus of the sources. A hyperprior distribution on the mixing parameters is considered and a procedure based on the expected Kullback–Leibler divergence is proposed to analytically calculate the hyperparameter values. Next, the experts’ prior beliefs are calibrated with respect to the combined posterior belief over the quantity of interest by using expected Kullback–Leibler divergences, which are estimated with a computationally low-cost method. Finally, it is remarkable that the proposed approach can be easily applied in practice, as it is shown with an application.     

可变抽样区间的二项EWMA控制图

在制造过程中,对产品的不合格品数进行监控时,通常选用计数性控制图-np图,它是基于过程服从二项分布建立的,一般对于过程中出现的较大波动效果明显。为了提高控制图对不合格品数较小波动的监控效果,本文设计了产品不合格品数服从二项分布的EWMA控制图。提出可变抽样区间的二项EWMA控制图,并采用马可夫链法计算其平均报警时间。对固定抽样区间以及可变抽样区间二项EWMA控制图对比研究,表明当过程失控时,可变抽样区间二项EWMA控制图具有较小的失控平均报警时间,能够迅速监测出过程中的异常波动,明显优于固定抽样区间的二项EWMA控制图。