Rejoinder on “Imprecise probability models for learning multinomial distributions from data. Applications to learning credal networks”

In this paper we answer to the comments provided by Fabio Cozman, Marco Zaffalon, Giorgio Corani, and Didier Dubois on our paper ‘Imprecise Probability Models for Learning Multinomial Distributions from Data. Applications to Learning Credal Networks’. The main topics we have considered are: regularity, the learning principle, the trade-off between prior imprecision and learning, strong symmetry, and the properties of ISSDM for learning graphical conditional independence models.     

A tree augmented classifier based on Extreme Imprecise Dirichlet Model

We present TANC, a TAN classifier (tree-augmented naive) based on imprecise probabilities. TANC models prior near-ignorance via the Extreme Imprecise Dirichlet Model (EDM). A first contribution of this paper is the experimental comparison between EDM and the global Imprecise Dirichlet Model using the naive credal classifier (NCC), with the aim of showing that EDM is a sensible approximation of the global IDM. TANC is able to deal with missing data in a conservative manner by considering all possible completions (without assuming them to be missing-at-random), but avoiding an exponential increase of the computational time. By experiments on real data sets, we show that TANC is more reliable than the Bayesian TAN and that it provides better performance compared to previous TANs based on imprecise probabilities. Yet, TANC is sometimes outperformed by NCC because the learned TAN structures are too complex; this calls for novel algorithms for learning the TAN structures, better suited for an imprecise probability classifier.     

Learning imprecise probability models: Conceptual and practical challenges

The paper by Masegosa and Moral, on “Imprecise probability models for learning multinomial distributions from data”, considers the combination of observed data and minimal prior assumptions so as to produce possibly interval-valued parameter estimates. We offer an evaluation of Masegosa and Moral's proposals.     

New prior near-ignorance models on the simplex

The aim of this paper is to derive new near-ignorance models on the probability simplex, which do not directly involve the Dirichlet distribution and, thus, are alternative to the Imprecise Dirichlet Model (IDM). We focus our investigation on a particular class of distributions on the simplex which is known as the class of Normalized Infinitely Divisible (NID) distributions; it includes the Dirichlet distribution as a particular case. For this class it is possible to derive general formulae for prior and posterior predictive inferences, by exploiting the Lévy–Khintchine representation theorem. This allows us to generally characterize the near-ignorance properties of the NID class. After deriving these general properties, we focus our attention on three members of this class. We will show that one of these near-ignorance models satisfies the representation invariance principle and, for a given value of the prior strength, always provides inferences that encompass those of the IDM. The other two models do not satisfy this principle, but their imprecision depends linearly or almost linearly on the number of observed categories; we argue that this is sometimes a desirable property for a predictive model.     

Infinite Dirichlet mixture models learning via expectation propagation

In this article, we propose a novel Bayesian nonparametric clustering algorithm based on a Dirichlet process mixture of Dirichlet distributions which have been shown to be very flexible for modeling proportional data. The idea is to let the number of mixture components increases as new data to cluster arrive in such a manner that the model selection problem (i.e. determination of the number of clusters) can be answered without recourse to classic selection criteria. Thus, the proposed model can be considered as an infinite Dirichlet mixture model. An expectation propagation inference framework is developed to learn this model by obtaining a full posterior distribution on its parameters. Within this learning framework, the model complexity and all the involved parameters are evaluated simultaneously. To show the practical relevance and efficiency of our model, we perform a detailed analysis using extensive simulations based on both synthetic and real data. In particular, real data are generated from three challenging applications namely images categorization, anomaly intrusion detection and videos summarization.     

A Gaussian Mixed Model for Learning Discrete Bayesian Networks

In this paper, we address the problem of learning discrete Bayesian networks from noisy data. A graphical model based on a mixture of Gaussian distributions with categorical mixing structure coming from a discrete Bayesian network is considered. The network learning is formulated as a maximum likelihood estimation problem and performed by employing an EM algorithm. The proposed approach is relevant to a variety of statistical problems for which Bayesian network models are suitable—from simple regression analysis to learning gene/protein regulatory networks from microarray data.     

多项分布与Dirichlet分布概率的相互表示

讨论了四种多项分布尾概率与四种Dirichlet分布尾概率的相互表示,并将结果应用于Majorization理论,得到了多项分布和Dirichlet分布对应的一些性质.同时,结果可应用于多项分布最大、最小参数的贝叶斯推断,在佛罗里达州沃尔顿县白人和黑人的职业状态调查结果中,求出最大、最小参数的后验分布函数以及95%贝叶斯区间估计,模拟结果表明提供的方法具有较好的表现.     

Bayesian MAP model selection of chain event graphs

Chain event graphs are graphical models that while retaining most of the structural advantages of Bayesian networks for model interrogation, propagation and learning, more naturally encode asymmetric state spaces and the order in which events happen than Bayesian networks do. In addition, the class of models that can be represented by chain event graphs for a finite set of discrete variables is a strict superset of the class that can be described by Bayesian networks. In this paper we demonstrate how with complete sampling, conjugate closed form model selection based on product Dirichlet priors is possible, and prove that suitable homogeneity assumptions characterise the product Dirichlet prior on this class of models. We demonstrate our techniques using two educational examples.     

Learning failure-free PRISM programs

PRISM is a probabilistic logic programming formalism which allows defining a probability distribution over possible worlds. This paper investigates learning a class of generative PRISM programs known as failure-free. The aim is to learn recursive PRISM programs which can be used to model stochastic processes. These programs generalise dynamic Bayesian networks by defining a halting distribution over the generative process. Dynamic Bayesian networks model infinite stochastic processes. Sampling from infinite process can only be done by specifying the length of sequences that the process generates. In this case, only observations of a fixed length of sequences can be obtained. On the other hand, the recursive PRISM programs considered in this paper are self-terminating upon some halting conditions. Thus, they generate observations of different lengths of sequences. The direction taken by this paper is to combine ideas from inductive logic programming and learning Bayesian networks to learn PRISM programs. It builds upon the inductive logic programming approach of learning from entailment.     

多级评分及其Bayes估计

对多级评分的测验题型 ,给出了其Bayes模型 ,在无信息先验分布或先验分布是Dirichlet分布情形下求出了参数的Bayes估计 ,并对后者在不同样本条件下给出了先验分布超参数的估计     

A Bayesian approach for quantile and response probability estimation with applications to reliability

In this paper we propose a Bayesian approach for the estimation of a potency curve which is assumed to be nondecreasing and concave or convex. This is done by assigning the Dirichlet as a prior distribution for transformations of some unknown parameters. We motivate our choice of the prior and investigate several aspects of the problem, including the numerical implementation of the suggested scheme. An approach for estimating the quantiles is also given. By casting the problem in a more general context, we argue that distributions which are IHR or IHRA can also be estimated via the suggested procedure. A problem from a government laboratory serves as an example to illustrate the use of our procedure in a realistic scenario.     

Optimal selection from distributions with unknown parameters: Robustness of Bayesian models

We consider the problem of making one choice from a known number of i.i.d. alternatives. It is assumed that the distribution of the alternatives has some unknown parameter. We follow a Bayesian approach to maximize the discounted expected value of the chosen alternative minus the costs for the observations. For the case of gamma and normal distribution we investigate the sensitivity of the solution with respect to the prior distributions. Our main objective is to derive monotonicity and continuity results for the dependence on parameters of the prior distributions. Thus we prove some sort of Bayesian robustness of the model.     

混合狄雷克利分布和高维表的贝叶斯估计

本文讨论多项分布情况下的高维列联表使用混合狄雷克利分布为先验分布时，贝叶斯估计的表达，以及独立性条件的表述．将文献［４］和［５］的结论推广到高维列联表中．     

Modelling and inference with Conditional Gaussian Probabilistic Decision Graphs

Probabilistic Decision Graphs (PDGs) are probabilistic graphical models that represent a factorisation of a discrete joint probability distribution using a “decision graph”-like structure over local marginal parameters. The structure of a PDG enables the model to capture some context specific independence relations that are not representable in the structure of more commonly used graphical models such as Bayesian networks and Markov networks. This sometimes makes operations in PDGs more efficient than in alternative models. PDGs have previously been defined only in the discrete case, assuming a multinomial joint distribution over the variables in the model. We extend PDGs to incorporate continuous variables, by assuming a Conditional Gaussian (CG) joint distribution. We also show how inference can be carried out in an efficient way.     

Inference of an oscillating model for the yeast cell cycle

High-throughput techniques allow measurement of hundreds of cell components simultaneously. The inference of interactions between cell components from these experimental data facilitates the understanding of complex regulatory processes. Differential equations have been established to model the dynamic behavior of these regulatory networks quantitatively. Usually traditional regression methods for estimating model parameters fail in this setting, since they overfit the data. This is even the case, if the focus is on modeling subnetworks of, at most, a few tens of components. In a Bayesian learning approach, this problem is avoided by a restriction of the search space with prior probability distributions over model parameters.This paper combines both differential equation models and a Bayesian approach. We model the periodic behavior of proteins involved in the cell cycle of the budding yeast Saccharomyces cerevisiae, with differential equations, which are based on chemical reaction kinetics. One property of these systems is that they usually converge to a steady state, and lots of efforts have been made to explain the observed periodic behavior. We introduce an approach to infer an oscillating network from experimental data. First, an oscillating core network is learned. This is extended by further components by using a Bayesian approach in a second step. A specifically designed hierarchical prior distribution over interaction strengths prevents overfitting, and drives the solutions to sparse networks with only a few significant interactions.We apply our method to a simulated and a real world dataset and reveal main regulatory interactions. Moreover, we are able to reconstruct the dynamic behavior of the network.     

A semiparametric Bayesian approach to the analysis of financial time series with applications to value at risk estimation

GARCH models are commonly used for describing, estimating and predicting the dynamics of financial returns. Here, we relax the usual parametric distributional assumptions of GARCH models and develop a Bayesian semiparametric approach based on modeling the innovations using the class of scale mixtures of Gaussian distributions with a Dirichlet process prior on the mixing distribution. The proposed specification allows for greater flexibility in capturing the usual patterns observed in financial returns. It is also shown how to undertake Bayesian prediction of the Value at Risk (VaR). The performance of the proposed semiparametric method is illustrated using simulated and real data from the Hang Seng Index (HSI) and Bombay Stock Exchange index (BSE30).     

Nonparametric Bayesian Modeling for Stochastic Order

In comparing two populations, sometimes a model incorporating stochastic order is desired. Customarily, such modeling is done parametrically. The objective of this paper is to formulate nonparametric (possibly semiparametric) stochastic order specifications providing richer, more flexible modeling. We adopt a fully Bayesian approach using Dirichlet process mixing. An attractive feature of the Bayesian approach is that full inference is available regarding the population distributions. Prior information can conveniently be incorporated. Also, prior stochastic order is preserved to the posterior analysis. Apart from the two sample setting, the approach handles the matched pairs problem, the k-sample slippage problem, ordered ANOVA and ordered regression models. We illustrate by comparing two rather small samples, one of diabetic men, the other of diabetic women. Measurements are of androstenedione levels. Males are anticipated to produce levels which will tend to be higher than those of females.     

A bayes procedure for selecting the population with the largestpth quantile

Summary The Bayes method is seldom applied to nonparametric statistical problems, for the reason that it is hard to find mathematically tractable prior distributions on a set of probability measures. However, it is found that the Dirichlet process generates randomly a family of probability distributions which can be taken as a family of prior distributions for an application of the Bayes method to such problems. This paper presents a Bayesian analysis of a nonparametric problem of selecting a distribution with the largestpth quantile value, fromk≧2 given distributions. It is assumed a priori that the given distributions have been generated from a Dirichlet process. This work was supported by the U.S. Office of Naval Research under Contract No. 00014-75-C-0451.     

Locally averaged Bayesian Dirichlet metrics for learning the structure and the parameters of Bayesian networks

The marginal likelihood of the data computed using Bayesian score metrics is at the core of score+search methods when learning Bayesian networks from data. However, common formulations of those Bayesian score metrics rely on free parameters which are hard to assess. Recent theoretical and experimental works have also shown that the commonly employed BDe score metric is strongly biased by the particular assignments of its free parameter known as the equivalent sample size. This sensitivity means that poor choices of this parameter lead to inferred BN models whose structure and parameters do not properly represent the distribution generating the data even for large sample sizes. In this paper we argue that the problem is that the BDe metric is based on assumptions about the BN model parameters distribution assumed to generate the data which are too strict and do not hold in real settings. To overcome this issue we introduce here an approach that tries to marginalize the meta-parameter locally, aiming to embrace a wider set of assumptions about these parameters. It is shown experimentally that this approach offers a robust performance, as good as that of the standard BDe metric with an optimum selection of its free parameter and, in consequence, this method prevents the choice of wrong settings for this widely applied Bayesian score metric.