High-Dimensional Mixed Graphical Models

While graphical models for continuous data (Gaussian graphical models) and discrete data (Ising models) have been extensively studied, there is little work on graphical models for datasets with both continuous and discrete variables (mixed data), which are common in many scientific applications. We propose a novel graphical model for mixed data, which is simple enough to be suitable for high-dimensional data, yet flexible enough to represent all possible graph structures. We develop a computationally efficient regression-based algorithm for fitting the model by focusing on the conditional log-likelihood of each variable given the rest. The parameters have a natural group structure, and sparsity in the fitted graph is attained by incorporating a group lasso penalty, approximated by a weighted lasso penalty for computational efficiency. We demonstrate the effectiveness of our method through an extensive simulation study and apply it to a music annotation dataset (CAL500), obtaining a sparse and interpretable graphical model relating the continuous features of the audio signal to binary variables such as genre, emotions, and usage associated with particular songs. While we focus on binary discrete variables for the main presentation, we also show that the proposed methodology can be easily extended to general discrete variables.     

Properties of graphical regression models for multidimensional categorical data

We propose a two-component graphical chain model, the discrete regression distribution, where a set of discrete random variables is modeled as a response to a set of categorical and continuous covariates. The proposed model is useful for modeling a set of discrete variables measured at multiple sites along with a set of continuous and/or discrete covariates. The proposed model allows for joint examination of the dependence structure of the discrete response and observed covariates and also accommodates site-to-site variability. We develop the graphical model properties and theoretical justifications of this model. Our model has several advantages over the traditional logistic normal model used to analyze similar compositional data, including site-specific random effect terms and the incorporation of discrete and continuous covariates.     

Modelling and inference with Conditional Gaussian Probabilistic Decision Graphs

Probabilistic Decision Graphs (PDGs) are probabilistic graphical models that represent a factorisation of a discrete joint probability distribution using a “decision graph”-like structure over local marginal parameters. The structure of a PDG enables the model to capture some context specific independence relations that are not representable in the structure of more commonly used graphical models such as Bayesian networks and Markov networks. This sometimes makes operations in PDGs more efficient than in alternative models. PDGs have previously been defined only in the discrete case, assuming a multinomial joint distribution over the variables in the model. We extend PDGs to incorporate continuous variables, by assuming a Conditional Gaussian (CG) joint distribution. We also show how inference can be carried out in an efficient way.     

Computational learning of the conditional phase-type (C-Ph) distribution

This paper presents a new algorithm for learning the structure of a special type of Bayesian network. The conditional phase-type (C-Ph) distribution is a Bayesian network that models the probabilistic causal relationships between a skewed continuous variable, modelled by the Coxian phase-type distribution, a special type of Markov model, and a set of interacting discrete variables. The algorithm takes a data set as input and produces the structure, parameters and graphical representations of the fit of the C-Ph distribution as output. The algorithm, which uses a greedy-search technique and has been implemented in MATLAB, is evaluated using a simulated data set consisting of 20,000 cases. The results show that the original C-Ph distribution is recaptured and the fit of the network to the data is discussed.     

Learning Interactions via Hierarchical Group-Lasso Regularization

We introduce a method for learning pairwise interactions in a linear regression or logistic regression model in a manner that satisfies strong hierarchy: whenever an interaction is estimated to be nonzero, both its associated main effects are also included in the model. We motivate our approach by modeling pairwise interactions for categorical variables with arbitrary numbers of levels, and then show how we can accommodate continuous variables as well. Our approach allows us to dispense with explicitly applying constraints on the main effects and interactions for identifiability, which results in interpretable interaction models. We compare our method with existing approaches on both simulated and real data, including a genome-wide association study, all using our R package glinternet.     

A Gaussian Mixed Model for Learning Discrete Bayesian Networks

In this paper, we address the problem of learning discrete Bayesian networks from noisy data. A graphical model based on a mixture of Gaussian distributions with categorical mixing structure coming from a discrete Bayesian network is considered. The network learning is formulated as a maximum likelihood estimation problem and performed by employing an EM algorithm. The proposed approach is relevant to a variety of statistical problems for which Bayesian network models are suitable—from simple regression analysis to learning gene/protein regulatory networks from microarray data.     

Asymptotic Bayesian structure learning using graph supports for Gaussian graphical models

The theory of Gaussian graphical models is a powerful tool for independence analysis between continuous variables. In this framework, various methods have been conceived to infer independence relations from data samples. However, most of them result in stepwise, deterministic, descent algorithms that are inadequate for solving this issue. More recent developments have focused on stochastic procedures, yet they all base their research on strong a priori knowledge and are unable to perform model selection among the set of all possible models. Moreover, convergence of the corresponding algorithms is slow, precluding applications on a large scale. In this paper, we propose a novel Bayesian strategy to deal with structure learning. Relating graphs to their supports, we convert the problem of model selection into that of parameter estimation. Use of non-informative priors and asymptotic results yield a posterior probability for independence graph supports in closed form. Gibbs sampling is then applied to approximate the full joint posterior density. We finally give three examples of structure learning, one from synthetic data, and the two others from real data.     

Bayesian MAP model selection of chain event graphs

Chain event graphs are graphical models that while retaining most of the structural advantages of Bayesian networks for model interrogation, propagation and learning, more naturally encode asymmetric state spaces and the order in which events happen than Bayesian networks do. In addition, the class of models that can be represented by chain event graphs for a finite set of discrete variables is a strict superset of the class that can be described by Bayesian networks. In this paper we demonstrate how with complete sampling, conjugate closed form model selection based on product Dirichlet priors is possible, and prove that suitable homogeneity assumptions characterise the product Dirichlet prior on this class of models. We demonstrate our techniques using two educational examples.     

Copula selection for graphical models in continuous Estimation of Distribution Algorithms

This paper presents the use of graphical models and copula functions in Estimation of Distribution Algorithms (EDAs) for solving multivariate optimization problems. It is shown in this work how the incorporation of copula functions and graphical models for modeling the dependencies among variables provides some theoretical advantages over traditional EDAs. By means of copula functions and two well known graphical models, this paper presents a novel approach for defining new EDAs. Either dependence is modeled by a copula function chosen from a predefined set of six functions that aim to cover a wide range of inter-relations. It is also shown how the use of mutual information in the learning of graphical models implies a natural way of employing copula entropies. The experimental results on separable and non-separable functions show that the two new EDAs, which adopt copula functions to model dependencies, perform better than their original version with Gaussian variables.     

Cycle-transitive comparison of independent random variables

The discrete dice model, previously introduced by the present authors, essentially amounts to the pairwise comparison of a collection of independent discrete random variables that are uniformly distributed on finite integer multisets. This pairwise comparison results in a probabilistic relation that exhibits a particular type of transitivity, called dice-transitivity. In this paper, the discrete dice model is generalized with the purpose of pairwisely comparing independent discrete or continuous random variables with arbitrary probability distributions. It is shown that the probabilistic relation generated by a collection of arbitrary independent random variables is still dice-transitive. Interestingly, this probabilistic relation can be seen as a graded alternative to the concept of stochastic dominance. Furthermore, when the marginal distributions of the random variables belong to the same parametric family of distributions, the probabilistic relation exhibits interesting types of isostochastic transitivity, such as multiplicative transitivity. Finally, the probabilistic relation generated by a collection of independent normal random variables is proven to be moderately stochastic transitive.     

Positivity for Gaussian graphical models

Gaussian graphical models are parametric statistical models for jointly normal random variables whose dependence structure is determined by a graph. In previous work, we introduced trek separation, which gives a necessary and sufficient condition in terms of the graph for when a subdeterminant is zero for all covariance matrices that belong to the Gaussian graphical model. Here we extend this result to give explicit cancellation-free formulas for the expansions of non-zero subdeterminants.     

Correlation when data are missing

Variable correlation is important for many operations research models. Manyinventory, revenue management, and queuing models presume uncorrelated demandbetween products, market segments, or time periods. The specific model applied,or the resulting policies of a model, can differ drastically depending onvariable correlation. Having missing data are a common problem for the realworld application of operations research models. This work is at the junction ofthe two topics of correlation and missing data. We propose a test ofindependence between two variables when data are missing. The typical method fordetermining correlation with missing data ignores all data pairs in which onepoint is missing. The test presented here incorporates all data. The test can beapplied when both variables are continuous, when both are discrete, or when onevariable is discrete and the other is continuous. The test makes no assumptionsabout the distribution of the two variables, and thus it can be used to extendapplication of non-parametric rank tests, such as Spearman's rankcorrelation, to the case where data are missing. An example is shown wherefailure to incorporate the incomplete data yields incorrect policies.     

Hybrid system identification using a structural approach and its model based control: An experimental validation

For implementation of various model based techniques such as in control and fault diagnosis, data-driven identification is key for enabling cheap and rapid development of models of hybrid systems of industrial interest. In the present work, a novel identification method is proposed for a class of hybrid systems which are linear and separable in the discrete variables (that is discrete states and discrete inputs). The method takes cognizance of the fact that the separable structure of the hybrid system constrains the evolution of system dynamics. In particular, the proposed method identifies models corresponding to a certain number of modes, far fewer than the total possible modes of the system. It then generates the models for the remaining modes without any further requirement for input–output data by exploiting the separable structure of the hybrid system. We experimentally validate the method by identifying the model for a three tank benchmark hybrid system followed by model predictive control using the identified model.     

Learning Bayesian networks from datasets joining continuous and discrete variables

The current paper addresses two problems observed in structure learning applications to computational biology.The first one is dealing with mixed data. Most optimization criteria for learning algorithms are applicable to either discrete or continuous data. Mixed datasets are usually handled by discretization of continuous data, which often leads to the loss of information. In order to address this problem, we adapted discrete scoring functions to continuous data. Consequently, the same score is used to both types of variables, and the network structure may be learned from mixed data directly.The second problem is the control of the type I error level. Usually, learning algorithms output a network that is the best according to some optimization criteria, but the reliability of particular relationships represented by this network is unknown. We address this problem by allowing the user to specify the expected error level and adjusting the parameters of the scoring criteria to this level.     

Precision Matrix Estimation by Inverse Principal Orthogonal Decomposition

We investigate the structure of a large precision matrix in Gaussian graphical models by decomposing it into a low rank component and a remainder part with sparse precision matrix.Based on the decomposition,we propose to estimate the large precision matrix by inverting a principal orthogonal decomposition(IPOD).The IPOD approach has appealing practical interpretations in conditional graphical models given the low rank component,and it connects to Gaussian graphical models with latent variables.Specifically,we show that the low rank component in the decomposition of the large precision matrix can be viewed as the contribution from the latent variables in a Gaussian graphical model.Compared with existing approaches for latent variable graphical models,the IPOD is conveniently feasible in practice where only inverting a low-dimensional matrix is required.To identify the number of latent variables,which is an objective of its own interest,we investigate and justify an approach by examining the ratios of adjacent eigenvalues of the sample covariance matrix?Theoretical properties,numerical examples,and a real data application demonstrate the merits of the IPOD approach in its convenience,performance,and interpretability.     

Discrete and continuous methods in ballistics

The kinematics of a projectile in flight provides an ideal opportunity for an introduction to (and a comparison between) discrete and continuous methods in applied mathematics. We use a graphical method in the discrete approach, which provides good physical insight and serves as an introduction to finite difference methods. The continuous approach is better in the no‐drag case, but the discrete approach is foundto be more effective when a nonlinear drag effect is included in the model.     

Efficient Bayesian Inference for Multivariate Probit Models With Sparse Inverse Correlation Matrices

We propose a Bayesian approach for inference in the multivariate probit model, taking into account the association structure between binary observations. We model the association through the correlation matrix of the latent Gaussian variables. Conditional independence is imposed by setting some off-diagonal elements of the inverse correlation matrix to zero and this sparsity structure is modeled using a decomposable graphical model. We propose an efficient Markov chain Monte Carlo algorithm relying on a parameter expansion scheme to sample from the resulting posterior distribution. This algorithm updates the correlation matrix within a simple Gibbs sampling framework and allows us to infer the correlation structure from the data, generalizing methods used for inference in decomposable Gaussian graphical models to multivariate binary observations. We demonstrate the performance of this model and of the Markov chain Monte Carlo algorithm on simulated and real datasets. This article has online supplementary materials.     

A PC algorithm variation for ordinal variables

Bayesian networks are graphical models that represent the joint distribution of a set of variables using directed acyclic graphs. The graph can be manually built by domain experts according to their knowledge. However, when the dependence structure is unknown (or partially known) the network has to be estimated from data by using suitable learning algorithms. In this paper, we deal with a constraint-based method to perform Bayesian networks structural learning in the presence of ordinal variables. We propose an alternative version of the PC algorithm, which is one of the most known procedures, with the aim to infer the network by accounting for additional information inherent to ordinal data. The proposal is based on a nonparametric test, appropriate for ordinal variables. A comparative study shows that, in some situations, the proposal discussed here is a slightly more efficient solution than the PC algorithm.     

Problems with the estimation of stochastic differential equations using structural equations models

The present paper deals with the identification and maximum likelihood estimation of systems of linear stochastic differential equations using panel data. So we only have a sample of discrete observations over time of the relevant variables for each individual. A popular approach in the social sciences advocates the estimation of the “exact discrete model” after a reparameterization with LISREL or similar programs for structural equations models. The “exact discrete model” corresponds to the continuous time model in the sense that observations at equidistant points in time that are generated by the latter system also satisfy the former. In the LISREL approach the reparameterized discrete time model is estimated first without taking into account the nonlinear mapping from the continuous to the discrete time parameters. In a second step, using the inverse mapping, the fundamental system parameters of the continuous time system in which we are interested, are inferred. However, some severe problems arise with this “indirect approach”. First, an identification problem may arise in multiple equation systems, since the matrix exponential function denning some of the new parameters is in general not one‐to‐one, and hence the inverse mapping mentioned above does not exist. Second, usually some sort of approximation of the time paths of the exogenous variables is necessary before the structural parameters of the system can be estimated with discrete data. Two simple approximation methods are discussed. In both approximation methods the resulting new discrete time parameters are connected in a complicated way. So estimating the reparameterized discrete model by OLS without restrictions does not yield maximum likelihood estimates of the desired continuous time parameters as claimed by some authors. Third, a further limitation of estimating the reparameterized model with programs for structural equations models is that even simple restrictions on the original fundamental parameters of the continuous time system cannot be dealt with. This issue is also discussed in some detail. For these reasons the “indirect method” cannot be recommended. In many cases the approach leads to misleading inferences. We strongly advocate the direct estimation of the continuous time parameters. This approach is more involved, because the exact discrete model is nonlinear in the original parameters. A computer program by Hermann Singer that provides appropriate maximum likelihood estimates is described.