Asymptotic Bayesian structure learning using graph supports for Gaussian graphical models

The theory of Gaussian graphical models is a powerful tool for independence analysis between continuous variables. In this framework, various methods have been conceived to infer independence relations from data samples. However, most of them result in stepwise, deterministic, descent algorithms that are inadequate for solving this issue. More recent developments have focused on stochastic procedures, yet they all base their research on strong a priori knowledge and are unable to perform model selection among the set of all possible models. Moreover, convergence of the corresponding algorithms is slow, precluding applications on a large scale. In this paper, we propose a novel Bayesian strategy to deal with structure learning. Relating graphs to their supports, we convert the problem of model selection into that of parameter estimation. Use of non-informative priors and asymptotic results yield a posterior probability for independence graph supports in closed form. Gibbs sampling is then applied to approximate the full joint posterior density. We finally give three examples of structure learning, one from synthetic data, and the two others from real data.     

Sparse principal component analysis via regularized low rank matrix approximation

Principal component analysis (PCA) is a widely used tool for data analysis and dimension reduction in applications throughout science and engineering. However, the principal components (PCs) can sometimes be difficult to interpret, because they are linear combinations of all the original variables. To facilitate interpretation, sparse PCA produces modified PCs with sparse loadings, i.e. loadings with very few non-zero elements. In this paper, we propose a new sparse PCA method, namely sparse PCA via regularized SVD (sPCA-rSVD). We use the connection of PCA with singular value decomposition (SVD) of the data matrix and extract the PCs through solving a low rank matrix approximation problem. Regularization penalties are introduced to the corresponding minimization problem to promote sparsity in PC loadings. An efficient iterative algorithm is proposed for computation. Two tuning parameter selection methods are discussed. Some theoretical results are established to justify the use of sPCA-rSVD when only the data covariance matrix is available. In addition, we give a modified definition of variance explained by the sparse PCs. The sPCA-rSVD provides a uniform treatment of both classical multivariate data and high-dimension-low-sample-size (HDLSS) data. Further understanding of sPCA-rSVD and some existing alternatives is gained through simulation studies and real data examples, which suggests that sPCA-rSVD provides competitive results.     

Clusters, outliers, and regression: fixed point clusters

Fixed point clustering is a new stochastic approach to cluster analysis. The definition of a single fixed point cluster (FPC) is based on a simple parametric model, but there is no parametric assumption for the whole dataset as opposed to mixture modeling and other approaches. An FPC is defined as a data subset that is exactly the set of non-outliers with respect to its own parameter estimators. This paper concentrates upon the theoretical foundation of FPC analysis as a method for clusterwise linear regression, i.e., the single clusters are modeled as linear regressions with normal errors. In this setup, fixed point clustering is based on an iteratively reweighted estimation with zero weight for all outliers. FPCs are non-hierarchical, but they may overlap and include each other. A specification of the number of clusters is not needed. Consistency results are given for certain mixture models of interest in cluster analysis. Convergence of a fixed point algorithm is shown. Application to a real dataset shows that fixed point clustering can highlight some other interesting features of datasets compared to maximum likelihood methods in the presence of deviations from the usual assumptions of model based cluster analysis.     

Some theoretical properties of Silverman's method for Smoothed functional principal component analysis

Principal component analysis (PCA) is one of the key techniques in functional data analysis. One important feature of functional PCA is that there is a need for smoothing or regularizing of the estimated principal component curves. Silverman’s method for smoothed functional principal component analysis is an important approach in a situation where the sample curves are fully observed due to its theoretical and practical advantages. However, lack of knowledge about the theoretical properties of this method makes it difficult to generalize it to the situation where the sample curves are only observed at discrete time points. In this paper, we first establish the existence of the solutions of the successive optimization problems in this method. We then provide upper bounds for the bias parts of the estimation errors for both eigenvalues and eigenfunctions. We also prove functional central limit theorems for the variation parts of the estimation errors. As a corollary, we give the convergence rates of the estimations for eigenvalues and eigenfunctions, where these rates depend on both the sample size and the smoothing parameters. Under some conditions on the convergence rates of the smoothing parameters, we can prove the asymptotic normalities of the estimations.     

The covariance structure and likelihood function for multivariate dyadic data

This paper extends the results in Li and Loken [A unified theory of statistical analysis and inference for variance component models for dyadic data, Statist. Sinica 12 (2002) 519-535] on the statistical analysis of measurements taken on dyads to the situations in which more than one attribute are measured on each dyad. Starting from the covariance structure for the univariate case obtained in Li and Loken (2002), the covariance structure for the multivariate case is derived based on the group symmetry induced by the assumed exchangeability in the units. Our primary objective is to document the Gaussian likelihood and the sufficient statistics for multivariate dyadic data in closed form, so that they can be referenced by researchers as they analyze those data. The derivation carried out can also serve as an example of multivariate extension of univariate models based on exchangeability.     

A note on bias due to fitting prospective multivariate generalized linear models to categorical outcomes ignoring retrospective sampling schemes

Outcome-dependent sampling designs are commonly used in economics, market research and epidemiological studies. Case-control sampling design is a classic example of outcome-dependent sampling, where exposure information is collected on subjects conditional on their disease status. In many situations, the outcome under consideration may have multiple categories instead of a simple dichotomization. For example, in a case-control study, there may be disease sub-classification among the “cases” based on progression of the disease, or in terms of other histological and morphological characteristics of the disease. In this note, we investigate the issue of fitting prospective multivariate generalized linear models to such multiple-category outcome data, ignoring the retrospective nature of the sampling design. We first provide a set of necessary and sufficient conditions for the link functions that will allow for equivalence of prospective and retrospective inference for the parameters of interest. We show that for categorical outcomes, prospective-retrospective equivalence does not hold beyond the generalized multinomial logit link. We then derive an approximate expression for the bias incurred when link functions outside this class are used. Most popular models for ordinal response fall outside the multiplicative intercept class and one should be cautious while performing a naive prospective analysis of such data as the bias could be substantial. We illustrate the extent of bias through a real data example, based on the ongoing Prostate, Lung, Colorectal and Ovarian (PLCO) cancer screening trial by the National Cancer Institute. The simulations based on the real study illustrate that the bias approximations work well in practice.     

Conditional independence models for seemingly unrelated regressions with incomplete data

We consider normal ≡ Gaussian seemingly unrelated regressions (SUR) with incomplete data (ID). Imposing a natural minimal set of conditional independence constraints, we find a restricted SUR/ID model whose likelihood function and parameter space factor into the product of the likelihood functions and the parameter spaces of standard complete data multivariate analysis of variance models. Hence, the restricted model has a unimodal likelihood and permits explicit likelihood inference. In the development of our methodology, we review and extend existing results for complete data SUR models and the multivariate ID problem.     

Asymptotic robustness of standard errors in multilevel structural equation models

Data in social and behavioral sciences are often hierarchically organized. Multilevel statistical methodology was developed to analyze such data. Most of the procedures for analyzing multilevel data are derived from maximum likelihood based on the normal distribution assumption. Standard errors for parameter estimates in these procedures are obtained from the corresponding information matrix. Because practical data typically contain heterogeneous marginal skewnesses and kurtoses, this paper studies how nonnormally distributed data affect the standard errors of parameter estimates in a two-level structural equation model. Specifically, we study how skewness and kurtosis in one level affect standard errors of parameter estimates within its level and outside its level. We also show that, parallel to asymptotic robustness theory in conventional factor analysis, conditions exist for asymptotic robustness of standard errors in a multilevel factor analysis model.     

Adequateness and interpretability of objective functions in ordinal data analysis

Objective functions that are applied in ordinal data analysis must be adequate, i.e. carefully adapted to the structure of the observed data. In addition, any analysis of data that is based upon objective functions must lead to interpretable results. After a general characterization of adequate objective functions in ordinal data analysis, therefore, the particular problems of constructing adequate and interpretable dissimilarity coefficients and correlation coefficients in ordinal data analysis, stress measures (stress functions) in non-metric scaling and generalized stress measures or correlation coefficients in any theory of rank estimation will be discussed.     

Hierarchical subspace models for contingency tables

For the statistical analysis of multiway contingency tables, we propose modeling interaction terms in each maximal compact component of a hierarchical model. By this approach we can search for parsimonious models with smaller degrees of freedom than the usual hierarchical model, while preserving the localization property of the inference in the hierarchical model. This approach also enables us to evaluate the localization property of a given log-affine model. We discuss estimation and exact tests of the proposed model and illustrate the advantage of the proposed modeling with some data sets.     

On the singularity of multivariate skew-symmetric models

In recent years, the skew-normal models introduced by Azzalini (1985) [1]-and their multivariate generalizations from Azzalini and Dalla Valle (1996) [4]-have enjoyed an amazing success, although an important literature has reported that they exhibit, in the vicinity of symmetry, singular Fisher information matrices and stationary points in the profile log-likelihood function for skewness, with the usual unpleasant consequences for inference. It has been shown (DiCiccio and Monti (2004) [23], DiCiccio and Monti (2009) [24] and Gómez et al. (2007) [25]) that these singularities, in some specific parametric extensions of skew-normal models (such as the classes of skew-t or skew-exponential power distributions), appear at skew-normal distributions only. Yet, an important question remains open: in broader semiparametric models of skewed distributions (such as the general skew-symmetric and skew-elliptical ones), which symmetric kernels lead to such singularities? The present paper provides an answer to this question. In very general (possibly multivariate) skew-symmetric models, we characterize, for each possible value of the rank of Fisher information matrices, the class of symmetric kernels achieving the corresponding rank. Our results show that, for strictly multivariate skew-symmetric models, not only Gaussian kernels yield singular Fisher information matrices. In contrast, we prove that systematic stationary points in the profile log-likelihood functions are obtained for (multi)normal kernels only. Finally, we also discuss the implications of such singularities on inference.     

Stochastic response restrictions

This paper considers the implementation of prior stochastic information on unknown outcomes of the response variables into estimation and forecasting of systems of linear regression equations in the context of time series, cross sections, pooled and longitudinal data models. The established approach proves particularly useful when only aggregated information on the response variables is available, as is frequently the case in applied statistics. We address the combination of prior stochastic and sample information as an extension of standard Gauss-Markov theory. Prior stochastic information could be given in the form of experts' expectations, or from estimations and/or projections of other models. A classical (i.e. non-Bayesian) regression framework for the incorporation of prior knowledge in generalized least-squares estimation and prediction is developed.     

Bayesian modeling of several covariance matrices and some results on propriety of the posterior for linear regression with correlated and/or heterogeneous errors

We explore simultaneous modeling of several covariance matrices across groups using the spectral (eigenvalue) decomposition and modified Cholesky decomposition. We introduce several models for covariance matrices under different assumptions about the mean structure. We consider ‘dependence’ matrices, which tend to have many parameters, as constant across groups and/or parsimoniously modeled via a regression formulation. For ‘variances’, we consider both unrestricted across groups and more parsimoniously modeled via log-linear models. In all these models, we explore the propriety of the posterior when improper priors are used on the mean and ‘variance’ parameters (and in some cases, on components of the ‘dependence’ matrices). The models examined include several common Bayesian regression models, whose propriety has not been previously explored, as special cases. We propose a simple approach to weaken the assumption of constant dependence matrices in an automated fashion and describe how to compute Bayes factors to test the hypothesis of constant ‘dependence’ across groups. The models are applied to data from two longitudinal clinical studies.     

Predictivistic characterizations of multivariate student-t models

De Finetti style theorems characterize models (predictive distributions) as mixtures of the likelihood function and the prior distribution, beginning from some judgment of invariance about observable quantities. The likelihood function generally has its functional form identified from invariance assumptions only. However, we need additional conditions on observable quantities (typically, assumptions on conditional expectations) to identify the prior distribution. In this paper, we consider some well-known invariance assumptions and establish additional conditions on observable quantities in order to obtain a predictivistic characterization of the multivariate and matrix-variate Student-t distributions as well as for the Student-t linear model. As a byproduct, a characterization for the Pearson type II distribution is provided.     

Grouped Dirichlet distribution: A new tool for incomplete categorical data analysis

Motivated by the likelihood functions of several incomplete categorical data, this article introduces a new family of distributions, grouped Dirichlet distributions (GDD), which includes the classical Dirichlet distribution (DD) as a special case. First, we develop distribution theory for the GDD in its own right. Second, we use this expanded family as a new tool for statistical analysis of incomplete categorical data. Starting with a GDD with two partitions, we derive its stochastic representation that provides a simple procedure for simulation. Other properties such as mixed moments, mode, marginal and conditional distributions are also derived. The general GDD with more than two partitions is considered in a parallel manner. Three data sets from a case-control study, a leprosy survey, and a neurological study are used to illustrate how the GDD can be used as a new tool for analyzing incomplete categorical data. Our approach based on GDD has at least two advantages over the commonly used approach based on the DD in both frequentist and conjugate Bayesian inference: (a) in some cases, both the maximum likelihood and Bayes estimates have closed-form expressions in the new approach, but not so when they are based on the commonly-used approach; and (b) even if a closed-form solution is not available, the EM and data augmentation algorithms in the new approach converge much faster than in the commonly-used approach.     

Symmetrised M-estimators of multivariate scatter

In this paper we introduce a family of symmetrised M-estimators of multivariate scatter. These are defined to be M-estimators only computed on pairwise differences of the observed multivariate data. Symmetrised Huber's M-estimator and Dümbgen's estimator serve as our examples. The influence functions of the symmetrised M-functionals are derived and the limiting distributions of the estimators are discussed in the multivariate elliptical case to consider the robustness and efficiency properties of estimators. The symmetrised M-estimators have the important independence property; they can therefore be used to find the independent components in the independent component analysis (ICA).     

Asymptotic normality of test statistics under alternative hypotheses

The aim of this paper is to present a framework for asymptotic analysis of likelihood ratio and minimum discrepancy test statistics. First order asymptotics are presented in a general framework under minimal regularity conditions and for not necessarily nested models. In particular, these asymptotics give sufficient and in a sense necessary conditions for asymptotic normality of test statistics under alternative hypotheses. Second order asymptotics, and their implications for bias corrections, are also discussed in a somewhat informal manner. As an example, asymptotics of test statistics in the analysis of covariance structures are discussed in detail.     

Some data-analytic modifications to Bayes-Stein estimation

Summary The usual Bayes-Stein shrinkages of maximum likelihood estimates towards a common value may be refined by taking fuller account of the locations of the individual observations. Under a Bayesian formulation, the types of shrinkages depend critically upon the nature of the common distribution assumed for the parameters at the second stage of the prior model. In the present paper this distribution is estimated empirically from the data, permitting the data to determine the nature of the shrinkages. For example, when the observations are located in two or more clearly distinct groups, the maximum likelihood estimates are roughly speaking constrained towards common values within each group. The method also detects outliers; an extreme observation will either the regarded as an outlier and not substantially adjusted towards the other observations, or it will be rejected as an outlier, in which case a more radical adjustment takes place. The method is appropriate for a wide range of sampling distributions and may also be viewed as an alternative to standard multiple comparisons, cluster analysis, and nonparametric kernel methods.     

A contribution to multivariate L-moments: L-comoment matrices

Multivariate statistical analysis relies heavily on moment assumptions of second order and higher. With increasing interest in heavy-tailed distributions, however, it is desirable to describe dispersion, skewness, and kurtosis under merely first order moment assumptions. Here, the univariate L-moments of Hosking [L-moments: analysis and estimation of distributions using linear combinations of order statistics, J. Roy. Statist. Soc. Ser. B 52 (1990) 105-124] are extended to “L-comoments” analogous to covariance. For certain models, the second order case yields correlational analysis coherent with classical correlation but also meaningful under just first moment assumptions. We develop properties and estimators for L-comoments, illustrate for several multivariate models, examine behavior of sample multivariate L-moments with heavy-tailed data, and discuss applications to financial risk analysis and regional frequency analysis.     

Power of edge exclusion tests for graphical log-linear models

Asymptotic multivariate normal approximations to the joint distributions of edge exclusion test statistics for saturated graphical log-linear models, with all variables binary, are derived. Non-signed and signed square-root versions of the likelihood ratio, Wald and score test statistics are considered. Non-central chi-squared approximations are also considered for the non-signed versions of the test statistics. Simulation results are used to assess the quality of the proposed approximations. These approximations are used to estimate the overall power of edge exclusion tests. Power calculations are illustrated using data on university admissions.