Bayesian modeling of several covariance matrices and some results on propriety of the posterior for linear regression with correlated and/or heterogeneous errors

We explore simultaneous modeling of several covariance matrices across groups using the spectral (eigenvalue) decomposition and modified Cholesky decomposition. We introduce several models for covariance matrices under different assumptions about the mean structure. We consider ‘dependence’ matrices, which tend to have many parameters, as constant across groups and/or parsimoniously modeled via a regression formulation. For ‘variances’, we consider both unrestricted across groups and more parsimoniously modeled via log-linear models. In all these models, we explore the propriety of the posterior when improper priors are used on the mean and ‘variance’ parameters (and in some cases, on components of the ‘dependence’ matrices). The models examined include several common Bayesian regression models, whose propriety has not been previously explored, as special cases. We propose a simple approach to weaken the assumption of constant dependence matrices in an automated fashion and describe how to compute Bayes factors to test the hypothesis of constant ‘dependence’ across groups. The models are applied to data from two longitudinal clinical studies.     

Wishart ratios with dependent structure: New members of the bimatrix beta type IV

In multivariate statistics under normality, the problems of interest are random covariance matrices (known as Wishart matrices) and “ratios” of Wishart matrices that arise in multivariate analysis of variance (MANOVA) (see 24). The bimatrix variate beta type IV distribution (also known in the literature as bimatrix variate generalised beta; matrix variate generalization of a bivariate beta type I) arises from “ratios” of Wishart matrices. In this paper, we add a further independent Wishart random variate to the “denominator” of one of the ratios; this results in deriving the exact expression for the density function of the bimatrix variate extended beta type IV distribution. The latter leads to the proposal of the bimatrix variate extended F distribution. Some interesting characteristics of these newly introduced bimatrix distributions are explored. Lastly, we focus on the bivariate extended beta type IV distribution (that is an extension of bivariate Jones’ beta) with emphasis on P(X₁<X₂) where X₁ is the random stress variate and X₂ is the random strength variate.     

Estimation of multivariate normal covariance and precision matrices in a star-shape model with missing data

In this paper, we study the problem of estimating the covariance matrix Σ and the precision matrix Ω (the inverse of the covariance matrix) in a star-shape model with missing data. By considering a type of Cholesky decomposition of the precision matrix Ω=Ψ^′Ψ, where Ψ is a lower triangular matrix with positive diagonal elements, we get the MLEs of the covariance matrix and precision matrix and prove that both of them are biased. Based on the MLEs, unbiased estimators of the covariance matrix and precision matrix are obtained. A special group G, which is a subgroup of the group consisting all lower triangular matrices, is introduced. By choosing the left invariant Haar measure on G as a prior, we obtain the closed forms of the best equivariant estimates of Ω under any of the Stein loss, the entropy loss, and the symmetric loss. Consequently, the MLE of the precision matrix (covariance matrix) is inadmissible under any of the above three loss functions. Some simulation results are given for illustration.     

Marginal permutation invariant covariance matrices with applications to linear models

The goal of the present paper is to perform a comprehensive study of the covariance structures in balanced linear models containing random factors which are invariant with respect to marginal permutations of the random factors. We shall focus on model formulation and interpretation rather than the estimation of parameters. It is proven that permutation invariance implies a specific structure for the covariance matrices. Useful results are obtained for the spectra of permutation invariant covariance matrices. In particular, the reparameterization of random effects, i.e., imposing certain constraints, will be considered. There are many possibilities to choose reparameterization constraints in a linear model, however not every reparameterization keeps permutation invariance. The question is if there are natural restrictions on the random effects in a given model, i.e., such reparameterizations which are defined by the covariance structure of the corresponding factor. Examining relationships between the reparameterization conditions applied to the random factors of the models and the spectrum of the corresponding covariance matrices when permutation invariance is assumed, restrictions on the spectrum of the covariance matrix are obtained which lead to “sum-to-zero” reparameterization of the corresponding factor.     

Analysis of the limiting spectral distribution of large dimensional information-plus-noise type matrices

A derivation of results on the analytic behavior of the limiting spectral distribution of sample covariance matrices of the “information-plus-noise” type, as studied in Dozier and Silverstein [On the empirical distribution of eigenvalues of large dimensional information-plus-noise type matrices, 2004, submitted for publication], is presented. It is shown that, away from zero, the limiting distribution possesses a continuous density. The density is analytic where it is positive and, for the most relevant cases of a in the boundary of its support, exhibits behavior closely resembling that of for x near a. A procedure to determine its support is also analyzed.     

Multivariate Behrens-Fisher problem with missing data

Inference about the difference between two normal mean vectors when the covariance matrices are unknown and arbitrary is considered. Assuming that the incomplete data are of monotone pattern, a pivotal quantity, similar to the Hotelling T² statistic, is proposed. A satisfactory moment approximation to the distribution of the pivotal quantity is derived. Hypothesis testing and confidence estimation based on the approximate distribution are outlined. The accuracy of the approximation is investigated using Monte Carlo simulation. Monte Carlo studies indicate that the approximate method is very satisfactory even for moderately small samples. The proposed methods are illustrated using an example.     

Randomly generating portfolio-selection covariance matrices with specified distributional characteristics

In portfolio selection, there is often the need for procedures to generate “realistic” covariance matrices for security returns, for example to test and benchmark optimization algorithms. For application in portfolio optimization, such a procedure should allow the entries in the matrices to have distributional characteristics which we would consider “realistic” for security returns. Deriving motivation from the fact that a covariance matrix can be viewed as stemming from a matrix of factor loadings, a procedure is developed for the random generation of covariance matrices (a) whose off-diagonal (covariance) entries possess a pre-specified expected value and standard deviation and (b) whose main diagonal (variance) entries possess a likely different pre-specified expected value and standard deviation. The paper concludes with a discussion about the futility one would likely encounter if one simply tried to invent a valid covariance matrix in the absence of a procedure such as in this paper.     

On testing for clusters using the sample covariance

This paper analyzes the problem of using the sample covariance matrix to detect the presence of clustering in p-variate data in the special case when the component covariance matrices are known up to a constant multiplier. For the case of testing one population against a mixture of two populations, tests are derived and shown to be optimal in a certain sense. Some of their distribution properties are derived exactly. Some remarks on the extensions of these tests to mixtures of k ≤ p populations are included. The paper is essentially a formal treatment (in a special case) of some well-known procedures. The methods used in deriving the distribution properties are applicable to a variety of other situations involving mixtures.     

A contribution to multivariate L-moments: L-comoment matrices

Multivariate statistical analysis relies heavily on moment assumptions of second order and higher. With increasing interest in heavy-tailed distributions, however, it is desirable to describe dispersion, skewness, and kurtosis under merely first order moment assumptions. Here, the univariate L-moments of Hosking [L-moments: analysis and estimation of distributions using linear combinations of order statistics, J. Roy. Statist. Soc. Ser. B 52 (1990) 105-124] are extended to “L-comoments” analogous to covariance. For certain models, the second order case yields correlational analysis coherent with classical correlation but also meaningful under just first moment assumptions. We develop properties and estimators for L-comoments, illustrate for several multivariate models, examine behavior of sample multivariate L-moments with heavy-tailed data, and discuss applications to financial risk analysis and regional frequency analysis.     

Likelihood ratio tests of correlated multivariate samples

We develop methods to compare multiple multivariate normally distributed samples which may be correlated. The methods are new in the context that no assumption is made about the correlations among the samples. Three types of null hypotheses are considered: equality of mean vectors, homogeneity of covariance matrices, and equality of both mean vectors and covariance matrices. We demonstrate that the likelihood ratio test statistics have finite-sample distributions that are functions of two independent Wishart variables and dependent on the covariance matrix of the combined multiple populations. Asymptotic calculations show that the likelihood ratio test statistics converge in distribution to central Chi-squared distributions under the null hypotheses regardless of how the populations are correlated. Following these theoretical findings, we propose a resampling procedure for the implementation of the likelihood ratio tests in which no restrictive assumption is imposed on the structures of the covariance matrices. The empirical size and power of the test procedure are investigated for various sample sizes via simulations. Two examples are provided for illustration. The results show good performance of the methods in terms of test validity and power.     

On the comparison of the pre-test and shrinkage phi-divergence test estimators for the symmetry model of categorical data

The estimation problem of the parameters in a symmetry model for categorical data has been considered for many authors in the statistical literature (for example, Bowker (1948) [1], Ireland et al. (1969) [2], Quade and Salama (1975) [3], Cressie and Read (1988) [4], Menéndez et al. (2005) [5]) without using uncertain prior information. It is well known that many new and interesting estimators, using uncertain prior information, have been studied by a host of researchers in different statistical models, and many papers have been published on this topic (see Saleh (2006) [9] and references therein). In this paper, we consider the symmetry model of categorical data and we study, for the first time, some new estimators when non-sample information about the symmetry of the probabilities is considered. The decision to use a “restricted” estimator or an “unrestricted” estimator is based on the outcome of a preliminary test, and then a shrinkage technique is used. It is interesting to note that we present a unified study in the sense that we consider not only the maximum likelihood estimator and likelihood ratio test or chi-square test statistic but we consider minimum phi-divergence estimators and phi-divergence test statistics. Families of minimum phi-divergence estimators and phi-divergence test statistics are wide classes of estimators and test statistics that contain as a particular case the maximum likelihood estimator, likelihood ratio test and chi-square test statistic. In an asymptotic set-up, the biases and the risk under the squared loss function for the proposed estimators are derived and compared. A numerical example clarifies the content of the paper.     

Finite sample tail behavior of multivariate location estimators

A finite sample performance measure of multivariate location estimators is introduced based on “tail behavior”. The tail performance of multivariate “monotone” location estimators and the halfspace depth based “non-monotone” location estimators including the Tukey halfspace median and multivariate L-estimators is investigated. The connections among the finite sample performance measure, the finite sample breakdown point, and the halfspace depth are revealed. It turns out that estimators with high breakdown point or halfspace depth have “appealing” tail performance. The tail performance of the halfspace median is very appealing and also robust against underlying population distributions, while the tail performance of the sample mean is very sensitive to underlying population distributions. These findings provide new insights into the notions of the halfspace depth and breakdown point and identify the important role of tail behavior as a quantitative measure of robustness in the multivariate location setting.     

On improved estimation of normal precision matrix and discriminant coefficients

The problem of estimating the precision matrix of a multivariate normal distribution model is considered with respect to a quadratic loss function. A number of covariance estimators originally intended for a variety of loss functions are adapted so as to obtain alternative estimators of the precision matrix. It is shown that the alternative estimators have analytically smaller risks than the unbiased estimator of the precision matrix. Through numerical studies of risk values, it is shown that the new estimators have substantial reduction in risk. In addition, we consider the problem of the estimation of discriminant coefficients, which arises in linear discriminant analysis when Fisher's linear discriminant function is viewed as the posterior log-odds under the assumption that two classes differ in mean but have a common covariance matrix. The above method is also adapted for this problem in order to obtain improved estimators of the discriminant coefficients under the quadratic loss function. Furthermore, a numerical study is undertaken to compare the properties of a collection of alternatives to the “unbiased” estimator of the discriminant coefficients.     

Relative efficiencies of estimates using patterned covariances or correlations in the multivariate normal estimation problem

Summary The relative efficiency of maximum likelihood estimates is studied when taking advantage of underlying linear patterns in the covariances or correlations when estimating covariance matrices. We compare the variances of estimates of the covariance matrix obtained under two nested patterns with the assumption that the more restricted pattern is the true state. Formulas for the asymptotic variances are given which are exact for linear covariance patterns when explicit maximum likelihood estimates exist. Several specific examples are given using complete symmetry, circular symmetry and general covariance patterns as well as an example involving a covariance matrix with a linear pattern in the correlations.     

Multivariate versus univariate Kriging metamodels for multi-response simulation models

To analyze the input/output behavior of simulation models with multiple responses, we may apply either univariate or multivariate Kriging (Gaussian process) metamodels. In multivariate Kriging we face a major problem: the covariance matrix of all responses should remain positive-definite; we therefore use the recently proposed “nonseparable dependence” model. To evaluate the performance of univariate and multivariate Kriging, we perform several Monte Carlo experiments that simulate Gaussian processes. These Monte Carlo results suggest that the simpler univariate Kriging gives smaller mean square error.     

Estimation of means of multivariate normal populations with order restriction

Multivariate isotonic regression theory plays a key role in the field of statistical inference under order restriction for vector valued parameters. Two cases of estimating multivariate normal means under order restricted set are considered. One case is that covariance matrices are known, the other one is that covariance matrices are unknown but are restricted by partial order. This paper shows that when covariance matrices are known, the estimator given by this paper always dominates unrestricted maximum likelihood estimator uniformly, and when covariance matrices are unknown, the plug-in estimator dominates unrestricted maximum likelihood estimator under the order restricted set of covariance matrices. The isotonic regression estimators in this paper are the generalizations of plug-in estimators in unitary case.     

Estimation of high-dimensional prior and posterior covariance matrices in Kalman filter variants

This work studies the effects of sampling variability in Monte Carlo-based methods to estimate very high-dimensional systems. Recent focus in the geosciences has been on representing the atmospheric state using a probability density function, and, for extremely high-dimensional systems, various sample-based Kalman filter techniques have been developed to address the problem of real-time assimilation of system information and observations. As the employed sample sizes are typically several orders of magnitude smaller than the system dimension, such sampling techniques inevitably induce considerable variability into the state estimate, primarily through prior and posterior sample covariance matrices. In this article, we quantify this variability with mean squared error measures for two Monte Carlo-based Kalman filter variants: the ensemble Kalman filter and the ensemble square-root Kalman filter. Expressions of the error measures are derived under weak assumptions and show that sample sizes need to grow proportionally to the square of the system dimension for bounded error growth. To reduce necessary ensemble size requirements and to address rank-deficient sample covariances, covariance-shrinking (tapering) based on the Schur product of the prior sample covariance and a positive definite function is demonstrated to be a simple, computationally feasible, and very effective technique. Rules for obtaining optimal taper functions for both stationary as well as non-stationary covariances are given, and optimal taper lengths are given in terms of the ensemble size and practical range of the forecast covariance. Results are also presented for optimal covariance inflation. The theory is verified and illustrated with extensive simulations.     

Asymptotic distributions of two “synthetic data” estimators for censored single-index models

The censored single-index model provides a flexible way for modelling the association between a response and a set of predictor variables when the response variable is randomly censored and the link function is unknown. It presents a technique for “dimension reduction” in semiparametric censored regression models and generalizes the existing accelerated failure time models for survival analysis. This paper proposes two methods for estimation of single-index models with randomly censored samples. We first transform the censored data into synthetic data or pseudo-responses unbiasedly, then obtain estimates of the index coefficients by the rOPG or rMAVE procedures of Xia (2006) [1]. Finally, we estimate the unknown nonparametric link function using techniques for univariate censored nonparametric regression. The estimators for the index coefficients are shown to be root-n consistent and asymptotically normal. In addition, the estimator for the unknown regression function is a local linear kernel regression estimator and can be estimated with the same efficiency as the parameters are known. Monte Carlo simulations are conducted to illustrate the proposed methodologies.     

A note on a one-sided Jacobi algorithm

Summary We propose a one-sided or implicit variant of the Jacobi diagonalization algorithm for positive definite matrices. The variant is based on a previous Cholesky decomposition and currently uses essentially one square array which, on output, contains the matrix of eigenvectors thus reaching the storage economy of the symmetric QL algorithm. The current array is accessed only columnwise which makes the algorithm attractive for various parallelized and/or vectorized implementations. Even on a serial computer our algorithm shows improved efficiency, in particular if the Cholesky step is made with diagonal pivoting. On matrices of ordern=25–200 our algorithm is about 2–3.5 times slower than QL thus being almost on the halfway between the standard Jacobi and QL algorithms. The previous Cholesky decomposition can be performed with higher precision without extra time or storage costs thus offering considerable gains in accuracy with highly conditioned input matrices.This work was partly done during the first author's visit to the Department of Mathematics, The University of Tennessee-Knoxville, while participating in the Special Year on Numerical Linear Algebra sponsored by the UTK Departments of Computer Science and Mathematics, and the ORNL Mathematical Sciences Section, Engineering Physics and Mathematics Division as well as during a second author visit to the Fernuniversität Hagen. Both authors gratefully acknowledge the support of the respective institutions