Computing p-values in conditional independence models for a contingency table

We present a Markov chain Monte Carlo (MCMC) method for generating Markov chains using Markov bases for conditional independence models for a four-way contingency table. We then describe a Markov basis characterized by Markov properties associated with a given conditional independence model and show how to use the Markov basis to generate random tables of a Markov chain. The estimates of exact p-values can be obtained from random tables generated by the MCMC method. Numerical experiments examine the performance of the proposed MCMC method in comparison with the χ ² approximation using large sparse contingency tables.     

A markov chain sampler for contingency table exact inference

Summary  In the inference of contingency table, when the cell counts are not large enough for asymptotic approximation, conditioning exact method is used and often computationally impractical for large tables. Instead, various sampling methods can be used. Based on permutation, the Monte Carlo sampling may become again impractical for large tables. For this, existing the Markov chain method is to sample a few elements of the table at each iteration and is inefficient. Here we consider a Markov chain, in which a sub-table of user specified size is updated at each iteration, and it achieves high sampling efficiency. Some theoretical properties of the chain and its applications to some commonly used tables are discussed. As an illustration, this method is applied to the exact test of the Hardy-Weinberg equilibrium in the population genetics context.     

A Markov chain Monte Carlo Algorithm for Approximating Exact Conditional Probabilities

Conditional inference eliminates nuisance parameters by conditioning on their sufficient statistics. For contingency tables conditional inference entails enumerating all tables with the same sufficient statistics as the observed data. For moderately sized tables and/or complex models, the computing time to enumerate these tables is often prohibitive. Monte Carlo approximations offer a viable alternative provided it is possible to obtain samples from the correct conditional distribution. This article presents an MCMC extension of the importance sampling algorithm, using a rounded normal candidate to update randomly chosen cells while leaving the remainder of the table fixed. This local approximation can greatly increase the efficiency of the rounded normal candidate. By choosing the number of cells to be updated at random, a balance is struck between dependency in the Markov chain and accuracy of the candidate.     

Efficiency of Monte Carlo computations in very high dimensional spaces

A standard measure for comparing different Monte Carlo estimators is the efficiency, which generally thought to be declining with increasing the number of dimensions. Here we give some numerical examples, ranging from one-hundred to one-thousand dimensional integration problems, that contradict this belief. Monte Carlo integrations carried out in one-thousand dimensional spaces is the other nontrivial result reported here. The examples concern the computation of the probabilities of convex sets (polyhedra and hyperellipsoids) in case of multidimensional normal probabilities.     

Exact Conditional P Value Calculation for the Quasi-Symmetry Model

Earlier literature introduced a network algorithm for computing an exact test of independence in a two-way contingency table. This article adapts that algorithm to tests of quasi-symmetry in square tables. The algorithm is generally faster than competing Monte Carlo methods, and essentially eliminates the need for asymptotic approximation of P values for assessing goodness-of-fit of the quasi-symmetry model. A macro written for the R computing package is available for implementing the method.     

Computing the confidence levels for a root-mean-square test of goodness-of-fit

The classic χ² statistic for testing goodness-of-fit has long been a cornerstone of modern statistical practice. The statistic consists of a sum in which each summand involves division by the probability associated with the corresponding bin in the distribution being tested for goodness-of-fit. Typically this division should precipitate rebinning to uniformize the probabilities associated with the bins, in order to make the test reasonably powerful. With the now widespread availability of computers, there is no longer any need for this. The present paper provides efficient black-box algorithms for calculating the asymptotic confidence levels of a variant on the classic χ² test which omits the problematic division. In many circumstances, it is also feasible to compute the exact confidence levels via Monte Carlo simulation.     

Conditional Inference on Tables With Structural Zeros

We develop a set of sequential importance sampling (SIS) strategies for sampling nearly uniformly from two-way zero-one or contingency tables with fixed marginal sums and a given set of structural zeros. The SIS procedure samples tables column by column or cell by cell by using appropriate proposal distributions, and enables us to approximate closely the null distributions of a number of test statistics involved in such tables. When structural zeros are on the diagonal or follow certain patterns, more efficient SIS algorithms are developed which guarantee that every generated table is valid. Examples show that our methods can be applied to make conditional inference on zero-one and contingency tables, and are more efficient than other existing Monte Carlo algorithms.     

Circulant Embedding of Approximate Covariances for Inference From Gaussian Data on Large Lattices

Recently proposed computationally efficient Markov chain Monte Carlo (MCMC) and Monte Carlo expectation–maximization (EM) methods for estimating covariance parameters from lattice data rely on successive imputations of values on an embedding lattice that is at least two times larger in each dimension. These methods can be considered exact in some sense, but we demonstrate that using such a large number of imputed values leads to slowly converging Markov chains and EM algorithms. We propose instead the use of a discrete spectral approximation to allow for the implementation of these methods on smaller embedding lattices. While our methods are approximate, our examples indicate that the error introduced by this approximation is small compared to the Monte Carlo errors present in long Markov chains or many iterations of Monte Carlo EM algorithms. Our results are demonstrated in simulation studies, as well as in numerical studies that explore both increasing domain and fixed domain asymptotics. We compare the exact methods to our approximate methods on a large satellite dataset, and show that the approximate methods are also faster to compute, especially when the aliased spectral density is modeled directly. Supplementary materials for this article are available online.     

Robust U-type test for high dimensional regression coefficients using refitted cross-validation variance estimation

This paper aims to develop a new robust U-type test for high dimensional regression coefficients using the estimated U-statistic of order two and refitted cross-validation error variance estimation. It is proved that the limiting null distribution of the proposed new test is normal under two kinds of ordinary models.We further study the local power of the proposed test and compare with other competitive tests for high dimensional data. The idea of refitted cross-validation approach is utilized to reduce the bias of sample variance in the estimation of the test statistic. Our theoretical results indicate that the proposed test can have even more substantial power gain than the test by Zhong and Chen(2011) when testing a hypothesis with outlying observations and heavy tailed distributions. We assess the finite-sample performance of the proposed test by examining its size and power via Monte Carlo studies. We also illustrate the application of the proposed test by an empirical analysis of a real data example.     

p-values of a test on homogeneous means in a multivariate isotonic regression

This paper considers a multivariate isotonic regression model in which the columns of the mean matrix are constrained by a quasi-ordering. For the test on the homogeneity of the means an ad hoc test statistic is derived. When the order restriction is confined to the elements of only one row, another statistic is obtained. A sufficient condition is established under which the p-value of the test is expressed through the distribution of the second statistic at a specified parameter point. This result not only simplifies the expression of the p-values but also makes the computation for the p-values by the Monte Carlo method possible.     

Estimation of partially specified spatial panel data models with random-effects

In this article, we study estimation of a partially specified spatial panel data linear regression with random-effects. Under the conditions of exogenous spatial weighting matrix and exogenous regressors, we give an instrumental variable estimation. Under certain sufficient assumptions, we show that the proposed estimator for the finite dimensional parameter is root-N consistent and asymptotically normally distributed and the proposed estimator for the unknown function is consistent and asymptotically distributed. Consistent estimators for the asymptotic variance-covariance matrices of both the parametric and unknown components are provided. The Monte Carlo simulation results verify our theory and suggest that the approach has some practical value.     

Estimation of a Bivariate Extreme Value Distribution

Several threshold methods have been proposed for the purpose of estimating a bivariate extreme value distribution from a sample of data whose distribution is only in its domain of attraction. An integrated view of these methods is presented which leads to the introduction of a new asymptotically consistent estimator of the dependence function characterizing the extreme dependence structure. Through Monte Carlo simulations, the new estimator is also shown to do as well as its competitors and to outperform them in cases of weak dependence. To the authors' knowledge, this is the first time that the small-sample behavior of nonparametric bivariate threshold methods has ever been investigated.     

The distribution of the maximum likelihood-ratio statistic used for testing a normal sample for three outliers

We study the exact distribution of the likelihood-ratio statistic used for testing a normal sample for three upper (lower) outliers. We obtain recursive correlations for the integral distribution function of this statistic. We apply the obtained correlations for calculating critical values of the likelihood-ratio statistic which appear to be close to critical values of this statistic simulated by the Monte Carlo method. We give an example of the joint use of the likelihood-ratio statistic for testing a sample for more than one outlier.     

Asymptotic sampling – a tool for efficient reliability computation in high dimensions

Monte Carlo methods are most versatile regarding applications to the reliability analysis of high-dimensional nonlinear structural systems. In addition to its versatility, the computational efficacy of Monte Carlo method is not adversely affected by the dimensionality of the problem. Crude Monte Carlo techniques, however, are very inefficient for extremely small failure probabilities such as typically required for sensitive structural systems. Therefore methods to increase the efficacy for small failure probability while keeping the adverse influence of dimensionality small are desirable. On such method is the asymptotic sampling method. Within the framework of this method, well-known asymptotic properties of the reliability index regarding the scaling of the basic variables are exploited to construct a regression model which allows to determine the reliability index for extremely small failure probabilities with high precision using a moderate number of Monte Carlo samples. (© 2015 Wiley-VCH Verlag GmbH & Co. KGaA, Weinheim)     

Empirical Likelihood Semiparametric Regression Analysis under Random Censorship

This paper considers large sample inference for the regression parameter in a partly linear model for right censored data. We introduce an estimated empirical likelihood for the regression parameter and show that its limiting distribution is a mixture of central chi-squared distributions. A Monte Carlo method is proposed to approximate the limiting distribution. This enables one to make empirical likelihood-based inference for the regression parameter. We also develop an adjusted empirical likelihood method which only appeals to standard chi-square tables. Finite sample performance of the proposed methods is illustrated in a simulation study.     

Robust <Emphasis Type="Italic">U</Emphasis>-type test for high dimensional regression coefficients using refitted cross-validation variance estimation

This paper aims to develop a new robust U-type test for high dimensional regression coefficients using the estimated U-statistic of order two and refitted cross-validation error variance estimation. It is proved that the limiting null distribution of the proposed new test is normal under two kinds of ordinary models. We further study the local power of the proposed test and compare with other competitive tests for high dimensional data. The idea of refitted cross-validation approach is utilized to reduce the bias of sample variance in the estimation of the test statistic. Our theoretical results indicate that the proposed test can have even more substantial power gain than the test by Zhong and Chen (2011) when testing a hypothesis with outlying observations and heavy tailed distributions. We assess the finite-sample performance of the proposed test by examining its size and power via Monte Carlo studies. We also illustrate the application of the proposed test by an empirical analysis of a real data example.     

An Estimate of the Odds Ratio That Always Exists

This article proposes an estimate of the odds ratio in a (2 × 2) table obtained from studies in which the row totals are fixed by design, such as a phase II clinical trial. Our estimate, based on the median unbiased estimate of the probabilities of success in the (2× 2) table, will always be in the interval (0, ∞). Another estimate of the odds ratio which has such properties is obtained when adding .5 to each cell of the table. Using simulations, we compared our proposed estimate to that obtained by adding .5 to every cell, and found that our estimate had smaller finite sample bias, and larger mean square error. We also propose the use of the bootstrap to form a confidence interval for the odds ratio based on our proposed estimate. Instead of a Monte Carlo bootstrap, one can easily calculate the “exact” bootstrap distribution of our estimate of the odds ratio, and use this distribution to calculate confidence intervals.     

Nonparametric Bayesian Modeling for Multivariate Ordinal Data

This article proposes a probability model for k-dimensional ordinal outcomes, that is, it considers inference for data recorded in k-dimensional contingency tables with ordinal factors. The proposed approach is based on full posterior inference, assuming a flexible underlying prior probability model for the contingency table cell probabilities. We use a variation of the traditional multivariate probit model, with latent scores that determine the observed data. In our model, a mixture of normals prior replaces the usual single multivariate normal model for the latent variables. By augmenting the prior model to a mixture of normals we generalize inference in two important ways. First, we allow for varying local dependence structure across the contingency table. Second, inference in ordinal multivariate probit models is plagued by problems related to the choice and resampling of cutoffs defined for these latent variables. We show how the proposed mixture model approach entirely removes these problems. We illustrate the methodology with two examples, one simulated dataset and one dataset of interrater agreement.     

Testing over-representation of observations in subsets of a DEA technology

This paper proposes a test for whether data are over-represented in a given production zone, i.e. a subset of a production possibility set which has been estimated using the non-parametric Data Envelopment Analysis (DEA) approach. A binomial test is used that relates the number of observations inside such a zone to a discrete probability weighted relative volume of that zone. A Monte Carlo simulation illustrates the performance of the proposed test statistic and provides good estimation of both facet probabilities and the assumed common inefficiency distribution in a three dimensional input space. Potential applications include tests for whether benchmark units dominate more (or less) observations than expected.     

Modelling life tables with advanced ages: An extreme value theory approach

We propose a new model – we call it a smoothed threshold life table (STLT) model – to generate life tables incorporating information on advanced ages. Our method allows a smooth mortality transition from non-extreme to extreme ages, and provides objectively determined highest attained ages with which to close the life table.We proceed by modifying the threshold life table (TLT) model developed by Li et al. (2008). In the TLT model, extreme value theory (EVT) is used to make optimal use of the relatively small number of observations at high ages, while the traditional Gompertz distribution is assumed for earlier ages. Our novel contribution is to constrain the hazard function of the two-part lifetime distribution to be continuous at the changeover point between the Gompertz and EVT models. This simple but far-reaching modification not only guarantees a smooth transition from non-extreme to extreme ages, but also provides a better and more robust fit than the TLT model when applied to a high quality Netherlands dataset. We show that the STLT model also compares favourably with other existing methods, including the Gompertz–Makeham model, logistic models, Heligman–Pollard model and Coale–Kisker method, and that a further generalisation, a time-dependent dynamic smooth threshold life table (DSTLT) model, generally has superior in-sample fitting as well as better out-of-sample forecasting performance, compared, for example, with the Cairns et al. (2006) model.