Asymptotics for estimation and testing procedures under loss of identifiability

Statistical analyses commonly make use of models that suffer from loss of identifiability. In this paper, we address important issues related to the parameter estimation and hypothesis testing in models with loss of identifiability. That is, there are multiple parameter points corresponding to the same true model. We refer the set of these parameter points to as the set of true parameter values. We consider the case where the set of true parameter values is allowed to be very large or even infinite, some parameter values may lie on the boundary of the parameter space, and the data are not necessarily independently and identically distributed. Our results are applicable to a large class of estimators and their related testing statistics derived from optimizing an objective function such as a likelihood. We examine three specific examples: (i) a finite mixture logistic regression model; (ii) stationary ARMA processes; (iii) general quadratic approximation using Hellinger distance. The applications to these examples demonstrate the applicability of our results in a broad range of difficult statistical problems.     

Fully Bayesian Analysis of Switching Gaussian State Space Models

In the present paper we study switching state space models from a Bayesian point of view. We discuss various MCMC methods for Bayesian estimation, among them unconstrained Gibbs sampling, constrained sampling and permutation sampling. We address in detail the problem of unidentifiability, and discuss potential information available from an unidentified model. Furthermore the paper discusses issues in model selection such as selecting the number of states or testing for the presence of Markov switching heterogeneity. The model likelihoods of all possible hypotheses are estimated by using the method of bridge sampling. We conclude the paper with applications to simulated data as well as to modelling the U.S./U.K. real exchange rate.     

Clustering via finite nonparametric ICA mixture models

We propose a novel extension of nonparametric multivariate finite mixture models by dropping the standard conditional independence assumption and incorporating the independent component analysis (ICA) structure instead. This innovation extends nonparametric mixture model estimation methods to situations in which conditional independence, a necessary assumption for the unique identifiability of the parameters in such models, is clearly violated. We formulate an objective function in terms of penalized smoothed Kullback–Leibler distance and introduce the nonlinear smoothed majorization-minimization independent component analysis algorithm for optimizing this function and estimating the model parameters. Our algorithm does not require any labeled observations a priori; it may be used for fully unsupervised clustering problems in a multivariate setting. We have implemented a practical version of this algorithm, which utilizes the FastICA algorithm, in the R package icamix. We illustrate this new methodology using several applications in unsupervised learning and image processing.

     

Bayesian Nonparametric Modeling for Multivariate Ordinal Regression

Univariate or multivariate ordinal responses are often assumed to arise from a latent continuous parametric distribution, with covariate effects that enter linearly. We introduce a Bayesian nonparametric modeling approach for univariate and multivariate ordinal regression, which is based on mixture modeling for the joint distribution of latent responses and covariates. The modeling framework enables highly flexible inference for ordinal regression relationships, avoiding assumptions of linearity or additivity in the covariate effects. In standard parametric ordinal regression models, computational challenges arise from identifiability constraints and estimation of parameters requiring nonstandard inferential techniques. A key feature of the nonparametric model is that it achieves inferential flexibility, while avoiding these difficulties. In particular, we establish full support of the nonparametric mixture model under fixed cut-off points that relate through discretization the latent continuous responses with the ordinal responses. The practical utility of the modeling approach is illustrated through application to two datasets from econometrics, an example involving regression relationships for ozone concentration, and a multirater agreement problem. Supplementary materials with technical details on theoretical results and on computation are available online.     

Variational Approximation for Mixtures of Linear Mixed Models

Mixtures of linear mixed models (MLMMs) are useful for clustering grouped data and can be estimated by likelihood maximization through the Expectation–Maximization algorithm. A suitable number of components is then determined conventionally by comparing different mixture models using penalized log-likelihood criteria such as Bayesian information criterion. We propose fitting MLMMs with variational methods, which can perform parameter estimation and model selection simultaneously. We describe a variational approximation for MLMMs where the variational lower bound is in closed form, allowing for fast evaluation and develop a novel variational greedy algorithm for model selection and learning of the mixture components. This approach handles algorithm initialization and returns a plausible number of mixture components automatically. In cases of weak identifiability of certain model parameters, we use hierarchical centering to reparameterize the model and show empirically that there is a gain in efficiency in variational algorithms similar to that in Markov chain Monte Carlo (MCMC) algorithms. Related to this, we prove that the approximate rate of convergence of variational algorithms by Gaussian approximation is equal to that of the corresponding Gibbs sampler, which suggests that reparameterizations can lead to improved convergence in variational algorithms just as in MCMC algorithms. Supplementary materials for the article are available online.     

Bayesian Methods for Wavelet Series in Single-Index Models

Single-index models have found applications in econometrics and biometrics, where multidimensional regression models are often encountered. This article proposes a nonparametric estimation approach that combines wavelet methods for nonequispaced designs with Bayesian models. We consider a wavelet series expansion of the unknown regression function and set prior distributions for the wavelet coefficients and the other model parameters. To ensure model identifiability, the direction parameter is represented via its polar coordinates. We employ ad hoc hierarchical mixture priors that perform shrinkage on wavelet coefficients and use Markov chain Monte Carlo methods for a posteriori inference. We investigate an independence-type Metropolis-Hastings algorithm to produce samples for the direction parameter. Our method leads to simultaneous estimates of the link function and of the index parameters. We present results on both simulated and real data, where we look at comparisons with other methods.     

Semiparametric mixtures of regressions with single-index for model based clustering

In this article, we propose two classes of semiparametric mixture regression models with single-index for model based clustering. Unlike many semiparametric/nonparametric mixture regression models that can only be applied to low dimensional predictors, the new semiparametric models can easily incorporate high dimensional predictors into the nonparametric components. The proposed models are very general, and many of the recently proposed semiparametric/nonparametric mixture regression models are indeed special cases of the new models. Backfitting estimates and the corresponding modified EM algorithms are proposed to achieve optimal convergence rates for both parametric and nonparametric parts. We establish the identifiability results of the proposed two models and investigate the asymptotic properties of the proposed estimation procedures. Simulation studies are conducted to demonstrate the finite sample performance of the proposed models. Two real data applications using the new models reveal some interesting findings.

     

System Identifiability (Symbolic Computation) and Parameter Estimation (Numerical Computation)

A system identification based on physical laws often involves a parameter estimation. Before performing an estimation problem, it is necessary to investigate its identifiability. This investigation leads often to painful calculations. Generally, the numerical computation of the parameters does not use these calculus. In this contribution we propose least-squares methods to link identifiability approaches with numerical parameter estimation.     

Identifiability issues of age–period and age–period–cohort models of the Lee–Carter type

The predominant way of modelling mortality rates is the Lee–Carter model and its many extensions. The Lee–Carter model and its many extensions use a latent process to forecast. These models are estimated using a two-step procedure that causes an inconsistent view on the latent variable. This paper considers identifiability issues of these models from a perspective that acknowledges the latent variable as a stochastic process from the beginning. We call this perspective the plug-in age–period or plug-in age–period–cohort model. Defining a parameter vector that includes the underlying parameters of this process rather than its realizations, we investigate whether the expected values and covariances of the plug-in Lee–Carter models are identifiable. It will be seen, for example, that even if in both steps of the estimation procedure we have identifiability in a certain sense it does not necessarily carry over to the plug-in models.     

An EM-Like Algorithm for Semi- and Nonparametric Estimation in Multivariate Mixtures

We propose an algorithm for nonparametric estimation for finite mixtures of multivariate random vectors that strongly resembles a true EM algorithm. The vectors are assumed to have independent coordinates conditional upon knowing from which mixture component they come, but otherwise their density functions are completely unspecified. Sometimes, the density functions may be partially specified by Euclidean parameters, a case we call semiparametric. Our algorithm is much more flexible and easily applicable than existing algorithms in the literature; it can be extended to any number of mixture components and any number of vector coordinates of the multivariate observations. Thus it may be applied even in situations where the model is not identifiable, so care is called for when using it in situations for which identifiability is difficult to establish conclusively. Our algorithm yields much smaller mean integrated squared errors than an alternative algorithm in a simulation study. In another example using a real dataset, it provides new insights that extend previous analyses. Finally, we present two different variations of our algorithm, one stochastic and one deterministic, and find anecdotal evidence that there is not a great deal of difference between the performance of these two variants. The computer code and data used in this article are available online.     

Non-linear model parameter estimation–estimating a feasible parameter set with respect to model use

This article deals with non-linear model parameter estimation from experimental data. As for non-linear models a rigorous identifiability analysis is difficult to perform, parameter estimation is performed in such a way that uncertainty in the estimated parameter values is represented by the range of model use results when the model is used for a certain purpose. Using this approach, the article presents a simulation study where the objective is to discover whether the estimation of model parameters can be improved, so that a small enough range of model use results is obtained. The results of the study indicate that from plant measurements available for the estimation of model parameters, it is possible to extract data that are important for the estimation of model parameters relative to a certain model use. If these data are improved by a proper measurement campaign (e.g. proper choice of measured variables, better accuracy, higher measurement frequency) it is to be expected that a valid model for a certain model use will be obtained. The simulation study is performed for an activated sludge model from wastewater treatment, while the estimation of model parameters is done by Monte Carlo simulation.     

A Skewed and Heavy-Tailed Latent Random Field Model for Spatial Extremes

This article develops Bayesian inference of spatial models with a flexible skew latent structure. Using the multivariate skew-normal distribution of Sahu et al., a valid random field model with stochastic skewing structure is proposed to take into account non-Gaussian features. The skewed spatial model is further improved via scale mixing to accommodate more extreme observations. Finally, the skewed and heavy-tailed random field model is used to describe the parameters of extreme value distributions. Bayesian prediction is done with a well-known Gibbs sampling algorithm, including slice sampling and adaptive simulation techniques. The model performance—as far as the identifiability of the parameters is concerned—is assessed by a simulation study and an analysis of extreme wind speeds across Iran. We conclude that our model provides more satisfactory results according to Bayesian model selection and predictive-based criteria. R code to implement the methods used is available as online supplementary material.     

Estimation of the position and time of emission of a source

We consider the problem of the estimation of the position on the plane and of the moment of beginning of emission of a source by observations from K detectors on the plane. We propose the conditions of regularity and identifiability, which allow us to use two general theorems by Ibragimov and Khasminskii and to describe the asymptotic behavior of the maximum likelihood and Bayes estimators. Then we propose the construction of a linear estimator of unknown parameters and study its properties in slightly more general situation. Special attention is payed to condition of identifiability.

     

Identifiability of intermediate variables on causal paths

We discuss the discovery of causal mechanisms and identifiability of intermediate variables on a causal path. Different from variable selection, we try to distinguish intermediate variables on the causal path from other variables. It is also different from ordinary model selection approaches which do not concern the causal relationships and do not contain unobserved variables. We propose an approach for selecting a causal mechanism depicted by a directed acyclic graph (DAG) with an unobserved variable. We consider several causal networks, and discuss their identifiability by observed data. We show that causal mechanisms of linear structural equation models are not identifiable. Furthermore, we present that causal mechanisms of nonlinear models are identifiable, and we demonstrate the identifiability of causal mechanisms of quadratic equation models. Sensitivity analysis is conducted for the identifiability.     

On Huber's contaminated model

Huber's contaminated model is a basic model for data with outliers. This paper aims at addressing several fundamental problems about this model. We first study its identifiability properties. Several theorems are presented to determine whether the model is identifiable for various situations. Based on these results, we discuss the problem of estimating the parameters with observations drawn from Huber's contaminated model. A definition of estimation consistency is introduced to handle the general case where the model may be unidentifiable. This consistency is a strong robustness property. After showing that existing estimators cannot be consistent in this sense, we propose a new estimator that possesses the consistency property under mild conditions. Its adaptive version, which can simultaneously possess this consistency property and optimal asymptotic efficiency, is also provided. Numerical examples show that our estimators have better overall performance than existing estimators no matter how many outliers in the data.     

Mixtures of biased sentiment analysers

Modelling bias is an important consideration when dealing with inexpert annotations. We are concerned with training a classifier to perform sentiment analysis on news media articles, some of which have been manually annotated by volunteers. The classifier is trained on the words in the articles and then applied to non-annotated articles. In previous work we found that a joint estimation of the annotator biases and the classifier parameters performed better than estimation of the biases followed by training of the classifier. An important question follows from this result: can the annotators be usefully clustered into either predetermined or data-driven clusters, based on their biases? If so, such a clustering could be used to select, drop or otherwise categorise the annotators in a crowdsourcing task. This paper presents work on fitting a finite mixture model to the annotators’ bias. We develop a model and an algorithm and demonstrate its properties on simulated data. We then demonstrate the clustering that exists in our motivating dataset, namely the analysis of potentially economically relevant news articles from Irish online news sources.     

Non-asymptotic Bandwidth Selection for Density Estimation of Discrete Data

We propose a new method for density estimation of categorical data. The method implements a non-asymptotic data-driven bandwidth selection rule and provides model sparsity not present in the standard kernel density estimation method. Numerical experiments with a well-known ten-dimensional binary medical data set illustrate the effectiveness of the proposed approach for density estimation, discriminant analysis and classification. Supported by the Australian Research Council, under grant number DP0558957.     

Estimation of parameters in latent class models using fuzzy clustering algorithms

A mixture approach to clustering is an important technique in cluster analysis. A mixture of multivariate multinomial distributions is usually used to analyze categorical data with latent class model. The parameter estimation is an important step for a mixture distribution. Described here are four approaches to estimating the parameters of a mixture of multivariate multinomial distributions. The first approach is an extended maximum likelihood (ML) method. The second approach is based on the well-known expectation maximization (EM) algorithm. The third approach is the classification maximum likelihood (CML) algorithm. In this paper, we propose a new approach using the so-called fuzzy class model and then create the fuzzy classification maximum likelihood (FCML) approach for categorical data. The accuracy, robustness and effectiveness of these four types of algorithms for estimating the parameters of multivariate binomial mixtures are compared using real empirical data and samples drawn from the multivariate binomial mixtures of two classes. The results show that the proposed FCML algorithm presents better accuracy, robustness and effectiveness. Overall, the FCML algorithm has the superiority over the ML, EM and CML algorithms. Thus, we recommend FCML as another good tool for estimating the parameters of mixture multivariate multinomial models.     

A stochastic volatility factor model of heston type. Statistical properties and estimation

In this paper we study the properties of a new multidimensional continuous-time stochastic covariance process, the Stochastic Volatility Factor model. Two helpful conditional characteristic functions, one needed for estimation and the other used in the pricing of financial derivatives are provided, together with the properties of the instantaneous dependence structure. Conditions for stationarity, ergodicity and mixing properties of the increments are studied. The estimation of the model is performed using the Continuum-Generalized Method of Moments (CGMM). A simulation exercise is included showing parameters are recovered, confirming identifiability. The model is also calibrated to exemplary financial data.     

Bayesian Analysis of Extreme Values by Mixture Modeling

Modeling of extreme values in the presence of heterogeneity is still a relatively unexplored area. We consider losses pertaining to several related categories. For each category, we view exceedances over a given threshold as generated by a Poisson process whose intensity is regulated by a specific location, shape and scale parameter. Using a Bayesian approach, we develop a hierarchical mixture prior, with an unknown number of components, for each of the above parameters. Computations are performed using Reversible Jump MCMC. Our model accounts for possible grouping effects and takes advantage of the similarity across categories, both for estimation and prediction purposes. Some guidance on the specification of the prior distribution is provided, together with an assessment of inferential robustness. The method is illustrated throughout using a data set on large claims against a well-known insurance company over a 15-year period.