Clustering Multivariate Longitudinal Observations: The Contaminated Gaussian Hidden Markov Model

The Gaussian hidden Markov model (HMM) is widely considered for the analysis of heterogenous continuous multivariate longitudinal data. To robustify this approach with respect to possible elliptical heavy-tailed departures from normality, due to the presence of outliers, spurious points, or noise (collectively referred to as bad points herein), the contaminated Gaussian HMM is here introduced. The contaminated Gaussian distribution represents an elliptical generalization of the Gaussian distribution and allows for automatic detection of bad points in the same natural way as observations are typically assigned to the latent states in the HMM context. Once the model is fitted, each observation has a posterior probability of belonging to a particular state and, inside each state, of being a bad point or not. In addition to the parameters of the classical Gaussian HMM, for each state we have two more parameters, both with a specific and useful interpretation: one controls the proportion of bad points and one specifies their degree of atypicality. A sufficient condition for the identifiability of the model is given, an expectation-conditional maximization algorithm is outlined for parameter estimation and various operational issues are discussed. Using a large-scale simulation study, but also an illustrative artificial dataset, we demonstrate the effectiveness of the proposed model in comparison with HMMs of different elliptical distributions, and we also evaluate the performance of some well-known information criteria in selecting the true number of latent states. The model is finally used to fit data on criminal activities in Italian provinces. Supplementary materials for this article are available online     

The Polya Tree Sampler: Towards Efficient and Automatic Independent Metropolis-Hastings Proposals

We present a simple, efficient, and computationally cheap sampling method for exploring an un-normalized multivariate density on ?(d), such as a posterior density, called the Polya tree sampler. The algorithm constructs an independent proposal based on an approximation of the target density. The approximation is built from a set of (initial) support points - data that act as parameters for the approximation - and the predictive density of a finite multivariate Polya tree. In an initial "warming-up" phase, the support points are iteratively relocated to regions of higher support under the target distribution to minimize the distance between the target distribution and the Polya tree predictive distribution. In the "sampling" phase, samples from the final approximating mixture of finite Polya trees are used as candidates which are accepted with a standard Metropolis-Hastings acceptance probability. Several illustrations are presented, including comparisons of the proposed approach to Metropolis-within-Gibbs and delayed rejection adaptive Metropolis algorithm.     

High breakdown mixture discriminant analysis

Robust S-estimation is proposed for multivariate Gaussian mixture models generalizing the work of Hastie and Tibshirani (J. Roy. Statist. Soc. Ser. B 58 (1996) 155). In the case of Gaussian Mixture models, the unknown location and scale parameters are estimated by the EM algorithm. In the presence of outliers, the maximum likelihood estimators of the unknown parameters are affected, resulting in the misclassification of the observations. The robust S-estimators of the unknown parameters replace the non-robust estimators from M-step of the EM algorithm. The results were compared with the standard mixture discriminant analysis approach using the probability of misclassification criterion. This comparison showed a slight reduction in the average probability of misclassification using robust S-estimators as compared to the standard maximum likelihood estimators.     

Rare Event Probability Estimation in the Presence of Epistemic Uncertainty on Input Probability Distribution Parameters

The accurate estimation of rare event probabilities is a crucial problem in engineering to characterize the reliability of complex systems. Several methods such as Importance Sampling or Importance Splitting have been proposed to perform the estimation of such events more accurately (i.e., with a lower variance) than crude Monte Carlo method. However, these methods assume that the probability distributions of the input variables are exactly defined (e.g., mean and covariance matrix perfectly known if the input variables are defined through Gaussian laws) and are not able to determine the impact of a change in the input distribution parameters on the probability of interest. The problem considered in this paper is the propagation of the input distribution parameter uncertainty defined by intervals to the rare event probability. This problem induces intricate optimization and numerous probability estimations in order to determine the upper and lower bounds of the probability estimate. The calculation of these bounds is often numerically intractable for rare event probability (say 10^?5), due to the high computational cost required. A new methodology is proposed to solve this problem with a reduced simulation budget, using the adaptive Importance Sampling. To this end, a method for estimating the Importance Sampling optimal auxiliary distribution is proposed, based on preceding Importance Sampling estimations. Furthermore, a Kriging-based adaptive Importance Sampling is used in order to minimize the number of evaluations of the computationally expensive simulation code. To determine the bounds of the probability estimate, an evolutionary algorithm is employed. This algorithm has been selected to deal with noisy problems since the Importance Sampling probability estimate is a random variable. The efficiency of the proposed approach, in terms of accuracy of the found results and computational cost, is assessed on academic and engineering test cases.     

Efficient simulation of discrete stochastic reaction systems with a splitting method

Stochastic reaction systems with discrete particle numbers are usually described by a continuous-time Markov process. Realizations of this process can be generated with the stochastic simulation algorithm, but simulating highly reactive systems is computationally costly because the computational work scales with the number of reaction events. We present a new approach which avoids this drawback and increases the efficiency considerably at the cost of a small approximation error. The approach is based on the fact that the time-dependent probability distribution associated to the Markov process is explicitly known for monomolecular, autocatalytic and certain catalytic reaction channels. More complicated reaction systems can often be decomposed into several parts some of which can be treated analytically. These subsystems are propagated in an alternating fashion similar to a splitting method for ordinary differential equations. We illustrate this approach by numerical examples and prove an error bound for the splitting error.     

A simulation-based multivariate Bayesian control chart for real time condition-based maintenance of complex systems

When complex systems are monitored, multi-observations from several sensors or sources may be available. These observations can be fused through Bayesian theory to give a posterior probabilistic estimate of the underlying state which is often not directly observable. This forms the basis of a Bayesian control chart where the estimated posterior probability of the state can be compared with a preset threshold level to assess whether a full inspection is needed or not. Maintenance can then be carried out if indicated as necessary by the inspection. This paper considers the design of such multivariate Bayesian control chart where both the transition between states and the relationship between observed information and the state are not Markovian. Since analytical or numerical solutions are difficult for the case considered in this paper, Monte Carlo simulation is used to obtain the optimal control chart parameters, which are the monitoring interval and the upper control limit. A two-stage failure process characterised by the delay time concept is used to describe the underlying state transition process and Bayesian theory is used to compute the posterior probability of the underlying state, which is embedded in the simulation algorithm. Extensive examples are shown to demonstrate the modelling idea.     

Stochastic optimization for allocation problems with shortfall risk constraints

One of the crucial aspects in asset allocation problems is the assumption concerning the probability distribution of asset returns. Financial managers generally suppose normal distribution, even if extreme realizations usually have an higher frequency than in the Gaussian case. The aim of this paper is to propose a general Monte Carlo simulation approach able to solve an asset allocation problem with shortfall constraint, and to evaluate the exact portfolio risk‐level when managers assume a misspecified return behaviour. We assume that returns are generated by a multivariate skewed Student‐t distribution where each marginal can have different degrees of freedom. The stochastic optimization allows us to value the effective risk for managers. In the empirical application we consider a symmetric and heterogeneous case, and interestingly note that a multivariate Student‐t with heterogeneous marginal distributions produces in the optimization problem a shortfall probability and a shortfall return level that can be adequately approximated by assuming a multivariate Student‐t with common degrees of freedom. Thus, the proposed simulation‐based approach could be an important instrument for investors who require a qualitative assessment of the reliability and sensitivity of their investment strategies in the case their models could be potentially misspecified. Copyright © 2007 John Wiley & Sons, Ltd.     

The excess-mass ellipsoid

The excess-mass ellipsoid is the ellipsoid that maximizes the difference between its probability content and a constant multiple of its volume, over all ellipsoids. When an empirical distribution determines the probability content, the sample excess-mass ellipsoid is a random set that can be used in contour estimation and tests for multimodality. Algorithms for computing the ellipsoid are provided, as well as comparative simulations. The asymptotic distribution of the parameters for the sample excess-mass ellipsoid are derived. It is found that a n^1/3 normalization of the center of the ellipsoid and lengths of its axes converge in distribution to the maximizer of a Gaussian process with quadratic drift. The generalization of ellipsoids to convex sets is discussed.     

Quantum-behaved particle swarm optimization with Gaussian distributed local attractor point

This paper proposes a novel variant of quantum-behaved particle swarm optimization (QPSO) algorithm with the local attractor point subject to a Gaussian probability distribution (GAQPSO). The local attractor point in QPSO plays an important in that determining the convergence behavior of an individual particle. As such, the mean value and standard deviation of the proposed Gaussian probability distribution in GAQPSO are carefully selected. The distributions and diversities of the local attractor points in GAQPSO and QPSO are evaluated and compared. For the purpose of comparison, two variants of the GAQPSO algorithm are proposed by using a mutation probability and other types of probability distribution. The GAQPSO has been comprehensively evaluated on the suite of CEC2005 benchmark functions, and the experimental results are compared with those of the PSO and QPSO algorithms based on different probability distributions. It is shown by the results that the GAQPSO algorithm is an effective approach that can improve the QPSO performance considerably, that is, the GAQPSO algorithm is less likely to be stuck in local optima and hence it can achieve better solutions in most cases.     

多元$t$分布数据的局部影响分析

对于多元$t$分布数据, 直接应用其概率密度进行影响分析是困难的\bd 本文通过引入服从Gamma分布的权重, 将其表示为特定多元正态分布的混合\bd 在此基础上, 进而将权重视为缺失数据, 引入EM算法; 从而利用基于完全数据似然函数的条件期望进行局部影响分析\bd 本文进一步系统研究了加权扰动模型下的局部影响分析, 得到了相应的诊断统计量; 并通过两个实例说明了这种方法的有效性.     

A hybrid analysis approach for finite-capacity queues with general inputs and phase type service

A new analysis method for queueing systems with general input stream and phase type service time distributions is introduced. The approach combines discrete event simulation and numerical analysis of continuous time Markov chains. Simulation is used to represent the arrival process, whereas the service process is analyzed with numerical techniques. In this way the state of the system is characterized by a probability vector rather than by a single state. The use of a distribution vector reduces the variance of result estimators such that the width of confidence intervals is often reduced compared to discrete event simulation. This, in particular, holds for measures based on rare events or states with a small probability. The analysis approach can be applied for a wide variety of result measures including stationary, transient and accumulated measures. This revised version was published online in June 2006 with corrections to the Cover Date.     

Penalized multinomial mixture logit model

Normal distribution based discriminant methods have been used for the classification of new entities into different groups based on a discriminant rule constructed from the learning set. In practice if the groups are not homogeneous, then mixture discriminant analysis of Hastie and Tibshirani (J R Stat Soc Ser B 58(1):155–176, 1996) is a useful approach, assuming that the distribution of the feature vectors is a mixture of multivariate normals. In this paper a new logistic regression model for heterogenous group structure of the learning set is proposed based on penalized multinomial mixture logit models. This approach is shown through simulation studies to be more effective. The results were compared with the standard mixture discriminant analysis approach using the probability of misclassification criterion. This comparison showed a slight reduction in the average probability of misclassification using this penalized multinomial mixture logit model as compared to the classical discriminant rules. It also showed better results when applied to practical life data problems producing smaller errors.     

基于混合分布单因子模型的CDO定价问题

本文讨论了基于混合分布单因子模型的CDO定价问题,在所研究的CDO抵押资产组合中,描述资产价值的市场共同因子和异质因子均服从标准高斯和NIG的混合分布,且相关系数为随机相关系数.通过半解析法给出了CDO分券层的公允价格公式,并利用傅里叶变换及其逆变换,得出了贝努利相关和三状态相关两种随机相关系数的情形下累积损失概率分布的求解方法.     

Nonparametric variable selection and its application to additive models

Variable selection for multivariate nonparametric regression models usually involves parameterized approximation for nonparametric functions in the objective function. However, this parameterized approximation often increases the number of parameters significantly, leading to the “curse of dimensionality” and inaccurate estimation. In this paper, we propose a novel and easily implemented approach to do variable selection in nonparametric models without parameterized approximation, enabling selection consistency to be achieved. The proposed method is applied to do variable selection for additive models. A two-stage procedure with selection and adaptive estimation is proposed, and the properties of this method are investigated. This two-stage algorithm is adaptive to the smoothness of the underlying components, and the estimation consistency can reach a parametric rate if the underlying model is really parametric. Simulation studies are conducted to examine the performance of the proposed method. Furthermore, a real data example is analyzed for illustration.

     

Asymptotic efficiency of the two-stage estimation method for copula-based models

For multivariate copula-based models for which maximum likelihood is computationally difficult, a two-stage estimation procedure has been proposed previously; the first stage involves maximum likelihood from univariate margins, and the second stage involves maximum likelihood of the dependence parameters with the univariate parameters held fixed from the first stage. Using the theory of inference functions, a partitioned matrix in a form amenable to analysis is obtained for the asymptotic covariance matrix of the two-stage estimator. The asymptotic relative efficiency of the two-stage estimation procedure compared with maximum likelihood estimation is studied. Analysis of the limiting cases of the independence copula and Fréchet upper bound help to determine common patterns in the efficiency as the dependence in the model increases. For the Fréchet upper bound, the two-stage estimation procedure can sometimes be equivalent to maximum likelihood estimation for the univariate parameters. Numerical results are shown for some models, including multivariate ordinal probit and bivariate extreme value distributions, to indicate the typical level of asymptotic efficiency for discrete and continuous data.     

A New Unified Approach for the Simulation of a Wide Class of Directional Distributions

The need for effective simulation methods for directional distributions has grown as they have become components in more sophisticated statistical models. A new acceptance–rejection method is proposed and investigated for the Bingham distribution on the sphere using the angular central Gaussian distribution as an envelope. It is shown that the proposed method has high efficiency and is also straightforward to use. Next, the simulation method is extended to the Fisher and Fisher–Bingham distributions on spheres and related manifolds. Together, these results provide a widely applicable and efficient methodology to simulate many of the standard models in directional data analysis. An R package simdd, available in the online supplementary material, implements these simulation methods.     

MCMC Estimation of Restricted Covariance Matrices

This article is motivated by the difficulty of applying standard simulation techniques when identification constraints or theoretical considerations induce covariance restrictions in multivariate models. To deal with this difficulty, we build upon a decomposition of positive definite matrices and show that it leads to straightforward Markov chain Monte Carlo samplers for restricted covariance matrices. We introduce the approach by reviewing results for multivariate Gaussian models without restrictions, where standard conjugate priors on the elements of the decomposition induce the usual Wishart distribution on the precision matrix and vice versa. The unrestricted case provides guidance for constructing efficient Metropolis–Hastings and accept-reject Metropolis–Hastings samplers in more complex settings, and we describe in detail how simulation can be performed under several important constraints. The proposed approach is illustrated in a simulation study and two applications in economics. Supplemental materials for this article (appendixes, data, and computer code) are available online.     

The Shadow Prior

In this article we consider posterior simulation in models with constrained parameter or sampling spaces. Constraints on the support of sampling and prior distributions give rise to a normalization constant in the complete conditional posterior distribution for the (hyper-) parameters of the respective distribution, complicating posterior simulation.

To mitigate the problem of evaluating normalization constants, we propose a computational approach based on model augmentation. We include an additional level in the probability model to separate the (hyper-) parameter from the constrained probability model, and we refer to this additional level in the probability model as a shadow prior. This approach can significantly reduce the overall computational burden if the original (hyper-) prior includes a complicated structure, but a simple form is chosen for the shadow prior, for example, if the original prior includes a mixture model or multivariate distribution, and the shadow prior defines a set of shadow parameters that are iid given the (hyper-) parameters. Although introducing the shadow prior changes the posterior inference on the original parameters, we argue that by appropriate choices of the shadow prior, the change is minimal and posterior simulation in the augmented probability model provides a meaningful approximation to the desired inference. Data used in this article are available online.     

James—Stein型置信椭球及其覆盖概率

本文构造了协方差阵具有非球型结构（未知）的多元正态分布均值的James-Stein型置信椭球,它能渐近一致改进通常置信椭球的覆盖概率,并给出了改进余项的一致阶,同时本质改进了文献中有关余项的一致阶。     

A sweep-plane algorithm for generating random tuples in simple polytopes

A sweep-plane algorithm of Lawrence for convex polytope computation is adapted to generate random tuples on simple polytopes. In our method an affine hyperplane is swept through the given polytope until a random fraction (sampled from a proper univariate distribution) of the volume of the polytope is covered. Then the intersection of the plane with the polytope is a simple polytope with smaller dimension.

In the second part we apply this method to construct a black-box algorithm for log-concave and -concave multivariate distributions by means of transformed density rejection.