The Mixturegram: A Visualization Tool for Assessing the Number of Components in Finite Mixture Models

Certain practical and theoretical challenges surround the estimation of finite mixture models. One such challenge is how to determine the number of components when this is not assumed a priori. Available methods in the literature are primarily numerical and lack any substantial visualization component. Traditional numerical methods include the calculation of information criteria and bootstrapping approaches; however, such methods have known technical issues regarding the necessary regularity conditions for testing the number of components. The ability to visualize an appropriate number of components for a finite mixture model could serve to supplement the results from traditional methods or provide visual evidence when results from such methods are inconclusive. Our research fills this gap through development of a visualization tool, which we call a mixturegram. This tool is easy to implement and provides a quick way for researchers to assess the number of components for their hypothesized mixture model. Mixtures of univariate or multivariate data can be assessed. We validate our visualization assessments by comparing with results from information criteria and an ad hoc selection criterion based on calculations used for the mixturegram. We also construct the mixturegram for two datasets.     

Variational Approximation for Mixtures of Linear Mixed Models

Mixtures of linear mixed models (MLMMs) are useful for clustering grouped data and can be estimated by likelihood maximization through the Expectation–Maximization algorithm. A suitable number of components is then determined conventionally by comparing different mixture models using penalized log-likelihood criteria such as Bayesian information criterion. We propose fitting MLMMs with variational methods, which can perform parameter estimation and model selection simultaneously. We describe a variational approximation for MLMMs where the variational lower bound is in closed form, allowing for fast evaluation and develop a novel variational greedy algorithm for model selection and learning of the mixture components. This approach handles algorithm initialization and returns a plausible number of mixture components automatically. In cases of weak identifiability of certain model parameters, we use hierarchical centering to reparameterize the model and show empirically that there is a gain in efficiency in variational algorithms similar to that in Markov chain Monte Carlo (MCMC) algorithms. Related to this, we prove that the approximate rate of convergence of variational algorithms by Gaussian approximation is equal to that of the corresponding Gibbs sampler, which suggests that reparameterizations can lead to improved convergence in variational algorithms just as in MCMC algorithms. Supplementary materials for the article are available online.     

Infinite Dirichlet mixture models learning via expectation propagation

In this article, we propose a novel Bayesian nonparametric clustering algorithm based on a Dirichlet process mixture of Dirichlet distributions which have been shown to be very flexible for modeling proportional data. The idea is to let the number of mixture components increases as new data to cluster arrive in such a manner that the model selection problem (i.e. determination of the number of clusters) can be answered without recourse to classic selection criteria. Thus, the proposed model can be considered as an infinite Dirichlet mixture model. An expectation propagation inference framework is developed to learn this model by obtaining a full posterior distribution on its parameters. Within this learning framework, the model complexity and all the involved parameters are evaluated simultaneously. To show the practical relevance and efficiency of our model, we perform a detailed analysis using extensive simulations based on both synthetic and real data. In particular, real data are generated from three challenging applications namely images categorization, anomaly intrusion detection and videos summarization.     

On Optimal Forecasting with Soft Computation for Nonlinear Time Series

In this paper we present a new approach on optimal forecasting by using the fuzzy set theory and soft computing methods for the dynamic data analysis. This research is based on the concepts of fuzzy membership function as well as the natural selection of evolution theory. Some discussions in the sensitivity of the design of fuzzy processing will be provided. Through the design of genetic evolution, the AIC criteria is used as the adjust function, and the fuzzy memberships function of each gene model are calculated. Simulation and empirical examples show that our proposed forecasting technique can give an optimal forecasting in time series analysis.     

基于EM算法的混合模型中子总体个数的研究

混合模型已成为数据分析中最流行的技术之一,由于拥有数学模型,它通常比聚类分析中的传统的方法产生的结果更精确,而关键因素是混合模型中子总体个数,它决定了数据分析的最终结果。期望最大化(EM)算法常用在混合模型的参数估计,以及机器学习和聚类领域中的参数估计中,是一种从不完全数据或者是有缺失值的数据中求解参数极大似然估计的迭代算法。学者们往往采用AIC和BIC的方法来确定子总体的个数,而这两种方法在实际的应用中的效果并不稳定,甚至可能会产生错误的结果。针对此问题,本文提出了一种利用似然函数的碎石图来确定混合模型中子总体的个数的新方法。实验结果表明,本文方法确定的子总体的个数在大部分理想的情况下可以得到与AIC、BIC方法确定的聚类个数相同的结果,而在一般的实际数据中或条件不理想的状态下,碎石图方法也可以得到更可靠的结果。随后,本文将新方法在选取的黄石公园喷泉数据的参数估计中进行了实际的应用。     

Empirical likelihood-based subset selection for partially linear autoregressive models

Based on the empirical likelihood method, the subset selection and hypothesis test for parameters in a partially linear autoregressive model are investigated. We show that the empirical log-likelihood ratio at the true parameters converges to the standard chi-square distribution. We then present the definitions of the empirical likelihood-based Bayes information criteria （EBIC） and Akaike information criteria （EAIC）. The results show that EBIC is consistent at selecting subset variables while EAIC is not. Simulation studies demonstrate that the proposed empirical likelihood confidence regions have better coverage probabilities than the least square method, while EBIC has a higher chance to select the true model than EAIC.     

Least squares generalized inferences in unbalanced two-component normal mixed linear model

In this paper, we make use of least squares idea to construct new fiducial generalized pivotal quantities of variance components in two-component normal mixed linear model, then obtain generalized confidence intervals for two variance components and the ratio of the two variance components. The simulation results demonstrate that the new method performs very well in terms of both empirical coverage probability and average interval length. The newly proposed method also is illustrated by a real data example.     

Efficient computer experiment-based optimization through variable selection

A computer experiment-based optimization approach employs design of experiments and statistical modeling to represent a complex objective function that can only be evaluated pointwise by running a computer model. In large-scale applications, the number of variables is huge, and direct use of computer experiments would require an exceedingly large experimental design and, consequently, significant computational effort. If a large portion of the variables have little impact on the objective, then there is a need to eliminate these before performing the complete set of computer experiments. This is a variable selection task. The ideal variable selection method for this task should handle unknown nonlinear structure, should be computationally fast, and would be conducted after a small number of computer experiment runs, likely fewer runs (N) than the number of variables (P). Conventional variable selection techniques are based on assumed linear model forms and cannot be applied in this “large P and small N” problem. In this paper, we present a framework that adds a variable selection step prior to computer experiment-based optimization, and we consider data mining methods, using principal components analysis and multiple testing based on false discovery rate, that are appropriate for our variable selection task. An airline fleet assignment case study is used to illustrate our approach.     

A high-dimensional Wilks phenomenon

A theorem by Wilks asserts that in smooth parametric density estimation the difference between the maximum likelihood and the likelihood of the sampling distribution converges toward a Chi-square distribution where the number of degrees of freedom coincides with the model dimension. This observation is at the core of some goodness-of-fit testing procedures and of some classical model selection methods. This paper describes a non-asymptotic version of the Wilks phenomenon in bounded contrast optimization procedures. Using concentration inequalities for general functions of independent random variables, it proves that in bounded contrast minimization (as for example in Statistical Learning Theory), the difference between the empirical risk of the minimizer of the true risk in the model and the minimum of the empirical risk (the excess empirical risk) satisfies a Bernstein-like inequality where the variance term reflects the dimension of the model and the scale term reflects the noise conditions. From a mathematical statistics viewpoint, the significance of this result comes from the recent observation that when using model selection via penalization, the excess empirical risk represents a minimum penalty if non-asymptotic guarantees concerning prediction error are to be provided. From the perspective of empirical process theory, this paper describes a concentration inequality for the supremum of a bounded non-centered (actually non-positive) empirical process. Combining the now classical analysis of M-estimation (building on Talagrand??s inequality for suprema of empirical processes) and versatile moment inequalities for functions of independent random variables, this paper develops a genuine Bernstein-like inequality that seems beyond the reach of traditional tools.     

On the dual of linear inverse problems

In linear inverse problems considered in this paper a vector with positive components is to be selected from a feasible set defined by linear constraints. The selection rule involves minimization of a certain function which is a measure of distance from a priori guess. Csiszar made an axiomatic approach towards defining a family of functions, we call it α-divergence, that can serve as logically consistent selection rules. In this paper we present an explicit and perfect dual of the resulting convex programming problem, prove the corresponding duality theorem and optimality criteria, and make some suggestions on an algorithmic solution.     

On Testing for the Number of Components in a Mixed Poisson Model

Poisson mixtures are usually used to describe overdispersed data. Finite Poisson mixtures are used in many practical situations where often it is of interest to determine the number of components in the mixture. Identifying how many components comprise a mixture remains a difficult problem. The likelihood ratio test (LRT) is a general statistical procedure to use. Unfortunately, a number of specific problems arise and the classical theory fails to hold. In this paper a new procedure is proposed that is based on testing whether a new component can be added to a finite Poisson mixture which eventually leads to the number of components in the mixture. It is a sequential testing procedure based on the well known LRT that utilises a resampling technique to construct the distribution of the test statistic. The application of the procedure to real data reveals some interesting features of the distribution of the test statistic.     

半参数变系数部分线性模型的统计推断

本文在多种复杂数据下, 研究一类半参数变系数部分线性模型的统计推断理论和方法. 首先在纵向数据和测量误差数据等复杂数据下, 研究半参数变系数部分线性模型的经验似然推断问题, 分别提出分组的和纠偏的经验似然方法. 该方法可以有效地处理纵向数据的组内相关性给构造经验似然比函数所带来的困难. 其次在测量误差数据和缺失数据等复杂数据下, 研究模型的变量选择问题, 分别提出一个“纠偏” 的和基于借补值的变量选择方法. 该变量选择方法可以同时选择参数分量及非参数分量中的重要变量, 并且变量选择与回归系数的估计同时进行. 通过选择适当的惩罚参数, 证明该变量选择方法可以相合地识别出真实模型, 并且所得的正则估计具有oracle 性质.     

Limit Distribution of the Mean Square deviation of the Gasser–Müller Nonparametric Estimate of the Regression Function

Asymptotic distribution of the mean square deviation of the Gasser–Müller estimate of the regression curve is investigated. The testing of hypotheses on regression function is considered. The asymptotic behaviour of the power of the proposed criteria under contiguous alternatives is studed.     

Risk bounds for model selection via penalization

Performance bounds for criteria for model selection are developed using recent theory for sieves. The model selection criteria are based on an empirical loss or contrast function with an added penalty term motivated by empirical process theory and roughly proportional to the number of parameters needed to describe the model divided by the number of observations. Most of our examples involve density or regression estimation settings and we focus on the problem of estimating the unknown density or regression function. We show that the quadratic risk of the minimum penalized empirical contrast estimator is bounded by an index of the accuracy of the sieve. This accuracy index quantifies the trade-off among the candidate models between the approximation error and parameter dimension relative to sample size. If we choose a list of models which exhibit good approximation properties with respect to different classes of smoothness, the estimator can be simultaneously minimax rate optimal in each of those classes. This is what is usually called adaptation. The type of classes of smoothness in which one gets adaptation depends heavily on the list of models. If too many models are involved in order to get accurate approximation of many wide classes of functions simultaneously, it may happen that the estimator is only approximately adaptive (typically up to a slowly varying function of the sample size). We shall provide various illustrations of our method such as penalized maximum likelihood, projection or least squares estimation. The models will involve commonly used finite dimensional expansions such as piecewise polynomials with fixed or variable knots, trigonometric polynomials, wavelets, neural nets and related nonlinear expansions defined by superposition of ridge functions. Received: 7 July 1995 / Revised version: 1 November 1997     

Return and Value at Risk using the Dirichlet Process

There exists a wide variety of models for return, and the chosen model determines the tool required to calculate the value at risk (VaR). This paper introduces an alternative methodology to model‐based simulation by using a Monte Carlo simulation of the Dirichlet process. The model is constructed in a Bayesian framework, using properties initially described by Ferguson. A notable advantage of this model is that, on average, the random draws are sampled from a mixed distribution that consists of a prior guess by an expert and the empirical process based on a random sample of historical asset returns. The method is relatively automatic and similar to machine learning tools, e.g. the estimate is updated as new data arrive.     

A mixed R&D projects and securities portfolio selection model

The business environment is full of uncertainty. Allocating the wealth among various asset classes may lower the risk of overall portfolio and increase the potential for more benefit over the long term. In this paper, we propose a mixed single-stage R&D projects and multi-stage securities portfolio selection model. Specifically, we present a bi-objective mixed-integer stochastic programming model. Moreover, we use semi-absolute deviation risk functions to measure the risk of mixed asset portfolio. Based on the idea of moments approximation method via linear programming, we propose a scenario generation approach for the mixed single-stage R&D projects and multi-stage securities portfolio selection problem. The bi-objective mixed-integer stochastic programming problem can be solved by transforming it into a single objective mixed-integer stochastic programming problem. A numerical example is given to illustrate the behavior of the proposed mixed single stage R&D projects and multi-stage securities portfolio selection model.     

车辆荷载作用下桥梁应变极值估计的阈值选取

采用过阈法估计车辆荷载作用下桥梁的应变极值,合理的阈值选取十分关键.阈值选取过大,信息量少,阈值选取过小,广义Pareto分布模型参数估计偏差大.常用的阈值选取方法不能较好地适用于车辆荷载作用下的应变极值估计.基于太平湖大桥车辆荷载作用下1年的应变数据,对拟合结果较好的3种混合分布进行Monte-Carlo(蒙特 卡洛)抽样,对比同一样本基于不同阈值的广义Pareto分布模型的极值估计结果,提出了一种经验式的阈值选取方法.与常用阈值选取方法相比,根据文中方法所得阈值估计的周应变极值分布与实测结果更为接近,估计结果更好.     

Bayesian analysis of mixture of autoregressive components with an application to financial market volatility

In this paper, we present a fully Bayesian analysis of a finite mixture of autoregressive components. Neither the number of mixture components nor the autoregressive order of each component have to be fixed, since we treat them as stochastic variables. Parameter estimation and model selection are performed using Markov chain Monte Carlo methods. This analysis allows us to take into account the stationarity conditions on the model parameters, which are often ignored by Bayesian approaches. Finally, the application to return volatility of financial markets will be illustrated. Our model seems to be consistent with some empirical facts concerning volatility such as persistence, clustering effects, nonsymmetrical dependencies. Copyright © 2006 John Wiley & Sons, Ltd.     

Efficient empirical-likelihood-based inferences for the single-index model

This article proposes the efficient empirical-likelihood-based inferences for the single component of the parameter and the link function in the single-index model. Unlike the existing empirical likelihood procedures for the single-index model, the proposed profile empirical likelihood for the parameter is constructed by using some components of the maximum empirical likelihood estimator (MELE) based on a semiparametric efficient score. The empirical-likelihood-based inference for the link function is also considered. The resulting statistics are proved to follow a standard chi-squared limiting distribution. Simulation studies are undertaken to assess the finite sample performance of the proposed confidence intervals. An application to real data set is illustrated.     

Poisson冲击下的$k/n(G)$系统的可靠性分析

本文研究了一类Poisson冲击下的$k/n(G)$系统(即$k$-out-of-$n$: $G$系统). 假定冲击的到达数形成一个参数为$\lambda$的Poisson过程, 且冲击的量服从某一分布. 当每次冲击到达时, 对系统中工作的部件独立地产生影响. 进而假定每一部件以一定的概率故障, 概率值是冲击量的函数. 且各次冲击独立地对系统造成损失, 直到工作部件数少于$k$系统故障为止. 在这些假定下, 我们获得了系统的可靠度函数和系统的平均工作时间. 进一步, 假定系统是可修的, 系统中有一个维修工, 并根据``先坏先修’’的维修规则对故障部件进行维修. 在维修时间服从指数分布的假设下, 系统状态转移服从Markov过程. 对该系统我们建立了状态转移方程, 并求得了系统可用度、稳态下的平均工作时间、平均停工时间和系统失效频率等可靠性指标. 最后, 我们还给出了一个简单例子来演示讨论的模型.