Variational Approximation for Mixtures of Linear Mixed Models

Mixtures of linear mixed models (MLMMs) are useful for clustering grouped data and can be estimated by likelihood maximization through the Expectation–Maximization algorithm. A suitable number of components is then determined conventionally by comparing different mixture models using penalized log-likelihood criteria such as Bayesian information criterion. We propose fitting MLMMs with variational methods, which can perform parameter estimation and model selection simultaneously. We describe a variational approximation for MLMMs where the variational lower bound is in closed form, allowing for fast evaluation and develop a novel variational greedy algorithm for model selection and learning of the mixture components. This approach handles algorithm initialization and returns a plausible number of mixture components automatically. In cases of weak identifiability of certain model parameters, we use hierarchical centering to reparameterize the model and show empirically that there is a gain in efficiency in variational algorithms similar to that in Markov chain Monte Carlo (MCMC) algorithms. Related to this, we prove that the approximate rate of convergence of variational algorithms by Gaussian approximation is equal to that of the corresponding Gibbs sampler, which suggests that reparameterizations can lead to improved convergence in variational algorithms just as in MCMC algorithms. Supplementary materials for the article are available online.     

基于EM算法的混合模型中子总体个数的研究

混合模型已成为数据分析中最流行的技术之一,由于拥有数学模型,它通常比聚类分析中的传统的方法产生的结果更精确,而关键因素是混合模型中子总体个数,它决定了数据分析的最终结果。期望最大化(EM)算法常用在混合模型的参数估计,以及机器学习和聚类领域中的参数估计中,是一种从不完全数据或者是有缺失值的数据中求解参数极大似然估计的迭代算法。学者们往往采用AIC和BIC的方法来确定子总体的个数,而这两种方法在实际的应用中的效果并不稳定,甚至可能会产生错误的结果。针对此问题,本文提出了一种利用似然函数的碎石图来确定混合模型中子总体的个数的新方法。实验结果表明,本文方法确定的子总体的个数在大部分理想的情况下可以得到与AIC、BIC方法确定的聚类个数相同的结果,而在一般的实际数据中或条件不理想的状态下,碎石图方法也可以得到更可靠的结果。随后,本文将新方法在选取的黄石公园喷泉数据的参数估计中进行了实际的应用。     

Robust estimation of the number of components for mixtures of linear regression models

In this paper, we investigate a robust estimation of the number of components in the mixture of regression models using trimmed information criteria. Compared to the traditional information criteria, the trimmed criteria are robust and not sensitive to outliers. The superiority of the trimmed methods in comparison with the traditional information criterion methods is illustrated through a simulation study. Two real data applications are also used to illustrate the effectiveness of the trimmed model selection methods.     

Bounds on Codes Derived by Counting Components in Varshamov Graphs

We are interested in improving the Varshamov bound for finite values of length n and minimum distance d. We employ a counting lemma to this end which we find particularly useful in relation to Varshamov graphs. Since a Varshamov graph consists of components corresponding to low weight vectors in the cosets of a code it is a useful tool when trying to improve the estimates involved in the Varshamov bound. We consider how the graph can be iteratively constructed and using our observations are able to achieve a reduction in the over-counting which occurs. This tightens the lower bound for any choice of parameters n, k, d or q and is not dependent on information such as the weight distribution of a code. This work is taken from the author’s thesis [10]     

From here to infinity: sparse finite versus Dirichlet process mixtures in model-based clustering

In model-based clustering mixture models are used to group data points into clusters. A useful concept introduced for Gaussian mixtures by Malsiner Walli et al. (Stat Comput 26:303–324, 2016) are sparse finite mixtures, where the prior distribution on the weight distribution of a mixture with K components is chosen in such a way that a priori the number of clusters in the data is random and is allowed to be smaller than K with high probability. The number of clusters is then inferred a posteriori from the data. The present paper makes the following contributions in the context of sparse finite mixture modelling. First, it is illustrated that the concept of sparse finite mixture is very generic and easily extended to cluster various types of non-Gaussian data, in particular discrete data and continuous multivariate data arising from non-Gaussian clusters. Second, sparse finite mixtures are compared to Dirichlet process mixtures with respect to their ability to identify the number of clusters. For both model classes, a random hyper prior is considered for the parameters determining the weight distribution. By suitable matching of these priors, it is shown that the choice of this hyper prior is far more influential on the cluster solution than whether a sparse finite mixture or a Dirichlet process mixture is taken into consideration.

     

Four-stage symplectic and P-stable SDIRKN methods with dispersion of high order

New SDIRKN methods specially adapted to the numerical integration of second-order stiff ODE systems with periodic solutions are obtained. Our interest is focused on the dispersion (phase errors) of the dominant components in the numerical oscillations when these methods are applied to the homogeneous linear test model. Based on this homogeneous test model we derive the dispersion and P-stability conditions for SDIRKN methods which are assumed to be zero dissipative. Two four-stage symplectic and P-stable methods with algebraic order 4 and high order of dispersion are obtained. One of the methods is symmetric and sixth-order dispersive whereas the other method is nonsymmetric and eighth-order dispersive. These methods have been applied to a number of test problems (linear as well as nonlinear) and some numerical results are presented to show their efficiency when they are compared with other methods derived by Sharp et al. [IMA J. Numer. Anal. 10 (1990) 489–504].     

Fourth-Order Symmetric DIRK Methods for Periodic Stiff Problems

New symmetric DIRK methods specially adapted to the numerical integration of first-order stiff ODE systems with periodic solutions are obtained. Our interest is focused on the dispersion (phase errors) of the dominant components in the numerical oscillations when these methods are applied to the homogeneous linear test model. Based on this homogeneous test model we derive the dispersion conditions for symmetric DIRK methods as well as symmetric stability functions with real poles and maximal dispersion order. Two new fourth-order symmetric methods with four and five stages are obtained. One of the methods is fourth-order dispersive whereas the other method is symplectic and sixth-order dispersive. These methods have been applied to a number of test problems (linear as well as nonlinear) and some numerical results are presented to show their efficiency when they are compared with the symplectic DIRK method derived by Sanz-Serna and Abia (SIAM J. Numer. Anal. 28 (1991) 1081–1096).     

A two-scale simulation framework for Internal Traverse Grinding

We present a hybrid framework which aims at the modelling and simulation of Internal Traverse Grinding of hardened 100Cr6/AISI 52100 by using electro plated cBN grinding wheels. We focus on the thermo-mechanical loading conditions on the workpiece that results from the interaction of workpiece and tool. The modelling framework basically consists of three components, namely representative plane strain adaptive finite element simulations on a meso-scale, capturing the proximity of a single cBN grain when cutting through the workpiece bulk. Secondly, we incorporate a kinematic simulation on the process level to gain detailed simulation-based information on the transient grain-bulk-interaction. The third component of the framework consists of a macro-scale process model that uses the superposed results of the former two components as thermo-mechanical boundary conditions. Using this framework, we target the prediction of metallurgical effects, such as white layers, on the workpiece on the macro scale in the near future. (© 2014 Wiley-VCH Verlag GmbH & Co. KGaA, Weinheim)     

Numerical Solutions for second-kind Volterra integral equations by Galerkin methods

In this paper, we study the global convergence for the numerical solutions of nonlinear Volterra integral equations of the second kind by means of Galerkin finite element methods. Global superconvergence properties are discussed by iterated finite element methods and interpolated finite element methods. Local superconvergence and iterative correction schemes are also considered by iterated finite element methods. We improve the corresponding results obtained by collocation methods in the recent papers [6] and [9] by H. Brunner, Q. Lin and N. Yan. Moreover, using an interpolation post-processing technique, we obtain a global superconvergence of the O(h ^2r )-convergence rate in the piecewise-polynomial space of degree not exceeding (r–1). As a by-product of our results, all these higher order numerical methods can also provide an a posteriori error estimator, which gives critical and useful information in the code development.     

Identifying Mixtures of Mixtures Using Bayesian Estimation

The use of a finite mixture of normal distributions in model-based clustering allows us to capture non-Gaussian data clusters. However, identifying the clusters from the normal components is challenging and in general either achieved by imposing constraints on the model or by using post-processing procedures. Within the Bayesian framework, we propose a different approach based on sparse finite mixtures to achieve identifiability. We specify a hierarchical prior, where the hyperparameters are carefully selected such that they are reflective of the cluster structure aimed at. In addition, this prior allows us to estimate the model using standard MCMC sampling methods. In combination with a post-processing approach which resolves the label switching issue and results in an identified model, our approach allows us to simultaneously (1) determine the number of clusters, (2) flexibly approximate the cluster distributions in a semiparametric way using finite mixtures of normals and (3) identify cluster-specific parameters and classify observations. The proposed approach is illustrated in two simulation studies and on benchmark datasets. Supplementary materials for this article are available online.     

Bayesian analysis of mixture of autoregressive components with an application to financial market volatility

In this paper, we present a fully Bayesian analysis of a finite mixture of autoregressive components. Neither the number of mixture components nor the autoregressive order of each component have to be fixed, since we treat them as stochastic variables. Parameter estimation and model selection are performed using Markov chain Monte Carlo methods. This analysis allows us to take into account the stationarity conditions on the model parameters, which are often ignored by Bayesian approaches. Finally, the application to return volatility of financial markets will be illustrated. Our model seems to be consistent with some empirical facts concerning volatility such as persistence, clustering effects, nonsymmetrical dependencies. Copyright © 2006 John Wiley & Sons, Ltd.     

Bayesian Dimensionality Reduction With PCA Using Penalized Semi-Integrated Likelihood

We discuss the problem of estimating the number of principal components in principal components analysis (PCA). Despite the importance of the problem and the multitude of solutions proposed in literature, it comes as a surprise that there does not exist a coherent asymptotic framework, which would justify different approaches depending on the actual size of the dataset. In this article, we address this issue by presenting an approximate Bayesian approach based on Laplace approximation and introducing a general method of developing criteria for model selection, called PEnalized SEmi-integrated Likelihood (PESEL). Our general framework encompasses a variety of existing approaches based on probabilistic models, like the Bayesian Information Criterion for Probabilistic PCA (PPCA), and enables the construction of new criteria, depending on the size of the dataset at hand and additional prior information. Specifically, we apply PESEL to derive two new criteria for datasets where the number of variables substantially exceeds the number of observations, which is out of the scope of currently existing approaches. We also report results of extensive simulation studies and real data analysis, which illustrate the desirable properties of our proposed criteria as compared to state-of-the-art methods and very recent proposals. Specifically, these simulations show that PESEL-based criteria can be quite robust against deviations from the assumptions of a probabilistic model. Selected PESEL-based criteria for the estimation of the number of principal components are implemented in the R package pesel, which is available on github (https://github.com/psobczyk/pesel). Supplementary material for this article, with additional simulation results, is available online. The code to reproduce all simulations is available at https://github.com/psobczyk/pesel_simulations.     

Adaptive methods for solvings minimax problems ∗

We consider finite-dimensional minimax problems for two traditional models: firstly,with box constraints at variables and,secondly,taking into account a finite number of tinear inequalities. We present finite exact primal and dual methods. These methods are adapted to a great extent to the specific structure of the cost function which is formed by a finite number of linear functions. During the iterations of the primal method we make use of the information from the dual problem, thereby increasing effectiveness. To improve the dual method we use the “long dual step” rule (the principle of ullrelaxation).The results are illustrated by numerical experiments.     

Asymptotic Analysis of Spectral Properties of Finite Capacity Processor Shared Queues

We consider sojourn (or response) times in processor‐shared queues that have a finite customer capacity. Computing the response time of a tagged customer involves solving a finite system of linear ODEs. Writing the system in matrix form, we study the eigenvectors and eigenvalues in the limit as the size of the matrix becomes large. This corresponds to finite capacity models where the system can only hold a large number K of customers. Using asymptotic methods we reduce the eigenvalue problem to that of a standard differential equation, such as the Airy equation. The dominant eigenvalue leads to the tail of a customer's sojourn time distribution. Some numerical results are given to assess the accuracy of the asymptotic results.     

Subdivision of simplices relative to a cutting plane and finite concave minimization

In this article, our primary concern is the classical problem of minimizing globally a concave function over a compact polyhedron (Problem (P)). We present a new simplicial branch and bound approach, which combines triangulations of intersections of simplices with halfspaces and ideas from outer approximation in such a way, that a class of finite algorithms for solving (P) results. For arbitrary compact convex feasible sets one obtains a not necessarily finite but convergent algorithm. Theoretical investigations include determination of the number of simplices in each applied triangulation step and bounds on the number of iterations in the resulting algorithms. Preliminary numerical results are given, and additional applications are sketched.     

Estimation of the number of components of finite mixtures of multivariate distributions

An estimator of the number of components of a finite mixture ofk-dimensional distributions is given on the basis of a one-dimensional independent random sample obtained by a transformation of ak-dimensional independent random sample. A consistency of the estimator is shown. Some simulation results are given in a case of finite mixtures of two-dimensional normal distributions.     

Sampling Bias Correction in the Model of Mixtures with Varying Concentrations

Model of mixture with varying concentrations is a generalization of the classical finite mixture model in which the mixing probabilities (concentrations) vary from observation to observation. We consider the case when the concentrations of the mixture components are known, but no assumptions on the distributions of the observed variable are made. The problem is to estimate the moments of the components’ distributions. We propose a modification of the Horvitz-Thompson weighting for moments estimation by observations from mixture with varying concentrations in presence of sampling bias. Consistency of obtained estimators is demonstrated. Results of simulations are presented.     

A non-standard finite difference method to solve a model of HIV–Malaria co-infection

In this paper, we design and analyse a non-standard finite difference numerical scheme for the numerical solution of the HIV–malaria co-infection model with a distributed delay representing the incubation period of malaria parasite in the mosquitoes vector. To come up with the efficient numerical method for the full co-infection model, we study a number of qualitative properties of sub-models and then use the information while designing the numerical methods for these sub-models. One of the salient features of these methods is that they preserve positivity of the solution which is very essential while studying epidemiological models. We also present numerical simulations to confirm the theoretical findings.     

Robust game theory

We present a distribution-free model of incomplete-information games, both with and without private information, in which the players use a robust optimization approach to contend with payoff uncertainty. Our ``robust game' model relaxes the assumptions of Harsanyi's Bayesian game model, and provides an alternative distribution-free equilibrium concept, which we call ``robust-optimization equilibrium,' to that of the ex post equilibrium. We prove that the robust-optimization equilibria of an incomplete-information game subsume the ex post equilibria of the game and are, unlike the latter, guaranteed to exist when the game is finite and has bounded payoff uncertainty set. For arbitrary robust finite games with bounded polyhedral payoff uncertainty sets, we show that we can compute a robust-optimization equilibrium by methods analogous to those for identifying a Nash equilibrium of a finite game with complete information. In addition, we present computational results. The research of the author was partially supported by a National Science Foundation Graduate Research Fellowship and by the Singapore-MIT Alliance. The research of the author was partially supported by the Singapore-MIT Alliance.     

Optimization of the number of components in the mixed model using multi-criteria decision-making

The distributions of empirical data are often complex. Such complexity cannot be sufficiently addressed by the individual theoretical statistical distribution function. Furthermore, the selection of the distribution function becomes more complicated when the empirical data present a multi-peak feature. In such a case, the multiple testing criteria and the mixed model must be considered during the selection of an appropriate distribution function. Aiming at this vague challenge, the present paper proposes a novel method for establishing a mixed model that can describe accurately the distribution characteristics of empirical data. Apart from combining the Akike and Bayesian information criteria to define the feasible solutions of the mixed model, this study also utilizes the root mean squared deviation, coefficient of determination, Kolmogorov–Smirnov test statistic, average deviation in cumulative distribution function, and average deviation in probability distribution function as the testing criteria. In addition, a non-linear programming is used to find the weighting factors of each criterion. The multi-criteria decision-making technology is adopted to comprehensively and objectively integrate these testing criteria into a synthetic indicator. Finally, an optimization algorithm is proposed to determine the optimal number of components in the mixed model. The illustrated results of the simulated data and measured signals confirm that this approach can estimate precisely the number of components as well as establish a highly accurate mixed model.