Minimum classification error training in example based speech and pattern recognition using sparse weight matrices

The Minimum Classification Error (MCE) criterion is a well-known criterion in pattern classification systems. The aim of MCE training is to minimize the resulting classification error when trying to classify a new data set. Usually, these classification systems use some form of statistical model to describe the data. These systems usually do not work very well when this underlying model is incorrect. Speech recognition systems traditionally use Hidden Markov Models (HMM) with Gaussian (or Gaussian mixture) probability density functions as their basic model. It is well known that these models make some assumptions that are not correct. In example based approaches, these statistical models are absent and are replaced by the pure data. The absence of statistical models has created the need for parameters to model the data space accurately. For this work, we use the MCE criterion to create a system that is able to work together with this example based approach. Moreover, we extend the locally scaled distance measure with sparse, block diagonal weight matrices resulting in a better model for the data space and avoiding the computational load caused by using full matrices. We illustrate the approach with some example experiments on databases from pattern recognition and with speech recognition.     

估计死亡率分布的一个最大熵模型

本文提出了一种估计死亡率分布的新模型一最大熵模型。该模型直接从样本信息出发，不需要对待估分布的概率密度函数或先验分布作任何假定，从而克服了极大似然估计和贝叶斯估计的不足。而且通过两个例子的计算结果，表明该方法与样本数据的拟合效果要好于其它两种方法。     

Automated rejection sampling from product of distributions

Summary  Sampling from probability density functions (pdfs) has become more and more important in many areas of applied science, and has therefore been the subject of great attention. Many sampling procedures proposed allow for approximate or asymptotic sampling. On the other hand, very few methods allow for exact sampling. Direct sampling of standard pdfs is feasible, but sampling of much more complicated pdfs is often required. Rejection sampling allows to exactly sample from univariate pdfs, but has the huge drawback of needing a case-by-case calculation of a comparison function that often reveals as a tremendous chore, whose results dramatically affect the efficiency of the sampling procedure. In this paper, we restrict ourselves to a pdf that is proportional to a product of standard distributions. From there, we show that an automated selection of both the comparison function and the upper bound is possible. Moreover, this choice is performed in order to optimize the sampling efficiency among a range of potential solutions. Finally, the method is illustrated on a few examples.     

Joint Modeling of Multiple Network Views

Latent space models (LSM) for network data rely on the basic assumption that each node of the network has an unknown position in a D-dimensional Euclidean latent space: generally the smaller the distance between two nodes in the latent space, the greater their probability of being connected. In this article, we propose a variational inference approach to estimate the intractable posterior of the LSM. In many cases, different network views on the same set of nodes are available. It can therefore be useful to build a model able to jointly summarize the information given by all the network views. For this purpose, we introduce the latent space joint model (LSJM) that merges the information given by multiple network views assuming that the probability of a node being connected with other nodes in each network view is explained by a unique latent variable. This model is demonstrated on the analysis of two datasets: an excerpt of 50 girls from “Teenage Friends and Lifestyle Study” data at three time points and the Saccharomyces cerevisiae genetic and physical protein–protein interactions. Supplementary materials for this article are available online.     

Maximum likelihood estimation from fuzzy data using the EM algorithm

A method is proposed for estimating the parameters in a parametric statistical model when the observations are fuzzy and are assumed to be related to underlying crisp realizations of a random sample. This method is based on maximizing the observed-data likelihood defined as the probability of the fuzzy data. It is shown that the EM algorithm may be used for that purpose, which makes it possible to solve a wide range of statistical problems involving fuzzy data. This approach, called the fuzzy EM (FEM) method, is illustrated using three classical problems: normal mean and variance estimation from a fuzzy sample, multiple linear regression with crisp inputs and fuzzy outputs, and univariate finite normal mixture estimation from fuzzy data.     

Large sample properties of estimates of a discrete grade of membership model

Increasingly, fuzzy partitions are being used in multivariate classification problems as an alternative to the crisp classification procedures commonly used. One such fuzzy partition, the grade of membership model, partitions individuals into fuzzy sets using multivariate categorical data. Although the statistical methods used to estimate fuzzy membership for this model are based on maximum likelihood methods, large sample properties of the estimation procedure are problematic for two reasons. First, the number of incidental parameters increases with the size of the sample. Second, estimated parameters fall on the boundary of the parameter space with non-zero probability. This paper examines the consistency of the likelihood approach when estimating the components of a particular probability model that gives rise to a fuzzy partition. The results of the consistency proof are used to determine the large sample distribution of the estimates. Common methods of classifying individuals based on multivariate observations attempt to place each individual into crisply defined sets. The fuzzy partition allows for individual to individual heterogeneity, beyond simply errors in measurement, by defining a set of pure type characteristics and determining each individual's distance from these pure types. Both the profiles of the pure types and the heterogeneity of the individuals must be estimated from data. These estimates empirically define the fuzzy partition. In the current paper, this data is assumed to be categorical data. Because of the large number of parameters to be estimated and the limitations of categorical data, one may be concerned about whether or not the fuzzy partition can be estimated consistently. This paper shows that if heterogeneity is measured with respect to a fixed number of moments of the grade of membership scores of each individual, the estimated fuzzy partition is consistent.     

Precomputing strategy for Hamiltonian Monte Carlo method based on regularity in parameter space

Markov Chain Monte Carlo (MCMC) algorithms play an important role in statistical inference problems dealing with intractable probability distributions. Recently, many MCMC algorithms such as Hamiltonian Monte Carlo (HMC) and Riemannian Manifold HMC have been proposed to provide distant proposals with high acceptance rate. These algorithms, however, tend to be computationally intensive which could limit their usefulness, especially for big data problems due to repetitive evaluations of functions and statistical quantities that depend on the data. This issue occurs in many statistic computing problems. In this paper, we propose a novel strategy that exploits smoothness (regularity) in parameter space to improve computational efficiency of MCMC algorithms. When evaluation of functions or statistical quantities are needed at a point in parameter space, interpolation from precomputed values or previous computed values is used. More specifically, we focus on HMC algorithms that use geometric information for faster exploration of probability distributions. Our proposed method is based on precomputing the required geometric information on a set of grids before running sampling algorithm and approximating the geometric information for the current location of the sampler using the precomputed information at nearby grids at each iteration of HMC. Sparse grid interpolation method is used for high dimensional problems. Tests on computational examples are shown to illustrate the advantages of our method.     

Degradation modeling with subpopulation heterogeneities based on the inverse Gaussian process

This study proposes a random effects model based on inverse Gaussian process, where the mixture normal distribution is used to account for both unit-specific and subpopulation-specific heterogeneities. The proposed model can capture heterogeneities due to subpopulations in the same population or the units from different batches. A new Expectation-Maximization (EM) algorithm is developed for point estimation and the bias-corrected bootstrap is used for interval estimation. We show that the EM algorithm updates the parameters based on the gradient of the loglikelihood function via a projection matrix. In addition, the convergence rate depends on the condition number that can be obtained by the projection matrix and the Hessian matrix of the loglikelihood function. A simulation study is conducted to assess the proposed model and the inference methods, and two real degradation datasets are analyzed for illustration.     

On modeling left-truncated loss data using mixtures of distributions

A new statistical methodology is developed for fitting left-truncated loss data by using the G-component finite mixture model with any combination of Gamma, Lognormal, and Weibull distributions. The EM algorithm, along with the emEM initialization strategy, is employed for model fitting. We propose a new grid map which considers the model selection criterion (AIC or BIC) and risk measures at the same time, by using the entire space of models under consideration. A simulation study validates our proposed approach. The application of the proposed methodology and use of new grid maps are illustrated through analyzing a real data set that includes left-truncated insurance losses.     

基于Gibbs抽样算法的贝叶斯动态面板数据模型分析

针对现有动态面板数据分析中存在偶发参数和没有考虑模型参数的不确定性风险问题,提出了基于Gibbs抽样算法的贝叶斯随机系数动态面板数据模型.假设初始值服从平稳分布,自回归系数服从Logit正态分布的条件下,设计了Markov链Monte Carlo数值计算程序,得到了模型参数的贝叶斯估计值.实证研究结果表明:基于Gibb...     

Granular fuzzy models: Analysis,design, and evaluation

The study is concerned with a design of granular fuzzy models. We exploit a concept of information granularity by developing a model coming as a network of intuitively structured collection of interval information granules described in the output space and a family of induced information granules (in the form of fuzzy sets) formed in the input space. In contrast to most fuzzy models encountered in the literature, the results produced by granular models are information granules rather than plain numeric entities. The design of the model concentrates on a construction of information granules that form a backbone of the overall construct. Interval information granules positioned in the output space are built by considering intervals of equal length, equal probability, and developing an optimized version of the intervals. The induced fuzzy information granules localized in the input space are realized by running a conditional Fuzzy C-Means (FCM). The performance of the model is assessed by considering criteria of coverage and information specificity (information granularity). Further optimization of the model is proposed along the line of an optimal re-distribution of input information granules induced by the individual interval information granules located in the output space. Experimental results involve some synthetic low-dimensional data and publicly available benchmark data sets.     

On New Multivariate Probability Distributions and Stochastic Processes with System Reliability and Maintenance Applications

Two distinct methods for construction of some interesting new classes of multivariate probability densities are described and applied. As common results of both procedures three n-variate pdf classes are obtained. These classes are considered as generalizations of the class of univariate Weibullian, gamma, and multivariate normal pdfs. An example of an application of the obtained n-variate pdfs to the problem of modeling the reliability of multicomponent systems with stochastically dependent life-times of their components is given. Obtaining sequences over n = 2, 3, ... of consistent n-variate pdfs, that obey a relatively simple common pattern, for each n, allows us to extend some of the constructions from random vectors to discrete time stochastic processes. Application of one, so obtained, class of highly non-Markovian, but still sufficiently simple, stochastic processes for modeling maintenance of systems with repair, is presented. These models allow us to describe and analyze repaired systems with histories of all past repairs.      

Analysis of Multi-Unit Variance Components Models with State Space Profiles

We apply the Kalman Filter to the analysis of multi-unit variance components models where each unit's response profile follows a state space model. We use mixed model results to obtain estimates of unit-specific random effects, state disturbance terms and residual noise terms. We use the signal extraction approach to smooth individual profiles. We show how to utilize the Kalman Filter to efficiently compute the restricted loglikelihood of the model. For the important special case where each unit's response profile follows a continuous structural time series model with known transition matrix we derive an EM algorithm for the restricted maximum likelihood (REML) estimation of the variance components. We present details for the case where individual profiles are modeled as local polynomial trends or polynomial smoothing splines.     

Testing Hypotheses on Discriminant Space in the Mixture Model of Gaussian Distributions

When clustering multivariate observations adhering the mixture model of Gaussian distributions, rather frequently projections of the observations onto a linear subspace of less dimensionality, called discriminant space (DS), contain all statistical information about the cluster structure of the model. In this case, the actual reduction of data dimensionality substantially facilitates a solution of various classification problems. In the paper, attention is devoted to statistical testing of hypotheses about DS and its dimension. The characterization of DS and methods of its identification are also briefly discussed.     

An optimum tolerance region for multivariate regression

A tolerance region is a map from the sample space of one statistical model to the event space of a second statistical model having the same parameter. This paper derives an optimum β-expectation tolerance region for the multivariate regression model. A measure of power is proposed and evaluated.     

Improved probabilistic point estimation schemes for uncertainty analysis

The probabilistic point estimation (PPE) methods replace the probability distribution of the random parameters of a model with a finite number of discrete points in sample space selected in such a way to preserve limit probabilistic information of involved random parameters. Most PPE methods developed thus far match the distribution of random parameters up to the third statistical moment and, in general, could provide reasonable accurate estimation only for the first two statistical moments of model output. This study proposes two optimization-based point selection schemes for the PPE methods to enhance the accuracy of higher-order statistical moments estimation for model output. Several test models of varying degrees of complexity and nonlinearity are used to examine the performance of the proposed point selection schemes. The results indicate that the proposed point selection schemes provide significantly more accurate estimation of model output uncertainty features than the existing schemes.     

Data-driven estimation of atomistic support for continuum stress using the Gaussian mixture model

The notion of stress being an inherent continuum concept has been a matter of discussion at the atomistic level. The atomistic stress measure at a given spatial position contains a space averaging volume over nearby atoms to provide an averaged macroscopic stress measure. Previous work on atomistic stress measures introduce the characteristic length as an a priori given parameter. In this contribution we learn the characteristic length directly from the atomistic data itself. Central to our proposed approach is the grouping of atoms with highly similar values of position and stress into the same atomistic sub-population. We hypothesise that atoms with similar values for position and stress are those atoms which harbour the greatest influence over each other and therefore should be contained within the same space averaging volume. Consequently the characteristic length can be computed directly from the discovered sub-populations by averaging over the maximum extent of each sub-population. We motivate the Gaussian mixture model (GMM) as a principled probabilistic method of estimating the similarity between atoms within position-stress space. The GMM parameters are learnt from the atomistic data using the Expectation Maximization (EM) algorithm. To form a parsimonious representation of the dataset we regularise our model using the Bayesian Information Criterion (BIC) which maintains a balance between too few and too many atomistic sub-populations. We use the GMM to segment the atoms into homogeneous sub-populations based on the probability of each atom belonging to a particular sub-population. Thorough evaluation is conducted on a numerical example of an edge dislocation in a single crystal. We derive estimates of the space averaging volume which are in very close agreement to the corresponding analytical solution. (© 2012 Wiley-VCH Verlag GmbH & Co. KGaA, Weinheim)     

n维数据统计信息增益的度量及应用

采样数据的增加究竟有多少相应的有效Fisher信息增益,这是测量数据处理、图像数据融合等应用领域中关心的问题.以(共轭)正态分布为基础,利用统计推断理论,导出一定相关性下样本数据的增加与统计信息(Fisher信息)增益之间的关系,并经一维航天测量数据和二维图像超分辨仿真算例验证.     

Generalized beta prior models on fraction defective in reliability test planning

In many reliability analyses, the probability of obtaining a defective unit in a production process should not be considered constant even though the process is stable and in control. Engineering experience or previous data of similar or related products may often be used in the proper selection of a prior model to describe the random fluctuations in the fraction defective. A generalized beta family of priors, several maximum entropy priors and other prior models are considered for this purpose. In order to determine the acceptability of a product based on the lifelengths of some test units, failure-censored reliability sampling plans for location-scale distributions using average producer and consumer risks are designed. Our procedure allows the practitioners to incorporate a restricted parameter space into the reliability analysis, and it is reasonably insensitive to small disturbances in the prior information. Impartial priors are used to reflect prior neutrality between the producer and the consumer when a consensus on the elicited prior model is required. Nonetheless, our approach also enables the producer and the consumer to assume their own prior distributions. The use of substantial prior information can, in many cases, significantly reduce the amount of testing required. However, the main advantage of utilizing a prior model for the fraction defective is not necessarily reduced sample size but improved assessment of the true sampling risks. An example involving shifted exponential lifetimes is considered to illustrate the results.     

Estimating a Resource Selection Function With Line Transect Sampling

A resource selection probability function is a function that gives the probability that a resource unit (e.g., a plot of land) that is described by a set of habitat variables X1 to Xp will be used by an animal or group of animals in a certain period of time. The estimation of a resource selection function is usually based on the comparison of a sample of resource units used by an animal with a sample of the resource units that were available for use, with both samples being assumed to be effectively randomly selected from the relevant populations. In this paper the possibility of using a modified sampling scheme is examined, with the used units obtained by line transect sampling. A logistic regression type of model is proposed, with estimation by conditional maximum likelihood. A simulation study indicates that the proposed method should be useful in practice.