Learning mixtures of polynomials of multidimensional probability densities from data using B-spline interpolation

Non-parametric density estimation is an important technique in probabilistic modeling and reasoning with uncertainty. We present a method for learning mixtures of polynomials (MoPs) approximations of one-dimensional and multidimensional probability densities from data. The method is based on basis spline interpolation, where a density is approximated as a linear combination of basis splines. We compute maximum likelihood estimators of the mixing coefficients of the linear combination. The Bayesian information criterion is used as the score function to select the order of the polynomials and the number of pieces of the MoP. The method is evaluated in two ways. First, we test the approximation fitting. We sample artificial datasets from known one-dimensional and multidimensional densities and learn MoP approximations from the datasets. The quality of the approximations is analyzed according to different criteria, and the new proposal is compared with MoPs learned with Lagrange interpolation and mixtures of truncated basis functions. Second, the proposed method is used as a non-parametric density estimation technique in Bayesian classifiers. Two of the most widely studied Bayesian classifiers, i.e., the naive Bayes and tree-augmented naive Bayes classifiers, are implemented and compared. Results on real datasets show that the non-parametric Bayesian classifiers using MoPs are comparable to the kernel density-based Bayesian classifiers. We provide a free R package implementing the proposed methods.     

Data Augmentation for Diffusions

The problem of formal likelihood-based (either classical or Bayesian) inference for discretely observed multidimensional diffusions is particularly challenging. In principle, this involves data augmentation of the observation data to give representations of the entire diffusion trajectory. Most currently proposed methodology splits broadly into two classes: either through the discretization of idealized approaches for the continuous-time diffusion setup or through the use of standard finite-dimensional methodologies discretization of the diffusion model. The connections between these approaches have not been well studied. This article provides a unified framework that brings together these approaches, demonstrating connections, and in some cases surprising differences. As a result, we provide, for the first time, theoretical justification for the various methods of imputing missing data. The inference problems are particularly challenging for irreducible diffusions, and our framework is correspondingly more complex in that case. Therefore, we treat the reducible and irreducible cases differently within the article. Supplementary materials for the article are available online.     

Estimating Mixture of Dirichlet Process Models

Abstract

Current Gibbs sampling schemes in mixture of Dirichlet process (MDP) models are restricted to using “conjugate” base measures that allow analytic evaluation of the transition probabilities when resampling configurations, or alternatively need to rely on approximate numeric evaluations of some transition probabilities. Implementation of Gibbs sampling in more general MDP models is an open and important problem because most applications call for the use of nonconjugate base measures. In this article we propose a conceptual framework for computational strategies. This framework provides a perspective on current methods, facilitates comparisons between them, and leads to several new methods that expand the scope of MDP models to nonconjugate situations. We discuss one in detail. The basic strategy is based on expanding the parameter vector, and is applicable for MDP models with arbitrary base measure and likelihood. Strategies are also presented for the important class of normal-normal MDP models and for problems with fixed or few hyperparameters. The proposed algorithms are easily implemented and illustrated with an application.     

On the evaluation of general sparse hybrid linear solvers

General sparse hybrid solvers are commonly used kernels for solving wide range of scientific and engineering problems. This work addresses the current problems of efficiently solving general sparse linear equations with direct/iterative hybrid solvers on many core distributed clusters. We briefly discuss the solution stages of Maphys, HIPS, and PDSLin hybrid solvers for large sparse linear systems with their major algorithmic differences. In this category of solvers, different methods with sophisticated preconditioning algorithms are suggested to solve the trade off between memory and convergence. Such solutions require a certain hierarchical level of parallelism more suitable for modern supercomputers that allow to scale for thousand numbers of processors using Schur complement framework. We study the effect of reordering and analyze the performance, scalability as well as memory for each solve phase of PDSLin, Maphys, and HIPS hybrid solvers using large set of challenging matrices arising from different actual applications and compare the results with SuperLU_DIST direct solver. We specifically focus on the level of parallel mechanisms used by the hybrid solvers and the effect on scalability. Tuning and Analysis Utilities (TAU) is employed to assess the efficient usage of heap memory profile and measuring communication volume. The tests are run on high performance large memory clusters using up to 512 processors.     

Towards a More General Concept of Inference

The main objective of this paper is to sketch unifying conceptual and formal framework for inference that is able to explain various proof techniques without implicitly changing the underlying notion of inference rules. We base this framework upon the so-called two-dimensional, i.e., deduction to deduction, account of inference introduced by Tichý in his seminal work The Foundation’s of Frege’s Logic (1988). Consequently, it will be argued that sequent calculus provides suitable basis for such general concept of inference and therefore should not be seen just as technical tool, but philosophically well-founded system that can rival natural deduction in terms of its “naturalness”.     

GRASP strategies for a bi-objective commercial territory design problem

A bi-objective commercial territory design problem motivated by a real-world application from the bottled beverage distribution industry is addressed. The problem considers territory compactness and balancing with respect to number of customers as optimization criteria. Previous work has focused on exact methods for small- to medium-scale instances. In this work, a GRASP framework is proposed for tackling considerably large instances. Within this framework two general schemes are developed. For each of these schemes two strategies are studied: (i) keeping connectivity as a hard constraint during construction and post-processing phases and, (ii) ignoring connectivity during the construction phase and adding this as another minimizing objective function during the post-processing phase. These strategies are empirically evaluated and compared to NSGA-II, one of the most successful evolutionary methods known in literature. Computational results show the superiority of the proposed strategies. In addition, one of the proposed GRASP strategies is successfully applied to a case study from industry.     

Bayesian Registration of Functions With a Gaussian Process Prior

We present a Bayesian framework for registration of real-valued functional data. At the core of our approach is a series of transformations of the data and functional parameters, developed under a differential geometric framework. We aim to avoid discretization of functional objects for as long as possible, thus minimizing the potential pitfalls associated with high-dimensional Bayesian inference. Approximate draws from the posterior distribution are obtained using a novel Markov chain Monte Carlo (MCMC) algorithm, which is well suited for estimation of functions. We illustrate our approach via pairwise and multiple functional data registration, using both simulated and real datasets. Supplementary material for this article is available online.     

On the convergence of inexact block coordinate descent methods for constrained optimization

We consider the problem of minimizing a smooth function over a feasible set defined as the Cartesian product of convex compact sets. We assume that the dimension of each factor set is huge, so we are interested in studying inexact block coordinate descent methods (possibly combined with column generation strategies). We define a general decomposition framework where different line search based methods can be embedded, and we state global convergence results. Specific decomposition methods based on gradient projection and Frank–Wolfe algorithms are derived from the proposed framework. The numerical results of computational experiments performed on network assignment problems are reported.     

Inference for SDE Models via Approximate Bayesian Computation

Models defined by stochastic differential equations (SDEs) allow for the representation of random variability in dynamical systems. The relevance of this class of models is growing in many applied research areas and is already a standard tool to model, for example, financial, neuronal, and population growth dynamics. However, inference for multidimensional SDE models is still very challenging, both computationally and theoretically. Approximate Bayesian computation (ABC) allows to perform Bayesian inference for models which are sufficiently complex that the likelihood function is either analytically unavailable or computationally prohibitive to evaluate. A computationally efficient ABC-MCMC algorithm is proposed, halving the running time in our simulations. Focus here is on the case where the SDE describes latent dynamics in state-space models; however, the methodology is not limited to the state-space framework. We consider simulation studies for a pharmacokinetics/pharmacodynamics model and for stochastic chemical reactions and we provide a Matlab package that implements our ABC-MCMC algorithm.     

Convergence of simple adaptive Galerkin schemes based on h − h/2 error estimators

We discuss several adaptive mesh-refinement strategies based on (h − h/2)-error estimation. This class of adaptive methods is particularly popular in practise since it is problem independent and requires virtually no implementational overhead. We prove that, under the saturation assumption, these adaptive algorithms are convergent. Our framework applies not only to finite element methods, but also yields a first convergence proof for adaptive boundary element schemes. For a finite element model problem, we extend the proposed adaptive scheme and prove convergence even if the saturation assumption fails to hold in general.     

On the Relationship Between Markov chain Monte Carlo Methods for Model Uncertainty

This article considers Markov chain computational methods for incorporating uncertainty about the dimension of a parameter when performing inference within a Bayesian setting. A general class of methods is proposed for performing such computations, based upon a product space representation of the problem which is similar to that of Carlin and Chib. It is shown that all of the existing algorithms for incorporation of model uncertainty into Markov chain Monte Carlo (MCMC) can be derived as special cases of this general class of methods. In particular, we show that the popular reversible jump method is obtained when a special form of Metropolis–Hastings (M–H) algorithm is applied to the product space. Furthermore, the Gibbs sampling method and the variable selection method are shown to derive straightforwardly from the general framework. We believe that these new relationships between methods, which were until now seen as diverse procedures, are an important aid to the understanding of MCMC model selection procedures and may assist in the future development of improved procedures. Our discussion also sheds some light upon the important issues of “pseudo-prior” selection in the case of the Carlin and Chib sampler and choice of proposal distribution in the case of reversible jump. Finally, we propose efficient reversible jump proposal schemes that take advantage of any analytic structure that may be present in the model. These proposal schemes are compared with a standard reversible jump scheme for the problem of model order uncertainty in autoregressive time series, demonstrating the improvements which can be achieved through careful choice of proposals.     

Unified description of cosmological and static solutions in affine generalized theories of gravity: Vecton-scalaron duality and its applications

We briefly describe the simplest class of affine theories of gravity in multidimensional space-times with symmetric connections and their reductions to two-dimensional dilaton-vecton gravity field theories. The distinctive feature of these theories is the presence of an absolutely neutral massive (or tachyonic) vector field (vecton) with an essentially nonlinear coupling to the dilaton gravity. We emphasize that the vecton field in dilaton-vecton gravity can be consistently replaced by a new effectively massive scalar field (scalaron) with an unusual coupling to the dilaton gravity. With this vecton-scalaron duality, we can use the methods and results of the standard dilaton gravity coupled to usual scalars in more complex dilaton-scalaron gravity theories equivalent to dilaton-vecton gravity. We present the dilaton-vecton gravity models derived by reductions of multidimensional affine theories and obtain one-dimensional dynamical systems simultaneously describing cosmological and static states in any gauge. Our approach is fully applicable to studying static and cosmological solutions in multidimensional theories and also in general one-dimensional dilaton-scalaron gravity models. We focus on general and global properties of the models, seeking integrals and analyzing the structure of the solution space. In integrable cases, it can be usefully visualized by drawing a “topological portrait” resembling the phase portraits of dynamical systems and simply exposing the global properties of static and cosmological solutions, including horizons, singularities, etc. For analytic approximations, we also propose an integral equation well suited for iterations.     

A highly accurate explicit symplectic ERKN method for multi-frequency and multidimensional oscillatory Hamiltonian systems

The numerical integration of Hamiltonian systems with multi-frequency and multidimensional oscillatory solutions is encountered frequently in many fields of the applied sciences. In this paper, we firstly summarize the extended Runge–Kutta–Nyström (ERKN) methods proposed by Wu et al. (Comput. Phys. Comm. 181:1873–1887, (2010)) for multi-frequency and multidimensional oscillatory systems and restate the order conditions and symplecticity conditions for the explicit ERKN methods. Secondly, we devote to exploring the explicit symplectic multi-frequency and multidimensional ERKN methods of order five based on the symplecticity conditions and order conditions. A five-stage explicit symplectic multi-frequency and multidimensional ERKN method of order five with some small residuals is proposed and its stability and phase properties are analyzed. It is shown that the new method is dispersive of order six. Numerical experiments are carried out and the numerical results demonstrate that the new method is much more efficient than the methods appeared in the scientific literature.     

Aggregation and consensus for preference relations based on fuzzy partial orders

We propose a framework for eliciting and aggregating pairwise preference relations based on the assumption of an underlying fuzzy partial order. We also propose some linear programming optimization methods for ensuring consistency either as part of the aggregation phase or as a pre- or post-processing task. We contend that this framework of pairwise-preference relations, based on the Kemeny distance, can be less sensitive to extreme or biased opinions and is also less complex to elicit from experts. We provide some examples and outline their relevant properties and associated concepts.     

The impact of diversity on the accuracy of evidential classifier ensembles

Diversity being inherent in classifiers is widely acknowledged as an important issue in constructing successful classifier ensembles. Although many statistics have been employed in measuring diversity among classifiers to ascertain whether it correlates with ensemble performance in the literature, most of these measures are incorporated and explained in a non-evidential context. In this paper, we provide a modelling for formulating classifier outputs as triplet mass functions and a uniform notation for defining diversity measures. We then assess the relationship between diversity obtained by four pairwise and non-pairwise diversity measures and the improvement in accuracy of classifiers combined in different orders by Demspter’s rule of combination, Smets’ conjunctive rule, the Proportion and Yager’s rules in the framework of belief functions. Our experimental results demonstrate that the accuracy of classifiers combined by Dempster’s rule is not strongly correlated with the diversity obtained by the four measures, and the correlation between the diversity and the ensemble accuracy made by Proportion and Yager’s rules is negative, which is not in favor of the claim that increasing diversity could lead to reduction of generalization error of classifier ensembles.     

A general framework for frequentist model averaging In Celebration of Professor Lincheng Zhao's 75th Birthday

Model selection strategies have been routinely employed to determine a model for data analysis in statistics, and further study and inference then often proceed as though the selected model were the true model that were known a priori. Model averaging approaches, on the other hand, try to combine estimators for a set of candidate models. Specifically, instead of deciding which model is the 'right' one, a model averaging approach suggests to fit a set of candidate models and average over the estimators using data adaptive weights.In this paper we establish a general frequentist model averaging framework that does not set any restrictions on the set of candidate models. It broaden, the scope of the existing methodologies under the frequentist model averaging development. Assuming the data is from an unknown model, we derive the model averaging estimator and study its limiting distributions and related predictions while taking possible modeling biases into account.We propose a set of optimal weights to combine the individual estimators so that the expected mean squared error of the average estimator is minimized. Simulation studies are conducted to compare the performance of the estimator with that of the existing methods. The results show the benefits of the proposed approach over traditional model selection approaches as well as existing model averaging methods.     

A survey of fuzzy decision tree classifier

Decision-tree algorithm provides one of the most popular methodologies for symbolic knowledge acquisition. The resulting knowledge, a symbolic decision tree along with a simple inference mechanism, has been praised for comprehensibility. The most comprehensible decision trees have been designed for perfect symbolic data. Over the years, additional methodologies have been investigated and proposed to deal with continuous or multi-valued data, and with missing or noisy features. Recently, with the growing popularity of fuzzy representation, some researchers have proposed to utilize fuzzy representation in decision trees to deal with similar situations. This paper presents a survey of current methods for Fuzzy Decision Tree (FDT) designment and the various existing issues. After considering potential advantages of FDT classifiers over traditional decision tree classifiers, we discuss the subjects of FDT including attribute selection criteria, inference for decision assignment and stopping criteria. To be best of our knowledge, this is the first overview of fuzzy decision tree classifier.     

Robust multicategory support vector machines using difference convex algorithm

The support vector machine (SVM) is one of the most popular classification methods in the machine learning literature. Binary SVM methods have been extensively studied, and have achieved many successes in various disciplines. However, generalization to multicategory SVM (MSVM) methods can be very challenging. Many existing methods estimate k functions for k classes with an explicit sum-to-zero constraint. It was shown recently that such a formulation can be suboptimal. Moreover, many existing MSVMs are not Fisher consistent, or do not take into account the effect of outliers. In this paper, we focus on classification in the angle-based framework, which is free of the explicit sum-to-zero constraint, hence more efficient, and propose two robust MSVM methods using truncated hinge loss functions. We show that our new classifiers can enjoy Fisher consistency, and simultaneously alleviate the impact of outliers to achieve more stable classification performance. To implement our proposed classifiers, we employ the difference convex algorithm for efficient computation. Theoretical and numerical results obtained indicate that for problems with potential outliers, our robust angle-based MSVMs can be very competitive among existing methods.     

Supervised box clustering

In this work we address a technique for effectively clustering points in specific convex sets, called homogeneous boxes, having sides aligned with the coordinate axes (isothetic condition). The proposed clustering approach is based on homogeneity conditions, not according to some distance measure, and, even if it was originally developed in the context of the logical analysis of data, it is now placed inside the framework of Supervised clustering. First, we introduce the basic concepts in box geometry; then, we consider a generalized clustering algorithm based on a class of graphs, called incompatibility graphs. For supervised classification problems, we consider classifiers based on box sets, and compare the overall performances to the accuracy levels of competing methods for a wide range of real data sets. The results show that the proposed method performs comparably with other supervised learning methods in terms of accuracy.