Selection between proportional and stratified hazards models based on expected log-likelihood

The problem of selecting between semi-parametric and proportional hazards models is considered. We propose to make this choice based on the expectation of the log-likelihood (ELL) which can be estimated by the likelihood cross-validation (LCV) criterion. The criterion is used to choose an estimator in families of semi-parametric estimators defined by the penalized likelihood. A simulation study shows that the ELL criterion performs nearly as well in this problem as the optimal Kullback–Leibler criterion in term of Kullback–Leibler distance and that LCV performs reasonably well. The approach is applied to a model of age-specific risk of dementia as a function of sex and educational level from the data of a large cohort study.     

Estimation of Kullback–Leibler Divergence by Local Likelihood

Motivated from the bandwidth selection problem in local likelihood density estimation and from the problem of assessing a final model chosen by a certain model selection procedure, we consider estimation of the Kullback–Leibler divergence. It is known that the best bandwidth choice for the local likelihood density estimator depends on the distance between the true density and the ‘vehicle’ parametric model. Also, the Kullback–Leibler divergence may be a useful measure based on which one judges how far the true density is away from a parametric family. We propose two estimators of the Kullback-Leibler divergence. We derive their asymptotic distributions and compare finite sample properties. Research of Young Kyung Lee was supported by the Brain Korea 21 Projects in 2004. Byeong U. Park’s research was supported by KOSEF through Statistical Research Center for Complex Systems at Seoul National University.     

Estimators for the binomial distribution that dominate the MLE in terms of Kullback–Leibler risk

Estimators based on the mode are introduced and shown empirically to have smaller Kullback–Leibler risk than the maximum likelihood estimator. For one of these, the midpoint modal estimator (MME), we prove the Kullback–Leibler risk is below \frac12{\frac{1}{2}} while for the MLE the risk is above \frac12{\frac{1}{2}} for a wide range of success probabilities that approaches the unit interval as the sample size grows to infinity. The MME is related to the mean of Fisher’s Fiducial estimator and to the rule of succession for Jefferey’s noninformative prior.     

Sample path moderate deviations for the cumulative fluid produced by an increasing number of exponential on-off sources

We consider the superposition of the cumulative fluid generated by an increasing number of stationary iid on-off sources with exponential iid on- and off-time distributions. We establish a family of sample path large deviation principles when the fluid is centered and then scaled with a factor between the inverse of the number of sources and its square root. The common rate function in this family also appears in a large deviation principle for the tail probabilities of an integrated Ornstein–Uhlenbeck process. When the produced fluid is centered and scaled with the square root of the inverse of the number of sources it converges to this integrated Ornstein–Uhlenbeck process in distribution. We discuss several representations of the rate function. We apply the results to queueing systems loaded with on-off traffic and approaching critical loading.      

Merging experts’ opinions: A Bayesian hierarchical model with mixture of prior distributions

In this paper, a general approach is proposed to address a full Bayesian analysis for the class of quadratic natural exponential families in the presence of several expert sources of prior information. By expressing the opinion of each expert as a conjugate prior distribution, a mixture model is used by the decision maker to arrive at a consensus of the sources. A hyperprior distribution on the mixing parameters is considered and a procedure based on the expected Kullback–Leibler divergence is proposed to analytically calculate the hyperparameter values. Next, the experts’ prior beliefs are calibrated with respect to the combined posterior belief over the quantity of interest by using expected Kullback–Leibler divergences, which are estimated with a computationally low-cost method. Finally, it is remarkable that the proposed approach can be easily applied in practice, as it is shown with an application.     

A modified EM algorithm for mixture models based on Bregman divergence

The EM algorithm is a sophisticated method for estimating statistical models with hidden variables based on the Kullback–Leibler divergence. A natural extension of the Kullback–Leibler divergence is given by a class of Bregman divergences, which in general enjoy robustness to contamination data in statistical inference. In this paper, a modification of the EM algorithm based on the Bregman divergence is proposed for estimating finite mixture models. The proposed algorithm is geometrically interpreted as a sequence of projections induced from the Bregman divergence. Since a rigorous algorithm includes a nonlinear optimization procedure, two simplification methods for reducing computational difficulty are also discussed from a geometrical viewpoint. Numerical experiments on a toy problem are carried out to confirm appropriateness of the simplifications.     

Converting information into probability measures with the Kullback–Leibler divergence

This paper uses a decision theoretic approach for updating a probability measure representing beliefs about an unknown parameter. A cumulative loss function is considered, which is the sum of two terms: one depends on the prior belief and the other one on further information obtained about the parameter. Such information is thus converted to a probability measure and the key to this process is shown to be the Kullback–Leibler divergence. The Bayesian approach can be derived as a natural special case. Some illustrations are presented.     

Bayesian prediction based on a class of shrinkage priors for location-scale models

A class of shrinkage priors for multivariate location-scale models is introduced. We consider Bayesian predictive densities for location-scale models and evaluate performance of them using the Kullback–Leibler divergence. We show that Bayesian predictive densities based on priors in the introduced class asymptotically dominate the best invariant predictive density.     

A note on the prior parameter choice in finite mixture models of distributions from exponential families

This work presents a new scheme to obtain the prior distribution parameters in the framework of Rufo et al. (Comput Stat 21:621–637, 2006). Firstly, an analytical expression of the proposed Kullback–Leibler divergence is derived for each distribution in the considered family. Therefore, no previous simulation technique is needed to estimate integrals and thus, the error related to this procedure is avoided. Secondly, a global optimization algorithm based on interval arithmetic is applied to obtain the prior parameters from the derived expression. The main advantage by using this approach is that all solutions are found and rightly bounded. Finally, an application comparing this strategy with the previous one illustrates the proposal.     

Small Noise Asymptotics of the Bayesian Estimator in Nonidentifiable Models

We study the asymptotic behavior of the Bayesian estimator for a deterministic signal in additive Gaussian white noise, in the case where the set of minima of the Kullback–Leibler information is a submanifold of the parameter space. This problem includes as a special case the study of the asymptotic behavior of the nonlinear filter, when the state equation is noise-free, and when the limiting deterministic system is nonobservable. As the noise intensity goes to zero, the posterior probability distribution of the parameter asymptotically concentrates on the submanifold of minima of the Kullback–Leibler information. We give an explicit expression of the limit, and we study the rate of convergence. We apply these results to a practical example where nonidentifiability occurs.     

A general divergence criterion for prior selection

The paper revisits the problem of selection of priors for regular one-parameter family of distributions. The goal is to find some “objective” or “default” prior by approximate maximization of the distance between the prior and the posterior under a general divergence criterion as introduced by Amari (Ann Stat 10:357–387, 1982) and Cressie and Read (J R Stat Soc Ser B 46:440–464, 1984). The maximization is based on an asymptotic expansion of this distance. The Kullback–Leibler, Bhattacharyya–Hellinger and Chi-square divergence are special cases of this general divergence criterion. It is shown that with the exception of one particular case, namely the Chi-square divergence, the general divergence criterion yields Jeffreys’ prior. For the Chi-square divergence, we obtain a prior different from that of Jeffreys and also from that of Clarke and Sun (Sankhya Ser A 59:215–231, 1997).     

Application of the cross-entropy method to clustering and vector quantization

We apply the cross-entropy (CE) method to problems in clustering and vector quantization. The CE algorithm for clustering involves the following iterative steps: (a) generate random clusters according to a specified parametric probability distribution, (b) update the parameters of this distribution according to the Kullback–Leibler cross-entropy. Through various numerical experiments, we demonstrate the high accuracy of the CE algorithm and show that it can generate near-optimal clusters for fairly large data sets. We compare the CE method with well-known clustering and vector quantization methods such as K-means, fuzzy K-means and linear vector quantization, and apply each method to benchmark and image analysis data.     

Bernstein polynomials and learning theory

When learning processes depend on samples but not on the order of the information in the sample, then the Bernoulli distribution is relevant and Bernstein polynomials enter into the analysis. We derive estimates of the approximation of the entropy function x log x that are sharper than the bounds from Voronovskaja's theorem. In this way we get the correct asymptotics for the Kullback–Leibler distance for an encoding problem.     

Improved prediction for a multivariate normal distribution with unknown mean and variance

The prediction problem for a multivariate normal distribution is considered where both mean and variance are unknown. When the Kullback–Leibler loss is used, the Bayesian predictive density based on the right invariant prior, which turns out to be a density of a multivariate t-distribution, is the best invariant and minimax predictive density. In this paper, we introduce an improper shrinkage prior and show that the Bayesian predictive density against the shrinkage prior improves upon the best invariant predictive density when the dimension is greater than or equal to three.     

Information Criteria in Model Selection for Mixing Processes

We present information criteria for statistical model evaluation problems for stochastic processes. The emphasis is put on the use of the asymptotic expansion of the distribution of an estimator based on the conditional Kullback–Leibler divergence for stochastic processes. Asymptotic properties of information criteria and their improvement are discussed. An application to a diffusion process is presented.     

Information Criteria for Small Diffusions via the Theory of Malliavin–Watanabe

Information criteria based on the expected Kullback–Leibler information are presented by means of the asymptotic expansions derived with the Malliavin calculus. We consider the evaluation problem of statistical models for diffusion processes with small noise. The correction terms are essentially different from the ones for ergodic diffusion models presented in Uchida and Yoshida [34, 35].     

Weak interaction limits for one-dimensional random polymers

 In this paper we present a new and flexible method to show that, in one dimension, various self-repellent random walks converge to self-repellent Brownian motion in the limit of weak interaction after appropriate space-time scaling. Our method is based on cutting the path into pieces of an appropriately scaled length, controlling the interaction between the different pieces, and applying an invariance principle to the single pieces. In this way, we show that the self-repellent random walk large deviation rate function for the empirical drift of the path converges to the self-repellent Brownian motion large deviation rate function after appropriate scaling with the interaction parameters. The method is considerably simpler than the approach followed in our earlier work, which was based on functional analytic arguments applied to variational representations and only worked in a very limited number of situations. We consider two examples of a weak interaction limit: (1) vanishing self-repellence, (2) diverging step variance. In example (1), we recover our earlier scaling results for simple random walk with vanishing self-repellence and show how these can be extended to random walk with steps that have zero mean and a finite exponential moment. Moreover, we show that these scaling results are stable against adding self-attraction, provided the self-repellence dominates. In example (2), we prove a conjecture by Aldous for the scaling of self-avoiding walk with diverging step variance. Moreover, we consider self-avoiding walk on a two-dimensional horizontal strip such that the steps in the vertical direction are uniform over the width of the strip and find the scaling as the width tends to infinity. Received: 6 March 2002 / Revised version: 11 October 2002 / Published online: 21 February 2003 Mathematics Subject Classification (2000): 60F05, 60F10, 60J55, 82D60 Key words or phrases: Self-repellent random walk and Brownian motion – Invariance principles – Large deviations – Scaling limits – Universality     

Large Deviations for Brownian Motion on Scale Irregular Sierpinski Gaskets

We study sample path large deviation principles for Brownian motion on scale irregular Sierpinski gaskets which are spatially homogeneous but do not have any exact self-similarity. One notable point of our study is that the rate function depends on a large deviation parameter and as such, we can only obtain an example of large deviations in an incomplete form. Instead of showing the large deviations principle we would expect to hold true, we show Varadhan’s integral lemma and exponential tightness by using an incomplete version of such large deviations.     

Universal bounds on the selfaveraging of random diffraction measures

 We consider diffraction at random point scatterers on general discrete point sets in ℝ^ν, restricted to a finite volume. We allow for random amplitudes and random dislocations of the scatterers. We investigate the speed of convergence of the random scattering measures applied to an observable towards its mean, when the finite volume tends to infinity. We give an explicit universal large deviation upper bound that is exponential in the number of scatterers. The rate is given in terms of a universal function that depends on the point set only through the minimal distance between points, and on the observable only through a suitable Sobolev-norm. Our proof uses a cluster expansion and also provides a central limit theorem. Received: 10 October 2001 / Revised version: 26 January 2003 / Published online: 15 April 2003 Work supported by the DFG Mathematics Subject Classification (2000): 78A45, 82B44, 60F10, 82B20 Key words or phrases: Diffraction theory – Random scatterers – Random point sets – Quasicrystals – Large deviations – Cluster expansions     

Large deviations for increasing sequences on the plane

We prove a large deviation principle with explicit rate functions for the length of the longest increasing sequence among Poisson points on the plane. The rate function for lower tail deviations is derived from a 1977 result of Logan and Shepp about Young diagrams of random permutations. For the upper tail we use a coupling with Hammersley's particle process and convex-analytic techniques. Along the way we obtain the rate function for the lower tail of a tagged particle in a totally asymmetric Hammersley's process. Received: 22 July 1997 / Revised version: 23 March 1998