Two-sample homogeneity tests based on divergence measures

The concept of f-divergences introduced by Ali and Silvey (J R Stat Soc (B) 28:131–142, 1996) provides a rich set of distance like measures between pairs of distributions. Divergences do not focus on certain moments of random variables, but rather consider discrepancies between the corresponding probability density functions. Thus, two-sample tests based on these measures can detect arbitrary alternatives when testing the equality of the distributions. We treat the problem of divergence estimation as well as the subsequent testing for the homogeneity of two-samples. In particular, we propose a nonparametric estimator for f-divergences in the case of continuous distributions, which is based on kernel density estimation and spline smoothing. As we show in extensive simulations, the new method performs stable and quite well in comparison to several existing non- and semiparametric divergence estimators. Furthermore, we tackle the two-sample homogeneity problem using permutation tests based on various divergence estimators. The methods are compared to an asymptotic divergence test as well as to several traditional parametric and nonparametric procedures under different distributional assumptions and alternatives in simulations. It turns out that divergence based methods detect discrepancies between distributions more often than traditional methods if the distributions do not differ in location only. The findings are illustrated on ion mobility spectrometry data.     

A review on consistency and robustness properties of support vector machines for heavy-tailed distributions

Support vector machines (SVMs) belong to the class of modern statistical machine learning techniques and can be described as M-estimators with a Hilbert norm regularization term for functions. SVMs are consistent and robust for classification and regression purposes if based on a Lipschitz continuous loss and a bounded continuous kernel with a dense reproducing kernel Hilbert space. For regression, one of the conditions used is that the output variable Y has a finite first absolute moment. This assumption, however, excludes heavy-tailed distributions. Recently, the applicability of SVMs was enlarged to these distributions by considering shifted loss functions. In this review paper, we briefly describe the approach of SVMs based on shifted loss functions and list some properties of such SVMs. Then, we prove that SVMs based on a bounded continuous kernel and on a convex and Lipschitz continuous, but not necessarily differentiable, shifted loss function have a bounded Bouligand influence function for all distributions, even for heavy-tailed distributions including extreme value distributions and Cauchy distributions. SVMs are thus robust in this sense. Our result covers the important loss functions ${\epsilon}$ -insensitive for regression and pinball for quantile regression, which were not covered by earlier results on the influence function. We demonstrate the usefulness of SVMs even for heavy-tailed distributions by applying SVMs to a simulated data set with Cauchy errors and to a data set of large fire insurance claims of Copenhagen Re.     

Estimating the intensity of a random measure by histogram type estimators

The purpose of this paper is to estimate the intensity of some random measure N on a set ${\mathcal{X}}$ by a piecewise constant function on a finite partition of ${\mathcal{X}}$ . Given a (possibly large) family ${\mathcal{M}}$ of candidate partitions, we build a piecewise constant estimator (histogram) on each of them and then use the data to select one estimator in the family. Choosing the square of a Hellinger-type distance as our loss function, we show that each estimator built on a given partition satisfies an analogue of the classical squared bias plus variance risk bound. Moreover, the selection procedure leads to a final estimator satisfying some oracle-type inequality, with, as usual, a possible loss corresponding to the complexity of the family ${\mathcal{M}}$ . When this complexity is not too high, the selected estimator has a risk bounded, up to a universal constant, by the smallest risk bound obtained for the estimators in the family. For suitable choices of the family of partitions, we deduce uniform risk bounds over various classes of intensities. Our approach applies to the estimation of the intensity of an inhomogenous Poisson process, among other counting processes, or the estimation of the mean of a random vector with nonnegative components.     

A sliced inverse regression approach for data stream

In this article, we focus on data arriving sequentially by blocks in a stream. A semiparametric regression model involving a common effective dimension reduction (EDR) direction \(\beta \) is assumed in each block. Our goal is to estimate this direction at each arrival of a new block. A simple direct approach consists of pooling all the observed blocks and estimating the EDR direction by the sliced inverse regression (SIR) method. But in practice, some disadvantages appear such as the storage of the blocks and the running time for large sample sizes. To overcome these drawbacks, we propose an adaptive SIR estimator of \(\beta \) based on the optimization of a quality measure. The corresponding approach is faster both in terms of computational complexity and running time, and provides data storage benefits. The consistency of our estimator is established and its asymptotic distribution is given. An extension to multiple indices model is proposed. A graphical tool is also provided in order to detect changes in the underlying model, i.e., drift in the EDR direction or aberrant blocks in the data stream. A simulation study illustrates the numerical behavior of our estimator. Finally, an application to real data concerning the estimation of physical properties of the Mars surface is presented.     

A general estimator for the right endpoint with an application to supercentenarian women’s records

We extend the setting of the right endpoint estimator introduced in Fraga Alves and Neves (Statist. Sinica 24, 1811–1835, 2014) to the broader class of light-tailed distributions with finite endpoint, belonging to some domain of attraction induced by the extreme value theorem. This stretch enables a general estimator for the finite endpoint, which does not require estimation of the (supposedly non-positive) extreme value index. A new testing procedure for selecting max-domains of attraction also arises in connection with the asymptotic properties of the general endpoint estimator. The simulation study conveys that the general endpoint estimator is a valuable complement to the most usual endpoint estimators, particularly when the true extreme value index stays above ?1/2, embracing the most common cases in practical applications. An illustration is provided via an extreme value analysis of supercentenarian women data.     

The Graphical Horseshoe Estimator for Inverse Covariance Matrices

We develop a new estimator of the inverse covariance matrix for high-dimensional multivariate normal data using the horseshoe prior. The proposed graphical horseshoe estimator has attractive properties compared to other popular estimators, such as the graphical lasso and the graphical smoothly clipped absolute deviation. The most prominent benefit is that when the true inverse covariance matrix is sparse, the graphical horseshoe provides estimates with small information divergence from the sampling model. The posterior mean under the graphical horseshoe prior can also be almost unbiased under certain conditions. In addition to these theoretical results, we also provide a full Gibbs sampler for implementing our estimator. MATLAB code is available for download from github at http://github.com/liyf1988/GHS. The graphical horseshoe estimator compares favorably to existing techniques in simulations and in a human gene network data analysis. Supplementary materials for this article are available online.     

Likelihood estimation of the extremal index

The article develops the approach of Ferro and Segers (J.R. Stat. Soc., Ser. B 65:545, 2003) to the estimation of the extremal index, and proposes the use of a new variable decreasing the bias of the likelihood based on the point process character of the exceedances. Two estimators are discussed: a maximum likelihood estimator and an iterative least squares estimator based on the normalized gaps between clusters. The first provides a flexible tool for use with smoothing methods. A diagnostic is given for condition , under which maximum likelihood is valid. The performance of the new estimators were tested by extensive simulations. An application to the Central England temperature series demonstrates the use of the maximum likelihood estimator together with smoothing methods.      

Birnbaum–Saunders power-exponential kernel density estimation and Bayes local bandwidth selection for nonnegative heavy tailed data

In this paper, we study the performance of the Birnbaum–Saunders-power-exponential (BS-PE) kernel and Bayesian local bandwidth selection in the context of kernel density estimation for nonnegative heavy tailed data. Our approach considers the BS-PE kernel estimator and treats locally the bandwidth h as a parameter with prior distribution. The posterior density of h at each point x (point where the density is estimated) is derived in closed form, and the Bayesian bandwidth selector is obtained by using popular loss functions. The performance evaluation of this new procedure is carried out by a simulation study and real data in web-traffic. The proposed method is very quick and very competitive in comparison with the existing global methods, namely biased cross-validation and unbiased cross-validation.     

Robust kernel-based regression with bounded influence for outliers

The kernel-based regression (KBR) method, such as support vector machine for regression (SVR) is a well-established methodology for estimating the nonlinear functional relationship between the response variable and predictor variables. KBR methods can be very sensitive to influential observations that in turn have a noticeable impact on the model coefficients. The robustness of KBR methods has recently been the subject of wide-scale investigations with the aim of obtaining a regression estimator insensitive to outlying observations. However, existing robust KBR (RKBR) methods only consider Y-space outliers and, consequently, are sensitive to X-space outliers. As a result, even a single anomalous outlying observation in X-space may greatly affect the estimator. In order to resolve this issue, we propose a new RKBR method that gives reliable result even if a training data set is contaminated with both Y-space and X-space outliers. The proposed method utilizes a weighting scheme based on the hat matrix that resembles the generalized M-estimator (GM-estimator) of conventional robust linear analysis. The diagonal elements of hat matrix in kernel-induced feature space are used as leverage measures to downweight the effects of potential X-space outliers. We show that the kernelized hat diagonal elements can be obtained via eigen decomposition of the kernel matrix. The regularized version of kernelized hat diagonal elements is also proposed to deal with the case of the kernel matrix having full rank where the kernelized hat diagonal elements are not suitable for leverage. We have shown that two kernelized leverage measures, namely, the kernel hat diagonal element and the regularized one, are related to statistical distance measures in the feature space. We also develop an efficiently kernelized training algorithm for the parameter estimation based on iteratively reweighted least squares (IRLS) method. The experimental results from simulated examples and real data sets demonstrate the robustness of our proposed method compared with conventional approaches.     

Rank Conditional Coverage and Confidence Intervals in High-Dimensional Problems

Confidence interval procedures used in low-dimensional settings are often inappropriate for high-dimensional applications. When many parameters are estimated, marginal confidence intervals associated with the most significant estimates have very low coverage rates: They are too small and centered at biased estimates. The problem of forming confidence intervals in high-dimensional settings has previously been studied through the lens of selection adjustment. In that framework, the goal is to control the proportion of noncovering intervals formed for selected parameters. In this article, we approach the problem by considering the relationship between rank and coverage probability. Marginal confidence intervals have very low coverage rates for the most significant parameters and high rates for parameters with more boring estimates. Many selection adjusted intervals have the same behavior despite controlling the coverage rate within a selected set. This relationship between rank and coverage rate means that the parameters most likely to be pursued further in follow-up or replication studies are the least likely to be covered by the constructed intervals. In this article, we propose rank conditional coverage (RCC) as a new coverage criterion for confidence intervals in multiple testing/covering problems. The RCC is the expected coverage rate of an interval given the significance ranking for the associated estimator. We also propose two methods that use bootstrapping to construct confidence intervals that control the RCC. Because these methods make use of additional information captured by the ranks of the parameter estimates, they often produce smaller intervals than marginal or selection adjusted methods. These methods are implemented in R (R Core Team, 2017 R Core Team (2017), R: A Language and Environment for Statistical Computing, Vienna, Austria: R Foundation for Statistical Computing. [Google Scholar]) in the package rcc available on CRAN at https://cran.r-project.org/web/packages/rcc/index.html. Supplementary material for this article is available online.     

Structural adaptive deconvolution under $${\mathbb{L}_p}$$-losses

In this paper, we address the problem of estimating a multidimensional density f by using indirect observations from the statistical model Y = X + ε. Here, ε is a measurement error independent of the random vector X of interest and having a known density with respect to Lebesgue measure. Our aim is to obtain optimal accuracy of estimation under \({\mathbb{L}_p}\)-losses when the error ε has a characteristic function with a polynomial decay. To achieve this goal, we first construct a kernel estimator of f which is fully data driven. Then, we derive for it an oracle inequality under very mild assumptions on the characteristic function of the error ε. As a consequence, we getminimax adaptive upper bounds over a large scale of anisotropic Nikolskii classes and we prove that our estimator is asymptotically rate optimal when p ∈ [2,+∞]. Furthermore, our estimation procedure adapts automatically to the possible independence structure of f and this allows us to improve significantly the accuracy of estimation.     

Tail fitting for truncated and non-truncated Pareto-type distributions

In extreme value analysis, natural upper bounds can appear that truncate the probability tail. At other instances ultimately at the largest data, deviations from a Pareto tail behaviour become apparent. This matter is especially important when extrapolation outside the sample is required. Given that in practice one does not always know whether the distribution is truncated or not, we consider estimators for extreme quantiles both under truncated and non-truncated Pareto-type distributions. We make use of the estimator of the tail index for the truncated Pareto distribution first proposed in Aban et al. (J. Amer. Statist. Assoc. 101(473), 270–277, 2006). We also propose a truncated Pareto QQ-plot and a formal test for truncation in order to help deciding between a truncated and a non-truncated case. In this way we enlarge the possibilities of extreme value modelling using Pareto tails, offering an alternative scenario by adding a truncation point T that is large with respect to the available data. In the mathematical modelling we hence let T→∞ at different speeds compared to the limiting fraction (k/n→0) of data used in the extreme value estimation. This work is motivated using practical examples from different fields, simulation results, and some asymptotic results.     

Optimal variance estimation based on lagged second-order difference in nonparametric regression

Differenced estimators of variance bypass the estimation of regression function and thus are simple to calculate. However, there exist two problems: most differenced estimators do not achieve the asymptotic optimal rate for the mean square error; for finite samples the estimation bias is also important and not further considered. In this paper, we estimate the variance as the intercept in a linear regression with the lagged Gasser-type variance estimator as dependent variable. For the equidistant design, our estimator is not only \(n^{1/2}\)-consistent and asymptotically normal, but also achieves the optimal bound in terms of estimation variance with less asymptotic bias. Simulation studies show that our estimator has less mean square error than some existing differenced estimators, especially in the cases of immense oscillation of regression function and small-sized sample.     

Zero-correlation attacks: statistical models independent of the number of approximations

Multiple and multidimensional zero-correlation linear cryptanalysis have been two of the most powerful cryptanalytic techniques for block ciphers, and it has been shown that the differentiating factor of these two statistical models is whether distinct plaintexts are assumed or not. Nevertheless, questions remain regarding how these analyses can be universalized without any limitations and can be used to accurately estimate the data complexity and the success probability. More concretely, the current models for multiple zero-correlation (MPZC) and multidimensional zero-correlation (MDZC) cryptanalysis are not valid in the setting with a limited number of approximations and the accuracy of the estimation for data complexity can not be guaranteed. Besides, in a lot of cases, using too many approximations may cause an exhaustive search when we want to launch key-recovery attacks. In order to generalize the original models using the normal approximation of the \(\chi ^2\)-distribution, we provide a more accurate approach to estimate the data complexity and the success probability for MPZC and MDZC cryptanalysis without such approximation. Since these new models directly rely on the \(\chi ^{2}\)-distribution, we call them the \(\chi ^{2}\) MPZC and MDZC models. An interesting thing is that the chi-square-multiple zero-correlation (\(\chi ^{2}\)-MPZC) model still works even though we only have a single zero-correlation linear approximation. This fact puts an end to the situation that the basic zero-correlation linear cryptanalysis requires the full codebook under the known-plaintext attack setting. As an illustration, we apply the \(\chi ^{2}\)-MPZC model to analyze TEA and XTEA. These new attacks cover more rounds than the previous MPZC attacks. Moreover, we reconsider the multidimensional zero-correlation (MDZC) attack on 14-round CLEFIA-192 by utilizing less zero-correlation linear approximations. In addition, some other ciphers which already have MDZC analytical results are reevaluated and the data complexities under the new model are all less than or equal to those under the original model. Some experiments are conducted in order to verify the validity of the new models, and the experimental results convince us that the new models provide more precise estimates of the data complexity and the success probability.     

Integral curves from noisy diffusion MRI data with closed-form uncertainty estimates

We present a method for estimating the trajectories of axon fibers through diffusion tensor MRI (DTI) data that provides theoretically rigorous estimates of trajectory uncertainty. We develop a three-step estimation procedure based on a kernel estimator for a tensor field based on the raw DTI measurements, followed by a plug-in estimator for the leading eigenvectors of the tensors, and a plug-in estimator for integral curves through the resulting vector field. The integral curve estimator is asymptotically normal; the covariance of the limiting Gaussian process allows us to construct confidence ellipsoids for fixed points along the curve. Complete trajectories of fibers are assembled by stopping integral curve tracing at locations with multiple viable leading eigenvector directions and tracing a new curve along each direction. Unlike probabilistic tractography approaches to this problem, we provide a rigorous, theoretically sound model of measurement uncertainty as it propagates from the raw MRI data, to the tensor field, to the vector field, to the integral curves. In addition, trajectory uncertainty is estimated in closed form while probabilistic tractography relies on sampling the space of tensors, vectors, or curves. We show that our estimator provides more realistic trajectory uncertainty estimates than a more simplified prior approach for closed-form trajectory uncertainty estimation due to Koltchinskii et al. (Ann Stat 35:1576–1607, 2007) and a popular probabilistic tractography method due to Behrens et al. (Magn Reson Med 50:1077–1088, 2003) using theory, simulation, and real DTI scans.     

Semiparametric estimation of the link function in binary-choice single-index models

We propose a new, easy to implement, semiparametric estimator for binary-choice single-index models which uses parametric information in the form of a known link (probability) function and nonparametrically corrects it. Asymptotic properties are derived and the finite sample performance of the proposed estimator is compared to those of the parametric probit and semiparametric single-index model estimators of Ichimura (J Econ 58:71–120, 1993) and Klein and Spady (Econometrica 61:387–421, 1993). Results indicate that if the parametric start is correct, the proposed estimator achieves significant bias reduction and efficiency gains compared to Ichimura (1993) and Klein and Spady (1993). Interestingly, the proposed estimator still achieves significant bias reduction and efficiency gains even if the parametric start is not correct.     

Approximation and estimation of very small probabilities of multivariate extreme events

This article discusses modelling of the tail of a multivariate distribution function by means of a large deviation principle (LDP), and its application to the estimation of the probability p _n of a multivariate extreme event from a sample of n iid random vectors, with \(p_{n}\in [n^{-\tau _{2}},n^{-\tau _{1}}]\) for some t ₁>1 and t ₂>t ₁. One way to view the classical tail limits is as limits of probability ratios. In contrast, the tail LDP provides asymptotic bounds or limits for log-probability ratios. After standardising the marginals to standard exponential, tail dependence is represented by a homogeneous rate function I. Furthermore, the tail LDP can be extended to represent both dependence and marginals, the latter implying marginal log-Generalised Weibull tail limits. A connection is established between the tail LDP and residual tail dependence (or hidden regular variation) and a recent extension of it. Under a smoothness assumption, they are implied by the tail LDP. Based on the tail LDP, a simple estimator for very small probabilities of extreme events is formulated. It avoids estimation of I by making use of its homogeneity. Strong consistency in the sense of convergence of log-probability ratios is proven. Simulations and an application illustrate the difference between the classical approach and the LDP-based approach.     

Robust multicategory support vector machines using difference convex algorithm

The support vector machine (SVM) is one of the most popular classification methods in the machine learning literature. Binary SVM methods have been extensively studied, and have achieved many successes in various disciplines. However, generalization to multicategory SVM (MSVM) methods can be very challenging. Many existing methods estimate k functions for k classes with an explicit sum-to-zero constraint. It was shown recently that such a formulation can be suboptimal. Moreover, many existing MSVMs are not Fisher consistent, or do not take into account the effect of outliers. In this paper, we focus on classification in the angle-based framework, which is free of the explicit sum-to-zero constraint, hence more efficient, and propose two robust MSVM methods using truncated hinge loss functions. We show that our new classifiers can enjoy Fisher consistency, and simultaneously alleviate the impact of outliers to achieve more stable classification performance. To implement our proposed classifiers, we employ the difference convex algorithm for efficient computation. Theoretical and numerical results obtained indicate that for problems with potential outliers, our robust angle-based MSVMs can be very competitive among existing methods.     

Two-Dimensional Solution Surface for Weighted Support Vector Machines

The support vector machine (SVM) is a popular learning method for binary classification. Standard SVMs treat all the data points equally, but in some practical problems it is more natural to assign different weights to observations from different classes. This leads to a broader class of learning, the so-called weighted SVMs (WSVMs), and one of their important applications is to estimate class probabilities besides learning the classification boundary. There are two parameters associated with the WSVM optimization problem: one is the regularization parameter and the other is the weight parameter. In this article, we first establish that the WSVM solutions are jointly piecewise-linear with respect to both the regularization and weight parameter. We then develop a state-of-the-art algorithm that can compute the entire trajectory of the WSVM solutions for every pair of the regularization parameter and the weight parameter at a feasible computational cost. The derived two-dimensional solution surface provides theoretical insight on the behavior of the WSVM solutions. Numerically, the algorithm can greatly facilitate the implementation of the WSVM and automate the selection process of the optimal regularization parameter. We illustrate the new algorithm on various examples. This article has online supplementary materials.