Aggregated estimators and empirical complexity for least square regression |
| |
Authors: | Jean-Yves Audibert |
| |
Institution: | a Université Paris VI Pierre et Marie Curie, laboratoire de probabilités et modèles aléatoires, 175, rue du Chevaleret, 75013, Paris, France;b CREST, laboratoire de finance et assurance, 15, bd Gabriel Péri, 92245, Malakoff Cedex, France |
| |
Abstract: | Numerous empirical results have shown that combining regression procedures can be a very efficient method. This work provides PAC bounds for the L2 generalization error of such methods. The interest of these bounds are twofold.First, it gives for any aggregating procedure a bound for the expected risk depending on the empirical risk and the empirical complexity measured by the Kullback–Leibler divergence between the aggregating distribution
and a prior distribution π and by the empirical mean of the variance of the regression functions under the probability
.Secondly, by structural risk minimization, we derive an aggregating procedure which takes advantage of the unknown properties of the best mixture
: when the best convex combination
of d regression functions belongs to the d initial functions (i.e. when combining does not make the bias decrease), the convergence rate is of order (logd)/N. In the worst case, our combining procedure achieves a convergence rate of order
which is known to be optimal in a uniform sense when
(see A. Nemirovski, in: Probability Summer School, Saint Flour, 1998; Y. Yang, Aggregating regression procedures for a better performance, 2001]).As in AdaBoost, our aggregating distribution tends to favor functions which disagree with the mixture on mispredicted points. Our algorithm is tested on artificial classification data (which have been also used for testing other boosting methods, such as AdaBoost). |
| |
Keywords: | Nonparametric regression Deviation inequalities Adaptive estimator Oracle inequalities Boosting |
本文献已被 ScienceDirect 等数据库收录! |
|