The convergence rates of Shannon sampling learning algorithms

In the present paper,we provide an error bound for the learning rates of the regularized Shannon sampling learning scheme when the hypothesis space is a reproducing kernel Hilbert space(RKHS) derived by a Mercer kernel and a determined net.We show that if the sample is taken according to the determined set,then,the sample error can be bounded by the Mercer matrix with respect to the samples and the determined net.The regularization error may be bounded by the approximation order of the reproducing kernel Hilbert space interpolation operator.The paper is an investigation on a remark provided by Smale and Zhou.     

Unregularized online learning algorithms with general loss functions

In this paper, we consider unregularized online learning algorithms in a Reproducing Kernel Hilbert Space (RKHS). Firstly, we derive explicit convergence rates of the unregularized online learning algorithms for classification associated with a general α-activating loss (see Definition 1 below). Our results extend and refine the results in [30] for the least square loss and the recent result [3] for the loss function with a Lipschitz-continuous gradient. Moreover, we establish a very general condition on the step sizes which guarantees the convergence of the last iterate of such algorithms. Secondly, we establish, for the first time, the convergence of the unregularized pairwise learning algorithm with a general loss function and derive explicit rates under the assumption of polynomially decaying step sizes. Concrete examples are used to illustrate our main results. The main techniques are tools from convex analysis, refined inequalities of Gaussian averages [5], and an induction approach.     

Derivative reproducing properties for kernel methods in learning theory

The regularity of functions from reproducing kernel Hilbert spaces (RKHSs) is studied in the setting of learning theory. We provide a reproducing property for partial derivatives up to order s when the Mercer kernel is C^2s. For such a kernel on a general domain we show that the RKHS can be embedded into the function space C^s. These observations yield a representer theorem for regularized learning algorithms involving data for function values and gradients. Examples of Hermite learning and semi-supervised learning penalized by gradients on data are considered.     

Gradient learning in a classification setting by gradient descent

Learning gradients is one approach for variable selection and feature covariation estimation when dealing with large data of many variables or coordinates. In a classification setting involving a convex loss function, a possible algorithm for gradient learning is implemented by solving convex quadratic programming optimization problems induced by regularization schemes in reproducing kernel Hilbert spaces. The complexity for such an algorithm might be very high when the number of variables or samples is huge. We introduce a gradient descent algorithm for gradient learning in classification. The implementation of this algorithm is simple and its convergence is elegantly studied. Explicit learning rates are presented in terms of the regularization parameter and the step size. Deep analysis for approximation by reproducing kernel Hilbert spaces under some mild conditions on the probability measure for sampling allows us to deal with a general class of convex loss functions.     

Behavior of a functional in learning theory

Let H be a Hilbert space, A ∈ L(H), y ∈ , and y ∉ R(A). We study the behavior of the distance square between y and A(B _τ), defined as a functional F(τ), as the radius τ of the ball B _τ of H tends to ∞. This problem is important in estimating the approximation error in learning theory. Our main result is to estimate the asymptotic behavior of F(τ) without the compactness assumption on the operator A. We also consider the Peetre K-functional and its convergence rates.      

Optimal rates for spectral algorithms with least-squares regression over Hilbert spaces

In this paper, we study regression problems over a separable Hilbert space with the square loss, covering non-parametric regression over a reproducing kernel Hilbert space. We investigate a class of spectral/regularized algorithms, including ridge regression, principal component regression, and gradient methods. We prove optimal, high-probability convergence results in terms of variants of norms for the studied algorithms, considering a capacity assumption on the hypothesis space and a general source condition on the target function. Consequently, we obtain almost sure convergence results with optimal rates. Our results improve and generalize previous results, filling a theoretical gap for the non-attainable cases.     

Learning Rates of Least-Square Regularized Regression

This paper considers the regularized learning algorithm associated with the least-square loss and reproducing kernel Hilbert spaces. The target is the error analysis for the regression problem in learning theory. A novel regularization approach is presented, which yields satisfactory learning rates. The rates depend on the approximation property and on the capacity of the reproducing kernel Hilbert space measured by covering numbers. When the kernel is C^∞ and the regression function lies in the corresponding reproducing kernel Hilbert space, the rate is m^ζ with ζ arbitrarily close to 1, regardless of the variance of the bounded probability distribution.     

Reproducing Kernel Hilbert Spaces Associated with Analytic Translation-Invariant Mercer Kernels

In this article we study reproducing kernel Hilbert spaces (RKHS) associated with translation-invariant Mercer kernels. Applying a special derivative reproducing property, we show that when the kernel is real analytic, every function from the RKHS is real analytic. This is used to investigate subspaces of the RKHS generated by a set of fundamental functions. The analyticity of functions from the RKHS enables us to derive some estimates for the covering numbers which form an essential part for the analysis of some algorithms in learning theory. The work is supported by City University of Hong Kong (Project No. 7001816), and National Science Fund for Distinguished Young Scholars of China (Project No. 10529101).     

Optimal learning rates for least squares regularized regression with unbounded sampling

A standard assumption in theoretical study of learning algorithms for regression is uniform boundedness of output sample values. This excludes the common case with Gaussian noise. In this paper we investigate the learning algorithm for regression generated by the least squares regularization scheme in reproducing kernel Hilbert spaces without the assumption of uniform boundedness for sampling. By imposing some incremental conditions on moments of the output variable, we derive learning rates in terms of regularity of the regression function and capacity of the hypothesis space. The novelty of our analysis is a new covering number argument for bounding the sample error.     

Hypercircle inequalities and sampling theory

In this article the well-known hypercircle inequality is extended to the Riesz bases setting. A natural application for this new inequality is given by the estimation of the truncation error in nonorthogonal sampling formulas. Examples including the estimation of the truncation error for wavelet sampling expansions or for nonorthogonal sampling formulas in Paley–Wiener spaces are exhibited.     

Spirals in Hilbert space: With an application in information theory

A (logarithmic) spiral of order is defined as a continuous path tx(t) in a real Hilbert space such that

For α=0 the spiral becomes a helix. The elegant proof by P. Masani of the spectral characterization of helices, due to Kolmogorov and to von Neumann and Schoenberg, is adapted here to spirals. As an application a conjecture by F. Topsøe that certain kernels on considered in information theory are negative definite, and hence are squares of metrics on , is confirmed.     

Error estimation for non-uniform sampling in shift invariant space

To reconstruct a function from its sampling value is not always exact, error may arise due to a lot of reasons, therefore error estimation is useful in reconstruction. For non-uniform sampling in shift invariant space, three kinds of errors of the reconstruction formula are discussed in this article. For every kind of error, we give an estimation. We find the accuracy of the reconstruction formula mainly depends on the decay property of the generator and the sampling function.     

Using a natural deconvolution for analysis of perturbed integer sampling in shift-invariant spaces

An important cornerstone of both wavelet and sampling theory is shift-invariant spaces, that is, spaces V spanned by a Riesz basis of integer-translates of a single function. Under some mild differentiability and decay assumptions on the Fourier transform of this function, we show that V also is generated by a function with Fourier transform for some g with . We explain why analysis of this particular generating function can be more likely to provide large jitter bounds ε such that any f∈V can be reconstructed from perturbed integer samples f(k+εk) whenever supk∈Z|εk|?ε. We use this natural deconvolution of to further develop analysis techniques from a previous paper. Then we demonstrate the resulting analysis method on the class of spaces for which g has compact support and bounded variation (including all spaces generated by Meyer wavelet scaling functions), on some particular choices of φ for which we know of no previously published bounds and finally, we use it to improve some previously known bounds for B-spline shift-invariant spaces.     

Learning gradients by a gradient descent algorithm

We propose a stochastic gradient descent algorithm for learning the gradient of a regression function from random samples of function values. This is a learning algorithm involving Mercer kernels. By a detailed analysis in reproducing kernel Hilbert spaces, we provide some error bounds to show that the gradient estimated by the algorithm converges to the true gradient, under some natural conditions on the regression function and suitable choices of the step size and regularization parameters.     

On optimal wavelet reconstructions from Fourier samples: Linearity and universality of the stable sampling rate

In this paper we study the problem of computing wavelet coefficients of compactly supported functions from their Fourier samples. For this, we use the recently introduced framework of generalized sampling. Our first result demonstrates that using generalized sampling one obtains a stable and accurate reconstruction, provided the number of Fourier samples grows linearly in the number of wavelet coefficients recovered. For the class of Daubechies wavelets we derive the exact constant of proportionality.Our second result concerns the optimality of generalized sampling for this problem. Under some mild assumptions we show that generalized sampling cannot be outperformed in terms of approximation quality by more than a constant factor. Moreover, for the class of so-called perfect methods, any attempt to lower the sampling ratio below a certain critical threshold necessarily results in exponential ill-conditioning. Thus generalized sampling provides a nearly-optimal solution to this problem.     

Learning and approximation by Gaussians on Riemannian manifolds

Learning function relations or understanding structures of data lying in manifolds embedded in huge dimensional Euclidean spaces is an important topic in learning theory. In this paper we study the approximation and learning by Gaussians of functions defined on a d-dimensional connected compact C ^∞ Riemannian submanifold of which is isometrically embedded. We show that the convolution with the Gaussian kernel with variance σ provides the uniform approximation order of O(σ ^s ) when the approximated function is Lipschitz s ∈(0, 1]. The uniform normal neighborhoods of a compact Riemannian manifold play a central role in deriving the approximation order. This approximation result is used to investigate the regression learning algorithm generated by the multi-kernel least square regularization scheme associated with Gaussian kernels with flexible variances. When the regression function is Lipschitz s, our learning rate is (log² m)/m) ^{s/(8 s + 4 d)} where m is the sample size. When the manifold dimension d is smaller than the dimension n of the underlying Euclidean space, this rate is much faster compared with those in the literature. By comparing approximation orders, we also show the essential difference between approximation schemes with flexible variances and those with a single variance. Supported partially by the Research Grants Council of Hong Kong [Project No. CityU 103405], City University of Hong Kong [Project No. 7001983], National Science Fund for Distinguished Young Scholars of China [Project No. 10529101], and National Basic Research Program of China [Project No. 973-2006CB303102].     

Operator theory on noncommutative varieties, II

An -tuple of operators on a Hilbert space is called a -constrained row contraction if and

where is a WOT-closed two-sided ideal of the noncommutative analytic Toeplitz algebra and is defined using the -functional calculus for row contractions.

We show that the constrained characteristic function associated with and is a complete unitary invariant for -constrained completely non-coisometric (c.n.c.) row contractions. We also provide a model for this class of row contractions in terms of the constrained characteristic functions. In particular, we obtain a model theory for -commuting c.n.c. row contractions.