Learning errors of linear programming support vector regression

In this paper, we give several results of learning errors for linear programming support vector regression. The corresponding theorems are proved in the reproducing kernel Hilbert space. With the covering number, the approximation property and the capacity of the reproducing kernel Hilbert space are measured. The obtained result (Theorem 2.1) shows that the learning error can be controlled by the sample error and regularization error. The mentioned sample error is summarized by the errors of learning regression function and regularizing function in the reproducing kernel Hilbert space. After estimating the generalization error of learning regression function (Theorem 2.2), the upper bound (Theorem 2.3) of the regularized learning algorithm associated with linear programming support vector regression is estimated.     

Learning Rates of Least-Square Regularized Regression

This paper considers the regularized learning algorithm associated with the least-square loss and reproducing kernel Hilbert spaces. The target is the error analysis for the regression problem in learning theory. A novel regularization approach is presented, which yields satisfactory learning rates. The rates depend on the approximation property and on the capacity of the reproducing kernel Hilbert space measured by covering numbers. When the kernel is C^∞ and the regression function lies in the corresponding reproducing kernel Hilbert space, the rate is m^ζ with ζ arbitrarily close to 1, regardless of the variance of the bounded probability distribution.     

Derivative reproducing properties for kernel methods in learning theory

The regularity of functions from reproducing kernel Hilbert spaces (RKHSs) is studied in the setting of learning theory. We provide a reproducing property for partial derivatives up to order s when the Mercer kernel is C^2s. For such a kernel on a general domain we show that the RKHS can be embedded into the function space C^s. These observations yield a representer theorem for regularized learning algorithms involving data for function values and gradients. Examples of Hermite learning and semi-supervised learning penalized by gradients on data are considered.     

Mercer theorem for RKHS on noncompact sets

Reproducing kernel Hilbert spaces are an important family of function spaces and play useful roles in various branches of analysis and applications including the kernel machine learning. When the domain of definition is compact, they can be characterized as the image of the square root of an integral operator, by means of the Mercer theorem. The purpose of this paper is to extend the Mercer theorem to noncompact domains, and to establish a functional analysis characterization of the reproducing kernel Hilbert spaces on general domains.     

Error analysis of multicategory support vector machine classifiers

The paper is related to the error analysis of Multicategory Support Vector Machine (MSVM) classifiers based on reproducing kernel Hilbert spaces. We choose the polynomial kernel as Mercer kernel and give the error estimate with De La Vallée Poussin means. We also introduce the standard estimation of sample error, and derive the explicit learning rate.     

学习算法的稳定性与泛化(Ⅱ):有界差分稳定框架

根据有界差分条件,提出了学习算法的有界差分稳定框架.依据新框架,研究了机器学习阈值选择算法,再生核Hilbert空间中的正则化学习算法,Ranking学习算法和Bagging算法,证明了对应学习算法的有界差分稳定性.所获结果断言了这些算法均具有有界差分稳定性,从而为这些算法的应用奠定了理论基础.     

无界抽样情形下不定核的系数正则化回归

本文讨论样本依赖空间中无界抽样情形下最小二乘损失函数的系数正则化问题. 这里的学习准则与之前再生核Hilbert空间的准则有着本质差异: 核除了满足连续性和有界性之外, 不需要再满足对称性和正定性; 正则化子是函数关于样本展开系数的l²-范数; 样本输出是无界的. 上述差异给误差分析增加了额外难度. 本文的目的是在样本输出不满足一致有界的情形下, 通过l²-经验覆盖数给出误差的集中估计(concentration estimates). 通过引入一个恰当的Hilbert空间以及l²-经验覆盖数的技巧, 得到了与假设空间的容量以及与回归函数的正则性有关的较满意的学习速率.     

Estimates of the norm of the Mercer kernel matrices with discrete orthogonal transforms

The paper is related to the lower and upper estimates of the norm for Mercer kernel matrices. We first give a presentation of the Lagrange interpolating operators from the view of reproducing kernel space. Then, we modify the Lagrange interpolating operators to make them bounded in the space of continuous function and be of the de la Vallée Poussin type. The order of approximation by the reproducing kernel spaces for the continuous functions is thus obtained, from which the lower and upper bounds of the Rayleigh entropy and the l ²-norm for some general Mercer kernel matrices are provided. As an example, we give the l ²-norm estimate for the Mercer kernel matrix presented by the Jacobi algebraic polynomials. The discussions indicate that the l ²-norm of the Mercer kernel matrices may be estimated with discrete orthogonal transforms. Supported by the national NSF (No: 10871226) of P.R. China.     

Learning gradients by a gradient descent algorithm

We propose a stochastic gradient descent algorithm for learning the gradient of a regression function from random samples of function values. This is a learning algorithm involving Mercer kernels. By a detailed analysis in reproducing kernel Hilbert spaces, we provide some error bounds to show that the gradient estimated by the algorithm converges to the true gradient, under some natural conditions on the regression function and suitable choices of the step size and regularization parameters.     

The covering number in learning theory

The covering number of a ball of a reproducing kernel Hilbert space as a subset of the continuous function space plays an important role in Learning Theory. We give estimates for this covering number by means of the regularity of the Mercer kernel K. For convolution type kernels K(x,t)=k(x−t) on [0,1]ⁿ, we provide estimates depending on the decay of k̂, the Fourier transform of k. In particular, when k̂ decays exponentially, our estimate for this covering number is better than all the previous results and covers many important Mercer kernels. A counter example is presented to show that the eigenfunctions of the Hilbert–Schmidt operator Lm_K associated with a Mercer kernel K may not be uniformly bounded. Hence some previous methods used for estimating the covering number in Learning Theory are not valid. We also provide an example of a Mercer kernel to show that L_K^1/2 may not be generated by a Mercer kernel.     

Learning rates of least-square regularized regression with polynomial kernels

This paper presents learning rates for the least-square regularized regression algorithms with polynomial kernels. The target is the error analysis for the regression problem in learning theory. A regularization scheme is given, which yields sharp learning rates. The rates depend on the dimension of polynomial space and polynomial reproducing kernel Hilbert space measured by covering numbers. Meanwhile, we also establish the direct approximation theorem by Bernstein-Durrmeyer operators in with Borel probability measure.      

Generalization errors of Laplacian regularized least squares regression

Semi-supervised learning is an emerging computational paradigm for machine learning,that aims to make better use of large amounts of inexpensive unlabeled data to improve the learning performance.While various methods have been proposed based on different intuitions,the crucial issue of generalization performance is still poorly understood.In this paper,we investigate the convergence property of the Laplacian regularized least squares regression,a semi-supervised learning algorithm based on manifold regularization.Moreover,the improvement of error bounds in terms of the number of labeled and unlabeled data is presented for the first time as far as we know.The convergence rate depends on the approximation property and the capacity of the reproducing kernel Hilbert space measured by covering numbers.Some new techniques are exploited for the analysis since an extra regularizer is introduced.     

Optimal learning rates for least squares regularized regression with unbounded sampling

A standard assumption in theoretical study of learning algorithms for regression is uniform boundedness of output sample values. This excludes the common case with Gaussian noise. In this paper we investigate the learning algorithm for regression generated by the least squares regularization scheme in reproducing kernel Hilbert spaces without the assumption of uniform boundedness for sampling. By imposing some incremental conditions on moments of the output variable, we derive learning rates in terms of regularity of the regression function and capacity of the hypothesis space. The novelty of our analysis is a new covering number argument for bounding the sample error.     

Learning Theory Estimates via Integral Operators and Their Approximations

The regression problem in learning theory is investigated with least square Tikhonov regularization schemes in reproducing kernel Hilbert spaces (RKHS). We follow our previous work and apply the sampling operator to the error analysis in both the RKHS norm and the L² norm. The tool for estimating the sample error is a Bennet inequality for random variables with values in Hilbert spaces. By taking the Hilbert space to be the one consisting of Hilbert-Schmidt operators in the RKHS, we improve the error bounds in the L² metric, motivated by an idea of Caponnetto and de Vito. The error bounds we derive in the RKHS norm, together with a Tsybakov function we discuss here, yield interesting applications to the error analysis of the (binary) classification problem, since the RKHS metric controls the one for the uniform convergence.     

W_2~m空间中样条插值算子与最佳逼近算子的一致性

This paper discusses generalized interpolating splines which determined by n order linear differential operators, and the best operators of interpolating approximation in W_2~m spaces, The explicit constructive method for the reproducing kernel in W_2~m space is presented, and proves the uniformity of spline interpolating operators and the best operators of interpolating approximation W_2~m space by reproducing kernel. The explicit expression of approximation error on a bounded ball in W_2~m space, and error estimation of spline operator of approximation are obtained.     

W_2~m空间中样条插值算子与最佳逼近算子的一致性

由线性微分算子确定的样条是连接多项式样条与希氏空间中抽象算子样条的重要环节,对微分算子样条的研究,既可从更高的观点揭示和概括多项式样条,又可启示我们去发现抽象算子样条的一些新的理论和应用． Ｇｒｅｅｎ函数是研究微分算子样条的重要工具 ［１］,但在微分算子插值样条的计算及将样条用于数值分析中,再生核方法起着更重要的作用．文献［２］［３］给出了与二阶线性微分算子插值样条有关的再生核解析表达式;由此得到了二阶微分算子插值样条与空间Ｗ_２～１［ａ,ｂ］中最佳插值逼近算子的一致性;而且还利用再生核给出了Ｈｉ…     

Approximation with polynomial kernels and SVM classifiers

This paper presents an error analysis for classification algorithms generated by regularization schemes with polynomial kernels. Explicit convergence rates are provided for support vector machine (SVM) soft margin classifiers. The misclassification error can be estimated by the sum of sample error and regularization error. The main difficulty for studying algorithms with polynomial kernels is the regularization error which involves deeply the degrees of the kernel polynomials. Here we overcome this difficulty by bounding the reproducing kernel Hilbert space norm of Durrmeyer operators, and estimating the rate of approximation by Durrmeyer operators in a weighted L¹ space (the weight is a probability distribution). Our study shows that the regularization parameter should decrease exponentially fast with the sample size, which is a special feature of polynomial kernels. Dedicated to Charlie Micchelli on the occasion of his 60th birthday Mathematics subject classifications (2000) 68T05, 62J02. Ding-Xuan Zhou: The first author is supported partially by the Research Grants Council of Hong Kong (Project No. CityU 103704).     

Solving nonlocal initial‐boundary value problems for parabolic and hyperbolic integro‐differential equations in reproducing kernel hilbert space

This article is concerned with a method for solving nonlocal initial‐boundary value problems for parabolic and hyperbolic integro‐differential equations in reproducing kernel Hilbert space. Convergence of the proposed method is studied under some hypotheses which provide the theoretical basis of the proposed method and some error estimates for the numerical approximation in reproducing kernel Hilbert space are presented. Finally, two numerical examples are considered to illustrate the computation efficiency and accuracy of the proposed method. © 2016 The Authors Numerical Methods for Partial Differential Equations Published by Wiley Periodicals, Inc. Numer Methods Partial Differential Eq 33: 174–198, 2017     

Reproducing Kernel Hilbert Spaces Associated with Analytic Translation-Invariant Mercer Kernels

In this article we study reproducing kernel Hilbert spaces (RKHS) associated with translation-invariant Mercer kernels. Applying a special derivative reproducing property, we show that when the kernel is real analytic, every function from the RKHS is real analytic. This is used to investigate subspaces of the RKHS generated by a set of fundamental functions. The analyticity of functions from the RKHS enables us to derive some estimates for the covering numbers which form an essential part for the analysis of some algorithms in learning theory. The work is supported by City University of Hong Kong (Project No. 7001816), and National Science Fund for Distinguished Young Scholars of China (Project No. 10529101).     

QMC Rules of Arbitrary High Order: Reproducing Kernel Hilbert Space Approach

In this paper we consider numerical integration of smooth functions lying in a particular reproducing kernel Hilbert space. We show that the worst-case error of numerical integration in this space converges at the optimal rate, up to some power of a log?N factor. A similar result is shown for the mean square worst-case error, where the bound for the latter is always better than the bound for the square worst-case error. Finally, bounds for integration errors of functions lying in the reproducing kernel Hilbert space are given. The paper concludes by illustrating the theory with numerical results.