Reproducing Kernel Hilbert Spaces and fractal interpolation

Reproducing Kernel Hilbert Spaces (RKHSs) are a very useful and powerful tool of functional analysis with application in many diverse paradigms, such as multivariate statistics and machine learning. Fractal interpolation, on the other hand, is a relatively recent technique that generalizes traditional interpolation through the introduction of self-similarity. In this work we show that the functional space of any family of (recurrent) fractal interpolation functions ((R)FIFs) constitutes an RKHS with a specific associated kernel function, thus, extending considerably the toolbox of known kernel functions and introducing fractals to the RKHS world. We also provide the means for the computation of the kernel function that corresponds to any specific fractal RKHS and give several examples.     

Reproducing Kernel Hilbert Spaces Associated with Analytic Translation-Invariant Mercer Kernels

In this article we study reproducing kernel Hilbert spaces (RKHS) associated with translation-invariant Mercer kernels. Applying a special derivative reproducing property, we show that when the kernel is real analytic, every function from the RKHS is real analytic. This is used to investigate subspaces of the RKHS generated by a set of fundamental functions. The analyticity of functions from the RKHS enables us to derive some estimates for the covering numbers which form an essential part for the analysis of some algorithms in learning theory. The work is supported by City University of Hong Kong (Project No. 7001816), and National Science Fund for Distinguished Young Scholars of China (Project No. 10529101).     

Learning Theory Estimates via Integral Operators and Their Approximations

The regression problem in learning theory is investigated with least square Tikhonov regularization schemes in reproducing kernel Hilbert spaces (RKHS). We follow our previous work and apply the sampling operator to the error analysis in both the RKHS norm and the L² norm. The tool for estimating the sample error is a Bennet inequality for random variables with values in Hilbert spaces. By taking the Hilbert space to be the one consisting of Hilbert-Schmidt operators in the RKHS, we improve the error bounds in the L² metric, motivated by an idea of Caponnetto and de Vito. The error bounds we derive in the RKHS norm, together with a Tsybakov function we discuss here, yield interesting applications to the error analysis of the (binary) classification problem, since the RKHS metric controls the one for the uniform convergence.     

Sobolev-type approximation rates for divergence-free and curl-free RBF interpolants

Recently, error estimates have been made available for divergence-free radial basis function (RBF) interpolants. However, these results are only valid for functions within the associated reproducing kernel Hilbert space (RKHS) of the matrix-valued RBF. Functions within the associated RKHS, also known as the ``native space' of the RBF, can be characterized as vector fields having a specific smoothness, making the native space quite small. In this paper we develop Sobolev-type error estimates when the target function is less smooth than functions in the native space.

     

Wavelet-RKHS-based functional statistical classification

A functional classification methodology, based on the Reproducing Kernel Hilbert Space (RKHS) theory, is proposed for discrimination of gene expression profiles. The parameter function involved in the definition of the functional logistic regression is univocally and consistently estimated, from the minimization of the penalized negative log-likelihood over a RKHS generated by a suitable wavelet basis. An iterative descendent method, the gradient method, is applied for solving the corresponding minimization problem, i.e., for computing the functional estimate. Temporal gene expression data involved in the yeast cell cycle are classified with the wavelet-RKHS-based discrimination methodology considered. A simulation study is developed for testing the performance of this statistical classification methodology in comparison with other statistical discrimination procedures.     

Sharp estimates for the covering numbers of the Weierstrass fractal kernel

In this paper, we present sharp estimates for the covering numbers of the embedding of the reproducing kernel Hilbert space (RKHS) associated with the Weierstrass fractal kernel into the space of continuous functions. The method we apply is based on the characterization of the infinite-dimensional RKHS generated by the Weierstrass fractal kernel and it requires estimates for the norm operator of orthogonal projections on the RKHS.     

Support vector machines regression with l1-regularizer

The classical support vector machines regression (SVMR) is known as a regularized learning algorithm in reproducing kernel Hilbert spaces (RKHS) with a $\epsilon$-insensitive loss function and an RKHS norm regularizer. In this paper, we study a new SVMR algorithm where the regularization term is proportional to $l^1$-norm of the coefficients in the kernel ensembles. We provide an error analysis of this algorithm, an explicit learning rate is then derived under some assumptions.     

Thresholding in Learning Theory

In this paper we investigate the problem of learning an unknown bounded function. We will emphasize special cases where it is possible to provide very simple (in terms of computation) estimates enjoying, in addition, the property of being universal, i.e. their construction does not depend on the a priori knowledge of regularity conditions on the unknown object and still they have almost optimal properties for a whole group of functions spaces. These estimates are constructed using a thresholding technique, which has proven in the last decade in statistics to have very good properties for recovering signals with inhomogeneous smoothness but has not been extensively developed in learning theory. We will basically consider two particular situations. In the first case, we consider the RKHS situation, where we produce a new algorithm and investigate its performances in . The exponential rates of convergences are proved to be almost optimal, and the regularity assumptions are expressed in simple terms. The second case considers a more specified situation where the X_i's are one-dimensional and the estimator is a wavelet thresholding estimate. The results are comparable in this setting to those obtained in the RKHS situation, as concerned the critical value and the exponential rates. The advantage here is that we are able to state the results in the -norm and the regularity conditions are expressed in terms of standard Holder spaces.     

Reproducing Properties of Differentiable Mercer-Like Kernels on the Sphere

We study differentiability of functions in the reproducing kernel Hilbert space (RKHS) associated with a smooth Mercer-like kernel on the sphere. We show that differentiability up to a certain order of the kernel yields both, differentiability up to the same order of the elements in the series representation of the kernel and a series representation for the corresponding derivatives of the kernel. These facts are used to embed the RKHS into spaces of differentiable functions and to deduce reproducing properties for the derivatives of functions in the RKHS. We discuss compactness and boundedness of the embedding and some applications to Gaussian-like kernels.     

Least-squares regularized regression with dependent samples and q-penalty

Least-squares regularized learning algorithms for regression were well-studied in the literature when the sampling process is independent and the regularization term is the square of the norm in a reproducing kernel Hilbert space (RKHS). Some analysis has also been done for dependent sampling processes or regularizers being the qth power of the function norm (q-penalty) with 0?q?≤?2. The purpose of this article is to conduct error analysis of the least-squares regularized regression algorithm when the sampling sequence is weakly dependent satisfying an exponentially decaying α-mixing condition and when the regularizer takes the q-penalty with 0?q?≤?2. We use a covering number argument and derive learning rates in terms of the α-mixing decay, an approximation condition and the capacity of balls of the RKHS.     

Orthogonality from disjoint support in reproducing kernel Hilbert spaces

We investigate reproducing kernel Hilbert spaces (RKHS) where two functions are orthogonal whenever they have disjoint support. Necessary and sufficient conditions in terms of feature maps for the reproducing kernel are established. We also present concrete examples of finite dimensional RKHS and RKHS with a translation invariant reproducing kernel. In particular, it is shown that a Sobolev space has the orthogonality from disjoint support property if and only if it is of integer index.     

Weighted Fourier–Laplace transforms in reproducing kernel Hilbert spaces on the sphere

We study the action of a weighted Fourier–Laplace transform on the functions in the reproducing kernel Hilbert space (RKHS) associated with a positive definite kernel on the sphere. After defining a notion of smoothness implied by the transform, we show that smoothness of the kernel implies the same smoothness for the generating elements (spherical harmonics) in the Mercer expansion of the kernel. We prove a reproducing property for the weighted Fourier–Laplace transform of the functions in the RKHS and embed the RKHS into spaces of smooth functions. Some relevant properties of the embedding are considered, including compactness and boundedness. The approach taken in the paper includes two important notions of differentiability characterized by weighted Fourier–Laplace transforms: fractional derivatives and Laplace–Beltrami derivatives.     

Risk Bounds for Random Regression Graphs

We consider the regression problem and describe an algorithm approximating the regression function by estimators piecewise constant on the elements of an adaptive partition. The partitions are iteratively constructed by suitable random merges and splits, using cuts of arbitrary geometry. We give a risk bound under the assumption that a "weak learning hypothesis" holds, and characterize this hypothesis in terms of a suitable RKHS. Two examples illustrate the general results in two particularly interesting cases.     

Recent developments in error estimates for scattered-data interpolation via radial basis functions

Error estimates for scattered data interpolation by shifts of a positive definite function for target functions in the associated reproducing kernel Hilbert space (RKHS) have been known for a long time. However, apart from special cases where data is gridded, these interpolation estimates do not apply when the target functions generating the data are outside of the associated RKHS, and in fact until very recently no estimates were known in such situations. In this paper, we review these estimates in cases where the underlying space is Rⁿ and the positive definite functions are radial basis functions (RBFs). AMS subject classification 41A25, 41A05, 41A63, 42B35Research supported by grant DMS-0204449 from the National Science Foundation.     

Online Gradient Descent Learning Algorithms

This paper considers the least-square online gradient descent algorithm in a reproducing kernel Hilbert space (RKHS) without an explicit regularization term. We present a novel capacity independent approach to derive error bounds and convergence results for this algorithm. The essential element in our analysis is the interplay between the generalization error and a weighted cumulative error which we define in the paper. We show that, although the algorithm does not involve an explicit RKHS regularization term, choosing the step sizes appropriately can yield competitive error rates with those in the literature.     

Sobolev Error Estimates and a Bernstein Inequality for Scattered Data Interpolation via Radial Basis Functions

Error estimates for scattered-data interpolation via radial basis functions (RBFs) for target functions in the associated reproducing kernel Hilbert space (RKHS) have been known for a long time. Recently, these estimates have been extended to apply to certain classes of target functions generating the data which are outside the associated RKHS. However, these classes of functions still were not "large" enough to be applicable to a number of practical situations. In this paper we obtain Sobolev-type error estimates on compact regions of Rⁿ when the RBFs have Fourier transforms that decay algebraically. In addition, we derive a Bernstein inequality for spaces of finite shifts of an RBF in terms of the minimal separation parameter.     

学习算法的稳定性与泛化(Ⅱ):有界差分稳定框架

根据有界差分条件,提出了学习算法的有界差分稳定框架.依据新框架,研究了机器学习阈值选择算法,再生核Hilbert空间中的正则化学习算法,Ranking学习算法和Bagging算法,证明了对应学习算法的有界差分稳定性.所获结果断言了这些算法均具有有界差分稳定性,从而为这些算法的应用奠定了理论基础.     

On restricted linear estimation for regression in stochastic processes

In this paper the problem of restricted linear estimation for regression in stochastic processes is analyzed from different viewpoints, using RKHS methods. Of special interest is a relationship with an extended regression problem. Applications of the results to finite dimensional situations are also given.     

A new nonlinear blind source separation method with chaos indicators for decoupling diagnosis of hybrid failures: A marine propulsion gearbox case with a large speed variation

The normal operation of propulsion gearboxes ensures the ship safety. Chaos indicators could efficiently indicate the state change of the gearboxes. However, accurate detection of gearbox hybrid faults using Chaos indicators is a challenging task and the detection under speed variation conditions is attracting considerable attentions. Literature review suggests that the gearbox vibration is a kind of nonlinear mixture of variant vibration sources and the blind source separation (BSS) is reported to be a promising technique for fault vibration analysis, but very limited work has addressed the nonlinear BSS approach for hybrid faults decoupling diagnosis. Aiming to enhance the fault detection performance of Chaos indicators, this work presents a new nonlinear BSS algorithm for gearbox hybrid faults detection under a speed variation condition. This new method appropriately introduces the kernel spectral regression (KSR) framework into the morphological component analysis (MCA). The original vibration data are projected into the reproducing kernel Hilbert space (RKHS) where the instinct nonlinear structure in the original data can be linearized by KSR. Thus the MCA is able to deal with nonlinear BSS in the KSR space. Reliable hybrid faults decoupling is then achieved by this new nonlinear MCA (NMCA). Subsequently, by calculating the Chaos indicators of the decoupled fault components and comparing them with benchmarks, the hybrid faults can be precisely identified. Two specially designed case studies were implemented to evaluate the proposed NMCA-Chaos method on hybrid gear faults decoupling diagnosis. The performance of the NMCA-Chaos was compared with state of art techniques. The analysis results show high performance of the proposed method on hybrid faults detection in a marine propulsion gearbox with large speed variations.     

Radial kernels and their reproducing kernel Hilbert spaces

We describe how to use Schoenberg’s theorem for a radial kernel combined with existing bounds on the approximation error functions for Gaussian kernels to obtain a bound on the approximation error function for the radial kernel. The result is applied to the exponential kernel and Student’s kernel. To establish these results we develop a general theory regarding mixtures of kernels. We analyze the reproducing kernel Hilbert space (RKHS) of the mixture in terms of the RKHS’s of the mixture components and prove a type of Jensen inequality between the approximation error function for the mixture and the approximation error functions of the mixture components.