Support vector machines regression with l1-regularizer

The classical support vector machines regression (SVMR) is known as a regularized learning algorithm in reproducing kernel Hilbert spaces (RKHS) with a $\epsilon$-insensitive loss function and an RKHS norm regularizer. In this paper, we study a new SVMR algorithm where the regularization term is proportional to $l^1$-norm of the coefficients in the kernel ensembles. We provide an error analysis of this algorithm, an explicit learning rate is then derived under some assumptions.     

Least-squares regularized regression with dependent samples and q-penalty

Least-squares regularized learning algorithms for regression were well-studied in the literature when the sampling process is independent and the regularization term is the square of the norm in a reproducing kernel Hilbert space (RKHS). Some analysis has also been done for dependent sampling processes or regularizers being the qth power of the function norm (q-penalty) with 0?q?≤?2. The purpose of this article is to conduct error analysis of the least-squares regularized regression algorithm when the sampling sequence is weakly dependent satisfying an exponentially decaying α-mixing condition and when the regularizer takes the q-penalty with 0?q?≤?2. We use a covering number argument and derive learning rates in terms of the α-mixing decay, an approximation condition and the capacity of balls of the RKHS.     

Learning Theory Estimates via Integral Operators and Their Approximations

The regression problem in learning theory is investigated with least square Tikhonov regularization schemes in reproducing kernel Hilbert spaces (RKHS). We follow our previous work and apply the sampling operator to the error analysis in both the RKHS norm and the L² norm. The tool for estimating the sample error is a Bennet inequality for random variables with values in Hilbert spaces. By taking the Hilbert space to be the one consisting of Hilbert-Schmidt operators in the RKHS, we improve the error bounds in the L² metric, motivated by an idea of Caponnetto and de Vito. The error bounds we derive in the RKHS norm, together with a Tsybakov function we discuss here, yield interesting applications to the error analysis of the (binary) classification problem, since the RKHS metric controls the one for the uniform convergence.     

The convergence rates of Shannon sampling learning algorithms

In the present paper,we provide an error bound for the learning rates of the regularized Shannon sampling learning scheme when the hypothesis space is a reproducing kernel Hilbert space(RKHS) derived by a Mercer kernel and a determined net.We show that if the sample is taken according to the determined set,then,the sample error can be bounded by the Mercer matrix with respect to the samples and the determined net.The regularization error may be bounded by the approximation order of the reproducing kernel Hilbert space interpolation operator.The paper is an investigation on a remark provided by Smale and Zhou.     

学习算法的稳定性与泛化(Ⅱ):有界差分稳定框架

根据有界差分条件,提出了学习算法的有界差分稳定框架.依据新框架,研究了机器学习阈值选择算法,再生核Hilbert空间中的正则化学习算法,Ranking学习算法和Bagging算法,证明了对应学习算法的有界差分稳定性.所获结果断言了这些算法均具有有界差分稳定性,从而为这些算法的应用奠定了理论基础.     

Primal and dual alternating direction algorithms for ℓ 1-ℓ 1-norm minimization problems in compressive sensing

In this paper, we propose, analyze and test primal and dual versions of the alternating direction algorithm for the sparse signal reconstruction from its major noise contained observation data. The algorithm minimizes a convex non-smooth function consisting of the sum of ? ₁-norm regularization term and ? ₁-norm data fidelity term. We minimize the corresponding augmented Lagrangian function alternatively from either primal or dual forms. Both of the resulting subproblems admit explicit solutions either by using a one-dimensional shrinkage or by an efficient Euclidean projection. The algorithm is easily implementable and it requires only two matrix-vector multiplications per-iteration. The global convergence of the proposed algorithm is established under some technical conditions. The extensions to the non-negative signal recovery problem and the weighted regularization minimization problem are also discussed and tested. Numerical results illustrate that the proposed algorithm performs better than the state-of-the-art algorithm YALL1.     

The Multichannel Deconvolution Problem: A Discrete Analysis

We study the multichannel deconvolution problem (MDP) in a discrete setting by developing the theory for converting the method used in the continuous setting in [36]. We give a method for solving the MDP when the convolvers are characteristic functions, derive the explicit form of the linear system, and obtain an upper bound on the condition number of the system in a particular case. We compare the Schiske reconstruction [28] to our solution in the discrete setting, and give an explicit formula for the corresponding error. We then give the algorithm for solving the general MDP and discuss in detail the local reconstruction aspects of the problem. Finally, we describe a method for improving the reconstruction by regularization and give some explicit estimates on error bounds in the presence of noise.     

Regularization of the Solution of the Cauchy Problem: The Quasi-Reversibility Method

Some regularization algorithm is proposed related to the problem of continuation of the wave field from the planar boundary into the half-plane. We consider a hyperbolic equation whose main part coincideswith the wave operator, whereas the lowest term contains a coefficient depending on the two spatial variables. The regularization algorithm is based on the quasi-reversibility method proposed by Lattes and Lions. We consider the solution of an auxiliary regularizing equation with a small parameter; the existence, the uniqueness, and the stability of the solution in the Cauchy data are proved. The convergence is substantiated of this solution to the exact solution as the small parameter vanishes. A solution of an auxiliary problem is constructed with the Cauchy data having some error. It is proved that, for a suitable choice of a small parameter, the approximate solution converges to the exact solution.     

求解带线性约束的凸优化的一类自适应不定线性化增广拉格朗日方法

增广拉格朗日方法是求解带线性约束的凸优化问题的有效算法.线性化增广拉格朗日方法通过线性化增广拉格朗日函数的二次罚项并加上一个临近正则项,使得子问题容易求解,其中正则项系数的恰当选取对算法的收敛性和收敛速度至关重要.较大的系数可保证算法收敛性,但容易导致小步长.较小的系数允许迭代步长增大,但容易导致算法不收敛.本文考虑求解带线性等式或不等式约束的凸优化问题.我们利用自适应技术设计了一类不定线性化增广拉格朗日方法,即利用当前迭代点的信息自适应选取合适的正则项系数,在保证收敛性的前提下尽量使得子问题步长选择范围更大,从而提高算法收敛速度.我们从理论上证明了算法的全局收敛性,并利用数值实验说明了算法的有效性.     

High-order total variation-based Poissonian image deconvolution with spatially adapted regularization parameter

Images captured by image acquisition systems using photon-counting devices such as astronomical imaging, positron emission tomography and confocal microscopy imaging, are often contaminated by Poisson noise. Total variation (TV) regularization, which is a classic regularization technique in image restoration, is well-known for recovering sharp edges of an image. Since the regularization parameter is important for a good recovery, Chen and Cheng (2012) proposed an effective TV-based Poissonian image deblurring model with a spatially adapted regularization parameter. However, it has drawbacks since the TV regularization produces staircase artifacts. In this paper, in order to remedy the shortcoming of TV of their model, we introduce an extra high-order total variation (HTV) regularization term. Furthermore, to balance the trade-off between edges and the smooth regions in the images, we also incorporate a weighting parameter to discriminate the TV and the HTV penalty. The proposed model is solved by an iterative algorithm under the framework of the well-known alternating direction method of multipliers. Our numerical results demonstrate the effectiveness and efficiency of the proposed method, in terms of signal-to-noise ratio (SNR) and relative error (RelRrr).     

Solving the Inverse Generalized Problem of Vertical Seismic Profiling

The dynamic inverse seismics problem is considered in a generalized setting. We investigate whether the wave propagation problem in a vertically nonhomogeneous medium is well-posed. We show that the regular part of the solution is an L ₂ function and the inverse problem, i.e., the determination of the reflection coefficient, is thus reducible to minimizing the error functional. The gradient of the functional is obtained in explicit form from the conjugate problem, and approximate formulas for its evaluation are derived. A regularization algorithm for the solution of the inverse problem is considered; simulation results using various excitation sources are reported.     

A generalized elastic net regularization with smoothed $$\ell _{q}$$ penalty for sparse vector recovery

In this paper, we propose an iterative algorithm for solving the generalized elastic net regularization problem with smoothed \(\ell _{q} (0<q \le 1)\) penalty for recovering sparse vectors. We prove the convergence result of the algorithm based on the algebraic method. Under certain conditions, we show that the iterative solutions converge to a local minimizer of the generalized elastic net regularization problem and we also present an error bound. Theoretical analysis and numerical results show that the proposed algorithm is promising.     

Numerical solutions of a two-phase membrane problem

In this paper different numerical methods for a two-phase free boundary problem are discussed. In the first method a novel iterative scheme for the two-phase membrane is considered. We study the regularization method and give an a posteriori error estimate which is needed for the implementation of the regularization method. Moreover, an efficient algorithm based on the finite element method is presented. It is shown that the sequence constructed by the algorithm is monotone and converges to the solution of the given free boundary problem. These methods can be applied for the one-phase obstacle problem as well.     

The Kirsch-Kress method for inverse scattering by infinite locally rough interfaces

This paper is concerned with the inverse problem of reconstructing an infinite, locally rough interface from the scattered field measured on line segments above and below the interface in two dimensions. We extend the Kirsch-Kress method originally developed for inverse obstacle scattering problems to the above inverse transmission problem with unbounded interfaces. To this end, we reformulate our inverse problem as a nonlinear optimization problem with a Tikhonov regularization term. We prove the convergence of the optimization problem when the regularization parameter tends to zero. Finally, numerical experiments are carried out to show the validity of the inversion algorithm.     

Learning rates of regularized regression on the unit sphere

This paper addresses the learning algorithm on the unit sphere. The main purpose is to present an error analysis for regression generated by regularized least square algorithms with spherical harmonics kernel. The excess error can be estimated by the sum of sample errors and regularization errors. Our study shows that by introducing a suitable spherical harmonics kernel, the regularization parameter can decrease arbitrarily fast with the sample size.     

Regularized Kernel-Based Reconstruction in Generalized Besov Spaces

We present a theoretical framework for reproducing kernel-based reconstruction methods in certain generalized Besov spaces based on positive, essentially self-adjoint operators. An explicit representation of the reproducing kernel is given in terms of an infinite series. We provide stability estimates for the kernel, including inverse Bernstein-type estimates for kernel-based trial spaces, and we give condition estimates for the interpolation matrix. Then, a deterministic error analysis for regularized reconstruction schemes is presented by means of sampling inequalities. In particular, we provide error bounds for a regularized reconstruction scheme based on a numerically feasible approximation of the kernel. This allows us to derive explicit coupling relations between the series truncation, the regularization parameters and the data set.     

Sobolev-type approximation rates for divergence-free and curl-free RBF interpolants

Recently, error estimates have been made available for divergence-free radial basis function (RBF) interpolants. However, these results are only valid for functions within the associated reproducing kernel Hilbert space (RKHS) of the matrix-valued RBF. Functions within the associated RKHS, also known as the ``native space' of the RBF, can be characterized as vector fields having a specific smoothness, making the native space quite small. In this paper we develop Sobolev-type error estimates when the target function is less smooth than functions in the native space.

     

A “Nonconvex+Nonconvex” approach for image restoration with impulse noise removal

In this paper, we propose a new method for image restoration problems, which are degraded by impulsive noise, with nonconvex data fitting term and nonconvex regularizer.The proposed method possesses the advantages of nonconvex data fitting and nonconvex regularizer simultaneously, namely, robustness for impulsive noise and efficiency for restoring neat edge images.Further, we propose an efficient algorithm to solve the “Nonconvex+Nonconvex” structure problem via using the alternating direction minimization, and prove that the algorithm is globally convergent when the regularization parameter is known. However, the regularization parameter is unavailable in general. Thereby, we combine the algorithm with the continuation technique and modified Morozov’s discrepancy principle to get an improved algorithm in which a suitable regularization parameter can be chosen automatically. The experiments reveal the superior performances of the proposed algorithm in comparison with some existing methods.     

A conjugate subgradient algorithm with adaptive preconditioning for the least absolute shrinkage and selection operator minimization

This paper describes a new efficient conjugate subgradient algorithm which minimizes a convex function containing a least squares fidelity term and an absolute value regularization term. This method is successfully applied to the inversion of ill-conditioned linear problems, in particular for computed tomography with the dictionary learning method. A comparison with other state-of-art methods shows a significant reduction of the number of iterations, which makes this algorithm appealing for practical use.     

Simultaneous identification of diffusion coefficient,spacewise dependent source and initial value for one‐dimensional heat equation

This paper deals with an inverse problem of determining the diffusion coefficient, spacewise dependent source term, and the initial value simultaneously for a one‐dimensional heat equation based on the boundary control, boundary measurement, and temperature distribution at a given single instant in time. By a Dirichlet series representation for the boundary observation, the identification of the diffusion coefficient and initial value can be transformed into a spectral estimation problem of an exponential series with measurement error, which is solved by the matrix pencil method. For the identification of the source term, a finite difference approximation method in conjunction with the truncated singular value decomposition is adopted, where the regularization parameter is determined by the generalized cross‐validation criterion. Numerical simulations are performed to verify the result of the proposed algorithm. Copyright © 2016 John Wiley & Sons, Ltd.