Multi-parameter Tikhonov regularization and model function approach to the damped Morozov principle for choosing regularization parameters

In this paper, we study the multi-parameter Tikhonov regularization method which adds multiple different penalties to exhibit multi-scale features of the solution. An optimal error bound of the regularization solution is obtained by a priori choice of multiple regularization parameters. Some theoretical results of the regularization solution about the dependence on regularization parameters are presented. Then, an a posteriori parameter choice, i.e., the damped Morozov discrepancy principle, is introduced to determine multiple regularization parameters. Five model functions, i.e., two hyperbolic model functions, a linear model function, an exponential model function and a logarithmic model function, are proposed to solve the damped Morozov discrepancy principle. Furthermore, four efficient model function algorithms are developed for finding reasonable multiple regularization parameters, and their convergence properties are also studied. Numerical results of several examples show that the damped discrepancy principle is competitive with the standard one, and the model function algorithms are efficient for choosing regularization parameters.     

Split Bregman iteration algorithm for total bounded variation regularization based image deblurring

Many existing algorithms taking the seminorm in BV(Ω) for regularization have achieved great success in image processing. However, this paper considers the total bounded variation regularization based approach to perform image deblurring. Based on this novel model, we introduce an extended split Bregman iteration to obtain the optimum solution quickly. We also provide the rigorous convergence analysis of the iterative algorithm here. Compared with the results of the ROF method, numerical simulations illustrate the more excellent reconstruction performance of the proposed algorithm.     

学习算法的稳定性与泛化(Ⅱ):有界差分稳定框架

根据有界差分条件,提出了学习算法的有界差分稳定框架.依据新框架,研究了机器学习阈值选择算法,再生核Hilbert空间中的正则化学习算法,Ranking学习算法和Bagging算法,证明了对应学习算法的有界差分稳定性.所获结果断言了这些算法均具有有界差分稳定性,从而为这些算法的应用奠定了理论基础.     

Adaptive iterative regularization schemes for two‐dimensional integral‐algebraic systems

The main idea of this paper is to utilize the adaptive iterative schemes based on regularization techniques for moderately ill‐posed problems that are obtained by a system of linear two‐dimensional Volterra integral equations with a singular matrix in the leading part. These problems may arise in the modeling of certain heat conduction processes as well as in the dynamic simulation packages such as compressible flow through a plant piping network. Owing to the ill‐posed nature of the first kind Volterra equation that appears in the system, we will focus on the two families of regularization algorithms, ie, the Landweber and Lavrentiev type methods, where we treat both the exact and perturbed data. Our aim is to work directly with the original Volterra equations without any kind of reduction. Two fast iterative algorithms with reasonable computational complexity are developed. Numerical experiments on a few test problems are used to illustrate the validity and efficiency of the proposed iterative methods in comparison with the classical regularization methods.     

Balancing principle in supervised learning for a general regularization scheme

We discuss the problem of parameter choice in learning algorithms generated by a general regularization scheme. Such a scheme covers well-known algorithms as regularized least squares and gradient descent learning. It is known that in contrast to classical deterministic regularization methods, the performance of regularized learning algorithms is influenced not only by the smoothness of a target function, but also by the capacity of a space, where regularization is performed. In the infinite dimensional case the latter one is usually measured in terms of the effective dimension. In the context of supervised learning both the smoothness and effective dimension are intrinsically unknown a priori. Therefore we are interested in a posteriori regularization parameter choice, and we propose a new form of the balancing principle. An advantage of this strategy over the known rules such as cross-validation based adaptation is that it does not require any data splitting and allows the use of all available labeled data in the construction of regularized approximants. We provide the analysis of the proposed rule and demonstrate its advantage in simulations.     

Learning rates for least square regressions with coefficient regularization

We analyze the learning rates for the least square regression with data dependent hypothesis spaces and coefficient regularization algorithms based on general kernels. Under a very mild regularity condition on the regression function, we obtain a bound for the approximation error by estimating the corresponding K-functional. Combining this estimate with the previous result of the sample error, we derive a dimensional free learning rate by the proper choice of the regularization parameter.     

The approximate regularization of index-2 differential-algebraic problems

The approximate regularization method is introduced for the solution of index-2 Hessenberg systems of differential-algebraic equations. This approach is based on using a Taylor-series-type method, which results in a singular perturbation ordinary differential systems. For linear problems, it is shown that this approach is stable. To compare with a partial regularization, the approximate regularization appears to be promising.     

Dynamical level set method for parameter identification of nonlinear parabolic distributed parameter systems

This article considers a dynamical level set method for the identification problem of the nonlinear parabolic distributed parameter system, which is based on the solvability and stability of the direct PDE (partial differential equation) in Sobolev space. The dynamical level set algorithms have been developed for ill-posed problems in Hilbert space. This method can be regarded as a asymptotical regularization method as long as a certain stopping rule is satisfied. Hence, the convergence analysis of the method is established similar to the proof of convergence of asymptotical regularization. The level set converges to a solution as the artificial time evolves to infinity. Furthermore, the proposed level set method is proved to be stable by using Lyapunov stability theorem, which is constructed in my previous article.Numerical tests are discussed to demonstrate the efficacy of the dynamical level set method, which consequently confirm the level set method to be a powerful tool for the identification of the parameter.     

Algorithms and Convergence Theorems for Mixed Equilibrium Problems in Hilbert Spaces

In this article, we study mixed equilibrium problems, and present algorithms and convergence theorems for the proposed algorithms, like proximal gradient method, Tikhonov regularization method, Mann’s type method, conjugate gradient method. As application, we study minimization problems.     

Shadow block iteration for solving linear systems obtained from wavelet transforms

Matrices resulting from wavelet transforms have a special “shadow” block structure, that is, their small upper left blocks contain their lower frequency information. Numerical solutions of linear systems with such matrices require special care. We propose shadow block iterative methods for solving linear systems of this type. Convergence analysis for these algorithms are presented. We apply the algorithms to three applications: linear systems arising in the classical regularization with a single parameter for the signal de-blurring problem, multilevel regularization with multiple parameters for the same problem and the Galerkin method of solving differential equations. We also demonstrate the efficiency of these algorithms by numerical examples in these applications.     

Prox-Regularization Methods for Generalized Fractional Programming

If a fractional program does not have a unique solution or the feasible set is unbounded, numerical difficulties can occur. By using a prox-regularization method that generates a sequence of auxiliary problems with unique solutions, these difficulties are avoided. Two regularization methods are introduced here. They are based on Dinkelbach-type algorithms for generalized fractional programming, but use a regularized parametric auxiliary problem. Convergence results and numerical examples are presented.     

解不适定问题的一种多尺度方法

1 前言 数学物理反问题是应用数学领域中成长和发展最快的领域之一.反问题大多是不适定的.对于不适定问题的解法已有不少的学者进行探索和研究,Tikhonov正则化方法是一种理论上最完备而在实践上行之有效的方法(参见[5,6,7,8,13]).     

Algorithms for the regularization of ill-conditioned least squares problems

Two regularization methods for ill-conditioned least squares problems are studied from the point of view of numerical efficiency. The regularization methods are formulated as quadratically constrained least squares problems, and it is shown that if they are transformed into a certain standard form, very efficient algorithms can be used for their solution. New algorithms are given, both for the transformation and for the regularization methods in standard form. A comparison to previous algorithms is made and it is shown that the overall efficiency (in terms of the number of arithmetic operations) of the new algorithms is better.     

Three-level schemes of the alternating triangular method

In this paper, the schemes of the alternating triangular method are set out in the class of splitting methods used for the approximate solution of Cauchy problems for evolutionary problems. These schemes are based on splitting the problem operator into two operators that are conjugate transposes of each other. Economical schemes for the numerical solution of boundary value problems for parabolic equations are designed on the basis of an explicit-implicit splitting of the problem operator. The alternating triangular method is also of interest for the construction of numerical algorithms that solve boundary value problems for systems of partial differential equations and vector systems. The conventional schemes of the alternating triangular method used for first-order evolutionary equations are two-level ones. The approximation properties of such splitting methods can be improved by transiting to three-level schemes. Their construction is based on a general principle for improving the properties of difference schemes, namely, on the regularization principle of A.A. Samarskii. The analysis conducted in this paper is based on the general stability (or correctness) theory of operator-difference schemes.     

Identification of distributed systems and the theory of the regularization

The identification problems, i.e., the problems of finding unknown parameters in distributed systems from the observations are very important in modern control theory. The solutions of these identification problems can be obtained by solving the equations of the first kind. However, the solutions are often unstable. In other words, they are not continuously dependent on the data. The regularization or Tihonov's regularization is known as one of the stabilizing algorithms to solve these non well-posed problems. In this paper is studied the regularization method for identification of distributed systems. Several approximation theorems are proved to solve the equations of the first kind. Then, identification problems are reduced to the minimization of quadratic cost functionals by virtue of these theorems. On the other hand, it is known that the statistical methods for identification such as the maximum likelihood lead to the minimization problems of certain quadratic functionals. Comparing these quadratic cost functionals, the relations between the regularization and the statistical methods are discussed. Further, numerical examples are given to show the effectiveness of this method.     

Regularized Adaptation of Fuzzy Inference Systems. Modelling the Opinion of a Medical Expert about Physical Fitness: An Application

This study presents a new approach to adaptation of Sugeno type fuzzy inference systems using regularization, since regularization improves the robustness of standard parameter estimation algorithms leading to stable fuzzy approximation. The proposed method can be used for modelling, identification and control of physical processes. A recursive method for on-line identification of fuzzy parameters employing Tikhonov regularization is suggested. The power of approach was shown by applying it to the modelling, identification, and adaptive control problems of dynamic processes. The proposed approach was used for modelling of human-decisions (experience) with a fuzzy inference system and for the fuzzy approximation of physical fitness with real world medical data.     

Modular solvers for image restoration problems using the discrepancy principle

Many problems in image restoration can be formulated as either an unconstrained non‐linear minimization problem, usually with a Tikhonov‐like regularization, where the regularization parameter has to be determined; or as a fully constrained problem, where an estimate of the noise level, either the variance or the signal‐to‐noise ratio, is available. The formulations are mathematically equivalent. However, in practice, it is much easier to develop algorithms for the unconstrained problem, and not always obvious how to adapt such methods to solve the corresponding constrained problem. In this paper, we present a new method which can make use of any existing convergent method for the unconstrained problem to solve the constrained one. The new method is based on a Newton iteration applied to an extended system of non‐linear equations, which couples the constraint and the regularized problem, but it does not require knowledge of the Jacobian of the irregularity functional. The existing solver is only used as a black box solver, which for a fixed regularization parameter returns an improved solution to the unconstrained minimization problem given an initial guess. The new modular solver enables us to easily solve the constrained image restoration problem; the solver automatically identifies the regularization parameter, during the iterative solution process. We present some numerical results. The results indicate that even in the worst case the constrained solver requires only about twice as much work as the unconstrained one, and in some instances the constrained solver can be even faster. Copyright © 2002 John Wiley & Sons, Ltd.     

Addressing the greediness phenomenon in Nonlinear Programming by means of Proximal Augmented Lagrangians

When one solves Nonlinear Programming problems by means of algorithms that use merit criteria combining the objective function and penalty feasibility terms, a phenomenon called greediness may occur. Unconstrained minimizers attract the iterates at early stages of the calculations and, so, the penalty parameter needs to grow excessively, in such a way that ill-conditioning harms the overall convergence. In this paper a regularization approach is suggested to overcome this difficulty. An Augmented Lagrangian method is defined with the addition of a regularization term that inhibits the possibility that the iterates go far from a reference point. Convergence proofs and numerical examples are given.     

Necessary and sufficient conditions for convergence of regularization methods for solving linear operator equations of the first kind

In this paper, we give necessary and sufficient conditions for weak and strong convergence of general regularization methods for the solution of linear ill-posed problems in Hilbert space. As special cases we obtain convergence criteria for Tychonoff regularization, for a regularization method based on Showalter's integral formula, and for regularization by truncated iteration     

An algorithm based on the regularization and integral mean value methods for the Fredholm integral equations of the first kind

In this paper, an algorithm based on the regularization and integral mean value methods, to handle the ill-posed multi-dimensional Fredholm equations, is introduced. The application of this algorithm is based on the transforming the first kind equation to a second kind equation by the regularization method. Then, by converting the first kind to a second kind, the integral mean value method is employed to handle the resulting Fredholm integral equations of the second kind. The efficiency of the approach will be shown by applying the procedure on some examples.