A generalized conditional gradient method and its connection to an iterative shrinkage method

This article combines techniques from two fields of applied mathematics: optimization theory and inverse problems. We investigate a generalized conditional gradient method and its connection to an iterative shrinkage method, which has been recently proposed for solving inverse problems. The iterative shrinkage method aims at the solution of non-quadratic minimization problems where the solution is expected to have a sparse representation in a known basis. We show that it can be interpreted as a generalized conditional gradient method. We prove the convergence of this generalized method for general class of functionals, which includes non-convex functionals. This also gives a deeper understanding of the iterative shrinkage method.     

A new proof of some polynomial inequalities related to pseudo-splines

Pseudo-splines of type I were introduced in [I. Daubechies, B. Han, A. Ron, Z. Shen, Framelets: MRA-based constructions of wavelet frames, Appl. Comput. Harmon. Anal. 14 (2003) 1–46] and [Selenick, Smooth wavelet tight frames with zero moments, Appl. Comput. Harmon. Anal. 10 (2000) 163–181] and type II were introduced in [B. Dong, Z. Shen, Pseudo-splines, wavelets and framelets, Appl. Comput. Harmon. Anal. 22 (2007) 78–104]. Both types of pseudo-splines provide a rich family of refinable functions with B-splines, interpolatory refinable functions and refinable functions with orthonormal shifts as special examples. In [B. Dong, Z. Shen, Pseudo-splines, wavelets and framelets, Appl. Comput. Harmon. Anal. 22 (2007) 78–104], Dong and Shen gave a regularity analysis of pseudo-splines of both types. The key to regularity analysis is Proposition 3.2 in [B. Dong, Z. Shen, Pseudo-splines, wavelets and framelets, Appl. Comput. Harmon. Anal. 22 (2007) 78–104], which also appeared in [A. Cohen, J.P. Conze, Régularité des bases d'ondelettes et mesures ergodiques, Rev. Mat. Iberoamericana 8 (1992) 351–365] and [I. Daubechies, Ten Lectures on Wavelets, CBMS-NSF Series in Applied Mathematics, SIAM, Philadelphia, 1992] for the case l=N−1. In this note, we will give a new insight into this proposition.     

Sparse approximate inverse preconditioning for dense linear systems arising in computational electromagnetics

We investigate the use of sparse approximate inverse preconditioners for the iterative solution of linear systems with dense complex coefficient matrices arising in industrial electromagnetic problems. An approximate inverse is computed via a Frobenius norm approach with a prescribed nonzero pattern. Some strategies for determining the nonzero pattern of an approximate inverse are described. The results of numerical experiments suggest that sparse approximate inverse preconditioning is a viable approach for the solution of large-scale dense linear systems on parallel computers. This revised version was published online in August 2006 with corrections to the Cover Date.     

New choices of preconditioning matrices for generalized inexact parameterized iterative methods

For large sparse saddle point problems, Chen and Jiang recently studied a class of generalized inexact parameterized iterative methods (see [F. Chen, Y.-L. Jiang, A generalization of the inexact parameterized Uzawa methods for saddle point problems, Appl. Math. Comput. 206 (2008) 765-771]). In this paper, the methods are modified and some choices of preconditioning matrices are given. These preconditioning matrices have advantages in solving large sparse linear system. Numerical experiments of a model Stokes problem are presented.     

SPAI Preconditioners for HPC Applications

Iterative methods to solve systems of linear equations Ax = b usually require preconditioners M to speed convergence. But the calculation of many preconditioners can be notoriously sequential. The sparse approximate inverse preconditioner (SPAI) has particular potential for high performance computing [1]. We have ported the SPAI algorithm to graphical processing units (GPUs) within NVIDIA's CUSP library [2] for sparse linear algebra. GPUs perform well on dense linear algebra problems where data resides for long periods on the device. Since the underlying minimization problems are independent, they are mapped to separate thread-blocks, and an optimized QR algorithm implemented using NVIDIA's CUBLAS library is employed on each. Traditionally the challenge has been to determine a sparsity pattern Sp( M ) of the preconditioner dynamically [3], which reduces the condition number of MA to a degree where a preconditioned iterative solver such as GMRES becomes computationally viable. Due to the extremely high performance of the GPU, it is possible to consider initial sparsity patterns much denser than have been previously considered. We therefore consider a fixed sparsity patterns to simplify the GPU implementation. We evaluate the performance of the resulting preconditioner on a standard set of sparse matrices, and compare SPAI to other preconditioners. (© 2012 Wiley-VCH Verlag GmbH & Co. KGaA, Weinheim)     

Highly sparse representations from dictionaries are unique and independent of the sparseness measure

The purpose of this paper is to study sparse representations of signals from a general dictionary in a Banach space. For so-called localized frames in Hilbert spaces, the canonical frame coefficients are shown to provide a near sparsest expansion for several sparseness measures. However, for frames which are not localized, this no longer holds true and sparse representations may depend strongly on the choice of the sparseness measure. A large class of admissible sparseness measures is introduced, and we give sufficient conditions for having a unique sparse representation of a signal from the dictionary w.r.t. such a sparseness measure. Moreover, we give sufficient conditions on a signal such that the simple solution of a linear programming problem simultaneously solves all the nonconvex (and generally hard combinatorial) problems of sparsest representation of the signal w.r.t. arbitrary admissible sparseness measures.     

An optimal control problem in image processing

As a starting point, we present a control problem in mammographic image processing which leads to non-standard penalty terms and involves a degenerate parabolic PDE which has to be controlled in the coefficients. We then discuss the classical conditional gradient method from constrained optimization and propose a generalization for non-convex functionals which covers the conditional gradient method as well as the recently proposed iterative shrinkage method of Daubechies, Defrise and De Mol for the solution of linear inverse problems with sparsity promoting penalty terms. We prove that this new algorithm converges. This also gives a deeper understanding of the iterative shrinkage method. Further, we show an application to the above-mentioned control problem in image processing. (© 2006 WILEY-VCH Verlag GmbH & Co. KGaA, Weinheim)     

Implicit complementarity problems on isotone projection cones

Isac and Németh [G. Isac and A. B. Németh, Projection methods, isotone projection cones and the complementarity problem, J. Math. Anal. Appl. 153 (1990), pp. 258–275] proved that solving a coincidence point equation (fixed point problem) in turn solves the corresponding implicit complementarity problem (nonlinear complementarity problem) and they exploited the isotonicity of the metric projection onto isotone projection cones to solve implicit complementarity problems (nonlinear complementarity problems) defined by these cones. In this article an iterative algorithm is studied in connection with an implicit complementarity problem. It is proved that if the sequence generated through the defined algorithm is convergent, then its limit is a solution of the coincidence point equation and thus solves the implicit complementarity problem. Sufficient conditions are given for this sequence to be convergent for implicit complementarity problems defined by isotone projection cones, extending the results of Németh [S.Z. Németh, Iterative methods for nonlinear complementarity problems on isotone projection cones, J. Math. Anal. Appl. 350 (2009), pp. 340–370]. Some existing concepts from the latter paper are extended to solve the problem of finding nonzero solutions of the implicit complementarity problem.     

Coordinate and subspace optimization methods for linear least squares with non-quadratic regularization

This work addresses the problem of regularized linear least squares (RLS) with non-quadratic separable regularization. Despite being frequently deployed in many applications, the RLS problem is often hard to solve using standard iterative methods. In a recent work [M. Elad, Why simple shrinkage is still relevant for redundant representations? IEEE Trans. Inform. Theory 52 (12) (2006) 5559–5569], a new iterative method called parallel coordinate descent (PCD) was devised. We provide herein a convergence analysis of the PCD algorithm, and also introduce a form of the regularization function, which permits analytical solution to the coordinate optimization. Several other recent works [I. Daubechies, M. Defrise, C. De-Mol, An iterative thresholding algorithm for linear inverse problems with a sparsity constraint, Comm. Pure Appl. Math. LVII (2004) 1413–1457; M.A. Figueiredo, R.D. Nowak, An EM algorithm for wavelet-based image restoration, IEEE Trans. Image Process. 12 (8) (2003) 906–916; M.A. Figueiredo, R.D. Nowak, A bound optimization approach to wavelet-based image deconvolution, in: IEEE International Conference on Image Processing, 2005], which considered the deblurring problem in a Bayesian methodology, also obtained element-wise optimization algorithms. We show that the last three methods are essentially equivalent, and the unified method is termed separable surrogate functionals (SSF). We also provide a convergence analysis for SSF. To further accelerate PCD and SSF, we merge them into a recently developed sequential subspace optimization technique (SESOP), with almost no additional complexity. A thorough numerical comparison of the denoising application is presented, using the basis pursuit denoising (BPDN) objective function, which leads all of the above algorithms to an iterated shrinkage format. Both with synthetic data and with real images, the advantage of the combined PCD-SESOP method is demonstrated.     

Stability theorems for Fourier frames and wavelet Riesz bases

In this paper we present two applications of a Stability Theorem of Hilbert frames to nonharmonic Fourier series and wavelet Riesz basis. The first result is an enhancement of the Paley-Wiener type constant for nonharmonic series given by Duffin and Schaefer in [6] and used recently in some applications (see [3]). In the case of an orthonormal basis, our estimate reduces to Kadec’ optimal 1/4 result. The second application proves that a phenomenon discovered by Daubechies and Tchamitchian [4] for the orthonormal Meyer wavelet basis (stability of the Riesz basis property under small changes of the translation parameter) actually holds for a large class of wavelet Riesz bases.     

A normalized sparse linear equations solver

Normalized factorization procedures for the solution of large sparse linear finite element systems have been recently introduced in [3]. In these procedures the large sparse symmetric coefficient matrix of irregular structure is factorized exactly to yield a normalized direct solution method. Additionally, approximate factorization procedures yield implicit iterative methods for the finite difference or finite element solution. The numerical implementation of these algorithms is presented here and FORTRAN subroutines for the efficient solution of the resulting large sparse symmetric linear systems of algebraic equations are given.     

Multilevel methods for mixed finite elements in three dimensions

In this paper we consider second order scalar elliptic boundary value problems posed over three–dimensional domains and their discretization by means of mixed Raviart–Thomas finite elements [18]. This leads to saddle point problems featuring a discrete flux vector field as additional unknown. Following Ewing and Wang [26], the proposed solution procedure is based on splitting the flux into divergence free components and a remainder. It leads to a variational problem involving solenoidal Raviart–Thomas vector fields. A fast iterative solution method for this problem is presented. It exploits the representation of divergence free vector fields as s of the –conforming finite element functions introduced by Nédélec [43]. We show that a nodal multilevel splitting of these finite element spaces gives rise to an optimal preconditioner for the solenoidal variational problem: Duality techniques in quotient spaces and modern algebraic multigrid theory [50, 10, 31] are the main tools for the proof. Received November 4, 1996 / Revised version received February 2, 1998     

Stopping criteria for iterations in finite element methods

Summary. This work extends the results of Arioli [1], [2] on stopping criteria for iterative solution methods for linear finite element problems to the case of nonsymmetric positive-definite problems. We show that the residual measured in the norm induced by the symmetric part of the inverse of the system matrix is relevant to convergence in a finite element context. We then use Krylov solvers to provide alternative ways of calculating or estimating this quantity and present numerical experiments which validate our criteria.Mathematics Subject Classification (2000): 65N30, 65F10, 65F35     

Estimation of intrinsic processes affected by additive fractal noise

Fractal Gaussian models have been widely used to represent the singular behavior of phenomena arising in different applied fields; for example, fractional Brownian motion and fractional Gaussian noise are considered as monofractal models in subsurface hydrology and geophysical studies Mandelbrot [The Fractal Geometry of Nature, Freeman Press, San Francisco, 1982 [13]]. In this paper, we address the problem of least-squares linear estimation of an intrinsic fractal input random field from the observation of an output random field affected by fractal noise (see Angulo et al. [Estimation and filtering of fractional generalised random fields, J. Austral. Math. Soc. A 69 (2000) 1-26 [2]], Ruiz-Medina et al. [Fractional generalized random fields on bounded domains, Stochastic Anal. Appl. 21 (2003a) 465-492], Ruiz-Medina et al. [Fractional-order regularization and wavelet approximation to the inverse estimation problem for random fields, J. Multivariate Anal. 85 (2003b) 192-216]. Conditions on the fractality order of the additive noise are studied to obtain a bounded inversion of the associated Wiener-Hopf equation. A stable solution is then obtained in terms of orthogonal bases of the reproducing kernel Hilbert spaces associated with the random fields involved. Such bases are constructed from orthonormal wavelet bases (see Angulo and Ruiz-Medina [Multiresolution approximation to the stochastic inverse problem, Adv. in Appl. Probab. 31 (1999) 1039-1057], Angulo et al. [Wavelet-based orthogonal expansions of fractional generalized random fields on bounded domains, Theoret. Probab. Math. Stat. (2004), in press]). A simulation study is carried out to illustrate the influence of the fractality orders of the output random field and the fractal additive noise on the stability of the solution derived.     

Pseudo-splines,wavelets and framelets

The first type of pseudo-splines were introduced in [I. Daubechies, B. Han, A. Ron, Z. Shen, Framelets: MRA-based constructions of wavelet frames, Appl. Comput. Harmon. Anal. 14 (1) (2003) 1–46; I. Selesnick, Smooth wavelet tight frames with zero moments, Appl. Comput. Harmon. Anal. 10 (2) (2001) 163–181] to construct tight framelets with desired approximation orders via the unitary extension principle of [A. Ron, Z. Shen, Affine systems in $L_2({\mathbb{R}}^d)$: The analysis of the analysis operator, J. Funct. Anal. 148 (2) (1997) 408–447]. In the spirit of the first type of pseudo-splines, we introduce here a new type (the second type) of pseudo-splines to construct symmetric or antisymmetric tight framelets with desired approximation orders. Pseudo-splines provide a rich family of refinable functions. B-splines are one of the special classes of pseudo-splines; orthogonal refinable functions (whose shifts form an orthonormal system given in [I. Daubechies, Orthonormal bases of compactly supported wavelets, Comm. Pure Appl. Math. 41 (1988) 909–996]) are another class of pseudo-splines; and so are the interpolatory refinable functions (which are the Lagrange interpolatory functions at $\mathbb{Z}$ and were first discussed in [S. Dubuc, Interpolation through an iterative scheme, J. Math. Anal. Appl. 114 (1986) 185–204]). The other pseudo-splines with various orders fill in the gaps between the B-splines and orthogonal refinable functions for the first type and between B-splines and interpolatory refinable functions for the second type. This gives a wide range of choices of refinable functions that meets various demands for balancing the approximation power, the length of the support, and the regularity in applications. This paper will give a regularity analysis of pseudo-splines of the both types and provide various constructions of wavelets and framelets. It is easy to see that the regularity of the first type of pseudo-splines is between B-spline and orthogonal refinable function of the same order. However, there is no precise regularity estimate for pseudo-splines in general. In this paper, an optimal estimate of the decay of the Fourier transform of the pseudo-splines is given. The regularity of pseudo-splines can then be deduced and hence, the regularity of the corresponding wavelets and framelets. The asymptotical regularity analysis, as the order of the pseudo-splines goes to infinity, is also provided. Furthermore, we show that in all tight frame systems constructed from pseudo-splines by methods provided both in [I. Daubechies, B. Han, A. Ron, Z. Shen, Framelets: MRA-based constructions of wavelet frames, Appl. Comput. Harmon. Anal. 14 (1) (2003) 1–46] and this paper, there is one tight framelet from the generating set of the tight frame system whose dilations and shifts already form a Riesz basis for $L_2(\mathbb{R})$.     

关于带W权Drazin逆的表示定理及迭代方法

To study singular linear system, Cline and Greville[8] proposed the concept of W-weighted Drazin inverse for the rectangular matrices,where the properties were also discussed. The computation for the W-weighted Drazin inverse is of much interest, which is mainly divided into two kinds of methods: direct method[2,4,6] and iterative method[3,5,7,9,12,13]. In this paper, we study the iterative method and successive matrix squaring(SMS) method for the W-weighted Drazin inverse and generalize the main results in [12,13].     

Construction of biorthogonal wavelets from pseudo-splines

Pseudo-splines constitute a new class of refinable functions with B-splines, interpolatory refinable functions and refinable functions with orthonormal shifts as special examples. Pseudo-splines were first introduced by Daubechies, Han, Ron and Shen in [Framelets: MRA-based constructions of wavelet frames, Appl. Comput. Harmon. Anal. 14(1) (2003), 1–46] and Selenick in [Smooth wavelet tight frames with zero moments, Appl. Comput. Harmon. Anal. 10(2) (2001) 163–181], and their properties were extensively studied by Dong and Shen in [Pseudo-splines, wavelets and framelets, 2004, preprint]. It was further shown by Dong and Shen in [Linear independence of pseudo-splines, Proc. Amer. Math. Soc., to appear] that the shifts of an arbitrarily given pseudo-spline are linearly independent. This implies the existence of biorthogonal dual refinable functions (of pseudo-splines) with an arbitrarily prescribed regularity. However, except for B-splines, there is no explicit construction of biorthogonal dual refinable functions with any given regularity. This paper focuses on an implementable scheme to derive a dual refinable function with a prescribed regularity. This automatically gives a construction of smooth biorthogonal Riesz wavelets with one of them being a pseudo-spline. As an example, an explicit formula of biorthogonal dual refinable functions of the interpolatory refinable function is given.     

An iterative thresholding algorithm for linear inverse problems with a sparsity constraint

We consider linear inverse problems where the solution is assumed to have a sparse expansion on an arbitrary preassigned orthonormal basis. We prove that replacing the usual quadratic regularizing penalties by weighted ??^p‐penalties on the coefficients of such expansions, with 1 ≤ p ≤ 2, still regularizes the problem. Use of such ??^p‐penalized problems with p < 2 is often advocated when one expects the underlying ideal noiseless solution to have a sparse expansion with respect to the basis under consideration. To compute the corresponding regularized solutions, we analyze an iterative algorithm that amounts to a Landweber iteration with thresholding (or nonlinear shrinkage) applied at each iteration step. We prove that this algorithm converges in norm. © 2004 Wiley Periodicals, Inc.     

不定最小二乘问题的改进的不完全双曲Gram-Schmidt预处理算法

应用改进的不完全双曲Gram-Schmidt(IHMGS)方法预处理不定最小二乘问题的共轭梯度法(CGILS)、正交分解法(ILSQR)与广义的最小剩余法(GMRES)等迭代算法来求解大型稀疏的不定最小二乘问题.数值实验表明,IHMGS预处理方法可有效提高相应算法的迭代速度,且当矩阵的条件数比较大时,效果更加显著.     

Algorithms for range restricted iterative methods for linear discrete ill-posed problems

Range restricted iterative methods based on the Arnoldi process are attractive for the solution of large nonsymmetric linear discrete ill-posed problems with error-contaminated data (right-hand side). Several derivations of this type of iterative methods are compared in Neuman et al. (Linear Algebra Appl. in press). We describe MATLAB codes for the best of these implementations. MATLAB codes for range restricted iterative methods for symmetric linear discrete ill-posed problems are also presented.