A new regularized stochastic approximation framework for stochastic inverse problems

We study the nonlinear inverse problem of estimating stochastic parameters in the fourth-order partial differential equation with random data. The primary focus is on developing a novel stochastic approximation framework for inverse problems consisting of three key components. As a first step, we reformulate the inverse problem into a stochastic convex optimization problem. The second step includes developing a new regularized stochastic extragradient framework for a nonlinear variational inequality, which subsumes the optimality conditions for the optimization formulation of the inverse problem. The third step involves modeling random variables by a Karhunen–Loève type finite-dimensional noise representation, allowing the direct and the inverse problems to be conveniently discretized. We show that the regularized extragradient methods are strongly convergent in a Hilbert space setting, and we also provide several auxiliary results for the inverse problem, including Lipschitz continuity and a derivative characterization of the solution map. We provide the outcome of computational experiments to estimate stochastic and deterministic parameters. The numerical results demonstrate the feasibility and effectiveness of the developed framework and validate stochastic approximation as an effective method for stochastic inverse problems.     

An iterative thresholding algorithm for linear inverse problems with a sparsity constraint

We consider linear inverse problems where the solution is assumed to have a sparse expansion on an arbitrary preassigned orthonormal basis. We prove that replacing the usual quadratic regularizing penalties by weighted ??^p‐penalties on the coefficients of such expansions, with 1 ≤ p ≤ 2, still regularizes the problem. Use of such ??^p‐penalized problems with p < 2 is often advocated when one expects the underlying ideal noiseless solution to have a sparse expansion with respect to the basis under consideration. To compute the corresponding regularized solutions, we analyze an iterative algorithm that amounts to a Landweber iteration with thresholding (or nonlinear shrinkage) applied at each iteration step. We prove that this algorithm converges in norm. © 2004 Wiley Periodicals, Inc.     

Statistically and Computationally Efficient Estimating Equations for Large Spatial Datasets

For Gaussian process models, likelihood-based methods are often difficult to use with large irregularly spaced spatial datasets, because exact calculations of the likelihood for n observations require O(n³) operations and O(n²) memory. Various approximation methods have been developed to address the computational difficulties. In this article, we propose new, unbiased estimating equations (EE) based on score equation approximations that are both computationally and statistically efficient. We replace the inverse covariance matrix that appears in the score equations by a sparse matrix to approximate the quadratic forms, then set the resulting quadratic forms equal to their expected values to obtain unbiased EE. The sparse matrix is constructed by a sparse inverse Cholesky approach to approximate the inverse covariance matrix. The statistical efficiency of the resulting unbiased EE is evaluated both in theory and by numerical studies. Our methods are applied to nearly 90,000 satellite-based measurements of water vapor levels over a region in the Southeast Pacific Ocean.     

A Tikhonov-based projection iteration for nonlinear Ill-posed problems with sparsity constraints

In this paper, we consider nonlinear inverse problems where the solution is assumed to have a sparse expansion with respect to a preassigned basis or frame. We develop a scheme which allows to minimize a Tikhonov functional where the usual quadratic regularization term is replaced by a one-homogeneous (typically weighted ℓ _p ) penalty on the coefficients (or isometrically transformed coefficients) of such expansions. For (p < 2), the regularized solution will have a sparser expansion with respect to the basis or frame under consideration. The computation of the regularized solution amounts in our setting to a Landweber-fixed-point iteration with a projection applied in each fixed-point iteration step. The performance of the resulting numerical scheme is demonstrated by solving the nonlinear inverse single photon emission computerized tomography (SPECT) problem.     

On the construction of bodies of constant width containing a given set

We discuss the problem of sparse representation of domains in ℝ ^d . We demonstrate how the recently developed general theory of greedy approximation in Banach spaces can be used in this problem. The use of greedy approximation has two important advantages: (1) it works for an arbitrary dictionary of sets used for sparse representation and (2) the method of approximation does not depend on smoothness properties of the domains and automatically provides a near optimal rate of approximation for domains with different smoothness properties. We also give some lower estimates of the approximation error and discuss a specific greedy algorithm for approximation of convex domains in ℝ².     

Matching-based preprocessing algorithms to the solution of saddle-point problems in large-scale nonconvex interior-point optimization

Interior-point methods are among the most efficient approaches for solving large-scale nonlinear programming problems. At the core of these methods, highly ill-conditioned symmetric saddle-point problems have to be solved. We present combinatorial methods to preprocess these matrices in order to establish more favorable numerical properties for the subsequent factorization. Our approach is based on symmetric weighted matchings and is used in a sparse direct LDL ^T factorization method where the pivoting is restricted to static supernode data structures. In addition, we will dynamically expand the supernode data structure in cases where additional fill-in helps to select better numerical pivot elements. This technique can be seen as an alternative to the more traditional threshold pivoting techniques. We demonstrate the competitiveness of this approach within an interior-point method on a large set of test problems from the CUTE and COPS sets, as well as large optimal control problems based on partial differential equations. The largest nonlinear optimization problem solved has more than 12 million variables and 6 million constraints.     

On Backward p(x)-Parabolic Equations for Image Enhancement

In this study, we investigate the backward p(x)-parabolic equation as a new methodology to enhance images. We propose a novel iterative regularization procedure for the backward p(x)-parabolic equation based on the nonlinear Landweber method for inverse problems. The proposed scheme can also be extended to the family of iterative regularization methods involving the nonlinear Landweber method. We also investigate the connection between the variable exponent p(x) in the proposed energy functional and the diffusivity function in the corresponding Euler-Lagrange equation. It is well known that the forward problems converges to a constant solution destroying the image. The purpose of the approach of the backward problems is twofold. First, solving the backward problem by a sequence of forward problems, we obtain a smooth image which is denoised. Second, by choosing the initial data properly, we try to reduce the blurriness of the image. The numerical results for denoising appear to give improvement over standard methods as shown by preliminary results.     

Estimating Precision in Functional Images

Abstract

Functional imaging of biologic parameters like in vivo tissue metabolism is made possible by Positron Emission Tomography (PET). Many techniques have been suggested for extracting such images from dynamic time-course sequences of reconstructed PET scans. Quantitating the precision of these estimates is important for drawing inferences on the biologic parameters. Analytic variance formulas are not immediate owing to the nonlinear methods used in extraction. The usual resampling approach is infeasible because each image reconstruction in PET is a computationally demanding solution to a high-dimensional linear inverse problem. We suggest an alternative simulation approach that approximates the distribution of reconstructed PET scans and performs a parametric bootstrap in the imaging domain. Results on a simplified model chosen to match the characteristics of PET reconstruction are very encouraging. Mixture analysis is used to estimate functional images; however, the suggested approach is general enough to extend to other techniques or imaging methods.     

L p estimates of boundary integral equations for some nonlinear boundary value problems

Summary Recently, Galerkin and collocation methods have been analyzed for boundary integral equation formulations of some potential problems in the plane with nonlinear boundary conditions, and stability results and error estimates in theH ^1/2-norm have been proved (Ruotsalainen and Wendland, and Ruotsalainen and Saranen). We show that these results extend toL ^p setting without any extra conditions. These extensions are proved by studying the uniform boundedness of the inverses of the linearized integral operators, and then considering the nonlinear equations. The fact that inH ^1/2 setting the nonlinear operator is a homeomorphism with Lipschitz continuous inverse plays a crucial role. Optimal error estimates for the Galerkin and collocation method inL ^p space then follow.This research was performed while the second author was visiting professor at the University of Delaware, spring 1989     

Local linear regression for functional data

We study a non-linear regression model with functional data as inputs and scalar response. We propose a pointwise estimate of the regression function that maps a Hilbert space onto the real line by a local linear method and derive its asymptotic mean square error. Computations involve a linear inverse problem as well as a representation of the small ball probability of the data and are based on recent advances in this area.     

Sparse time-frequency representation of nonlinear and nonstationary data

Adaptive data analysis provides an important tool in extracting hidden physical information from multiscale data that arise from various applications. In this paper, we review two data-driven time-frequency analysis methods that we introduced recently to study trend and instantaneous frequency of nonlinear and nonstationary data. These methods are inspired by the empirical mode decomposition method (EMD) and the recently developed compressed (compressive) sensing theory. The main idea is to look for the sparsest representation of multiscale data within the largest possible dictionary consisting of intrinsic mode functions of the form {a(t) cos(θ(t))}, where a is assumed to be less oscillatory than cos(θ(t)) and θ′ ? 0. This problem can be formulated as a nonlinear l ⁰ optimization problem. We have proposed two methods to solve this nonlinear optimization problem. The first one is based on nonlinear basis pursuit and the second one is based on nonlinear matching pursuit. Convergence analysis has been carried out for the nonlinear matching pursuit method. Some numerical experiments are given to demonstrate the effectiveness of the proposed methods.     

An Isometric Representation of the Dual of {\user1{\mathcal{C}}}{\left( {X,\mathbb{R}} \right)}

Many structures in functional analysis are introduced as the limit of an inverse (aka projective) system of seminormed spaces [2, 3, 8]. In these situations, the dual is moreover equipped with a seminorm. Although the topology of the inverse limit is seldom metrizable, there is always a natural overlying locally convex approach structure. We provide a method for computing the adjoint of this space, by showing that the dual of a limit of locally convex approach spaces becomes a co-limit in the category of seminormed spaces. As an application we obtain an isometric representation of the dual space of real valued continuous functions on a locally compact Hausdorff space X, equipped with the compact open structure.     

Inverse problem for Sturm-Liouville and hill equations

Summary We discuss the inverse Sturm-Liouville problem on a finite interval by the method of transformation kernel. The -function, the Fredholm determinant of the transformation kernel, is explicitly written down in terms of the spectral data, from which a very explicit representation formula for the potential is deduced, and well-posedness of the inverse problem is established. The above method is also applicated to the inverse problem for Hill equations, in particular to the isospectral problem. We obtain an analog of FIT formula and a regularity theorem.     

A modified greedy analysis pursuit algorithm for the cosparse analysis model

In the past decade, the sparse representation synthesis model has been deeply researched and widely applied in signal processing. Recently, a cosparse analysis model has been introduced as an interesting alternative to the sparse representation synthesis model. The sparse synthesis model pay attention to non-zero elements in a representation vector x, while the cosparse analysis model focuses on zero elements in the analysis representation vector Ωx. This paper mainly considers the problem of the cosparse analysis model. Based on the greedy analysis pursuit algorithm, by constructing an adaptive weighted matrix W _k?1, we propose a modified greedy analysis pursuit algorithm for the sparse recovery problem when the signal obeys the cosparse model. Using a weighted matrix, we fill the gap between greedy algorithm and relaxation techniques. The standard analysis shows that our algorithm is convergent. We estimate the error bound for solving the cosparse analysis model, and then the presented simulations demonstrate the advantage of the proposed method for the cosparse inverse problem.     

Canonical dual least square method for solving general nonlinear systems of quadratic equations

This paper presents a canonical dual approach for solving general nonlinear algebraic systems. By using least square method, the nonlinear system of m-quadratic equations in n-dimensional space is first formulated as a nonconvex optimization problem. We then proved that, by the canonical duality theory developed by the second author, this nonconvex problem is equivalent to a concave maximization problem in ℝ ^m , which can be solved easily by well-developed convex optimization techniques. Both existence and uniqueness of global optimal solutions are discussed, and several illustrative examples are presented.     

Rational approximation to the Fermi–Dirac function with applications in density functional theory

We are interested in computing the Fermi–Dirac matrix function in which the matrix argument is the Hamiltonian matrix arising from density functional theory (DFT) applications. More precisely, we are really interested in the diagonal of this matrix function. We discuss rational approximation methods to the problem, specifically the rational Chebyshev approximation and the continued fraction representation. These schemes are further decomposed into their partial fraction expansions, leading ultimately to computing the diagonal of the inverse of a shifted matrix over a series of shifts. We describe Lanczos and sparse direct methods to address these systems. Each approach has advantages and disadvantages that are illustrated with experiments.     

TV‐like regularization for backward parabolic problems

This paper investigates an inverse problem for parabolic equations backward in time, which is solved by total‐variation‐like (TV‐like, in abbreviation) regularization method with cost function ∥u_x∥². The existence, uniqueness and stability estimate for the regularization problem are deduced in the linear case. For numerical illustration, the variational adjoint method, which presents a simple method to derive the gradient of the optimization functional, is introduced to reconstruct the unknown initial condition for both linear and nonlinear parabolic equations. The conjugate gradient method is used to iteratively search for the optimal approximation. Numerical results validate the feasibility and effectiveness of the proposed algorithm. Copyright © 2016 John Wiley & Sons, Ltd.     

Compactly supported shearlets are optimally sparse

Cartoon-like images, i.e., C² functions which are smooth apart from a C² discontinuity curve, have by now become a standard model for measuring sparse (nonlinear) approximation properties of directional representation systems. It was already shown that curvelets, contourlets, as well as shearlets do exhibit sparse approximations within this model, which are optimal up to a log-factor. However, all those results are only applicable to band-limited generators, whereas, in particular, spatially compactly supported generators are of uttermost importance for applications.In this paper, we present the first complete proof of optimally sparse approximations of cartoon-like images by using a particular class of directional representation systems, which indeed consists of compactly supported elements. This class will be chosen as a subset of (non-tight) shearlet frames with shearlet generators having compact support and satisfying some weak directional vanishing moment conditions.     

On sparse reconstruction from Fourier and Gaussian measurements

This paper improves upon best‐known guarantees for exact reconstruction of a sparse signal f from a small universal sample of Fourier measurements. The method for reconstruction that has recently gained momentum in the sparse approximation theory is to relax this highly nonconvex problem to a convex problem and then solve it as a linear program. We show that there exists a set of frequencies Ω such that one can exactly reconstruct every r‐sparse signal f of length n from its frequencies in Ω, using the convex relaxation, and Ω has size A random set Ω satisfies this with high probability. This estimate is optimal within the log logn and log³r factors. We also give a relatively short argument for a similar problem with k(r, n) ≈ r[12 + 8 log(n/r)] Gaussian measurements. We use methods of geometric functional analysis and probability theory in Banach spaces, which makes our arguments quite short. © 2007 Wiley Periodicals, Inc.     

On the Neumann problem for the Helmholtz equation in a plane angle

We prove that the solution of the Neumann problem for the Helmholtz equation in a plane angle Ω with boundary conditions from the space H_−1/2(Γ), where Γ is the boundary of Ω, which is provided by the well‐known Sommerfeld integral, belongs to the Sobolev space H₁(Ω) and depends continuously on the boundary values. To this end, we use another representation of the solution given by the inverse two‐dimensional Fourier transform of an analytic function depending on the Cauchy data of the solution. Copyright © 2000 John Wiley & Sons, Ltd.