Two-Subspace Projection Method for Coherent Overdetermined Systems

We present a Projection onto Convex Sets (POCS) type algorithm for solving systems of linear equations. POCS methods have found many applications ranging from computer tomography to digital signal and image processing. The Kaczmarz method is one of the most popular solvers for overdetermined systems of linear equations due to its speed and simplicity. Here we introduce and analyze an extension of the Kaczmarz method that iteratively projects the estimate onto a solution space given by two randomly selected rows. We show that this projection algorithm provides exponential convergence to the solution in expectation. The convergence rate improves upon that of the standard randomized Kaczmarz method when the system has correlated rows. Experimental results confirm that in this case our method significantly outperforms the randomized Kaczmarz method.     

A generalized cyclic iterative method for solving variational inequalities over the solution set of a split common fixed point problem

For solving the large-scale linear least-squares problem, we propose a block version of the randomized extended Kaczmarz method, called the two-subspace randomized extended Kaczmarz method, which does not require any row or column paving. Theoretical analysis and numerical results show that the two-subspace randomized extended Kaczmarz method is much more efficient than the randomized extended Kaczmarz method. When the coefficient matrix is of full column rank, the two-subspace randomized extended Kaczmarz method can also outperform the randomized coordinate descent method. If the linear system is consistent, we remove one of the iteration sequences in the two-subspace randomized extended Kaczmarz method, which approximates the projection of the right-hand side vector onto the orthogonal complement space of the range space of the coefficient matrix, and obtain the generalized two-subspace randomized Kaczmarz method, which is actually a generalization of the two-subspace randomized Kaczmarz method without the assumptions of unit row norms and full column rank on the coefficient matrix. We give the upper bound for the convergence rate of the generalized two-subspace randomized Kaczmarz method which also leads to a better upper bound for the convergence rate of the two-subspace randomized Kaczmarz method.

     

Variant of Greedy Randomized Gauss-Seidel Method for Ridge Regression

The variants of randomized Kaczmarz and randomized Gauss-Seidel algorithms are two effective stochastic iterative methods for solving ridge regression problems. For solving ordinary least squares regression problems, the greedy randomized Gauss-Seidel (GRGS) algorithm always performs better than the randomized Gauss-Seidel algorithm (RGS) when the system is overdetermined. In this paper, inspired by the greedy modification technique of the GRGS algorithm, we extend the variant of the randomized Gauss-Seidel algorithm, obtaining a variant of greedy randomized Gauss-Seidel (VGRGS) algorithm for solving ridge regression problems. In addition, we propose a relaxed VGRGS algorithm and the corresponding convergence theorem is established. Numerical experiments show that our algorithms outperform the VRK-type and the VRGS algorithms when $m > n$.     

On the Kaczmarz iterative method and its generalizations

Under study is the family of iterative projection methods that stems from the work of Kaczmarz in 1937. We propose some alternating block algorithms of Kaczmarz type in Krylov subspaces with a relaxation parameter and acceleration in Krylov spaces. The results are presented and discussed of numerical experiments for some model problems.     

On greedy randomized coordinate descent methods for solving large linear least‐squares problems

For solving large scale linear least‐squares problem by iteration methods, we introduce an effective probability criterion for selecting the working columns from the coefficient matrix and construct a greedy randomized coordinate descent method. It is proved that this method converges to the unique solution of the linear least‐squares problem when its coefficient matrix is of full rank, with the number of rows being no less than the number of columns. Numerical results show that the greedy randomized coordinate descent method is more efficient than the randomized coordinate descent method.     

Acceleration of randomized Kaczmarz method via the Johnson–Lindenstrauss Lemma

The Kaczmarz method is an algorithm for finding the solution to an overdetermined consistent system of linear equations Ax = b by iteratively projecting onto the solution spaces. The randomized version put forth by Strohmer and Vershynin yields provably exponential convergence in expectation, which for highly overdetermined systems even outperforms the conjugate gradient method. In this article we present a modified version of the randomized Kaczmarz method which at each iteration selects the optimal projection from a randomly chosen set, which in most cases significantly improves the convergence rate. We utilize a Johnson–Lindenstrauss dimension reduction technique to keep the runtime on the same order as the original randomized version, adding only extra preprocessing time. We present a series of empirical studies which demonstrate the remarkable acceleration in convergence to the solution using this modified approach.     

A Randomized Kaczmarz Algorithm with Exponential Convergence

The Kaczmarz method for solving linear systems of equations is an iterative algorithm that has found many applications ranging from computer tomography to digital signal processing. Despite the popularity of this method, useful theoretical estimates for its rate of convergence are still scarce. We introduce a randomized version of the Kaczmarz method for consistent, overdetermined linear systems and we prove that it converges with expected exponential rate. Furthermore, this is the first solver whose rate does not depend on the number of equations in the system. The solver does not even need to know the whole system but only a small random part of it. It thus outperforms all previously known methods on general extremely overdetermined systems. Even for moderately overdetermined systems, numerical simulations as well as theoretical analysis reveal that our algorithm can converge faster than the celebrated conjugate gradient algorithm. Furthermore, our theory and numerical simulations confirm a prediction of Feichtinger et al. in the context of reconstructing bandlimited functions from nonuniform sampling. T. Strohmer was supported by NSF DMS grant 0511461. R. Vershynin was supported by the Alfred P. Sloan Foundation and by NSF DMS grant 0401032.     

Projection methods for the numerical solution of non-self-adjoint elliptic partial differential equations

We compare the relative performances of two iterative schemes based on projection techniques for the solution of large sparse nonsymmetric systems of linear equations, encountered in the numerical solution of partial differential equations. The Block–Symmetric Successive Over-Relaxation (Block-SSOR) method and the Symmetric–Kaczmarz method are derived from the simplest of projection methods, that is, the Kaczmarz method. These methods are then accelerated using the conjugate gradient method, in order to improve their convergence. We study their behavior on various test problems and comment on the conditions under which one method would be better than the other. We show that while the conjugate-gradient-accelerated Block-SSOR method is more amenable to implementation on vector and parallel computers, the conjugate-gradient accelerated Symmetric–Kaczmarz method provides a viable alternative for use on a scalar machine.     

Tight upper bounds for the convergence of the randomized extended Kaczmarz and Gauss–Seidel algorithms

The randomized extended Kaczmarz and Gauss–Seidel algorithms have attracted much attention because of their ability to treat all types of linear systems (consistent or inconsistent, full rank or rank deficient). In this paper, we present tight upper bounds for the convergence of the randomized extended Kaczmarz and Gauss–Seidel algorithms. Numerical experiments are given to illustrate the theoretical results.     

Randomized Kaczmarz solver for noisy linear systems

The Kaczmarz method is an iterative algorithm for solving systems of linear equations Ax=b. Theoretical convergence rates for this algorithm were largely unknown until recently when work was done on a randomized version of the algorithm. It was proved that for overdetermined systems, the randomized Kaczmarz method converges with expected exponential rate, independent of the number of equations in the system. Here we analyze the case where the system Ax=b is corrupted by noise, so we consider the system Ax≈b+r where r is an arbitrary error vector. We prove that in this noisy version, the randomized method reaches an error threshold dependent on the matrix A with the same rate as in the error-free case. We provide examples showing our results are sharp in the general context.     

The randomized Kaczmarz method with mismatched adjoint

This paper investigates the randomized version of the Kaczmarz method to solve linear systems in the case where the adjoint of the system matrix is not exact—a situation we refer to as “mismatched adjoint”. We show that the method may still converge both in the over- and underdetermined consistent case under appropriate conditions, and we calculate the expected asymptotic rate of linear convergence. Moreover, we analyze the inconsistent case and obtain results for the method with mismatched adjoint as for the standard method. Finally, we derive a method to compute optimized probabilities for the choice of the rows and illustrate our findings with numerical examples.     

A derandomization approach to recovering bandlimited signals across a wide range of random sampling rates

Reconstructing bandlimited functions from random sampling is an important problem in signal processing. Strohmer and Vershynin obtained good results for this problem by using a randomized version of the Kaczmarz algorithm (RK) and assigning to every equation a probability weight proportional to the average distance of the sample from its two nearest neighbors. However, their results are valid only for moderate to high sampling rates; in practice, it may not always be possible to obtain many samples. Experiments show that the number of projections required by RK and other Kaczmarz variants rises seemingly exponentially when the equations/variables ratio (EVR) falls below 5. CGMN, which is a CG acceleration of Kaczmarz, provides very good results for low values of EVR and it is much better than CGNR and CGNE. A derandomization method, based on an extension of the bit-reversal permutation, is combined with the weights and shown to improve the performance of CGMN and the regular (cyclic) Kaczmarz, which even outperforms RK. A byproduct of our results is the finding that signals composed mainly of high-frequency components are easier to recover.     

The optimal relaxation parameter for the SOR method applied to the Poisson equation in any space dimensions

The finite difference discretization of the Poisson equation with Dirichlet boundary conditions leads to a large, sparse system of linear equations for the solution values at the interior mesh points. This problem is a popular and useful model problem for performance comparisons of iterative methods for the solution of linear systems. To use the successive overrelaxation (SOR) method in these comparisons, a formula for the optimal value of its relaxation parameter is needed. In standard texts, this value is only available for the case of two space dimensions, even though the model problem is also instructive in higher dimensions. This note extends the derivation of the optimal relaxation parameter to any space dimension and confirms its validity by means of test calculations in three dimensions.     

Greedy approaches for a class of nonlinear Generalized Assignment Problems

The traditional Generalized Assignment Problem (GAP) seeks an assignment of customers to facilities that minimizes the sum of the assignment costs while respecting the capacity of each facility. We consider a nonlinear GAP where, in addition to the assignment costs, there is a nonlinear cost function associated with each facility whose argument is a linear function of the customers assigned to the facility. We propose a class of greedy algorithms for this problem that extends a family of greedy algorithms for the GAP. The effectiveness of these algorithms is based on our analysis of the continuous relaxation of our problem. We show that there exists an optimal solution to the continuous relaxation with a small number of fractional variables and provide a set of dual multipliers associated with this solution. This set of dual multipliers is then used in the greedy algorithm. We provide conditions under which our greedy algorithm is asymptotically optimal and feasible under a stochastic model of the parameters.     

On the Acceleration of Kaczmarz Projection Algorithm

In the paper [1], a direct version of the classical Kaczmarz algorithm was proposed, which gives us in only one iteration a solution of an arbitrary consistent system of linear equations. Unfortunately, as any direct method applied to large sparse matrices, this algorithm is based on some modifications of the system matrix sparsity structure such that a big fill-in appears. In order to overcome this difficulty, in the present paper we propose a modified version of this direct Kaczmarz algorithm in which the transformations applied to the system matrix try to conserve the initial sparsity structure. This transformations are done via clustering using Jaccard and Hamming distances. The modified Kaczmarz algorithm is no more a direct method, but we obtain an acceleration of convergence with respect to the classical Kaczmarz algorithm. Numerical experiments which ilustrate the efficiency of our algorithm are also presented. (© 2013 Wiley-VCH Verlag GmbH & Co. KGaA, Weinheim)     

On block Gaussian sketching for the Kaczmarz method

Numerical Algorithms - The Kaczmarz algorithm is one of the most popular methods for solving large-scale over-determined linear systems due to its simplicity and computational efficiency. This...     

Selectable Set Randomized Kaczmarz

The Randomized Kaczmarz method (RK) is a stochastic iterative method for solving linear systems that has recently grown in popularity due to its speed and low memory requirement. Selectable Set Randomized Kaczmarz is a variant of RK that leverages existing information about the Kaczmarz iterate to identify an adaptive “selectable set” and thus yields an improved convergence guarantee. In this article, we propose a general perspective for selectable set approaches and prove a convergence result for that framework. In addition, we define two specific selectable set sampling strategies that have competitive convergence guarantees to those of other variants of RK. One selectable set sampling strategy leverages information about the previous iterate, while the other leverages the orthogonality structure of the problem via the Gramian matrix. We complement our theoretical results with numerical experiments that compare our proposed rules with those existing in the literature.     

Sampling Kaczmarz-Motzkin method for linear feasibility problems: generalization and acceleration

Mathematical Programming - Randomized Kaczmarz, Motzkin Method and Sampling Kaczmarz Motzkin (SKM) algorithms are commonly used iterative techniques for solving a system of linear inequalities...     

A note on Hermitian splitting induced relaxation methods for convection-diffusion equations

The solution of the linear system Ax = b by iterative methods requires a splitting of the coefficient matrix in the form A = M − N where M is usually chosen to be a diagonal or a triangular matrix. In this article we study relaxation methods induced by the Hermitian and skew-Hermitian splittings for the solution of the linear system arising from a compact fourth order approximation to the one dimensional convection-diffusion equation and compare the convergence rates of these relaxation methods to that of the widely used successive overrelaxation (SOR) method. Optimal convergence parameters are derived for each method and numerical experiments are given to supplement the theoretical estimates. For certain values of the diffusion parameter, a relaxation method based on the Hermitian splitting converges faster than SOR. For two-dimensional problems a block form of the iterative algorithm is presented. © 1998 John Wiley & Sons, Inc. Numer Methods Partial Differential Eq 14: 581–591, 1998     

Projection method for solving a singular system of linear equations and its applications

Summary The iterative method for solving system of linear equations, due to Kaczmarz [2], is investigated. It is shown that the method works well for both singular and non-singular systems and it determines the affine space formed by the solutions if they exist. The method also provides an iterative procedure for computing a generalized inverse of a matrix.