An alternating projected gradient algorithm for nonnegative matrix factorization

Due to the extensive applications of nonnegative matrix factorizations (NMFs) of nonnegative matrices, such as in image processing, text mining, spectral data analysis, speech processing, etc., algorithms for NMF have been studied for years. In this paper, we propose a new algorithm for NMF, which is based on an alternating projected gradient (APG) approach. In particular, no zero entries appear in denominators in our algorithm which implies no breakdown occurs, and even if some zero entries appear in numerators new updates can always be improved in our algorithm. It is shown that the effect of our algorithm is better than that of Lee and Seung’s algorithm when we do numerical experiments on two known facial databases and one iris database.     

An alternating direction algorithm for matrix completion with nonnegative factors

This paper introduces an algorithm for the nonnegative matrix factorization-and-completion problem, which aims to find nonnegative low-rank matrices X and Y so that the product XY approximates a nonnegative data matrix M whose elements are partially known (to a certain accuracy). This problem aggregates two existing problems: (i) nonnegative matrix factorization where all entries of M are given, and (ii) low-rank matrix completion where nonnegativity is not required. By taking the advantages of both nonnegativity and low-rankness, one can generally obtain superior results than those of just using one of the two properties. We propose to solve the non-convex constrained least-squares problem using an algorithm based on the classical alternating direction augmented Lagrangian method. Preliminary convergence properties of the algorithm and numerical simulation results are presented. Compared to a recent algorithm for nonnegative matrix factorization, the proposed algorithm produces factorizations of similar quality using only about half of the matrix entries. On tasks of recovering incomplete grayscale and hyperspectral images, the proposed algorithm yields overall better qualities than those produced by two recent matrix-completion algorithms that do not exploit nonnegativity.     

A very fast algorithm for matrix factorization

We present a very fast algorithm for general matrix factorization of a data matrix for use in the statistical analysis of high-dimensional data via latent factors. Such data are prevalent across many application areas and generate an ever-increasing demand for methods of dimension reduction in order to undertake the statistical analysis of interest. Our algorithm uses a gradient-based approach which can be used with an arbitrary loss function provided the latter is differentiable. The speed and effectiveness of our algorithm for dimension reduction is demonstrated in the context of supervised classification of some real high-dimensional data sets from the bioinformatics literature.     

An alternating direction method for linear‐constrained matrix nuclear norm minimization

The aim of the nuclear norm minimization problem is to find a matrix that minimizes the sum of its singular values and satisfies some constraints simultaneously. Such a problem has received more attention largely because it is closely related to the affine rank minimization problem, which appears in many control applications including controller design, realization theory, and model reduction. In this paper, we first propose an exact version alternating direction method for solving the nuclear norm minimization problem with linear equality constraints. At each iteration, the method involves a singular value thresholding and linear matrix equations which are solved exactly. Convergence of the proposed algorithm is followed directly. To broaden the capacity of solving larger problems, we solve approximately the subproblem by an iterative method with the Barzilai–Borwein steplength. Some extensions to the noisy problems and nuclear norm regularized least‐square problems are also discussed. Numerical experiments and comparisons with the state‐of‐the‐art method FPCA show that the proposed method is effective and promising. Copyright © 2011 John Wiley & Sons, Ltd.     

An LDLT factorization based ADI algorithm for solving large-scale differential matrix equations

Large-scale differential matrix equations appear in many applications like optimal control of partial differential equations, balanced truncation model order reduction of linear time varying systems etc. Here, we will focus on matrix Riccati differential equations (RDE). Solving such matrix valued ordinary differential equations (ODE) is a highly storage and time consuming process. Therefore, it is necessary to develop efficient solution strategies minimizing both. We present an LDL^T factorization based ADI method for solving algebraic Lyapunov equations (ALE) arising in the innermost iteration during the application of Rosenbrock ODE solvers to RDEs. We show that the LDL^T-type decomposition avoids complex arithmetic, as well as cancellation effects arising from indefinite right hand sides of the ALEs appearing in the classic ZZ^T based approach. Additionally, a certain number of linear system solves can be saved within the ADI algorithm by reducing the number of column blocks in the right hand sides while the full accuracy of the standard low-rank ADI is preserved. (© 2014 Wiley-VCH Verlag GmbH & Co. KGaA, Weinheim)     

An alternating direction method for general variational inequalities

In this paper, we present an alternating direction method for structured general variational inequalities. This method only needs functional values for given variables in the solution process and does not require the estimate of the co-coercive modulus. All the computing process are easily implemented and the global convergence is also presented under mild assumptions. Some preliminary computational results are given.     

An oracle inequality for quasi-Bayesian nonnegative matrix factorization

The aim of this paper is to provide some theoretical understanding of quasi-Bayesian aggregation methods of nonnegative matrix factorization. We derive an oracle inequality for an aggregated estimator. This result holds for a very general class of prior distributions and shows how the prior affects the rate of convergence.     

On the convergence of von Neumann''s alternating projection algorithm for two sets

We give several unifying results, interpretations, and examples regarding the convergence of the von Neumann alternating projection algorithm for two arbitrary closed convex nonempty subsets of a Hilbert space. Our research is formulated within the framework of Fejér monotonicity, convex and set-valued analysis. We also discuss the case of finitely many sets.     

On the convergence of von Neumann's alternating projection algorithm for two sets

We give several unifying results, interpretations, and examples regarding the convergence of the von Neumann alternating projection algorithm for two arbitrary closed convex nonempty subsets of a Hilbert space. Our research is formulated within the framework of Fejér monotonicity, convex and set-valued analysis. We also discuss the case of finitely many sets.     

解一类结构变分不等式问题的非精确并行交替方向法

带线性约束的具有两分块结构的单调变分不等式问题, 出现在许多现代应用中, 如交通和经济问题等. 基于该问题良好的可分结构, 分裂型算法被广泛研究用于其求解. 提出新的带回代的非精确并行交替方向法解该类问题, 在每一步迭代中,首先以并行模式通过投影得到预测点, 然后对其校正得到下一步的迭代点. 在压缩型算法的理论框架下, 在适当条件下证明了所提算法的全局收敛性. 数值结果表明了算法的有效性. 此外, 该算法可推广到求解具有多分块结构的问题.     

An alternating direction implicit scheme for discrete ordinate transport equations

This paper concerns a finite difference approximation of the discrete ordinate equations for the time-dependent linear transport equation posed in a multi-dimensional rectangular parallelepiped with partially reflecting walls. We present an unconditionally stable alternating direction implicit finite difference scheme, show how to solve the difference equations, and establish the following properties of the scheme.If a sequence of difference approximations is considered in which the time and space increments approach zero, then the corresponding sequence of solutions has a subsequence which converges continuously to a strong solution of the discrete ordinate equations. Provided that the time increment is sufficiently small, independently of the space and velocity increment sizes: the solution of the difference equations is bounded by an exponential function of time; in the subcritical case the coefficient of t in this exponential bound is zero or negative; and if the constituent functions are all nonnegative, then the solution of the difference equations will also be nonnegative. This last result implies a monotonicity principle for solutions of related difference problems.     

An alternating direction implicit algorithm for the solution of linear complementarity problems arising from free boundary problems

We show that the alternating direction implicit algorithm of Peaceman—Rachford can be adapted to solve linear complementarity problems arising from free boundary problems. The alternating direction implicit algorithm is significantly faster than modified SOR algorithms. Some numerical results are given.This work was partially supported by the National Science Foundation under grant number MCS 77-27632.     

An improved Monte Carlo factorization algorithm

Pollard's Monte Carlo factorization algorithm usually finds a factor of a composite integerN inO(N ^1/4) arithmetic operations. The algorithm is based on a cycle-finding algorithm of Floyd. We describe a cycle-finding algorithm which is about 36 percent faster than Floyd's (on the average), and apply it to give a Monte Carlo factorization algorithm which is similar to Pollard's but about 24 percent faster.     

A class of fractional-order multi-scale variational models and alternating projection algorithm for image denoising

The total variation model proposed by Rudin, Osher and Fatemi performs very well for removing noise while preserving edges. However, it favors a piecewise constant solution in BV space which often leads to the staircase effect, and small details such as textures are often filtered out with noise in the process of denoising. To preserve the textures and eliminate the staircase effect, we improve the total variation model in this paper. This is accomplished by the following steps: (1) we define a new space of functions of fractional-order bounded variation called the BV_α space by using the Grünwald–Letnikov definition of fractional-order derivative; (2) we model the structure of the image as a function belonging to the BV_α space, and the textures in different scales as functions belonging to different negative Sobolev spaces. Thus, we propose a class of fractional-order multi-scale variational models for image denoising. (3) We analyze some properties of the fraction-order total variation operator and its conjugate operator. By using these properties, we develop an alternation projection algorithm for the new model and propose an efficient condition of the convergence of the algorithm. The numerical results show that the fractional-order multi-scale variational model can improve the peak signal to noise ratio of image, preserve textures and eliminate the staircase effect efficiently in the process of denoising.     

An extension ofA-stability to alternating direction implicit methods

Completely implicit, noniterative, finite-difference schemes have recently been developed by several authors for nonlinear, multidimensional systems of hyperbolic and mixed hyperbolic-parabolic partial differential equations. The method of Douglas and Gunn or the method of approximate factorization can be used to reduce the computational problem to a sequence of one-dimensional or alternating direction implicit (ADI) steps. Since the eigenvalues of partial differential equations (for example, the equations of compressible fluid dynamics) are often widely distributed with large imaginary parts,A-stable integration formulas provide ideal time-differencing approximations. In this paper it is shown that if anA-stable linear multistep method is used to integrate a model two-dimensional hyperbolic-parabolic partial differential equation, then one can always construct an ADI scheme by the method of approximate factorization which is alsoA-stable, i.e., unconditionally stable. A more restrictive result is given for three spatial dimensions. Since necessary and sufficient conditions forA-stability can easily be determined by using the theory of positive real functions, the stability analysis of the factored partial difference equations is reduced to a simple algebraic test.The main results of this paper were presented at the SIAM National Meeting, Madison, Wis., May 24 to 26, 1978, and section 9 was part of a presentation at the 751st Meeting of the American Mathematical Society, San Luis Obispo, California, Nov. 11 to 12, 1977.     

Accelerated parallel and distributed algorithm using limited internal memory for nonnegative matrix factorization

Nonnegative matrix factorization (NMF) is a powerful technique for dimension reduction, extracting latent factors and learning part-based representation. For large datasets, NMF performance depends on some major issues such as fast algorithms, fully parallel distributed feasibility and limited internal memory. This research designs a fast fully parallel and distributed algorithm using limited internal memory to reach high NMF performance for large datasets. Specially, we propose a flexible accelerated algorithm for NMF with all its \(L_1\) \(L_2\) regularized variants based on full decomposition, which is a combination of exact line search, greedy coordinate descent, and accelerated search. The proposed algorithm takes advantages of these algorithms to converges linearly at an over-bounded rate \((1-\frac{\mu }{L})(1 - \frac{\mu }{rL})^{2r}\) in optimizing each factor matrix when fixing the other factor one in the sub-space of passive variables, where r is the number of latent components, and \(\mu \) and L are bounded as \(\frac{1}{2} \le \mu \le L \le r\). In addition, the algorithm can exploit the data sparseness to run on large datasets with limited internal memory of machines, which is is advanced compared to fast block coordinate descent methods and accelerated methods. Our experimental results are highly competitive with seven state-of-the-art methods about three significant aspects of convergence, optimality and average of the iteration numbers.     

An alternating direction implicit method for orthogonal spline collocation linear systems

Summary An Alternating Direction Implicit method is analyzed for the solution of linear systems arising in high-order, tensor-product orthogonal spline collocation applied to some separable, second order, linear, elliptic partial differential equations in rectangles. On anNxN partition, with Jordan's selection of the acceleration parameters, the method requiresO(N ² ln ² N) arithmetic operations to produce an approximation whose accuracy, in theH ¹-norm, is that of the collocation solution.     

An alternating direction Galerkin method for elliptic problems with gradient nonlinearity

Several problems in many applications involve the solution of partial differential equations with gradient dependent nonlinearity. The numerical solution of the resulting nonlinear system is rather expensive. We present an alternating direction Galerkin method that allows much faster solution of the nonlinear system. The alternating direction formulation help reduce the problem into a sequence of nonlinear systems that may be solved very efficiently. Theoretical study of the convergence of the method will also be presented. (© 2008 WILEY-VCH Verlag GmbH & Co. KGaA, Weinheim)     

An alternating projection that does not converge in norm

This paper proves that the method of alternating projections between two closed convex intersecting sets does not always converge in norm. Weak convergence was established by Bregman (Soviet Math. Dokl. 6 (1965) 688), but the status of norm convergence was undetermined. An explicit counterexample is provided.