Structure methods for solving the nearest correlation matrix problem

The nearest correlation matrix problem is to find a positive semidefinite matrix with unit diagonal, that is, nearest in the Frobenius norm to a given symmetric matrix A. This problem arises in the finance industry, where the correlations are between stocks. In this paper, we formulate this problem as a smooth unconstrained minimization problem, for which rapid convergence can be obtained. Other methods are also studied. Comparative numerical results are reported.     

Alternating convex projection methods for discrete-time covariance control design

The problem of designing a controller for a linear, discretetime system is formulated as a problem of designing an appropriate plant-state covariance matrix. Closed-loop stability and multiple-output performance constraints are expressed geometrically as requirements that the covariance matrix lies in the intersection of some specified closed, convex sets in the space of symmetric matrices. We solve a covariance feasibility problem to determine the existence and compute a covariance matrix to satisty assignability and output-norm performance constraints. In addition, we can treat a covariance optimization problem to construct an assignable covariance matrix which satisfies output performance constraints and is as close as possible to a given desired covariance. We can also treat inconsistent constraints, where we look for an assignable covariance which best approximates desired but unachievable output performance objectives; we call this the infeasible covariance optimization problem. All these problems are of a convex nature, and alternating convex projection methods are proposed to solve them, exploiting the geometric formulation of the problem. To this end, analytical expressions for the projections onto the covariance assignability and the output covariance inequality constraint sets are derived. Finally, the problem of designing low-order dynamic controllers using alternating projections is discussed, and a numerical technique using alternating projections is suggested for a solution, although convergence of the algorithm is not guaranteed in this case. A control design example for a fighter aircraft model illustrates the method.This research was completed while the first author was with the Space Systems Control Laboratory at Purdue University. Support from the Army Research Office Grant ARO-29029-EG is gratefully acknowledged.     

Acceleration of the EM algorithm using the Vector Aitken method and its Steffensen form

Based on Vector Aitken (VA) method, we propose an acceleration Expectation-Maximization (EM) algorithm, VA-accelerated EM algorithm, whose convergence speed is faster than that of EM algorithm. The VA-accelerated EM algorithm does not use the information matrix but only uses the sequence of estimates obtained from iterations of the EM algorithm, thus it keeps the flexibility and simplicity of the EM algorithm. Considering Steffensen iterative process, we have also given the Steffensen form of the VA-accelerated EM algorithm. It can be proved that the reform process is quadratic convergence. Numerical analysis illustrate the proposed methods are efficient and faster than EM algorithm.     

The method of alternating projections and the method of subspace corrections in Hilbert space

A new identity is given in this paper for estimating the norm of the product of nonexpansive operators in Hilbert space. This identity can be applied for the design and analysis of the method of alternating projections and the method of subspace corrections. The method of alternating projections is an iterative algorithm for determining the best approximation to any given point in a Hilbert space from the intersection of a finite number of subspaces by alternatively computing the best approximations from the individual subspaces which make up the intersection. The method of subspace corrections is an iterative algorithm for finding the solution of a linear equation in a Hilbert space by approximately solving equations restricted on a number of closed subspaces which make up the entire space. The new identity given in the paper provides a sharpest possible estimate for the rate of convergence of these algorithms. It is also proved in the paper that the method of alternating projections is essentially equivalent to the method of subspace corrections. Some simple examples of multigrid and domain decomposition methods are given to illustrate the application of the new identity.

     

Accelerated reflection projection algorithm and its application to the LMI problem

We discuss accelerated version of the alternating projection method which can be applied to solve the linear matrix inequality (LMI) problem. The alternating projection method is a well-known algorithm for the convex feasibility problem, and has many generalizations and extensions. Bauschke and Kruk proposed a reflection projection algorithm for computing a point in the intersection of an obtuse cone and a closed convex set. We carry on this research in two directions. First, we present an accelerated version of the reflection projection algorithm, and prove its weak convergence in a Hilbert space; second, we prove the finite termination of an algorithm which is based on the proposed algorithm and provide an explicit upper bound for the required number of iterations under certain assumptions. Numerical experiments for the LMI problem are provided to demonstrate the effectiveness and merits of the proposed algorithms.     

Unconstrained Optimization Techniques for the Acceleration of Alternating Projection Methods

Alternating projection methods have been extensively used to find the closest point, to a given point, in the intersection of several given sets that belong to a Hilbert space. One of the characteristics of these schemes is the slow convergence that can be observed in practical applications. To overcome this difficulty, several techniques, based on different ideas, have been developed to accelerate their convergence. Recently, a successful acceleration scheme was developed specially for Cimmino's method when applied to the solution of large-scale saddle point problems. This specialized acceleration scheme is based on the use of the well-known conjugate gradient method for minimizing a related convex quadratic map. In this work, we extend and further analyze this optimization approach for several alternating projection methods on different scenarios. In particular, we include a specialized analysis and treatment for the acceleration of von Neumann-Halperin's method and Cimmino's method on subspaces, and Kaczmarz method on linear varieties. For some specific applications we illustrate the advantages of our acceleration schemes with encouraging numerical experiments.     

Block relaxation and majorization methods for the nearest correlation matrix with factor structure

We propose two numerical methods, namely the alternating block relaxation method and the alternating majorization method, for the problem of nearest correlation matrix with factor structure, which is highly nonconvex. In the block relaxation method, the subproblem is of the standard trust region problem, which is solved by Steighaug’s truncated conjugate gradient method or by the exact trust region method. In the majorization method, the subproblem has a closed-form solution. We then apply the majorization method to the case where nonnegative factors are required. The numerical results confirm that the proposed methods work quite well and are competitive against the best available methods.     

Additional Results on Convergence of Alternating Iterations Involving Rectangular Matrices

Different classes of matrix splittings of semi-monotone matrices lead to many comparison results, which are useful tools in examining the convergence rate of iterative methods for solving rectangular systems of linear equations in a faster way. In this context, the theory of alternating iterations for rectangular matrices was introduced recently, in order to arrive at the desired solution of accuracy or at the exact solution. In this article, we expand convergence theory of such alternating iterations and obtain comparison results for such iterations.     

关于Schwarz交替法的收敛因子

§1.Schwarz交替法的收敛因子 我们就二阶自共轭椭圆型方程的Dirichlet问题来讨论.设Ω?R~2为一多边形区域, a(u,v)=(f,v),v∈H_0~1(Ω),f∈H~(-1)(Ω), u∈H_0~1(Ω)是定义在其上的边值问题的变分形式,双线性型时a(·,·)满足     

A general iterative scheme with applications to convex optimization and related fields

A general iterative scheme including relaxation and a corresponding problem class are presented. Some global convergence results are given. The acceleration of convergence is discussed, The scheme comprises a lot of known iterative methods such as subgradient methods and methods of successive orthogonal projections with relaxation. Applications to convex optimization, convex feasibility problems, systems of convex inequalities, variational inequalities, operator equations and systems of linear equations are given.     

Modified Alternating Direction Methods for the Modified Multiple-Sets Split Feasibility Problems

In this paper, we propose two new multiple-sets split feasibility problem models and new solution methods. The first model is more separable than the original one, which enables us to apply a modified alternating direction method with parallel steps to solve it. Then, to overcome the difficulty of computing projections onto the constraint sets, a special version of this modified method with the strategy of projection onto half-space is given. The second model consists in finding a least Euclidean norm solution of the multiple-sets split feasibility problem, for which we provide another modified alternating direction method. Numerical results presented at the last show the efficiency of our methods.     

A note on the sufficient initial condition ensuring the convergence of directly extended 3-block ADMM for special semidefinite programming

ABSTRACT

In this note, we consider three types of problems, H-weighted nearest correlation matrix problem and two types of important doubly non-negative semidefinite programming, derived from the binary integer quadratic programming and maximum cut problem. The dual of these three types of problems is a 3-block separable convex optimization problem with a coupling linear equation constraint. It is known that, the directly extended 3-block alternating direction method of multipliers (ADMM3d) is more efficient than many of its variants for solving these convex optimization, but its convergence is not guaranteed. By choosing initial points properly, we obtain the convergence of ADMM3d for solving the dual of these three types of problems. Furthermore, we simplify the iterative scheme of ADMM3d and show the equivalence of ADMM3d to the 2-block semi-proximal ADMM for solving the dual's reformulation, under these initial conditions.     

基于交替投影算法求解单变量线性约束矩阵方程问题

研究如下线性约束矩阵方程求解问题:给定A∈R~(m×n),B∈R~(n×p)和C∈R~(m×p),求矩阵X∈R(?)R~(n×n)"使得A×B=C以及相应的最佳逼近问题,其中集合R为如对称阵,Toeplitz阵等构成的线性子空间,或者对称半(ε)正定阵,(对称)非负阵等构成的闭凸集.给出了在相容条件下求解该问题的交替投影算法及算法收敛性分析.通过大量数值算例说明该算法的可行性和高效性,以及该算法较传统的矩阵形式的Krylov子空间方法(可行前提下)在迭代效率上的明显优势,本文也通过寻求加速技巧进一步提高算法的收敛速度.     

Newton’s Method for Computing the Nearest Correlation Matrix with a Simple Upper Bound

The standard nearest correlation matrix can be efficiently computed by exploiting a recent development of Newton’s method (Qi and Sun in SIAM J. Matrix Anal. Appl. 28:360–385, 2006). Two key mathematical properties, that ensure the efficiency of the method, are the strong semismoothness of the projection operator onto the positive semidefinite cone and constraint nondegeneracy at every feasible point. In the case where a simple upper bound is enforced in the nearest correlation matrix in order to improve its condition number, it is shown, among other things, that constraint nondegeneracy does not always hold, meaning Newton’s method may lose its quadratic convergence. Despite this, the numerical results show that Newton’s method is still extremely efficient even for large scale problems. Through regularization, the developed method is applied to semidefinite programming problems with simple bounds.     

On some methods for entropy maximization and matrix scaling

We describe and survey in this paper iterative algorithms for solving the discrete maximum entropy problem with linear equality constraints. This problem has applications e.g. in image reconstruction from projections, transportation planning, and matrix scaling. In particular we study local convergence and asymptotic rate of convergence as a function of the iteration parameter. For the trip distribution problem in transportation planning and the equivalent problem of scaling a positive matrix to achieve a priori given row and column sums, it is shown how the iteration parameters can be chosen in an optimal way. We also consider the related problem of finding a matrix X, diagonally similar to a given matrix, such that corresponding row and column norms in X are all equal. Reports of some numerical tests are given.     

On the squared Smith method for large‐scale Stein equations

A squared Smith type algorithm for solving large‐scale discrete‐time Stein equations is developed. The algorithm uses restarted Krylov spaces to compute approximations of the squared Smith iterations in low‐rank factored form. Fast convergence results when very few iterations of the alternating direction implicit method are applied to the Stein equation beforehand. The convergence of the algorithm is discussed and its performance is demonstrated by several test examples. Copyright © 2013 John Wiley & Sons, Ltd.     

On the optimization of linear,time-variant,multivariable control systems using the contraction mapping principle

The problem of optimizing a linear, time-variant, multivariable control system with quadratic cost functional is first phrased in the context of functional analysis. It is shown that the optimum control is given implicitly as the solution of a matrix integral equation. After the fixed-point contraction mapping theorem is invoked, an iterative method for solving this equation is developed and conditions for its convergence to a unique optimum are derived. Techniques for transforming the system operator are discussed so that, when convergence of the original sequence of iterations cannot be assured, that of the transformed system can. However, it is shown specifically that there is always a finite optimization interval for which the procedure may be used. Bounds are also given for the errors, in the sense of norms, between the control aftern iterations and its ultimate value and between the cost functional aftern iterations and its ultimate value. These bounds are used to decide when to terminate the sequence. Solutions of the iterative scheme using a hybrid computer in parallel and serial modes are discussed and the delays inherent in both methods calculated. It is concluded that the method can be used to track an optimum control system, which drifts from optimum because of parameter variations, with little delay and particularly when the optimization interval is extrapolated only a little into the future. Comparison of the proposed scheme with the steepest-descent approach developed by Balakrishnan shows that the present scheme requires one-third of the computations per step and, therefore, may converge more quickly.The author is indebted to the Principal and the Governors of Sunderland Polytechnic for the facilities placed at his disposal and permission to publish this work.     

交替方向法求解带线性约束的变分不等式

１引言变分不等式是一个有广泛应用的数学问题,它的一般形式是：确定一个向量,使其满足这里ｆ是一个从到自身的一个映射,Ｓ是Ｒ中的一个闭凸集．在许多实际问题中集合Ｓ往往具有如下结构其中ＡｂＫ是中的一个简单闭凸集．例如一个正卦限,一个框形约束结构,或者一个球简言之,Ｓ是Ｒ中的一个超平面与一个简单闭凸集的交．求解问题（１）－（２）,往往是通过对线性约束Ａ引人Ｌａｇｒａｎｇｅ乘子,将原问题化为如下的变分不等式：确定使得我们记问题（３）－（４）为ＶＩ（Ｆ）．熟知［３］,ＶＩ（,Ｆ）等价于投影方程其中凡（·）表…     

Low-rank and sparse matrix recovery from noisy observations via 3-block ADMM algorithm

Recovering low-rank and sparse matrix from a given matrix arises in many applications, such as image processing, video background substraction, and so on. The 3-block alternating direction method of multipliers (ADMM) has been applied successfully to solve convex problems with 3-block variables. However, the existing sufficient conditions to guarantee the convergence of the 3-block ADMM usually require the penalty parameter $\gamma$ to satisfy a certain bound, which may affect the performance of solving the large scale problem in practice. In this paper, we propose the 3-block ADMM to recover low-rank and sparse matrix from noisy observations. In theory, we prove that the 3-block ADMM is convergent when the penalty parameters satisfy a certain condition and the objective function value sequences generated by 3-block ADMM converge to the optimal value. Numerical experiments verify that proposed method can achieve higher performance than existing methods in terms of both efficiency and accuracy.     

Incomplete oblique projections for solving large inconsistent linear systems

The authors introduced in previously published papers acceleration schemes for Projected Aggregation Methods (PAM), aiming at solving consistent linear systems of equalities and inequalities. They have used the basic idea of forcing each iterate to belong to the aggregate hyperplane generated in the previous iteration. That scheme has been applied to a variety of projection algorithms for solving systems of linear equalities or inequalities, proving that the acceleration technique can be successfully used for consistent problems. The aim of this paper is to extend the applicability of those schemes to the inconsistent case, employing incomplete projections onto the set of solutions of the augmented system Ax − r = b. These extended algorithms converge to the least squares solution. For that purpose, oblique projections are used and, in particular, variable oblique incomplete projections are introduced. They are defined by means of matrices that penalize the norm of the residuals very strongly in the first iterations, decreasing their influence with the iteration counter in order to fulfill the convergence conditions. The theoretical properties of the new algorithms are analyzed, and numerical experiences are presented comparing their performance with several well-known projection methods. Dedicated to Clovis Gonzaga on the occassion of his 60th birthday.