Space alternating penalized Kullback proximal point algorithms for maximizing likelihood with nondifferentiable penalty

The EM algorithm is a widely used methodology for penalized likelihood estimation. Provable monotonicity and convergence are the hallmarks of the EM algorithm and these properties are well established for smooth likelihood and smooth penalty functions. However, many relaxed versions of variable selection penalties are not smooth. In this paper, we introduce a new class of space alternating penalized Kullback proximal extensions of the EM algorithm for nonsmooth likelihood inference. We show that the cluster points of the new method are stationary points even when they lie on the boundary of the parameter set. We illustrate the new class of algorithms for the problems of model selection for finite mixtures of regression and of sparse image reconstruction.     

On the numerical solution of a logarithmic integral equation of the first kind for the Helmholtz equation

Summary We describe a quadrature method for the numerical solution of the logarithmic integral equation of the first kind arising from the single-layer approach to the Dirichlet problem for the two-dimensional Helmholtz equation in smooth domains. We develop an error analysis in a Sobolev space setting and prove fast convergence rates for smooth boundary data.     

Convergence analysis of an accelerated expectation‐maximization algorithm for ill‐posed integral equations

The maximum‐likelihood expectation‐maximization (EM) algorithm has attracted considerable interest in single‐photon emission computed tomography, because it produces superior images in addition to be being flexible, simple, and allowing a physical interpretation. However, it often needs a large number of calculations because of the algorithm's slow rate of convergence. Therefore, there is a large body of literature concerning the EM algorithm's acceleration. One of the accelerated means is increasing an overrelaxation parameter, whereas we have not found any analysis in this method that would provide an immediate answer to the questions of the convergence. In this paper, our main focus is on the continuous version of an accelerated EM algorithm based on Lewitt and Muehllenner. We extend their conclusions to the infinite‐dimensional space and interpret and analyze the convergence of the accelerated EM algorithm. We also obtain some new properties of the modified algorithm. Copyright © 2015 John Wiley & Sons, Ltd.     

Monotonicity of control volume methods

Robustness of numerical methods for multiphase flow problems in porous media is important for development of methods to be used in a wide range of applications. Here, we discuss monotonicity for a simplified problem of single-phase flow, but where the simulation grids and media are allowed to be general, posing challenges to control-volume methods. We discuss discrete formulations of the maximum principle and derive sufficient criteria for discrete monotonicity for arbitrary nine-point control-volume discretizations for conforming quadrilateral grids in 2D. These criteria are less restrictive than the M-matrix property. It is shown that it is impossible to construct nine-point methods which unconditionally satisfy the monotonicity criteria when the discretization satisfies local conservation and exact reproduction of linear potential fields. Numerical examples are presented which show the validity of the criteria for monotonicity. Further, the impact of nonmonotonicity is studied. Different behavior for different discretization methods is illuminated, and simple ideas are presented for improvement in terms of monotonicity.     

Monotonieeigenschaften des Steffensen-Verfahrens

Steffensen's method is slightly generalized by introducing a real parameter. In this way one can get different monotonicity properties, depending on the choice of this parameter. These monotonicity statements give the possibility of bracketing the solution of a given problem. In a special case they even ensure the convergence and the existence of a solution. Furthermore there are given sufficient conditions, which show that Steffensen's method converges at least as quickly as Newton's method. A numerical example shows the efficiency of the theorems and compares Steffensen's and Newton's method.

     

ML Estimation of the MultivariatetDistribution and the EM Algorithm

Maximum likelihood estimation of the multivariatetdistribution, especially with unknown degrees of freedom, has been an interesting topic in the development of the EM algorithm. After a brief review of the EM algorithm and its application to finding the maximum likelihood estimates of the parameters of thetdistribution, this paper provides new versions of the ECME algorithm for maximum likelihood estimation of the multivariatetdistribution from data with possibly missing values. The results show that the new versions of the ECME algorithm converge faster than the previous procedures. Most important, the idea of this new implementation is quite general and useful for the development of the EM algorithm. Comparisons of different methods based on two datasets are presented.     

One-step smoothing Newton method for solving the mixed complementarity problem with a function

The mixed complementarity problem (denote by MCP(F)) can be reformulated as the solution of a smooth system of equations. In the paper, based on a perturbed mid function, we propose a new smoothing function, which has an important property, not satisfied by many other smoothing function. The existence and continuity of a smooth path for solving the mixed complementarity problem with a P₀ function are discussed. Then we presented a one-step smoothing Newton algorithm to solve the MCP with a P₀ function. The global convergence of the proposed algorithm is verified under mild conditions. And by using the smooth and semismooth technique, the rate of convergence of the method is proved under some suitable assumptions.     

Predictor-Corrector Smoothing Newton Method,Based on a New Smoothing Function,for Solving the Nonlinear Complementarity Problem with a P 0 Function

By smoothing a perturbed minimum function, we propose in this paper a new smoothing function. The existence and continuity of a smooth path for solving the nonlinear complementarity problem (NCP) with a P ₀ function are discussed. We investigate the boundedness of the iteration sequence generated by noninterior continuation/smoothing methods under the assumption that the solution set of the NCP is nonempty and bounded. Based on the new smoothing function, we present a predictor-corrector smoothing Newton algorithm for solving the NCP with a P ₀ function, which is shown to be globally linearly and locally superlinearly convergent under suitable assumptions. Some preliminary computational results are reported.     

A Geometric Approach to Maximum Likelihood Estimation of the Functional Principal Components From Sparse Longitudinal Data

In this article, we consider the problem of estimating the eigenvalues and eigenfunctions of the covariance kernel (i.e., the functional principal components) from sparse and irregularly observed longitudinal data. We exploit the smoothness of the eigenfunctions to reduce dimensionality by restricting them to a lower dimensional space of smooth functions. We then approach this problem through a restricted maximum likelihood method. The estimation scheme is based on a Newton–Raphson procedure on the Stiefel manifold using the fact that the basis coefficient matrix for representing the eigenfunctions has orthonormal columns. We also address the selection of the number of basis functions, as well as that of the dimension of the covariance kernel by a second-order approximation to the leave-one-curve-out cross-validation score that is computationally very efficient. The effectiveness of our procedure is demonstrated by simulation studies and an application to a CD4+ counts dataset. In the simulation studies, our method performs well on both estimation and model selection. It also outperforms two existing approaches: one based on a local polynomial smoothing, and another using an EM algorithm. Supplementary materials including technical details, the R package fpca, and data analyzed by this article are available online.     

Iterative refinement for constrained and weighted linear least squares

We present an algorithm for mixed precision iterative refinement on the constrained and weighted linear least squares problem, the CWLSQ problem. The approximate solution is obtained by solving the CWLSQ problem with the weightedQR factorization [6]. With backward errors for the weightedQR decomposition together with perturbation bounds for the CWLSQ problem we analyze the convergence behaviour of the iterative refinement procedure.In the unweighted case the initial convergence rate of the error of the iteratively refined solution is determined essentially by the condition number. For the CWLSQ problem the initial convergence behaviour is more complicated. The analysis shows that the initial convergence is dependent both on the condition of the problem related to the solution,x, and the vector =Wr, whereW is the weight matrix andr is the residual.We test our algorithm on two examples where the solution is known and the condition number of the problem can be varied. The computational test confirms the theoretical results and verifies that mixed precision iterative refinement, using the system matrix and the weightedQR decomposition, is an effective way of improving an approximate solution to the CWLSQ problem.     

Reliable parallel solution of bidiagonal systems

Summary. This paper presents a new efficient algorithm for solving bidiagonal systems of linear equations on massively parallel machines. We use a divide and conquer approach to compute a representative subset of the solution components after which we solve the complete system in parallel with no communication overhead. We address the numerical properties of the algorithm in two ways: we show how to verify the à posteriori backward stability at virtually no additional cost, and prove that the algorithm is à priori forward stable. We then show how we can use the algorithm in order to bound the possible perturbations in the solution components. Received March 13, 1998 / Revised version received December 21, 1999 / Published online June 20, 2001     

Homotopy method for solving ball-constrained variational inequalities

In this paper, we present a smoothing homotopy method for solving ball-constrained variational inequalities by utilizing a similar Chen-Harker-Kanzow-Smale function to smooth Robinson’s normal equation. Without any monotonicity condition on the defining map F, for the starting point chosen almost everywhere in Rⁿ, the existence and convergence of the homotopy pathway are proven. Numerical experiments illustrate that the method is feasible and effective.     

A mixed-step algorithm for the approximation of the stationary regime of a diffusion

In some recent papers, some procedures based on some weighted empirical measures related to decreasing-step Euler schemes have been investigated to approximate the stationary regime of a diffusion (possibly with jumps) for a class of functionals of the process. This method is efficient but needs the computation of the function at each step. To reduce the complexity of the procedure (especially for functionals), we propose in this paper to study a new scheme, called the mixed-step scheme, where we only keep some regularly time-spaced values of the Euler scheme. Our main result is that, when the coefficients of the diffusion are smooth enough, this alternative does not change the order of the rate of convergence of the procedure. We also investigate a Richardson–Romberg method to speed up the convergence and show that the variance of the original algorithm can be preserved under a uniqueness assumption for the invariant distribution of the “duplicated” diffusion, condition which is extensively discussed in the paper. Finally, we conclude by giving sufficient “asymptotic confluence” conditions for the existence of a smooth solution to a discrete version of the associated Poisson equation, condition which is required to ensure the rate of convergence results.     

A full-Newton step non-interior continuation algorithm for a class of complementarity problems

In this paper, we investigate a class of nonlinear complementarity problems arising from the discretization of the free boundary problem, which was recently studied by Sun and Zeng [Z. Sun, J. Zeng, A monotone semismooth Newton type method for a class of complementarity problems, J. Comput. Appl. Math. 235 (5) (2011) 1261–1274]. We propose a new non-interior continuation algorithm for solving this class of problems, where the full-Newton step is used in each iteration. We show that the algorithm is globally convergent, where the iteration sequence of the variable converges monotonically. We also prove that the algorithm is globally linearly and locally superlinearly convergent without any additional assumption, and locally quadratically convergent under suitable assumptions. The preliminary numerical results demonstrate the effectiveness of the proposed algorithm.     

A Functional EM Algorithm for Mixing Density Estimation via Nonparametric Penalized Likelihood Maximization

When the true mixing density is known to be continuous, the maximum likelihood estimate of the mixing density does not provide a satisfying answer due to its degeneracy. Estimation of mixing densities is a well-known ill-posed indirect problem. In this article, we propose to estimate the mixing density by maximizing a penalized likelihood and call the resulting estimate the nonparametric maximum penalized likelihood estimate (NPMPLE). Using theory and methods from the calculus of variations and differential equations, a new functional EM algorithm is derived for computing the NPMPLE of the mixing density. In the algorithm, maximizers in M-steps are found by solving an ordinary differential equation with boundary conditions numerically. Simulation studies show the algorithm outperforms other existing methods such as the popular EMS algorithm. Some theoretical properties of the NPMPLE and the algorithm are also discussed. Computer code used in this article is available online.     

Iterative refinement enhances the stability ofQR factorization methods for solving linear equations

Iterative refinement is a well-known technique for improving the quality of an approximate solution to a linear system. In the traditional usage residuals are computed in extended precision, but more recent work has shown that fixed precision is sufficient to yield benefits for stability. We extend existing results to show that fixed precision iterative refinement renders anarbitrary linear equations solver backward stable in a strong, componentwise sense, under suitable assumptions. Two particular applications involving theQR factorization are discussed in detail: solution of square linear systems and solution of least squares problems. In the former case we show that one step of iterative refinement suffices to produce a small componentwise relative backward error. Our results are weaker for the least squares problem, but again we find that iterative refinement improves a componentwise measure of backward stability. In particular, iterative refinement mitigates the effect of poor row scaling of the coefficient matrix, and so provides an alternative to the use of row interchanges in the HouseholderQR factorization. A further application of the results is described to fast methods for solving Vandermonde-like systems.     

The EM algorithm for ML Estimators under nonlinear inequalities restrictions on the parameters

One of the most powerful algorithms for obtaining maximum likelihood estimates for many incomplete-data problems is the EM algorithm. However, when the parameters satisfy a set of nonlinear restrictions, It is difficult to apply the EM algorithm directly. In this paper,we propose an asymptotic maximum likelihood estimation procedure under a set of nonlinear inequalities restrictions on the parameters, in which the EM algorithm can be used. Essentially this kind of estimation problem is a stochastic optimization problem in the M-step. We make use of methods in stochastic optimization to overcome the difficulty caused by nonlinearity in the given constraints.     

An interior point multiplicative method for optimization under positivity constraints

We analyze an algorithm for the problem minf(x) s.t.x 0 suggested, without convergence proof, by Eggermont. The iterative step is given by x _j ^k+1 =x _j ^k (1-_kf(x ^k)_j) with _k > 0 determined through a line search. This method can be seen as a natural extension of the steepest descent method for unconstrained optimization, and we establish convergence properties similar to those known for steepest descent, namely weak convergence to a KKT point for a generalf, weak convergence to a solution for convexf and full convergence to the solution for strictly convexf. Applying this method to a maximum likelihood estimation problem, we obtain an additively overrelaxed version of the EM Algorithm. We extend the full convergence results known for EM to this overrelaxed version by establishing local Fejér monotonicity to the solution set.Research for this paper was partially supported by CNPq grant No 301280/86.     

Efficient fourth orderP-stable formulae

In this paper, a family of fourth orderP-stable methods for solving second order initial value problems is considered. When applied to a nonlinear differential system, all the methods in the family give rise to a nonlinear system which may be solved using a modified Newton method. The classical methods of this type involve at least three (new) function evaluations per iteration (that is, they are 3-stage methods) and most involve using complex arithmetic in factorising their iteration matrix. We derive methods which require only two (new) function evaluations per iteration and for which the iteration matrix is a true real perfect square. This implies that real arithmetic will be used and that at most one real matrix must be factorised at each step. Also we consider various computational aspects such as local error estimation and a strategy for changing the step size.