Lower Rank Approximation of Matrices Based on Fast and Robust Alternating Regression

We introduce fast and robust algorithms for lower rank approximation to given matrices based on robust alternating regression. The alternating least squares regression, also called criss-cross regression, was used for lower rank approximation of matrices, but it lacks robustness against outliers in these matrices. We use robust regression estimators and address some of the complications arising from this approach. We find it helpful to use high breakdown estimators in the initial iterations, followed by M estimators with monotone score functions in later iterations towards convergence. In addition to robustness, the computational speed is another important consideration in the development of our proposed algorithm, because alternating robust regression can be computationally intensive for large matrices. Based on a mix of the least trimmed squares (LTS) and Huber's M estimators, we demonstrate that fast and robust lower rank approximations are possible for modestly large matrices.     

CP decomposition for tensors via alternating least squares with QR decomposition

The CP tensor decomposition is used in applications such as machine learning and signal processing to discover latent low-rank structure in multidimensional data. Computing a CP decomposition via an alternating least squares (ALS) method reduces the problem to several linear least squares problems. The standard way to solve these linear least squares subproblems is to use the normal equations, which inherit special tensor structure that can be exploited for computational efficiency. However, the normal equations are sensitive to numerical ill-conditioning, which can compromise the results of the decomposition. In this paper, we develop versions of the CP-ALS algorithm using the QR decomposition and the singular value decomposition, which are more numerically stable than the normal equations, to solve the linear least squares problems. Our algorithms utilize the tensor structure of the CP-ALS subproblems efficiently, have the same complexity as the standard CP-ALS algorithm when the input is dense and the rank is small, and are shown via examples to produce more stable results when ill-conditioning is present. Our MATLAB implementation achieves the same running time as the standard algorithm for small ranks, and we show that the new methods can obtain lower approximation error.     

Derivative free analogues of the Levenberg-Marquardt and Gauss algorithms for nonlinear least squares approximation

In this paper we give two derivative-free computational algorithms for nonlinear least squares approximation. The algorithms are finite difference analogues of the Levenberg-Marquardt and Gauss methods. Local convergence theorems for the algorithms are proven. In the special case when the residuals are zero at the minimum, we show that certain computationally simple choices of the parameters lead to quadratic convergence. Numerical examples are included.On leave 1970–71 at Yale UniversityThe work of this author was supported in part by the National Science Foundation under Grant GJ-844.     

On choosing “optimal” shape parameters for RBF approximation

Many radial basis function (RBF) methods contain a free shape parameter that plays an important role for the accuracy of the method. In most papers the authors end up choosing this shape parameter by trial and error or some other ad hoc means. The method of cross validation has long been used in the statistics literature, and the special case of leave-one-out cross validation forms the basis of the algorithm for choosing an optimal value of the shape parameter proposed by Rippa in the setting of scattered data interpolation with RBFs. We discuss extensions of this approach that can be applied in the setting of iterated approximate moving least squares approximation of function value data and for RBF pseudo-spectral methods for the solution of partial differential equations. The former method can be viewed as an efficient alternative to ridge regression or smoothing spline approximation, while the latter forms an extension of the classical polynomial pseudo-spectral approach. Numerical experiments illustrating the use of our algorithms are included.     

Recursive prediction of chaotic time series

Summary Considerable progress has been made in recent years in the analysis of time series arising from chaotic systems. In particular, a variety of schemes for the short-term prediction of such time series has been developed. However, hitherto all such algorithms have used batch processing and have not been able to continuously update their estimate of the dynamics using new observations as they are made. This severely limits their usefulness in real time signal processing applications. In this paper we present a continuous update prediction scheme for chaotic time series that overcomes this difficulty. It is based on radial basis function approximation combined with a recursive least squares estimation algorithm. We test this scheme using simulated data and comment on its relationship to adaptive transversal filters, which are widely used in conventional signal processing.     

Weighted least squares solutions to general coupled Sylvester matrix equations

This paper is concerned with weighted least squares solutions to general coupled Sylvester matrix equations. Gradient based iterative algorithms are proposed to solve this problem. This type of iterative algorithm includes a wide class of iterative algorithms, and two special cases of them are studied in detail in this paper. Necessary and sufficient conditions guaranteeing the convergence of the proposed algorithms are presented. Sufficient conditions that are easy to compute are also given. The optimal step sizes such that the convergence rates of the algorithms, which are properly defined in this paper, are maximized and established. Several special cases of the weighted least squares problem, such as a least squares solution to the coupled Sylvester matrix equations problem, solutions to the general coupled Sylvester matrix equations problem, and a weighted least squares solution to the linear matrix equation problem are simultaneously solved. Several numerical examples are given to illustrate the effectiveness of the proposed algorithms.     

Extreme logistic regression

Kernel logistic regression (KLR) is a very powerful algorithm that has been shown to be very competitive with many state-of the art machine learning algorithms such as support vector machines (SVM). Unlike SVM, KLR can be easily extended to multi-class problems and produces class posterior probability estimates making it very useful for many real world applications. However, the training of KLR using gradient based methods or iterative re-weighted least squares can be unbearably slow for large datasets. Coupled with poor conditioning and parameter tuning, training KLR can quickly design matrix become infeasible for some real datasets. The goal of this paper is to present simple, fast, scalable, and efficient algorithms for learning KLR. First, based on a simple approximation of the logistic function, a least square algorithm for KLR is derived that avoids the iterative tuning of gradient based methods. Second, inspired by the extreme learning machine (ELM) theory, an explicit feature space is constructed through a generalized single hidden layer feedforward network and used for training iterative re-weighted least squares KLR (IRLS-KLR) and the newly proposed least squares KLR (LS-KLR). Finally, for large-scale and/or poorly conditioned problems, a robust and efficient preconditioned learning technique is proposed for learning the algorithms presented in the paper. Numerical results on a series of artificial and 12 real bench-mark datasets show first that LS-KLR compares favorable with SVM and traditional IRLS-KLR in terms of accuracy and learning speed. Second, the extension of ELM to KLR results in simple, scalable and very fast algorithms with comparable generalization performance to their original versions. Finally, the introduced preconditioned learning method can significantly increase the learning speed of IRLS-KLR.     

A Dynamic Programming Algorithm for the Fused Lasso and L 0-Segmentation

We propose a dynamic programming algorithm for the one-dimensional Fused Lasso Signal Approximator (FLSA). The proposed algorithm has a linear running time in the worst case. A similar approach is developed for the task of least squares segmentation, and simulations indicate substantial performance improvement over existing algorithms. Examples of R and C implementations are provided in the online Supplementary materials, posted on the journal web site.     

非线性最小二乘法的算法

本给出非线性最小二乘的优化条件和几何特征．     

Randomized algorithms for total least squares problems

Motivated by the recently popular probabilistic methods for low‐rank approximations and randomized algorithms for the least squares problems, we develop randomized algorithms for the total least squares problem with a single right‐hand side. We present the Nyström method for the medium‐sized problems. For the large‐scale and ill‐conditioned cases, we introduce the randomized truncated total least squares with the known or estimated rank as the regularization parameter. We analyze the accuracy of the algorithm randomized truncated total least squares and perform numerical experiments to demonstrate the efficiency of our randomized algorithms. The randomized algorithms can greatly reduce the computational time and still maintain good accuracy with very high probability.     

A genetic algorithm approach to nonlinear least squares estimation

A common type of problem encountered in mathematics is optimizing nonlinear functions. Many popular algorithms that are currently available for finding nonlinear least squares estimators, a special class of nonlinear problems, are sometimes inadequate. They might not converge to an optimal value, or if they do, it could be to a local rather than global optimum. Genetic algorithms have been applied successfully to function optimization and therefore would be effective for nonlinear least squares estimation. This paper provides an illustration of a genetic algorithm applied to a simple nonlinear least squares example.     

基于偏最小二乘变换与Kriging模型的全局优化方法

针对传统Kriging模型在多变量(高维)输入全局优化中因超参数过多而引发收敛速度慢,精度低,建模效率不高问题,提出了基于偏最小二乘变换技术和Kriging模型的有效全局优化方法.首先,构造偏最小二乘高斯核函数;其次,借助差分进化算法寻找满足期望改进准则最大化条件的新样本点;然后,将不同核函数和期望改进准则组合,构建四种有效全局优化算法并进行比较;最后,数值算例结果表明,基于偏最小二乘变换的Kriging全局优化方法在解决高维全局优化问题方面相比于标准的全局优化算法在收敛精度及收敛速度方面更具优势.     

Toeplitz矩阵填充的四种流形逼近算法比较

 本文提出Toeplitz矩阵填充的四种流形逼近算法。在左奇异向量空间中对已知部分运用最小二乘法逼近,形成新的可行矩阵;并将对角线上的元素分别用均值,l₁范数,l_∞范数和中间数四种方法逼近使得迭代后的矩阵仍保持Toeplitz结构,节约了奇异向量空间的分解时间。最终找到合理的低秩矩阵来逼近未知的高秩矩阵,进而精确地完成Toeplitz矩阵的填充。理论上,分析了在一定条件下算法的收敛性。实验上,通过取不同的采样密度进行数值实验展示了四种算法的优劣。实验结果说明均值算法和l_∞范数算法大多用的时间较少,但是当采样密度和矩阵规模较大时,中间数算法的精度较高。     

On the ill conditioning of locating transmission zeros in least squares ARMA filtering

The algorithms of Levinson-Schur and Nevanlinna-Pick are briefly reviewed. Both produce least squares predictive filters. By minimizing the least squares error with respect to the interpolation points of the Nevanlinna-Pick algorithm we find the transmission zeros of an ARMA filter. It is shown by some simple examples that this is an ill conditioned problem.     

Data-driven recursive least squares methods for non-affined nonlinear discrete-time systems

Aiming at identifying nonlinear systems, one of the most challenging problems in system identification, a class of data-driven recursive least squares algorithms are presented in this work. First, a full form dynamic linearization based linear data model for nonlinear systems is derived. Consequently, a full form dynamic linearization-based data-driven recursive least squares identification method for estimating the unknown parameter of the obtained linear data model is proposed along with convergence analysis and prediction of the outputs subject to stochastic noises. Furthermore, a partial form dynamic linearization-based data-driven recursive least squares identification algorithm is also developed as a special case of the full form dynamic linearization based algorithm. The proposed two identification algorithms for the nonlinear nonaffine discrete-time systems are flexible in applications without relying on any explicit mechanism model information of the systems. Additionally, the number of the parameters in the obtained linear data model can be tuned flexibly to reduce computation complexity. The validity of the two identification algorithms is verified by rigorous theoretical analysis and simulation studies.     

Two-stage least squares and indirect least squares algorithms for simultaneous equations models

This paper analyzes the solution of simultaneous equations models. Efficient algorithms for the two-stage least squares method using QR-decomposition are developed and studied. The reduction of the execution time when the structure of the matrices in each equation is exploited is analyzed theoretically and experimentally. An efficient algorithm for the indirect least squares method is developed. Some techniques are used to accelerate the solution of the problem: parallel versions for multicore systems, and extensive use of the MKL library, thus obtaining efficient, portable versions of the algorithms.     

Meshless Finite Difference Operators from Moving Least Squares Interpolation: Applications to PDEs and Convergence Results

We give a brief introduction to moving least squares interpolation, which is followed by some reflections on the construction of meshless finite difference operators for derivative approximation and present finally a Korovkin–type convergence result for the pure interpolation approach. (© 2008 WILEY-VCH Verlag GmbH & Co. KGaA, Weinheim)     

A Sequential Least Squares Method for Elliptic Equations in Non-Divergence Form

We develop a new least squares method for solving the second-order elliptic equations in non-divergence form. Two least-squares-type functionals are proposed for solving the equation in two sequential steps. We first obtain a numerical approximation to the gradient in a piecewise irrotational polynomial space. Then together with the numerical gradient, we seek a numerical solution of the primitive variable in the continuous Lagrange finite element space. The variational setting naturally provides an a posteriori error which can be used in an adaptive refinement algorithm. The error estimates under the $L^2$ norm and the energy norm for both two unknowns are derived. By a series of numerical experiments, we verify the convergence rates and show the efficiency of the adaptive algorithm.     

Extensions of system signatures to dependent lifetimes: Explicit expressions and interpretations

The concept of system signature was introduced by Samaniego for systems whose components have i.i.d. lifetimes. We consider its extension to the continuous dependent case and give an explicit expression for this extension as a difference of weighted means of the structure function values. We then derive a formula for the computation of the coefficients of these weighted means in the special case of independent continuous lifetimes. Finally, we interpret this extended concept of signature through a natural least squares approximation problem.     

Recursive computational formulas of the least squares criterion functions for scalar system identification

The paper discusses recursive computation problems of the criterion functions of several least squares type parameter estimation methods for linear regression models, including the well-known recursive least squares (RLS) algorithm, the weighted RLS algorithm, the forgetting factor RLS algorithm and the finite-data-window RLS algorithm without or with a forgetting factor. The recursive computation formulas of the criterion functions are derived by using the recursive parameter estimation equations. The proposed recursive computation formulas can be extended to the estimation algorithms of the pseudo-linear regression models for equation error systems and output error systems. Finally, the simulation example is provided.