Residual Smoothing Techniques: Do They Improve the Limiting Accuracy of Iterative Solvers?

Many iterative methods for solving linear systems, in particular the biconjugate gradient (BiCG) method and its squared version CGS (or BiCGS), produce often residuals whose norms decrease far from monotonously, but fluctuate rather strongly. Large intermediate residuals are known to reduce the ultimately attainable accuracy of the method, unless special measures are taken to counteract this effect. One measure that has been suggested is residual smoothing: by application of simple recurrences, the iterates x _n and the corresponding residuals r _n : b – Ax _n are replaced by smoothed iterates y _n and corresponding residuals s _n : b– Ay _n. We address the question whether the smoothed residuals can ultimately become markedly smaller than the primary ones. To investigate this, we present a roundoff error analysis of the smoothing algorithms. It shows that the ultimately attainable accuracy of the smoothed iterates, measured in the norm of the corresponding residuals, is, in general, not higher than that of the primary iterates. Nevertheless, smoothing can be used to produce certain residuals, most notably those of the minimum residual method, with higher attainable accuracy than by other frequently used algorithms.     

Confidence Intervals for Discrete Approximations to Ill-Posed Problems

We consider the linear model Y = Xβ + ε that is obtained by discretizing a system of first-kind integral equations describing a set of physical measurements. The n vector β represents the desired quantities, the m x n matrix X represents the instrument response functions, and the m vector Y contains the measurements actually obtained. These measurements are corrupted by random measuring errors ε drawn from a distribution with zero mean vector and known variance matrix. Solution of first-kind integral equations is an ill-posed problem, so the least squares solution for the above model is a highly unstable function of the measurements, and the classical confidence intervals for the solution are too wide to be useful. The solution can often be stabilized by imposing physically motivated nonnegativity constraints. In a previous article (O'Leary and Rust 1986) we developed a method for computing sets of nonnegatively constrained simultaneous confidence intervals. In this article we briefly review the simultaneous intervals and then show how to compute nonnegativity constrained one-at-a-time confidence intervals. The technique gives valid confidence intervals even for problems with m < n. We demonstrate the methods using both an overdetermined and an underdetermined problem obtained by discretizing an equation of Phillips (Phillips 1962).     

On ecological aspects of dynamics for zero slope regression for water pollution in Chile

Zero slope regression is an important problem in chemometrics, ranging from challenges of intercept-bias and slope ‘corrections’ in spectrometry, up to analysis of administrative data on chemical pollution in water in the region of Arica and Parinacota. Such issue is really complex and it integrates problems of optimal design, symmetry of errors, stabilization of the variability of estimators, dynamical system for errors up to an administrative data challenges. In this article we introduce a realistic approach to zero slope regression problem from dynamical point of view. Linear regression is a widely used approach for data fitting under assumption of normally distributed residuals. Many times non-normal residuals are observed and also theoretically justified. Our solution to such problem uses the recently introduced inference function called score function of distribution. As a minimization criterion, the minimum information of residuals criterion is used. The score regression appears to be a direct generalization of the least-squares regression for an arbitrary known (believed) distribution of residuals. The score estimation is also distribution sensitive version of M-estimation. The capability of the method is demonstrated by water pollution data examples.     

Normal/Independent Distributions and Their Applications in Robust Regression

Abstract

Maximum likelihood estimation with nonnormal error distributions provides one method of robust regression. Certain families of normal/independent distributions are particularly attractive for adaptive, robust regression. This article reviews the properties of normal/independent distributions and presents several new results. A major virtue of these distributions is that they lend themselves to EM algorithms for maximum likelihood estimation. EM algorithms are discussed for least L_p regression and for adaptive, robust regression based on the t, slash, and contaminated normal families. Four concrete examples illustrate the performance of the different methods on real data.     

A Unified Approach to Second Order Optimality Criteria in Nonlinear Design of Experiments

In the nonlinear regression model we consider the optimal design problem with a second order design D-criterion. Our purpose is to present a general approach to this problem, which includes the asymptotic second order bias and variance criterion of the least squares estimator and criteria using the volume of confidence regions based on different statistics. Under assumptions of regularity for these statistics a second order approximation of the volume of these regions is derived which is proposed as a quadratic optimality criterion. These criteria include volumes of confidence regions based on the u _n - representable statistics. An important difference between the criteria presented in this paper and the second order criteria commonly employed in the recent literature is that the former criteria are independent of the vector of residuals. Moreover, a refined version of the commonly applied criteria is obtained, which also includes effects of nonlinearity caused by third derivatives of the response function.     

MODEL SELECTION METHOD BASED ON MAXIMAL INFORMATION COEFFICIENT OF RESIDUALS

The traditional model selection criterions try to make a balance between fitted error and model complexity. Assumptions on the distribution of the response or the noise, which may be misspecified, should be made before using the traditional ones. In this ar- ticle, we give a new model selection criterion, based on the assumption that noise term in the model is independent with explanatory variables, of minimizing the association strength between regression residuals and the response, with fewer assumptions. Maximal Information Coe~cient （MIC）, a recently proposed dependence measure, captures a wide range of associ- ations, and gives almost the same score to different type of relationships with equal noise, so MIC is used to measure the association strength. Furthermore, partial maximal information coefficient （PMIC） is introduced to capture the association between two variables removing a third controlling random variable. In addition, the definition of general partial relationship is given.     

Thank God That Regressing Y on X is Not the Same as Regressing X on Y: Direct and Indirect Residual Augmentations

What does regressing Y on X versus regressing X on Y have to do with Markov chain Monte Carlo (MCMC)? It turns out that many strategies for speeding up data augmentation (DA) type algorithms can be understood as fostering independence or “de-correlation” between a regression function and the corresponding residual, thereby reducing or even eliminating dependence among MCMC iterates. There are two general classes of algorithms, those corresponding to regressing parameters on augmented data/auxiliary variables and those that operate the other way around. The interweaving strategy of Yu and Meng provides a general recipe to automatically take advantage of both, and it is the existence of two different types of residuals that makes the interweaving strategy seemingly magical in some cases and promising in general. The concept of residuals—which depends on actual data—also highlights the potential for substantial improvements when DA schemes are allowed to depend on the observed data. At the same time, there is an intriguing phase transition type of phenomenon regarding choosing (partially) residual augmentation schemes, reminding us once more of the prevailing issue of trade-off between robustness and efficiency. This article reports on these latest theoretical investigations (using a class of normal/independence models) and empirical findings (using a posterior sampling for a probit regression) in the search for effective residual augmentations—and ultimately more MCMC algorithms—that meet the 3-S criterion: simple, stable, and speedy. Supplementary materials for the article are available online.     

Model selection method based on maximal information coefficient of residuals

The traditional model selection criterions try to make a balance between fitted error and model complexity. Assumptions on the distribution of the response or the noise, which may be misspecified, should be made before using the traditional ones. In this article, we give a new model selection criterion, based on the assumption that noise term in the model is independent with explanatory variables, of minimizing the association strength between regression residuals and the response, with fewer assumptions. Maximal Information Coefficient (MIC), a recently proposed dependence measure, captures a wide range of associations, and gives almost the same score to different type of relationships with equal noise, so MIC is used to measure the association strength. Furthermore, partial maximal information coefficient (PMIC) is introduced to capture the association between two variables removing a third controlling random variable. In addition, the definition of general partial relationship is given.     

Weak consistency of the Support Vector Machine Quantile Regression approach when covariates are functions

This paper deals with a nonparametric estimation of conditional quantile regression when the explanatory variable X takes its values in a bounded subspace of a functional space X and the response Y takes its values in a compact of the space Y?R. The functional observations, X₁,…,X_n, are projected onto a finite dimensional subspace having a suitable orthonormal system. The X_i’s will be characterized by their coordinates in this basis. We perform the Support Vector Machine Quantile Regression approach in finite dimension with the selected coefficients. Then we establish weak consistency of this estimator. The various parameters needed for the construction of this estimator are automatically selected by data-splitting and by penalized empirical risk minimization.     

随机缺失数据下样本分位数估计

分位数的估计在生物医学、社会经济调查等领域有着广泛的应用,然而在实际问题的研究中,往往由于各种人为或不可控因素造成数据收集不完全.本文在随机缺失(MAR)假设条件下,利用非参数核补法和局部多重插补法给出了响应变量缺失时样本分位数的估计,并利用经验过程等理论证明了由这两种方法得到的分位数估计的大样本性质,同时,使用重抽样方法给出了估计的渐近方差的估计,模拟结果验证了这两种方法的有效性.文章所提两种方法的优点在于:首先,所提出的缺失修正方法不需要对缺失概率的模型做任何假设;其次,方法亦适用于其他有关参数不可微的估计目标函数;最后,方法很容易地推广到一般M估计的情况,并可以对多个分位数同时进行估计.     

Control charts based on randomized quantile residuals

In practice, quality characteristics do not always follow a normal distribution, and quality control processes sometimes generate non‐normal response outcomes, including continuous non‐normal data and discrete count data. Thus, achieving better results in such situations requires a new control chart derived from various types of response variables. This study proposes a procedure for monitoring response variables that uses control charts based on randomized quantile residuals obtained from a fitted regression model. Simulation studies demonstrate the performance of the proposed control charts under various situations. We illustrate the procedure using two real‐data examples, based on normal and negative binomial regression models, respectively. The simulation and real‐data results support our proposed procedure.     

The Latent Scale Covariogram: A Tool for Exploring the Spatial Dependence Structure of Nonnormal Responses

When modeling spatially distributed normal responses Y_i in terms of vectors x_i of explanatory variables, one may fit a linear model assuming independence, and then use the empirical variogram of the residuals to determine an appropriate parametric form for the autocorrelation function. Suppose, however, that the responses are not normally distributed—for example, Poisson or Bernoulli. One may model spatial dependence using a hierarchical generalized linear model in which, conditional on a latent Gaussian field Z = {Z_i}, the Y_i have independent distributions from the exponential family, with an appropriate link function connecting their conditional means with the linear predictors x^t_iβ + Z_i. The question then is how to determine an appropriate model for the autocorrelation function of Z. The empirical variogram of the Y_i is no longer appropriate, since (unless the link function is the identity) it is on the wrong scale. We propose here an alternative, the latent scale covariogram, whose graph reflects the autocorrelation structure of the underlying normal field. We illustrate its use on several real datasets, together with a simulated dataset, and obtain results quite different from those obtained using the variogram. Supplementary materials for this article are available online.     

Fast Algorithms for Nonparametric Curve Estimation

Abstract

Naive implementations of local polynomial fits and kernel estimators require almost O(n ²) operations. In this article two fast O(n) algorithms for nonparametric local polynomial fitting are presented. They are based on updating normal equations. Numerical stability is guaranteed by controlling ill-conditioned situations for small bandwidths and data-tuned restarting of the updating procedure. Restarting at every output point results in a moderately fast but highly stable O(n ^7/5) algorithm. Applicability of algorithms is evaluated for estimation of regression curves and their derivatives. The idea is also applied to kernel estimators of regression curves and densities.     

On the Homology and Fiber Cone of Ideals

In this article, we give a unified approach for several results concerning the fiber cone. Our novel idea is to use the complex C(x _k , ? _{I 1; I 2} , (1, n)). We improve earlier results obtained by several researchers and get some new results. We give a more general definition of ideals of minimal multiplicity and of ideals of almost minimal multiplicity. We also compute the Hilbert series of the fiber cone for these ideals.     

A posteriori error estimates for H 1-Galerkin mixed finite-element method for parabolic problems

The purpose of this article is to derive a posteriori error estimates for the H ¹-Galerkin mixed finite element method for parabolic problems. We study both semidiscrete and fully discrete a posteriori error analyses using standard energy argument. A fully discrete a posteriori error analysis based on the backward Euler method is analysed and upper bounds for the errors are derived. The estimators yield upper bounds for the errors which are global in space and time. Our analysis is based on residual approach and the estimators are free from edge residuals.     

Estimation of central shapes of error distributions in linear regression problems

Consider a linear regression model subject to an error distribution which is symmetric about 0 and varies regularly at 0 with exponent ζ. We propose two estimators of ζ, which characterizes the central shape of the error distribution. Both methods are motivated by the well-known Hill estimator, which has been extensively studied in the related problem of estimating tail indices, but substitute reciprocals of small L _p residuals for the extreme order statistics in its original definition. The first method requires careful choices of p and the number k of smallest residuals employed for calculating the estimator. The second method is based on subsampling and works under less restrictive conditions on p and k. Both estimators are shown to be consistent for ζ and asymptotically normal. A simulation study is conducted to compare our proposed procedures with alternative estimates of ζ constructed using resampling methods designed for convergence rate estimation.     

Penalized Nonparametric Scalar-on-Function Regression via Principal Coordinates

A number of classical approaches to nonparametric regression have recently been extended to the case of functional predictors. This article introduces a new method of this type, which extends intermediate-rank penalized smoothing to scalar-on-function regression. In the proposed method, which we call principal coordinate ridge regression, one regresses the response on leading principal coordinates defined by a relevant distance among the functional predictors, while applying a ridge penalty. Our publicly available implementation, based on generalized additive modeling software, allows for fast optimal tuning parameter selection and for extensions to multiple functional predictors, exponential family-valued responses, and mixed-effects models. In an application to signature verification data, principal coordinate ridge regression, with dynamic time warping distance used to define the principal coordinates, is shown to outperform a functional generalized linear model. Supplementary materials for this article are available online.     

Hazard Rate Regression Using Ordinary Nonparametric Regression Smoothers

Abstract

This article proposes a method for nonparametric estimation of hazard rates as a function of time and possibly multiple covariates. The method is based on dividing the time axis into intervals, and calculating number of event and follow-up time contributions from the different intervals. The number of event and follow-up time data are then separately smoothed on time and the covariates, and the hazard rate estimators obtained by taking the ratio. Pointwise consistency and asymptotic normality are shown for the hazard rate estimators for a certain class of smoothers, which includes some standard approaches to locally weighted regression and kernel regression. It is shown through simulation that a variance estimator based on this asymptotic distribution is reasonably reliable in practice. The problem of how to select the smoothing parameter is considered, but a satisfactory resolution to this problem has not been identified. The method is illustrated using data from several breast cancer clinical trials.     

Delete-group Jackknife Estimate in Partially Linear Regression Models with Heteroscedasticity

Abstract Consider a partially linear regression model with an unknown vector parameter β,an unknownfunction g(.),and unknown heteroscedastic error variances.Chen,You proposed a semiparametric generalizedleast squares estimator(SGLSE)for β,which takes the heteroscedasticity into account to increase efficiency.Forinference based on this SGLSE,it is necessary to construct a consistent estimator for its asymptotic covariancematrix.However,when there exists within-group correlation, the traditional delta method and the delete-1jackknife estimation fail to offer such a consistent estimator.In this paper, by deleting grouped partial residualsa delete-group jackknife method is examined.It is shown that the delete-group jackknife method indeed canprovide a consistent estimator for the asymptotic covariance matrix in the presence of within-group correlations.This result is an extension of that in[21].