The Convergence of a Levenberg–Marquardt Method for Nonlinear Inequalities

In this paper, we consider the least l ₂-norm solution for a possibly inconsistent system of nonlinear inequalities. The objective function of the problem is only first-order continuously differentiable. By introducing a new smoothing function, the problem is approximated by a family of parameterized optimization problems with twice continuously differentiable objective functions. Then a Levenberg–Marquardt algorithm is proposed to solve the parameterized smooth optimization problems. It is proved that the algorithm either terminates finitely at a solution of the original inequality problem or generates an infinite sequence. In the latter case, the infinite sequence converges to a least l ₂-norm solution of the inequality problem. The local quadratic convergence of the algorithm was produced under some conditions.     

On Adaptive Combination of Regression Estimators

Consider the problem of choosing between two estimators of the regression function, where one estimator is based on stronger assumptions than the other and thus the rates of convergence are different. We propose a linear combination of the estimators where the weights are estimated by Mallows' C _L . The adaptive estimator retains the optimal rates of convergence and is an extension of Stein-type estimators considered by Li and Hwang (1984, Ann. Statist., 12, 887-897) and related to an estimator in Burman and Chaudhuri (1999, Ann. Inst. Statist. Math. (to appear)).     

Adaptive Semiparametric Estimation of the Memory Parameter

In Giraitis, Robinson, and Samarov (1997), we have shown that the optimal rate for memory parameter estimators in semiparametric long memory models with degree of “local smoothness” β is n^−r(β), r(β)=β/(2β+1), and that a log-periodogram regression estimator (a modified Geweke and Porter-Hudak (1983) estimator) with maximum frequency m=m(β)n^2r(β) is rate optimal. The question which we address in this paper is what is the best obtainable rate when β is unknown, so that estimators cannot depend on β. We obtain a lower bound for the asymptotic quadratic risk of any such adaptive estimator, which turns out to be larger than the optimal nonadaptive rate n^−r(β) by a logarithmic factor. We then consider a modified log-periodogram regression estimator based on tapered data and with a data-dependent maximum frequency m=m(β), which depends on an adaptively chosen estimator β of β, and show, using methods proposed by Lepskii (1990) in another context, that this estimator attains the lower bound up to a logarithmic factor. On one hand, this means that this estimator has nearly optimal rate among all adaptive (free from β) estimators, and, on the other hand, it shows near optimality of our data-dependent choice of the rate of the maximum frequency for the modified log-periodogram regression estimator. The proofs contain results which are also of independent interest: one result shows that data tapering gives a significant improvement in asymptotic properties of covariances of discrete Fourier transforms of long memory time series, while another gives an exponential inequality for the modified log-periodogram regression estimator.     

Strict L<Subscript>∞</Subscript> Isotonic Regression

Given a function f and weights w on the vertices of a directed acyclic graph G, an isotonic regression of (f,w) is an order-preserving real-valued function that minimizes the weighted distance to f among all order-preserving functions. When the distance is given via the supremum norm there may be many isotonic regressions. One of special interest is the strict isotonic regression, which is the limit of p-norm isotonic regression as p approaches infinity. Algorithms for determining it are given. We also examine previous isotonic regression algorithms in terms of their behavior as mappings from weighted functions over G to isotonic functions over G, showing that the fastest algorithms are not monotonic mappings. In contrast, the strict isotonic regression is monotonic.     

A sequence of improvements over the James-Stein estimator

In this article, we consider the problem of estimating a p-variate (p ≥ 3) normal mean vector in a decision-theoretic setup. Using a simple property of the noncentral chi-square distribution, we have produced a sequence of smooth estimators dominating the James-Stein estimator and each improved estimator is better than the previous one. It is also shown by using a technique of [5]. J. Multivariate Anal.36 121–126) that our smooth estimators can be dominated by non-smooth estimators.     

A posteriori error estimates of constrained optimal control problem governed by convection diffusion equations

In this paper, we study a posteriori error estimates of the edge stabilization Galerkin method for the constrained optimal control problem governed by convection-dominated diffusion equations. The residual-type a posteriori error estimators yield both upper and lower bounds for control u measured in L ²-norm and for state y and costate p measured in energy norm. Two numerical examples are presented to illustrate the effectiveness of the error estimators provided in this paper.      

Consistent nonparametric regression from recursive partitioning schemes

We here extend our results on asymptotically Bayes risk efficient classification to the general regression scenario. More precisely, we find L^p consistent estimators for an arbitrary regression function provided only that the dependent variable has a finite absolute pth moment. The estimators are truncated and untruncated local means derived from recursive partitioning schemes.     

Convergence of a fourth-order singular perturbation of the n-dimensional radially symmetric Monge–Ampère equation

This paper concerns with the convergence analysis of a fourth-order singular perturbation of the Dirichlet Monge–Ampère problem in the n-dimensional radial symmetric case. A detailed study of the fourth- order problem is presented. In particular, various a priori estimates with explicit dependence on the perturbation parameter ε are derived, and a crucial convexity property is also proved for the solution of the fourth-order problem. Using these estimates and the convexity property, we prove that the solution of the perturbed problem converges uniformly and compactly to the unique convex viscosity solution of the Dirichlet Monge–Ampère problem. Rates of convergence in the H^k-norm for k = 0, 1, 2 are also established.     

Sharp asymptotics for isotonic regression

 The asymptotic behavior of the isotonic estimator of a monotone regression function (that is the least-squares estimator under monotonicity restriction) is investigated. In particular it is proved that the ?₁-distance between the isotonic estimator and the true function is of magnitude n ^-1/3. Moreover, it is proved that a centered version of this ?₁-distance converges at the n ^1/2 rate to a Gaussian variable with fixed variance. Received: 20 September 1999 / Revised version: 10 May 2001 / Published online: 19 December 2001     

Sharp adaptive estimation of quadratic functionals

Estimation of a quadratic functional of a function observed in the Gaussian white noise model is considered. A data-dependent method for choosing the amount of smoothing is given. The method is based on comparing certain quadratic estimators with each other. It is shown that the method is asymptotically sharp or nearly sharp adaptive simultaneously for the “regular” and “irregular” region. We consider l_p bodies and construct bounds for the risk of the estimator which show that for p=4 the estimator is exactly optimal and for example when p ∈[3,100], then the upper bound is at most 1.055 times larger than the lower bound. We show the connection of the estimator to the theory of optimal recovery. The estimator is a calibration of an estimator which is nearly minimax optimal among quadratic estimators. Writing of this article was financed by Deutsche Forschungsgemeinschaft under project MA1026/6-2, CIES, France, and Jenny and AnttiWihuri Foundation.     

Generalized mirror averaging and <Emphasis Type="Italic">D</Emphasis>-convex aggregation

We study the problem of aggregation of estimators. Given a collection of M different estimators, we construct a new estimator, called aggregate, which is nearly as good as the best linear combination over an l ₁-ball of ℝ^M of the initial estimators. The aggregate is obtained by a particular version of the mirror averaging algorithm. We show that our aggregation procedure statisfies sharp oracle inequalities under general assumptions. Then we apply these results to a new aggregation problem: D-convex aggregation. Finally we implement our procedure in a Gaussian regression model with random design and we prove its optimality in a minimax sense up to a logarithmic factor.      

A Maximum Principle for an Explicit Finite Difference Scheme Approximating the Hodgkin-Huxley Model

We analyze an explicit finite difference scheme for the general form of the Hodgkin-Huxley model, which is a nonlinear partial differential equation coupled to a set of ODEs. The system of equations describes propagation of an electrical signal in excitable cells. We prove that the numerical solution is bounded in the L^∞-norm and L² converges to a unique solution. The L^∞-bound, which is the key point of our analysis, is proved by showing that the discrete solutions are invariant in a physically relevant bounded region. For the convergence proof we use the compactness method. AMS subject classification (2000) 65F20     

On universal estimators in learning theory

This paper addresses the problem of constructing and analyzing estimators for the regression problem in supervised learning. Recently, there has been great interest in studying universal estimators. The term “universal” means that, on the one hand, the estimator does not depend on the a priori assumption that the regression function f _ρ belongs to some class F from a collection of classes F and, on the other hand, the estimation error for f _ρ is close to the optimal error for the class F. This paper is an illustration of how the general technique of constructing universal estimators, developed in the author’s previous paper, can be applied in concrete situations. The setting of the problem studied in the paper has been motivated by a recent paper by Smale and Zhou. The starting point for us is a kernel K(x, u) defined on X × Ω. On the base of this kernel, we build an estimator that is universal for classes defined in terms of nonlinear approximations with regard to the system {K(·, u)} _uεΩ. To construct an easily implementable estimator, we apply the relaxed greedy algorithm. Published in Russian in Trudy Matematicheskogo Instituta imeni V.A. Steklova, 2006, Vol. 255, pp. 256–272.     

An improved estimation to make Markowitz’s portfolio optimization theory users friendly and estimation accurate with application on the US stock market investment

Using the Markowitz mean–variance portfolio optimization theory, researchers have shown that the traditional estimated return greatly overestimates the theoretical optimal return, especially when the dimension to sample size ratio p/n is large. Bai et al. (2009) propose a bootstrap-corrected estimator to correct the overestimation, but there is no closed form for their estimator. To circumvent this limitation, this paper derives explicit formulas for the estimator of the optimal portfolio return. We also prove that our proposed closed-form return estimator is consistent when n → ∞ and p/n → y ∈ (0, 1). Our simulation results show that our proposed estimators dramatically outperform traditional estimators for both the optimal return and its corresponding allocation under different values of p/n ratios and different inter-asset correlations ρ, especially when p/n is close to 1. We also find that our proposed estimators perform better than the bootstrap-corrected estimators for both the optimal return and its corresponding allocation. Another advantage of our improved estimation of returns is that we can also obtain an explicit formula for the standard deviation of the improved return estimate and it is smaller than that of the traditional estimate, especially when p/n is large. In addition, we illustrate the applicability of our proposed estimate on the US stock market investment.     

Optimal solution approximation for infinite positive-definite quadratic programming

We consider a general doubly-infinite, positive-definite, quadratic programming problem. We show that the sequence of unique optimal solutions to the natural finite-dimensional subproblems strongly converges to the unique optimal solution. This offers the opportunity to arbitrarily well approximate the infinite-dimensional optimal solution by numerically solving a sufficiently large finite-dimensional version of the problem. We then apply our results to a general time-varying, infinite-horizon, positive-definite, LQ control problem.This work was supported in part by the National Science Foundation under Grants ECS-8700836, DDM-9202849, and DDM-9214894.     

Wavelet Threshold Estimation of a Regression Function with Random Design

The wavelet threshold estimator of a regression function for the random design is constructed. The optimal uniform convergence rate of the estimator in a ball of Besov Space B^s_p, q is proved under general assumptions. The adaptive wavelet threshold estimator with near-optimal convergence rate in a wide range of Besov scale is also constructed.     

Sequential Identification of Linear Dynamic Systems with Memory

This paper presents a sequential estimator for some unknown parameters in stochastic linear systems with memory. As examples stochastic differential equations with time delayed drift are considered. Based on the maximum likelihood method, we construct an estimation procedure with given accuracy in the sense of the L _p -norm (p 2). It is shown, that this procedure works also in certain cases, when the normalized information matrix of the observed process is asymptotically degenerated. The almost surely consistency of the proposed estimators and the asymptotic behavior of the length of observations are derived.     

Empirical Bayes estimation in a multiple linear regression model

Summary Estimation of the vector β of the regression coefficients in a multiple linear regressionY=Xβ+ε is considered when β has a completely unknown and unspecified distribution and the error-vector ε has a multivariate standard normal distribution. The optimal estimator for β, which minimizes the overall mean squared error, cannot be constructed for use in practice. UsingX, Y and the information contained in the observation-vectors obtained fromn independent past experiences of the problem, (empirical Bayes) estimators for β are exhibited. These estimators are compared with the optimal estimator and are shown to be asymptotically optimal. Estimators asymptotically optimal with rates nearO(n ⁻¹) are constructed. Supported in part by a Natural Sciences and Engineering Research Council of Canada grant.     

Optimal variance estimation based on lagged second-order difference in nonparametric regression

Differenced estimators of variance bypass the estimation of regression function and thus are simple to calculate. However, there exist two problems: most differenced estimators do not achieve the asymptotic optimal rate for the mean square error; for finite samples the estimation bias is also important and not further considered. In this paper, we estimate the variance as the intercept in a linear regression with the lagged Gasser-type variance estimator as dependent variable. For the equidistant design, our estimator is not only \(n^{1/2}\)-consistent and asymptotically normal, but also achieves the optimal bound in terms of estimation variance with less asymptotic bias. Simulation studies show that our estimator has less mean square error than some existing differenced estimators, especially in the cases of immense oscillation of regression function and small-sized sample.     

Admissible and minimax multiparameter estimation in exponential families

Consider p independent distributions each belonging to the one parameter exponential family with distribution functions absolutely continuous with respect to Lebesgue measure. For estimating the natural parameter vector with p ≥ p₀ (p₀ is typically 2 or 3), a general class of estimators dominating the minimum variance unbiased estimator (MVUE) or an estimator which is a known constant multiple of the MVUE is produced under different weighted squared error losses. Included as special cases are some results of Hudson [13] and Berger [5]. Also, for a subfamily of the general exponential family, a class of estimators dominating the MVUE of the mean vector or an estimator which is a known constant multiple of the MVUE is produced. The major tool is to obtain a general solution to a basic differential inequality.