绝对值函数的一致光滑逼近函数

绝对值函数是一个非光滑函数,研究了绝对值函数的光滑逼近函数.给出了绝对值函数的上方一致光滑逼近函数和下方一致光滑逼近函数,分别研究了其性质,并通过图像展示了逼近效果.     

F格上的逼近算子

粗集理论是波兰学者Pawlak提出的知识表示新理论.Pawlak代数是粗集理论中粗集系统的抽象,其公理系统包含了知识粗表示所必须的全部性质.本文深入研究了F格上的逼近算子,建立了F格上弱逼近算子之间的某些代数运算,从而从理论上建立了各种知识粗表示之间的联系.我们还定义了逼近算子的闭包,进而用逼近算子导出拓扑,为信息系统的近似提供必要的数学基础.最后,作为特例,我们研究了粗集理论中由相似关系导出逼近算子的某些性质.     

局部凸空间的余逼近

研究了在局部凸空间中的ｆ－余逼近和强ｆ－余逼近的一些性质．     

L^p与△^p空间的小波逼近定理

定义了线性小波算子序列,运用其概率性质研究其对Lp与△^p空间函数的逼近性质,建立了相应的逼近等价定理。     

关于混合指数型积分算子的加权逼近

1987年,Z.Ditzian和V.Totik在[1]中研究了指数型算子的加权逼近问题,1989年,陈文忠教授在[2]中研究了混合指数型积分算子在C-空间的逼近性质,本文则是研究一类混合指数型积分算子在L_p空间的加权逼近问题。     

L~p与△~p空间的小波逼近定理

定义了线性小波算子序列 ,运用其概率性质研究其对Lp 与△p 空间函数的逼近性质 ,建立了相应的逼近等价定理 .     

强CHIP性质和广义限制域逼近的特征

本文研究了广义限制域的最佳逼近问题,在允许有有限个节点的情况下,引入次强内点条件的概念,并将优化理论中的强CHIP性质等概念应用到本文所研究的问题中,刻画了次强内点、强CHIP性质和最佳逼近的特征之间的关系.作为推论,我们得到了广义限制最佳逼近的Kolmogorov型和“零属于凸包”型等特征定理.     

Fourier-Laplace级数的Vallee Poussin平均逼近

本文研究了Fourier-Laplace级数Vall閑 Poussin平均的逼近性质,建立了Vall閑Poussin平均的一致逼近度估计和几乎处处逼近的阶.     

论非线性多重网格法的逼近性质

多重网格法是一种求解椭圆边值问题离散所得的大型线性或非线性方程组的“最优”解法。在有限元离散情形,Hackbusch提出了一种多重网格法的收敛分析方法,即把线性或非线性的多重网格法收敛率的估计问题归结为所谓“光滑性质”与“逼近性质”的研究。在线性情形,若已知有限元解的误差估计,一般容易得到多重网格法的“逼近性质”。但对非线性多重网格法的“逼近性质”在什么条件下成立,尚未见到这方面的工     

广义算子半群的遍历及概率逼近定理

首先给出了广义算子半群的Abel-遍历和Cesaro-遍历的定义,对两种遍历的性质进行了刻画,研究了两种遍历的等价条件.其次,利用Pettis积分、算子值数学期望及广义连续修正模等工具给出广义算子半群的概率逼近表达式.     

随机梯度下降法的收敛速度(英文)

本文研究了正则化格式下随机梯度下降法的收敛速度.利用线性迭代的方法,并通过参数选择,得到了随机梯度下降法的收敛速度.     

小批量随机块坐标下降算法

针对机器学习中广泛存在的一类问题：结构化随机优化问题（其中“结构化”是指问题的可行域具有块状结构,且目标函数的非光滑正则化部分在变量块之间是可分离的）,我们研究了小批量随机块坐标下降算法（mSBD）。按照求解非复合问题和复合问题分别给出了基本的mSBD和它的变体,对于非复合问题,分析了算法在没有一致有界梯度方差假设情况下的收敛性质。而对于复合问题,在不需要通常的Lipschitz梯度连续性假设条件下得到了算法的收敛性。最后通过数值实验验证了mSBD的有效性。     

Non-convergence of stochastic gradient descent in the training of deep neural networks

Deep neural networks have successfully been trained in various application areas with stochastic gradient descent. However, there exists no rigorous mathematical explanation why this works so well. The training of neural networks with stochastic gradient descent has four different discretization parameters: (i) the network architecture; (ii) the amount of training data; (iii) the number of gradient steps; and (iv) the number of randomly initialized gradient trajectories. While it can be shown that the approximation error converges to zero if all four parameters are sent to infinity in the right order, we demonstrate in this paper that stochastic gradient descent fails to converge for ReLU networks if their depth is much larger than their width and the number of random initializations does not increase to infinity fast enough.     

Stochastic quasi-gradient methods: variance reduction via Jacobian sketching

Mathematical Programming - We develop a new family of variance reduced stochastic gradient descent methods for minimizing the average of a very large number of smooth functions. Our...     

Accelerating variance-reduced stochastic gradient methods

Mathematical Programming - Variance reduction is a crucial tool for improving the slow convergence of stochastic gradient descent. Only a few variance-reduced methods, however, have yet been shown...     

Alternate step gradient method*

The Barzilai and Borwein (BB) gradient method does not guarantee a descent in the objective function at each iteration, but performs better than the classical steepest descent (SD) method in practice. So far, the BB method has found many successful applications and generalizations in linear systems, unconstrained optimization, convex-constrained optimization, stochastic optimization, etc. In this article, we propose a new gradient method that uses the SD and the BB steps alternately. Hence the name “alternate step (AS) gradient method.” Our theoretical and numerical analyses show that the AS method is a promising alternative to the BB method for linear systems. Unconstrained optimization algorithms related to the AS method are also discussed. Particularly, a more efficient gradient algorithm is provided by exploring the idea of the AS method in the GBB algorithm by Raydan (1997).

To establish a general R-linear convergence result for gradient methods, an important property of the stepsize is drawn in this article. Consequently, R-linear convergence result is established for a large collection of gradient methods, including the AS method. Some interesting insights into gradient methods and discussion about monotonicity and nonmonotonicity are also given.     

Meshfree Approximation for Stochastic Optimal Control Problems

In this work,we study the gradient projection method for solving a class of stochastic control problems by using a mesh free approximation ap-proach to implement spatial dimension approximation.Our main contribu-tion is to extend the existing gradient projection method to moderate high-dimensional space.The moving least square method and the general radial basis function interpolation method are introduced as showcase methods to demonstrate our computational framework,and rigorous numerical analysis is provided to prove the convergence of our meshfree approximation approach.We also present several numerical experiments to validate the theoretical re-sults of our approach and demonstrate the performance meshfree approxima-tion in solving stochastic optimal control problems.     

On Early Stopping in Gradient Descent Learning

In this paper we study a family of gradient descent algorithms to approximate the regression function from reproducing kernel Hilbert spaces (RKHSs), the family being characterized by a polynomial decreasing rate of step sizes (or learning rate). By solving a bias-variance trade-off we obtain an early stopping rule and some probabilistic upper bounds for the convergence of the algorithms. We also discuss the implication of these results in the context of classification where some fast convergence rates can be achieved for plug-in classifiers. Some connections are addressed with Boosting, Landweber iterations, and the online learning algorithms as stochastic approximations of the gradient descent method.     

Learning gradients by a gradient descent algorithm

We propose a stochastic gradient descent algorithm for learning the gradient of a regression function from random samples of function values. This is a learning algorithm involving Mercer kernels. By a detailed analysis in reproducing kernel Hilbert spaces, we provide some error bounds to show that the gradient estimated by the algorithm converges to the true gradient, under some natural conditions on the regression function and suitable choices of the step size and regularization parameters.     

Scheduling arrivals to a stochastic service mechanism

We consider the problem of scheduling the arrivals of a fixed number of customers to a stochastic service mechanism to minimize an expected cost associated with operating the system. We consider the special case of exponentially distributed service times and the problems in general associated with obtaining exact analytic solutions. For general service time distributions we obtain approximate numerical solutions using a stochastic version of gradient search employing Infinitesimal Perturbation Analysis estimates of the objective function gradient obtained via simulation.