Approximating smooth and sparse functions by deep neural networks: Optimal approximation rates and saturation

Constructing neural networks for function approximation is a classical and longstanding topic in approximation theory. In this paper, we aim at constructing deep neural networks with three hidden layers using a sigmoidal activation function to approximate smooth and sparse functions. Specifically, we prove that the constructed deep nets with controllable magnitude of free parameters can reach the optimal approximation rate in approximating both smooth and sparse functions. In particular, we prove that neural networks with three hidden layers can avoid the phenomenon of saturation, i.e., the phenomenon that for some neural network architectures, the approximation rate stops improving for functions of very high smoothness.     

The estimate for approximation error of spherical neural networks

Compared with planar hyperplane, fitting data on the sphere has been an important and an active issue in geoscience, metrology, brain imaging, and so on. In this paper, with the help of the Jackson‐type theorem of polynomial approximation on the sphere, we construct spherical feed‐forward neural networks to approximate the continuous function defined on the sphere. As a metric, the modulus of smoothness of spherical function is used to measure the error of the approximation, and a Jackson‐type theorem on the approximation is established. Copyright © 2011 John Wiley & Sons, Ltd.     

Chebyshev approximation by discrete superposition. Application to neural networks

In this paper, we develop two algorithms for Chebyshev approximation of continuous functions on [0, 1] ⁿ using the modulus of continuity and the maximum norm estimated by a given finite data system. The algorithms are based on constructive versions of Kolmogorov's superposition theorem. One of the algorithms we apply to neural networks.     

On the near optimality of the stochastic approximation of smooth functions by neural networks

We consider the problem of approximating the Sobolev class of functions by neural networks with a single hidden layer, establishing both upper and lower bounds. The upper bound uses a probabilistic approach, based on the Radon and wavelet transforms, and yields similar rates to those derived recently under more restrictive conditions on the activation function. Moreover, the construction using the Radon and wavelet transforms seems very natural to the problem. Additionally, geometrical arguments are used to establish lower bounds for two types of commonly used activation functions. The results demonstrate the tightness of the bounds, up to a factor logarithmic in the number of nodes of the neural network. This revised version was published online in June 2006 with corrections to the Cover Date.     

The capability of approximation for neural networks interpolant on the sphere

Compared with planar hyperplane, fitting data on the sphere has been an important and active issue in geoscience, metrology, brain imaging, and so on. In this paper, using a functional approach, we rigorously prove that for given distinct samples on the unit sphere there exists a feed‐forward neural network with single hidden layer which can interpolate the samples, and simultaneously near best approximate the target function in continuous function space. Also, by using the relation between spherical positive definite radial basis functions and the basis function on the Euclidean space ?^{d + 1}, a similar result in a spherical Sobolev space is established. Copyright © 2010 John Wiley & Sons, Ltd.     

Approximation by radial bases and neural networks

In this paper, we study approximation by radial basis functions including Gaussian, multiquadric, and thin plate spline functions, and derive order of approximation under certain conditions. Moreover, neural networks are also constructed by wavelet recovery formula and wavelet frames.     

The rate of approximation of Gaussian radial basis neural networks in continuous function space

There have been many studies on the dense theorem of approximation by radial basis feedforword neural networks, and some approximation problems by Gaussian radial basis feedforward neural networks(GRBFNs)in some special function space have also been investigated. This paper considers the approximation by the GRBFNs in continuous function space. It is proved that the rate of approximation by GRNFNs with n~d neurons to any continuous function f defined on a compact subset K(R~d)can be controlled by ω(f, n~(-1/2)), where ω(f, t)is the modulus of continuity of the function f .     

基于散乱阈值结点的周期神经网络的逼近阶

§1Introduction Inrecentyearstherehasbeengrowinginterestintheproblemofneuralnetworkand relatedapproximation,manyimportantresultsareobtained.Becauseofitsabilityof parallelcomputationinlargescaleandofperfectself-adaptingandapproximation,the neuralnetworkhasbeenwidelyapplied.Theapproximationabilityoftheneuralnetwork dependsonitstopologicalstructure.LetRsbeans-dimensionalEuclidSpaceand(x)isa realfunctiondefinedonRs.When(x)isanexcitationfunctionandx∈Rsisaninput vector,thesimpleneuralnetwork…     

Approximation by neural networks with sigmoidal functions

In this paper, we introduce a type of approximation operators of neural networks with sigmodal functions on compact intervals, and obtain the pointwise and uniform estimates of the ap- proximation. To improve the approximation rate, we further introduce a type of combinations of neurM networks. Moreover, we show that the derivatives of functions can also be simultaneously approximated by the derivatives of the combinations. We also apply our method to construct approximation operators of neural networks with sigmodal functions on infinite intervals.     

The construction and approximation of some neural networks operators

In this paper,the technique of approximate partition of unity is used to construct a class of neural networks operators with sigmoidal functions.Using the modulus of continuity of function as a metric,...     

Universality of deep convolutional neural networks

Deep learning has been widely applied and brought breakthroughs in speech recognition, computer vision, and many other domains. Deep neural network architectures and computational issues have been well studied in machine learning. But there lacks a theoretical foundation for understanding the approximation or generalization ability of deep learning methods generated by the network architectures such as deep convolutional neural networks. Here we show that a deep convolutional neural network (CNN) is universal, meaning that it can be used to approximate any continuous function to an arbitrary accuracy when the depth of the neural network is large enough. This answers an open question in learning theory. Our quantitative estimate, given tightly in terms of the number of free parameters to be computed, verifies the efficiency of deep CNNs in dealing with large dimensional data. Our study also demonstrates the role of convolutions in deep CNNs.     

The construction and approximation of feedforward neural network with hyp erb olic tangent function

In this paper, we discuss some analytic properties of hyperbolic tangent function and estimate some approximation errors of neural network operators with the hyperbolic tangent activation functionFirstly, an equation of partitions of unity for the hyperbolic tangent function is givenThen, two kinds of quasi-interpolation type neural network operators are constructed to approximate univariate and bivariate functions, respectivelyAlso, the errors of the approximation are estimated by means of the modulus of continuity of functionMoreover, for approximated functions with high order derivatives, the approximation errors of the constructed operators are estimated.     

Limitations of the approximation capabilities of neural networks with one hidden layer

Lets1 be an integer andW be the class of all functions having integrable partial derivatives on [0, 1] ^s . We are interested in the minimum number of neurons in a neural network with a single hidden layer required in order to provide a mean approximation order of a preassigned>0 to each function inW. We prove that this number cannot be if a spline-like localization is required. This cannot be improved even if one allows different neurons to evaluate different activation functions, even depending upon the target function. Nevertheless, for any>0, a network with neurons can be constructed to provide this order of approximation, with localization. Analogous results are also valid for otherL ^p norms.The research of this author was supported by NSF Grant # DMS 92-0698.The research of this author was supported, in part, by AFOSR Grant #F49620-93-1-0150 and by NSF Grant #DMS 9404513.     

Non-convergence of stochastic gradient descent in the training of deep neural networks

Deep neural networks have successfully been trained in various application areas with stochastic gradient descent. However, there exists no rigorous mathematical explanation why this works so well. The training of neural networks with stochastic gradient descent has four different discretization parameters: (i) the network architecture; (ii) the amount of training data; (iii) the number of gradient steps; and (iv) the number of randomly initialized gradient trajectories. While it can be shown that the approximation error converges to zero if all four parameters are sent to infinity in the right order, we demonstrate in this paper that stochastic gradient descent fails to converge for ReLU networks if their depth is much larger than their width and the number of random initializations does not increase to infinity fast enough.     

Complexity of Gaussian-radial-basis networks approximating smooth functions

Complexity of Gaussian-radial-basis-function networks, with varying widths, is investigated. Upper bounds on rates of decrease of approximation errors with increasing number of hidden units are derived. Bounds are in terms of norms measuring smoothness (Bessel and Sobolev norms) multiplied by explicitly given functions a(r,d)$a(r,d)$ of the number of variables d$d$ and degree of smoothness r$r$. Estimates are proven using suitable integral representations in the form of networks with continua of hidden units computing scaled Gaussians and translated Bessel potentials. Consequences on tractability of approximation by Gaussian-radial-basis function networks are discussed.     

Convergence analysis of a space-time approximation to a two-dimensional system of shallow water equations

A characteristic-Galerkin approximation of a new two-dimensional Shallow Water viscous parabolic model is presented and a semi-implicit-Lagrangian finite element scheme is used. A convergence result is proved and numerical experiments on academic tests are shown.