A note on the expressive power of deep rectified linear unit networks in high‐dimensional spaces

We investigate the ability of deep deep rectified linear unit (ReLU) networks to approximate multivariate functions. Specially, we establish the approximation error estimate on a class of bandlimited functions; in this case, ReLU networks can overcome the “curse of dimensionality.”     

Approximating smooth and sparse functions by deep neural networks: Optimal approximation rates and saturation

Constructing neural networks for function approximation is a classical and longstanding topic in approximation theory. In this paper, we aim at constructing deep neural networks with three hidden layers using a sigmoidal activation function to approximate smooth and sparse functions. Specifically, we prove that the constructed deep nets with controllable magnitude of free parameters can reach the optimal approximation rate in approximating both smooth and sparse functions. In particular, we prove that neural networks with three hidden layers can avoid the phenomenon of saturation, i.e., the phenomenon that for some neural network architectures, the approximation rate stops improving for functions of very high smoothness.     

DEEP RELU NETWORKS OVERCOME THE CURSE OF DIMENSIONALITY FOR GENERALIZED BANDLIMITED FUNCTIONS

We prove a theorem concerning the approximation of generalized bandlimited mul-tivariate functions by deep ReLU networks for which the curse of the dimensionality is overcome.Our theorem is based on a result by Maurey and on the ability of deep ReLU networks to approximate Chebyshev polynomials and analytic functions efficiently.     

The ridge function representation of polynomials and an application to neural networks

The first goal of this paper is to establish some properties of the ridge function representation for multivariate polynomials, and the second one is to apply these results to the problem of approximation by neural networks. We find that for continuous functions, the rate of approximation obtained by a neural network with one hidden layer is no slower than that of an algebraic polynomial.     

Adaptive finite element methods for elliptic equations over hierarchical T-meshes

Isogeometric analysis using NURBS (Non-Uniform Rational B-Splines) as basis functions gives accurate representation of the geometry and the solution but it is not well suited for local refinement. In this paper, we use the polynomial splines over hierarchical T-meshes (PHT-splines) to construct basis functions which not only share the nice smoothness properties as the B-splines, but also allow us to effectively refine meshes locally. We develop a residual-based a posteriori error estimate for the finite element discretization of elliptic equations using PHT-splines basis functions and study their approximation properties. In addition, we conduct numerical experiments to verify the theory and to demonstrate the effectiveness of the error estimate and the high order approximations provided by the numerical solution.     

The estimate for approximation error of spherical neural networks

Compared with planar hyperplane, fitting data on the sphere has been an important and an active issue in geoscience, metrology, brain imaging, and so on. In this paper, with the help of the Jackson‐type theorem of polynomial approximation on the sphere, we construct spherical feed‐forward neural networks to approximate the continuous function defined on the sphere. As a metric, the modulus of smoothness of spherical function is used to measure the error of the approximation, and a Jackson‐type theorem on the approximation is established. Copyright © 2011 John Wiley & Sons, Ltd.     

Weierstrass formula and zero-finding methods

Summary. Classical Weierstrass' formula [29] has been often the subject of investigation of many authors. In this paper we give some further applications of this formula for finding the zeros of polynomials and analytic functions. We are concerned with the problems of localization of polynomial zeros and the construction of iterative methods for the simultaneous approximation and inclusion of these zeros. Conditions for the safe convergence of Weierstrass' method, depending only on initial approximations, are given. In particular, we study polynomials with interval coefficients. Using an interval version of Weierstrass' method enclosures in the form of disks for the complex-valued set containing all zeros of a polynomial with varying coefficients are obtained. We also present Weierstrass-like algorithm for approximating, simultaneously, all zeros of a class of analytic functions in a given closed region. To demonstrate the proposed algorithms, three numerical examples are included. Received September 13, 1993     

Approximating Gradients with Continuous Piecewise Polynomial Functions

Motivated by conforming finite element methods for elliptic problems of second order, we analyze the approximation of the gradient of a target function by continuous piecewise polynomial functions over a simplicial mesh. The main result is that the global best approximation error is equivalent to an appropriate sum in terms of the local best approximation errors on elements. Thus, requiring continuity does not downgrade local approximation capability and discontinuous piecewise polynomials essentially do not offer additional approximation power, even for a fixed mesh. This result implies error bounds in terms of piecewise regularity over the whole admissible smoothness range. Moreover, it allows for simple local error functionals in adaptive tree approximation of gradients.     

On an approximation family of discrete polynomial operators

We present a new family of linear discrete polynomial operators giving a Timan type approximation theorem for functions of arbitrary smoothness. Using this we construct two families of operators of this kind to extend Freud type approximation results to functions of higher smoothness.     

Numerical solution of the Poisson equation over hypercubes using reduced Chebyshev polynomial bases

We describe how to use new reduced size polynomial approximations for the numerical solution of the Poisson equation over hypercubes. Our method is based on a non-standard Galerkin method which allows test functions which do not verify the boundary conditions. Numerical examples are given in dimensions up to 8 on solutions with different smoothness using the same approximation basis for both situations. A special attention is paid on conditioning problems.     

Approximation by Entire Functions on a Countable Union of Segments on the Real Axis: 3. Further Generalization

In this paper, an approximation of functions of extensive classes set on a countable unit of segments of a real axis using the entire functions of exponential type is considered. The higher the type of the approximating function is, the higher the rate of approximation near segment ends can be made, compared with their inner points. The general approximation scale, which is nonuniform over its segments, depending on the type of the entire function, is similar to the scale set out for the first time in the study of the approximation of the function by polynomials. For cases with one segment and its approximation by polynomials, this scale has allowed us to connect the so-called direct theorems, which state a possible rate of smooth function approximation by polynomials, and the inverse theorems, which give the smoothness of a function approximated by polynomials at a given rate. The approximations by entire functions on a countable unit of segments for the case of Hölder spaces have been studied by the authors in two preceding papers. This paper significantly expands the class of spaces for the functions, which are used to plot an approximation that engages the entire functions with the required properties.     

多层前向正则模糊神经网络的逼近能力

Abstract. Four-layer feedforward regular fuzzy neural networks are constructed. Universal ap-proximations to some continuous fuzzy functions defined on (R)“ by the four-layer fuzzyneural networks are shown. At first,multivariate Bernstein polynomials associated with fuzzyvalued functions are empolyed to approximate continuous fuzzy valued functions defined on eachcompact set of R“. Secondly,by introducing cut-preserving fuzzy mapping,the equivalent condi-tions for continuous fuzzy functions that can be arbitrarily closely approximated by regular fuzzyneural networks are shown. Finally a few of sufficient and necessary conditions for characteriz-ing approximation capabilities of regular fuzzy neural networks are obtained. And some concretefuzzy functions demonstrate our conclusions.     

Lower bounds of error estimates for singular perturbation problems with Jacobi-Spectral approximations

With weighted orthogonal Jacobi polynomials, we study spectral approximations for singular perturbation problems on an interval. The singular parameters of the model are included in the basis functions, and then its stiff matrix is diagonal. Considering the estimations for weighted orthogonal coefficients, a special technique is proposed to investigate the a posteriori error estimates. In view of the difficulty of a posteriori error estimates for spectral approximations, we employ a truncation projection to study lower bounds for the models. Specially, we present the lower bounds of a posteriori error estimates with two different weighted norms in details.     

Adaptive finite element methods for elliptic equations over hierarchical T-meshes

Isogeometric analysis using NURBS (Non-Uniform Rational B-Splines) as basis functions gives accurate representation of the geometry and the solution but it is not well suited for local refinement. In this paper, we use the polynomial splines over hierarchical T-meshes (PHT-splines) to construct basis functions which not only share the nice smoothness properties as the B-splines, but also allow us to effectively refine meshes locally. We develop a residual-based a posteriori error estimate for the finite element discretization of elliptic equations using PHT-splines basis functions and study their approximation properties. In addition, we conduct numerical experiments to verify the theory and to demonstrate the effectiveness of the error estimate and the high order approximations provided by the numerical solution.     

Complexity of Neural Network Approximation with Limited Information: A Worst Case Approach

In neural network theory the complexity of constructing networks to approximate input-output functions is of interest. We study this in the more general context of approximating elements f of a normed space F using partial information about f. We assume information about f and the size of the network are limited, as is typical in radial basis function networks. We show complexity can be essentially split into two independent parts, information ε-complexity and neural ε-complexity. We use a worst case setting, and integrate elements of information-based complexity and nonlinear approximation. We consider deterministic and/or randomized approximations using information possibly corrupted by noise. The results are illustrated by examples including approximation by piecewise polynomial neural networks.     

On the optimization of flux limiter schemes for hyperbolic conservation laws

A classic strategy to obtain high‐quality discretizations of hyperbolic partial differential equations is to use flux limiter (FL) functions for blending two types of approximations: a monotone first‐order scheme that deals with discontinuous solution features and a higher order method for approximating smooth solution parts. In this article, we study a new approach to FL methods. Relying on a classification of input data with respect to smoothness, we associate specific basis functions with the individual smoothness notions. Then, we construct a limiter as a linear combination of the members of parameter‐dependent families of basis functions, and we explore the possibility to optimize the parameters in interesting model situations to find a corresponding optimal limiter. © 2012 Wiley Periodicals, Inc. Numer Methods Partial Differential Eq, 2013     

Non-convergence of stochastic gradient descent in the training of deep neural networks

Deep neural networks have successfully been trained in various application areas with stochastic gradient descent. However, there exists no rigorous mathematical explanation why this works so well. The training of neural networks with stochastic gradient descent has four different discretization parameters: (i) the network architecture; (ii) the amount of training data; (iii) the number of gradient steps; and (iv) the number of randomly initialized gradient trajectories. While it can be shown that the approximation error converges to zero if all four parameters are sent to infinity in the right order, we demonstrate in this paper that stochastic gradient descent fails to converge for ReLU networks if their depth is much larger than their width and the number of random initializations does not increase to infinity fast enough.     

Approximating Unknown Mappings: An Experimental Evaluation

Different methodologies have been introduced in recent years with the aim of approximating unknown functions. Basically, these methodologies are general frameworks for representing non-linear mappings from several input variables to several output variables. Research into this problem occurs in applied mathematics (multivariate function approximation), statistics (nonparametric multiple regression) and computer science (neural networks). However, since these methodologies have been proposed in different fields, most of the previous papers treat them in isolation, ignoring contributions in the other areas. In this paper we consider five well known approaches for function approximation. Specifically we target polynomial approximation, general additive models (Gam), local regression (Loess), multivariate additive regression splines (Mars) and artificial neural networks (Ann).Neural networks can be viewed as models of real systems, built by tuning parameters known as weights. In training the net, the problem is to find the weights that optimize its performance (i.e. to minimize the error over the training set). Although the most popular method for Ann training is back propagation, other optimization methods based on metaheuristics have recently been adapted to this problem, outperforming classical approaches. In this paper we propose a short term memory tabu search method, coupled with path relinking and BFGS (a gradient-based local NLP solver) to provide high quality solutions to this problem. The experimentation with 15 functions previously reported shows that a feed-forward neural network with one hidden layer, trained with our procedure, can compete with the best-known approximating methods. The experimental results also show the effectiveness of a new mechanism to avoid overfitting in neural network training.     

球面带形平移网络逼近的Jackson定理

研究了球面带型平移网络逼近阶用球面调和多项式的最佳逼近及光滑模的刻画问题．借助于球调和多项式的最佳逼近多项式和Riesz平均构造出了单位球面Sq上的带形平移网络,并建立了球面带形平移网络对Lp(Sq)中函数一致逼近的Jackson型定理．所得结果表明球面带形平移网络可以达到球调和多项式的逼近阶．     

RELU DEEP NEURAL NETWORKS AND LINEAR FINITE ELEMENTS

In this paper, we investigate the relationship between deep neural networks (DNN) with rectified linear unit (ReLU) function as the activation function and continuous piecewise linear (CPWL) functions, especially CPWL functions from the simplicial linear finite element method (FEM). We first consider the special case of FEM. By exploring the DNN representation of its nodal basis functions, we present a ReLU DNN representation of CPWL in FEM. We theoretically establish that at least $2$ hidden layers are needed in a ReLU DNN to represent any linear finite element functions in $\Omega \subseteq \mathbb{R}^d$ when $d\ge2$. Consequently, for $d=2,3$ which are often encountered in scientific and engineering computing, the minimal number of two hidden layers are necessary and sufficient for any CPWL function to be represented by a ReLU DNN. Then we include a detailed account on how a general CPWL in $\mathbb R^d$ can be represented by a ReLU DNN with at most $\lceil\log_2(d+1)\rceil$ hidden layers and we also give an estimation of the number of neurons in DNN that are needed in such a representation. Furthermore, using the relationship between DNN and FEM, we theoretically argue that a special class of DNN models with low bit-width are still expected to have an adequate representation power in applications. Finally, as a proof of concept, we present some numerical results for using ReLU DNNs to solve a two-point boundary problem to demonstrate the potential of applying DNN for numerical solution of partial differential equations.