Approximating smooth and sparse functions by deep neural networks: Optimal approximation rates and saturation |
| |
| Affiliation: | 1. École Polytechnique, LIX, 91128 Palaiseau Cedex, France;2. Instituto Superior Técnico, Universidade de Lisboa, 1049-001 Lisbon, Portugal;3. Instituto de Telecomunicações, Lisbon, Portugal;4. Universidade do Algarve, C. Gambelas, 8005-139 Faro, Portugal;5. Université Paris Cité, CNRS, IRIF, F-75013, Paris, France;1. School of Mathematics, Hangzhou Normal University, Hangzhou, 311121, China;2. School of Data Science, Zhejiang University of Finance & Economics, Hangzhou, 310018, China |
| |
| Abstract: | Constructing neural networks for function approximation is a classical and longstanding topic in approximation theory. In this paper, we aim at constructing deep neural networks with three hidden layers using a sigmoidal activation function to approximate smooth and sparse functions. Specifically, we prove that the constructed deep nets with controllable magnitude of free parameters can reach the optimal approximation rate in approximating both smooth and sparse functions. In particular, we prove that neural networks with three hidden layers can avoid the phenomenon of saturation, i.e., the phenomenon that for some neural network architectures, the approximation rate stops improving for functions of very high smoothness. |
| |
| Keywords: | Approximation theory Deep learning Deep neural networks Localized approximation Sparse approximation |
| 本文献已被 ScienceDirect 等数据库收录！ |
|
