Topological Properties of the Set of Functions Generated by Neural Networks of Fixed Size |
| |
Authors: | Petersen Philipp Raslan Mones Voigtlaender Felix |
| |
Institution: | 1.Faculty of Mathematics and Research Platform Data Science @ Uni Vienna, University of Vienna, Oskar Morgenstern Platz 1, 1090, Vienna, Austria ;2.Institute of Mathematics, Technical University of Berlin, Straße des 17. Juni 136, 10623, Berlin, Germany ;3.Department of Scientific Computing, Catholic University of Eichstätt-Ingolstadt, Kollegiengebäude I Bau B, Ostenstraße 26, 85072, Eichstätt, Germany ; |
| |
Abstract: | We analyze the topological properties of the set of functions that can be implemented by neural networks of a fixed size. Surprisingly, this set has many undesirable properties. It is highly non-convex, except possibly for a few exotic activation functions. Moreover, the set is not closed with respect to \(L^p\)-norms, \(0< p < \infty \), for all practically used activation functions, and also not closed with respect to the \(L^\infty \)-norm for all practically used activation functions except for the ReLU and the parametric ReLU. Finally, the function that maps a family of weights to the function computed by the associated network is not inverse stable for every practically used activation function. In other words, if \(f_1, f_2\) are two functions realized by neural networks and if \(f_1, f_2\) are close in the sense that \(\Vert f_1 - f_2\Vert _{L^\infty } \le \varepsilon \) for \(\varepsilon > 0\), it is, regardless of the size of \(\varepsilon \), usually not possible to find weights \(w_1, w_2\) close together such that each \(f_i\) is realized by a neural network with weights \(w_i\). Overall, our findings identify potential causes for issues in the training procedure of deep learning such as no guaranteed convergence, explosion of parameters, and slow convergence. |
| |
Keywords: | |
本文献已被 SpringerLink 等数据库收录! |
|