首页 | 本学科首页   官方微博 | 高级检索  
     检索      


Topological Properties of the Set of Functions Generated by Neural Networks of Fixed Size
Authors:Petersen  Philipp  Raslan  Mones  Voigtlaender  Felix
Institution:1.Faculty of Mathematics and Research Platform Data Science @ Uni Vienna, University of Vienna, Oskar Morgenstern Platz 1, 1090, Vienna, Austria
;2.Institute of Mathematics, Technical University of Berlin, Straße des 17. Juni 136, 10623, Berlin, Germany
;3.Department of Scientific Computing, Catholic University of Eichstätt-Ingolstadt, Kollegiengebäude I Bau B, Ostenstraße 26, 85072, Eichstätt, Germany
;
Abstract:

We analyze the topological properties of the set of functions that can be implemented by neural networks of a fixed size. Surprisingly, this set has many undesirable properties. It is highly non-convex, except possibly for a few exotic activation functions. Moreover, the set is not closed with respect to \(L^p\)-norms, \(0< p < \infty \), for all practically used activation functions, and also not closed with respect to the \(L^\infty \)-norm for all practically used activation functions except for the ReLU and the parametric ReLU. Finally, the function that maps a family of weights to the function computed by the associated network is not inverse stable for every practically used activation function. In other words, if \(f_1, f_2\) are two functions realized by neural networks and if \(f_1, f_2\) are close in the sense that \(\Vert f_1 - f_2\Vert _{L^\infty } \le \varepsilon \) for \(\varepsilon > 0\), it is, regardless of the size of \(\varepsilon \), usually not possible to find weights \(w_1, w_2\) close together such that each \(f_i\) is realized by a neural network with weights \(w_i\). Overall, our findings identify potential causes for issues in the training procedure of deep learning such as no guaranteed convergence, explosion of parameters, and slow convergence.

Keywords:
本文献已被 SpringerLink 等数据库收录!
设为首页 | 免责声明 | 关于勤云 | 加入收藏

Copyright©北京勤云科技发展有限公司  京ICP备09084417号