Training Multilayer Perceptrons Via Minimization of Sum of Ridge Functions

Motivated by the problem of training multilayer perceptrons in neural networks, we consider the problem of minimizing E(x)= _i=1 ⁿ f _i ( _i x), where _i R ^s , 1in, and each f _i ( _i x) is a ridge function. We show that when n is small the problem of minimizing E can be treated as one of minimizing univariate functions, and we use the gradient algorithms for minimizing E when n is moderately large. For large n, we present the online gradient algorithms and especially show the monotonicity and weak convergence of the algorithms.