基于改进sigmoid激活函数的深度神经网络训练算法研究

针对深度神经网络训练过程中残差随着其传播深度越来越小而使底层网络无法得到有效训练的问题,通过分析传统sigmoid激活函数应用于深度神经网络的局限性,提出双参数sigmoid激活函数。一个参数保证激活函数的输入集中坐标原点两侧,避免了激活函数进入饱和区,一个参数抑制残差衰减的速度,双参数结合有效的增强了深度神经网络的训练。结合DBN对MNIST数据集进行数字分类实验,实验表明双参数 sigmoid激活函数能够直接应用于无预训练深度神经网络,而且提高了sigmoid激活函数在有预训练深度神经网络中的训练效果。

A Study of Training Algorithm in Deep Neural Networks based on Sigmoid Activation Function

Aiming at the problem that residual error gets smaller with the depth of propagation increasing and the bottom of DNN trains ineffective, by investigating the limitations of sigmoid activation function in DNN, a sigmoid activation function with two parameters is proposed. One parameter makes the input of sigmoid activation function concentrate in sides of the origin, and another parameter restrains the decreasing speed of residual error. The combination of two parameters enhances the training of DNN. Do number classification experiments on MNIST using deep belief networks(DBN), the results show that sigmoid activation function with two parameters can be used in DNN directly without pre-training and improve the performance in DNN with pre-training.