偏最小二乘回归的贝叶斯正则化神经网络集成模型在证券分析预测中的应用

神经网络集成技术能有效地提高神经网络的预测精度和泛化能力,已经成为机器学习和神经计算领域的一个研究热点.利用Bagging技术和不同的神经网络算法生成集成个体,并用偏最小二乘回归方法从中提取集成因子,再利用贝叶斯正则化神经网络对其集成,以此建立上证指数预测模型.通过上证指数开、收盘价进行实例分析,计算结果表明该方法预测精度高、稳定性好.     

Subsampling bias and the best-discrepancy systematic cross validation

Statistical machine learning models should be evaluated and validated before putting to work.Conventional k-fold Monte Carlo cross-validation(MCCV)procedure uses a pseudo-random sequence to partition instances into k subsets,which usually causes subsampling bias,inflates generalization errors and jeopardizes the reliability and effectiveness of cross-validation.Based on ordered systematic sampling theory in statistics and low-discrepancy sequence theory in number theory,we propose a new k-fold cross-validation procedure by replacing a pseudo-random sequence with a best-discrepancy sequence,which ensures low subsampling bias and leads to more precise expected-prediction-error(EPE)estimates.Experiments with 156 benchmark datasets and three classifiers(logistic regression,decision tree and na?ve bayes)show that in general,our cross-validation procedure can extrude subsampling bias in the MCCV by lowering the EPE around 7.18%and the variances around 26.73%.In comparison,the stratified MCCV can reduce the EPE and variances of the MCCV around 1.58%and 11.85%,respectively.The leave-one-out(LOO)can lower the EPE around 2.50%but its variances are much higher than the any other cross-validation(CV)procedure.The computational time of our cross-validation procedure is just 8.64%of the MCCV,8.67%of the stratified MCCV and 16.72%of the LOO.Experiments also show that our approach is more beneficial for datasets characterized by relatively small size and large aspect ratio.This makes our approach particularly pertinent when solving bioscience classification problems.Our proposed systematic subsampling technique could be generalized to other machine learning algorithms that involve random subsampling mechanism.     

基于支持向量机的粒子群神经网络集成股市预测模型

为有效提高神经网络集成的泛化能力,先利用量子粒子群和主成分分析提高集成个体的泛化能力,再利用泛化能力强的支持向量机回归集成生成输出结论,建立一个基于支持向量机的粒子群神经网络集成股市预测模型.试验表明,该模型能有效提高神经网络集成系统的泛化能力,预测精度高,稳定性好.     

拟自适应分类随机森林算法

本文给出了集成学习模型可以收敛的集成学习算法,拟自适应分类随机森林算法。拟自适应分类随机森林算法综合了Adaboost算法和随机森林算法的优势,实验数据分析表明,训练集较大时,拟自适应随机森林算法的效果会好于随机森林算法。另外,拟自适应分类随机森林算法的收敛性确保它的推广误差可以通过训练集估计,所以,对于实际数据,拟自适应分类随机森林算法不需要把数据划分为训练集和测试集,从而,可以有效的利用数据信息。     

Improved Generalization via Tolerant Training

Theoretical and computational justification is given for improved generalization when the training set is learned with less accuracy. The model used for this investigation is a simple linear one. It is shown that learning a training set with a tolerance improves generalization, over zero-tolerance training, for any testing set satisfying a certain closeness condition to the training set. These results, obtained via a mathematical programming formulation, are placed in the context of some well-known machine learning results. Computational confirmation of improved generalization is given for linear systems (including nine of the twelve real-world data sets tested), as well as for nonlinear systems such as neural networks for which no theoretical results are available at present. In particular, the tolerant training method improves generalization on noisy, sparse, and overparameterized problems.     

A CS-AdaBoost-BP model for product quality inspection

Quality inspection is essential in preventing defective products from entering the market. Due to the typically low percentage of defective products, it is generally challenging to detect them using algorithms that aim for the overall classification accuracy. To help solve this problem, we propose an ensemble learning classification model, where we employ adaptive boosting (AdaBoost) to cascade multiple backpropagation (BP) neural networks. Furthermore, cost-sensitive (CS) learning is introduced to adjust the loss function of the basic classifier of the BP neural network. For clarity, this model is called a CS-AdaBoost-BP model. To empirically verify its effectiveness, we use data from home appliance production lines from Bosch. We carry out tenfold cross-validation to evaluate and compare the performance between the CS-AdaBoost-BP model and three existing models: BP neural network, BP neural network based on sampling, and AdaBoost-BP. The results show that our proposed model not only performs better than the other models but also significantly improves the ability to identify defective products. Furthermore, based on the mean value of the Youden index, our proposed model has the highest stability.

     

Adaptive network-based fuzzy inference system with leave-one-out cross-validation approach for prediction of surface roughness

The prediction of surface roughness is a challengeable problem. In order to improve the prediction accuracy in end milling process, an improved approach is proposed to model surface roughness with adaptive network-based fuzzy inference system (ANFIS) and leave-one-out cross-validation (LOO-CV) approach. This approach focuses on both architecture and parameter optimization. LOO-CV, which is an effective measure to evaluate the generalization capability of mode, is employed to find the most suitable membership function and the optimal rule base of ANFIS model for the issue of surface roughness prediction. To find the optimal rule base of ANFIS, a new “top down” rules reduction method is suggested. Three machining parameters, the spindle speed, feed rate and depth of cut are used as inputs in the model. Based on the same experimental data, the predictive results of ANFIS with LOO-CV are compared with the results reported recently in the literature and ANFIS with clustering methods. The comparisons indicate that the presented approach outperforms the opponent methods, and the prediction accuracy can be improved to 96.38%. ANFIS with LOO-CV approach is an effective approach for prediction of surface roughness in end milling process.     

Convergence of Online Gradient Method with Penalty for BP Neural Networks

Online gradient method has been widely used as a learning algorithm for training feedforward neural networks. Penalty is often introduced into the training procedure to improve the generalization performance and to decrease the magnitude of network weights. In this paper, some weight boundedness and deterministic con- vergence theorems are proved for the online gradient method with penalty for BP neural network with a hidden layer, assuming that the training samples are supplied with the network in a fixed order within each epoch. The monotonicity of the error function with penalty is also guaranteed in the training iteration. Simulation results for a 3-bits parity problem are presented to support our theoretical results.     

Environmentally induced amplitude death and firing provocation in large-scale networks of neuronal systems

We study the dynamics of multielement neuronal systems taking into account both the direct interaction between the cells via linear coupling and nondiffusive cell-to-cell communication via common environment. For the cells exhibiting individual bursting behavior, we have revealed the dependence of the network activity on its scale. Particularly, we show that small-scale networks demonstrate the inability to maintain complicated oscillations: for a small number of elements in an ensemble, the phenomenon of amplitude death is observed. The existence of threshold network scales and mechanisms causing firing in artificial and real multielement neural networks, as well as their significance for biological applications, are discussed.     

Optimization of the Local Search in the Training for SAMANN Neural Network

In this paper, we discuss the visualization of multidimensional data. A well-known procedure for mapping data from a high-dimensional space onto a lower-dimensional one is Sammon’s mapping. This algorithm preserves as well as possible all interpattern distances. We investigate an unsupervised backpropagation algorithm to train a multilayer feed-forward neural network (SAMANN) to perform the Sammon’s nonlinear projection. Sammon mapping has a disadvantage. It lacks generalization, which means that new points cannot be added to the obtained map without recalculating it. The SAMANN network offers the generalization ability of projecting new data, which is not present in the original Sammon’s projection algorithm. To save computation time without losing the mapping quality, we need to select optimal values of control parameters. In our research the emphasis is put on the optimization of the learning rate. The experiments are carried out both on artificial and real data. Two cases have been analyzed: (1) training of the SAMANN network with full data set, (2) retraining of the network when the new data points appear.     

基于辛格式的深度哈密尔顿神经网络

HNN是一类基于物理先验学习哈密尔顿系统的神经网络.本文通过误差分析解释使用不同积分器作为超参数对HNN的影响.如果我们把网络目标定义为在任意训练集上损失为零的映射,那么传统的积分器无法保证HNN存在网络目标.我们引进反修正方程,并严格证明基于辛格式的HNN具有网络目标,且它与原哈密尔顿量之差依赖于数值格式的精度.数值实验表明,由辛HNN得到的哈密尔顿系统的相流不能精确保持原哈密尔顿量,但保持网络目标;网络目标在训练集、测试集上的损失远小于原哈密尔顿量的损失;在预测问题上辛HNN较非辛HNN具备更强大的泛化能力和更高的精度.因此,辛格式对于HNN是至关重要的.     

Image compression with a dynamic autoassociative neural network

Image compression using neural networks has been attempted with some promise. Among the architectures, feedforward backpropagation networks (FFBPN) have been used in several attempts. Although it is demonstrated that using the mean quadratic error function is equivalent to applying the Karhunen-Loeve transformation method, promise still arises from directed learning possibilities, generalization abilities and performance of the network once trained. In this paper we propose an architecture and an improved training method to attempt to solve some of the shortcomings of traditional data compression systems based on feedforward neural networks trained with backpropagation—the dynamic autoassociation neural network (DANN).The successful application of neural networks to any task requires proper training of the network. In this research, this issue is taken as the main consideration in the design of DANN. We emphasize the convergence of the learning process proposed by DANN. This process provides an escape mechanism, by adding neurons in a random state, to avoid the local minima trapping seen in traditional PFBPN. Also, DANN's training algorithm constrains the error for every pattern to an allowed interval to balance the training for every pattern, thus improving the performance rates in recognition and generalization. The addition of these two mechanisms to DANN's training algorithm has the result of improving the final quality of the images processed by DANN.The results of several tasks presented to DANN-based compression are compared and contrasted with the performance of an FFBPN-based system applied to the same task. These results indicate that DANN is superior to FFBPN when applied to image compression.     

白骨化尸体死亡时间的推断

本文分析了15具白骨化尸体标本的股骨汞(Hg),铅(Pb),镉(Cd)元素含量数据,在三年的时间内采集了3次,一共收集到45个数据。首先将这组数据看着纵向数据,利用线性随机效应混合模型、Cox随机混合效应模型进行分析,结果显示,如果对每个白骨化尸体标本建立线性模型,可以精确预测出死亡时间,而且不需要采集铅元素含量数据。混合效应模型的预测效果也很好,最大误差不会超过1个月。其次我们对数据不作任何假设,利用机器学习中随机森林方法分析数据,并利用5折交叉验证方法来判断结果的可靠性,训练集和测试集的NMSE分别为0.1205944,0.5604286,因此可以用训练出的模型来预测死亡时间。     

Toward a solution of a class of non-linear stochastic perturbed PDEs using automated WHEP algorithm

This paper introduces a generalization and automation of the Wiener Hermite expansion with perturbation (WHEP) technique to solve a class of stochastic nonlinear partial differential equations with a perturbed nonlinearity. The automated algorithm generates the deterministic resultant linear equations according to the application of a general linear differential operator and the input parameters. Sample output with different nonlinearities, orders and corrections are presented. The resultant equations are solved numerically and the ensemble average and variance are computed and compared with previous research work. Higher order solutions with higher corrections are computed to show the importance of the generalization of the WHEP technique. The current work extends the use of WHEP for solving stochastic nonlinear differential equations.     

A comparative analysis of optimization and generalization properties of two-layer neural network and random feature models under gradient descent dynamics

A fairly comprehensive analysis is presented for the gradient descent dynamics for training two-layer neural network models in the situation when the parameters in both layers are updated. General initialization schemes as well as general regimes for the network width and training data size are considered. In the overparametrized regime, it is shown that gradient descent dynamics can achieve zero training loss exponentially fast regardless of the quality of the labels. In addition, it is proved that throughout the training process the functions represented by the neural network model are uniformly close to those of a kernel method. For general values of the network width and training data size, sharp estimates of the generalization error are established for target functions in the appropriate reproducing kernel Hilbert space.     

阿尔茨海默病诊断的自适应BP神经网络模型

阿尔茨海默病(AD)和轻度认知功能损伤(MCI)具有患者多、诊断难的特点,改进BP神经网络,提出自适应BP神经网络(ABP)进行100次AD和MCI诊断模拟,ABP神经网络的诊断正确率显著高于BP和RBF神经网络.采用留一法将101例正常人、200例MCI和90例AD患者的样本分为训练集和检测集,用ABP神经网络对其进行诊断模拟,总正确率达到73.91%.     

对部分线性模型用惩罚最小二乘估计时最优光滑化的注记

部分线性模型也就是响应变量关于一个或者多个协变量是线性的, 但对于其他的协变量是非线性的关系\bd 对于部分线性模型中的参数和非参数部分的估计方法, 惩罚最小二乘估计是重要的估计方法之一\bd 对于这种估计方法, 广义交叉验证法提供了一种确定光滑参数的方法\bd 但是, 在部分线性模型中, 用广义交叉验证法确定光滑参数的最优性还没有被证明\bd 本文证明了利用惩罚最小二乘估计对于部分线性模型估计时, 用广义交叉验证法选择光滑参数的最优性\bd 通过模拟验证了本文中所提出的用广义交叉验证法选择光滑参数具有很好的效果, 同时, 本文在模拟部分比较了广义交叉验证和最小二乘交叉验证的优劣.     

Discriminant Analysis When a Block of Observations is Missing

We consider the problem of classifying a p× 1 observation into one of two multivariate normal populations when the training samples contain a block of missing observations. A new classification procedure is proposed which is a linear combination of two discriminant functions, one based on the complete samples and the other on the incomplete samples. The new discriminant function is easy to use. We consider the estimation of error rate of the linear combination classification procedure by using the leave-one-out estimation and bootstrap estimation. A Monte Carlo study is conducted to evaluate the error rate and the estimation of it. A numerical example is given tof illustrate its use.     

A Note on the Optimality of Generalized Cross-validation Bandwidth Selection in Partially Linear Models with Kernel Smoothing Estimator

The issue of selection of bandwidth in kernel smoothing method is considered within the context of partially linear models, hi this paper, we study the asymptotic behavior of the bandwidth choice based on generalized cross-validation （CCV） approach and prove that this bandwidth choice is asymptotically optimal. Numerical simulation are also conducted to investigate the empirical performance of generalized cross-valldation.     

Time series forecasting with neural network ensembles: an application for exchange rate prediction

This paper investigates the use of neural network combining methods to improve time series forecasting performance of the traditional single keep-the-best (KTB) model. The ensemble methods are applied to the difficult problem of exchange rate forecasting. Two general approaches to combining neural networks are proposed and examined in predicting the exchange rate between the British pound and US dollar. Specifically, we propose to use systematic and serial partitioning methods to build neural network ensembles for time series forecasting. It is found that the basic ensemble approach created with non-varying network architectures trained using different initial random weights is not effective in improving the accuracy of prediction while ensemble models consisting of different neural network structures can consistently outperform predictions of the single ‘best’ network. Results also show that neural ensembles based on different partitions of the data are more effective than those developed with the full training data in out-of-sample forecasting. Moreover, reducing correlation among forecasts made by the ensemble members by utilizing data partitioning techniques is the key to success for the neural ensemble models. Although our ensemble methods show considerable advantages over the traditional KTB approach, they do not have significant improvement compared to the widely used random walk model in exchange rate forecasting.