A hybrid algorithm to optimize RBF network architecture and parameters for nonlinear time series prediction

This paper proposes a novel hybrid algorithm for automatic selection of the proper input variables, the number of hidden nodes of the radial basis function (RBF) network, and optimizing network parameters (weights, centers and widths) simultaneously. In the proposed algorithm, the inputs and the number of hidden nodes of the RBF network are represented by binary-coded strings and evolved by a genetic algorithm (GA). Simultaneously, for each chromosome with fixed inputs and number of hidden nodes, the corresponding parameters of the network are real-coded and optimized by a gradient-based fast-converging parameter estimation method. Performance of the presented hybrid approach is evaluated by several benchmark time series modeling and prediction problems. Experimental results show that the proposed approach produces parsimonious RBF networks, and obtains better modeling accuracy than some other algorithms.     

Model selection in Neural Networks: Some difficulties

This paper considers two related issues regarding feedforward Neural Networks (NNs). The first involves the question of whether the network weights corresponding to the best fitting network are unique. Our empirical tests suggest an answer in the negative, whether using standard Backpropagation algorithm or our preferred direct (non-gradient-based) search procedure. We also offer a theoretical analysis which suggests that there will almost inevitably be functional relationships between network weights. The second issue concerns the use of standard statistical approaches to testing the significance of weights or groups of weights. Treating feedforward NNs as an interesting way to carry out nonlinear regression suggests that statistical tests should be employed. According to our results, however, statistical tests can in practice be indeterminate. It is rather difficult to choose either the number of hidden layers or the number of nodes on this basis.     

Global sensitivity analysis using support vector regression

Global sensitivity analysis (GSA) plays an important role in exploring the respective effects of input variables on response variables. In this paper, a new kernel function derived from orthogonal polynomials is proposed for support vector regression (SVR). Based on this new kernel function, the Sobol’ global sensitivity indices can be computed analytically by the coefficients of the surrogate model built by SVR. In order to improve the performance of the SVR model, a kernel function iteration scheme is introduced further. Due to the excellent generalization performance and structural risk minimization principle, the SVR possesses the advantages of solving non-linear prediction problems with small samples. Thus, the proposed method is capable of computing the Sobol’ indices with a relatively limited number of model evaluations. The proposed method is examined by several examples, and the sensitivity analysis results are compared with the sparse polynomial chaos expansion (PCE), high dimensional model representation (HDMR) and Gaussian radial basis (RBF) SVR model. The examined examples show that the proposed method is an efficient approach for GSA of complex models.     

Gaussian kernel approximation algorithm for feedforward neural network design

A Gaussian kernel approximation algorithm for a feedforward neural network is presented. The approach used by the algorithm, which is based on a constructive learning algorithm, is to create the hidden units directly so that automatic design of the architecture of neural networks can be carried out. The algorithm is defined using the linear summation of input patterns and their randomized input weights. Hidden-layer nodes are defined so as to partition the input space into homogeneous regions, where each region contains patterns belonging to the same class. The largest region is used to define the center of the corresponding Gaussian hidden nodes. The algorithm is tested on three benchmark data sets of different dimensionality and sample sizes to compare the approach presented here with other algorithms. Real medical diagnoses and a biological classification of mushrooms are used to illustrate the performance of the algorithm. These results confirm the effectiveness of the proposed algorithm.     

Switching Control of Nonlinear Systems Based on the Quasi-ARX Model and the SVR Algorithm北大核心CSCD

该文基于改进的含有外部输入项的准线性自回归(准ARX)径向基函数(RBF)网络模型和支持向量回归(SVR)算法,提出了一种非线性切换控制方法.改进的准ARX模型非线性部分采用RBF网络.控制系统设计过程分为三个部分:首先,利用聚类方法确定模型的非线性参数;然后,采用线性SVR算法来解决控制系统的鲁棒性问题;接下来,基于控制误差给出切换判定函数,确定切换律给出控制序列.最后通过数值仿真验证了该方法的有效性.     

Minimax designs for linear regression models with bias in a reproducing kernel Hilbert space in a discrete set

Consider the design problem for estimation and extrapolation in approximately linear regression models with possible misspecification. The design space is a discrete set consisting of finitely many points, and the model bias comes from a reproducing kernel Hilbert space. Two different design criteria are proposed by applying the minimax approach for estimating the parameters of the regression response and extrapolating the regression response to points outside of the design space. A simulated annealing algorithm is applied to construct the minimax designs.These minimax designs are compared with the classical D-optimal designs and all-bias extrapolation designs. Numerical results indicate that the simulated annealing algorithm is feasible and the minimax designs are robust against bias caused by model misspecification.     

基于支持向量回归和核岭回归对血糖值预测的对比分析

为了对比支持向量回归(SVR)和核岭回归(KRR)预测血糖值的效果,本文选择人工智能辅助糖尿病遗传风险的相关数据进行实证分析.首先对数据进行预处理,将处理后的数据导入Python.其次,为了使SVR和KRR的对比结果具有客观性,使用了三种有代表性的核方法(线性核函数,径向基核函数和sigmod核函数).然后,在训练集上采用网格搜索自动调参分别建立SVR和KRR的最优模型,对血糖值进行预测.最后,在测试集上对比分析SVR和KRR预测的均方误差(MSE)和拟合时间等指标.结果表明:均方误差(MSE)都小于0.006,且KRR的MSE比SVR的小0.0002,KRR的预测精度比SVR更高;而SVR的预测时间比KRR的少0.803秒,SVR的预测效率比KRR好.     

Volatility forecasting via SVR–GARCH with mixture of Gaussian kernels

The support vector regression (SVR) is a supervised machine learning technique that has been successfully employed to forecast financial volatility. As the SVR is a kernel-based technique, the choice of the kernel has a great impact on its forecasting accuracy. Empirical results show that SVRs with hybrid kernels tend to beat single-kernel models in terms of forecasting accuracy. Nevertheless, no application of hybrid kernel SVR to financial volatility forecasting has been performed in previous researches. Given that the empirical evidence shows that the stock market oscillates between several possible regimes, in which the overall distribution of returns it is a mixture of normals, we attempt to find the optimal number of mixture of Gaussian kernels that improve the one-period-ahead volatility forecasting of SVR based on GARCH(1,1). The forecast performance of a mixture of one, two, three and four Gaussian kernels are evaluated on the daily returns of Nikkei and Ibovespa indexes and compared with SVR–GARCH with Morlet wavelet kernel, standard GARCH, Glosten–Jagannathan–Runkle (GJR) and nonlinear EGARCH models with normal, student-t, skew-student-t and generalized error distribution (GED) innovations by using mean absolute error (MAE), root mean squared error (RMSE) and robust Diebold–Mariano test. The results of the out-of-sample forecasts suggest that the SVR–GARCH with a mixture of Gaussian kernels can improve the volatility forecasts and capture the regime-switching behavior.     

Rotorcraft parameter estimation using radial basis function neural network

Increased emphasis on rotorcraft performance and operational capabilities has resulted in accurate computation of aerodynamic stability and control parameters. System identification is one such tool in which the model structure and parameters such as aerodynamic stability and control derivatives are derived. In the present work, the rotorcraft aerodynamic parameters are computed using radial basis function neural networks (RBFN) in the presence of both state and measurement noise. The effect of presence of outliers in the data is also considered. RBFN is found to give superior results compared to finite difference derivatives for noisy data.     

基于散乱阈值结点的周期神经网络的逼近阶

§1Introduction Inrecentyearstherehasbeengrowinginterestintheproblemofneuralnetworkand relatedapproximation,manyimportantresultsareobtained.Becauseofitsabilityof parallelcomputationinlargescaleandofperfectself-adaptingandapproximation,the neuralnetworkhasbeenwidelyapplied.Theapproximationabilityoftheneuralnetwork dependsonitstopologicalstructure.LetRsbeans-dimensionalEuclidSpaceand(x)isa realfunctiondefinedonRs.When(x)isanexcitationfunctionandx∈Rsisaninput vector,thesimpleneuralnetwork…     

A robust support vector regression with a linear-log concave loss function

Support vector regression (SVR) is one of the most popular nonlinear regression techniques with the aim to approximate a nonlinear system with a good generalization capability. However, SVR has a major drawback in that it is sensitive to the presence of outliers. The ramp loss function for robust SVR has been introduced to resolve this problem, but SVR with ramp loss function has a non-differentiable and non-convex formulation, which is not easy to solve. Consequently, SVR with the ramp loss function requires smoothing and Concave-Convex Procedure techniques, which transform the non-differentiable and non-convex optimization to a differentiable and convex one. We present a robust SVR with linear-log concave loss function (RSLL), which does not require the transformation technique, where the linear-log concave loss function has a similar effect as the ramp loss function. The zero norm approximation and the difference of convex functions problem are employed for solving the optimization problem. The proposed RSLL approach is used to develop a robust and stable virtual metrology (VM) prediction model, which utilizes the status variables of process equipment to predict the process quality of wafer level in semiconductor manufacturing. We also compare the proposed approach to existing SVR-based methods in terms of the root mean squared error of prediction using both synthetic and real data sets. Our experimental results show that the proposed approach performs better than existing SVR-based methods regardless of the data set and type of outliers (ie, X-space and Y-space outliers), implying that it can be used as a useful alternative when the regression data contain outliers.     

Efficient robust estimation of time-series regression models

The paper studies a new class of robust regression estimators based on the two-step least weighted squares (2S-LWS) estimator which employs data-adaptive weights determined from the empirical distribution or quantile functions of regression residuals obtained from an initial robust fit. Just like many existing two-step robust methods, the proposed 2S-LWS estimator preserves robust properties of the initial robust estimate. However, contrary to the existing methods, the first-order asymptotic behavior of 2S-LWS is fully independent of the initial estimate under mild conditions. We propose data-adaptive weighting schemes that perform well both in the cross-section and time-series data and prove the asymptotic normality and efficiency of the resulting procedure. A simulation study documents these theoretical properties in finite samples.     

Hybrid evolutionary algorithms in a SVR traffic flow forecasting model

Accurate urban traffic flow forecasting is critical to intelligent transportation system developments and implementations, thus, it has been one of the most important issues in the research on road traffic congestion. Due to complex nonlinear data pattern of the urban traffic flow, there are many kinds of traffic flow forecasting techniques in literature, thus, it is difficult to make a general conclusion which forecasting technique is superior to others. Recently, the support vector regression model (SVR) has been widely used to solve nonlinear regression and time series problems. This investigation presents a SVR traffic flow forecasting model which employs the hybrid genetic algorithm-simulated annealing algorithm (GA-SA) to determine its suitable parameter combination. Additionally, a numerical example of traffic flow data from northern Taiwan is used to elucidate the forecasting performance of the proposed SVRGA-SA model. The forecasting results indicate that the proposed model yields more accurate forecasting results than the seasonal autoregressive integrated moving average (SARIMA), back-propagation neural network (BPNN), Holt-Winters (HW) and seasonal Holt-Winters (SHW) models. Therefore, the SVRGA-SA model is a promising alternative for forecasting traffic flow.     

Robust nonparametric regression estimation

In this paper we define a robust conditional location functional without requiring any moment condition. We apply the nonparametric proposals considered by C. Stone (Ann. Statist. 5 (1977), 595–645) to this functional equation in order to obtain strongly consistent, robust nonparametric estimates of the regression function. We give some examples by using nearest neighbor weights or weights based on kernel methods under no assumptions whatsoever on the probability measure of the vector (X,Y). We also derive strong convergence rates and the asymptotic distribution of the proposed estimates.     

具有随机隐单元的渐增前馈神经网络的$L^2(R^d)$逼近能力

This paper studies approximation capability to L2(Rd) functions of incremental constructive feedforward neural networks(FNN) with random hidden units.Two kinds of therelayered feedforward neural networks are considered:radial basis function(RBF) neural networks and translation and dilation invariant(TDI) neural networks.In comparison with conventional methods that existence approach is mainly used in approximation theories for neural networks,we follow a constructive approach to prove that one may simply randomly choose parameters of hidden units and then adjust the weights between the hidden units and the output unit to make the neural network approximate any function in L2(Rd) to any accuracy.Our result shows given any non-zero activation function g :R+→R and g(x Rd) ∈ L2(Rd) for RBF hidden units,or any non-zero activation function g(x) ∈ L2(Rd) for TDI hidden units,the incremental network function fn with randomly generated hidden units converges to any target function in L2(Rd) with probability one as the number of hidden units n→∞,if one only properly adjusts the weights between the hidden units and output unit.     

Identification of nonlinear system using extreme learning machine based Hammerstein model

In this paper, a new method for nonlinear system identification via extreme learning machine neural network based Hammerstein model (ELM-Hammerstein) is proposed. The ELM-Hammerstein model consists of static ELM neural network followed by a linear dynamic subsystem. The identification of nonlinear system is achieved by determining the structure of ELM-Hammerstein model and estimating its parameters. Lipschitz quotient criterion is adopted to determine the structure of ELM-Hammerstein model from input–output data. A generalized ELM algorithm is proposed to estimate the parameters of ELM-Hammerstein model, where the parameters of linear dynamic part and the output weights of ELM neural network are estimated simultaneously. The proposed method can obtain more accurate identification results with less computation complexity. Three simulation examples demonstrate its effectiveness.     

Approximating Unknown Mappings: An Experimental Evaluation

Different methodologies have been introduced in recent years with the aim of approximating unknown functions. Basically, these methodologies are general frameworks for representing non-linear mappings from several input variables to several output variables. Research into this problem occurs in applied mathematics (multivariate function approximation), statistics (nonparametric multiple regression) and computer science (neural networks). However, since these methodologies have been proposed in different fields, most of the previous papers treat them in isolation, ignoring contributions in the other areas. In this paper we consider five well known approaches for function approximation. Specifically we target polynomial approximation, general additive models (Gam), local regression (Loess), multivariate additive regression splines (Mars) and artificial neural networks (Ann).Neural networks can be viewed as models of real systems, built by tuning parameters known as weights. In training the net, the problem is to find the weights that optimize its performance (i.e. to minimize the error over the training set). Although the most popular method for Ann training is back propagation, other optimization methods based on metaheuristics have recently been adapted to this problem, outperforming classical approaches. In this paper we propose a short term memory tabu search method, coupled with path relinking and BFGS (a gradient-based local NLP solver) to provide high quality solutions to this problem. The experimentation with 15 functions previously reported shows that a feed-forward neural network with one hidden layer, trained with our procedure, can compete with the best-known approximating methods. The experimental results also show the effectiveness of a new mechanism to avoid overfitting in neural network training.     

Robust Depth-Weighted Wavelet for Nonparametric Regression Models

In the nonparametric regression models, the original regression estimators including kernel estimator, Fourier series estimator and wavelet estimator are always constructed by the weighted sum of data, and the weights depend only on the distance between the design points and estimation points. As a result these estimators are not robust to the perturbations in data. In order to avoid this problem, a new nonparametric regression model, called the depth-weighted regression model, is introduced and then the depth-weighted wavelet estimation is defined. The new estimation is robust to the perturbations in data, which attains very high breakdown value close to 1/2. On the other hand, some asymptotic behaviours such as asymptotic normality are obtained. Some simulations illustrate that the proposed wavelet estimator is more robust than the original wavelet estimator and, as a price to pay for the robustness, the new method is slightly less efficient than the original method.     

检测网络社团结构的基于最短路径的$k$-均值算法

我们考虑复杂网络社团结构的检测问题,即检测出那些具有高于平均密度的边所连接的节点的集合.本文我们利用模拟退火策略来极大化可表示为稳定效益函数的模量(modularity),并结合基于最短路径的$k$-均值迭代过程来对网络进行分区.该算法不仅能检测出社团,而且能够识别出在最短路径度量下,该社团中位于中心位置的节点.社团的最优数目可以在无需任何关于网络结构的先验信息下自动确定.对人工生成网络和真实世界中的网络的成功应用表明了算法的有效性.