学习算法的稳定性与泛化(Ⅱ):有界差分稳定框架

根据有界差分条件,提出了学习算法的有界差分稳定框架.依据新框架,研究了机器学习阈值选择算法,再生核Hilbert空间中的正则化学习算法,Ranking学习算法和Bagging算法,证明了对应学习算法的有界差分稳定性.所获结果断言了这些算法均具有有界差分稳定性,从而为这些算法的应用奠定了理论基础.     

The Design and Analysis of Benchmark Experiments

The assessment of the performance of learners by means of benchmark experiments is an established exercise. In practice, benchmark studies are a tool to compare the performance of several competing algorithms for a certain learning problem. Cross-validation or resampling techniques are commonly used to derive point estimates of the performances which are compared to identify algorithms with good properties. For several benchmarking problems, test procedures taking the variability of those point estimates into account have been suggested. Most of the recently proposed inference procedures are based on special variance estimators for the cross-validated performance. We introduce a theoretical framework for inference problems in benchmark experiments and show that standard statistical test procedures can be used to test for differences in the performances. The theory is based on well-defined distributions of performance measures which can be compared with established tests. To demonstrate the usefulness in practice, the theoretical results are applied to regression and classification benchmark studies based on artificial and real world data.     

Novel biogeography-based optimization algorithm with hybrid migration and global-best Gaussian mutation

The Biogeography-Based Optimization algorithm and its variants have been used widely for optimization problems. To get better performance, a novel Biogeography-Based Optimization algorithm with Hybrid migration and global-best Gaussian mutation is proposed in this paper. Firstly, a linearly dynamic random heuristic crossover strategy and an exponentially dynamic random differential mutation one are presented to form a hybrid migration operator, and the former is used to get stronger local search ability and the latter strengthen the global search ability. Secondly, a new global-best Gaussian mutation operator is put forward to balance exploration and exploitation better. Finally, a random opposition learning strategy is merged to avoid getting stuck in local optima. The experiments on the classical benchmark functions and the complexity functions from CEC-2013 and CEC-2017 test sets, and the Wilcoxon, Bonferroni-Holm and Friedman statistical tests are used to evaluate our algorithm. The results show that our algorithm obtains better performance and faster running speed compared with quite a few state-of-the-art competitive algorithms. In addition, experimental results on Minimum Spanning Tree and K-means clustering optimization show that our algorithm can cope with these two problems better than the comparison algorithms.     

Decision making with imprecise parameters

We analyze the impact of imprecise parameters on performance of an uncertainty-modeling tool presented in this paper. In particular, we present a reliable and efficient uncertainty-modeling tool, which enables dynamic capturing of interval-valued clusters representations sets and functions using well-known pattern recognition and machine learning algorithms. We mainly deal with imprecise learning parameters in identifying uncertainty intervals of membership value distributions and imprecise functions. In the experiments, we use the proposed system as a decision support tool for a production line process. Simulation results indicate that in comparison to benchmark methods such as well-known type-1 and type-2 system modeling tools, and statistical machine-learning algorithms, proposed interval-valued imprecise system modeling tool is more robust with less error.     

Global Optimization by Monotonic Transformation

This paper addresses the problem of global optimization by means of a monotonic transformation. With an observation on global optimality of functions under such a transformation, we show that a simple and effective algorithm can be derived to search within possible regions containing the global optima. Numerical experiments are performed to compare this algorithm with one that does not incorporate transformed information using several benchmark problems. These results are also compared to best known global search algorithms in the literature. In addition, the algorithm is shown to be useful for several neural network learning problems, which possess much larger parameter spaces.     

A global best artificial bee colony algorithm for global optimization

The artificial bee colony (ABC) algorithm is a relatively new optimization technique which has been shown to be competitive to other population-based algorithms. However, there is still an insufficiency in the ABC algorithm regarding its solution search equation, which is good at exploration but poor at exploitation. Inspired by differential evolution (DE), we propose a modified ABC algorithm (denoted as ABC/best), which is based on that each bee searches only around the best solution of the previous iteration in order to improve the exploitation. In addition, to enhance the global convergence, when producing the initial population and scout bees, both chaotic systems and opposition-based learning method are employed. Experiments are conducted on a set of 26 benchmark functions. The results demonstrate good performance of ABC/best in solving complex numerical optimization problems when compared with two ABC based algorithms.     

Learning theory: stability is sufficient for generalization and necessary and sufficient for consistency of empirical risk minimization

Solutions of learning problems by Empirical Risk Minimization (ERM) – and almost-ERM when the minimizer does not exist – need to be consistent, so that they may be predictive. They also need to be well-posed in the sense of being stable, so that they might be used robustly. We propose a statistical form of stability, defined as leave-one-out (LOO) stability. We prove that for bounded loss classes LOO stability is (a) sufficient for generalization, that is convergence in probability of the empirical error to the expected error, for any algorithm satisfying it and, (b) necessary and sufficient for consistency of ERM. Thus LOO stability is a weak form of stability that represents a sufficient condition for generalization for symmetric learning algorithms while subsuming the classical conditions for consistency of ERM. In particular, we conclude that a certain form of well-posedness and consistency are equivalent for ERM. Dedicated to Charles A. Micchelli on his 60th birthday Mathematics subject classifications (2000) 68T05, 68T10, 68Q32, 62M20. Tomaso Poggio: Corresponding author.     

Multiple kernel boosting framework based on information measure for classification

The performance of kernel-based method, such as support vector machine (SVM), is greatly affected by the choice of kernel function. Multiple kernel learning (MKL) is a promising family of machine learning algorithms and has attracted many attentions in recent years. MKL combines multiple sub-kernels to seek better results compared to single kernel learning. In order to improve the efficiency of SVM and MKL, in this paper, the Kullback–Leibler kernel function is derived to develop SVM. The proposed method employs an improved ensemble learning framework, named KLMKB, which applies Adaboost to learning multiple kernel-based classifier. In the experiment for hyperspectral remote sensing image classification, we employ feature selected through Optional Index Factor (OIF) to classify the satellite image. We extensively examine the performance of our approach in comparison to some relevant and state-of-the-art algorithms on a number of benchmark classification data sets and hyperspectral remote sensing image data set. Experimental results show that our method has a stable behavior and a noticeable accuracy for different data set.     

Generalization bounds of ERM algorithm with Markov chain samples

One of the main goals of machine learning is to study the generalization performance of learning algorithms. The previous main results describing the generalization ability of learning algorithms are usually based on independent and identically distributed （i.i.d.） samples. However, independence is a very restrictive concept for both theory and real-world applications. In this paper we go far beyond this classical framework by establishing the bounds on the rate of relative uniform convergence for the Empirical Risk Minimization （ERM） algorithm with uniformly ergodic Markov chain samples. We not only obtain generalization bounds of ERM algorithm, but also show that the ERM algorithm with uniformly ergodic Markov chain samples is consistent. The established theory underlies application of ERM type of learning algorithms.     

Target re-identification based on adaptive incremental KISS measure learning

Target re-identification from across cameras is a difficult problem in multi-camera surveillance, which needs to be urgently solved. Traditional solutions, in addition to relying on the statistical characteristics of targets’ appearance, are more often using excellent measurement algorithms. Among many such algorithms, the Keep It Simple and Stupid Measure Learning (KISSME) algorithm based on statistical probability is an outstanding one. But it has a problem that the eigenvalue is not stable, and the actual matching rate is relatively low. So, in this paper, we optimize the measurement algorithms based on large scale Keep It Simple and Stupid (KISS) measure learning. From elements, such as inadequate sample, size and smaller or larger eigenvalues, we introduce eigenvalue stabilization technique, and finally form our algorithm which can be called Adaptive Incremental Keep It Simple and Stupid Measure Learning (AIKISSME). Finally, through many experiments based on Viewpoint Invariant Pedestrian Recognition (VIPeR) and by comparing with other algorithms, this work concludes that AIKISSME achieves the best overall performance.     

A novel multi-objective memetic algorithm based on opposition-based self-adaptive differential evolution

Under the framework of evolutionary paradigms, many evolutionary algorithms have been designed for handling multi-objective optimization problems. Each of the different algorithms may display exceptionally good performance in certain optimization problems, but none of them can be completely superior over one another. As such, different evolutionary algorithms are being synthesized to complement each other in view of their strengths and the limitations inherent in them. In this study, the novel memetic algorithm known as the Opposition-based Self-adaptive Hybridized Differential Evolution algorithm (OSADE) is being comprehensively investigated through a comparative study with some state-of-the-art algorithms, such as NSGA-II, non-dominated sorting Differential Evolution (NSDE), MOEA/D-SBX, MOEA/D-DE and the Multi-objective Evolutionary Gradient Search (MO-EGS) by using a suite of different benchmark problems. Through the experimental results that are presented by employing the Inverted Generational Distance (IGD) and the Hausdorff Distance performance indicators, it is seen that OSADE is able to achieve competitive, if not better, performance when compared to the other algorithms in this study.     

Experimental Testing of Advanced Scatter Search Designs for Global Optimization of Multimodal Functions

Scatter search is an evolutionary method that, unlike genetic algorithms, operates on a small set of solutions and makes only limited use of randomization as a proxy for diversification when searching for a globally optimal solution. The scatter search framework is flexible, allowing the development of alternative implementations with varying degrees of sophistication. In this paper, we test the merit of several scatter search designs in the context of global optimization of multimodal functions. We compare these designs among themselves and choose one to compare against a well-known genetic algorithm that has been specifically developed for this class of problems. The testing is performed on a set of benchmark multimodal functions with known global minima.     

AN ADAPTIVE MEMBRANE ALGORITHM FOR SOLVING COMBINATORIAL OPTIMIZATION PROBLEMS

Membrane algorithms （MAs）, which inherit from P systems, constitute a new parallel and distribute framework for approximate computation. In the paper, a membrane algorithm is proposed with the improvement that the involved parameters can be adaptively chosen. In the algorithm, some membranes can evolve dynamically during the computing process to specify the values of the requested parameters. The new algorithm is tested on a well-known combinatorial optimization problem, the travelling salesman problem. The em-pirical evidence suggests that the proposed approach is efficient and reliable when dealing with 11 benchmark instances, particularly obtaining the best of the known solutions in eight instances. Compared with the genetic algorithm, simulated annealing algorithm, neural net-work and a fine-tuned non-adaptive membrane algorithm, our algorithm performs better than them. In practice, to design the airline network that minimize the total routing cost on the CAB data with twenty-five US cities, we can quickly obtain high quality solutions using our algorithm.     

An adaptive membrane algorithm for solving combinatorial optimization problems

Membrane algorithms (MAs), which inherit from P systems, constitute a new parallel and distribute framework for approximate computation. In the paper, a membrane algorithm is proposed with the improvement that the involved parameters can be adaptively chosen. In the algorithm, some membranes can evolve dynamically during the computing process to specify the values of the requested parameters. The new algorithm is tested on a well-known combinatorial optimization problem, the travelling salesman problem. The empirical evidence suggests that the proposed approach is efficient and reliable when dealing with 11 benchmark instances, particularly obtaining the best of the known solutions in eight instances. Compared with the genetic algorithm, simulated annealing algorithm, neural network and a fine-tuned non-adaptive membrane algorithm, our algorithm performs better than them. In practice, to design the airline network that minimize the total routing cost on the CAB data with twenty-five US cities, we can quickly obtain high quality solutions using our algorithm.     

Robust generalized eigenvalue classifier with ellipsoidal uncertainty

Uncertainty is a concept associated with data acquisition and analysis, usually appearing in the form of noise or measure error, often due to some technological constraint. In supervised learning, uncertainty affects classification accuracy and yields low quality solutions. For this reason, it is essential to develop machine learning algorithms able to handle efficiently data with imprecision. In this paper we study this problem from a robust optimization perspective. We consider a supervised learning algorithm based on generalized eigenvalues and we provide a robust counterpart formulation and solution in case of ellipsoidal uncertainty sets. We demonstrate the performance of the proposed robust scheme on artificial and benchmark datasets from University of California Irvine (UCI) machine learning repository and we compare results against a robust implementation of Support Vector Machines.     

Parameter adaptive harmony search algorithm for unimodal and multimodal optimization problems

This paper presents a parameter adaptive harmony search algorithm (PAHS) for solving optimization problems. The two important parameters of harmony search algorithm namely Harmony Memory Consideration Rate (HMCR) and Pitch Adjusting Rate (PAR), which were either kept constant or the PAR value was dynamically changed while still keeping HMCR fixed, as observed from literature, are both being allowed to change dynamically in this proposed PAHS. This change in the parameters has been done to get the global optimal solution. Four different cases of linear and exponential changes have been explored. The change has been allowed during the process of improvization. The proposed algorithm is evaluated on 15 standard benchmark functions of various characteristics. Its performance is investigated and compared with three existing harmony search algorithms. Experimental results reveal that proposed algorithm outperforms the existing approaches when applied to 15 benchmark functions. The effects of scalability, noise, and harmony memory size have also been investigated on four approaches of HS. The proposed algorithm is also employed for data clustering. Five real life datasets selected from UCI machine learning repository are used. The results show that, for data clustering, the proposed algorithm achieved results better than other algorithms.     

Two-stage based ensemble optimization framework for large-scale global optimization

Large-scale global optimization (LSGO) is a very important and challenging task in optimization domain, which is embedded in many scientific and engineering applications. In order to strengthen both effectiveness and efficiency of LSGO algorithm, this paper designs a two-stage based ensemble optimization evolutionary algorithm (EOEA) framework, which serially implements two sub-optimizers. These two sub-optimizers mainly focus on exploration and exploitation separately. The EOEA framework can be easily generated, flexibly altered and modified, according to different implementation conditions. In order to analyze the effects of EOEA’s components, we compare its performance on diverse kinds of problems with its two sub-optimizers and three variants. To show its superiorities over the previous LSGO algorithms, we compare its performance with six classical LSGO algorithms on the LSGO test functions of IEEE Congress of Evolutionary Computation (CEC 2008). The performance of EOEA is further evaluated by experimental comparison with four state-of-the-art LSGO algorithms on the test functions of CEC 2010 LSGO competition. To benchmark the practical applicability of EOEA, we adopt EOEA to the parameter calibration problem of water pipeline system. Based on the experimental results on diverse scales of systems, EOEA performs steadily and robustly.     

Solving a Network Design Problem

Industrial optimization applications must be “robust” i.e., they must provide good solutions to problem instances of different size and numerical characteristics, and continue to work well when side constraints are added. This paper presents a case study that addresses this requirement and its consequences on the applicability of different optimization techniques. An extensive benchmark suite, built on real network design data, is used to test multiple algorithms for robustness against variations in problem size, numerical characteristics, and side constraints. The experimental results illustrate the performance discrepancies that have occurred and how some have been corrected. In the end, the results suggest that we shall remain very humble when assessing the adequacy of a given algorithm for a given problem, and that a new generation of public optimization benchmark suites is needed for the academic community to attack the issue of algorithm robustness as it is encountered in industrial settings.     

A General Framework for Mining Massive Data Streams

In many domains, data now arrive faster than we are able to mine it. To avoid wasting these data, we must switch from the traditional “one-shot” data mining approach to systems that are able to mine continuous, high-volume, open-ended data streams as they arrive. In this article we identify some desiderata for such systems, and outline our framework for realizing them. A key property of our approach is that it minimizes the time required to build a model on a stream while guaranteeing (as long as the data are iid) that the model learned is effectively indistinguishable from the one that would be obtained using infinite data. Using this framework, we have successfully adapted several learning algorithms to massive data streams, including decision tree induction, Bayesian network learning, k-means clustering, and the EM algorithm for mixtures of Gaussians. These algorithms are able to process on the order of billions of examples per day using off-the-shelf hardware. Building on this, we are currently developing software primitives for scaling arbitrary learning algorithms to massive data streams with minimal effort.     

An alternating direction and projection algorithm for structure-enforced matrix factorization

Structure-enforced matrix factorization (SeMF) represents a large class of mathematical models appearing in various forms of principal component analysis, sparse coding, dictionary learning and other machine learning techniques useful in many applications including neuroscience and signal processing. In this paper, we present a unified algorithm framework, based on the classic alternating direction method of multipliers (ADMM), for solving a wide range of SeMF problems whose constraint sets permit low-complexity projections. We propose a strategy to adaptively adjust the penalty parameters which is the key to achieving good performance for ADMM. We conduct extensive numerical experiments to compare the proposed algorithm with a number of state-of-the-art special-purpose algorithms on test problems including dictionary learning for sparse representation and sparse nonnegative matrix factorization. Results show that our unified SeMF algorithm can solve different types of factorization problems as reliably and as efficiently as special-purpose algorithms. In particular, our SeMF algorithm provides the ability to explicitly enforce various combinatorial sparsity patterns that, to our knowledge, has not been considered in existing approaches.