An interpretability improvement for fuzzy rule bases obtained by the iterative rule learning approach

Interpretability is one of the key concepts in many of the applications using the fuzzy rule-based approach. It is well known that there are many different criteria around this concept, the complexity being one of them. In this paper, we focus our efforts in reducing the complexity of the fuzzy rule sets. One of the most interesting approaches for learning fuzzy rules is the iterative rule learning approach. It is mainly characterized by obtaining rules covering few examples in final stages, being in most cases useless to represent the knowledge. This behavior is due to the specificity of the extracted rules, which eventually creates more complex set of rules. Thus, we propose a modified version of the iterative rule learning algorithm in order to extract simple rules relaxing this natural trend. The main idea is to change the rule extraction process to be able to obtain more general rules, using pruned searching spaces together with a knowledge simplification scheme able to replace learned rules. The experimental results prove that this purpose is achieved. The new proposal reduces the complexity at both, the rule and rule base levels, maintaining the accuracy regarding to previous versions of the algorithm.     

An efficient multi-objective evolutionary fuzzy system for regression problems

During the last years, multi-objective evolutionary algorithms (MOEAs) have been extensively employed as optimization tools for generating fuzzy rule-based systems (FRBSs) with different trade-offs between accuracy and interpretability from data. Since the size of the search space and the computational cost of the fitness evaluation depend on the number of input variables and instances, respectively, managing high-dimensional and large datasets is a critical issue.In this paper, we focus on MOEAs applied to learn concurrently the rule base and the data base of Mamdani FRBSs and propose to tackle the issue by exploiting the synergy between two different techniques. The first technique is based on a novel method which reduces the search space by learning rules not from scratch, but rather from a heuristically generated rule base. The second technique performs an instance selection by exploiting a co-evolutionary approach where cyclically a genetic algorithm evolves a reduced training set which is used in the evolution of the MOEA.The effectiveness of the synergy has been tested on twelve datasets. Using non-parametric statistical tests we show that, although achieving statistically equivalent solutions, the adoption of this synergy allows saving up to 97.38% of the execution time with respect to a state-of-the-art multi-objective evolutionary approach which learns rules from scratch.     

Genetic learning of accurate and compact fuzzy rule based systems based on the 2-tuples linguistic representation

One of the problems that focus the research in the linguistic fuzzy modeling area is the trade-off between interpretability and accuracy. To deal with this problem, different approaches can be found in the literature. Recently, a new linguistic rule representation model was presented to perform a genetic lateral tuning of membership functions. It is based on the linguistic 2-tuples representation that allows the lateral displacement of a label considering an unique parameter. This way to work involves a reduction of the search space that eases the derivation of optimal models and therefore, improves the mentioned trade-off.Based on the 2-tuples rule representation, this work proposes a new method to obtain linguistic fuzzy systems by means of an evolutionary learning of the data base a priori (number of labels and lateral displacements) and a simple rule generation method to quickly learn the associated rule base. Since this rule generation method is run from each data base definition generated by the evolutionary algorithm, its selection is an important aspect. In this work, we also propose two new ad hoc data-driven rule generation methods, analyzing the influence of them and other rule generation methods in the proposed learning approach. The developed algorithms will be tested considering two different real-world problems.     

A fast genetic method for inducting descriptive fuzzy models

Under certain inference mechanisms, fuzzy rule bases can be regarded as extended additive models. This relationship can be applied to extend some statistical techniques to learn fuzzy models from data. The interest in this parallelism is twofold: theoretical and practical. First, extended additive models can be estimated by means of the matching pursuit algorithm, which has been related to Support Vector Machines, Boosting and Radial Basis neural networks learning; this connection can be exploited to better understand the learning of fuzzy models. In particular, the technique we propose here can be regarded as the counterpart to boosting fuzzy classifiers in the field of fuzzy modeling. Second, since matching pursuit is very efficient in time, we can expect to obtain faster algorithms to learn fuzzy rules from data. We show that the combination of a genetic algorithm and the backfitting process learns faster than ad hoc methods in certain datasets.     

Supervised learning algorithms for multi-class classification problems with partial class memberships

In several application domains such as biology, computer vision, social network analysis and information retrieval, multi-class classification problems arise in which data instances not simply belong to one particular class, but exhibit a partial membership to several classes. Existing machine learning or fuzzy set approaches for representing this type of fuzzy information mainly focus on unsupervised methods. In contrast, we present in this article supervised learning algorithms for classification problems with partial class memberships, where class memberships instead of crisp class labels serve as input for fitting a model to the data. Using kernel logistic regression (KLR) as a baseline method, first a basic one-versus-all approach is proposed, by replacing the binary-coded label vectors with [0,1]-valued class memberships in the likelihood. Subsequently, we use this KLR extension as base classifier to construct one-versus-one decompositions, in which partial class memberships are transformed and estimated in a pairwise manner. Empirical results on synthetic data and a real-world application in bioinformatics confirm that our approach delivers promising results. The one-versus-all method yields the best computational efficiency, while the one-versus-one methods are preferred in terms of predictive performance, especially when the observed class memberships are heavily unbalanced.     

Interpretability assessment of fuzzy knowledge bases: A cointension based approach

Computing with words (CWW) relies on linguistic representation of knowledge that is processed by operating at the semantical level defined through fuzzy sets. Linguistic representation of knowledge is a major issue when fuzzy rule based models are acquired from data by some form of empirical learning. Indeed, these models are often requested to exhibit interpretability, which is normally evaluated in terms of structural features, such as rule complexity, properties on fuzzy sets and partitions. In this paper we propose a different approach for evaluating interpretability that is based on the notion of cointension. The interpretability of a fuzzy rule-based model is measured in terms of cointension degree between the explicit semantics, defined by the formal parameter settings of the model, and the implicit semantics conveyed to the reader by the linguistic representation of knowledge. Implicit semantics calls for a representation of user’s knowledge which is difficult to externalise. Nevertheless, we identify a set of properties - which we call “logical view” - that is expected to hold in the implicit semantics and is used in our approach to evaluate the cointension between explicit and implicit semantics. In practice, a new fuzzy rule base is obtained by minimising the fuzzy rule base through logical properties. Semantic comparison is made by evaluating the performances of the two rule bases, which are supposed to be similar when the two semantics are almost equivalent. If this is the case, we deduce that the logical view is applicable to the model, which can be tagged as interpretable from the cointension viewpoint. These ideas are then used to define a strategy for assessing interpretability of fuzzy rule-based classifiers (FRBCs). The strategy has been evaluated on a set of pre-existent FRBCs, acquired by different learning processes from a well-known benchmark dataset. Our analysis highlighted that some of them are not cointensive with user’s knowledge, hence their linguistic representation is not appropriate, even though they can be tagged as interpretable from a structural point of view.     

Intelligent data analysis and model interpretation with spectral analysis fuzzy symbolic modeling

This paper proposes fuzzy symbolic modeling as a framework for intelligent data analysis and model interpretation in classification and regression problems. The fuzzy symbolic modeling approach is based on the eigenstructure analysis of the data similarity matrix to define the number of fuzzy rules in the model. Each fuzzy rule is associated with a symbol and is defined by a Gaussian membership function. The prototypes for the rules are computed by a clustering algorithm, and the model output parameters are computed as the solutions of a bounded quadratic optimization problem. In classification problems, the rules’ parameters are interpreted as the rules’ confidence. In regression problems, the rules’ parameters are used to derive rules’ confidences for classes that represent ranges of output variable values. The resulting model is evaluated based on a set of benchmark datasets for classification and regression problems. Nonparametric statistical tests were performed on the benchmark results, showing that the proposed approach produces compact fuzzy models with accuracy comparable to models produced by the standard modeling approaches. The resulting model is also exploited from the interpretability point of view, showing how the rule weights provide additional information to help in data and model understanding, such that it can be used as a decision support tool for the prediction of new data.     

Online identification of Takagi–Sugeno fuzzy models based on self-adaptive hierarchical particle swarm optimization algorithm

This paper presents an approach for online learning of Takagi–Sugeno (T-S) fuzzy models. A novel learning algorithm based on a Hierarchical Particle Swarm Optimization (HPSO) is introduced to automatically extract all fuzzy logic system (FLS)’s parameters of a T–S fuzzy model. During online operation, both the consequent parameters of the T–S fuzzy model and the PSO inertia weight are continually updated when new data becomes available. By applying this concept to the learning algorithm, a new type T–S fuzzy modeling approach is constructed where the proposed HPSO algorithm includes an adaptive procedure and becomes a self-adaptive HPSO (S-AHPSO) algorithm usable in real-time processes. To improve the computational time of the proposed HPSO, particles positions are initialized by using an efficient unsupervised fuzzy clustering algorithm (UFCA). The UFCA combines the K-nearest neighbour and fuzzy C-means methods into a fuzzy modeling method for partitioning of the input–output data and identifying the antecedent parameters of the fuzzy system, enhancing the HPSO’s tuning. The approach is applied to identify the dynamical behavior of the dissolved oxygen concentration in an activated sludge reactor within a wastewater treatment plant. The results show that the proposed approach can identify nonlinear systems satisfactorily, and reveal superior performance of the proposed methods when compared with other state of the art methods. Moreover, the methodologies proposed in this paper can be involved in wider applications in a number of fields such as model predictive control, direct controller design, unsupervised clustering, motion detection, and robotics.     

A learning process for fuzzy control rules using genetic algorithms

The purpose of this paper is to present a genetic learning process for learning fuzzy control rules from examples. It is developed in three stages: the first one is a fuzzy rule genetic generating process based on a rule learning iterative approach, the second one combines two kinds of rules, experts rules if there are and the previously generated fuzzy control rules, removing the redundant fuzzy rules, and the thrid one is a tuning process for adjusting the membership functions of the fuzzy rules. The three components of the learning process are developed formulating suitable genetic algorithms.     

A semi-automated design of instance-based fuzzy parameter tuning for metaheuristics based on decision tree induction

Two main concepts are established in the literature for the Parameter Setting Problem of metaheuristics: Parameter Tuning Strategies (PTS) and Parameter Control Strategies (PCS). While PTS result in a fixed parameter setting for a set of problem instances, PCS are incorporated into the metaheuristic and adapt parameter values according to instance-specific performance feedback. The idea of Instance-specific Parameter Tuning Strategies (IPTS) is aiming to combine advantages of both tuning and control strategies by enabling the adoption of parameter values tailored to instance-specific characteristics a priori to running the metaheuristic. This requires, however, a significant knowledge about the impact of instance characteristics on heuristic performance. This paper presents an approach that semi-automatically designs the fuzzy logic rule base to obtain instance-specific parameter values by means of decision trees. This enables the user to automate the process of converting insights about instance-specific information and its impact on heuristic performance into a fuzzy rule base IPTS system. The system incorporates the decision maker’s preference about the trade-off between computational time and solution quality.     

Learning concurrently partition granularities and rule bases of Mamdani fuzzy systems in a multi-objective evolutionary framework

In this paper we propose a multi-objective evolutionary algorithm to generate Mamdani fuzzy rule-based systems with different good trade-offs between complexity and accuracy. The main novelty of the algorithm is that both rule base and granularity of the uniform partitions defined on the input and output variables are learned concurrently. To this aim, we introduce the concepts of virtual and concrete rule bases: the former is defined on linguistic variables, all partitioned with a fixed maximum number of fuzzy sets, while the latter takes into account, for each variable, a number of fuzzy sets as determined by the specific partition granularity of that variable. We exploit a chromosome composed of two parts, which codify the variables partition granularities, and the virtual rule base, respectively. Genetic operators manage virtual rule bases, whereas fitness evaluation relies on an appropriate mapping strategy between virtual and concrete rule bases. The algorithm has been tested on two real-world regression problems showing very promising results.     

Evolving fuzzy rule based controllers using genetic algorithms

The synthesis of genetics-based machine learning and fuzzy logic is beginning to show promise as a potent tool in solving complex control problems in multi-variate non-linear systems. In this paper an overview of current research applying the genetic algorithm to fuzzy rule based control is presented. A novel approach to genetics-based machine learning of fuzzy controllers, called a Pittsburgh Fuzzy Classifier System # 1 (P-FCS1) is proposed. P-FCS1 is based on the Pittsburgh model of learning classifier systems and employs variable length rule-sets and simultaneously evolves fuzzy set membership functions and relations. A new crossover operator which respects the functional linkage between fuzzy rules with overlapping input fuzzy set membership functions is introduced. Experimental results using P-FCS 1 are reported and compared with other published results. Application of P-FCS1 to a distributed control problem (dynamic routing in computer networks) is also described and experimental results are presented.     

The bi-objective fuzzy c-means cluster analysis for TSK fuzzy system identification

For conventional fuzzy clustering-based approaches to fuzzy system identification, a fuzzy function is used for cluster formation and another fuzzy function is used for cluster validation to determine the number and location of the clusters which define IF parts of the rule base. However, the different fuzzy functions used for cluster formation and validation may not indicate the same best number and location of the clusters. This potential disparity motivates us to propose a new fuzzy clustering-based approach to fuzzy system identification based on the bi-objective fuzzy c-means (BOFCM) cluster analysis. In this approach, we use the BOFCM function for both cluster formation and validation to simultaneously determine the number and location of the clusters which we hope can efficiently and effectively define IF parts of the rule base. The proposed approach is validated by applying it to the truck backer-upper problem with an obstacle in the center of the field.     

Clustering of search trajectory and its application to parameter tuning

This paper is concerned with automated classification of Combinatorial Optimization Problem instances for instance-specific parameter tuning purpose. We propose the CluPaTra Framework, a generic approach to CLUster instances based on similar PAtterns according to search TRAjectories and apply it on parameter tuning. The key idea is to use the search trajectory as a generic feature for clustering problem instances. The advantage of using search trajectory is that it can be obtained from any local-search based algorithm with small additional computation time. We explore and compare two different search trajectory representations, two sequence alignment techniques (to calculate similarities) as well as two well-known clustering methods. We report experiment results on two classical problems: Travelling Salesman Problem and Quadratic Assignment Problem and industrial case study.     

An estimation of distribution algorithm for nurse scheduling

Schedules can be built in a similar way to a human scheduler by using a set of rules that involve domain knowledge. This paper presents an Estimation of Distribution Algorithm (EDA) for the nurse scheduling problem, which involves choosing a suitable scheduling rule from a set for the assignment of each nurse. Unlike previous work that used Genetic Algorithms (GAs) to implement implicit learning, the learning in the proposed algorithm is explicit, i.e. we identify and mix building blocks directly. The EDA is applied to implement such explicit learning by building a Bayesian network of the joint distribution of solutions. The conditional probability of each variable in the network is computed according to an initial set of promising solutions. Subsequently, each new instance for each variable is generated by using the corresponding conditional probabilities, until all variables have been generated, i.e. in our case, a new rule string has been obtained. Another set of rule strings will be generated in this way, some of which will replace previous strings based on fitness selection. If stopping conditions are not met, the conditional probabilities for all nodes in the Bayesian network are updated again using the current set of promising rule strings. Computational results from 52 real data instances demonstrate the success of this approach. It is also suggested that the learning mechanism in the proposed approach might be suitable for other scheduling problems.     

Synthesis of distillation trains with heat integration by a combined fuzzy and graphical approach

After passing some results of fuzzy synthesis of heat exchanger networks in review the application of this approach to the synthesis of distillation trains for the separation of multicomponent mixtures is demonstrated. The available heuristic rules are transformed into fuzzy implications dependent on different criteria and that competing rule, which is to be applied, is selected by the fuzzy algorithm.The theoretically possible amount of heat recovery within the networks generated in this way is estimated by a graphical method and included in the final cost evaluation. Regarding operating parameters like column pressure and reflux ratio, according to the mathematical effort at least qualitative statements can be made in result of the estimation of possible heat recovery.As already in the case of heat exchanger networks a following learning algorithm can serve to overcome the disadvantages of this sequential approach, representing at the same time the first step of a further evolutionary improvement of the initial distillation train.     

Quadratic Majorization for Nonconvex Loss with Applications to the Boosting Algorithm

Classical robust statistical methods dealing with noisy data are often based on modifications of convex loss functions. In recent years, nonconvex loss-based robust methods have been increasingly popular. A nonconvex loss can provide robust estimation for data contaminated with outliers. The significant challenge is that a nonconvex loss can be numerically difficult to optimize. This article proposes quadratic majorization algorithm for nonconvex (QManc) loss. The QManc can decompose a nonconvex loss into a sequence of simpler optimization problems. Subsequently, the QManc is applied to a powerful machine learning algorithm: quadratic majorization boosting algorithm (QMBA). We develop QMBA for robust classification (binary and multi-category) and regression. In high-dimensional cancer genetics data and simulations, the QMBA is comparable with convex loss-based boosting algorithms for clean data, and outperforms the latter for data contaminated with outliers. The QMBA is also superior to boosting when directly implemented to optimize nonconvex loss functions. Supplementary material for this article is available online.     

On strong consistency of the fuzzy generalized nearest neighbor rule

The k nearest neighbor rule (k-NNR) is a well-known nonparametric decision rule in pattern classification. Fuzzy set theory has been widely used to represent the uncertainty of class membership. Several researchers extended conventional k-NNR to fuzzy k-NNR, such as Bezdek et al. [Fuzzy Sets and Systems 18 (1986) 237–256], Keller et al. [IEEE Trans. Syst. Man, and Cybern. 15(4) (1985) 580–585], Béreau and Dubuisson [Fuzzy Sets and Systems 44 (1991) 17–32]. In this paper, we describe a fuzzy generalized k-NN algorithm. This algorithm is a unified approach to a variety of fuzzy k-NNR's. Then we create the strong consistency of posterior risk of the fuzzy generalized NNR.