Towards Generative Design of Computationally Efficient Mathematical Models with Evolutionary Learning

In this paper, we describe the concept of generative design approach applied to the automated evolutionary learning of mathematical models in a computationally efficient way. To formalize the problems of models’ design and co-design, the generalized formulation of the modeling workflow is proposed. A parallelized evolutionary learning approach for the identification of model structure is described for the equation-based model and composite machine learning models. Moreover, the involvement of the performance models in the design process is analyzed. A set of experiments with various models and computational resources is conducted to verify different aspects of the proposed approach.     

基于机器学习的合金催化剂表面吸附能预测

近年来,机器学习方法逐渐成为多相催化中的一种关键研究手段. 二元合金材料作为重要的催化剂之一,在双功能催化剂的筛选中受到了广泛的关注. 本文提出了一个将机器学习方法应用在预测催化性质上的整体框架,从而快速预测原子、分子在金属和二元合金表面的吸附能. 通过测试不同的机器学习方法来评估它们对于该问题的适用性,并将树集成的方法与压缩感知方法相结合,利用约6×10⁴个吸附能数据构建了预测模型. 相对于线性比例关系,该方法可以更准确地预测大量合金上的吸附能(预测的均方根误差降低一半),并且更通用地预测各种吸附物的能量,为发现新的双金属催化剂铺平了道路.     

Assessment of robustness and transferability of classification models built for cancer diagnostics using Raman spectroscopy

Over recent years, Raman spectroscopy has been demonstrated as a prospective tool for application in cancer diagnostics. The use of Raman spectroscopy for this purpose relies on pattern recognition methods that have been developed to perform well on data achieved under laboratory conditions. However, the application of Raman spectroscopy as a routine clinical tool is likely to result in imperfect data due to instrument‐to‐instrument variation. Such corruption to the pure tissue spectral data is expected to negatively impact the classification performance of the diagnostic model. In this paper, we present a thorough assessment of the robustness of the Raman approach. This was achieved by perturbing a set of spectra in different ways, including various linear shifts, nonlinear shifts and random noise and using previously optimised classification models to predict the class membership of each spectrum in a testing set. The loss of predictive power with increased corruption was used to calculate a score, which allows an easy comparison of the model robustness. For this approach, three different types of classification models, including linear discriminant analysis (LDA), partial least square discriminant analysis (PLS‐DA) and support vector machine (SVM), built for lymph node diagnostics were the subject of the robustness testing. The results showed that a linear perturbation had the highest impact on the performance of all classification models. Among all linear corruption methods, a gradient y‐shift resulted in the highest performance loss. Thus, the factor most likely to affect the predictive outcome of models when using different systems is a gradient y‐shift. Copyright © 2010 John Wiley & Sons, Ltd.     

基于高斯过程的混沌时间序列单步与多步预测

针对混沌时间序列单步和多步预测,提出基于复合协方差函数的高斯过程 (GP)模型方法.GP模型的确立由协方差函数决定,通过对训练数据集的学习,在证据最大化框架内,利用矩阵运算和优化算法自适应地确定协方差函数和均值函数中的超参数.GP模型与神经网络、模糊模型相比,其可调整参数很少.将不同复合协方差函数的GP模型应用在混沌时间序列单步及多步提前预测中,并与单一协方差函数的GP、支持向量机、最小二乘支持向量机、径向基函数神经网络等方法进行了比较.仿真结果表明,基于不同复合协方差函数的GP方法能精确地预测混沌时间序 关键词： 高斯过程 混沌时间序列 预测 模型比较     

Probabilistic Predictions with Federated Learning

Probabilistic predictions with machine learning are important in many applications. These are commonly done with Bayesian learning algorithms. However, Bayesian learning methods are computationally expensive in comparison with non-Bayesian methods. Furthermore, the data used to train these algorithms are often distributed over a large group of end devices. Federated learning can be applied in this setting in a communication-efficient and privacy-preserving manner but does not include predictive uncertainty. To represent predictive uncertainty in federated learning, our suggestion is to introduce uncertainty in the aggregation step of the algorithm by treating the set of local weights as a posterior distribution for the weights of the global model. We compare our approach to state-of-the-art Bayesian and non-Bayesian probabilistic learning algorithms. By applying proper scoring rules to evaluate the predictive distributions, we show that our approach can achieve similar performance as the benchmark would achieve in a non-distributed setting.     

The Resolved Mutual Information Function as a Structural Fingerprint of Biomolecular Sequences for Interpretable Machine Learning Classifiers

In the present article we propose the application of variants of the mutual information function as characteristic fingerprints of biomolecular sequences for classification analysis. In particular, we consider the resolved mutual information functions based on Shannon-, Rényi-, and Tsallis-entropy. In combination with interpretable machine learning classifier models based on generalized learning vector quantization, a powerful methodology for sequence classification is achieved which allows substantial knowledge extraction in addition to the high classification ability due to the model-inherent robustness. Any potential (slightly) inferior performance of the used classifier is compensated by the additional knowledge provided by interpretable models. This knowledge may assist the user in the analysis and understanding of the used data and considered task. After theoretical justification of the concepts, we demonstrate the approach for various example data sets covering different areas in biomolecular sequence analysis.     

An Appraisal of Incremental Learning Methods

As a special case of machine learning, incremental learning can acquire useful knowledge from incoming data continuously while it does not need to access the original data. It is expected to have the ability of memorization and it is regarded as one of the ultimate goals of artificial intelligence technology. However, incremental learning remains a long term challenge. Modern deep neural network models achieve outstanding performance on stationary data distributions with batch training. This restriction leads to catastrophic forgetting for incremental learning scenarios since the distribution of incoming data is unknown and has a highly different probability from the old data. Therefore, a model must be both plastic to acquire new knowledge and stable to consolidate existing knowledge. This review aims to draw a systematic review of the state of the art of incremental learning methods. Published reports are selected from Web of Science, IEEEXplore, and DBLP databases up to May 2020. Each paper is reviewed according to the types: architectural strategy, regularization strategy and rehearsal and pseudo-rehearsal strategy. We compare and discuss different methods. Moreover, the development trend and research focus are given. It is concluded that incremental learning is still a hot research area and will be for a long period. More attention should be paid to the exploration of both biological systems and computational models.     

可见-近红外多光谱数据对水稻种子成活率的判定

种子活性受到存储条件的影响很大。收集了真实情况下受到不同存储条件影响的种子,通过发芽实验验证了其成活率存在差异。再从中选择适量的种子样本,采集其单颗种子的可见-近红外反射光谱,运用不同的光谱预处理技术,结合不同的机器学习建模手段,以区分不同成活率的种子。比较了不同的光谱预处理方法,比如标准反射光谱校正、多元散射校正等。从识别准确度的角度,认为标准反射光谱校正的方法,能够很大程度上提升不同存活率种子的光谱差异性,从而经过机器学习判断达到更高的识别准确度。同时比较了支持向量机、 K邻近和距离判别分析等机器监督学习建模方法,发现利用标准反射光谱校正的方法结合距离判别分析,能够对种子样本实现完全准确的判定。更进一步,为了满足实际运用中快速识别的要求,将高分辨率的光谱数据压缩成为低分辨率多通道带通光谱数据,这样可以大大降低的光谱数据长度,节约各种机器学习器在训练和判断中所用的时间。使用简化过后的多通道带通光谱数据判定种子存活率,其识别准确度仍然接近90%。充分说明了,利用多通道宽带光谱数据,并选择合适的机器学习建模方式,足以满足实际选种产业的一般性需求,有潜力作为未来粮种成活率快速鉴别的技术手段。还采用了多种带通宽度以简化光谱,分析比较不同带通宽度对识别精度的影响。总体来说由于带宽增大,数据量减少,识别速度更快,但是识别精度降低。从10~50 nm改变光谱带宽,标准反射校正后的简化光谱的识别精度从87.50%下降到58.75%。在实际运用中,需要权衡识别速率和预期识别精度,合理的选择带宽。验证了根据简化后的可见近红外反射光谱,能够较快速且准确的识别水稻种子存活率,为以后的基于带通滤波片的快速种子存活率识别奠定了基础。     

Prediction carbon dioxide solubility in ionic liquids based on deep learning

Ionic liquids have a great potential in capture and separation of carbon dioxide (CO₂), and the solubility of CO₂ in ionic liquids is one of key data for engineering applications. In this paper, the critical properties of ionic liquids are combined with deep learning models (CP-DNN, CP-CNN, CP-RNN) to establish theoretical prediction models of CO₂ solubility in ionic liquids. The predictive performance of these framworks is able to meet or exceed the predicted effects of the method based on thermodynamic models (PR,SRK) and machine learning method (XGBoost). For CP-RNN, the coefficient of determination (R²) between experimental and predicted values is 0.988, CP-CNN is 0.999, and CP-DNN is 0.984. This research can avoid complex computational characterisation, it is to provide a theoretical method to further enrich and improve the data information analysis of the solubility of CO₂ in ionic liquids.     

A data-efficient self-supervised deep learning model for design and characterization of nanophotonic structures

With its tremendous success in many machine learning and pattern recognition tasks, deep learning, as one type of data-driven models, has also led to many breakthroughs in other disciplines including physics, chemistry and material science. Nevertheless,the supremacy of deep learning over conventional optimization approaches heavily depends on the huge amount of data collected in advance to train the model, which is a common bottleneck of such a data-driven technique. In this work, we present a comprehensive deep learning model for the design and characterization of nanophotonic structures, where a self-supervised learning mechanism is introduced to alleviate the burden of data acquisition. Taking reflective metasurfaces as an example, we demonstrate that the self-supervised deep learning model can effectively utilize randomly generated unlabeled data during training, with the total test loss and prediction accuracy improved by about 15% compared with the fully supervised counterpart.The proposed self-supervised learning scheme provides an efficient solution for deep learning models in some physics-related tasks where labeled data are limited or expensive to collect.     

FASTENER Feature Selection for Inference from Earth Observation Data

In this paper, a novel feature selection algorithm for inference from high-dimensional data (FASTENER) is presented. With its multi-objective approach, the algorithm tries to maximize the accuracy of a machine learning algorithm with as few features as possible. The algorithm exploits entropy-based measures, such as mutual information in the crossover phase of the iterative genetic approach. FASTENER converges to a (near) optimal subset of features faster than other multi-objective wrapper methods, such as POSS, DT-forward and FS-SDS, and achieves better classification accuracy than similarity and information theory-based methods currently utilized in earth observation scenarios. The approach was primarily evaluated using the earth observation data set for land-cover classification from ESA’s Sentinel-2 mission, the digital elevation model and the ground truth data of the Land Parcel Identification System from Slovenia. For land cover classification, the algorithm gives state-of-the-art results. Additionally, FASTENER was tested on open feature selection data sets and compared to the state-of-the-art methods. With fewer model evaluations, the algorithm yields comparable results to DT-forward and is superior to FS-SDS. FASTENER can be used in any supervised machine learning scenario.     

Ensemble Neuroevolution-Based Approach for Multivariate Time Series Anomaly Detection

Multivariate time series anomaly detection is a widespread problem in the field of failure prevention. Fast prevention means lower repair costs and losses. The amount of sensors in novel industry systems makes the anomaly detection process quite difficult for humans. Algorithms that automate the process of detecting anomalies are crucial in modern failure prevention systems. Therefore, many machine learning models have been designed to address this problem. Mostly, they are autoencoder-based architectures with some generative adversarial elements. This work shows a framework that incorporates neuroevolution methods to boost the anomaly detection scores of new and already known models. The presented approach adapts evolution strategies for evolving an ensemble model, in which every single model works on a subgroup of data sensors. The next goal of neuroevolution is to optimize the architecture and hyperparameters such as the window size, the number of layers, and the layer depths. The proposed framework shows that it is possible to boost most anomaly detection deep learning models in a reasonable time and a fully automated mode. We ran tests on the SWAT and WADI datasets. To the best of our knowledge, this is the first approach in which an ensemble deep learning anomaly detection model is built in a fully automatic way using a neuroevolution strategy.     

Assessment of Classification Models and Relevant Features on Nonalcoholic Steatohepatitis Using Random Forest

Nonalcoholic fatty liver disease (NAFLD) is the hepatic manifestation of metabolic syndrome and is the most common cause of chronic liver disease in developed countries. Certain conditions, including mild inflammation biomarkers, dyslipidemia, and insulin resistance, can trigger a progression to nonalcoholic steatohepatitis (NASH), a condition characterized by inflammation and liver cell damage. We demonstrate the usefulness of machine learning with a case study to analyze the most important features in random forest (RF) models for predicting patients at risk of developing NASH. We collected data from patients who attended the Cardiovascular Risk Unit of Mostoles University Hospital (Madrid, Spain) from 2005 to 2021. We reviewed electronic health records to assess the presence of NASH, which was used as the outcome. We chose RF as the algorithm to develop six models using different pre-processing strategies. The performance metrics was evaluated to choose an optimized model. Finally, several interpretability techniques, such as feature importance, contribution of each feature to predictions, and partial dependence plots, were used to understand and explain the model to help obtain a better understanding of machine learning-based predictions. In total, 1525 patients met the inclusion criteria. The mean age was 57.3 years, and 507 patients had NASH (prevalence of 33.2%). Filter methods (the chi-square and Mann–Whitney–Wilcoxon tests) did not produce additional insight in terms of interactions, contributions, or relationships among variables and their outcomes. The random forest model correctly classified patients with NASH to an accuracy of 0.87 in the best model and to 0.79 in the worst one. Four features were the most relevant: insulin resistance, ferritin, serum levels of insulin, and triglycerides. The contribution of each feature was assessed via partial dependence plots. Random forest-based modeling demonstrated that machine learning can be used to improve interpretability, produce understanding of the modeled behavior, and demonstrate how far certain features can contribute to predictions.     

Task’s Choice: Pruning-Based Feature Sharing (PBFS) for Multi-Task Learning

In most of the existing multi-task learning (MTL) models, multiple tasks’ public information is learned by sharing parameters across hidden layers, such as hard sharing, soft sharing, and hierarchical sharing. One promising approach is to introduce model pruning into information learning, such as sparse sharing, which is regarded as being outstanding in knowledge transferring. However, the above method performs inefficiently in conflict tasks, with inadequate learning of tasks’ private information, or through suffering from negative transferring. In this paper, we propose a multi-task learning model (Pruning-Based Feature Sharing, PBFS) that merges a soft parameter sharing structure with model pruning and adds a prunable shared network among different task-specific subnets. In this way, each task can select parameters in a shared subnet, according to its requirements. Experiments are conducted on three benchmark public datasets and one synthetic dataset; the impact of the different subnets’ sparsity and tasks’ correlations to the model performance is analyzed. Results show that the proposed model’s information sharing strategy is helpful to transfer learning and superior to the several comparison models.     

最小角回归结合竞争性自适应重加权采样的近红外光谱波长选择

近红外光谱分析技术对检测样品无损伤且检测速度快、精度高,因此被广泛应用在了药品检测、石油化工等领域,尤其近年来机器学习和深度学习建模方法的深入应用使其具备了更准确的检测性能.然而,样品的近红外光谱数据具有比较高的维度且存在谱间重合、共线性和噪声等问题,对近红外光谱模型的性能产生消极影响,此时样品有效特征波长的筛选极为重...     

Federated Quantum Machine Learning

Distributed training across several quantum computers could significantly improve the training time and if we could share the learned model, not the data, it could potentially improve the data privacy as the training would happen where the data is located. One of the potential schemes to achieve this property is the federated learning (FL), which consists of several clients or local nodes learning on their own data and a central node to aggregate the models collected from those local nodes. However, to the best of our knowledge, no work has been done in quantum machine learning (QML) in federation setting yet. In this work, we present the federated training on hybrid quantum-classical machine learning models although our framework could be generalized to pure quantum machine learning model. Specifically, we consider the quantum neural network (QNN) coupled with classical pre-trained convolutional model. Our distributed federated learning scheme demonstrated almost the same level of trained model accuracies and yet significantly faster distributed training. It demonstrates a promising future research direction for scaling and privacy aspects.     

TNT: An Interpretable Tree-Network-Tree Learning Framework using Knowledge Distillation

Deep Neural Networks (DNNs) usually work in an end-to-end manner. This makes the trained DNNs easy to use, but they remain an ambiguous decision process for every test case. Unfortunately, the interpretability of decisions is crucial in some scenarios, such as medical or financial data mining and decision-making. In this paper, we propose a Tree-Network-Tree (TNT) learning framework for explainable decision-making, where the knowledge is alternately transferred between the tree model and DNNs. Specifically, the proposed TNT learning framework exerts the advantages of different models at different stages: (1) a novel James–Stein Decision Tree (JSDT) is proposed to generate better knowledge representations for DNNs, especially when the input data are in low-frequency or low-quality; (2) the DNNs output high-performing prediction result from the knowledge embedding inputs and behave as a teacher model for the following tree model; and (3) a novel distillable Gradient Boosted Decision Tree (dGBDT) is proposed to learn interpretable trees from the soft labels and make a comparable prediction as DNNs do. Extensive experiments on various machine learning tasks demonstrated the effectiveness of the proposed method.     

The emergence of coordination in public good games

In physical models it is well understood that the aggregate behaviour of a system is not in one to one correspondence with the behaviour of the average individual element of that system. Yet, in many economic models the behaviour of aggregates is thought of as corresponding to that of an individual. A typical example is that of public goods experiments. A systematic feature of such experiments is that, with repetition, people contribute less to public goods. A typical explanation is that people “learn to play Nash” or something approaching it. To justify such an explanation, an individual learning model is tested on average or aggregate data. In this paper we will examine this idea by analysing average and individual behaviour in a series of public goods experiments. We analyse data from a series of games of contributions to public goods and as is usual, we test a learning model on the average data. We then look at individual data, examine the changes that this produces and see if some general model such as the EWA (Expected Weighted Attraction) with varying parameters can account for individual behaviour. We find that once we disaggregate data such models have poor explanatory power. Groups do not learn as supposed, their behaviour differs markedly from one group to another, and the behaviour of the individuals who make up the groups also varies within groups. The decline in aggregate contributions cannot be explained by resorting to a uniform model of individual behaviour. However, the Nash equilibrium of such a game is a total payment for all the individuals and there is some convergence of the group in this respect. Yet the individual contributions do not converge. How the individuals “self-organsise” to coordinate, even in this limited way remains to be explained.