Attention Mechanisms and Their Applications to Complex Systems

Deep learning models and graphics processing units have completely transformed the field of machine learning. Recurrent neural networks and long short-term memories have been successfully used to model and predict complex systems. However, these classic models do not perform sequential reasoning, a process that guides a task based on perception and memory. In recent years, attention mechanisms have emerged as a promising solution to these problems. In this review, we describe the key aspects of attention mechanisms and some relevant attention techniques and point out why they are a remarkable advance in machine learning. Then, we illustrate some important applications of these techniques in the modeling of complex systems.     

Deep Learning Methods for Heart Sounds Classification: A Systematic Review

The automated classification of heart sounds plays a significant role in the diagnosis of cardiovascular diseases (CVDs). With the recent introduction of medical big data and artificial intelligence technology, there has been an increased focus on the development of deep learning approaches for heart sound classification. However, despite significant achievements in this field, there are still limitations due to insufficient data, inefficient training, and the unavailability of effective models. With the aim of improving the accuracy of heart sounds classification, an in-depth systematic review and an analysis of existing deep learning methods were performed in the present study, with an emphasis on the convolutional neural network (CNN) and recurrent neural network (RNN) methods developed over the last five years. This paper also discusses the challenges and expected future trends in the application of deep learning to heart sounds classification with the objective of providing an essential reference for further study.     

从单分子到凝聚相水体系的理论描述：势能面和分子动力学最新进展

本文回顾了近十年来水体系的势能面与分子动力学理论研究的最新进展,包括水分子参与的气相反应,固体表面上的吸附与解离动力学,以及从团簇到凝聚相水的结构、振动光谱与统计力学模拟. 近年来再次发展起来的机器学习技术,例如结合置换不变多项式的神经网络,或结合基本不变量的神经网络,已被成熟应用于气相与固体表面体系的高精度势能面构造中. 对于团簇甚至凝聚相水体系,原子中心神经网络方法或基于核的高斯过程方法应用更为广泛. 此外,在多体展开框架下,在气相体系中发展起来的的方法也组成了高维度体系势函数构造的高精度方案. 当前凝聚相水体系面临的主要问题是高精度从头算数据集的积累,兼顾计算精度与效率的双杂化密度泛函是一种可能的解决方案. 在动力学理论方面,无论是化学反应截面计算还是振动光谱模拟,往往需要合理描述水分子中氢原子的量子效应,才能得到较为可靠的理论计算结果. 量子波包动力学方法已经在气相反应机理研究方面有深入的应用,也在包含数个水分子的团簇振动分析中有初步应用. 基于路径积分的分子动力学方法正在较大水团簇以及凝聚相水的结构与谱学模拟方面发挥重要作用.     

Neural networks: A biased overview

An overview of recent activity in the field of neural networks is presented. The long-range aim of this research is to understand how the brain works. First some of the problems are stated and terminology defined; then an attempt is made to explain why physicists are drawn to the field, and their main potential contribution. In particular, in recent years some interesting models have been introduced by physicists. A small subset of these models is described, with particular emphasis on those that are analytically soluble. Finally a brief review of the history and recent developments of single- and multilayer perceptrons is given, bringing the situation up to date regarding the central immediate problem of the field: search for a learning algorithm that has an associated convergence theorem.     

Supervised Deep Learning in High Energy Phenomenology: a Mini Review *

Deep learning, a branch of machine learning, has been recently applied to high energy experimental and phenomenological studies. In this note we give a brief review on those applications using supervised deep learning. We first describe various learning models and then recapitulate their applications to high energy phenomenological studies. Some detailed applications are delineated in details, including the machine learning scan in the analysis of new physics parameter space, the graph neural networks in the search of top-squark production and in the $CP$ measurement of the top-Higgs coupling at the LHC.      

An Overview of Infrared Spectroscopy Based on Continuous Wavelet Transform Combined with Machine Learning Algorithms: Application to Chinese Medicines,Plant Classification,and Cancer Diagnosis

Abstract

Infrared spectroscopy has been a workhorse technique for materials analysis and can result in positively identifying many different types of material. In recent years there have been reports using wavelet analysis and machine learning algorithms to extract features of Fourier transform infrared spectrometry (FTIR). The machine learning algorithms contain back-propagation neural network (BPNN), radial basis function neural network (RBFNN), and support vector machine (SVM). This article reviews the important advances in FTIR analysis employing a continuous wavelet transform (CWT) and machine learning algorithms, especially in the applications of the method for Chinese medicine identification, plant classification, and cancer diagnosis.     

机器学习在大型粒子加速器中的应用回顾与展望

机器学习技术在近十几年发展迅猛,并被广泛地用于解决复杂的科学和工程问题。最近十年间,基于机器学习的粒子加速器相关研究也开始呈现出井喷式发展趋势。国际上许多加速器实验室开始尝试用机器学习和大数据技术处理加速器中的海量复杂数据,以期解决加速器及其子系统中的诸多物理和技术问题。不过,迄今为止,机器学习在加速器中的应用仍处于初步探索阶段,不同机器学习算法在解决具体加速器问题的效果及其适用范围尚待摸索,机器学习在实际加速器中的应用仍非常有限。因此,有必要对加速器领域中的机器学习研究做一个整体回顾和总结。将回顾机器学习在大型粒子加速器（以储存环加速器和直线加速器为主）中的加速器技术、束流物理以及加速器整体性能优化等研究方向中已取得的研究成果,并探讨机器学习在加速器领域的未来发展方向和应用前景。     

Co-Training for Visual Object Recognition Based on Self-Supervised Models Using a Cross-Entropy Regularization

Automatic recognition of visual objects using a deep learning approach has been successfully applied to multiple areas. However, deep learning techniques require a large amount of labeled data, which is usually expensive to obtain. An alternative is to use semi-supervised models, such as co-training, where multiple complementary views are combined using a small amount of labeled data. A simple way to associate views to visual objects is through the application of a degree of rotation or a type of filter. In this work, we propose a co-training model for visual object recognition using deep neural networks by adding layers of self-supervised neural networks as intermediate inputs to the views, where the views are diversified through the cross-entropy regularization of their outputs. Since the model merges the concepts of co-training and self-supervised learning by considering the differentiation of outputs, we called it Differential Self-Supervised Co-Training (DSSCo-Training). This paper presents some experiments using the DSSCo-Training model to well-known image datasets such as MNIST, CIFAR-100, and SVHN. The results indicate that the proposed model is competitive with the state-of-art models and shows an average relative improvement of 5% in accuracy for several datasets, despite its greater simplicity with respect to more recent approaches.     

A Transformer-Based Capsule Network for 3D Part–Whole Relationship Learning

Learning the relationship between the part and whole of an object, such as humans recognizing objects, is a challenging task. In this paper, we specifically design a novel neural network to explore the local-to-global cognition of 3D models and the aggregation of structural contextual features in 3D space, inspired by the recent success of Transformer in natural language processing (NLP) and impressive strides in image analysis tasks such as image classification and object detection. We build a 3D shape Transformer based on local shape representation, which provides relation learning between local patches on 3D mesh models. Similar to token (word) states in NLP, we propose local shape tokens to encode local geometric information. On this basis, we design a shape-Transformer-based capsule routing algorithm. By applying an iterative capsule routing algorithm, local shape information can be further aggregated into high-level capsules containing deeper contextual information so as to realize the cognition from the local to the whole. We performed classification tasks on the deformable 3D object data sets SHREC10 and SHREC15 and the large data set ModelNet40, and obtained profound results, which shows that our model has excellent performance in complex 3D model recognition and big data feature learning.     

The Large Hadron Collider: A Marvel of Technology,edited by Lyndon Evans

Machine learning algorithms learn a desired input-output relation from examples in order to interpret new inputs. This is important for tasks such as image and speech recognition or strategy optimisation, with growing applications in the IT industry. In the last couple of years, researchers investigated if quantum computing can help to improve classical machine learning algorithms. Ideas range from running computationally costly algorithms or their subroutines efficiently on a quantum computer to the translation of stochastic methods into the language of quantum theory. This contribution gives a systematic overview of the emerging field of quantum machine learning. It presents the approaches as well as technical details in an accessible way, and discusses the potential of a future theory of quantum learning.     

New technical developments in magnetic resonance imaging of epilepsy

Within the last several years a number of technical developments have been made in magnetic resonance imaging (MRI) that can potentially impact clinical and research MR imaging applications in epilepsy. These include developments in instrumentation and in pulse sequences. Advances in instrumentation include higher capacity gradient systems and multiple receiver coils as directed to brain imaging. Advances in pulse sequence include use of fast or turbo-spin-echo techniques, variants of echo-planar imaging, and sequences such as fluid-attenuation inversion recovery (FLAIR) targeted to specific applications of brain imaging. The purpose of this paper is to review several of these developments.     

An Appraisal of Incremental Learning Methods

As a special case of machine learning, incremental learning can acquire useful knowledge from incoming data continuously while it does not need to access the original data. It is expected to have the ability of memorization and it is regarded as one of the ultimate goals of artificial intelligence technology. However, incremental learning remains a long term challenge. Modern deep neural network models achieve outstanding performance on stationary data distributions with batch training. This restriction leads to catastrophic forgetting for incremental learning scenarios since the distribution of incoming data is unknown and has a highly different probability from the old data. Therefore, a model must be both plastic to acquire new knowledge and stable to consolidate existing knowledge. This review aims to draw a systematic review of the state of the art of incremental learning methods. Published reports are selected from Web of Science, IEEEXplore, and DBLP databases up to May 2020. Each paper is reviewed according to the types: architectural strategy, regularization strategy and rehearsal and pseudo-rehearsal strategy. We compare and discuss different methods. Moreover, the development trend and research focus are given. It is concluded that incremental learning is still a hot research area and will be for a long period. More attention should be paid to the exploration of both biological systems and computational models.     

Survey on Self-Supervised Learning: Auxiliary Pretext Tasks and Contrastive Learning Methods in Imaging

Although deep learning algorithms have achieved significant progress in a variety of domains, they require costly annotations on huge datasets. Self-supervised learning (SSL) using unlabeled data has emerged as an alternative, as it eliminates manual annotation. To do this, SSL constructs feature representations using pretext tasks that operate without manual annotation, which allows models trained in these tasks to extract useful latent representations that later improve downstream tasks such as object classification and detection. The early methods of SSL are based on auxiliary pretext tasks as a way to learn representations using pseudo-labels, or labels that were created automatically based on the dataset’s attributes. Furthermore, contrastive learning has also performed well in learning representations via SSL. To succeed, it pushes positive samples closer together, and negative ones further apart, in the latent space. This paper provides a comprehensive literature review of the top-performing SSL methods using auxiliary pretext and contrastive learning techniques. It details the motivation for this research, a general pipeline of SSL, the terminologies of the field, and provides an examination of pretext tasks and self-supervised methods. It also examines how self-supervised methods compare to supervised ones, and then discusses both further considerations and ongoing challenges faced by SSL.     

Evaluating the Learning Procedure of CNNs through a Sequence of Prognostic Tests Utilising Information Theoretical Measures

Deep learning has proven to be an important element of modern data processing technology, which has found its application in many areas such as multimodal sensor data processing and understanding, data generation and anomaly detection. While the use of deep learning is booming in many real-world tasks, the internal processes of how it draws results is still uncertain. Understanding the data processing pathways within a deep neural network is important for transparency and better resource utilisation. In this paper, a method utilising information theoretic measures is used to reveal the typical learning patterns of convolutional neural networks, which are commonly used for image processing tasks. For this purpose, training samples, true labels and estimated labels are considered to be random variables. The mutual information and conditional entropy between these variables are then studied using information theoretical measures. This paper shows that more convolutional layers in the network improve its learning and unnecessarily higher numbers of convolutional layers do not improve the learning any further. The number of convolutional layers that need to be added to a neural network to gain the desired learning level can be determined with the help of theoretic information quantities including entropy, inequality and mutual information among the inputs to the network. The kernel size of convolutional layers only affects the learning speed of the network. This study also shows that where the dropout layer is applied to has no significant effects on the learning of networks with a lower dropout rate, and it is better placed immediately after the last convolutional layer with higher dropout rates.     

Winsorization for Robust Bayesian Neural Networks

With the advent of big data and the popularity of black-box deep learning methods, it is imperative to address the robustness of neural networks to noise and outliers. We propose the use of Winsorization to recover model performances when the data may have outliers and other aberrant observations. We provide a comparative analysis of several probabilistic artificial intelligence and machine learning techniques for supervised learning case studies. Broadly, Winsorization is a versatile technique for accounting for outliers in data. However, different probabilistic machine learning techniques have different levels of efficiency when used on outlier-prone data, with or without Winsorization. We notice that Gaussian processes are extremely vulnerable to outliers, while deep learning techniques in general are more robust.     

Behavioral and computational aspects of language and its acquisition

One of the greatest challenges facing the cognitive sciences is to explain what it means to know a language, and how the knowledge of language is acquired. The dominant approach to this challenge within linguistics has been to seek an efficient characterization of the wealth of documented structural properties of language in terms of a compact generative grammar—ideally, the minimal necessary set of innate, universal, exception-less, highly abstract rules that jointly generate all and only the observed phenomena and are common to all human languages. We review developmental, behavioral, and computational evidence that seems to favor an alternative view of language, according to which linguistic structures are generated by a large, open set of constructions of varying degrees of abstraction and complexity, which embody both form and meaning and are acquired through socially situated experience in a given language community, by probabilistic learning algorithms that resemble those at work in other cognitive modalities.     

Efficient detection of refugees and migrants in Turkey using convolutional neural network

At the start of the Syrian Civil War in 2011, NGOs played a big part in giving refugees access to aid and distributing that aid so that people could go to school, get a job, or get medical care. Within the last few years, when tensions rose between Syrian refugees and the Turkish community, many non-governmental organizations switched their attention to fostering community among refugees in Turkey. Over the past two decades, family displacement has become a big problem in various countries due to a rise in the frequency with which natural catastrophes, military conflicts, and terrorist strikes occur. It poses severe difficulties for governing bodies and the organizations that oversee them. This research aims to identify and track refugees in surveillance zones by utilizing artificial intelligence. Refugees are vulnerable to acts of nature and human aggression, which makes their random relocation or encampments challenging to manage. To overcome these challenges, a convolutional neural network deep learning model has been proposed to identify and track refugees in surveillance zones. The proposed solution is integrated with Internet of Things (IoT) technology by equipping the system with IoT sensors to capture real-time data on the location and movements of refugees. This combination of AI and IoT has the potential to improve the efficiency and effectiveness of refugee management efforts. The suggested solution uses a convolutional neural network deep learning model, which can quickly identify a refugee’s face. To assist the government in locating a specific refugee, the system simultaneously connects with the refugees and requests that they regularly update their location. The system alerts security to identify the missing immigrant since the refugee does not update their whereabouts. Without human intervention, the deep learning algorithm makes it simple to recognize immigrants and keep an eye on them.     

How machine learning conquers the unitary limit

Machine learning has become a premier tool in physics and other fields of science.It has been shown that the quantum mechanical scattering problem cannot only be solved with such techniques,but it was argued that the underlying neural network develops the Bom series for shallow potentials.However,classical machine learning algorithms fail in the unitary limit of an infinite scattering length.The unitary limit plays an important role in our understanding of bound strongly interacting fermionic systems and can be realized in cold atom experiments.Here,we develop a formalism that explains the unitary limit in terms of what we define as unitary limit surfaces.This not only allows to investigate the unitary limit geometrically in potential space,but also provides a numerically simple approach towards unnaturally large scattering lengths with standard multilayer perceptrons.Its scope is therefore not limited to applications in nuclear and atomic physics,but includes all systems that exhibit an unnaturally large scale.     

Nuclear magnetic resonance for quantum computing: Techniques and recent achievements

Rapid developments in quantum information processing have been made, and remarkable achievements have been obtained in recent years, both in theory and experiments. Coherent control of nuclear spin dynamics is a powerful tool for the experimental implementation of quantum schemes in liquid and solid nuclear magnetic resonance(NMR) system,especially in liquid-state NMR. Compared with other quantum information processing systems, the NMR platform has the advantages such as the long coherence time, the precise manipulation, and well-developed quantum control techniques,which make it possible to accurately control a quantum system with up to 12-qubits. Extensive applications of liquid-state NMR spectroscopy in quantum information processing such as quantum communication, quantum computing, and quantum simulation have been thoroughly studied over half a century. This article introduces the general principles of NMR quantum information processing, and reviews the new-developed techniques. The review will also include the recent achievements of the experimental realization of quantum algorithms for machine learning, quantum simulations for high energy physics, and topological order in NMR. We also discuss the limitation and prospect of liquid-state NMR spectroscopy and the solid-state NMR systems as quantum computing in the article.