Memristor-based multi-synaptic spiking neuron circuit for spiking neural network

Spiking neural networks (SNNs) are widely used in many fields because they work closer to biological neurons. However, due to its computational complexity, many SNNs implementations are limited to computer programs. First, this paper proposes a multi-synaptic circuit (MSC) based on memristor, which realizes the multi-synapse connection between neurons and the multi-delay transmission of pulse signals. The synapse circuit participates in the calculation of the network while transmitting the pulse signal, and completes the complex calculations on the software with hardware. Secondly, a new spiking neuron circuit based on the leaky integrate-and-fire (LIF) model is designed in this paper. The amplitude and width of the pulse emitted by the spiking neuron circuit can be adjusted as required. The combination of spiking neuron circuit and MSC forms the multi-synaptic spiking neuron (MSSN). The MSSN was simulated in PSPICE and the expected result was obtained, which verified the feasibility of the circuit. Finally, a small SNN was designed based on the mathematical model of MSSN. After the SNN is trained and optimized, it obtains a good accuracy in the classification of the IRIS-dataset, which verifies the practicability of the design in the network.     

Continuous Viewpoint Planning in Conjunction with Dynamic Exploration for Active Object Recognition

Active object recognition (AOR) aims at collecting additional information to improve recognition performance by purposefully adjusting the viewpoint of an agent. How to determine the next best viewpoint of the agent, i.e., viewpoint planning (VP), is a research focus. Most existing VP methods perform viewpoint exploration in the discrete viewpoint space, which have to sample viewpoint space and may bring in significant quantization error. To address this challenge, a continuous VP approach for AOR based on reinforcement learning is proposed. Specifically, we use two separate neural networks to model the VP policy as a parameterized Gaussian distribution and resort the proximal policy optimization framework to learn the policy. Furthermore, an adaptive entropy regularization based dynamic exploration scheme is presented to automatically adjust the viewpoint exploration ability in the learning process. To the end, experimental results on the public dataset GERMS well demonstrate the superiority of our proposed VP method.     

Evaluating the Learning Procedure of CNNs through a Sequence of Prognostic Tests Utilising Information Theoretical Measures

Deep learning has proven to be an important element of modern data processing technology, which has found its application in many areas such as multimodal sensor data processing and understanding, data generation and anomaly detection. While the use of deep learning is booming in many real-world tasks, the internal processes of how it draws results is still uncertain. Understanding the data processing pathways within a deep neural network is important for transparency and better resource utilisation. In this paper, a method utilising information theoretic measures is used to reveal the typical learning patterns of convolutional neural networks, which are commonly used for image processing tasks. For this purpose, training samples, true labels and estimated labels are considered to be random variables. The mutual information and conditional entropy between these variables are then studied using information theoretical measures. This paper shows that more convolutional layers in the network improve its learning and unnecessarily higher numbers of convolutional layers do not improve the learning any further. The number of convolutional layers that need to be added to a neural network to gain the desired learning level can be determined with the help of theoretic information quantities including entropy, inequality and mutual information among the inputs to the network. The kernel size of convolutional layers only affects the learning speed of the network. This study also shows that where the dropout layer is applied to has no significant effects on the learning of networks with a lower dropout rate, and it is better placed immediately after the last convolutional layer with higher dropout rates.     

Sentiment Analysis of Persian Movie Reviews Using Deep Learning

Sentiment analysis aims to automatically classify the subject’s sentiment (e.g., positive, negative, or neutral) towards a particular aspect such as a topic, product, movie, news, etc. Deep learning has recently emerged as a powerful machine learning technique to tackle the growing demand for accurate sentiment analysis. However, the majority of research efforts are devoted to English-language only, while information of great importance is also available in other languages. This paper presents a novel, context-aware, deep-learning-driven, Persian sentiment analysis approach. Specifically, the proposed deep-learning-driven automated feature-engineering approach classifies Persian movie reviews as having positive or negative sentiments. Two deep learning algorithms, convolutional neural networks (CNN) and long-short-term memory (LSTM), are applied and compared with our previously proposed manual-feature-engineering-driven, SVM-based approach. Simulation results demonstrate that LSTM obtained a better performance as compared to multilayer perceptron (MLP), autoencoder, support vector machine (SVM), logistic regression and CNN algorithms.     

高效选取线性独立的原子特征构建精确机器学习势函数

机器学习势由于具有与第一性原理计算相当的准确性,且低得多的计算成本,在原子模拟中极具前景. 然而原子机器学习势的可靠性、速度和可迁移性在很大程度上取决于原子构型的表示. 适当地选取用作机器学习程序输入的描述符是一个成功的机器学习表示的关键. 本文发展了一种简单有效的方法,可以基于训练数据固有的相关性,从大量待选的描述符中自动选取一组最佳的线性独立原子特征. 通过对几个具有较少冗余线性独立嵌入密度描述符的基准分子构建嵌入原子神经网络势的应用,证明了这种新方法的有效性和准确性. 该算法可以大大简化原子特征的初始选取,并极大地提高原子机器学习势的性能.     

An Entropy Metric for Regular Grammar Classification and Learning with Recurrent Neural Networks

Recently, there has been a resurgence of formal language theory in deep learning research. However, most research focused on the more practical problems of attempting to represent symbolic knowledge by machine learning. In contrast, there has been limited research on exploring the fundamental connection between them. To obtain a better understanding of the internal structures of regular grammars and their corresponding complexity, we focus on categorizing regular grammars by using both theoretical analysis and empirical evidence. Specifically, motivated by the concentric ring representation, we relaxed the original order information and introduced an entropy metric for describing the complexity of different regular grammars. Based on the entropy metric, we categorized regular grammars into three disjoint subclasses: the polynomial, exponential and proportional classes. In addition, several classification theorems are provided for different representations of regular grammars. Our analysis was validated by examining the process of learning grammars with multiple recurrent neural networks. Our results show that as expected more complex grammars are generally more difficult to learn.     

Multimodal Fake-News Recognition Using Ensemble of Deep Learners

Social networks have drastically changed how people obtain information. News in social networks is accompanied by images and videos and thus receives more attention from readers as opposed to traditional sources. Unfortunately, fake-news publishers often misuse these advantages to spread false information rapidly. Therefore, the early detection of fake news is crucial. The best way to address this issue is to design an automatic detector based on fake-news content. Thus far, many fake-news recognition systems, including both traditional machine learning and deep learning models, have been proposed. Given that manual feature-extraction methods are very time-consuming, deep learning methods are the preferred tools. This study aimed to enhance the performance of existing approaches by utilizing an ensemble of deep learners based on attention mechanisms. To a great extent, the success of an ensemble model depends on the variety of its learners. To this end, we propose a novel loss function that enforces each learner to attend to different parts of news content on the one hand and obtain good classification accuracy on the other hand. Also, the learners are built on a common deep-feature extractor and only differ in their attention modules. As a result, the number of parameters is reduced efficiently and the overfitting problem is addressed. We conducted several experiments on some widely used fake-news detection datasets. The results confirm that the proposed method consistently surpasses the existing peer methods.     

Information Field Theory and Artificial Intelligence

Information field theory (IFT), the information theory for fields, is a mathematical framework for signal reconstruction and non-parametric inverse problems. Artificial intelligence (AI) and machine learning (ML) aim at generating intelligent systems, including such for perception, cognition, and learning. This overlaps with IFT, which is designed to address perception, reasoning, and inference tasks. Here, the relation between concepts and tools in IFT and those in AI and ML research are discussed. In the context of IFT, fields denote physical quantities that change continuously as a function of space (and time) and information theory refers to Bayesian probabilistic logic equipped with the associated entropic information measures. Reconstructing a signal with IFT is a computational problem similar to training a generative neural network (GNN) in ML. In this paper, the process of inference in IFT is reformulated in terms of GNN training. In contrast to classical neural networks, IFT based GNNs can operate without pre-training thanks to incorporating expert knowledge into their architecture. Furthermore, the cross-fertilization of variational inference methods used in IFT and ML are discussed. These discussions suggest that IFT is well suited to address many problems in AI and ML research and application.     

Regularization,Bayesian Inference,and Machine Learning Methods for Inverse Problems

Classical methods for inverse problems are mainly based on regularization theory, in particular those, that are based on optimization of a criterion with two parts: a data-model matching and a regularization term. Different choices for these two terms and a great number of optimization algorithms have been proposed. When these two terms are distance or divergence measures, they can have a Bayesian Maximum A Posteriori (MAP) interpretation where these two terms correspond to the likelihood and prior-probability models, respectively. The Bayesian approach gives more flexibility in choosing these terms and, in particular, the prior term via hierarchical models and hidden variables. However, the Bayesian computations can become very heavy computationally. The machine learning (ML) methods such as classification, clustering, segmentation, and regression, based on neural networks (NN) and particularly convolutional NN, deep NN, physics-informed neural networks, etc. can become helpful to obtain approximate practical solutions to inverse problems. In this tutorial article, particular examples of image denoising, image restoration, and computed-tomography (CT) image reconstruction will illustrate this cooperation between ML and inversion.     

自然循环流动不稳定性的多目标优化极限学习机预测方法

极限学习机是近年来提出的一种前向单隐层神经网络训练算法,具有训练速度快、不会陷入局部最优等优点,但其性能会受到随机选取的输入权值和阈值的影响.针对这一问题,提出一种基于多目标优化的改进极限学习机,将训练误差和输出层权值的均方最小化同时作为优化目标,采用带精英策略的快速非支配排序遗传算法对极限学习机的输入层到隐层的权值和阈值进行优化.将该算法应用于摇摆工况下自然循环系统不规则复合型流量脉动的多步滚动预测,分析了训练误差和输出层权值对不同步长预测效果的影响.仿真结果表明,优化极限学习机预测误差可以用较小的网络规模获得很好的泛化能力.为流动不稳定性的实时预测提供了一种准确度较高的途径,其预测结果可以作为核动力系统操作员的参考.     

Taming the Chaos in Neural Network Time Series Predictions

Machine learning methods, such as Long Short-Term Memory (LSTM) neural networks can predict real-life time series data. Here, we present a new approach to predict time series data combining interpolation techniques, randomly parameterized LSTM neural networks and measures of signal complexity, which we will refer to as complexity measures throughout this research. First, we interpolate the time series data under study. Next, we predict the time series data using an ensemble of randomly parameterized LSTM neural networks. Finally, we filter the ensemble prediction based on the original data complexity to improve the predictability, i.e., we keep only predictions with a complexity close to that of the training data. We test the proposed approach on five different univariate time series data. We use linear and fractal interpolation to increase the amount of data. We tested five different complexity measures for the ensemble filters for time series data, i.e., the Hurst exponent, Shannon’s entropy, Fisher’s information, SVD entropy, and the spectrum of Lyapunov exponents. Our results show that the interpolated predictions consistently outperformed the non-interpolated ones. The best ensemble predictions always beat a baseline prediction based on a neural network with only a single hidden LSTM, gated recurrent unit (GRU) or simple recurrent neural network (RNN) layer. The complexity filters can reduce the error of a random ensemble prediction by a factor of 10. Further, because we use randomly parameterized neural networks, no hyperparameter tuning is required. We prove this method useful for real-time time series prediction because the optimization of hyperparameters, which is usually very costly and time-intensive, can be circumvented with the presented approach.     

基于自适应遗忘因子极限学习机的高炉煤气预测

高炉煤气是钢铁企业重要的二次能源,其产生量和消耗量的实时准确预测对高炉煤气系统的平衡调度具有重要作用。但由于高炉煤气系统工况多变、产消量数据波动较大,给高炉煤气产消量的准确预测带来了很大的挑战。为此,通过对煤气产消量数据特征的深入分析,提出了一种基于自适应遗忘因子极限学习机（AF-ELM）的在线预测算法。在序贯极限学习机的基础上,引入遗忘因子逐步遗忘旧样本,通过预测误差反馈机制,自适应的调节遗忘因子,从而提高预测方法对系统工况的动态变化的适应能力,提高预测精度。将该算法应用于钢铁企业的高炉煤气产消量在线预测,实验结果表明与序贯极限学习机相比,该预测方法在系统工况变化的情况下能保持较高的预测精度,更适合于高炉煤气产消量的在线预测。     

水下目标多模态深度学习分类识别研究

水下目标的分类识别对于水声探测具有重要意义。提出一种水下目标多模态深度学习分类识别方法。针对水声信号的一维时域模态和二维频域模态特征建立一种多模态特征融合的深度学习结构,结合长短时记忆网络和卷积神经网络的优点,对一维时域信号和二维频谱信号分别进行并行处理,对输出进行典型相关分析,形成特征融合表示,并利用相邻帧的相关性进行参数优化。利用实测水声信号对算法进行了验证。结果表明：提出的算法对于水下目标识别的精度有显著的提高。     

An Optimized Black-Box Adversarial Simulator Attack Based on Meta-Learning

Much research on adversarial attacks has proved that deep neural networks have certain security vulnerabilities. Among potential attacks, black-box adversarial attacks are considered the most realistic based on the the natural hidden nature of deep neural networks. Such attacks have become a critical academic emphasis in the current security field. However, current black-box attack methods still have shortcomings, resulting in incomplete utilization of query information. Our research, based on the newly proposed Simulator Attack, proves the correctness and usability of feature layer information in a simulator model obtained by meta-learning for the first time. Then, we propose an optimized Simulator Attack+ based on this discovery. Our optimization methods used in Simulator Attack+ include: (1) a feature attentional boosting module that uses the feature layer information of the simulator to enhance the attack and accelerate the generation of adversarial examples; (2) a linear self-adaptive simulator-predict interval mechanism that allows the simulator model to be fully fine-tuned in the early stage of the attack and dynamically adjusts the interval for querying the black-box model; and (3) an unsupervised clustering module to provide a warm-start for targeted attacks. Results from experiments on the CIFAR-10 and CIFAR-100 datasets clearly show that Simulator Attack+ can further reduce the number of consuming queries to improve query efficiency while maintaining the attack.     

Forest Fire Detection via Feature Entropy Guided Neural Network

Forest fire detection from videos or images is vital to forest firefighting. Most deep learning based approaches rely on converging image loss, which ignores the content from different fire scenes. In fact, complex content of images always has higher entropy. From this perspective, we propose a novel feature entropy guided neural network for forest fire detection, which is used to balance the content complexity of different training samples. Specifically, a larger weight is given to the feature of the sample with a high entropy source when calculating the classification loss. In addition, we also propose a color attention neural network, which mainly consists of several repeated multiple-blocks of color-attention modules (MCM). Each MCM module can extract the color feature information of fire adequately. The experimental results show that the performance of our proposed method outperforms the state-of-the-art methods.     

Meta-Strategy for Learning Tuning Parameters with Guarantees

Online learning methods, similar to the online gradient algorithm (OGA) and exponentially weighted aggregation (EWA), often depend on tuning parameters that are difficult to set in practice. We consider an online meta-learning scenario, and we propose a meta-strategy to learn these parameters from past tasks. Our strategy is based on the minimization of a regret bound. It allows us to learn the initialization and the step size in OGA with guarantees. It also allows us to learn the prior or the learning rate in EWA. We provide a regret analysis of the strategy. It allows to identify settings where meta-learning indeed improves on learning each task in isolation.     

一种新型广义RBF神经网络在混沌时间序列预测中的研究

提出了一种新颖的广义径向基函数神经网络模型,其径向基函数(RBF)的形式由生成函数确定.然后,给出了易实现的梯度学习算法,同时为了进一步提高网络的收敛速度和网络性能,又给出了基于卡尔曼滤波的动态学习算法.为了验证网络的学习性能，采用基于卡尔曼滤波算法的新型广义RBF网络预测模型对Mackey-Glass混沌时间序列和Henon映射进行了仿真.结果表明,所提出的新型广义RBF神经网络模型能快速、精确地预测混沌时间序列,是研究复杂非线性动力系统辨识和控制的一种有效方法. 关键词： 广义径向基函数神经网络 卡尔曼滤波 梯度下降学习算法 混沌时间序列 预测     

A Color Image Encryption Algorithm Based on Double Fractional Order Chaotic Neural Network and Convolution Operation

A color image encryption algorithm based on double fractional order chaotic neural network (CNN), interlaced dynamic deoxyribonucleic acid (DNA) encoding and decoding, zigzag confusion, bidirectional bit-level diffusion and convolution operation is proposed. Firstly, two fractional order chaotic neural networks (CNNs) are proposed to explore the application of fractional order CNN in image encryption. Meanwhile, spectral entropy (SE) algorithm shows that the sequence generated by the proposed fractional order CNNs has better randomness. Secondly, a DNA encoding and decoding encryption scheme with evolutionary characteristics is adopted. In addition, convolution operation is utilized to improve the key sensitivity. Finally, simulation results and security analysis illustrate that the proposed algorithm has high security performance and can withstand classical cryptanalysis attacks.     

Dynamic Topology Reconfiguration of Boltzmann Machines on Quantum Annealers

Boltzmann machines have useful roles in deep learning applications, such as generative data modeling, initializing weights for other types of networks, or extracting efficient representations from high-dimensional data. Most Boltzmann machines use restricted topologies that exclude looping connectivity, as such connectivity creates complex distributions that are difficult to sample. We have used an open-system quantum annealer to sample from complex distributions and implement Boltzmann machines with looping connectivity. Further, we have created policies mapping Boltzmann machine variables to the quantum bits of an annealer. These policies, based on correlation and entropy metrics, dynamically reconfigure the topology of Boltzmann machines during training and improve performance.