A data-efficient self-supervised deep learning model for design and characterization of nanophotonic structures

With its tremendous success in many machine learning and pattern recognition tasks, deep learning, as one type of data-driven models, has also led to many breakthroughs in other disciplines including physics, chemistry and material science. Nevertheless,the supremacy of deep learning over conventional optimization approaches heavily depends on the huge amount of data collected in advance to train the model, which is a common bottleneck of such a data-driven technique. In this work, we present a comprehensive deep learning model for the design and characterization of nanophotonic structures, where a self-supervised learning mechanism is introduced to alleviate the burden of data acquisition. Taking reflective metasurfaces as an example, we demonstrate that the self-supervised deep learning model can effectively utilize randomly generated unlabeled data during training, with the total test loss and prediction accuracy improved by about 15% compared with the fully supervised counterpart.The proposed self-supervised learning scheme provides an efficient solution for deep learning models in some physics-related tasks where labeled data are limited or expensive to collect.     

Benchmarking Attention-Based Interpretability of Deep Learning in Multivariate Time Series Predictions

The adaptation of deep learning models within safety-critical systems cannot rely only on good prediction performance but needs to provide interpretable and robust explanations for their decisions. When modeling complex sequences, attention mechanisms are regarded as the established approach to support deep neural networks with intrinsic interpretability. This paper focuses on the emerging trend of specifically designing diagnostic datasets for understanding the inner workings of attention mechanism based deep learning models for multivariate forecasting tasks. We design a novel benchmark of synthetically designed datasets with the transparent underlying generating process of multiple time series interactions with increasing complexity. The benchmark enables empirical evaluation of the performance of attention based deep neural networks in three different aspects: (i) prediction performance score, (ii) interpretability correctness, (iii) sensitivity analysis. Our analysis shows that although most models have satisfying and stable prediction performance results, they often fail to give correct interpretability. The only model with both a satisfying performance score and correct interpretability is IMV-LSTM, capturing both autocorrelations and crosscorrelations between multiple time series. Interestingly, while evaluating IMV-LSTM on simulated data from statistical and mechanistic models, the correctness of interpretability increases with more complex datasets.     

从二维视图识别三维目标的多网络融合方法

提出了一种从二维视图识别三维目标的多网络融合方法,基于单个网络分类的置信度概念,有效地结合多个网络的输出结果作出最终分类判决,应用三个多层前向网络（隐层神经元数,初始权值等取不同值）,设计了基于分类确认度的多网络融合结构,对四类车辆目标进行的识别实验表明,所提出的多网络融合方法明显优于单个网络的识别性能。     

深度学习在超声检测缺陷识别中的应用与发展*

深度学习（Deep Learning）是目前最强大的机器学习算法之一,其中卷积神经网络（Convolutional Neural Network, CNN）模型具有自动学习特征的能力,在图像处理领域较其他深度学习模型有较大的性能优势。本文先简述了深度学习的发展史,然后综述了深度学习在超声检测缺陷识别中的应用与发展,从早期浅层神经网络到现在深度学习的应用现状,并借鉴医学影像识别和射线图像识别领域的方法,分析了卷积神经网络对超声图像缺陷识别的适用性。最后,探讨归纳了目前在超声检测图像识别中使用CNN存在的一些问题,及其主要应对策略的研究方向。     

Application of Improved Asynchronous Advantage Actor Critic Reinforcement Learning Model on Anomaly Detection

Anomaly detection research was conducted traditionally using mathematical and statistical methods. This topic has been widely applied in many fields. Recently reinforcement learning has achieved exceptional successes in many areas such as the AlphaGo chess playing and video gaming etc. However, there were scarce researches applying reinforcement learning to the field of anomaly detection. This paper therefore aimed at proposing an adaptable asynchronous advantage actor-critic model of reinforcement learning to this field. The performances were evaluated and compared among classical machine learning and the generative adversarial model with variants. Basic principles of the related models were introduced firstly. Then problem definitions, modelling processes and testing were detailed. The proposed model differentiated the sequence and image from other anomalies by proposing appropriate neural networks of attention mechanism and convolutional network for the two kinds of anomalies, respectively. Finally, performances with classical models using public benchmark datasets (NSL-KDD, AWID and CICIDS-2017, DoHBrw-2020) were evaluated and compared. Experiments confirmed the effectiveness of the proposed model with the results indicating higher rewards and lower loss rates on the datasets during training and testing. The metrics of precision, recall rate and F1 score were higher than or at least comparable to the state-of-the-art models. We concluded the proposed model could outperform or at least achieve comparable results with the existing anomaly detection models.     

Probabilistic Models with Deep Neural Networks

Recent advances in statistical inference have significantly expanded the toolbox of probabilistic modeling. Historically, probabilistic modeling has been constrained to very restricted model classes, where exact or approximate probabilistic inference is feasible. However, developments in variational inference, a general form of approximate probabilistic inference that originated in statistical physics, have enabled probabilistic modeling to overcome these limitations: (i) Approximate probabilistic inference is now possible over a broad class of probabilistic models containing a large number of parameters, and (ii) scalable inference methods based on stochastic gradient descent and distributed computing engines allow probabilistic modeling to be applied to massive data sets. One important practical consequence of these advances is the possibility to include deep neural networks within probabilistic models, thereby capturing complex non-linear stochastic relationships between the random variables. These advances, in conjunction with the release of novel probabilistic modeling toolboxes, have greatly expanded the scope of applications of probabilistic models, and allowed the models to take advantage of the recent strides made by the deep learning community. In this paper, we provide an overview of the main concepts, methods, and tools needed to use deep neural networks within a probabilistic modeling framework.     

Supervised Deep Learning in High Energy Phenomenology: a Mini Review *

Deep learning, a branch of machine learning, has been recently applied to high energy experimental and phenomenological studies. In this note we give a brief review on those applications using supervised deep learning. We first describe various learning models and then recapitulate their applications to high energy phenomenological studies. Some detailed applications are delineated in details, including the machine learning scan in the analysis of new physics parameter space, the graph neural networks in the search of top-squark production and in the $CP$ measurement of the top-Higgs coupling at the LHC.      

利用深度神经网络和小波包变换进行缺陷类型分析

超声检测中对缺陷进行类型分析通常取决于操作人员对于特定专业知识的了解及检测经验,从而导致其分析结果的不稳定性和个体差异性。本文提出了一种使用小波包变换提取缺陷特征信息,并应用深度神经网络对得到的信息进行分类识别的方法。利用超声相控阵系统对于不锈钢试块上的通孔、斜通孔和平底孔进行超声检测,并对得到的超声回波波形按照新方法进行分析。实验结果表明,使用小波包变换后的数据进行分类识别能够在提高识别准确率的同时降低神经网络的学习时间,而使用深度神经网络相比通用的BP神经网络以可接受延长学习时间的代价提高了识别的准确率。采用新方法后,缺陷分类正确率提高了21.66%,而网络学习时间只延长了91.9s。在超声检测中使用小波包变换和深度神经网络来对于缺陷进行类型分析,能够排除人为干扰,增加识别准确率,对于实际应用有着极大的意义。      

深度学习在核物理中的应用

深度学习是目前最好的模式识别工具，预期会在核物理领域帮助科学家从大量复杂数据中寻找与某些物理最相关的特征。本文综述了深度学习技术的分类，不同数据结构对应的最优神经网络架构，黑盒模型的可解释性与预测结果的不确定性。介绍了深度学习在核物质状态方程、核结构、原子核质量、衰变与裂变方面的应用，并展示如何训练神经网络预测原子核质量。结果发现使用实验数据训练的神经网络模型对未参与训练的实验数据拥有良好的预测能力。基于已有的实验数据外推，神经网络对丰中子的轻原子核质量预测结果与宏观微观液滴模型有较大偏离。此区域可能存在未被宏观微观液滴模型包含的新物理，需要进一步的实验数据验证。     

Multi-task learning approach for modulation and wireless signal classification for 5G and beyond: Edge deployment via model compression

Future communication networks must address the scarce spectrum to accommodate extensive growth of heterogeneous wireless devices. Efforts are underway to address spectrum coexistence, enhance spectrum awareness, and bolster authentication schemes. Wireless signal recognition is becoming increasingly more significant for spectrum monitoring, spectrum management, secure communications, among others. Consequently, comprehensive spectrum awareness on the edge has the potential to serve as a key enabler for the emerging beyond 5G (fifth generation) networks. State-of-the-art studies in this domain have (i) only focused on a single task – modulation or signal (protocol) classification – which in many cases is insufficient information for a system to act on, (ii) consider either radar or communication waveforms (homogeneous waveform category), and (iii) does not address edge deployment during neural network design phase. In this work, for the first time in the wireless communication domain, we exploit the potential of deep neural networks based multi-task learning (MTL) framework to simultaneously learn modulation and signal classification tasks while considering heterogeneous wireless signals such as radar and communication waveforms in the electromagnetic spectrum. The proposed MTL architecture benefits from the mutual relation between the two tasks in improving the classification accuracy as well as the learning efficiency with a lightweight neural network model. We additionally include experimental evaluations of the model with over-the-air collected samples and demonstrate first-hand insight on model compression along with deep learning pipeline for deployment on resource-constrained edge devices. We demonstrate significant computational, memory, and accuracy improvement of the proposed model over two reference architectures. In addition to modeling a lightweight MTL model suitable for resource-constrained embedded radio platforms, we provide a comprehensive heterogeneous wireless signals dataset for public use.     

UWB LOS/NLOS identification in multiple indoor environments using deep learning methods

Indoor location-aware service is booming in daily life and business activities, making the demand for precise indoor positioning systems thrive. The identification between line-of-sight (LOS) and non-line-of-sight (NLOS) is critical for wireless indoor time-of-arrival-based localization methods. Ultra-Wide-Band (UWB) is considered low cost among the many wireless positioning systems. It can resolve multi-path and have high penetration ability. This contribution addresses UWB NLOS/LOS identification problem in multiple environments. We propose a LOS/NLOS identification method using Convolutional Neural Network parallel with Gate Recurrent Unit, named Indoor NLOS/LOS identification Neural Network. The Convolutional Neural Network extracts spatial features of UWB channel impulse response data. While the Gate Recurrent Unit is an effective approach for designing deep recurrent neural networks which can extract temporal features. By integrating squeeze-and-extraction blocks into these architectures we can assign weights on channel-wise features. We simulated UWB channel impulse response signals in residential, office, and industrial scenarios based on the IEEE 802.15.4a channel model report. The presented network was tested in simulation scenarios and an open-source real-time measured dataset. Our method can solve NLOS identification problems for multiple indoor environments. Thus more versatile compare with networks only working in one scenario. Popular machine learning methods and deep learning methods are compared against our method. The test results show that the proposed network outperforms benchmark methods in simulation datasets and real-time measured datasets.     

Learning from Scarce Information: Using Synthetic Data to Classify Roman Fine Ware Pottery

In this article, we consider a version of the challenging problem of learning from datasets whose size is too limited to allow generalisation beyond the training set. To address the challenge, we propose to use a transfer learning approach whereby the model is first trained on a synthetic dataset replicating features of the original objects. In this study, the objects were smartphone photographs of near-complete Roman terra sigillata pottery vessels from the collection of the Museum of London. Taking the replicated features from published profile drawings of pottery forms allowed the integration of expert knowledge into the process through our synthetic data generator. After this first initial training the model was fine-tuned with data from photographs of real vessels. We show, through exhaustive experiments across several popular deep learning architectures, different test priors, and considering the impact of the photograph viewpoint and excessive damage to the vessels, that the proposed hybrid approach enables the creation of classifiers with appropriate generalisation performance. This performance is significantly better than that of classifiers trained exclusively on the original data, which shows the promise of the approach to alleviate the fundamental issue of learning from small datasets.     

Using physics-informed enhanced super-resolution generative adversarial networks for subfilter modeling in turbulent reactive flows

Turbulence is still one of the main challenges in accurate prediction of reactive flows. Therefore, the development of new turbulence closures that can be applied to combustion problems is essential. Over the last few years, data-driven modeling has become popular in many fields as large, often extensively labeled datasets are now available and training of large neural networks has become possible on graphics processing units (GPUs) that speed up the learning process tremendously. However, the successful application of deep neural networks in fluid dynamics, such as in subfilter modeling in the context of large-eddy simulations (LESs), is still challenging. Reasons for this are the large number of degrees of freedom in natural flows, high requirements of accuracy and error robustness, and open questions, for example, regarding the generalization capability of trained neural networks in such high-dimensional, physics-constrained scenarios. This work presents a novel subfilter modeling approach based on a generative adversarial network (GAN), which is trained with unsupervised deep learning (DL) using adversarial and physics-informed losses. A two-step training method is employed to improve the generalization capability, especially extrapolation, of the network. The novel approach gives good results in a priori and a posteriori tests with decaying turbulence including turbulent mixing, and the importance of the physics-informed continuity loss term is demonstrated. The applicability of the network in complex combustion scenarios is furthermore discussed by employing it in reactive and inert LESs of the Spray A case defined by the Engine Combustion Network (ECN).     

稀疏降噪自编码算法用于近红外光谱鉴别真假药的研究

近红外光谱分析技术作为一种快速、无损检测技术十分适用于真假药品现场鉴别。自编码网络作为当前机器学习领域研究的热点受到广泛关注,自编码网络是一种典型的深度学习网络模型,它比传统的潜层学习方法具有更强的模型表示能力。自编码网络使用贪婪逐层预训练算法,通过最小化各层网络的重构误差,依次训练网络的每一层,进而训练整个网络。通过对数据进行白化预处理并使用无监督算法对输入数据进行逐层重构,使网络更有效的学习到数据的内部结构特征。之后使用带标签数据通过监督学习算法对整个网络进行调优。首先对真假琥乙红霉素片的近红外光谱数据进行预处理以及白化预处理,通过白化处理降低数据特征之间的相关性,使数据各特征具有相同的方差。数据处理之后利用稀疏降噪自编码网络针对真假药品光谱数据建立分类模型,并将稀疏降噪自编码网络模型与BP神经网络以及SVM算法在分类准确率及算法稳定性方面进行对比。结果表明对光谱数据进行白化预处理能有效提升稀疏降噪自编码网络的分类准确率。并且自编码网络分类准确率在不同训练样本数量下均高于BP神经网络,SVM算法在少量训练样本的情况下更有优势,但在训练数据集样本数达到一定数量后,自编码网络的分类准确率将优于SVM算法。在算法稳定性方面,自编码网络较之BP神经网络和SVM算法也更稳定。使用稀疏降噪自编码网络对真假药品近红外光谱数据进行建模,能对真假药品进行有效的鉴别。     

Deep transfer reinforcement learning for resource allocation in hybrid multiple access systems

This paper proposes a resource allocation scheme for hybrid multiple access involving both orthogonal multiple access and non-orthogonal multiple access (NOMA) techniques. The proposed resource allocation scheme employs multi-agent deep reinforcement learning (MA-DRL) to maximize the sum-rate for all users. More specifically, the MA-DRL-based scheme jointly allocates subcarrier and power resources for users by utilizing deep Q networks and multi-agent deep deterministic policy gradient networks. Meanwhile, an adaptive learning determiner mechanism is introduced into our allocation scheme to achieve better sum-rate performance. However, the above deep reinforcement learning adopted by our scheme cannot optimize parameters quickly in the new communication model. In order to better adapt to the new environment and make the resource allocation strategy more robust, we propose a transfer learning scheme based on deep reinforcement learning (T-DRL). The T-DRL-based scheme allows us to transfer the subcarrier allocation network and the power allocation network collectively or independently. Simulation results show that the proposed MA-DRL-based resource allocation scheme can achieve better sum-rate performance. Furthermore, the T-DRL-based scheme can effectively improve the convergence speed of the deep resource allocation network.     

一种基于无监督主动学习的苹果品质光谱无损检测模型构建方法

利用光谱技术实现农产品、食品品质无损检测的实质是建立样本光谱信息与样本品质参数之间的机器学习模型。为了获得具有良好泛化性能的机器学习模型,通常需要大量的标记样本,然而,获取样本的光谱信息相对容易,但标注样本品质参数的过程往往涉及到大量的时间和经济成本,并且具有破坏性。主动学习是一种减少训练集有标记样本数量的方法,通过选择最有价值的样本进行标记,而不是随机选择。因此,主动学习能够控制向训练集添加哪些样本,模型不再是被动地接受用于建模的样本。在分类任务中已经提出较多关于主动学习的算法,但回归任务中的研究却相对较少,且现有的用于回归任务的主动学习算法大多是有监督的,即需要少量有标记样本训练初始模型。本文提出了一种基于无监督主动学习方法的训练样本选择策略。该方法首先通过层次凝聚聚类对无标记（标准值）光谱数据集进行多样性划分,获得不同的聚类簇;然后通过局部线性重建算法在每个聚类簇中选择最具代表性的样本构成训练样本集,最后基于训练集构建模型。利用两个年份三个品种苹果的近红外光谱数据,构建了其可溶性固形物含量和硬度的偏最小二乘预测模型,用于验证所提出方法的有效性。实验结果表明：所提出的方法要优于已有的样本选择策略,可以有效地提高模型精度,减少在模型训练中的破坏性理化实验。同时,与随机采样（RS）、Kennard-Stone算法（KS）、光谱-理化值共生距离算法（SPXY）这三种光谱领域常用的样本选择算法相比,该研究所提出的方法表现出了最佳的性能, 基于所提出的无监督主动学习算法选取200个样本作为训练集所建立的可溶性固形物含量预测模型的预测均方根误差相对于其他三种算法降低了2.0%~13.2%,硬度预测模型的预测均方根误差相对降低了1.2%~15.7%。     

神经网络的声场景自动分类方法*

声场景探察和自动分类能帮助人类制定应对特定环境的正确策略,具有重要的研究价值。随着卷积神经网络的发展,出现了许多基于卷积神经网络的声场景分类方法。其中时频卷积神经网络(TS-CNN)采用了时频注意力模块,是目前声场景分类效果最好的网络之一。为了在保持网络复杂度不变的前提下进一步提高网络的声场景分类性能,该文提出了一种基于协同学习的时频卷积神经网络模型(TSCNN-CL)。具体地说,该文首先建立了基于同构结构的辅助分支参与网络的训练。其次,提出了一种基于KL散度的协同损失函数,实现了分支与主干的知识协同,最后,在测试过程中,为了不增加推理计算量,该文提出的模型只使用主干网络预测结果。在ESC-10、ESC-50和UrbanSound8k数据集的综合实验表明,该模型分类效果要优于TS-CNN模型以及当前大部分的主流方法。     

Self-Supervised Variational Auto-Encoders

Density estimation, compression, and data generation are crucial tasks in artificial intelligence. Variational Auto-Encoders (VAEs) constitute a single framework to achieve these goals. Here, we present a novel class of generative models, called self-supervised Variational Auto-Encoder (selfVAE), which utilizes deterministic and discrete transformations of data. This class of models allows both conditional and unconditional sampling while simplifying the objective function. First, we use a single self-supervised transformation as a latent variable, where the transformation is either downscaling or edge detection. Next, we consider a hierarchical architecture, i.e., multiple transformations, and we show its benefits compared to the VAE. The flexibility of selfVAE in data reconstruction finds a particularly interesting use case in data compression tasks, where we can trade-off memory for better data quality and vice-versa. We present the performance of our approach on three benchmark image data (Cifar10, Imagenette64, and CelebA).     

基于密集连接时延神经网络的说话人识别算法*

说话人识别技术是一项重要的生物特征识别技术。近年来,使用深度神经网络提取发声特征的说话人识别算法取得了突出成果。时延神经网络作为其中的典型代表之一已被证明具有出色的特征提取能力。为进一步提升识别准确率并节约计算资源,通过对现有的说话人识别算法进行研究,提出一种带有注意力机制的密集连接时延神经网络用于说话人识别。密集连接的网络结构在增强不同网络层之间的信息复用的同时能有效控制模型体积。通道注意力机制和帧注意力机制帮助网络聚焦于更关键的细节特征,使得通过统计池化提取出的说话人特征更具有代表性。实验结果表明,在VoxCeleb1测试数据集上取得了1.40%的等错误率（EER）和0.15的最小检测代价标准（DCF）,证明了在说话人识别任务上的有效性。