Information-Theoretic Generalization Bounds for Meta-Learning and Applications

Meta-learning, or “learning to learn”, refers to techniques that infer an inductive bias from data corresponding to multiple related tasks with the goal of improving the sample efficiency for new, previously unobserved, tasks. A key performance measure for meta-learning is the meta-generalization gap, that is, the difference between the average loss measured on the meta-training data and on a new, randomly selected task. This paper presents novel information-theoretic upper bounds on the meta-generalization gap. Two broad classes of meta-learning algorithms are considered that use either separate within-task training and test sets, like model agnostic meta-learning (MAML), or joint within-task training and test sets, like reptile. Extending the existing work for conventional learning, an upper bound on the meta-generalization gap is derived for the former class that depends on the mutual information (MI) between the output of the meta-learning algorithm and its input meta-training data. For the latter, the derived bound includes an additional MI between the output of the per-task learning procedure and corresponding data set to capture within-task uncertainty. Tighter bounds are then developed for the two classes via novel individual task MI (ITMI) bounds. Applications of the derived bounds are finally discussed, including a broad class of noisy iterative algorithms for meta-learning.     

Meta-Learning-Based Physics-Informed Neural Network: Numerical Simulations of Initial Value Problems of Nonlinear Dynamical Systems without Labeled Data and Correlation Analyses

There are several main challenges in solving nonlinear differential equations with artificial neural networks (ANNs), such as a nonlinear system''s sensitivity to its initial values, discretization, and strategies for incorporating physics-based information into ANNs. As for the first issue, this paper addresses the initial value problems of nonlinear dynamical systems (a Duffing oscillator and a Burger''s equation), which cause large global truncation errors in sub-domains with a significant reduction in the influence of initial constraints, using meta-learning-based physics-informed neural networks (MPINNs). The MPINNs with dual learners outperform physics-informed neural networks with a single learner (no fine reinitialization capability). As a result, the former approach improves solution convergence by 98.83\% in the sub-time domain (III) of a Duffing oscillator, and by 85.89\% at $t = 45$ in a Burger''s equation problem, compared to the latter one. Model accuracy is highly dependent on the adaptability of the initial parameters in the first hidden layers of the meta-models. From correlation analyses, it is obvious that the parameters become less (the Duffing oscillator) or more (the Burger''s equation) correlated during fine reinitialization, as the update manner differs or is similar to the one used in pre-initialization. In the first example, the MPINN achieves both the mitigation of model sensitivity to its output and the improvement of model accuracy. Conversely, the second example shows that the proposed approach is not enough to solve both issues simultaneously, as increased model sensitivity to its output leads to higher model accuracy. The application of transfer learning reduces the number of iterative pre-meta-trainings.     

三维小样本元学习模型的大豆食心虫虫害高光谱检测

为降低大豆食心虫对大豆产量以及品质的影响,实现对大豆食心虫虫害的快速检测,提出了一种基于三维关系网络小样本元学习（3D-RN）模型的大豆食心虫虫害的检测方法。首先分别对附着虫卵的,附着食心虫幼虫的,被啃食的及正常的大豆各20颗进行高光谱图像采集,提取感兴趣区,建立基于高光谱图像的3D-RN模型。最终模型的正确率达82%±2.50%。对比与模型无关的元学习和匹配网络元学习模型,3D-RN模型能够充分度量样本特征间的距离,识别效果大大提升。研究表明,基于高光谱图像的3D-RN模型能够在少量样本情况下实现对大豆食心虫虫害的检测,将小样本元学习与高光谱结合的方法为虫害检测提供一种新思路。     

Meta-Strategy for Learning Tuning Parameters with Guarantees

Online learning methods, similar to the online gradient algorithm (OGA) and exponentially weighted aggregation (EWA), often depend on tuning parameters that are difficult to set in practice. We consider an online meta-learning scenario, and we propose a meta-strategy to learn these parameters from past tasks. Our strategy is based on the minimization of a regret bound. It allows us to learn the initialization and the step size in OGA with guarantees. It also allows us to learn the prior or the learning rate in EWA. We provide a regret analysis of the strategy. It allows to identify settings where meta-learning indeed improves on learning each task in isolation.     

Robust Spike-Based Continual Meta-Learning Improved by Restricted Minimum Error Entropy Criterion

The spiking neural network (SNN) is regarded as a promising candidate to deal with the great challenges presented by current machine learning techniques, including the high energy consumption induced by deep neural networks. However, there is still a great gap between SNNs and the online meta-learning performance of artificial neural networks. Importantly, existing spike-based online meta-learning models do not target the robust learning based on spatio-temporal dynamics and superior machine learning theory. In this invited article, we propose a novel spike-based framework with minimum error entropy, called MeMEE, using the entropy theory to establish the gradient-based online meta-learning scheme in a recurrent SNN architecture. We examine the performance based on various types of tasks, including autonomous navigation and the working memory test. The experimental results show that the proposed MeMEE model can effectively improve the accuracy and the robustness of the spike-based meta-learning performance. More importantly, the proposed MeMEE model emphasizes the application of the modern information theoretic learning approach on the state-of-the-art spike-based learning algorithms. Therefore, in this invited paper, we provide new perspectives for further integration of advanced information theory in machine learning to improve the learning performance of SNNs, which could be of great merit to applied developments with spike-based neuromorphic systems.     

高一学生化学元学习能力调查与对策

对2所中学的高一学生进行问卷调查发现,高一学生化学元学习能力处于中等水平,其中计划性、意识性、方法性、总结性等维度正向比例较低;男生的化学元学习能力水平总体上比女生略高,主要体现在计划性、补救性和总结性等3个维度,但女生的反馈性比男生略高;优秀学生、中等学生和学困生的化学元学习能力存在显著差异。提出鼓励自我设定学习目标培养学习计划性,提高元学习的意识性,加强元学习能力知识和策略的直接指导等措施培养元学习能力。