Information-Theoretic Generalization Bounds for Meta-Learning and Applications

Meta-learning, or “learning to learn”, refers to techniques that infer an inductive bias from data corresponding to multiple related tasks with the goal of improving the sample efficiency for new, previously unobserved, tasks. A key performance measure for meta-learning is the meta-generalization gap, that is, the difference between the average loss measured on the meta-training data and on a new, randomly selected task. This paper presents novel information-theoretic upper bounds on the meta-generalization gap. Two broad classes of meta-learning algorithms are considered that use either separate within-task training and test sets, like model agnostic meta-learning (MAML), or joint within-task training and test sets, like reptile. Extending the existing work for conventional learning, an upper bound on the meta-generalization gap is derived for the former class that depends on the mutual information (MI) between the output of the meta-learning algorithm and its input meta-training data. For the latter, the derived bound includes an additional MI between the output of the per-task learning procedure and corresponding data set to capture within-task uncertainty. Tighter bounds are then developed for the two classes via novel individual task MI (ITMI) bounds. Applications of the derived bounds are finally discussed, including a broad class of noisy iterative algorithms for meta-learning.     

Hotel Performance in the UK: The Role of Information Entropy in a Novel Slack-Based Data Envelopment Analysis

Previous hotel performance studies neglected the role of information entropy in feedback processes between input and output management. This paper focuses on this gap by exploring the relationship between hotel performance at the industry level and the capability of learning by doing and adopting best practices using a sample of 153 UK hotels over a 10-year period between 2008–2017. Besides, this research also fills a literature gap by addressing the issues of measuring hotel performance in light of negative outputs. In order to achieve this, we apply a novel Modified slack-based model for the efficiency analysis and Least Absolute Shrinkage and Selection Operator to examine the influence of entropy related variable on efficiency score. The Results indicate that less can be learnt from inputs than from outputs to improve efficiency levels and resource allocation is more balanced than cash flow and liquidity. The findings suggest that market dynamics explains the cash flow generation potential and liquidity. We find that market conditions are increasingly offering the opportunities for learning and improving hotel efficiency. The results report that the distinctive characteristic of superior performance in hotel operations is the capability to match the cash flow generation potential with market opportunities.     

The Role of Information in Managing Interactions from a Multifractal Perspective

In the framework of the multifractal hydrodynamic model, the correlations informational entropy–cross-entropy manages attractive and repulsive interactions through a multifractal specific potential. The classical dynamics associated with them imply Hubble-type effects, Galilei-type effects, and dependences of interaction constants with multifractal degrees at various scale resolutions, while the insertion of the relativistic amendments in the same dynamics imply multifractal transformations of a generalized Lorentz-type, multifractal metrics invariant to these transformations, and an estimation of the dimension of the multifractal Universe. In such a context, some correspondences with standard cosmologies are analyzed. Since the same types of interactions can also be obtained as harmonics mapping between the usual space and the hyperbolic plane, two measures with uniform and non-uniform temporal flows become functional, temporal measures analogous with Milne’s temporal measures in a more general manner. This work furthers the analysis published recently by our group in “Towards Interactions through Information in a Multifractal Paradigm”.     

Evidence of Strange Attractors in Class C Amplifier with Single Bipolar Transistor: Polynomial and Piecewise-Linear Case

This paper presents and briefly discusses recent observations of dynamics associated with isolated generalized bipolar transistor cells. A mathematical model of this simple system is considered on the highest level of abstraction such that it comprises many different network topologies. The key property of the analyzed structure is its bias point since the transistor is modeled via two-port admittance parameters. A necessary but not sufficient condition for the evolution of autonomous complex behavior is the nonlinear bilateral nature of the transistor with arbitrary reason that causes this effect. It is proved both by numerical analysis and experimental measurement that chaotic motion is miscellaneous, robust, and it is neither numerical artifact nor long transient motion.     

Survival of Virus Particles in Water Droplets: Hydrophobic Forces and Landauer’s Principle

Many small biological objects, such as viruses, survive in a water environment and cannot remain active in dry air without condensation of water vapor. From a physical point of view, these objects belong to the mesoscale, where small thermal fluctuations with the characteristic kinetic energy of k_BT (where k_B is the Boltzmann’s constant and T is the absolute temperature) play a significant role. The self-assembly of viruses, including protein folding and the formation of a protein capsid and lipid bilayer membrane, is controlled by hydrophobic forces (i.e., the repulsing forces between hydrophobic particles and regions of molecules) in a water environment. Hydrophobic forces are entropic, and they are driven by a system’s tendency to attain the maximum disordered state. On the other hand, in information systems, entropic forces are responsible for erasing information, if the energy barrier between two states of a switch is on the order of k_BT, which is referred to as Landauer’s principle. We treated hydrophobic interactions responsible for the self-assembly of viruses as an information-processing mechanism. We further showed a similarity of these submicron-scale processes with the self-assembly in colloidal crystals, droplet clusters, and liquid marbles.     

A Bootstrap Framework for Aggregating within and between Feature Selection Methods

In the past decade, big data has become increasingly prevalent in a large number of applications. As a result, datasets suffering from noise and redundancy issues have necessitated the use of feature selection across multiple domains. However, a common concern in feature selection is that different approaches can give very different results when applied to similar datasets. Aggregating the results of different selection methods helps to resolve this concern and control the diversity of selected feature subsets. In this work, we implemented a general framework for the ensemble of multiple feature selection methods. Based on diversified datasets generated from the original set of observations, we aggregated the importance scores generated by multiple feature selection techniques using two methods: the Within Aggregation Method (WAM), which refers to aggregating importance scores within a single feature selection; and the Between Aggregation Method (BAM), which refers to aggregating importance scores between multiple feature selection methods. We applied the proposed framework on 13 real datasets with diverse performances and characteristics. The experimental evaluation showed that WAM provides an effective tool for determining the best feature selection method for a given dataset. WAM has also shown greater stability than BAM in terms of identifying important features. The computational demands of the two methods appeared to be comparable. The results of this work suggest that by applying both WAM and BAM, practitioners can gain a deeper understanding of the feature selection process.     

A Unified Approach to Local Quantum Uncertainty and Interferometric Power by Metric Adjusted Skew Information

Local quantum uncertainty and interferometric power were introduced by Girolami et al. as geometric quantifiers of quantum correlations. The aim of the present paper is to discuss their properties in a unified manner by means of the metric adjusted skew information defined by Hansen.     

Fisher Information of Free-Electron Landau States

An electron in a constant magnetic field has energy levels, known as the Landau levels. One can obtain the corresponding radial wavefunction of free-electron Landau states in cylindrical polar coordinates. However, this system has not been explored so far in terms of an information-theoretical viewpoint. Here, we focus on Fisher information associated with these Landau states specified by the two quantum numbers. Fisher information provides a useful measure of the electronic structure in quantum systems, such as hydrogen-like atoms and under some potentials. By numerically evaluating the generalized Laguerre polynomials in the radial densities, we report that Fisher information increases linearly with the principal quantum number that specifies energy levels, but decreases monotonically with the azimuthal quantum number m. We also present relative Fisher information of the Landau states against the reference density with m=0, which is proportional to the principal quantum number. We compare it with the case when the lowest Landau level state is set as the reference.     

Maximum Entropy Approach to Reliability of Multi-Component Systems with Non-Repairable or Repairable Components

The degradation and recovery processes are multi-scale phenomena in many physical, engineering, biological, and social systems, and determine the aging of the entire system. Therefore, understanding the interplay between the two processes at the component level is the key to evaluate the reliability of the system. Based on the principle of maximum entropy, an approach is proposed to model and infer the processes at the component level, and is applied to repairable and non-repairable systems. By incorporating the reliability block diagram, this approach allows for integrating the information of network connectivity and statistical moments to infer the hazard or recovery rates of the degradation or recovery processes. The overall approach is demonstrated with numerical examples.     

A Structural Entropy Measurement Principle of Propositional Formulas in Conjunctive Normal Form

The satisfiability (SAT) problem is a core problem in computer science. Existing studies have shown that most industrial SAT instances can be effectively solved by modern SAT solvers while random SAT instances cannot. It is believed that the structural characteristics of different SAT formula classes are the reasons behind this difference. In this paper, we study the structural properties of propositional formulas in conjunctive normal form (CNF) by the principle of structural entropy of formulas. First, we used structural entropy to measure the complex structure of a formula and found that the difficulty solving the formula is related to the structural entropy of the formula. The smaller the compressing information of a formula, the more difficult it is to solve the formula. Secondly, we proposed a λ-approximation strategy to approximate the structural entropy of large formulas. The experimental results showed that the proposed strategy can effectively approximate the structural entropy of the original formula and that the approximation ratio is more than 92%. Finally, we analyzed the structural properties of a formula in the solution process and found that a local search solver tends to select variables in different communities to perform the next round of searches during a search and that the structural entropy of a variable affects the probability of the variable being flipped. By using these conclusions, we also proposed an initial candidate solution generation strategy for a local search for SAT, and the experimental results showed that this strategy effectively improves the performance of the solvers CCAsat and Sparrow2011 when incorporated into these two solvers.