Scalable and Transferable Reinforcement Learning for Multi-Agent Mixed Cooperative–Competitive Environments Based on Hierarchical Graph Attention

Most previous studies on multi-agent systems aim to coordinate agents to achieve a common goal, but the lack of scalability and transferability prevents them from being applied to large-scale multi-agent tasks. To deal with these limitations, we propose a deep reinforcement learning (DRL) based multi-agent coordination control method for mixed cooperative–competitive environments. To improve scalability and transferability when applying in large-scale multi-agent systems, we construct inter-agent communication and use hierarchical graph attention networks (HGAT) to process the local observations of agents and received messages from neighbors. We also adopt the gated recurrent units (GRU) to address the partial observability issue by recording historical information. The simulation results based on a cooperative task and a competitive task not only show the superiority of our method, but also indicate the scalability and transferability of our method in various scale tasks.     

Optimal Energy-Storage Configuration for Microgrids Based on SOH Estimation and Deep Q-Network

Energy storage is an important adjustment method to improve the economy and reliability of a power system. Due to the complexity of the coupling relationship of elements such as the power source, load, and energy storage in the microgrid, there are problems of insufficient performance in terms of economic operation and efficient dispatching. In view of this, this paper proposes an energy storage configuration optimization model based on reinforcement learning and battery state of health assessment. Firstly, a quantitative assessment of battery health life loss based on deep learning was performed. Secondly, on the basis of considering comprehensive energy complementarity, a two-layer optimal configuration model was designed to optimize the capacity configuration and dispatch operation. Finally, the feasibility of the proposed method in microgrid energy storage planning and operation was verified by experimentation. By integrating reinforcement learning and traditional optimization methods, the proposed method did not rely on the accurate prediction of the power supply and load and can make decisions based only on the real-time information of the microgrid. In this paper, the advantages and disadvantages of the proposed method and existing methods were analyzed, and the results show that the proposed method can effectively improve the performance of dynamic planning for energy storage in microgrids.     

Andrographolide Inhibits Biofilm and Virulence in Listeria monocytogenes as a Quorum-Sensing Inhibitor

Listeria monocytogenes is a major foodborne pathogen that can cause listeriosis in humans and animals. Andrographolide is known as a natural antibiotic and exhibits good antibacterial activity. We aimed to investigate the effect of andrographolide on two quorum-sensing (QS) systems, LuxS/AI-2 and Agr/AIP of L. monocytogenes, as well as QS-controlled phenotypes in this study. Our results showed that neither luxS expression nor AI-2 production was affected by andrographolide. Nevertheless, andrographolide significantly reduced the expression levels of the agr genes and the activity of the agr promoter P₂. Results from the crystal violet staining method, confocal laser scanning microscopy (CLSM), and field emission scanning electron microscopy (FE-SEM) demonstrated that andrographolide remarkably inhibited the biofilm-forming ability of L. monocytogenes 10403S. The preformed biofilms were eradicated when exposed to andrographolide, and reduced surviving cells were also observed in treated biofilms. L. monocytogenes treated with andrographolide exhibited decreased ability to secrete LLO and adhere to and invade Caco-2 cells. Therefore, andrographolide is a potential QS inhibitor by targeting the Agr QS system to reduce biofilm formation and virulence of L. monocytogenes.     

Multi-Classifier Fusion Based on MI–SFFS for Cross-Subject Emotion Recognition

With the widespread use of emotion recognition, cross-subject emotion recognition based on EEG signals has become a hot topic in affective computing. Electroencephalography (EEG) can be used to detect the brain’s electrical activity associated with different emotions. The aim of this research is to improve the accuracy by enhancing the generalization of features. A Multi-Classifier Fusion method based on mutual information with sequential forward floating selection (MI_SFFS) is proposed. The dataset used in this paper is DEAP, which is a multi-modal open dataset containing 32 EEG channels and multiple other physiological signals. First, high-dimensional features are extracted from 15 EEG channels of DEAP after using a 10 s time window for data slicing. Second, MI and SFFS are integrated as a novel feature-selection method. Then, support vector machine (SVM), k-nearest neighbor (KNN) and random forest (RF) are employed to classify positive and negative emotions to obtain the output probabilities of classifiers as weighted features for further classification. To evaluate the model performance, leave-one-out cross-validation is adopted. Finally, cross-subject classification accuracies of 0.7089, 0.7106 and 0.7361 are achieved by the SVM, KNN and RF classifiers, respectively. The results demonstrate the feasibility of the model by splicing different classifiers’ output probabilities as a portion of the weighted features.     

Synthesis of PNIPAAm-g-P4VP Microgel as Draw Agent in Forward Osmosis by RAFT Polymerization and Reverse Suspension Polymerization to Improve Water Flux

Microgels have unique and versatile properties allowing their use in forward osmosis areas as a draw agent. In this contribution, poly(4-vinylpyridine) (P4VP) was synthesized via RAFT polymerization and then grafted to a poly(N-Isopropylacrylamide) (PNIPAAm) crosslinking network by reverse suspension polymerization. P4VP was successfully obtained by the quasiliving polymerization with the result of nuclear magnetic resonance and gel permeation chromatography characterization. The particle size and particle size distribution of the PNIPAAm-g-P4VP microgels containing 0, 5, 10, 15 and 20 wt% P4VP were measured by means of a laser particle size analyzer. It was found that all the microgels were of micrometer scale and the particle size was increased with the P4VP load. Inter/intra-molecular-specific interactions, i.e., hydrogen bond interactions were then investigated by Fourier infrared spectroscopy. In addition, the water flux measurements showed that all the PNIPAAm-g-P4VP microgels can draw water more effectively than a blank PNIPAAm microgel. For the copolymer microgel incorporating 20 wt% P4VP, the water flux was measured to be 7.48 L∙m⁻²∙h⁻¹.     

Space-Air-Ground Integrated Mobile Crowdsensing for Partially Observable Data Collection by Multi-Scale Convolutional Graph Reinforcement Learning

Mobile crowdsensing (MCS) is attracting considerable attention in the past few years as a new paradigm for large-scale information sensing. Unmanned aerial vehicles (UAVs) have played a significant role in MCS tasks and served as crucial nodes in the newly-proposed space-air-ground integrated network (SAGIN). In this paper, we incorporate SAGIN into MCS task and present a Space-Air-Ground integrated Mobile CrowdSensing (SAG-MCS) problem. Based on multi-source observations from embedded sensors and satellites, an aerial UAV swarm is required to carry out energy-efficient data collection and recharging tasks. Up to date, few studies have explored such multi-task MCS problem with the cooperation of UAV swarm and satellites. To address this multi-agent problem, we propose a novel deep reinforcement learning (DRL) based method called Multi-Scale Soft Deep Recurrent Graph Network (ms-SDRGN). Our ms-SDRGN approach incorporates a multi-scale convolutional encoder to process multi-source raw observations for better feature exploitation. We also use a graph attention mechanism to model inter-UAV communications and aggregate extra neighboring information, and utilize a gated recurrent unit for long-term performance. In addition, a stochastic policy can be learned through a maximum-entropy method with an adjustable temperature parameter. Specifically, we design a heuristic reward function to encourage the agents to achieve global cooperation under partial observability. We train the model to convergence and conduct a series of case studies. Evaluation results show statistical significance and that ms-SDRGN outperforms three state-of-the-art DRL baselines in SAG-MCS. Compared with the best-performing baseline, ms-SDRGN improves 29.0% reward and 3.8% CFE score. We also investigate the scalability and robustness of ms-SDRGN towards DRL environments with diverse observation scales or demanding communication conditions.     

KEADA: Identifying Key Classes in Software Systems Using Dynamic Analysis and Entropy-Based Metrics

Software maintenance is indispensable in the software development process. Developers need to spend a lot of time and energy to understand the software when maintaining the software, which increases the difficulty of software maintenance. It is a feasible method to understand the software through the key classes of the software. Identifying the key classes of the software can help developers understand the software more quickly. Existing techniques on key class identification mainly use static analysis techniques to extract software structure information. Such structure information may contain redundant relationships that may not exist when the software runs and ignores the actual interaction times between classes. In this paper, we propose an approach based on dynamic analysis and entropy-based metrics to identify key classes in the Java GUI software system, called KEADA (identifying KEy clAsses based on Dynamic Analysis and entropy-based metrics). First, KEADA extracts software structure information by recording the calling relationship between classes during the software running process; such structure information takes into account the actual interaction of classes. Second, KEADA represents the structure information as a weighted directed network and further calculates the importance of each node using an entropy-based metric OSE (One-order Structural Entropy). Third, KEADA ranks classes in descending order according to their OSE values and selects a small number of classes as the key class candidates. In order to verify the effectiveness of our approach, we conducted experiments on three Java GUI software systems and compared them with seven state-of-the-art approaches. We used the Friedman test to evaluate all approaches, and the results demonstrate that our approach performs best in all software systems.     

Linker engineering in metal–organic frameworks for dark photocatalysis

Dark reactions featuring continuous activity under light off conditions play a critical role in natural photosynthesis. However, most artificial photocatalysts are inactive upon the removal of the light source, and the artificial photocatalysts with dark photocatalysis abilities have been rarely explored. Herein, we report a Ti-based metal–organic framework (MOF), MIL-125, exhibiting the capability of dark photocatalytic hydrogen production. Remarkably, the introduction of different functional groups onto the linkers enables distinctly different activities of the resulting MOFs (MIL-125-X, X = NH₂, NO₂, Br). Dynamic and thermodynamic investigations indicate that the production and lifetime of the Ti³⁺ intermediate are the key factors, due to the electron-donating/-withdrawing effect of the functional groups. As far as we know, this is the first report on dark photocatalysis over MOFs, providing new insights into the storage of irradiation energy and demonstrating their great potential in dark photocatalysis due to the great MOF diversity.

A Ti-based MOF with long-lived Ti³⁺ can achieve dark photocatalysis. The different groups on the organic linker modulate electron storage ability and the lifetime of Ti³⁺, significantly regulating dark photocatalytic activity in H₂ production.     

Application of Two Novel Acoustic Emission Parameters on Identifying the Instability of Granite

The issue of monitoring and early warning of rock instability has received increasing critical attention in the study of rock engineering. To investigate the damage evolution process of granite under triaxial compression tests, acoustic emission (AE) tests were performed simultaneously. This study firstly introduced two novel parameters, i.e., the coefficient of variation (CoV) of the information entropy and correlation dimension of the amplitude data from the AE tests, to identify the precursor of the failure of granite. Then the relationship between the changes in these parameters and the stress-time curve was compared and analyzed. The results of this study show that: (1) There is a strong correlation between the CoV of the information entropy and the failure process of granite. The granite failed when the CoV curve raised to a plateau, which could be used as an indicator of rock instability. (2) The fluctuation of the correlation dimension indicates the different stages during the loading process, i.e., the initial compaction stage, the linear elastic stage, the yield stage, and the failure stage. Each stage contains a descending and a rising process in the correlation dimension curve, and the exhibited starting point or the bottom point at the correlation dimension curve could be selected as the indicator point for the rock instability. (3) The combined analysis of the Information entropy and Correlation dimension can improve the accuracy of rock instability prediction. This study provides new insights into the prediction of rock instability, which has theoretical implications for the stability of subsurface engineering rock masses.