Deep transfer reinforcement learning for resource allocation in hybrid multiple access systems

This paper proposes a resource allocation scheme for hybrid multiple access involving both orthogonal multiple access and non-orthogonal multiple access (NOMA) techniques. The proposed resource allocation scheme employs multi-agent deep reinforcement learning (MA-DRL) to maximize the sum-rate for all users. More specifically, the MA-DRL-based scheme jointly allocates subcarrier and power resources for users by utilizing deep Q networks and multi-agent deep deterministic policy gradient networks. Meanwhile, an adaptive learning determiner mechanism is introduced into our allocation scheme to achieve better sum-rate performance. However, the above deep reinforcement learning adopted by our scheme cannot optimize parameters quickly in the new communication model. In order to better adapt to the new environment and make the resource allocation strategy more robust, we propose a transfer learning scheme based on deep reinforcement learning (T-DRL). The T-DRL-based scheme allows us to transfer the subcarrier allocation network and the power allocation network collectively or independently. Simulation results show that the proposed MA-DRL-based resource allocation scheme can achieve better sum-rate performance. Furthermore, the T-DRL-based scheme can effectively improve the convergence speed of the deep resource allocation network.     

A multi-timescale resource allocation algorithm based on self-learning for distributed fog radio access networks

Faced with limited network resources, diverse service requirements and complex network structures, how to efficiently allocate resources and improve network performances is an important issue that needs to be addressed in 5G or future 6G networks. In this paper, we propose a multi-timescale collaboration resource allocation algorithm for distributed fog radio access networks (F-RANs) based on self-learning. This algorithm uses a distributed computing architecture for parallel optimization and each optimization model includes large time-scale resource allocation and small time-scale resource scheduling. First, we establish a large time-scale resource allocation model based on long-term average information such as historical bandwidth requirements for each network slice in F-RAN by long short-term memory network (LSTM) to obtain its next period required bandwidth. Then, based on the allocated bandwidth, we establish a resource scheduling model based on short-term instantaneous information such as channel gain by reinforcement learning (RL) which can interact with the environment to realize adaptive resource scheduling. And the cumulative effects of small time-scale resource scheduling will trigger another round large time-scale resource reallocation. Thus, they constitute a self-learning resource allocation closed loop optimization. Simulation results show that compared with other algorithms, the proposed algorithm can significantly improve resource utilization.     

Improved Q-Learning Algorithm Based on Approximate State Matching in Agricultural Plant Protection Environment

An Unmanned Aerial Vehicle (UAV) can greatly reduce manpower in the agricultural plant protection such as watering, sowing, and pesticide spraying. It is essential to develop a Decision-making Support System (DSS) for UAVs to help them choose the correct action in states according to the policy. In an unknown environment, the method of formulating rules for UAVs to help them choose actions is not applicable, and it is a feasible solution to obtain the optimal policy through reinforcement learning. However, experiments show that the existing reinforcement learning algorithms cannot get the optimal policy for a UAV in the agricultural plant protection environment. In this work we propose an improved Q-learning algorithm based on similar state matching, and we prove theoretically that there has a greater probability for UAV choosing the optimal action according to the policy learned by the algorithm we proposed than the classic Q-learning algorithm in the agricultural plant protection environment. This proposed algorithm is implemented and tested on datasets that are evenly distributed based on real UAV parameters and real farm information. The performance evaluation of the algorithm is discussed in detail. Experimental results show that the algorithm we proposed can efficiently learn the optimal policy for UAVs in the agricultural plant protection environment.     

Profit maximization in cache-aided intelligent computing networks

This paper focuses on the profit maximization problem in a reconfigurable intelligent surfaces (RIS) aided computing network, where multiple heterogeneous users offload their computational tasks to one computational access point (CAP) for seeking computing acceleration at the cost of profit. In particular, the CAP can also pre-store a part of the computing task to speed up computing, and the system has limited communication and computing resources, where heterogeneous users have different offloading requirements and the CAP can dynamically allocate the system resources to meet the requirements of users to earn profits. To maximize the system profit, we devise the system by proposing a resource allocation scheme which employs a genetic algorithm (GA), based on statistical channel state information (CSI) of wireless links. The proposed algorithm maximizes the long-term profit of the system by optimizing resource allocation among users. Finally, simulation results are provided to verify the proposed scheme. The results show that our proposed resource allocation scheme outperforms the conventional ones.     

Deep reinforcement learning based IRS-assisted mobile edge computing under physical-layer security

In this paper, we investigate an intelligent reflecting surface (IRS)-assisted mobile edge computing (MEC) network under physical-layer security, where users can partially offload confidential and compute-intensive tasks to a computing access point (CAP) with the help of the IRS. We consider an eavesdropping environment, where an eavesdropper steals information from the communication. For the considered MEC network, we firstly design a secure data transmission rate to ensure physical-layer security. Moreover, we formulate the optimization target as minimizing the system cost linearized by the latency and energy consumption (ENCP). In further, we employ a deep deterministic policy gradient (DDPG) to optimize the system performance by allocating the offloading ratio and wireless bandwidth and computational capability to users. Finally, considering the impacts from different resources, based on DDPG, seeing our optimization strategy as one criterion, we designed other criteria with different resource allocation schemes. And some simulation results are given to demonstrate that our proposed criterion outperforms other criteria.     

Quality of Service based resource allocation in D2D enabled 5G-CNs with network slicing

With the rapid new advancements in technology, there is an enormous increase in devices and their versatile need for services. Fifth-generation (5G) cellular networks (5G-CNs) with network slicing (NS) have emerged as a necessity for future mobile communication. The available network is partitioned logically into multiple virtual networks to provide an enormous range of users’ specific services. Efficient resource allocation methods are critical to delivering the customers with their required Quality of Service (QoS) priorities. In this work, we have investigated a QoS based resource allocation (RA) scheme considering two types of 5G slices with different service requirements; (1) enhanced Mobile Broadband (eMBB) slice that requires a very high data rate and (2) massive Machine Type Communication (mMTC) slice that requires extremely low latency. We investigated the device-to-device (D2D) enabled 5G-CN model with NS to assign resources to users based on their QoS needs while considering the cellular and D2D user’s data rate requirements. We have proposed a Distributed Algorithm (DA) with edge computation to solve the optimization problem, which is novel as edge routers will solve the problem locally using the augmented Lagrange method. They then send this information to the central server to find the global optimum solution utilizing a consensus algorithm. Simulation analysis proves that this scheme is efficient as it assigns resources based on their QoS requirements. This scheme is excellent in reducing the central load and computational time.     

Power control based on the Stackelberg game in two-tier femtocell networks

In this paper, the resource allocation strategy is investigated for a spectrum sharing two-tier femtocell networks, in which a central macrocell is underlaid with distributed femtocells. The spectral radius is introduced to address the conditions that any feasible set of users’ signal-to-interference-plus-noise ratio requirements should satisfy in femtocell networks. To develop power allocation scheme with the derived conditions, a Stackelberg game is formulated, which aims at the utility maximization both of the macrocell user and femtocell users. The distributed power control algorithm is given to reduce the cross-tier interference between the macrocell and femtocell with same channel. At last, admission control algorithm is proposed, aiming to exploit the network resource effectively. Numerical results show that the proposed resource allocation schemes are effective in reducing power consumption and more suitable in the densely deployed scenario of the femtocell networks. Meanwhile, it also presents that the distributed power allocation scheme combined with admission control can protect the performance of all active femtocell users in a robust manner.     

Macrocell–femtocells resource allocation with hybrid access motivational model

Femtocell technology has emerged as an efficient cost-effective solution not only to solve the indoor coverage problem but also to cope with the growing demand requirements. This paper investigates two major design concerns in two tier networks: resource allocation and femtocell access. Base station selection together with dual bandwidth and power allocation among the two tiers is investigated under shared spectrum usage. To achieve fair and efficient resource optimization, our model assumes that the hybrid access mode is applied in the femtocells. The hybrid access mode is beneficial for system performance as (1) it lessens interference caused by nearby public users, (2) it allows public users to connect to near femtocells and get better Quality of Service (QoS) and (3) it increases system capacity as it allows the macrocell to serve more users. However, femtocells’ owners can behave selfishly by denying public access to avoid any performance reduction in subscribers’ transmissions. Such a problem needs a motivation scheme to assure the cooperation of femtocells’ owners. In this paper, we propose a game-theoretical hybrid access motivational model. The proposed model encourages femtocells’ owners to share resources with public users, thus, more efficient resource allocation can be obtained. We optimize the resource allocation by means of the Genetic Algorithm (GA). The objective of the formulated optimization problem is the maximization of network throughput that is calculated by means of Shannon’s Capacity Law. Simulations are conducted where a modified version of the Weighted Water Filling (WWF) algorithm is used as a benchmark. Our proposed model, compared to WWF, achieves more efficient resource allocation in terms of system throughput and resources utilization.     

Chance-constrained resource allocation for cognitive wireless-powered backscatter communication networks

With the rapid development of the Internet of Things (IoT) and the increasing number of wireless nodes, the problems of scare spectrum and energy supply of nodes have become main issues. To achieve green IoT techniques and resolve the challenge of wireless power supply, wireless-powered backscatter communication as a promising transmission paradigm has been concerned by many scholars. In wireless-powered backscatter communication networks, the passive backscatter nodes can harvest the ambient radio frequency signals for the devices’ wireless charging and also reflect some information signals to the information receiver in a low-power-consumption way. To balance the relationship between the amount of energy harvesting and the amount of information rate, resource allocation is a key technique in wireless-powered backscatter communication networks. However, most of the current resource allocation algorithms assume available perfect channel state information and limited spectrum resource, it is impractical for actual backscatter systems due to the impact of channel delays, the nonlinearity of hardware circuits and quantization errors that may increase the possibility of outage probability. To this end, we investigate a robust resource allocation problem to improve system robustness and spectrum efficiency in a cognitive wireless-powered backscatter communication network, where secondary transmitters can work at the backscattering transmission mode and the harvest-then-transmit mode by a time division multiple access manner. The total throughput of the secondary users is maximized by jointly optimizing the transmission time, the transmit power, and the reflection coefficients of secondary transmitters under the constraints on the throughput outage probability of the users. To tackle the non-convex problem, we design a robust resource allocation algorithm to obtain the optimal solution by using the proper variable substitution method and Lagrange dual theory. Simulation results verify the effectiveness of the proposed algorithm in terms of lower outage probabilities.     

Joint user grouping and power allocation in VLC-NOMA system

With the energy consumption of wireless networks increasing, visible light communication (VLC) has been regarded as a promising technology to realize energy conservation. Due to the massive terminals access and increased traffic demand, the implementation of non-orthogonal multiple access (NOMA) technology in VLC networks has become an inevitable trend. In this paper, we aim to maximize the energy efficiency in VLC-NOMA networks. Assuming perfect knowledge of the channel state information of user equipment, the energy efficiency maximization problem is formulated as a mixed integer nonlinear programming problem. To solve this problem, the joint user grouping and power allocation (JUGPA) is proposed including user grouping and power allocation. In user grouping phase, we utilize the average of channel gain among all user equipment and propose a dynamic user grouping algorithm with low complexity. The proposed scheme exploits the channel gain differences among users and divides them into multiple groups. In power allocation phase, we proposed a power allocation algorithm for maximizing the energy efficiency for a given NOMA group. Thanks to the objective function is fraction form and non-convex, we firstly transform it to difference form and convex function. Then, we derive the closed-form optimal power allocation expression that maximizes the energy efficiency by Dinkelbach method and Lagrange dual decomposition method. Simulation results show that the JUGPA can effectively improve energy efficiency of the VLC-NOMA networks.     

Optimal resource scheduling algorithm for cell boundaries users in heterogenous 5G networks

Deployment of small cells over the existing cellular network is an effective solution to improve the system coverage and throughput of fifth generation (5G) mobile communication networks. The arrival of the 5G mobile networks have demonstrated the importance of advanced scheduling techniques to manage the limited frequency spectrum available while achieving 5G transmission requirements. Cellular networks of the future necessitate the formulation of efficient resource allocation schemes that mitigate the interference between the different cells. In this research work, we formulate an optimization problem for heterogenous networks (HetNets) for resource allocation to maximize the system throughput among the cell center users (CCUs) and cell edge users (CEUs). We solve the optimization problem by effective utilization of the weight factors distribution for resource allocation. A novel Utility-based Resource Scheduling Algorithm (URSA) optimizes the resource sharing among the users with better delay budget of each application. The designed URSA ameliorates fairness along with reduced cross layer interference for real and non-real time applications. Performance of the URSA has been evaluated and compared most relevant state of art algorithms using the matlab based simulators. Furthermore, simulation results validate the superiority of the proposed scheduling scheme against conventional techniques in terms of throughput, fairness, and spectral efficiency.     

Fairness optimization in IRS-assisted MEC computational offloading

Nowadays, more and more multimedia services are supported by Mobile Edge Computing (MEC). However, the instability of the wireless environment brings a lot of uncertainty to the computational offloading. Additionally, intelligent reflecting surface (IRS) is considered as a potential technology to enhance Quality of Service (QoS). Therefore, in this paper, we establish a framework for IRS-assisted MEC computational offloading to solve this problem and take fairness optimization as a key point involving communication and computing resources. Minimize user consumption by optimizing bandwidth allocation, task offloading ratio, edge computing resources, transmission power and IRS phase shifts. Firstly, we decompose the problem into three aspects, such as bandwidth allocation, computing resource allocation, transmission power and IRS phase shifts. Then, an alternative optimization algorithm is proposed to find the optimum solution and its convergence is proved. Secondly, since the optimization problem on transmission power and IRS phase shifts is non-convex, we propose Riemann gradient descent (R-SGD) algorithm to solve it. Finally, numerical results show that our proposed algorithm performs better than other algorithms and achieves a superiority in the framework.     

Caching popularity-unknown contents for massive communication channels via deep Q-learning

In order to reduce the backhaul link pressure of wireless networks, edge caching technology has been regarded as a promising solution. However, with massive and dynamical communication connections, it is challenging to provide analytical caching solution to achieve the best performance, particularly when the requested contents are changing and their popularities are unknown. In this paper, we propose a deep Q-learning (DQN) method to address the issue of caching placement. Considering a content caching network containing multiple cooperating SBSs with unknown content popularity, we need to determine which content to cache and where to cache. Therefore, the learning network has to be designed for dual aims, one of which is to estimate the content popularities while the other is to assign contents to the proper channels. An elaborate DQN is proposed to make decisions to cache contents with limited storage space of base-station by considering channel conditions. Specifically, the content requests of users are first collected as one of the training samples of the learning network. Second, the channel state information for the massive links are estimated as the other training samples. Then, we train the network based on the proposed method thereby improving spectral efficiency of the entire system and reducing bit-error rate. Our major contribution is to contrive a caching strategy for enhanced performance in massive connection communications without knowing the content popularity. Numerical studies are performed to show that the proposed method acquires apparent performance gain over random caching in terms of average spectral efficiency and bit-error rate of the network.     

Resource allocation for LTE-based cognitive radio network with queue stability and interference constraints

We consider the problem of interference management and resource allocation in a cognitive radio network (CRNs) where the licensed spectrum holders (primary users) share their spare capacity with the non-licensed spectrum holders (secondary users). Under such shared spectrum usage the transmissions of the secondary users should have a minimal impact on the quality of service (QoS) and the operating conditions of the primary users. Therefore, it is important to distinguish the two types of users, and formulate the problem of resource allocation considering hard restrictions on the user-perceived QoS (such as packet end-to-end delay and loss) and physical-layer channel characteristics (such as noise and interference) of the primary users. To achieve this goal, we propose to assign the bandwidth and transmission power to minimize the total buffer occupancy in the system subject to capacity constraints, queue stability constraints, and interference requirements of the primary users. We apply this approach for resource allocation in a CRN built upon a Third Generation Partnership Project (3GPP) long-term evolution (LTE) standard platform. Performance of the algorithm is evaluated using simulations in OPNET environment. The algorithm shows consistent performance improvement when compared with other relevant resource allocation techniques.     

基于改进的布谷鸟算法在云计算资源的研究

如何进行更好地资源调度一直都是云计算研究的热点,本文在云计算资源算法中引入布谷鸟算法,针对布谷鸟算法中出现的收敛速度快,容易局部震荡等现象,本文首先引入高斯变异算子来处理每一个阶段中的鸟窝最佳位置的选择,然后通过自适应动态因子来调整不同阶段中的鸟窝位置的选择,使得改进后的算法收敛精度提高,通过适应度函数的平衡以及遗传算法中的三种操作,使得本文算法能够有效的提高云计算环境下的资源分配效率,降低了网络消耗。在Cloudsim平台仿真实验中,通过三个方面的比较,本文算法在性能上、资源调度效率和任务调度方面都有很大改进,有效提高了云计算系统的资源调度能力。     

基于膜计算和蚁群算法的融合算法在云计算资源调度中的研究

针对云计算下的资源调度的问题,提出将蚁群算法的个体与云计算中的可行性资源调度进行对应,首先对云计算资源调度进行描述,其次针对蚁群算法的路径选择引入了平衡因子,对信息素进行了局部研究和全局研究,将蚁群个体引入到膜计算中,通过膜内运算和膜间运算,提高了算法的局部和全局收敛的能力,最后在云计算资源分配中,引入匹配表概念,将云计算任务和资源进行匹配,融合后的算法提高了算法的整体性能.仿真实验说明在网络消耗,成本消耗,能量消耗上有了明显的降低,提高了资源分配效率。     

Multi-channel sensing and resource allocation in energy constrained cognitive radio networks

We consider a cognitive radio network in a multi-channel licensed environment. Secondary user transmits in a channel if the channel is sensed to be vacant. This results in a tradeoff between sensing time and transmission time. When secondary users are energy constrained, energy available for transmission is less if more energy is used in sensing. This gives rise to an energy tradeoff. For multiple primary channels, secondary users must decide appropriate sensing time and transmission power in each channel to maximize average aggregate-bit throughput in each frame duration while ensuring quality-of-service of primary users. Considering time and energy as limited resources, we formulate this problem as a resource allocation problem. Initially a single secondary user scenario is considered and solution is obtained using decomposition and alternating optimization techniques. Later we extend the analysis for the case of multiple secondary users. Simulation results are presented to study effect of channel occupancy, fading and energy availability on performance of proposed method.     

Distributed dynamic spectrum access through multi-agent deep recurrent Q-learning in cognitive radio network

This paper addresses the problem of distributed dynamic spectrum access in a cognitive radio (CR) environment utilizing deep recurrent reinforcement learning. Specifically, the network consists of multiple primary users (PU) transmitting intermittently in their respective channels, while the secondary users (SU) attempt to access the channels when PUs are not transmitting. The problem is challenging considering the decentralized nature of CR network where each SU attempts to access a vacant channel, without coordination with other SUs, which result in collision and throughput loss. To address this issue, a multi-agent environment is considered where each of the SUs perform independent reinforcement learning to learn the appropriate policy to transmit opportunistically so as to minimize collisions with other users. In this article, we propose two long short-term memory (LSTM) based deep recurrent Q-network (DRQN) architectures for exploiting the temporal correlation in the transmissions by various nodes in the network. Furthermore, we investigate the effect of the architecture on success rate with varying number of users in the network and partial channel observations. Simulation results are compared with other existing reinforcement learning based techniques to establish the superiority of the proposed method.     

A joint resource allocation method for hybrid NOMA MEC offloading

In this article, a joint resource allocation of power, time, and sub-channels that minimizes the total energy consumption of users for Hybrid NOMA MEC Offloading is proposed. By formulating and solving the joint optimization problem, first we propose a novel optimal Hybrid NOMA scheme referred to as Switched Hybrid NOMA (SH-NOMA) for power and time allocation. Subsequently, we address sub-channel allocation as a three-dimensional assignment problem, and propose the Total-Reward Exchange Stable (TES) algorithm to solve it. Analytically, we show that SH-NOMA is more energy efficient than the Hybrid NOMA scheme in the literature and that the TES algorithm converges to a solution with less energy consumption than the widely used two-sided exchange stable algorithm. Finally, via simulations we demonstrate that the proposed methods outperform the results in the literature.     

Resource optimization for UAV-assisted mobile edge computing system based on deep reinforcement learning

Computational efficiency is a direction worth considering in moving edge computing (MEC) systems. However, the computational efficiency of UAV-assisted MEC systems is rarely studied. In this paper, we maximize the computational efficiency of the MEC network by optimizing offloading decisions, UAV flight paths, and allocating users’ charging and offloading time reasonably. The method of deep reinforcement learning is used to optimize the resources of UAV-assisted MEC system in complex urban environment, and the user’s computation-intensive tasks are offloaded to the UAV-mounted MEC server, so that the overloaded tasks in the whole system can be alleviated. We study and design a framework algorithm that can quickly adapt to task offload decision making and resource allocation under changing wireless channel conditions in complex urban environments. The optimal offloading decisions from state space to action space is generated through deep reinforcement learning, and then the user’s own charging time and offloading time are rationally allocated to maximize the weighted sum computation rate. Finally, combined with the radio map to optimize the UAC trajectory to improve the overall weighted sum computation rate of the system. Simulation results show that the proposed DRL+TO framework algorithm can significantly improve the weighted sum computation rate of the whole MEC system and save time. It can be seen that the MEC system resource optimization scheme proposed in this paper is feasible and has better performance than other benchmark schemes.