Defense decision-making method based on incomplete information stochastic game and Q-learning

Most of the existing stochastic games are based on the assumption of complete information,which are not consistent with the fact of network attack and defense.Aiming at this problem,the uncertainty of the attacker’s revenue was transformed to the uncertainty of the attacker type,and then a stochastic game model with incomplete information was constructed.The probability of network state transition is difficult to determine,which makes it impossible to determine the parameter needed to solve the equilibrium.Aiming at this problem,the Q-learning was introduced into stochastic game,which allowed defender to get the relevant parameter by learning in network attack and defense and to solve Bayesian Nash equilibrium.Based on the above,a defense decision algorithm that could learn online was designed.The simulation experiment proves the effectiveness of the proposed method.     

一种基于Q学习的单路口交通信号控制方法

利用Watkins(1989)提出的Q学习算法,基于不同相位的动态最小时间需求,并以相位通行需求时间为研究对象,以动态相位时间差为状态转换控制目标,建立了单路口交通信号控制的一种动态智能控制方法.计算机模拟仿真表明这种控制方法的好的控制效果.     

认知无线电网络中提高传输层端到端吞吐率的跨层参数配置

在认知无线电网络中, 传输层端到端(TCP)吞吐率是衡量网络性能的重要指标. 前期相关研究大都具有以下两方面缺点: 第一, 大部分研究只考虑了协议底层参数来优化物理链路性能, 对传输层性能有所忽略; 第二, 目前的研究大都基于马尔可夫决策过程建模, 这需要网络具有完全知识, 使得这类模型的应用受到很大限制. 针对以上问题, 本文提出一种新的算法: 网络中每个节点通过联合配置物理层调制方式、发射功率、 链路层信道接入和TCP拥塞控制因子来找到传输层端到端近似最优吞吐率. 由于无线设备对环境感知存在误差, 本文将网络模型建模为部分可观测马尔可夫决策过程, 并将其转换成信念状态马尔可夫决策过程, 采用Q值迭代找到近似最优策略. 仿真分析表明, 提出的算法能在动态无线环境下以一定的误差限收敛于最优策略, 能在功率受限条件下, 有效提高传输层端到端吞吐率.     

基于反馈机制的自愈算法

提出了一种网络自愈算法,当网络中的节点发生故障或链路出现拥塞时,该算法利用Q学习的反馈机制、多QoS约束的评价函数和基于Boltzmann-Gibbs分布的路径选择策略,自适应地选择恢复路径,降低了选择发生故障和拥塞路径的概率,从而实现了自愈。仿真结果表明,该算法在恢复率、区分业务能力和网络资源优化等方面,表现出了良好的性能。     

Femtocell双层网络中基于Q-learning的子信道分配方案

在Femtocell家庭基站(Femtocell Base Station, FBS)组成的异构网络中,为提升网络的频谱效率,FBS与Macrocell宏基站(Macrocell Base Station, MBS)一般要求是同频部署,然而同频部署会产生同信道干扰。为了实现FBS的大规模部署,降低网络同信道干扰影响变得尤为重要。该文提出一种基于Q-learning的子信道分配方案,既保证大量部署的FBS不会对MBS带来过高的跨层干扰,同时也降低了FBS之间的同层干扰。同时针对FBS稀疏部署和密集部署的场景,分别进行了算法的仿真验证,其仿真结果表明该算法降低了干扰,验证了理论的正确性。     

车联网中基于Q学习的地理位置路由协议

由于道路拓扑结构的限制以及车辆节点的快速变化,车联网路由协议正面临着很多挑战,例如道路的低连通性、较大的延时以及高开销等。为解决此类问题,提出了一种基于Q学习的地理位置路由协议。该协议将地理区域划分成大小一致的正方形,称为网格。在给定目的地的情况下,根据历史交通流信息计算出车辆从当前网格向不同方向的邻居网格移动的Q值,每辆车存储Q值表,通过查询Q值表选择最优下一跳网格。在选定的下一跳网格中,选择距离目的地最近的车辆,当最优下一跳网格中没有邻居车辆时,选择次优下一跳网格中的车辆。仿真结果表明,与其他基于地理位置的路由协议相比,所提协议能够提高分组投递率,降低传输延时并减少通信跳数。     

Research on Q-learning based rate control approach for HTTP adaptive streaming

HTTP adaptive streaming (HAS) has become the standard for adaptive video streaming service.In changing network environments,current hardcoded-based rate adaptation algorithm was less flexible,and it is insufficient to consider the quality of experience (QoE).To optimize the QoE of users,a rate control approach based on Q-learning strategy was proposed.the client environments of HTTP adaptive video streaming was modeled and the state transition rule was defined.Three parameters related to QoE were quantified and a novel reward function was constructed.The experiments were employed by the Q-learning rate control approach in two typical HAS algorithms.The experiments show the rate control approach can enhance the stability of rate switching in HAS clients.     

Proficient link state routing in mobile ad hoc network-based deep Q-learning network optimized with chaotic bat swarm optimization algorithm

Mobile ad-hoc network (MANET) is a category of ad-hoc network that can be reconfigurable its network. MANETS are self-organized networks, that can use the wireless links to connect various networks via mobile nodes: but it consumes more energy and it also has routing problems. This is the major drawback of being connected with the MANET technology. Therefore, this study proposes a new protocol as deep Q-learning network optimized with chaotic bat swarm optimization algorithm (CBS)-based optimized link state routing (OLSR) (CBS-OLSR) for MANET. This protocol reduces MANET energy usage and adopts OLSR multi-point relay (MPR) technology. MANET's OLSR and the CBS algorithm utilize a similar method to locate the best optimum path from source to destination node. By embedding the new improved deep Q-learning and OLSR algorithms, both are used for optimizing the MPR sets selection, it can efficiently diminish the energy consumption in the network topology, but automatically increase the lifespan of the network. It also enhances the package delivery ratio and decreases end-to-end delay. The experimental outcomes prove that the proposed protocol is reliable and proficient that is appropriate for numerous MANET applications.     

Q-Learning Algorithms with Random Truncation Bounds and Applications to Effective Parallel Computing

Motivated by an important problem of load balancing in parallel computing, this paper examines a modified algorithm to enhance Q-learning methods, especially in asynchronous recursive procedures for self-adaptive load distribution at run-time. Unlike the existing projection method that utilizes a fixed region, our algorithm employs a sequence of growing truncation bounds to ensure the boundedness of the iterates. Convergence and rates of convergence of the proposed algorithm are established. This class of algorithms has broad applications in signal processing, learning, financial engineering, and other related fields. G. Yin’s research was supported in part by the National Science Foundation under Grants DMS-0603287 and DMS-0624849 and in part by the National Security Agency under Grant MSPF-068-029. C.Z. Xu’s research was supported in part by the National Science Foundation under Grants CCF-0611750, DMS-0624849, CNS-0702488, and CRI-0708232. L.Y. Wang’s research was supported in part by the National Science Foundation under Grants ECS-0329597 and DMS-0624849 and by the Michigan Economic Development Council.     

强化学习算法在雷达智能抗干扰中的应用

雷达在工作过程中所应对的干扰场景复杂且多变,所具有的反干扰措施难以穷举。人工设计的反干扰流程与抑制策略在面对这些对抗场景时,由于受限于专家的经验知识,其反干扰性能难以保证。对此,文中从雷达抗干扰的应用需求出发,通过引入强化学习方法,提出一种基于强化学习模型的智能抗干扰方法。分别利用Q学习与Sarsa两种典型的强化学习算法对反干扰模型中的值函数进行了计算并迭代,使得反干扰策略具备了自主更新与优化功能。仿真结果表明,强化学习算法在训练过程中能够收敛并实现反干扰策略的优化。相比于传统的反干扰设计手段,雷达反干扰的智能化程度得到了有效提升。