reinforcement learining game
52824204
On Reinforcement Learning for Full length Game of StarCraft
StarCraft II poses a grand challenge for reinforcement learning. The main difficulties include huge state space, varying action space, long horizon, etc. In this paper, we investigate a set of techniques of reinforcement learning for the full length game of StarCraft II. We investigate a hierarchical approach, where the hierarchy involves two levels of abstraction. One is the macro actions extracted from expert's demonstration trajectories, which can reduce the action space in an order of magnitude yet remain effective. The other is a two layer hierarchical architecture, which is modular and easy to scale. We also investigate a curriculum transfer learning approach that trains the agent from the simplest opponent to harder ones. On a 64x64 map and using restrictive units, we train the agent on a single machine with 4 GPUs and 48 CPU threads. We achieve a winning rate of more than 99% against the difficulty level 1 built in AI. Through the curriculum transfer learning algorithm and a mixture of combat model, we can achieve over 93% winning rate against the most difficult noncheating built in AI (level 7) within days. We hope this study could shed some light on the future research of large scale reinforcement learning.
AAAI
Zhen-Jia Pang Ruo-Ze Liu Zhou-Yu Meng Y Zhang Yang Yu Tong Lu
58544242
A Multistage Game in Smart Grid Security: A Reinforcement Learning Solution
Existing smart grid security research investigates different attack techniques and cascading failures from the attackers' viewpoints, while the defenders' or the operators' protection strategies are somehow neglected. Game theoretic methods are applied for the attacker defender games in the smart grid security area. Yet, most of the existing works only use the one shot game and do not consider the dynamic process of the electric power grid. In this paper, we propose a new solution for a multistage game (also called a dynamic game) between the attacker and the defender based on reinforcement learning to identify the optimal attack sequences given certain objectives (e.g. transmission line outages or generation loss) Different from a one shot game, the attacker here learns a sequence of attack actions applying for the transmission lines and the defender protects a set of selected lines. After each time step, the cascading failure will be measured, and the line outage (and/or generation loss) will be used as the feedback for the attacker to generate the next action. The performance is evaluated on W&W 6 bus and IEEE 39 bus systems. A comparison between a multistage attack and a one shot attack is conducted to show the significance of the multistage attack. Furthermore, different protection strategies are evaluated in simulation, which shows that the proposed reinforcement learning solution can identify optimal attack sequences under several attack objectives. It also indicates that attacker's learned information helps the defender to enhance the security of the system.
IEEE Transactions on Neural Networks and Learning Systems
Zhen Ni Shuva Paul
196179566
Hierarchical Game Theoretic and Reinforcement Learning Framework for Computational Offloading in UAV Enabled Mobile Edge Computing Networks With Multiple Service Providers
We present a novel game theoretic (GT) and reinforcement learning (RL) framework for computational offloading in the mobile edge computing (MEC) network operated by multiple service providers (SPs) The network is formed by MEC servers installed at stationary base stations (BSs) and unmanned aerial vehicles (UAVs) deployed as quasi stationary BSs. Since computing powers of MEC servers are limited, the BSs in proximity can form coalitions with shared data processing resources to serve their users more efficiently. However, as BSs can be privately owned or controlled by different SPs, in any coalition, the BSs: 1) take only the actions that maximize their long term payoffs and 2) do not coordinate their actions with other BSs in the coalition. That is, inside each coalition, BSs act in an independent and self interested manner. Therefore, the interactions among BSs cannot be described by conventional coalitional games. Instead, the network operation is modeled by a two level hierarchical model. The upper level is a cooperative game that defines the process of coalition formation. The lower level comprises the set of noncooperative subgames to represent a self interested and independent behavior of BSs in coalitions. To enable each BS to select a coalition and decide on its action maximizing its long term payoff, we propose two algorithms that combine coalition formation with RL and prove that these algorithms converge to the states where the coalitional structure is strongly stable and the strategies of BSs are in the mixed strategy Nash equilibrium (NE)
IEEE Internet of Things Journal
Alia Asheralieva Dusit Tao Niyato
126175296
Approximate Nash Solutions for Multiplayer Mixed Zero Sum Game With Reinforcement Learning
Inspired by Nash game theory, a multiplayer mixed zero sum (MZS) nonlinear game considering both two situations [zero sum and nonzero sum (NZS) Nash games] is proposed in this paper. A synchronous reinforcement learning (RL) scheme based on the identifier critic structure is developed to learn the Nash equilibrium solution of the proposed MZS game. First, the MZS game formulation is presented, where the performance indexes for players 1 to <inline formula> <tex math notation="LaTeX"{N} /tex math>/inline formula> 1 and <inline formula> <tex math notation="LaTeX"{N} /tex math>/inline formula> NZS Nash game are presented, and another performance index for players <inline formula> <tex math notation="LaTeX"{N} /tex math>/inline formula> and <inline formula> <tex math notation="LaTeX"{N} /tex math>/inline formula> 1 zero sum game is presented, such that player <inline formula> <tex math notation="LaTeX"{N} /tex math>/inline formula> cooperates with players 1 to <inline formula> <tex math notation="LaTeX"{N} /tex math>/inline formula> 1, while competes with player <inline formula> <tex math notation="LaTeX"{N} /tex math>/inline formula> 1, which leads to a Nash equilibrium of all players. A single layer neural network (NN) is then used to approximate the unknown dynamics of the nonlinear game system. Finally, an RL scheme based on NNs is developed to learn the optimal performance indexes, which can be used to produce the optimal control policy of every player such that Nash equilibrium can be obtained. Thus, the widely used actor NN in RL literature is not needed. To this end, a recently proposed adaptive law is used to estimate the unknown identifier coefficient vectors, and an improved adaptive law with the error performance index is further developed to update the critic coefficient vectors. Both linear and nonlinear simulations are presented to demonstrate the existence of Nash equilibrium for MZS game and performance of the proposed algorithm.
IEEE Transactions on Systems, Man, and Cybernetics: Systems
Yongfeng Lv Xuemei Ren
46965189
Video game addiction, ADHD symptomatology, and video game reinforcement
ABSTRACT Background: Up to 23% of people who play video games report symptoms of addiction. Individuals with attention deficit hyperactivity disorder (ADHD) may be at increased risk for video game addiction, especially when playing games with more reinforcing properties. Objectives: The current study tested whether level of video game reinforcement (type of game) places individuals with greater ADHD symptom severity at higher risk for developing video game addiction. Methods: Adult video game players (N 2,801; Mean age 22.43, SD 4.70; 93.30% male; 82.80% Caucasian) completed an online survey. Hierarchical multiple linear regression analyses were used to test type of game, ADHD symptom severity, and the interaction between type of game and ADHD symptomatology as predictors of video game addiction severity, after controlling for age, gender, and weekly time spent playing video games. Results: ADHD symptom severity was positively associated with increased addiction severity (b .73 and .68, ps 0.001) Type of game played or preferred the most was not associated with addiction severity, ps .05. The relationship between ADHD symptom severity and addiction severity did not depend on the type of video game played or preferred most, ps .05. Conclusion: Gamers who have greater ADHD symptom severity may be at greater risk for developing symptoms of video game addiction and its negative consequences, regardless of type of video game played or preferred most. Individuals who report ADHD symptomatology and also identify as gamers may benefit from psychoeducation about the potential risk for problematic play.
The American journal of drug and alcohol abuse
Christine L Mathews Holly E R Morrell Jon E Molle
210694979
Wuji: Automatic Online Combat Game Testing Using Evolutionary Deep Reinforcement Learning
Game testing has been long recognized as a notoriously challenging task, which mainly relies on manual playing and scripting based testing in game industry. Even until recently, automated game testing still remains to be largely untouched niche. A key challenge is that game testing often requires to play the game as a sequential decision process. A bug may only be triggered until completing certain difficult intermediate tasks, which requires a certain level of intelligence. The recent success of deep reinforcement learning (DRL) sheds light on advancing automated game testing, without human competitive intelligent support. However, the existing DRLs mostly focus on winning the game rather than game testing. To bridge the gap, in this paper, we first perform an in depth analysis of 1349 real bugs from four real world commercial game products. Based on this, we propose four oracles to support automated game testing, and further propose Wuji, an on the fly game testing framework, which leverages evolutionary algorithms, DRL and multi objective optimization to perform automatic game testing. Wuji balances between winning the game and exploring the space of the game. Winning the game allows the agent to make progress in the game, while space exploration increases the possibility of discovering bugs. We conduct a large scale evaluation on a simple game and two popular commercial games. The results demonstrate the effectiveness of Wuji in exploring space and detecting bugs. Moreover, Wuji found 3 previously unknown bugs, which have been confirmed by the developers, in the commercial games.
2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE)
Yan Zheng Changjie Fan Xiaofei Xie Ting Su L Ma Jianye Hao Zhaopeng Meng Yang Liu Ruimin Shen Yingfeng Chen
26529085
A Secure Mobile Crowdsensing Game With Deep Reinforcement Learning
Mobile crowdsensing (MCS) is vulnerable to faked sensing attacks, as selfish smartphone users sometimes provide faked sensing results to the MCS server to save their sensing costs and avoid privacy leakage. In this paper, the interactions between an MCS server and a number of smartphone users are formulated as a Stackelberg game, in which the server as the leader first determines and broadcasts its payment policy for each sensing accuracy. Each user as a follower chooses the sensing effort and thus the sensing accuracy afterward to receive the payment based on the payment policy and the sensing accuracy estimated by the server. The Stackelberg equilibria of the secure MCS game are presented, disclosing conditions to motivate accurate sensing. Without knowing the smartphone sensing models in a dynamic version of the MCS game, an MCS system can apply deep Q network (DQN) which is a deep reinforcement learning technique combining reinforcement learning and deep learning techniques, to derive the optimal MCS policy against faked sensing attacks. The DQN based MCS system uses a deep convolutional neural network to accelerate the learning process with a high dimensional state space and action set, and thus improve the MCS performance against selfish users. Simulation results show that the proposed MCS system stimulates high quality sensing services and suppresses faked sensing attacks, compared with a Q learning based MCS system.
IEEE Transactions on Information Forensics and Security
Liang Xiao Yanda Li Guoan Han Huaiyu Dai H Vincent Poor
174803821
Grounding natural language commands to StarCraft II game states for narration guided reinforcement learning
While deep reinforcement learning techniques have led to agents that are successfully able to learn to perform a number of tasks that had been previously unlearnable, these techniques are still susceptible to the longstanding problem of reward sparsity. This is especially true for tasks such as training an agent to play StarCraft II, a real time strategy game where reward is only given at the end of a game which is usually very long. While this problem can be addressed through reward shaping, such approaches typically require a human expert with specialized knowledge. Inspired by the vision of enabling reward shaping through the more accessible paradigm of natural language narration, we investigate to what extent we can contextualize these narrations by grounding them to the goal specific states. We present a mutual embedding model using a multi input deep neural network that projects a sequence of natural language commands into the same high dimensional representation space as corresponding goal states. We show that using this model we can learn an embedding space with separable and distinct clusters that accurately maps natural language commands to corresponding game states We also discuss how this model can allow for the use of narrations as a robust form of reward shaping to improve RL performance and efficiency.
Defense Commercial Sensing
Nicholas R Waytowich Sean L Barton Vernon J Lawhern Ethan Stump Garrett Warnell
64720661
Hierarchical Reinforcement Learning With Monte Carlo Tree Search in Computer Fighting Game
Fighting games are complex environments where challenging action selection problems arise, mainly due to a diversity of opponents and possible actions. In this paper, we present the design and evaluation of a fighting player on top of the FightingICE platform that is used in the Fighting Game Artificial Intelligence (FTGAI) competition. Our proposal is based on hierarchical reinforcement learning (HRL) in combination with Monte Carlo tree search (MCTS) designed as options. By using the FightingICE framework, we evaluate our player against state of the art FTGAIs. We train our player against the current FTGAI champion (GigaThunder) The resulting learned policy is comparable with the champion in direct confront in regard to the number of victories, with the advantage of having less need for expert knowledge. We also evaluate the proposed player against the runners up and show that adaptation to the strategies of each opponent is necessary for building stronger fighting players.
IEEE Transactions on Games
Ivan J P Pinto Luciano R Coutinho
201146122
Deep Reinforcement Learning in Match 3 Game
An increasing number of algorithms in deep reinforcement learning area creates new challenges for environments, particularly, for their comprehensive analysis and searching application areas. The key purpose of this article is to provide an extensible environment for researches. We consider a Match 3 game, which has simple gameplay, but challenging game design for engaging players. The article provides metrics for evaluation of agents and corresponding baselines in different scenarios.
2019 IEEE Conference on Games (CoG)
Ildar Kamaldinov Ilya Makarov