deep reinforcement learning
6875312
Asynchronous Methods for Deep Reinforcement Learning
We propose a conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network controllers. We present asynchronous variants of four standard reinforcement learning algorithms and show that parallel actor learners have a stabilizing effect on training allowing all four methods to successfully train neural network controllers. The best performing method, an asynchronous variant of actor critic, surpasses the current state of the art on the Atari domain while training for half the time on a single multi core CPU instead of a GPU. Furthermore, we show that asynchronous actor critic succeeds on a wide variety of continuous motor control problems as well as on a new task of navigating random 3D mazes using a visual input.
ICML
Volodymyr Mnih Adria Puigdomenech Badia Mehdi Mirza Alex Graves Timothy P Lillicrap Tim Harley David Silver Koray Kavukcuoglu
205242740
Human level control through deep reinforcement learning
The theory of reinforcement learning provides a normative account, deeply rooted in psychological and neuroscientific perspectives on animal behaviour, of how agents may optimize their control of an environment. To use reinforcement learning successfully in situations approaching real world complexity, however, agents are confronted with a difficult task: they must derive efficient representations of the environment from high dimensional sensory inputs, and use these to generalize past experience to new situations. Remarkably, humans and other animals seem to solve this problem through a harmonious combination of reinforcement learning and hierarchical sensory processing systems, the former evidenced by a wealth of neural data revealing notable parallels between the phasic signals emitted by dopaminergic neurons and temporal difference reinforcement learning algorithms. While reinforcement learning agents have achieved some successes in a variety of domains, their applicability has previously been limited to domains in which useful features can be handcrafted, or to domains with fully observed, low dimensional state spaces. Here we use recent advances in training deep neural networks to develop a novel artificial agent, termed a deep Q network, that can learn successful policies directly from high dimensional sensory inputs using end to end reinforcement learning. We tested this agent on the challenging domain of classic Atari 2600 games. We demonstrate that the deep Q network agent, receiving only the pixels and the game score as inputs, was able to surpass the performance of all previous algorithms and achieve a level comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters. This work bridges the divide between high dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks.
Nature
Volodymyr Mnih Koray Kavukcuoglu David Silver Andrei A Rusu Joel Veness Marc G Bellemare Alex Graves Martin A Riedmiller Andreas Fidjeland Georg Ostrovski Stig Petersen Charlie Beattie Amir Sadik Ioannis Antonoglou Helen King Dharshan Kumaran Daan Wierstra Shane Legg Demis Hassabis
16326763
Continuous control with deep reinforcement learning
We adapt the ideas underlying the success of Deep Q Learning to the continuous action domain. We present an actor critic, model free algorithm based on the deterministic policy gradient that can operate over continuous action spaces. Using the same learning algorithm, network architecture and hyper parameters, our algorithm robustly solves more than 20 simulated physics tasks, including classic problems such as cartpole swing up, dexterous manipulation, legged locomotion and car driving. Our algorithm is able to find policies whose performance is competitive with those found by a planning algorithm with full access to the dynamics of the domain and its derivatives. We further demonstrate that for many of the tasks the algorithm can learn policies end to end: directly from raw pixel inputs.
ICLR
Timothy P Lillicrap Jonathan J Hunt Alexander Pritzel Nicolas Manfred Otto Heess Tom Erez Yuval Tassa David Silver Daan Wierstra
15238391
Playing Atari with Deep Reinforcement Learning
We present the first deep learning model to successfully learn control policies directly from high dimensional sensory input using reinforcement learning. The model is a convolutional neural network, trained with a variant of Q learning, whose input is raw pixels and whose output is a value function estimating future rewards. We apply our method to seven Atari 2600 games from the Arcade Learning Environment, with no adjustment of the architecture or learning algorithm. We find that it outperforms all previous approaches on six of the games and surpasses a human expert on three of them.
ArXiv
Volodymyr Mnih Koray Kavukcuoglu David Silver Alex Graves Ioannis Antonoglou Daan Wierstra Martin A Riedmiller
28202810
Soft Actor Critic: Off Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
Model free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning. Both of these challenges severely limit the applicability of such methods to complex, real world domains. In this paper, we propose soft actor critic, an off policy actor critic deep RL algorithm based on the maximum entropy reinforcement learning framework. In this framework, the actor aims to maximize expected reward while also maximizing entropy. That is, to succeed at the task while acting as randomly as possible. Prior deep RL methods based on this framework have been formulated as Q learning methods. By combining off policy updates with a stable stochastic actor critic formulation, our method achieves state of the art performance on a range of continuous control benchmark tasks, outperforming prior on policy and off policy methods. Furthermore, we demonstrate that, in contrast to other off policy algorithms, our approach is very stable, achieving very similar performance across different random seeds.
ICML
Tuomas Haarnoja Aurick Zhou P Abbeel Sergey Levine
6208256
Deep Reinforcement Learning with Double Q Learning
The popular Q learning algorithm is known to overestimate action values under certain conditions. It was not previously known whether, in practice, such overestimations are common, whether they harm performance, and whether they can generally be prevented. In this paper, we answer all these questions affirmatively. In particular, we first show that the recent DQN algorithm, which combines Q learning with a deep neural network, suffers from substantial overestimations in some games in the Atari 2600 domain. We then show that the idea behind the Double Q learning algorithm, which was introduced in a tabular setting, can be generalized to work with large scale function approximation. We propose a specific adaptation to the DQN algorithm and show that the resulting algorithm not only reduces the observed overestimations, as hypothesized, but that this also leads to much better performance on several games.
AAAI
H V Hasselt Arthur Guez David Silver
5389801
Dueling Network Architectures for Deep Reinforcement Learning
In recent years there have been many successes of using deep representations in reinforcement learning. Still, many of these applications use conventional architectures, such as convolutional networks, LSTMs, or auto encoders. In this paper, we present a new neural network architecture for model free reinforcement learning. Our dueling network represents two separate estimators: one for the state value function and one for the state dependent action advantage function. The main benefit of this factoring is to generalize learning across actions without imposing any change to the underlying reinforcement learning algorithm. Our results show that this architecture leads to better policy evaluation in the presence of many similar valued actions. Moreover, the dueling architecture enables our RL agent to outperform the state of the art on the Atari 2600 domain.
ICML
Ziyun Wang Tom Schaul Matteo Hessel H V Hasselt Marc Lanctot Nando de Freitas
54457299
Off Policy Deep Reinforcement Learning without Exploration
Many practical applications of reinforcement learning constrain agents to learn from a fixed batch of data which has already been gathered, without offering further possibility for data collection. In this paper, we demonstrate that due to errors introduced by extrapolation, standard off policy deep reinforcement learning algorithms, such as DQN and DDPG, are incapable of learning with data uncorrelated to the distribution under the current policy, making them ineffective for this fixed batch setting. We introduce a novel class of off policy algorithms, batch constrained reinforcement learning, which restricts the action space in order to force the agent towards behaving close to on policy with respect to a subset of the given data. We present the first continuous control deep reinforcement learning algorithm which can learn effectively from arbitrary, fixed batch data, and empirically demonstrate the quality of its behavior in several tasks.
ICML
Scott Fujimoto David Meger Doina Precup
211258956
Reconfigurable Intelligent Surface Assisted Multiuser MISO Systems Exploiting Deep Reinforcement Learning
Recently, the reconfigurable intelligent surface (RIS) benefited from the breakthrough on the fabrication of programmable meta material, has been speculated as one of the key enabling technologies for the future six generation (6G) wireless communication systems scaled up beyond massive multiple input multiple output (Massive MIMO) technology to achieve smart radio environments. Employed as reflecting arrays, RIS is able to assist MIMO transmissions without the need of radio frequency chains resulting in considerable reduction in power consumption. In this paper, we investigate the joint design of transmit beamforming matrix at the base station and the phase shift matrix at the RIS, by leveraging recent advances in deep reinforcement learning (DRL) We first develop a DRL based algorithm, in which the joint design is obtained through trial and error interactions with the environment by observing predefined rewards, in the context of continuous state and action. Unlike the most reported works utilizing the alternating optimization techniques to alternatively obtain the transmit beamforming and phase shifts, the proposed DRL based algorithm obtains the joint design simultaneously as the output of the DRL neural network. Simulation results show that the proposed algorithm is not only able to learn from the environment and gradually improve its behavior, but also obtains the comparable performance compared with two state of the art benchmarks. It is also observed that, appropriate neural network parameter settings will improve significantly the performance and convergence rate of the proposed algorithm.
IEEE Journal on Selected Areas in Communications
Chongwen Huang Ronghong Mo Chau Yuen
57189444
Deep Reinforcement Learning for Multiagent Systems: A Review of Challenges, Solutions, and Applications
Reinforcement learning (RL) algorithms have been around for decades and employed to solve various sequential decision making problems. These algorithms, however, have faced great challenges when dealing with high dimensional environments. The recent development of deep learning has enabled RL methods to drive optimal policies for sophisticated and capable agents, which can perform efficiently in these challenging environments. This article addresses an important aspect of deep RL related to situations that require multiple agents to communicate and cooperate to solve complex tasks. A survey of different approaches to problems related to multiagent deep RL (MADRL) is presented, including nonstationarity, partial observability, continuous state and action spaces, multiagent training schemes, and multiagent transfer learning. The merits and demerits of the reviewed methods will be analyzed and discussed with their corresponding applications explored. It is envisaged that this review provides insights about various MADRL methods and can lead to the future development of more robust and highly useful multiagent learning methods for solving real world problems.
IEEE Transactions on Cybernetics
Thanh Nguyen Ngoc Duy Nguyen Saeid Nahavandi