pommerman
52305458
Pommerman: A Multi Agent Playground
We present Pommerman, a multi agent environment based on the classic console game Bomberman. Pommerman consists of a set of scenarios, each having at least four players and containing both cooperative and competitive aspects. We believe that success in Pommerman will require a diverse set of tools and methods, including planning, opponent/teammate modeling, game theory, and communication, and consequently can serve well as a multi agent benchmark. To date, we have already hosted one competition, and our next one will be featured in the NIPS 2018 competition track.
AIIDE Workshops
Cinjon Resnick Wes Eldridge David R Ha Denny Britz Jakob N Foerster Julian Togelius Kyunghyun Cho Joan Bruna
146120627
Skynet: A Top Deep RL Agent in the Inaugural Pommerman Team Competition
The Pommerman Team Environment is a recently proposed benchmark which involves a multi agent domain with challenges such as partial observability, decentralized execution (without communication) and very sparse and delayed rewards. The inaugural Pommerman Team Competition held at NeurIPS 2018 hosted 25 participants who submitted a team of 2 agents. Our submission nn_team_skynet955_skynet955 won 2nd place of the "learning agents' category. Our team is composed of 2 neural networks trained with state of the art deep reinforcement learning algorithms and makes use of concepts like reward shaping, curriculum learning, and an automatic reasoning module for action pruning. Here, we describe these elements and additionally we present a collection of open sourced agents that can be used for training and testing in the Pommerman environment. Code available at: this https URL
ArXiv
Chao Gao Pablo Hernandez-Leal Bilal Kartal Matthew E Taylor
119189183
Safer Deep RL with Shallow MCTS: A Case Study in Pommerman
Safe reinforcement learning has many variants and it is still an open research problem. Here, we focus on how to use action guidance by means of a non expert demonstrator to avoid catastrophic events in a domain with sparse, delayed, and deceptive rewards: the recently proposed multi agent benchmark of Pommerman. This domain is very challenging for reinforcement learning (RL) past work has shown that model free RL algorithms fail to achieve significant learning. In this paper, we shed light into the reasons behind this failure by exemplifying and analyzing the high rate of catastrophic events (i.e. suicides) that happen under random exploration in this domain. While model free random exploration is typically futile, we propose a new framework where even a non expert simulated demonstrator, e.g. planning algorithms such as Monte Carlo tree search with small number of rollouts, can be integrated to asynchronous distributed deep reinforcement learning methods. Compared to vanilla deep RL algorithms, our proposed methods both learn faster and converge to better policies on a two player mini version of the Pommerman game.
ArXiv
Bilal Kartal Pablo Hernandez-Leal Chao Gao Matthew E Taylor
204944675
Analysis of Statistical Forward Planning Methods in Pommerman
Pommerman is a complex multi player and partially observable game where agents try to be the last standing to win. This game poses very interesting challenges to AI, such as collaboration, learning and planning. In this paper, we compare two Statistical Forward Planning algorithms, Monte Carlo Tree Search (MCTS) and Rolling Horizon Evolutionary Algorithm (RHEA) in Pommerman. We provide insights on how the agents actually play the game, inspecting their behaviours to explain their performance. Results show that MCTS outperforms RHEA in several game settings, but leaving room for multiple avenues of future work: tuning these methods, improving opponent modelling, identifying trap moves and introducing of assumptions for partial observability settings.
AIIDE
Diego Perez Liebana Raluca D Gaina Olve Drageset Ercument Ilhan Martin Balla Simon M M Lucas
198967668
On Hard Exploration for Reinforcement Learning: a Case Study in Pommerman
How to best explore in domains with sparse, delayed, and deceptive rewards is an important open problem for reinforcement learning (RL) This paper considers one such domain, the recently proposed multi agent benchmark of Pommerman. This domain is very challenging for RL past work has shown that model free RL algorithms fail to achieve significant learning without artificially reducing the environment's complexity. In this paper, we illuminate reasons behind this failure by providing a thorough analysis on the hardness of random exploration in Pommerman. While model free random exploration is typically futile, we develop a model based automatic reasoning module that can be used for safer exploration by pruning actions that will surely lead the agent to death. We empirically demonstrate that this module can significantly improve learning.
AIIDE
Chao Gao Bilal Kartal Pablo Hernandez-Leal Matthew E Taylor
204852524
Real time tree search with pessimistic scenarios: Winning the NeurIPS 2018 Pommerman Competition
Autonomous agents need to make decisions in a sequential manner, under partially observable environment, and in consideration of how other agents behave. In critical situations, such decisions need to be made in real time for example to avoid collisions and recover to safe conditions. We propose a technique of tree search where a deterministic and pessimistic scenario is used after a specified depth. Because there is no branching with the deterministic scenario, the proposed technique allows us to take into account the events that can occur far ahead in the future. The effectiveness of the proposed technique is demonstrated in Pommerman, a multi agent environment used in a NeurIPS 2018 competition, where the agents that implement the proposed technique have won the first and third places.
ACML
Takayuki Osogami Toshihiro Takahashi
207863593
Accelerating Training in Pommerman with Imitation and Reinforcement Learning
The Pommerman simulation was recently developed to mimic the classic Japanese game Bomberman, and focuses on competitive gameplay in a multi agent setting. We focus on the 2$\times$2 team version of Pommerman, developed for a competition at NeurIPS 2018. Our methodology involves training an agent initially through imitation learning on a noisy expert policy, followed by a proximal policy optimization (PPO) reinforcement learning algorithm. The basic PPO approach is modified for stable transition from the imitation learning phase through reward shaping, action filters based on heuristics, and curriculum learning. The proposed methodology is able to beat heuristic and pure reinforcement learning baselines with a combined 100,000 training games, significantly faster than other non tree search methods in literature. We present results against multiple agents provided by the developers of the simulation, including some that we have enhanced. We include a sensitivity analysis over different parameters, and highlight undesirable effects of some strategies that initially appear promising. Since Pommerman is a complex multi agent competitive environment, the strategies developed here provide insights into several real world problems with characteristics such as partial observability, decentralized execution (without communication) and very sparse and delayed rewards.
ArXiv
Hardik Meisheri Omkar Shelke Richa Verma Harshad Khadilkar
196623480
Automated Gaming Pommerman: FFA
Our game Pommerman is based on the console game Bommerman. The game starts on an 11 by 11 platform. Pommerman is a multi agent environment and is made up of a set of different situations and contains four agents.
ArXiv
Navya Singh Anshul Dhull S BarathMohan Bhavish Pahwa Komal Sharma
52293304
A hybrid search agent in pommerman
In this paper, we explore the possibility of search based agents in games with resource intensive forward models. We implemented a player agent in the Pommerman framework and put it against the baseline agent to measure its performance. We implemented a heuristic agent and improved it by enabling depth limited tree search in specific gameplay moments. We also compared different node selection methods during depth limited tree search. Our result shows that depth limited tree search is still viable when presented with inefficient forward models and exploitation driven selection method is the most efficient in this specific domain.
FDG
Hongwei Zhou Yichen Gong Luvneesh Mugrai Ahmed Khalifa Andy Nealen Julian Togelius
56176640
Continual Match Based Training in Pommerman: Technical Report
Continual learning is the ability of agents to improve their capacities throughout multiple tasks continually. While recent works in the literature of continual learning mostly focused on developing either particular loss functions or specialized structures of neural network explaining the episodic memory or neural plasticity, we study continual learning from the perspective of the training mechanism. Specifically, we propose a COnitnual Match BAsed Training (COMBAT) framework for training a population of advantage actor critic (A2C) agents in Pommerman, a partially observable multi agent environment with no communication. Following the COMBAT framework, we trained an agent, namely, Navocado, that won the title of the top 1 learning agent in the NeurIPS 2018 Pommerman Competition. Two critical features of our agent are worth mentioning. Firstly, our agent did not learn from any demonstrations. Secondly, our agent is highly reproducible. As a technical report, we articulate the design of state space, action space, reward, and most importantly, the COMBAT framework for our Pommerman agent. We show in the experiments that Pommerman is a perfect environment for studying continual learning, and the agent can improve its performance by continually learning new skills without forgetting the old ones. Finally, the result in the Pommerman Competition verifies the robustness of our agent when competing with various opponents.
ArXiv
Peng Peng Liang Pang Yufeng Yuan Chao Gao