---
abstract: 'In this paper, we introduce `ChainerRL`, an open-source Deep Reinforcement Learning (DRL) library built using Python and the `Chainer` deep learning framework. `ChainerRL` implements a comprehensive set of DRL algorithms and techniques drawn from the state-of-the-art research in the field. To foster reproducible research, and for instructional purposes, provides scripts that closely replicate the original papers’ experimental settings and reproduce published benchmark results for several algorithms. Lastly, `ChainerRL` offers a visualization tool that enables the qualitative inspection of trained agents. The `ChainerRL` source code can be found on GitHub: <https://github.com/chainer/chainerrl>.'
author:
- |
    Yasuhiro Fujita\
    Preferred Networks, Inc.\
    Tokyo, Japan\
    `fujita@preferred.jp`\
    Toshiki Kataoka\
    Preferred Networks, Inc.\
    Tokyo, Japan\
    `kataoka@preferred.jp`\
    Prabhat Nagarajan\
    Preferred Networks, Inc.\
    Tokyo, Japan\
    `prabhat@preferred.jp`\
    Takahiro Ishikawa\
    The University of Tokyo\
    Tokyo, Japan\
    `sykwer@g.ecc.u-tokyo.ac.jp`\
bibliography:
- 'neurips/citations.bib'
title: 'ChainerRL: A Deep Reinforcement Learning Library'
---

Introduction
============

In recent years, significant strides have been made in numerous complex sequential decision-making problems including game-playing [@dqn; @alphazero] and robotic manipulation [@Levine2016; @Kalashnikov2018]. These advances have been enabled through Deep Reinforcement Learning (DRL), which has undergone tremendous progress since its resurgence in 2013, with the introduction of deep Q-networks [@dqn2013]. Since then, the body of literature on DRL algorithms has rapidly grown. This growing body of algorithms, coupled with the emergence of common benchmark tasks [@ale; @mujoco] begets the need for comprehensive libraries, tools, and implementations that can aid RL-based research and development.

While DRL has demonstrated several successes, as a field it still faces significant impediments. DRL algorithms are often sensitive to hyperparameter settings and implementation details, which often go unreported in published work, raising concerns about the state of reproducibility in the field [@deeprlmatters]. These seemingly minor implementation details have striking effects in DRL algorithms since the data collection process is closely tied to the parameter updates, as opposed to the typical supervised learning setting. Such issues make the task of reproducing published results challenging when the original implementation is not open-sourced.

Many open-source software packages have been developed to alleviate these issues by providing reference algorithm implementations. However, as the state-of-the-art rapidly advances, it is difficult for DRL libraries to keep apace. Even when most open-source software packages make re-implementations available, they seldom provide comprehensive benchmarks or implementations that faithfully replicate the original settings of published results.

In this paper, we introduce , an open-source DRL library, built using Python and the `Chainer` [@chainer] deep learning framework. has the following features/characteristics:

Comprehensive

:   aims for comprehensiveness as a DRL library. It spans algorithms for both discrete-action and continuous-action tasks, from foundational algorithms such as DQN [@dqn] and DDPG [@ddpg] to state-of-the-art algorithms such as IQN [@Dabney2018], Rainbow [@rainbow], Soft-Actor-Critic [@sac], and TD3 [@Fujimoto2018a]. Moreover, supports multiple training paradigms, including standard serial algorithms (e.g. DQN), asynchronous parallel algorithms (e.g. A3C [@a3c]), and synchronous parallel training (e.g. A2C [@a2c]).

Faithful reproduction

:   To provide reliable baselines as a starting point for future research, emphasizes the faithfulness of its algorithm implementations to their corresponding original papers or implementations, if any. For several state-of-the-art algorithms, we replicate the published training and evaluation details as closely as possible, successfully reproducing the results from the original paper, both in the Atari 2600 [@ale] and `OpenAI Gym` [@gym] MuJoCo benchmarks. We provide training scripts with full reported scores and training times as well as comparisons against the original implementations for each reproduction.

Visualizer

:   is accompanied by the [`ChainerRL Visualizer`]{}, an agent visualization tool that enables users to effortlessly visualize the behavior of trained agents, both discrete-action and continuous-action. For tasks with image-based observations, it can show saliency maps [@visualizingatari] to indicate which part of the image the neural network is attending to.

Individually, these features/characteristics may not be entirely novel, but achieving all of them within a single, unified open-source platform gives a unique value that sets it apart from existing libraries.

This paper is organized as follows. In Section \[related\], we review related work on reproducibility in RL and existing DRL libraries. In Section \[overview\], we explain the design and functionalities of and introduce the [`ChainerRL Visualizer`]{}, our companion visualization tool. In Section \[reproducibility\], we describe our efforts to reproduce published benchmark results, and provide concluding remarks in Section \[conclusion\].

Related work {#related}
============

Recent work has raised a number of concerns about the state of reproducibility in the field of DRL  [@Islam2017; @deeprlmatters]. Minor differences in hyperparameters, architectures, reward scales, etc. can have dramatic effects on the performance of an agent. Even minor implementational differences can cause significant differences between different open-source implementations of the same algorithm. It is also known that there can be high variance in performance among training trials even with identical settings save for the random seeds, necessitating evaluation with multiple random seeds and statistical testing [@Colas2018b].

Many DRL libraries have been released to provide reference implementations of algorithms with different focuses. `rllab` [@rllab] and its successor, `garage`, provide a thorough set of continuous-action algorithms and their own benchmark environments for systematic benchmarking. `Dopamine` [@dopamine] primarily focuses on DQN and its extensions for discrete-action environments. `rlpyt` [@rlpyt] is comprehensive as it supports both discrete and continuous-action algorithms from the three classes: policy gradient (with V-functions), deep Q-learning, and policy gradient with Q-functions. Several other libraries also support diverse sets of algorithms [@baselines; @coach; @stablebaselines; @rayrllib]. While it is difficult to compare the comprehensiveness of libraries that are actively developed, ’s support of a wide range of algorithms and functionality provides a competitive edge over the other libraries, as we detail in Section \[overview\]. `catalyst.RL` [@catalyst_rl] aims to address reproducibility issues in RL via deterministic evaluations and by tracking code changes for continuous-action algorithms. addresses the reproducibility challenge by providing implementations that closely follow the original papers’ descriptions and experimental settings. These implementations are then extensively benchmarked to best reproduce the original reported scores.

Design of ChainerRL {#overview}
===================

![A depiction of . In , DRL algorithms, called agents, are written by implementing the abstract `Agent` interface, typically using the offered building blocks. Such agents can be trained with the experiment utilities and inspected with the [`ChainerRL Visualizer`]{}. also provides a set of scripts for reproducing published results for agents. []{data-label="fig:crl"}](figures/CRLDiagram2.pdf){width="0.95\linewidth"}

Agents
------

In , each DRL algorithm is written as a class that implements the `Agent` interface. The `Agent` interface provides a mechanism through which an agent interacts with an environment, e.g. through an abstract method `Agent.act_and_train(obs, reward, done)` that takes the current observation, the previous step’s immediate reward, and a flag for episode termination, and returns the agent’s action to execute in the environment. By implementing such methods, both the update rule and the action-selection procedure are specified for an algorithm.

An agent’s internals consist of any model parameters needed for decision-making and model updating. includes several built-in agents that implement popular algorithms including DQN [@dqn], IQN [@Dabney2018], Rainbow [@rainbow], A2C [@a2c], A3C [@a3c], ACER [@acer], DDPG [@ddpg], PPO [@Schulman2017b], TRPO [@trpo], TD3 [@Fujimoto2018a], and SAC [@sac]. [^1]

Experiments
-----------

While users can directly use agents via the interface for maximum flexibility, provides an `experiments` module that manages the interactions between the agent and the environment as well as the training and evaluation schedule. The module supports any environment that is compatible with the `Env` class of `OpenAI Gym` [@gym]. An experiment takes as input an agent and an environment, queries the agent for actions, executes them in the environment, and feeds the agent the rewards for training updates. Moreover, an experiment can periodically perform evaluations, possibly in a separate evaluation environment, storing relevant statistics regarding the agent’s performance.

Through the `experiments` module, supports batch or asynchronous training, enabling agents to act and train synchronously or asynchronously in several environments in parallel. Asynchronous training, where multiple agents interact with multiple environments while sharing the model parameters, is supported for A3C, ACER, and $n$-step Q-learning [@a3c]. Synchronous parallel training, where a single agent interacts with multiple environments synchronously, known to practitioners for A2C [@a2c], is supported to leverage GPUs not only for A2C but also for the majority of agents including IQN [@Dabney2018] and Soft Actor-Critic [@sac].

Developing a new agent
----------------------

The `Agent` interface is defined very abstractly and flexibly so that users can easily implement new algorithms while leveraging the `experiments` utility and parallel training infrastructure. The general flow for developing and evaluating a new agent is as follows. First, a class that inherits `Agent` is created. Next, the learning update rules and the algorithm’s action-selection mechanisms are implemented, employing the many building blocks that provides for building agents (see Section \[features\]). Once an agent is created, one can use any `Gym`-like environment combined with our experimental and evaluation utilities in `experiments` to easily train and evaluate the agent within the specified environment.

Agent building blocks {#features}
---------------------

We have described at a high level how agents interact with the environment in , as well as some of the built-in agents and experimental utilities offered. However, these built-in agents are typically built with a set of reusable components that offers. While a comprehensive treatment of the features built into is beyond the scope of this paper, we highlight here some of the building blocks that demonstrate the flexibility and reusability of .

Explorers

:   To easily develop an agent’s action-selection mechanisms during training, has several built-in `Explorer`s including $\epsilon$-greedy, Boltzmann exploration, additive Gaussian noise, and additive Ornstein-Uhlenbeck noise [@ddpg].

Replay buffers

:   Replay buffers [@Lin1992; @dqn] have become standard tools in off-policy DRL. In addition to the standard replay buffer that uniformly samples transitions, supports episodic buffers that sample past (sub-)episodes for recurrent models, and prioritized buffers that implement prioritized experience replay [@prioritizeddqn]. also supports sampling $N$ steps of transitions, allowing for the easy implementation of algorithm variants based on $N$-step returns.

Neural networks

:   While supports any neural network model that is implemented in `Chainer` [@chainer] as `chainer.Link`, it has several pre-defined architectures, including DQN architectures, dueling network architectures [@duelingdqn], noisy networks [@noisynetworks], and multi-layer perceptrons. Recurrent models are supported for many algorithms, including DQN and IQN.

Distributions

:   `Distribution`s are parameterized objects used to model action distributions. Neural network models that return a `Distribution` object are considered a policy. Supported policies include Gaussian, Softmax, Mellowmax [@Asadi2016a], and deterministic policies.

Action values

:   Similarly to `Distribution`s, `ActionValue`s parameterizing the values of actions are used as outputs of neural networks to model Q-functions. Supported Q-functions include the one that evaluates discrete actions typical for DQN as well as categorical [@distributionaldqn] and quantile [@Dabney2018] Q-functions for distributional RL. For continuous action spaces, quadratic Q-functions called Normalized Advantage Functions (NAFs) [@Gu2016b] are also supported.

The set of algorithms that can be developed by combining the agent building blocks of is large. One notable example is Rainbow [@rainbow], which combines double updating [@ddqn], prioritized replay [@prioritizeddqn], $N$-step learning, dueling architectures [@duelingdqn], and Categorical DQN [@distributionaldqn] into a single agent. The following pseudocode depicts the simplicity of creating and training a Rainbow agent with .

\[ frame=lines, bgcolor=LightGray, linenos, breaklines \][python]{} import chainerrl as crl import gym

q\_func = crl.q\_functions.DistributionalDuelingDQN(…)\# dueling crl.links.to\_factorized\_noisy(q\_func) \# noisy networks \# Prioritized Experience Replay Buffer with a 3-step reward per = crl.replay\_buffers.PrioritizedReplayBuffer(num\_step\_return=3,…) \# Create a rainbow agent rainbow = crl.agents.CategoricalDoubleDQN(per, q\_func,…) num\_envs = 5 \# Train in five environments env = crl.envs.MultiprocessVectorEnv( \[gym.make(“Breakout”) for \_ in range(num\_envs)\])

\# Train the agent and collect evaluation statistics crl.experiments.train\_agent\_batch\_with\_evaluation(rainbow, env, steps=…)

We first create a distributional dueling Q-function, and then in a single line, convert it to a noisy network. We then initialize a prioritized replay buffer configured to use $N$-step rewards. We pass this replay buffer to a `DoubleCategoricalDQN` agent — which is a built-in agent — to produce a Rainbow agent. Moreover, with , users can easily specify the number of environments in which to train the Rainbow agent in parallel processes, and the `experiments` module will automatically manage the training loops, evaluation statistics, logging, and saving of the agent.

Visualization
-------------

![ The [`ChainerRL Visualizer`]{}. With the [`ChainerRL Visualizer`]{}, the user can closely investigate an agent’s behaviors within a browser window. The top image is the visualization of a trained A3C agent on <span style="font-variant:small-caps;">Breakout</span>, while the bottom one is that of a C51 agent trained on <span style="font-variant:small-caps;">Seaquest</span>. []{data-label="fig:viz1"}](figures/crlviz1.png "fig:"){width="0.95\linewidth"} ![ The [`ChainerRL Visualizer`]{}. With the [`ChainerRL Visualizer`]{}, the user can closely investigate an agent’s behaviors within a browser window. The top image is the visualization of a trained A3C agent on <span style="font-variant:small-caps;">Breakout</span>, while the bottom one is that of a C51 agent trained on <span style="font-variant:small-caps;">Seaquest</span>. []{data-label="fig:viz1"}](figures/crlviz2.png "fig:"){width="0.95\linewidth"}

is accompanied by a visualizer: [`ChainerRL Visualizer`]{}, which takes as input an environment and an agent, and allows users to inspect their agents from a browser UI easily. Figure \[fig:viz1\] depicts some of the key features available in the [`ChainerRL Visualizer`]{}. The top of the figure depicts a trained A3C agent in the Atari game <span style="font-variant:small-caps;">Breakout</span>. With the visualizer, one can visualize the portions of the pixel input that the agent is attending to as a saliency map [@visualizingatari]. Additionally, users can perform careful, controlled investigations of agents by manually stepping through an episode, or can alternatively view rollouts of agents. Since A3C is an actor-critic agent, we can view the probabilities with which the agent will perform a specific action, as well as the agent’s predicted state values. If the agent learns Q-values or a distribution of Q-values, the predicted Q-value or Q-value distribution for each action can be displayed, as shown in the bottom of Figure \[fig:viz1\].

Reproducibility
===============

                                          **DQN** [@dqn]    **IQN** [@Dabney2018]   **Rainbow** [@rainbow]   **A3C** [@noisynetworks]
  -------------------------------------- ----------------- ----------------------- ------------------------ --------------------------
  **Evaluation Frequency** (timesteps)         250K                 250K                     250K                      250K
  **Evaluation Phase** (timesteps)             125K                 125K                     125K                      125K
  **Evaluation Episode Length** (time)         5 min               30 min                   30 min                    30 min
  **Evaluation Episode Policy**           $\epsilon=0.05$     $\epsilon=0.001$          $\epsilon=0.0$                 N/A
  **Reporting Protocol**                     *re-eval*           *best-eval*              *re-eval*                *best-eval*

Many open-source DRL libraries offer high-quality implementations of algorithms but often deviate from the original paper’s implementation details. We currently provide a set of compact examples, i.e., single files, of paper implementations written with that users can run to reproduce the results of the original research paper. These “reproducibility scripts” are carefully written to replicate as closely as possible the original paper’s (or in some cases, another published paper’s) implementation and evaluation details. For each of these scripts, we provide the training times of the script (in our repository), full tables of our achieved scores, and comparisons of these scores against those reported in the literature.

Though has high-quality implementations of dozens of algorithms, we currently have created such “reproducibility scripts” for 9 algorithms. In the Atari benchmark [@ale], we have successfully reproduced DQN, IQN, Rainbow, and A3C. For the OpenAI Gym Mujoco benchmark tasks, we have successfully reproduced DDPG, TRPO, PPO, TD3, and SAC.

The reproducibility scripts emphasize correctly reproducing evaluation protocols, which are particularly relevant when evaluating Atari agents. During a typical Atari agent’s 50 million timesteps of training, it is evaluated periodically in an offline evaluation phase for a specified number of timesteps before resuming training. Oftentimes, since the agent performs some form of exploratory policy during training, the agent sometimes changes policies specifically for evaluations. Unfortunately, evaluation protocols tend to vary, and consequently results are often inconsistently reported across the literature [@revisitingale], significantly impacting results. The critical details of standard Atari evaluation protocols are as follows:

Evaluation frequency

:   the frequency (in timesteps) with which the evaluation phase occurs

Evaluation phase length

:   the number of timesteps in the offline evaluation

Evaluation episode length

:   the maximum duration of an evaluation episode

Evaluation policy

:   The policy to follow during an evaluation episode

Reporting protocol

:   Each intermediate evaluation phase outputs some score, representing the mean score of all evaluation episodes during that evaluation phase. Papers typically report scores according to one of the following reporting protocols:

    1.  *best-eval*: Papers using the *best-eval* protocol report the highest mean score across all intermediate evaluation phases.

    2.  *re-eval*: Papers using the *re-eval* protocol report the score of a re-evaluation of the network parameters that produced the *best-eval*.

Each of these details, especially the reporting protocols, can significantly influence the results, and thus are critical details to hold consistent for a fair comparison between algorithms.

Table \[atariresults\] lists the results obtained by ’s reproducibility scripts for DQN, IQN, Rainbow, and A3C on the Atari benchmark, with comparisons against a published result. Table \[evalprotocolatari\] depicts the evaluation protocol used for each algorithm, with a citation of the source paper whose results we compare against. Note that the results for the A3C [@a3c] algorithm do not come from the original A3C paper, but from another [@noisynetworks]. For continuous-action algorithms, the results on `OpenAI Gym` MuJoCo tasks for DDPG [@ddpg], TRPO [@trpo], PPO [@Schulman2017b], TD3 [@Fujimoto2018a], and SAC [@sac] are reported in Table \[mujocoresults\].

The reproducibility scripts are produced through a combination of reading released source code and studying published hyperparameters, implementation details, and evaluation protocols. We also have extensive email correspondences with authors to clarify ambiguities, omitted details, or inconsistencies that may exist in papers.

As seen in both the Atari and MuJoCo reproducibility results, sometimes a reproduction effort cannot be directly compared against the original paper’s reported results. For example, the reported scores in the original paper introducing the A3C algorithm [@a3c] utilize demonstrations that are not publicly available, making it impossible to accurately compare a re-implementation’s scores to the original paper. In such scenarios, we seek out high-quality published research [@noisynetworks; @deeprlmatters; @Fujimoto2018a] from which faithful reproductions are indeed possible, and compare against these.

Conclusion
==========

In this paper, we introduced a reproducibility-focused deep reinforcement library, , and its companion visualizer, [`ChainerRL Visualizer`]{}. We hope that ’s comprehensive suite of algorithms, flexible APIs, visualization features, and faithful reproductions can accelerate research in DRL as well as foster its application to a wide range of new and interesting sequential decision-making problems.

has been in active development since 2017 with extensive plans laid out for continued expansion and improvement with novel algorithms, functionality, and paper reproductions. We are currently in the process of releasing a significant number of trained agent models for users to utilize in research and development. We are also adding functionality to enable large scale distributed RL. Lastly, we plan to expand beyond pure reinforcement learning approaches for sequential decision-making by including algorithms that can learn from demonstrations.

We look forward to continuing our collaboration with the open-source community in developing and accelerating RL research.

### Acknowledgments {#acknowledgments .unnumbered}

We thank Avinash Ummadisingu, Mario Ynocente Castro, Keisuke Nakata, Lester James V. Miranda, and all the open source contributors for their contributions to the development of ChainerRL. We thank Kohei Hayashi and Jason Naradowsky for useful comments on how to improve the paper. We thank the many authors who fielded our questions when reproducing their papers, especially George Ostrovski.

[^1]: Within the Deep Q-networks (DQN) [@dqn] family of algorithms, has: DQN [@dqn], Double DQN [@ddqn], Categorical DQN [@distributionaldqn], Rainbow [@rainbow], Implicit Quantile Networks (IQN) [@Dabney2018], Off-policy SARSA, and (Persistent) Advantage Learning [@Bellemare2016c]. Within policy gradient methods, has: (Asynchronous) Advantage Actor-Critic (A2C [@a2c] and A3C [@a3c]), Actor-Critic with Experience Replay (ACER) [@acer], Deep Deterministic Policy Gradients (DDPG) [@ddpg], Twin-delayed double DDPG (TD3) [@Fujimoto2018a], Proximal Policy Optimization (PPO) [@Schulman2017b], REINFORCE [@Williams1992], Trust Region Policy Optimization (TRPO) [@trpo], and Soft Actor-Critic (SAC) [@sac].
