\section{Introduction}
\label{sec:introduction}

There are significant benefits to using a single neural sequence model across all tasks.
It reduces the need for hand crafting policy models with appropriate inductive biases for each domain.
It increases the amount and diversity of training data since the sequence model can ingest any data that can be serialized into a flat sequence.
Furthermore, its performance continues to improve even at the frontier of data, compute and model scale~\citep{kaplan2020scaling, hoffmann2022training}.
Historically, generic models that are better at leveraging computation have also tended to overtake more specialized domain-specific approaches~\citep{sutton2019bitter}, eventually.

In this paper, we describe the current iteration of a general-purpose agent which we call {Gato}{}, instantiated as a single, large, transformer sequence model.
With a single set of weights, {Gato}{} can engage in dialogue, caption images, stack blocks with a real robot arm, outperform humans at playing Atari games, navigate in simulated 3D environments, follow instructions, and more.

While no agent can be expected to excel in all imaginable control tasks, especially those far outside of its training distribution, we here test the hypothesis that training an agent which is generally capable on a \emph{large number} of tasks is possible; and that this general agent can be adapted with little extra data to succeed at an even larger number of tasks.
We hypothesize that such an agent can be obtained through scaling data, compute and model parameters, continually broadening the training distribution while maintaining performance, towards covering any task, behavior and embodiment of interest.
In this setting, natural language can act as a common grounding across otherwise incompatible embodiments, unlocking combinatorial generalization to new behaviors.

We focus our training at the operating point of model scale that allows real-time control of real-world robots, currently around 1.2B parameters in the case of~{Gato}{}.
As hardware and model architectures improve, this operating point will naturally increase the feasible model size, pushing generalist models higher up the scaling law curve.
For simplicity~{Gato}{} was trained offline in a purely supervised manner; however, in principle, there is no reason it could not also be trained with either offline or online reinforcement learning (RL).

\begin{figure*}[t]
    \raggedleft
    \includegraphics[width=0.99\linewidth]{figures/diagram_train.pdf}
    \caption{{\bf Training phase of {Gato}{}}. Data from different tasks and modalities is serialized into a flat sequence of tokens, batched, and processed by a transformer neural network akin to a large language model. Masking is used such that the loss function is applied only to target outputs, i.e. text and various actions.}
    \label{fig:calm-diagram}
\end{figure*}

\section{Model}
\label{sec:model}

The guiding design principle of {Gato}{} is to train on the widest variety of relevant data possible, including diverse modalities such as images, text, proprioception, joint torques, button presses, and other discrete and continuous observations and actions.
To enable processing this multi-modal data, we serialize all data into a flat sequence of tokens.
In this representation, {Gato}{} can be trained and sampled from akin to a standard large-scale language model.
During deployment, sampled tokens are assembled into dialogue responses, captions, button presses, or other actions based on the context.
In the following subsections, we describe {Gato}{}'s tokenization, network architecture, loss function, and deployment.

\subsection{Tokenization}\label{sec:tokenization}

There are infinite possible ways to transform data into tokens, including directly using the raw underlying byte stream.
Below we report the tokenization scheme we found to produce the best results for {Gato}{} at the current scale using contemporary hardware and model architectures.
\begin{itemize}
    \item Text is encoded via SentencePiece~\citep{kudo-richardson-2018-sentencepiece} with 32000 subwords into the integer range [0, 32000).
    \item Images are first transformed into sequences of non-overlapping $16\times 16$ patches in raster order, as done in ViT~\citep{dosovitskiy2020image}. Each pixel in the image patches is then normalized between $[-1, 1]$ and divided by the square-root of the patch size (i.e.\ $\sqrt{16} = 4$).
    \item Discrete values, e.g. Atari button presses, are flattened into sequences of integers in row-major order. The tokenized result is a sequence of integers within the range of $[0, 1024)$.
    \item Continuous values, e.g. proprioceptive inputs or joint torques, are first flattened into sequences of floating point values in row-major order. The values are mu-law encoded to the range  $[-1, 1]$ if not already there (see Figure~\ref{fig:proprio_tokenization} for details), then discretized to 1024 uniform bins. The discrete integers are then shifted to the range of $[32000, 33024)$.
\end{itemize}

\noindent After converting data into tokens, we use the following canonical sequence ordering.
\begin{itemize}
    \item Text tokens in the same order as the raw input text.
    \item Image patch tokens in raster order.
    \item Tensors in row-major order.
    \item Nested structures in lexicographical order by key.
    \item Agent timesteps as observation tokens followed by a separator, then action tokens.
    \item Agent episodes as timesteps in time order.
\end{itemize}

\noindent Further details on tokenizing agent data are presented in the supplementary material (Section~\ref{sec:tokenization_appendix}).

\subsection{Embedding input tokens and setting output targets}
\label{sec:embed}

After tokenization and sequencing, we apply a parameterized embedding function $f(\cdot; \theta_e)$ to each token (i.e. it is applied to both observations and actions) to produce the final model input.
To enable efficient learning from our multi-modal input sequence $s_{1:L}$ the embedding function performs different operations depending on the modality the token stems from:

\begin{itemize}
\item Tokens belonging to text, discrete- or continuous-valued observations or actions for any time-step are embedded via a lookup table into a learned vector embedding space. Learnable position encodings are added for all tokens based on their local token position within their corresponding time-step.
\item Tokens belonging to image patches for any time-step are embedded using a single ResNet~\citep{he2016deep} block to obtain a vector per patch. For image patch token embeddings, we also add a learnable within-image position encoding vector.
\end{itemize}
We refer to appendix Section~\ref{sec:position_encodings} for full details on the embedding function.

As we model the data autoregressively, each token is potentially also a target label given the previous tokens.
Text tokens, discrete and continuous values, and actions can be directly set as targets after tokenization.
Image tokens and agent observations are not currently predicted in {Gato}{}, although that may be an interesting direction for future work.
Targets for these non-predicted tokens are set to an unused value and their contribution to the loss is masked out.

\subsection{Training}
Given a sequence of tokens $s_{1:L}$ and parameters $\theta$, we model the data using the chain rule of probability:
\begin{align}
     \log p_{\theta}(s_1,\ldots,s_L) = \sum_{l=1}^L \log p_{\theta}(s_l|s_1,\ldots,s_{l-1}),
\end{align}
Let $b$ index a training batch of sequences $\mathcal{B}$.
We define a masking function $m$ such that $m(b,l)=1$ if the token at index $l$ is either from text or from the logged action of an agent, and $0$ otherwise.
The training loss for a batch $\mathcal{B}$ can then be written as
\begin{align}
    \mathcal{L}(\theta, \mathcal{B}) = - \sum_{b=1}^{|\mathcal{B}|} \sum_{l=1}^L m\left(b, l\right) \log p_\theta\left(s^{(b)}_l|s^{(b)}_1,\ldots,s^{(b)}_{l-1}\right)
\end{align}

As described above, {Gato}{}'s network architecture has two main components: the parameterized embedding function which transforms tokens to token embeddings, and the sequence model which outputs a distribution over the next discrete token.
While any general sequence model can work for next token prediction, we chose a transformer~\citep{vaswani2017attention} for simplicity and scalability.
{Gato}{} uses a 1.2B parameter decoder-only transformer with 24 layers, an embedding size of 2048, and a post-attention feedforward hidden size of 8196 (more details in Section~\ref{app:trans_hparams}).

Because distinct tasks within a domain can share identical embodiments, observation formats and action specifications, the model sometimes needs further context to disambiguate tasks.
Rather than providing e.g. one-hot task identifiers, we instead take inspiration from ~\citep{sanh2022multitask,wei2021finetuned,brown2020language} and use prompt conditioning.
During training, for $25\%$ of the sequences in each batch, a prompt sequence is prepended, coming from an episode generated by the same source agent on the same task.
Half of the prompt sequences are from the end of the episode, acting as a form of goal conditioning for many domains; and the other half are uniformly sampled from the episode.
During evaluation, the agent can be prompted using a successful demonstration of the desired task, which we do by default in all control results that we present here.

\begin{figure*}[t]
	\centering
	\includegraphics[width=0.8\textwidth]{figures/eval4x.png}
	\caption{\textbf{Running {Gato}{} as a control policy.} {Gato}{} consumes a sequence of interleaved tokenized observations, separator tokens, and previously sampled actions to produce the next action in standard autoregressive manner. The new action is applied to the environment -- a game console in this illustration, a new set of observations is obtained, and the process repeats.}
	\label{fig:evaluation}
\end{figure*}

Training of the model is performed on a 16x16 TPU v3 slice for 1M steps with batch size 512 and token sequence length $L = 1024$, which takes about 4 days.
Architecture details can be found in Section~\ref{app:model_arch}.
Because agent episodes and documents can easily contain many more tokens than fit into context, we randomly sample subsequences of $L$ tokens from the available episodes.
Each batch mixes subsequences approximately uniformly over domains (e.g. Atari, MassiveWeb, etc.), with some manual upweighting of larger and higher quality datasets (see Table~\ref{table:datasets} in Section~\ref{sec:datasets} for details).

\subsection{Deployment}
\label{sec:evaluation_protocol}

Deploying {Gato}{} as a policy is illustrated in Figure~\ref{fig:evaluation}.
First a prompt, such as a demonstration, is tokenized, forming the initial sequence.
By default, we take the first 1024 tokens of the demonstration.
Next the environment yields the first observation which is tokenized and appended to the sequence.
{Gato}{} samples the action vector autoregressively one token at a time.
Once all tokens comprising the action vector have been sampled (determined by the action specification of the environment), the action is decoded by inverting the tokenization procedure described in Section~\ref{sec:tokenization}.
This action is sent to the environment which steps and yields a new observation.
The procedure repeats.
The model always sees all previous observations and actions in its context window of 1024 tokens.
We found it beneficial to use transformer XL memory during deployment, although it was not used during training~\citep{dai2019transformer}.

\section{Datasets}
\label{sec:datasets}

\begin{table*}
    \caption{
    {\bf Datasets.}
    Left: Control datasets used to train {Gato}{}. Right: Vision \& language datasets. Sample weight means the proportion of each dataset, on average, in the training sequence batches.\label{table:datasets}}
    \centering
    \scalebox{0.85}{
    \begin{tabular}[t]{|l|c|c|c|c|}
          \hline
          Control environment & Tasks & Episodes & \makecell{Approx. \\ Tokens} & \makecell{Sample \\ Weight} \\
          \hline
          {DM Lab}{} & 254 & 16.4M & 194B & 9.35\% \\
          {ALE Atari}{} & 51  & 63.4K & 1.26B & 9.5\% \\
          {ALE Atari Extended}{} & 28  & 28.4K & 565M & 10.0\% \\
          {Sokoban}{} & 1   & 27.2K & 298M & 1.33\% \\
          {BabyAI}{} & 46  & 4.61M & 22.8B & 9.06\% \\
          {DM Control Suite}{}  & 30  & 395K  & 22.5B & 4.62\% \\
          {DM Control Suite Pixels}{} & 28  & 485K & 35.5B & 7.07\% \\
          {DM Control Suite Random Small}{} & 26  & 10.6M & 313B & 3.04\% \\
          {DM Control Suite Random Large}{} & 26  & 26.1M & 791B & 3.04\% \\
          {Meta-World}{} & 45  & 94.6K & 3.39B & 8.96\% \\
          {Procgen Benchmark}{} & 16  & 1.6M  & 4.46B & 5.34\% \\
          {RGB Stacking simulator}{} & 1   & 387K  & 24.4B & 1.33\% \\
          {RGB Stacking real robot}{} & 1   & 15.7K & 980M  & 1.33\% \\
          {Modular RL}{} & 38  & 843K  & 69.6B & 8.23\% \\
          {DM Manipulation Playground}{} & 4   & 286K  & 6.58B & 1.68\% \\
          {Playroom}{} & 1 & 829K & 118B & 1.33\% \\
          \hline
          Total               & 596 & 63M & 1.5T & 85.3\% \\
          \hline
    \end{tabular}}
    \scalebox{0.85}{
    \begin{tabular}[t]{|l|c|}
        \hline
        Vision / language dataset & \makecell{Sample \\ Weight} \\
        \hline
        MassiveText & 6.7\%  \\
        M3W & 4\% \\
        ALIGN & 0.67\% \\
        MS-COCO Captions  & 0.67\% \\
        Conceptual Captions & 0.67\% \\
        LTIP& 0.67\% \\
        OKVQA & 0.67\% \\
        VQAV2 & 0.67\% \\
        \hline
        Total & 14.7\% \\
        \hline
    \end{tabular}
    }
\end{table*}
{Gato}{} is trained on a large number of datasets comprising agent experience in both simulated and real world environments, as well as a variety of natural language and image datasets.
The datasets we use and their attributes are listed in Table~\ref{table:datasets}.
The approximate number of tokens per control dataset is computed assuming the tokenization mechanism described in Section~\ref{sec:tokenization}.

\subsection{Simulated control tasks}

Our control tasks consist of datasets generated by specialist SoTA or near-SoTA reinforcement learning agents trained on a variety of different environments.
For each environment we record a subset of the experience the agent generates (states, actions, and rewards) while it is training.

The simulated environments include
{Meta-World}{}~\citep{yu2020meta} introduced to benchmark meta-reinforcement learning and multi-task learning,
{Sokoban}{}~\citep{racaniere2017imagination} proposed as a planning problem,
{BabyAI}{}~\citep{chevalier2018babyai} for language instruction following in grid-worlds,
the {DM Control Suite}{}~\citep{tunyasuvunakool2020dmcontrol} for continuous control, as well as
{DM Lab}{}~\citep{beattie2016deepmind} designed to teach agents navigation and 3D vision from raw pixels with an egocentric viewpoint.
We also use the Arcade Learning Environment~\citep{bellemare2013arcade} with classic Atari games (we use two sets of games that we call {ALE Atari}{} and {ALE Atari Extended}{}, see Section~\ref{sec:atari_details} for details).
We as well include the {Procgen Benchmark}{}~\citep{cobbe2020leveraging} and  {Modular RL}{}~\citep{huang2020one}.
We also include four tasks using a simulated Kinova Jaco arm from {DM Manipulation Playground}{}, as introduced in~\cite{zolna2020offline}.
Section~\ref{sec:data-collection} includes a more in-depth description of these control tasks, along with what RL agent was used to generate the data.

We found it effective to train on a filtered set of episodes with returns at least 80\% of the expert return for the task.
The expert return measures the maximum sustained performance that the expert agent can achieve.
We define it as the maximum over the set of all windowed average returns calculated over all the collected episodes for a task:
\begin{align*}
\label{eq:expert-return}
    \max_{j \in [0, 1, ..., N - W]} \left( \sum_{i=j}^{j+L-1} \frac{R_i}{W} \right)
\end{align*}
where $N$ it the total number of collected episodes for the task, $W$ is the window size, and $R_i$ is the total return for episode $i$.
To obtain accurate estimates, in practice, we set $W$ to be $10 \%$ of the total data amount or a minimum of 1000 episodes (i.e. $W = \min(1000, 0.1 \times N)$).

\subsection{Vision and language}
\label{sec:vision-and-language}
{Gato}{} is trained on MassiveText~\citep{rae2021scaling}, a collection of large English-language text datasets from multiple sources: web pages, books, news articles, and code.

We also included several vision-language datasets in {Gato}{}'s training.
ALIGN~\citep{jia2021scaling} consists of 1.8B images and their alternative text (alt-text) annotations.
LTIP (Long Text \& Image Pairs), consists of 312 million images with captions~\citep{Alayrac2022FlamingoAV}.
Conceptual captions~\citep{sharma-etal-2018-conceptual} and COCO captions~\citep{chen2015microsoft} are captioning datasets with 3.3M and 120k image-text pairs respectively.
The MultiModal MassiveWeb (M3W) dataset~\citep{Alayrac2022FlamingoAV} includes 43M webpages where both text and images were extracted.
We also included visual question-answering datasets. In particular OKVQA~\citep{marino2019ok} and VQAv2~\citep{antol2015vqa}
with 9K and 443K triplets of images, questions, and answers.
To form a training episode from these, we sample five (image, text) pairs, tokenize them, concatenate, and then pad or randomly crop to the required training sequence length.

\subsection{Robotics - RGB Stacking Benchmark (real and sim)}

As a testbed for taking physical actions in the real world, we chose the robotic block stacking environment introduced by~\citet{lee2021beyond}. The environment consists of a Sawyer robot arm with 3-DoF cartesian velocity control, an additional DoF for velocity, and a discrete gripper action. The robot's workspace contains three plastic blocks colored red, green and blue with varying shapes.
The available observations include two 128 $\times$ 128 camera images, robot arm and gripper joint angles as well as the robot's end-effector pose. Notably, ground truth state information for the three objects in the basket is not observed by the agent. Episodes have a fixed length of 400 timesteps at 20 Hz for a total of 20 seconds, and at the end of an episode block positions are randomly re-positioned within the workspace. The robot in action is shown in Figure~\ref{fig:stack_strike_cage}. There are two challenges in this benchmark: \emph{Skill Mastery} (where the agent is provided data from the 5 test object triplets it is later tested on) and \emph{Skill Generalization} (where data can only be obtained from a set of training objects that excludes the 5 test sets).

We used several sources of training data for these tasks. In Skill Generalization, for both simulation and real, we use data collected by the best generalist sim2real agent from \citet{lee2021beyond}. We collected data only when interacting with the designated RGB-stacking \emph{training objects} (this amounts to a total of 387k successful trajectories in simulation and 15k trajectories in real). For Skill Mastery we used data from the best per group experts from \citet{lee2021beyond} in simulation and from the best sim2real policy on the real robot (amounting to 219k trajectories in total). Note that this data is only included for specific Skill Mastery experiments in Section~\ref{sec:skill_mastery}.

\begin{figure}
    \includegraphics[width=\textwidth]{figures/skill_generalization_challenge}
    \caption{{\bf RGB Stacking environment with the Sawyer robot arm.} Blocks vary along several shape axes, with 5 held out test triplets. The goal is to stack red on blue, ignoring green. \label{fig:stack_strike_cage}}
\end{figure}

\section{Capabilities of the generalist agent}
\label{sec:results}

In this section, we summarize the performance of {Gato}{} when trained on the above described data. That is, all results across all tasks are derived from a single pretrained model with a single set of weights. Results with fine-tuning will be presented in Section~\ref{sec:additional_results}.

\subsection{Simulated control tasks}\label{sec:simulated_control_task}
Figure \ref{fig:indist_barplot} shows the number of distinct control tasks for which {Gato}{} performs above a given score threshold, relative to expert performance demonstrated in {Gato}{}'s training data.

We report performance as a percentage, where $100\%$ corresponds to the per-task expert and $0\%$ to a random policy.
For each simulated control task we trained our model on, we roll out the {Gato}{} policy on the corresponding environment 50 times and average the defined scores.
As shown in Figure~\ref{fig:indist_barplot}, {Gato}{} performs over 450 out of 604 tasks at over a $50\%$ expert score threshold.

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{figures/barplot_domains}
    \caption{
    {\bf {Gato}{}'s performance on simulated control tasks.} Number of tasks where the performance of the pretrained model is above a percentage of expert score, grouped by domain. Here values on the x-axis represent a specific percentage of expert score, where 0 corresponds to random agent performance. The y-axis is the number of tasks where the pretrained model's mean performance is equal to or above that percentage. That is, the width of each colour band indicates the number of tasks where {Gato}{}'s mean performance is above a percentage of the maximum score obtained by a task-specific expert.
    \label{fig:indist_barplot}}
\end{figure}

In {ALE Atari}{}~\citep{bellemare2013arcade} {Gato}{} achieves the average human (or better) scores for 23 Atari games\footnote{The full list of games: Assault, Atlantis, Bank heist, Battle zone, Bowling, Crazy climber, Defender, Fishing derby, Gopher, Hero, Ice hockey, Jamesbond, Kangaroo, Kung fu master, Name this game, Pong, Road runner, Robotank, Tennis, Time pilot, Up n down, Wizard of wor, Zaxxon.}, achieving over twice human score for 11 games.
While the single-task online RL agents which generated the data still outperform {Gato}{}, this may be overcome by adding capacity or using offline RL training rather than purely supervised (see Section~\ref{sec:specialist_agents} where we present a specialist single domain {ALE Atari}{} agent achieving better than human scores for 44 games).

On BabyAI~\citep{chevalier2018babyai} {Gato}{} achieves over 80\% of expert score for nearly all levels\footnote{The only three tasks below 80\% success rate are GoToImpUnlock (59\%), Unlock (74\%), and BossLevel (75\%).}.
For the most difficult task, called BossLevel, {Gato}{} scores 75\%.
The two other published baselines we could find, BabyAI 1.0 and BabyAI 1.1~\citep{hui2020babyai}, scored 77\% and 90\%, respectively, having trained on this single task alone using a million demonstrations.

On Meta-World~\citep{yu2020meta} {Gato}{} achieves more than 50\% for all 44 out of 45 tasks that we trained on, over 80\% for 35 tasks, and over 90\% for 3 tasks.
On canonical DM Control Suite~\citep{tassa2018deepmind}, {Gato}{} achieves better than 50\% of the expert score on 21 out of 30 tasks from state, and more than 80\% for 18 tasks.

\subsection{Robotics}
\label{robotics_zeroshot}

First person teleoperation enables the collection of expert demonstrations. However, such demonstrations are slow and costly to collect.
Data-efficient behavior cloning methods are therefore desirable for training a generalist robot manipulator and offline pretraining is thus a well-motivated area of research.
To that end, we evaluated {Gato}{} on the established RGB Stacking benchmark for robotics.

\subsubsection*{Skill Generalization Performance}

The Skill Generalization challenge from the RGB Stacking robotics benchmark tests the agent's ability to stack objects of previously unseen shapes. The agent is trained on a dataset consisting of episodes of the robot stacking objects with a variety of different shapes. Five triplets of object shapes are, however, not included in the training data and serve as test triplets.
We evaluated the trained generalist for 200 episodes per test triplet on the real robot. Table~\ref{table:skill_generalization} shows that our generalist agent's success rate on each test triplet is comparable to the single task BC-IMP (filtered BC) baseline in \cite{lee2021beyond}.

\begin{table*}
    \caption{{\bf {Gato}{} real robot Skill Generalization results.} In addition to performing hundreds of other tasks, {Gato}{} also stacks competitively with the comparable published baseline. \label{table:skill_generalization}}
    \centering
    \sc
    \scalebox{0.85}{
    \begin{tabular}{l|r|r|r|r|r|r}
          \toprule
          Agent & Group 1 & Group 2 & Group 3 & Group 4 & Group 5 & Average \\
          \midrule
          {Gato}{}                     & \textbf{24.5}\% & 33\%            & \textbf{50.5}\% & 76.5\%          & \textbf{66.5}\% & \textbf{50.2}\% \\
          BC-IMP~\citep{lee2021beyond} & 23\%            & \textbf{39.3}\% & 39.3\%          & \textbf{77.5}\% & 66\% & 49\% \\
          \bottomrule
    \end{tabular}}
\end{table*}

\subsection{Text samples}

\begin{figure*}[hbtp]
    \centering
    \includegraphics[width=\linewidth]{figures/image_captions_v3.pdf}
    \caption{{\bf Image captions generated by {Gato}{}.} {Gato}{} prompted to be an image captioner, describing the first several held-out images from MS-COCO. We report the first three captions sampled using temperature 0.9, without cherry-picking. The prompt is shown in the appendix.
    \label{fig:vlm_caption_samples}}
    %
    \bigskip
    %
    \includegraphics[width=\linewidth]{figures/dialogue_examples_g1.pdf}
    \caption{\textbf{Chitchat with {Gato}{}.} Dialogues with {Gato}{} when it is prompted to be a chat bot. Usually {Gato}{} replies with a relevant response, but is often superficial or factually incorrect, which could likely be improved with further scaling. We used the same prompt as in~\citet{rae2021scaling}.
    \label{fig:chatbot_samples}}
\end{figure*}
The model demonstrates rudimentary dialogue and image captioning capabilities. Figure \ref{fig:vlm_caption_samples} contains a representative sample of {Gato}{}'s image captioning performance. Figure \ref{fig:chatbot_samples} shows some hand-picked examples of plain text dialogue exchange.

\section{Analysis}
\label{sec:additional_results}

\subsection{Scaling Laws Analysis}\label{sec:scaling_law_analysis}

\begin{figure*}[t]
    \centering
    \includegraphics[width=0.47\linewidth]{figures/paper_scaling_law}
    \caption{
    {\bf Model size scaling laws results.} In-distribution performance as a function of tokens processed for 3 model scales.
    Performance is first mean-aggregated within each separate control domain, and then mean-aggregated across all domains.
    We can see a consistent improvement as model capacity is increased for a fixed number of tokens.
    }
    \label{fig:scaling_law}
\end{figure*}

In Figure~\ref{fig:scaling_law}, we analyze the aggregate in-distribution performance of the pretrained model as a function of the number of parameters in order to get insight into how performance could improve with increased model capacity.
We evaluated 3 different model sizes (measured in  parameter count):
a \texttt{79M model},
a \texttt{364M model},
and a \texttt{1.18B model} ({Gato}{}).
We refer to Section~\ref{app:model_arch} for details on the three model architectures.

Here, for all three model sizes we plot the normalized return as training progresses.
To get this single value, for each task we calculate the performance of the model as a percentage of expert score (the same as done in Section~\ref{sec:simulated_control_task}).
Then for each domain listed in Table~\ref{table:datasets} we average the percentage scores across all tasks for that domain.
Finally, we mean-aggregate the percentage scores across all domains.
We can see that for an equivalent token count, there is a significant performance improvement with increased scale.

\subsection{Out of distribution tasks}\label{sec:ood_results}

In this section we want to answer the following question: \emph{Can our agent be used to solve a completely new task efficiently?}
For this reason, we held-out all data for four tasks from our pre-training set: \texttt{cartpole.swingup} ({DM Control Suite}{} domain), \texttt{assembly-v2} ({Meta-World}{} domain), \texttt{order\_of\_apples\_forage\_simple} ({DM Lab}{} domain), and \texttt{boxing} ({ALE Atari}{} domain).
These four tasks will serve as testbeds for evaluating the out-of-distribution capabilities of {Gato}{}.

Ideally, the agent could potentially learn to adapt to a new task via conditioning on a prompt including demonstrations of desired behaviour.
However, due to accelerator memory constraints and the extremely long sequence lengths of tokenized demonstrations, the  maximum context length possible does not allow the agent to attend over an informative-enough context.
Therefore, to adapt the agent to new tasks or behaviours, we choose to fine-tune the agent's parameters on a limited number of demonstrations of a single task, and then evaluate the fine-tuned model's performance in the environment.
Fine-tuning is very similar to pretraining with minor changes, such as different learning rate schedule; see Section~\ref{sec:finetuning_setup} for details.

We want to measure how choice of data used during pretraining influences post-fine-tuning performance.
To this end, we compare {Gato}{} (trained on \textit{all data}) to variants trained on ablated datasets:
\begin{enumerate}
\item A model pretrained only on data from the same domain as the task to be fine-tuned on, \textit{same domain only data}.
\item A model pretrained only on non-control data, \textit{no control data}.
\item A model fine-tuned from scratch, i.e.\ no pretraining at all, \textit{scratch}.
\end{enumerate}

Considering as all these experiments require training a new model from scratch and then also fine-tuning, we present results using the less compute-intensive \texttt{364M} parameter architecture described in Section~\ref{sec:scaling_law_analysis}.
Results are shown in Figure~\ref{fig:ood_ablations}.
\begin{figure*}[t]
    \includegraphics[width=\linewidth]{figures/ood_ablations}
    \caption{
    {\bf Few-shot performance, ablating over various pretraining settings.} Orange corresponds to the base {Gato}{} pretrained on all data. Red is trained from scratch only on the few-shot data. \texttt{364M} parameter variants of {Gato}{} were used for this experiment to save compute.
    \label{fig:ood_ablations}}
\end{figure*}

Fine-tuning performance on both \texttt{cartpole.swingup} and \texttt{assembly-v2} tasks, both of which do not require image processing, present similar trends.
Pretraining on all the datasets yields the best results, followed by pretraining on the same domain only. This difference is smaller for \texttt{assembly-v2} but consistent for all few shot datasets.
For these non-image-based environments, we see either no benefit (\texttt{cartpole.swingup}) or even negative transfer (\texttt{assembly-v2}) when pretraining on \textit{no control} datasets, which only contain images and text data.

Results for {DM Lab}{} \texttt{order\_of\_apples\_forage\_simple} are slightly different.
Pretraining on {DM Lab}{} data only is already enough to approach the maximum reward of 19 and hence there is no observable benefit of adding data from different environments.
What is different when compared to previously analysed no-vision environments is that pretraining on \textit{no control} data helps,
which can be possibly explained by the fact that agents in the {DM Lab}{} environment are fed images which, despite being simulated, are natural looking.
Therefore, transfer from image captioning or visual grounded question answering tasks is possible.

We were not able to observe any benefit from pretraining on \texttt{boxing}.
The randomly initialized model seems to work better than any of the pretrained variants considered.
We hypothesise that this is caused by the game's input images being visually very distinct from the other data, suggesting transfer is difficult.
We discuss this Atari challenge further in our related work section.

\subsection{Fine-tuning on Robotic Stacking Tasks}
\label{sec:skill_gen}
Section~\ref{robotics_zeroshot} demonstrates that the base {Gato}{} capable of a diverse array of tasks can perform competitively on the RGB Stacking Skill Generalization benchmark.
In this section, we would like to answer the following question:
\emph{How does our agent improve on robotics tasks when allowed to fine-tune similarly to how we fine-tune on new tasks in Section~\ref{sec:ood_results}?}
We consider different model sizes and analyse the impact of pretraining datasets on the Skill Generalization benchmark, as well as a novel out of distribution task. Further analysis of fine-tuning with dataset ablations is in Appendix \ref{sec:sim_ablations_appendix}.

\subsubsection*{Skill Generalization}

First, we would like to show that fine-tuning on object-specific data, similarly to what was done by \citet{lee2022spend}, is beneficial.
Therefore, we fine-tuned {Gato}{} separately on five subsets of demonstrations from the \textit{test} dataset.
Each subset was obtained by random partitioning of a test dataset consisting of demonstrations gathered by a generalist sim-to-real agent stacking real test objects.
We consider this setting, which is comparable to the fine-tuning baselines on RGB stacking tasks from~\citep{lee2022spend}; and use the 5k dataset that their behavior cloning 5k results are obtained with.
To best match their experiments, we change our return filtering scheme during training: instead of using only successful stacks, we condition on the normalized return of the episode.

Figure \ref{fig:skill_generalization} compares the success rate of {Gato}{} across different fine-tuning data regimes to the sim-to-real expert and a Critic-Regularized Regression (CRR)~\citep{wang2020critic} agent trained on 35k episodes of all test triplets.
{Gato}{}, in both reality and simulation (red curves on the left and right figure, respectively), recovers the expert's performance with only 10 episodes, and peaks at 100 or 1000 episodes of fine-tuning data, where it exceeds the expert.
After this point (at 5000), performance degrades slightly but does not drop far below the expert's performance.

\begin{figure*}[t]
    \centering
    \begin{tabular}{p{0.47\linewidth} p{0.47\linewidth}}
      \vspace{0pt} \includegraphics[width=1\linewidth]{figures/skill_gen_real_fewshot} &
      \vspace{0pt} \includegraphics[width=1\linewidth]{figures/skill_gen_sim_ablations}
    \end{tabular}
    \caption{
    {\bf Robotics fine-tuning results.}
    Left: Comparison of real robot Skill Generalization success rate averaged across test triplets for {Gato}{},
    expert, and CRR trained on 35k expert episodes (upper bound).
    Right: Comparison of simulated robot Skill Generalization success rate averaged across test triplets for a series of ablations on the number of parameters, including scores for expert and a BC baseline trained on 5k episodes.
    }
    \label{fig:skill_generalization}
\end{figure*}

\subsubsection*{Fine-tuning and Model Size}

To better understand the benefit of large models for few-shot adaptation in robotics domains, we conducted an ablation on model parameter size.
This section focuses on in-simulation evaluation.
Figure \ref{fig:skill_generalization} compares the full \texttt{1.18B} parameter {Gato}{} with the smaller \texttt{364M} and \texttt{79M} parameter variants for varying amounts of fine-tuning data.
Although the \texttt{364M} model overfits on one episode, causing performance to drop, there is a clear trend towards better adaptation with fewer episodes as the number of parameters is scaled up.
The \texttt{79M} model performs clearly worse than its bigger counterparts.
The results suggest that the model's greater capacity allows the model to use representations learned from the diverse training data at test time.

\subsubsection*{Adaptation to Perceptual Variations}

While the Skill Generalization task is an effective benchmark for motor Skill Generalization to shape variations, it does not test the agent's ability to adapt to perceptual variations and permutations in the objective specification.
To further evaluate {Gato}{}'s generalization capabilities, we devised a new task in the RGB stacking benchmark where the goal is to stack the blue object on the green object, for test triplet 1 (see Figure~\ref{fig:stack_strike_cage}).
First, we used a 3D mouse to collect 500 demonstrations of this task on the real robot, for a total of 2 hours and 45 minutes of demonstration data, and fine-tuned {Gato}{} on these episodes.
Notably, all of the simulated and real robotics data in the pretraining set shows the robot successfully stacking the red object on the blue object, and the data does not include the object shapes in the test set.
We found that additionally adding simulated demonstrations of the stack blue on green task to the fine-tuning dataset improved performance, and 10\% was an ideal sampling ratio for this data.
\begin{figure*}
    \centering
    \includegraphics[width=0.98\linewidth]{figures/real_robot_blue_on_green}
    \caption{
    {\bf Comparing training/test task goal variations.}
    Top: the standard ``stack red on blue'' task tested in the Skill Generalization benchmark. Bottom: the novel ``stack blue on green'' task demonstrating {Gato}{}'s out of distribution adaptation to perceptual variations.
    \label{fig:real_robot_blue_on_green}}
\end{figure*}

We achieved a final 60\% success rate after evaluating fine-tuned {Gato}{} on the real robot, while a BC baseline trained from scratch on the blue-on-green data achieved only 0.5\% success (1/200 episodes).
Qualitatively, the BC baseline would consistently move towards the blue object and occasionally pick it up and place it on top of the green object, but a full, stable stack was almost never achieved.

\subsection{Robotics: Skill Mastery}
\label{sec:skill_mastery}
Similarly to the Skill Generalization challenge discussed in Section~\ref{robotics_zeroshot}, the Skill Mastery challenge consists in training a robotic arm to stack blocks of different shapes.
However, the Skill Mastery allows the agent to train on data involving the object shapes used for evaluation, i.e. the \emph{test} set in Skill Generalization becomes a part of the Skill Mastery \emph{training} set.
Thus, this challenge serves to measure {Gato}{}'s performance on in-distribution tasks (possibly with initial conditions not seen in the training demonstrations).
Our Skill Mastery results use an earlier version of the {Gato}{} architecture described in Appendix \ref{sec:skill_mastery_architecture}, with no fine-tuning.

\begin{table*}[t]
    \caption{{\bf Real robot Skill Mastery results.} {Gato}{} is competitive with the filtered BC baseline. \label{table:skill_mastery}}
    \centering
    \sc
    \scalebox{0.85}{
    \begin{tabular}{l|r|r|r|r|r|r}
    \toprule
    Agent                        & Group 1         & Group 2         & Group 3         & Group 4        & Group 5         & Average \\
    \midrule
    {Gato}{}                     & 58\%            & 57.6\%          & \textbf{78.5}\% & \textbf{89} \% & \textbf{95.1}\% & \textbf{75.6}\% \\
    BC-IMP~\citep{lee2021beyond} & \textbf{75.6}\% & \textbf{60.8}\% & 70.8\%          & 87.8\%         & 78.3\%          & 74.6\% \\
    \bottomrule
    \end{tabular}
    }
\end{table*}

Table~\ref{table:skill_mastery} compares the group-wise success percentage and the average success across object groups for {Gato}{} and the established BC-IMP baseline.
{Gato}{} exceeds or closely matches BC-IMP's performance on all but one training triplet.

\subsection{Specialist single-domain multi-task agents}\label{sec:specialist_agents}

In this section we show results obtained with two specialist (rather than generalist) agents.
Both of them were trained on data from a single domain only and rolled out 500 times for each training task without any per-task fine-tuning.

\subsubsection*{{Meta-World}}

The first agent uses the smallest architecture introduced in Section~\ref{sec:scaling_law_analysis}, i.e. \texttt{79M} parameters, and is trained on all 50 {Meta-World}{} tasks.
While {Gato}{} has access to the state of the MuJoCo physics engine and unlimited task seeds, the agent presented here has no access to any extra features or tasks and uses the canonical API as in~\citep{yu2020meta}.
This experiment is to show that the architecture proposed in our paper can be used to obtain state-of-the-art agents also at small scale.
The training procedure was to train single-task MPO \citep{abdolmaleki2018maximum} experts on each of the MT-50 tasks individually, recording the trajectories produced while training.
This experience is then combined, or distilled, into a single agent, which achieves 96.6\% success rate averaged over all 50 tasks. To the best of our knowledge there is no single multi-task agent that approaches 100\% success rate.
See Table~\ref{tab:metaworld} in the supplementary material (Section~\ref{sec:mt50_results}) for the full list of tasks and corresponding success rates of our agent.

\subsubsection*{{ALE Atari}{}}

We also trained a specialist agent on all 51 {ALE Atari}{} tasks.
As the Atari domain is much more challenging than {Meta-World}{}, we used the {Gato}{} architecture with \texttt{1.18B} parameters.

The resulting agent performs better than the average human for 44 games (see Section~\ref{sec:simulated_control_task} for details on our evaluation and scoring).
We want to note that the performance of online experts used to generate training data for the other 7 games were also below the average human.
Hence, the specialist Atari agent achieved better than human performance for all games where data contained super-human episodes.

The specialist Atari agent outperforms our generalist agent {Gato}{}, which achieved super-human performance on 23 games.
It suggests that scaling {Gato}{} may result in even better performance.
We, however, purposely restricted {Gato}{}'s size such that it can be run in real-time on the real robot.

\section{Related Work}
\label{sec:related}

The most closely related architectures to that of {Gato}{} are Decision Transformers~\citep{chen2021decision,reid2022can,zheng2022online,furuta2021generalized} and Trajectory Transformer~\citep{janner2021offline}, which showed the usefulness of highly generic LM-like architectures for a variety of control problems.
~{Gato}{} also uses an LM-like architecture for control, but with design differences chosen to support multi-modality, multi-embodiment, large scale and general purpose deployment.
Pix2Seq~\citep{chen2021pix2seq} also uses an LM-based architecture for object detection.
Perceiver IO~\citep{jaegle2021perceiver} uses a transformer-derived architecture specialized for very long sequences, to model any modality as a sequence of bytes.
This and similar architectures could be used to expand the range of modalities supported by future generalist models.

{Gato}{} was inspired by works such as GPT-3~\citep{brown2020language} and Gopher~\citep{rae2021scaling}, pushing the limits of generalist language models; and more recently the  Flamingo~\citep{Alayrac2022FlamingoAV} generalist visual language model.
\citet{chowdhery2022palm} developed the 540B parameter Pathways Language Model (PalM) explicitly as a generalist few-shot learner for hundreds of text tasks.
Future work should consider how to unify these text capabilities into one fully generalist agent that can also act in real time in the real world, in diverse environments and embodiments.

{Gato}{} also takes inspiration from recent works on multi-embodiment continuous control.
\citet{huang2020one} used message passing graph networks to build a single locomotor controller for many simulated 2D walker variants.
\citet{kurin2020my} showed that transformers can outperform graph based approaches for incompatible (i.e.\ varying embodiment) control, despite not encoding any morphological inductive biases.
\citet{devin2017learning} learn a modular policy for multi-task and multi-robot transfer in simulated 2D manipulation environments.
\citet{chen2018hardware} train a universal policy conditioned on a vector representation of robot hardware, showing successful transfer both to simulated held out robot arms, and to a real world sawyer robot arm.

A variety of earlier generalist models have been developed that, like {Gato}{}, operate across highly distinct domains and modalities.
NPI~\citep{reed2016neural} trained a single LSTM~\citep{hochreiter1997long} to execute diverse programs such as sorting an array and adding two numbers, such that the network is able to generalize to larger problem instances than those seen during training.
\citet{kaiser2017one} developed the MultiModel that trains jointly on 8 distinct speech, image and text processing tasks including classification, image captioning and translation.
Modality-specific encoders were used to process text, images, audio and categorical data, while the rest of the network parameters are shared across tasks.
\citet{schmidhuber2018one} proposed ``\emph{one big net for everything}'', describing a method for the incremental training of an increasingly general problem solver.
\citet{keskar2019ctrl} proposed controllable multi-task language models that can be directed according to language domain, subdomain, entities, relationships
between entities, dates, and task-specific behavior.

In this discussion, it is important to distinguish between one single multi-task network architecture versus one single neural network with the same weights for all tasks.
Several poplar RL agents achieve good multi-task RL results within single domains such as Atari57 and DMLab~\citep{espeholt2018impala,song2019v,hessel2019multi}.
However, it is much more common to use the same policy architecture and hyper-parameters across tasks, but the policy parameters are different in each task \citep{mnih2015human,tassa2018deepmind}.
This is also true of state-of-the-art RL methods applied to board games \citep{schrittwieser2020mastering}. Moreover, this choice has been adopted by off-line RL benchmarks \citep{gulcehre2020rl,fu2020d4rl} and recent works on large sequence neural networks for control, including decision transformers \citep{chen2021decision,reid2022can,zheng2022online} and the Trajectory Transformer of \cite{janner2021offline}. In contrast, in this work we learn a single network with the same weights across a diverse set of tasks.

Recent position papers advocate for highly generalist models, notably \cite{schmidhuber2018one} proposing one big net for everything, and \cite{bommasani2021opportunities} on foundation models.
However, to our knowledge there has not yet been reported a single generalist trained on hundreds of vision, language and control tasks using modern transformer networks at scale.

``Single-brain''-style models have interesting connections to neuroscience.
\cite{mountcastle1978organizing} famously stated that ``\emph{the processing function of neocortical modules is qualitatively similar in all neocortical regions.
Put shortly, there is nothing intrinsically motor about the motor cortex, nor sensory about the sensory cortex}''.
Mountcastle found that columns of neurons in the cortex behave similarly whether associated with vision, hearing or motor control.
This has motivated arguments that we may only need one algorithm or model to build intelligence \citep{hawkins2004intelligence}.

Sensory substitution provides another argument for a single model \citep{bach2003sensory}. For example, it is possible to build tactile visual aids for blind people as follows. The signal captured by a camera can be sent via an electrode array on the tongue to the brain. The visual cortex learns to process and interpret these tactile signals, endowing the person with some form of ``vision''. Suggesting that, no matter the type of input signal, the same network can process it to useful effect.

Our work is based on deep autoregressive models, which have a long history and can be found in generative models of text, images, video and audio. Combining autoregressive generation with transformers \citep{vaswani2017attention,devlin2018bert} has been of enormous impact in language modelling \citep{brown2020language,rae2021scaling}, protein folding \citep{jumper2021highly}, vision-language models \citep{tsimpoukelli2021multimodal, wang2021simvlm,Alayrac2022FlamingoAV}, code generation \citep{chen2021evaluating,li2022competition}, dialogue systems with retrieval capabilities \citep{nakano2021webgpt,thoppilan2022lamda}, speech recognition~\citep{pratap2020massively}, neural machine translation~\citep{johnson2019massively} and more \citep{bommasani2021opportunities}. Recently researchers have explored task decomposition and grounding with language models \citep{huang2022language,ahn2022can}.

\cite{li2022pre} construct a control architecture, consisting of a sequence tokenizer, a pretrained language model and a task-specific feed-forward network. They apply it to VirtualHome and BabyAI tasks, and find that the inclusion of the pretrained language model improves generalisation to novel tasks. Similarly, \cite{parisi2022unsurprising} demonstrate that vision models pretrained with self-supervised learning, especially crop segmentations and momentum contrast \citep{he2020momentum}, can be effectively incorporated into control policies.

As mentioned earlier, transfer in Atari is challenging.
\cite{rusu2016progressive} researched transfer between randomly selected Atari games. They found that Atari is a difficult domain for transfer because of pronounced differences in the visuals, controls and strategy among the different games.
Further difficulties that arise when applying behaviour cloning to video games like Atari are discussed by \cite{kanervisto2020benchmarking}.

There has been great recent interest in data-driven robotics~\citep{cabi2019scaling,chen2021learning}. However,
\cite{bommasani2021opportunities} note that in robotics ``\emph{the key stumbling block is collecting the right data. Unlike language and vision data, robotics data is neither plentiful nor representative of a sufficiently diverse array of
embodiments, tasks, and environments}''. Moreover, every time we update the hardware in a robotics lab, we need to collect new data and retrain. We argue that this is precisely why we need a generalist agent that can adapt to new embodiments and learn new tasks with few data.

Generating actions using an autoregressive model can lead to causal ``self-delusion'' biases when there are confounding variables \citep{ortega2021shaking}. For example, sampling actions can condition the model to solve the wrong task when multiple tasks share similar observation and actions specifications. As explained in Section~\ref{sec:model}, we use prompt engineering in ambiguous tasks, conditioning our model on a successful demonstration. This screens off confounding variables, reducing self-delusions. Another solution which we did not explore in this work is to use counterfactual teaching, where we train a model online using instantaneous expert feedback. We leave this for future investigation.

\section{Broader Impact}
\label{sec:safety}

Although generalist agents are still only an emerging area of research, their potential impact on society calls for a thorough interdisciplinary analysis of their risks and benefits.
For the sake of transparency, we document the intended use cases of {Gato}{} in the model card in Appendix \ref{app:model_card}.
However, the tools for mitigating harms of generalist agents are relatively underdeveloped, and require further research before these agents are deployed.

Since our generalist agent can act as a vision-language model, it inherits similar concerns as discussed in~\citep{weidinger2021ethical,bommasani2021opportunities,rae2021scaling,Alayrac2022FlamingoAV}.
In addition, generalist agents can take actions in the the physical world; posing new challenges that may require novel mitigation strategies.
For example, physical embodiment could lead to users anthropomorphizing the agent, leading to misplaced trust in the case of a malfunctioning system, or be exploitable by bad actors.
Additionally, while cross-domain knowledge transfer is often a goal in ML research, it could create unexpected and undesired outcomes if certain behaviors (e.g. arcade game fighting) are transferred to the wrong context.
The ethics and safety considerations of knowledge transfer may require substantial new research as generalist systems advance.

Technical AGI safety~\citep{bostrom2017superintelligence} may also become more challenging when considering generalist agents that operate in many embodiments.
For this reason, preference learning, uncertainty modeling and value alignment~\citep{russell2019human} are especially important for the design of human-compatible generalist agents.
It may be possible to extend some of the value alignment approaches for language~\citep{ouyang2022training, kenton2021alignment} to generalist agents.
However, even as technical solutions are developed for value alignment, generalist systems could still have negative societal impacts even with the intervention of well-intentioned designers, due to unforeseen circumstances or limited oversight~\citep{amodei2016concrete}.
This limitation underscores the need for a careful design and a deployment process that incorporates multiple disciplines and viewpoints.

Understanding how the models process information, and any emergent capabilities, requires significant experimentation.
External retrieval~\citep{borgeaud2021improving,menick2022teaching,nakano2021webgpt,thoppilan2022lamda} has been shown to improve both interpretability and performance, and hence should be considered in future designs of generalist agents.

Although still at the proof-of-concept stage, the recent progress in generalist models suggests that safety researchers, ethicists, and most importantly, the general public, should consider their risks and benefits.
We are not currently deploying {Gato}{} to any users, and so anticipate no immediate societal impact.
However, given their potential impact, generalist models should be developed thoughtfully and deployed in a way that promotes the health and vitality of humanity.

\section{Conclusions}
\label{sec:conclusions}

Transformer sequence models are effective as multi-task multi-embodiment policies, including for real-world text, vision and robotics tasks.
They show promise as well in few-shot out-of-distribution task learning.
In the future, such models could be used as a default starting point via prompting or fine-tuning to learn new behaviors, rather than training  from scratch.

Given scaling law trends, the performance across all tasks including dialogue will increase with scale in parameters, data and compute.
Better hardware and network architectures will allow training bigger models while maintaining real-time robot control capability.
By scaling up and iterating on this same basic approach, we can build a useful general-purpose agent.

\newpage
\section*{Acknowledgements}
We would like to thank Dan Horgan, Manuel Kroiss, Mantas Pajarskas, and Thibault Sottiaux for their help with data storage infrastructure; Jean-Baptiste~Lespiau and Fan Yang for help on concurrent evaluation; Joel Veness for advising on the model design; Koray Kavukcuoglu for helping inspire the project and facilitating feedback; Tom~Erez for advising on the agent design and task selection for continuous control; Igor~Babuschkin for helping code the initial prototype;  Jack~Rae for advising on the transformer language model codebase; Thomas~Lampe for building robot infrastructure and advising on real robotics experiments; Boxi~Wu for input on ethics and safety considerations; Pedro A. Ortega for advice in regard to causality and self-delusion biases.
\section*{Author Contributions}
\noindent \textbf{Scott Reed} developed the project concept, wrote the initial prototype, and led the project overall. \\
\noindent \textbf{Konrad \.Zo\l{}na} led architecture development for vision and text, built infrastructure for tokenization and prompting, and contributed heavily to overall agent development and evaluation. \\
\noindent \textbf{Emilio Parisotto} led work on optimizing the transformer architecture, ran the largest number of experiments, and analyzed scaling law properties and in-distribution agent performance. \\
\noindent \textbf{Sergio Gómez Colmenarejo} was the technical lead, responsible for creating a scalable data loader and evaluator supporting hundreds of tasks at once, and for the initial robot integration with {Gato}{}. \\
\noindent \textbf{Alexander Novikov} developed the model including the sampler for the initial prototype, carried out experiments focusing on robotics, and created visualizations. \\
\noindent \textbf{Gabriel Barth-Maron} built scalable storage infrastructure to provide {Gato}{} with SoTA-level agent experience in Atari and other domains. \\
\noindent \textbf{Mai Giménez} conducted large scale agent data collection, built substantial data loading infrastructure, and integrated large scale visual-language datasets into the training of {Gato}{}. \\
\noindent \textbf{Yury Sulsky} contributed broadly to the {Gato}{} codebase including a bespoke distributed training sequence loader, and led the development of benchmarks for out-of-distribution generalization, and the training of competitive baseline agents. \\
\noindent \textbf{Jackie Kay} supported physical robotics infrastructure, conducted numerous evaluations and experiments to analyze the generalization properties of {Gato}{}, and contemplated broader ethical impact. \\
\noindent \textbf{Jost Tobias Springenberg} guided {Gato}{}'s deployment to the physical robot, provided strong existing baselines for block stacking, and advised on model development and experimental design. \\
\noindent \textbf{Tom Eccles} developed the {Gato}{} dialogue and image captioning demonstrations, allowing users to easily probe the vision and language capacities of agents in development. \\
\noindent \textbf{Jake Bruce} contributed to agent design as well as control datasets and environments with randomized physics and morphology variations. \\
\noindent \textbf{Ali Razavi} helped in exploring vision architectures. \\
\noindent \textbf{Ashley Edwards} contributed to the first prototype of {Gato}{} that worked on Atari, in addition to exploring alternative network architectures and training objectives. \\
\noindent \textbf{Nicolas Heess} advised on agent design, experiment design and task selection, especially for continuous control applications.  \\
\noindent \textbf{Yutian Chen} advised on model design and experiments, and provided feedback in regular meetings. \\
\noindent \textbf{Raia Hadsell} advised on the design and planning of robotics efforts.\\
\noindent \textbf{Oriol Vinyals} advised on all aspects of the project, especially model architecture, training strategies and benchmark design. \\
\noindent \textbf{Mahyar Bordbar} was the primary project manager; eliciting key goals, tracking progress, facilitating presentations and feedback, and coordinating resource planning. \\
\noindent \textbf{Nando de Freitas} oversaw the project from its inception.

\bibliographystyle{abbrvnat}
\nobibliography*


\section{Introduction}
COVID-19 was declared as a global health pandemic by the WHO, and it can be very well noticed that social media has played a very significant role much before the spread of the virus. As various countries around the world went into lockdown for long periods, it was noticed that social media became a very important platform for people to share information, post their views and emotions in short amount of texts. It has been seen that a study of these texts have resulted in various novel applications which are not only limited to, political opinion detection as seen in \cite{Maynard2012}, stock market monitoring as seen in \cite{Abdullah2013}, and analysing user reviews of a product as seen in \cite{DBLP:journals/corr/abs-1904-04096}. The wide usage of figurative language like hashtags, emotes, abbreviations, and slangs makes it even more difficult to comprehend the text being used on these social platforms, making Natural Language Processing a more challenging task.
It has been seen that techniques like Latent Topic Clustering \cite{Lee2018}, Cultivating deep decision trees \cite{Ignatov2017}, performing Fine grained sentiment analysis \cite{DBLP:journals/corr/abs-1904-04096}, and ensemble techniques \cite{Balikas2017} have given competitive results in language understanding tasks in NLP. In this paper we present a similar Deep Learning technique which competed in AAAI SHARED TASK @ CONSTRAINT 2021 'COVID 19 Fake News Detection in English' and 'Hostile Post Detection in Hindi'. The overview of above Shared Task has been explain in this\cite{patwa2021overview}. We explored differentiated layer training technique, where different sections of the layers were frozen and unfrozen during the training. This was combined with the training procedure as discussed in ULMFiT \cite{howard2018universal}. The complete training procedure is explained in the coming sections. The paper is divided into sections, the next section discusses the task at hand, details of the dataset provided and the preprocessing steps that were taken.

\section{Overview}
This section contains details of the given task, the dataset provided, and the preprocessing steps taken to clean the dataset.
\subsection{Task Description and Dataset}
\textbf{Task Definition Sub-task 1}  This subtask focuses on the detection of COVID19-related fake news in English. The sources of data are various social-media platforms such as Twitter, Facebook, Instagram, etc. Given a social media post, the objective of the shared task is to classify it into either fake or real news.
The dataset provided for the task is discussed in \cite{patwa2020fighting}. The dataset contains a total of 6420 labeled tweets for training, 2140 labeled tweets for validation and 2140 unlabeled tweets were given during the test phase. The complete class distribution for the dataset is shown in Fig. 1 (a).
The image shows that the distribution of the classes was almost balanced, hence no under-sampling or over-sampling techniques were used during the preprocessing to balance the dataset.\\
\textbf{Task Definition Sub-task 2} This subtask focuses on a variety of hostile posts in Hindi Devanagari script collected from Twitter and Facebook. The set of valid categories are fake news, hate speech, offensive, defamation, and non-hostile posts. It is a multi-label multi-class classification problem where each post can belong to one or more of these hostile classes.
The dataset for this  sub-task covers four hostility dimentions: fake news, hate speech, offensive, and defamation posts, along with a non-hostile label. Dataset is multi labelled due to overlap of different hostility classes. The dataset is further described here \cite{bhardwaj2020hostility} .
The dataset provided 5728 labeled posts for training, 811 labeled post for validation, and 1653 unlabeled for test phase. The labeled distribution for train set is shown in Fig. 1 (b).
\begin{figure}[h]
\centering
\includegraphics[scale=0.35]{templates/images/englishDatasetDistribution.png}
\includegraphics[scale=0.35]{templates/images/Label_distribution.png}
\caption{Label distribution for training dataset (a)"COVID19 Fake News Detection" (b)"Hostile Post Detection in Hindi"}
\end{figure}
\subsection{Preprocesing}
The various steps used during the preprocessing of the dataset are mentioned below.
\subsubsection{Replacing Emojis}  Since tweets from twitter are mostly accompanied with graphics (emojis) which are supposed to help a user express his thoughts, our first task was to replace these emojis with their text counterpart. While a machine cannot understand the emoji, it's text counterpart can easily be interpreted as discussed in \cite{2016} and \cite{Azhan2020}. We used the emoji library\footnote{https://pypi.org/project/emoji/} for converting emojis to their English textual meanings. For the Hindi dataset we created our own library 'Emot Hindi' \footnote{https://github.com/ahmadkhan242/emot\_hindi} similar to the emoji library discussed above which contains emojis and their Hindi textual meanings. This was a common step for both sub-tasks. A few examples of sample emojis and their meanings are shown in Fig. 2.
\begin{figure}[h]
\begin{center}
\includegraphics[]{templates/images/hindiEmojiExample.jpg}
\includegraphics[]{templates/images/emojidemoji.png}
\caption{Example: Emoji and text counterpart (a)Emoji to Hindi (b) Emoji to English}
\end{center}
\end{figure}
\subsubsection{Addressing hashtags} Hashtags are word or phases preceded by a hash sign '\#' which are used to identify texts regarding a specific topic of discussion. It has been seen that the attached hashtags to a post or tweet tell what the text is relevant to, this has been discussed in \cite{Azhan2020} and \cite{Alfina2017}. For the given tweets a white space was added between the hash symbol and the following word for the model to comprehend it easily. This was also a common step for both sub-tasks.
 \subsubsection{Adding special tokens} We replaced specific parts of the text with special tokens as discussed in the fastai library\footnote{https://docs.fast.ai/text.core.html}. The special tokens and their usage are mentioned in the list below.
\begin{itemize}
    \item \textbf\{TK\_REP\} This token was used to replace characters that were occurring more than thrice repeatedly. This special token was used for both sub-tasks. For example 'This was a verrrryyyyyyy tiring trip' will be replaced with 'This was a ve\{TK\_WREP\} 4 r \{TK\_WREP\} 7 y tiring trip'.
    \item \textbf{\{TK\_WREP\}} This token was used to replace words occuring three or more times consecutively. This special token was used for both sub-tasks. For example 'This is a very very very very very sad news' will be replaced with 'This is a \{TK\_WRPEP\} 5 very sad news'.
    \item \textbf{\{TK\_UP\}} This token was used to replace words using all caps. Since the Devnagri script used for Hindi has no uppercase alphabetsm this special token was used for the English sub-task only. For example 'I AM SHOUTING' becomes '\{TK\_UP\} i \{TK\_UP\} am \{TK\_UP\} shouting'.
    \item \textbf{\{TK\_MAJ\}} Used to replace characters in words which started with an upper case except for when it is the starting of a sentence. Again, this special token was used for the English subtask only. For example, 'I am Kaleen Bhaiya' becomes 'i am \{TK\_MAJ\} kaleen \{TK\_MAJ\} bhaiya'.
\end{itemize}
\subsubsection{Normalization} These steps included removing extra spacing between words, correcting hmtl format from texts if any, adding white space between special characters and alphabets, and replacing texts with lower case. The above preprocessing steps were taken for both subtasks.
\subsubsection{Tokenization} Once the preprocessing of the dataset was complete, we performed tokenization. For the ULMFiT training the ULMFiT tokenizer was used, similarly the text for the customized RoBERTa model was tokenized using RoBERTa tokenizer, and for the Random Forest Classifier (English and Hindi sub-task) and Linear Regression (Hindi sub-task) the text was tokenized using the nltk library for both the languages.

\section{Model Description}
Next, we provide an in detail description of the training strategies that were used to achieve the results. The test results obtained using each technique is mentioned in the results section. Each technique is discussed in the coming sub-sections.
\subsection{Layer Differentiated ULMFiT Training}
As discussed in \cite{howard2018universal} inductive training has shown incredible performance in Computer Vision tasks where the model is first pretrained on large datasets like ImageNet, MS-COCO, and others. The same idea was implemented during the training of the ULMFiT model, only it was modified using a pretrained language model. Traditional transfer learning language models used to pretrain the language model on a relatively larger dataset, this language model was then used to create the classifier model which will again pretrain on the large dataset, at the final step the classifier model was fine-tuned on the target dataset. ULMFiT introduced LM Pretraining and Fine-tuning to make sure that the language model used to pretrain the classifier consisted of extracted features from the target domain. This part of the training procedure is exactly same as discussed in \cite{howard2018universal}. The image below shows the training of both Language model and classifier as in \cite{howard2018universal}.
\begin{figure}[ht]
\centering
\includegraphics[scale=0.22]{templates/images/ulmfitclassifier.jpg}
\caption{ULMFiT Traditional Training}
\end{figure}
We introduced a layer differentiated training procedure, which gradually unfreezed the layers for training them. This differentiated training procedure was implemented for training both, the language model and the classifier model for both of the sub-tasks.
Fig. 4 shows a plot between the training and validation losses as the training progressed for the English sub-task. The graph shows a spike after every 100 batches which is then followed by a sharp decline. These spikes are the parts where the layers were unfreezed. As the layers were unfreezed, the untrained layers led to an increase in the training loss, which gradually decreased as the training progressed. This also made sure that the final layers were trained longer as compared to initial layers so that the initial layers don\'t start overfitting and the model doesn\'t drops out any important features.
\begin{figure}[h]
\centering
\includegraphics[scale=0.75]{templates/images/trainingLossesULMFiTEnglish.png}
\caption{Loss vs Batches Progressed: LaDiff ULMFiT}
\end{figure}
This concludes our discussion for the LaDiff ULMFiT training. We now move forward with our next technique.
\subsection{Customized RoBERTa}
RoBERTa\cite{DBLP:journals/corr/abs-1907-11692} is a robustly optimized pretraining approach designed for BERT\cite{DBLP:journals/corr/abs-1810-04805}. BERT stands for Bidirectional Encoder Representations from Transformers, and it introduced the use of transformers for language training tasks. RoBERTa aimed at improvising the training methodology as introduced in BERT using dynamic masking, provising full sentences rather than using next sentence prediction, training with a large number of batches having small sizes and a larger byte-level Byte-Pair Encoding. For our customized model, we used the RoBERTa uncased model pre-trained on various larger twitter dataset. We then added a few customized layers to the model. This training procedure was implemented on the English sub-task only.
\subsection{Random Forest Classifiers and Logistic Regression}
While the above two approaches have shown how language modelling and using text transformers give exceptionally high performance, our idea behind trying these approach was to understand where do simple language classifiers lack as compared to deep neural networks. While the baseline results as presented in the English dataset paper \cite{patwa2020fighting} and Hindi dataset paper \cite{bhardwaj2020hostility} use an SVM Classifier, we decided to use various Machine Learning techniques, and submit the one which has the highest score in the validation set. In our case, we achieved the best results using a Random Forest Classifier, having n\_estimators set as 1000, min\_samples\_split as 15 and a random\_state of 42. The same classifier hyper-parameters were passed to both of the classification models and trained separately.
The Logistic Regression Classifier was used only for the Hindi sub-task.
This brings an end to our discussion for the various approaches used. We now move forward to the results obtained and compare them with the available baseline results\cite{patwa2020fighting} \cite{bhardwaj2020hostility}.

\section{Results}
We first present the results obtained for the English sub-task "COVID19  Fake  News  Detection  in English".The table given below gives the accuracy, precision, recall and f1-score of our approaches and compares them with the available baseline results.
\begin{table}[]
\begin{tabular}{|l|r|r|r|r|}
\hline
\textbf{Model} & \multicolumn{1}{l|}{\textbf{Accuracy}} & \multicolumn{1}{l|}{\textbf{Precision}} & \multicolumn{1}{l|}{\textbf{Recall}} & \multicolumn{1}{l|}{\textbf{f1-score}} \\ \hline
\textbf{LaDiff ULMFiT} & \textbf{0.96728972} & \textbf{0.967908486} & \textbf{0.96728972} & \textbf{0.967324832} \\ \hline
\textbf{Baseline Model} & 93.46 & 93.46 & 93.46 & 93.46 \\ \hline
\textbf{Customized RoBERTa} & 0.929906542 & 0.929906542 & 0.929906542 & 0.929906542 \\ \hline
\textbf{Random Forest Classifier} & 0.91728972 & 0.917382499 & 0.91728972 & 0.917311831 \\ \hline
\end{tabular}
\caption{Comparison Results on Test Set: LaDiff ULMFiT vs Customized RoBERTa vs Random Forest Classifier vs Baseline Model- Sub-task 1}
\end{table}
Our best approach, LaDiff-ULMFiT ranked 61st out of 167 submissions on the final leaderboard.\\
We now present our results for the Hindi sub-task "Hostile Post Detection in Hindi" shown in Table 2.
\begin{table}[]

\begin{tabular}{|m{10em}|m{4.5em}|m{6em}|m{3.5em}|m{3.5em}|m{5em}|m{5em}|}
\hline
\textbf{Model} &
  \textbf{Coarse Grained Hostility f1 Score} &
  \textbf{Defamation f1 Score} &
  \textbf{Fake f1 Score} &
  \textbf{Hate f1 Score} &
  \textbf{Offensive f1 Score} &
  \textbf{Weighted Fine Grained f1 Score} \\ \hline
\textbf{LaDiff ULMFiT}            & \textbf{90.87} & 27.31        & \textbf{73.83} & 44.93       & \textbf{51.39} & \textbf{0.53} \\ \hline
\textbf{Baseline Results}         & 84.11             & \textbf{43.57} & 68.15             & \textbf{47.49} & 41.98             &                   \\ \hline
\textbf{Logistic Regression}      & 76.56          & 24.8          & 54.71          & 40.65       & 40.58          & 42.74          \\ \hline
\textbf{Random Forest Classifier} & 76.56          & 24.8          & 54.71          & 40.65       & 40.58          & 42.74          \\ \hline
\end{tabular}

\caption{Comparison Results on Test Set: LaDiff ULMFiT vs Logistic Regression vs Random Forest Classifier vs Baseline Results- Sub-task 2}
\label{tab:my-table}
\end{table}
The results ranked 18th for the Coarse Grained f1 Score and 25th for the Fine Grained f1 Score. We now proceed with our conclusions.
\section{Conclusions}
From the achieved results as shown in Table 2, the following conclusions can be drawn:
\begin{itemize}
    \item Fine-tuned language model used with a simple classifier (LaDiff-ULMFiT) outperforms transformers used with sophisticated networks (Customized RoBERTa).
    \item The losses trend seen in Fig. 4 also signifies the fact that target domain fine tuned on a pre-trained model done at when trained at gradual steps leads to faster decrease in losses.
    \item We also conclude that, tweets containing hashtags and short texts can also be confidently classified using Machine Learning techniques.
\end{itemize}
Finally, we make all our approaches and their source codes completely available for the open source community, to reproduce the results and facilitate further experimentation in the field.

\bibliographystyle{plain}

\chapter*{Preface}
This textbook is intended for use by students of physics, physical
chemistry, and theoretical chemistry. The reader is presumed to have a
basic knowledge of atomic and quantum physics at the level provided, for
example, by the first few chapters in our book {\it The Physics of Atoms
and Quanta}. The student of physics will find here material which should
be included in the basic education of every physicist. This book should
furthermore allow students to acquire an appreciation of the breadth and
variety within the field of molecular physics and its future as a
fascinating area of research.

\vspace{1cm}
\begin{flushright}\noindent
June 2016\hfill Walter Olthoff\\
Program Chair\\
ECOOP 2016
\end{flushright}
\chapter*{Organization}
ECOOP 2016 is organized by the department of Computer Science, Univeristy
of \AA rhus and AITO (association Internationa pour les Technologie
Object) in cooperation with ACM/SIGPLAN.
\section*{Executive Commitee}
\begin{tabular}{@{}p{5cm}@{}p{7.2cm}@{}}
Conference Chair:&Ole Lehrmann Madsen (\AA rhus University, DK)\\
Program Chair:   &Walter Olthoff (DFKI GmbH, Germany)\\
Organizing Chair:&J\o rgen Lindskov Knudsen (\AA rhus University, DK)\\
Tutorials:&Birger M\o ller-Pedersen\hfil\break
(Norwegian Computing Center, Norway)\\
Workshops:&Eric Jul (University of Kopenhagen, Denmark)\\
Panels:&Boris Magnusson (Lund University, Sweden)\\
Exhibition:&Elmer Sandvad (\AA rhus University, DK)\\
Demonstrations:&Kurt N\o rdmark (\AA rhus University, DK)
\end{tabular}
\section*{Program Commitee}
\begin{tabular}{@{}p{5cm}@{}p{7.2cm}@{}}
Conference Chair:&Ole Lehrmann Madsen (\AA rhus University, DK)\\
Program Chair:   &Walter Olthoff (DFKI GmbH, Germany)\\
Organizing Chair:&J\o rgen Lindskov Knudsen (\AA rhus University, DK)\\
Tutorials:&Birger M\o ller-Pedersen\hfil\break
(Norwegian Computing Center, Norway)\\
Workshops:&Eric Jul (University of Kopenhagen, Denmark)\\
Panels:&Boris Magnusson (Lund University, Sweden)\\
Exhibition:&Elmer Sandvad (\AA rhus University, DK)\\
Demonstrations:&Kurt N\o rdmark (\AA rhus University, DK)
\end{tabular}
\begin{multicols}{3}[\section*{Referees}]
V.~Andreev\\
B\"arwolff\\
E.~Barrelet\\
H.P.~Beck\\
G.~Bernardi\\
E.~Binder\\
P.C.~Bosetti\\
Braunschweig\\
F.W.~B\"usser\\
T.~Carli\\
A.B.~Clegg\\
G.~Cozzika\\
S.~Dagoret\\
Del~Buono\\
P.~Dingus\\
H.~Duhm\\
J.~Ebert\\
S.~Eichenberger\\
R.J.~Ellison\\
Feltesse\\
W.~Flauger\\
A.~Fomenko\\
G.~Franke\\
J.~Garvey\\
M.~Gennis\\
L.~Goerlich\\
P.~Goritchev\\
H.~Greif\\
E.M.~Hanlon\\
R.~Haydar\\
R.C.W.~Henderso\\
P.~Hill\\
H.~Hufnagel\\
A.~Jacholkowska\\
Johannsen\\
S.~Kasarian\\
I.R.~Kenyon\\
C.~Kleinwort\\
T.~K\"ohler\\
S.D.~Kolya\\
P.~Kostka\\
U.~Kr\"uger\\
J.~Kurzh\"ofer\\
M.P.J.~Landon\\
A.~Lebedev\\
Ch.~Ley\\
F.~Linsel\\
H.~Lohmand\\
Martin\\
S.~Masson\\
K.~Meier\\
C.A.~Meyer\\
S.~Mikocki\\
J.V.~Morris\\
B.~Naroska\\
Nguyen\\
U.~Obrock\\
G.D.~Patel\\
Ch.~Pichler\\
S.~Prell\\
F.~Raupach\\
V.~Riech\\
P.~Robmann\\
N.~Sahlmann\\
P.~Schleper\\
Sch\"oning\\
B.~Schwab\\
A.~Semenov\\
G.~Siegmon\\
J.R.~Smith\\
M.~Steenbock\\
U.~Straumann\\
C.~Thiebaux\\
P.~Van~Esch\\
from Yerevan Ph\\
L.R.~West\\
G.-G.~Winter\\
T.P.~Yiou\\
M.~Zimmer\end{multicols}
\section*{Sponsoring Institutions}
V. Meyer Inc., Reading, MA, USA\\
The Hofmann-International Company, San Louis Obispo, CA, USA\\
Kramer Industries, Heidelberg, Germany
\tableofcontents
\mainmatter
\part{Hamiltonian Mechanics}
\title{Hamiltonian Mechanics unter besonderer Ber\"ucksichtigung der
h\"ohereren Lehranstalten}
\titlerunning{Hamiltonian Mechanics}
\author{Ivar Ekeland\inst{1} \and Roger Temam\inst{2}
Jeffrey Dean \and David Grove \and Craig Chambers \and Kim~B.~Bruce \and
Elsa Bertino}
\authorrunning{Ivar Ekeland et al.}
\tocauthor{Ivar Ekeland, Roger Temam, Jeffrey Dean, David Grove,
Craig Chambers, Kim B. Bruce, and Elisa Bertino}
\index{Ekeland, I.}
\index{Temam, R.}
\index{Dean, J.}
\index{Grove, D.}
\index{Chambers, C.}
\index{Kim, B.}
\index{Bertino, E.}
\institute{Princeton University, Princeton NJ 08544, USA,\\
\email{I.Ekeland@princeton.edu},\\ WWW home page:
\texttt{http://users/\homedir iekeland/web/welcome.html}
\and
Universit\'{e} de Paris-Sud,
Laboratoire d'Analyse Num\'{e}rique, B\^{a}timent 425,\\
F-91405 Orsay Cedex, France}

\maketitle

\begin{abstract}
The abstract should summarize the contents of the paper
using at least 70 and at most 150 words. It will be set in 9-point
font size and be inset 1.0 cm from the right and left margins.
There will be two blank lines before and after the Abstract. \dots
\keywords{computational geometry, graph theory, Hamilton cycles}
\end{abstract}
\section{Fixed-Period Problems: The Sublinear Case}
With this chapter, the preliminaries are over, and we begin the search
for periodic solutions to Hamiltonian systems. All this will be done in
the convex case; that is, we shall study the boundary-value problem
\begin{eqnarray*}
  \dot{x}&=&JH' (t,x)\\
  x(0) &=& x(T)
\end{eqnarray*}
with $H(t,\cdot)$ a convex function of $x$, going to $+\infty$ when
$\left\|x\right\| \to \infty$.

\subsection{Autonomous Systems}
In this section, we will consider the case when the Hamiltonian $H(x)$
is autonomous. For the sake of simplicity, we shall also assume that it
is $C^{1}$.

We shall first consider the question of nontriviality, within the
general framework of
$\left(A_{\infty},B_{\infty}\right)$-subquadratic Hamiltonians. In
the second subsection, we shall look into the special case when $H$ is
$\left(0,b_{\infty}\right)$-subquadratic,
and we shall try to derive additional information.
\subsubsection{The General Case: Nontriviality.}
We assume that $H$ is
$\left(A_{\infty},B_{\infty}\right)$-sub\-qua\-dra\-tic at infinity,
for some constant symmetric matrices $A_{\infty}$ and $B_{\infty}$,
with $B_{\infty}-A_{\infty}$ positive definite. Set:
\begin{eqnarray}
\gamma :&=&{\rm smallest\ eigenvalue\ of}\ \ B_{\infty} - A_{\infty} \\
  \lambda : &=& {\rm largest\ negative\ eigenvalue\ of}\ \
  J \frac{d}{dt} +A_{\infty}\ .
\end{eqnarray}

Theorem~\ref{ghou:pre} tells us that if $\lambda +\gamma < 0$, the
boundary-value problem:
\begin{equation}
\begin{array}{rcl}
  \dot{x}&=&JH' (x)\\
  x(0)&=&x (T)
\end{array}
\end{equation}
has at least one solution
$\overline{x}$, which is found by minimizing the dual
action functional:
\begin{equation}
  \psi (u) = \int_{o}^{T} \left[\frac{1}{2}
  \left(\Lambda_{o}^{-1} u,u\right) + N^{\ast} (-u)\right] dt
\end{equation}
on the range of $\Lambda$, which is a subspace $R (\Lambda)_{L}^{2}$
with finite codimension. Here
\begin{equation}
  N(x) := H(x) - \frac{1}{2} \left(A_{\infty} x,x\right)
\end{equation}
is a convex function, and
\begin{equation}
  N(x) \le \frac{1}{2}
  \left(\left(B_{\infty} - A_{\infty}\right) x,x\right)
  + c\ \ \ \forall x\ .
\end{equation}

\begin{proposition}
Assume $H'(0)=0$ and $ H(0)=0$. Set:
\begin{equation}
  \delta := \liminf_{x\to 0} 2 N (x) \left\|x\right\|^{-2}\ .
  \label{eq:one}
\end{equation}

If $\gamma < - \lambda < \delta$,
the solution $\overline{u}$ is non-zero:
\begin{equation}
  \overline{x} (t) \ne 0\ \ \ \forall t\ .
\end{equation}
\end{proposition}
\begin{proof}
Condition (\ref{eq:one}) means that, for every
$\delta ' > \delta$, there is some $\varepsilon > 0$ such that
\begin{equation}
  \left\|x\right\| \le \varepsilon \Rightarrow N (x) \le
  \frac{\delta '}{2} \left\|x\right\|^{2}\ .
\end{equation}

It is an exercise in convex analysis, into which we shall not go, to
show that this implies that there is an $\eta > 0$ such that
\begin{equation}
  f\left\|x\right\| \le \eta
  \Rightarrow N^{\ast} (y) \le \frac{1}{2\delta '}
  \left\|y\right\|^{2}\ .
  \label{eq:two}
\end{equation}

\begin{figure}
\vspace{2.5cm}
\caption{This is the caption of the figure displaying a white eagle and
a white horse on a snow field}
\end{figure}

Since $u_{1}$ is a smooth function, we will have
$\left\|hu_{1}\right\|_\infty \le \eta$
for $h$ small enough, and inequality (\ref{eq:two}) will hold,
yielding thereby:
\begin{equation}
  \psi (hu_{1}) \le \frac{h^{2}}{2}
  \frac{1}{\lambda} \left\|u_{1} \right\|_{2}^{2} + \frac{h^{2}}{2}
  \frac{1}{\delta '} \left\|u_{1}\right\|^{2}\ .
\end{equation}

If we choose $\delta '$ close enough to $\delta$, the quantity
$\left(\frac{1}{\lambda} + \frac{1}{\delta '}\right)$
will be negative, and we end up with
\begin{equation}
  \psi (hu_{1}) < 0\ \ \ \ \ {\rm for}\ \ h\ne 0\ \ {\rm small}\ .
\end{equation}

On the other hand, we check directly that $\psi (0) = 0$. This shows
that 0 cannot be a minimizer of $\psi$, not even a local one.
So $\overline{u} \ne 0$ and
$\overline{u} \ne \Lambda_{o}^{-1} (0) = 0$. \qed
\end{proof}
\begin{corollary}
Assume $H$ is $C^{2}$ and
$\left(a_{\infty},b_{\infty}\right)$-subquadratic at infinity. Let
$\xi_{1},\allowbreak\dots,\allowbreak\xi_{N}$  be the
equilibria, that is, the solutions of $H' (\xi ) = 0$.
Denote by $\omega_{k}$
the smallest eigenvalue of $H'' \left(\xi_{k}\right)$, and set:
\begin{equation}
  \omega : = {\rm Min\,} \left\{\omega_{1},\dots,\omega_{k}\right\}\ .
\end{equation}
If:
\begin{equation}
  \frac{T}{2\pi} b_{\infty} <
  - E \left[- \frac{T}{2\pi}a_{\infty}\right] <
  \frac{T}{2\pi}\omega
  \label{eq:three}
\end{equation}
then minimization of $\psi$ yields a non-constant $T$-periodic solution
$\overline{x}$.
\end{corollary}

We recall once more that by the integer part $E [\alpha ]$ of
$\alpha \in \bbbr$, we mean the $a\in \bbbz$
such that $a< \alpha \le a+1$. For instance,
if we take $a_{\infty} = 0$, Corollary 2 tells
us that $\overline{x}$ exists and is
non-constant provided that:

\begin{equation}
  \frac{T}{2\pi} b_{\infty} < 1 < \frac{T}{2\pi}
\end{equation}
or
\begin{equation}
  T\in \left(\frac{2\pi}{\omega},\frac{2\pi}{b_{\infty}}\right)\ .
  \label{eq:four}
\end{equation}

\begin{proof}
The spectrum of $\Lambda$ is $\frac{2\pi}{T} \bbbz +a_{\infty}$. The
largest negative eigenvalue $\lambda$ is given by
$\frac{2\pi}{T}k_{o} +a_{\infty}$,
where
\begin{equation}
  \frac{2\pi}{T}k_{o} + a_{\infty} < 0
  \le \frac{2\pi}{T} (k_{o} +1) + a_{\infty}\ .
\end{equation}
Hence:
\begin{equation}
  k_{o} = E \left[- \frac{T}{2\pi} a_{\infty}\right] \ .
\end{equation}

The condition $\gamma < -\lambda < \delta$ now becomes:
\begin{equation}
  b_{\infty} - a_{\infty} <
  - \frac{2\pi}{T} k_{o} -a_{\infty} < \omega -a_{\infty}
\end{equation}
which is precisely condition (\ref{eq:three}).\qed
\end{proof}

\begin{lemma}
Assume that $H$ is $C^{2}$ on $\bbbr^{2n} \setminus \{ 0\}$ and
that $H'' (x)$ is non-de\-gen\-er\-ate for any $x\ne 0$. Then any local
minimizer $\widetilde{x}$ of $\psi$ has minimal period $T$.
\end{lemma}
\begin{proof}
We know that $\widetilde{x}$, or
$\widetilde{x} + \xi$ for some constant $\xi
\in \bbbr^{2n}$, is a $T$-periodic solution of the Hamiltonian system:
\begin{equation}
  \dot{x} = JH' (x)\ .
\end{equation}

There is no loss of generality in taking $\xi = 0$. So
$\psi (x) \ge \psi (\widetilde{x} )$
for all $\widetilde{x}$ in some neighbourhood of $x$ in
$W^{1,2} \left(\bbbr / T\bbbz ; \bbbr^{2n}\right)$.

But this index is precisely the index
$i_{T} (\widetilde{x} )$ of the $T$-periodic
solution $\widetilde{x}$ over the interval
$(0,T)$, as defined in Sect.~2.6. So
\begin{equation}
  i_{T} (\widetilde{x} ) = 0\ .
  \label{eq:five}
\end{equation}

Now if $\widetilde{x}$ has a lower period, $T/k$ say,
we would have, by Corollary 31:
\begin{equation}
  i_{T} (\widetilde{x} ) =
  i_{kT/k}(\widetilde{x} ) \ge
  ki_{T/k} (\widetilde{x} ) + k-1 \ge k-1 \ge 1\ .
\end{equation}

This would contradict (\ref{eq:five}), and thus cannot happen.\qed
\end{proof}
\paragraph{Notes and Comments.}
The results in this section are a
refined version of \cite{smit:wat};
the minimality result of Proposition
14 was the first of its kind.

To understand the nontriviality conditions, such as the one in formula
(\ref{eq:four}), one may think of a one-parameter family
$x_{T}$, $T\in \left(2\pi\omega^{-1}, 2\pi b_{\infty}^{-1}\right)$
of periodic solutions, $x_{T} (0) = x_{T} (T)$,
with $x_{T}$ going away to infinity when $T\to 2\pi \omega^{-1}$,
which is the period of the linearized system at 0.

\begin{table}
\caption{This is the example table taken out of {\it The
\TeX{}book,} p.\,246}
\begin{center}
\begin{tabular}{r@{\quad}rl}
\hline
\multicolumn{1}{l}{\rule{0pt}{12pt}
                   Year}&\multicolumn{2}{l}{World population}\\[2pt]
\hline\rule{0pt}{12pt}
8000 B.C.  &     5,000,000& \\
  50 A.D.  &   200,000,000& \\
1650 A.D.  &   500,000,000& \\
1945 A.D.  & 2,300,000,000& \\
1980 A.D.  & 4,400,000,000& \\[2pt]
\hline
\end{tabular}
\end{center}
\end{table}
\begin{theorem} [Ghoussoub-Preiss]\label{ghou:pre}
Assume $H(t,x)$ is
$(0,\varepsilon )$-subquadratic at
infinity for all $\varepsilon > 0$, and $T$-periodic in $t$
\begin{equation}
  H (t,\cdot )\ \ \ \ \ {\rm is\ convex}\ \ \forall t
\end{equation}
\begin{equation}
  H (\cdot ,x)\ \ \ \ \ {\rm is}\ \ T{\rm -periodic}\ \ \forall x
\end{equation}
\begin{equation}
  H (t,x)\ge n\left(\left\|x\right\|\right)\ \ \ \ \
  {\rm with}\ \ n (s)s^{-1}\to \infty\ \ {\rm as}\ \ s\to \infty
\end{equation}
\begin{equation}
  \forall \varepsilon > 0\ ,\ \ \ \exists c\ :\
  H(t,x) \le \frac{\varepsilon}{2}\left\|x\right\|^{2} + c\ .
\end{equation}

Assume also that $H$ is $C^{2}$, and $H'' (t,x)$ is positive definite
everywhere. Then there is a sequence $x_{k}$, $k\in \bbbn$, of
$kT$-periodic solutions of the system
\begin{equation}
  \dot{x} = JH' (t,x)
\end{equation}
such that, for every $k\in \bbbn$, there is some $p_{o}\in\bbbn$ with:
\begin{equation}
  p\ge p_{o}\Rightarrow x_{pk} \ne x_{k}\ .
\end{equation}
\qed
\end{theorem}
\begin{example} [{{\rm External forcing}}]
Consider the system:
\begin{equation}
  \dot{x} = JH' (x) + f(t)
\end{equation}
where the Hamiltonian $H$ is
$\left(0,b_{\infty}\right)$-subquadratic, and the
forcing term is a distribution on the circle:
\begin{equation}
  f = \frac{d}{dt} F + f_{o}\ \ \ \ \
  {\rm with}\ \ F\in L^{2} \left(\bbbr / T\bbbz; \bbbr^{2n}\right)\ ,
\end{equation}
where $f_{o} : = T^{-1}\int_{o}^{T} f (t) dt$. For instance,
\begin{equation}
  f (t) = \sum_{k\in \bbbn} \delta_{k} \xi\ ,
\end{equation}
where $\delta_{k}$ is the Dirac mass at $t= k$ and
$\xi \in \bbbr^{2n}$ is a
constant, fits the prescription. This means that the system
$\dot{x} = JH' (x)$ is being excited by a
series of identical shocks at interval $T$.
\end{example}
\begin{definition}
Let $A_{\infty} (t)$ and $B_{\infty} (t)$ be symmetric
operators in $\bbbr^{2n}$, depending continuously on
$t\in [0,T]$, such that
$A_{\infty} (t) \le B_{\infty} (t)$ for all $t$.

A Borelian function
$H: [0,T]\times \bbbr^{2n} \to \bbbr$
is called
$\left(A_{\infty} ,B_{\infty}\right)$-{\it subquadratic at infinity}
if there exists a function $N(t,x)$ such that:
\begin{equation}
  H (t,x) = \frac{1}{2} \left(A_{\infty} (t) x,x\right) + N(t,x)
\end{equation}
\begin{equation}
  \forall t\ ,\ \ \ N(t,x)\ \ \ \ \
  {\rm is\ convex\ with\  respect\  to}\ \ x
\end{equation}
\begin{equation}
  N(t,x) \ge n\left(\left\|x\right\|\right)\ \ \ \ \
  {\rm with}\ \ n(s)s^{-1}\to +\infty\ \ {\rm as}\ \ s\to +\infty
\end{equation}
\begin{equation}
  \exists c\in \bbbr\ :\ \ \ H (t,x) \le
  \frac{1}{2} \left(B_{\infty} (t) x,x\right) + c\ \ \ \forall x\ .
\end{equation}

If $A_{\infty} (t) = a_{\infty} I$ and
$B_{\infty} (t) = b_{\infty} I$, with
$a_{\infty} \le b_{\infty} \in \bbbr$,
we shall say that $H$ is
$\left(a_{\infty},b_{\infty}\right)$-subquadratic
at infinity. As an example, the function
$\left\|x\right\|^{\alpha}$, with
$1\le \alpha < 2$, is $(0,\varepsilon )$-subquadratic at infinity
for every $\varepsilon > 0$. Similarly, the Hamiltonian
\begin{equation}
H (t,x) = \frac{1}{2} k \left\|k\right\|^{2} +\left\|x\right\|^{\alpha}
\end{equation}
is $(k,k+\varepsilon )$-subquadratic for every $\varepsilon > 0$.
Note that, if $k<0$, it is not convex.
\end{definition}

\paragraph{Notes and Comments.}
The first results on subharmonics were
obtained by Rabinowitz in \cite{fo:kes:nic:tue}, who showed the existence of
infinitely many subharmonics both in the subquadratic and superquadratic
case, with suitable growth conditions on $H'$. Again the duality
approach enabled Clarke and Ekeland in \cite{may:ehr:stein} to treat the
same problem in the convex-subquadratic case, with growth conditions on
$H$ only.

Recently, Michalek and Tarantello (see \cite{fost:kes} and \cite{czaj:fitz})
have obtained lower bound on the number of subharmonics of period $kT$,
based on symmetry considerations and on pinching estimates, as in
Sect.~5.2 of this article.