Accurate video understanding involves reasoning about the relationships
between actors, objects and their environment, often over long temporal
intervals. In this paper, we propose a message passing graph neural network
that explicitly models these spatio-temporal relations and can use explicit
representations of objects, when supervision is available, and implicit
representations otherwise. Our formulation generalises previous structured
models for video understanding, and allows us to study how different design
choices in graph structure and representation affect the model's performance.
We demonstrate our method on two different tasks requiring relational reasoning
in videos -- spatio-temporal action detection on AVA and UCF101-24, and video
scene graph classification on the recent Action Genome dataset -- and achieve
state-of-the-art results on all three datasets. Furthermore, we show
quantitatively and qualitatively how our method is able to more effectively
model relationships between relevant entities in the scene.
Anurag Arnab Chen Sun Cordelia Schmid
Uniﬁed Graph Structured Models for Video Understanding
Anurag Arnab Chen Sun Cordelia Schmid
Google Research
faarnab, chensun, cordelias g@google.com
Abstract
Accurate video understanding involves reasoning about
the relationships between actors, objects and their environ-
ment, often over long temporal intervals. In this paper,
we propose a message passing graph neural network that
explicitly models these spatio-temporal relations and can
use explicit representations of objects, when supervision is
available, and implicit representations otherwise. Our for-
mulation generalises previous structured models for video
understanding, and allows us to study how different de-
sign choices in graph structure and representation affect
the model’s performance. We demonstrate our method on
two different tasks requiring relational reasoning in videos
– spatio-temporal action detection on AVA and UCF101-
24, and video scene graph classiﬁcation on the recent Ac-
tion Genome dataset – and achieve state-of-the-art results
on all three datasets. Furthermore, we show quantitatively
and qualitatively how our method is able to more effectively
model relationships between relevant entities in the scene.
1. Introduction
Deep learning has enabled rapid advances in many im-
age understanding tasks such as image classiﬁcation [20],
object detection [46] and semantic segmentation [7]. How-
ever, progress on recent video understanding datasets such
as A V A [17] and Charades [49] has lagged behind in com-
parison. Further progress in the video understanding tasks
posed by these datasets would facilitate applications in au-
tonomous vehicles, health monitoring and automated media
analysis and production among others.
A reason why video understanding is so challenging is
because, as shown in Fig. 1, it requires understanding the
interactions between actors, objects and other context in the
scene. Furthermore, these interactions are not always ob-
servable from a single frame, and thus require reasoning
over long temporal intervals. This is illustrated in Fig. 1,
where understanding the actions of the person in the centre-
frame is not possible from the target keyframe alone. In or-
der to know that the woman is “listening”, we need to con-sider the man who is speaking but no longer in the scene.
And to correctly infer that the woman is “driving” the car,
rather than “riding” like the man, we must take into account
that she is later holding the steering wheel.
Video is a signiﬁcantly higher-dimensional signal than
single images, due to its additional temporal axis, and so we
believe learning these unlabelled interactions directly from
current datasets with large convolutional networks is not
feasible. In this paper, we propose a structured graph neural
network to explicitly model these spatio-temporal interac-
tions. We model actors and objects (explicitly with bound-
ing boxes when we have supervision, and implicitly other-
wise) as nodes in our spatio-temporal graph and perform
message passing inference to directly model their relations.
Although a wide range of graph-structured models have
been proposed for action recognition, we note that there has
been no unifying formulation for these models. As such,
some works only model spatial relations between actors
and objects [14, 54], but not how these interactions evolve
over time. Other approaches model long-range temporal
interactions [61], but do not capture spatial relations and
are not trained end-to-end. And whilst some methods do
model spatio-temporal interactions of objects [3, 60], their
explicit representations of objects need additional supervi-
sion, and are not evaluated on spatio-temporal localisation
tasks which requires detailed understanding and is neces-
sary for analysing untrimmed videos.
Our graph network formulation based on the message-
passing neural network [13] abstraction, allows us to ex-
plicitly model interactions between actors, objects and
the scene, and how these interactions evolve over time.
Our ﬂexible model allows us to use explicit object rep-
resentations from a pretrained Region Proposal Network
(RPN) [46], and/or implicitly from convolutional feature
maps without additional supervision. Moreover, our general
formulation allows us to interpret previous work [14, 54, 60,
61, 71] as special cases, and thus understand how different
design choices in object representation, graph connectivity
and message passing functions affect the model’s perfor-
mance. We demonstrate our versatile model on two dif-
ferent tasks: spatio-temporal action detection on A V A [17]arXiv:2103.15662v1  [cs.CV]  29 Mar 2021
TimeTarget keyframeLabels: listen, drive, sit…………Figure 1: Understanding videos requires reasoning over long-term spatio-temporal interactions between actors, objects and the environ-
ment. The actions of the woman in the centre frame are ambiguous given the nearby frames which typical 3D CNN architectures consider.
However, by considering her interactions with the man from nearby frames, we know she is “listening” to him. And the fact that she is later
holding the steering wheel indicates she is “driving” in contrast to the man who is “riding”. In this paper, we propose a spatio-temporal
graph on which we perform message passing to explicitly model these spatio-temporal interactions. Example from the A V A dataset [17].
and UCF101-24 [53], and video scene graph prediction on
the recent Action Genome [23] dataset. Both of these tasks
require modelling the spatio-temporal interactions between
the actors and/or objects in the scene, and we show con-
sistent improvements from using each component of our
model and achieve state-of-the-art results on each dataset.
Furthermore, we observe that the largest improvements
in A V A are indeed achieved on action classes involving
human-to-human and human-to-object interactions, and vi-
sualisations of our network show that it is focusing on scene
context that is intuitively relevant to its action classiﬁcation.
2. Related Work
Modelling contextual relationships has a long history in
scene understanding. Relevant examples of early works in
this area included modelling interactions between humans
and objects [18, 68], different objects [45] and relationships
between human actions and scene context [41, 16]. Further-
more, it has also been shown that human vision is reliant on
context too [43]. In this paper, we consider video under-
standing tasks, speciﬁcally spatio-temporal action recogni-
tion and video scene graph parsing, which involve reasoning
about interactions between actors, objects and their environ-
ment in both space and time.
Early work in action recognition used hand-crafted fea-
tures to encode motion information [32, 58]. Advances
in deep learning ﬁrst saw repurposing of 2D image con-
volutional neural networks (CNNs) for video as “two
stream” networks [27, 50] followed by spatio-temporal 3D
CNNs [6, 12, 55, 72]. However, these architectures fo-
cus on extracting coarse, video-level features and are not
suitable for learning the ﬁne-grained relations depicted in
Fig. 1. Consequently, whilst initial approaches to spatio-
temporal action detection involved extending 2D object de-
tectors [38, 46] temporally [26, 51, 1, 65], current leading
methods [14, 61, 71] on the A V A dataset [17] all explicitly
model relationships, with approaches which we show can
be interpreted as variants of graph neural networks.
Graph neural networks (GNNs) explicitly model rela-
tions between entities by modelling them as nodes in adirected or undirected graph [5, 48, 29], which interact
via a neighbourhood deﬁned for each node. The self-
attention [56] and Non-local [59] operators can also be
thought of as GNNs [5, 13] where each element in a fea-
ture map is a node, and all nodes are fully-connected to
each other. Such attention-based models have excelled at
a number of natural language processing and computer vi-
sion tasks and have inspired many follow-up methods [21,
8, 9, 62, 70].
Many structured models have also recently been em-
ployed in video understanding. However, there has pre-
viously been no coherent framework to unify these ap-
proaches. Consequently, some works only model spatial re-
lations between actors and objects, but not how these evolve
over time [54, 14]. And whilst LFB [61] models long range
temporal interactions, it does not capture spatial relation-
ships within a keyframe. Moreover, in order to model long-
range interactions, [61] uses precomputed features and is
thus not trained end-to-end. Our proposed method, which is
based on the message passing neural network (MPNN) [13]
framework, coherently models both spatial and temporal
interactions. After describing our model in Sec. 3, we
show how previous structured models for action detec-
tion [54, 14, 60, 61, 71] can be thought of as special cases
of our model. Furthermore, our ﬂexible model can reason
about objects both when we have explicit supervision for
them, and when we do not. Prior work in video understand-
ing, which have proposed less generic models, have either
assumed the case of explicit object supervision [3, 60] or
not [54, 14, 71], but have not considered the scenario when
both options are available. Furthermore, our general formu-
lation allows us to ablate graph modelling design choices,
such as the object representation, message passing func-
tions and temporal context, in a manner not possible with
these previous, more speciﬁc approaches. Moreover, we
note that some approaches which have employed spatio-
temporal graphs in video have only considered frame-to-
frame interactions [22, 60, 3, 40, 42, 44], rather than long-
range relations, and have not evaluated on spatio-temporal
localisation which requires more detailed understanding
3D CNN
sit, answer phone, carry/hold 
object
sit, touch (an object), watch (a 
person), talk (a person)
sit, touch (an object), watch (a 
person), listen (a person)
Readout Function, 

Spatio
-
temporal graph 
construction and message passing
Foreground node
Implicit context
Explicit contextFigure 2: Overview of our method: We construct a spatio-temporal graph, and perform message-passing inference on it, to model interac-
tions between actors, objects and their environment. Foreground nodes (circles) have readout functions associated with them for the task of
interest ( i.e. for action recognition, the nodes represent person bounding boxes which are classiﬁed into actions). Context nodes (squares)
model additional information, and can either be implicit, as cells of the original feature map, or explicit by ROI-pooling external region
proposals (RPN [46] not shown for clarity). The initial state of each node is a spatio-temporal feature vector extracted from a 3D CNN.
and is essential for analysing untrimmed videos.
We also note that scene graph parsing [25, 30] is an-
other task that evaluates a model’s ability to reason about
the interactions between different objects by representing
objects as nodes and relationships as edges in a graph. Al-
though the task was originally posed for single images [30],
the recent Action Genome [23] dataset extends this task
to video by adding annotations to Charades [49]. While
GNN-based approaches have also been used in scene graph
parsing [10, 33, 64, 66] for single images, to our knowl-
edge, they have not been applied to model temporal rela-
tions in video. Moreover, the same model has not also been
demonstrated on spatio-temporal action recognition like our
method.
3. Proposed Approach
Our model aims to build a structured representation of
a video by representing it as a graph of actors, objects
and contextual elements in the scene, as shown in Fig. 2.
This structured representation is then used to perform tasks
which require understanding the interactions between the
elements in the graph, such as action recognition and scene
graph prediction. Note that we do not assume we have an-
notations for relevant scene context.
Our approach is based on Message Passing Neural Net-
works (MPNN) [13], as it is a ﬂexible framework that gen-
eralises many previous graph neural network algorithms [4,
5, 29, 57] We review this approach, and describe how we
adapt it for video understanding in Sec. 3.1, before detail-
ing our model in Sec. 3.2 through 3.5. Finally, we dis-cuss how previous structured models for video understand-
ing [14, 54, 60, 61, 71] can be regarded as speciﬁc instanti-
ations of our model in Sec. 3.6.
3.1. Message Passing Neural Networks (MPNNs)
MPNNs operate on a directed or undirected graph, G
consisting of nodes, v2V, and a neighbourhood for each
node,Nv, that deﬁnes the graph’s connectivity. For video
models we distinguish between the spatial, Sv, and tempo-
ral,Tv, neighbourhoods for node v(Nv=Sv[Tv). Each
node,v, is associated with a latent state, hv. Inference in
this model consists of a message passing phase, and a ﬁ-
nal readout phase. In the message passing phase, messages
for each node, mv, are ﬁrst computed by applying spatial
and temporal message passing functions, MsandMtre-
spectively, to all nodes in its neighbourhood as described in
Eq. (1). An update function, U, then aggregates the received
messages to update the latent state, hv,
mi+1
v=X
w2SvMs(hi
v;hi
w;i
s) +X
u2TvMt(hi
v;hi
u;i
t)(1)
hi+1
v=U(mi+1
v;hi
v); (2)
wheredenotes learnable function parameters. Intuitively,
the state of a node, hv, is updated by aggregating the mes-
sages passed from its neighbours. Finally, after I1iter-
ations of message passing, a readout function, R, uses the
updated node features for the classiﬁcation task of interest
y=R(fhi
vgjv2G): (3)
As illustrated in Fig. 2, our graph consists of a set of
“Foreground” nodes, F=ff1;f2;:::;fngand “Context”
nodes,C=fc1;c2;:::;cmgwherenandmvary for each
video. The “Foreground” nodes, F, have readout and loss
functions associated with them, and correspond to the ob-
jects that will subsequently be classiﬁed. For the task of ac-
tion recognition,Fcorresponds to bounding boxes of each
actor in the keyframe. Whilst for scene graph prediction, F
refers to bounding boxes for all potential objects of inter-
est. The “Context” nodes, C, capture additional information
extracted from the scene for relational reasoning. The rep-
resentations of these nodes, and their spatial connectivity,
are detailed next.
3.2. Spatial Model
The spatial connections in our graph model relation-
ships between actors, objects and scene context in the same
frame. For example, recognising an action such as “give
object to a person” in the A V A dataset [17] requires under-
standing both people involved in the action and also the ob-
ject being transferred.
To model such interactions, we ﬁrst represent the fore-
ground nodes of our model, F, by extracting convolu-
tional features from the last layer of the network, X2
Rthwc, and using ROI-Align [19] followed by spatio-
temporal pooling and a linear projection to obtain fi2Rd.
We model scene context by considering the features from
each spatial position in the feature map, X, as a contextual
node,ci, in our graph. Note that these features are projected
toRd. A similar representation was used by [54], which we
refer to as an implicit object model as it enables the network
to encode information about the scene and relevant objects
without any extra supervision. This approach is also known
as “grid features” in visual question answering [4, 24].
It is also possible to augment our set of contextual nodes
with an explicit object representation by computing class-
agnostic object proposals with a Region Proposal Network
(RPN) [46]. We use an RPN pretrained on the OpenImages
dataset [31] and obtain a d-dimensional feature from each
proposal by using ROI-Align and a linear projection as for
the foreground nodes. A similar idea of using explicit object
representations in video has also been employed by [3] and
[60], though not for spatio-temporal action recognition.
Messages are then computed and passed to foreground
nodes from both foreground and contextual nodes in the
graph. We only update the foreground nodes during mes-
sage passing, as these are the nodes which are subsequently
classiﬁed in the ﬁnal Readout phase. Concretely, the spatial
neighbourhood for each foreground node in our graph is
Sv=F[C8v2F: (4)
ContextForegroundFrames
⚫
⚫
⚫𝜏"𝜏#Figure 3: Illustration of temporal connectivity. Foreground (blue)
and context (green) nodes have initial states computed from spatio-
temporal features over tframes (shaded in yellow), centred on
a keyframe (denoted by a black circle). scontrols the distance
between selected keyframes; cis the total number of keyframes
considered. Here, c= 3,s= 7frames andt= 5 frames.
3.3. Temporal Model
We also include temporal connections in our graph to
model long-range interactions between actors and objects.
As shown in Fig. 1, understanding actions often requires
reasoning about actors who are no longer visible in the cur-
rent frame, thus requiring large temporal contexts.
We model these temporal interactions by connecting
foreground nodes in keyframe twith all other foreground
nodes in neighbouring keyframes t02T . Concretely, we
deﬁne the temporal context, c, and temporal stride, s, hy-
perparameters. As shown in Fig. 3, cis the total number of
number of keyframes in the video which we consider in our
temporal graph. sis the sampling rate at which we select
keyframes, as s1allows us to consider a wider temporal
interval in a more computationally efﬁcient manner. This is
necessary to train the entire model end-to-end. Moreover,
as each foreground feature node in the graph, fi, is itself
a spatio-temporal feature computed over a period of tby
a 3D CNN, selecting adjacent keyframes (and effectively
settings= 1 keyframe) could result in redundant infor-
mation being captured by the temporal connections in the
graph. Note that our deﬁnition of keyframe follows com-
mon datasets [17, 23] as a frame we aim to classify given
surrounding temporal context.
More formally, we can describe the neighbourhood of
each foreground node, v, for temporal message passing as
Tv=bc=2c[
dt=c=2eFts: (5)
Here, we use the superscript to denote the temporal index,
and without loss of generality consider t= 0 to be the
centre keyframe, meaning that negative time indices corre-
spond to frames in the past. We set cto be an odd, positive
integer to employ an equal-sized temporal window on either
side of the centre keyframe.
We ﬁrst perform spatial message passing, before pass-
ing messages temporally. This allows information from the
context nodes in frame t0,ct0
j, to efﬁciently propagate to a
foreground node ft
i(wheret6=t0) viaft0
j, as the foreground
nodes are fully-connected temporally.
The following section now describes the messages that
are passed along the graph described above.
3.4. Message Passing Functions
We ﬁrst observe that the Non-local operator or self-
attention [56, 59] can be considered as a message passing
function in a fully-connected graph, where each node, hv,
is an element in the input feature map, H, and the neigh-
bourhood comprises of all other feature elements. And
since self-attention is employed with a residual connec-
tion [59, 56] and layer normalisation [2, 56], the update
function of Non-local when viewed as an MPNN is
U=LN(hv+Self-Attention (H)): (6)
Similar analysis [5, 13] has shown that Graph Attention
Networks (GAT) [57], Relational Networks [47] and many
other graph neural networks [48, 29, 35, 4] can also be inter-
preted as MPNNs. In this paper, we consider Non-local [59]
and Graph Attention (GAT) [57] as message passing func-
tions within the update function, Eq. (6).
Non-local We modify Non-local [59] to pass messages
from all Foreground and Context nodes to only Foreground
nodes (Eq. 4), as these are the nodes which are used for the
ﬁnal classiﬁcation in the Readout phase,
M=SoftmaxQK>
p
d
V (7)
Q=AWqK= [AjjC]WkV= [AjjC]Wv:
Here,A2RndandC2Rmdare matrices where each
row is a foreground and context feature node respectively,
Mis theRndmatrix of messages received by each node in
A,[AjjC]denotes the concatenation of these matrices and
Wq,WkandWvare learnable ddprojection matrices.
Graph attention (GAT) The graph attention [57] mes-
sage for a node vis computed as
mv=0
@X
j2NvijWahj1
A (8)
ij=Softmax

w>
b[hijjhj]
(9)
whereis a ReLU non-linearity and Waandwbare a
learnable matrix and vector respectively.
Parallel messages It is also possible to compute multiple
incoming messages in parallel for a node, hv. When usingmultiple Non-local or Graph Attention functions, this corre-
sponds to multi-headed attention [56, 57]. By viewing Non-
local and GAT as message passing functions, we can also
aggregate messages from a combination of these two. In
these cases, we aggregate the messages using an attention-
weighted convex combination, as performed in Eq. (9).
3.5. Readout Function
AfterI1iterations of message passing, a readout
function is applied on the Foreground nodes to obtain the
ﬁnal predictions. For action detection, the readout func-
tion is a linear classiﬁer operating on each element of F,
where each foreground node, fi, corresponds to the features
of an actor in the keyframe. For scene graph prediction, the
readout function consists of two classiﬁers: The ﬁrst linear
classiﬁer predicts the object-class label of each foreground
node. The second is a function of each pair of foreground
nodes, and predicts the relationship label between them.
3.6. Discussion
We note that many previous structured models for video
understanding can be considered as special cases of our pro-
posed MPNN framework:
Girdhar et al . [14] only consider a spatial model, i.e.
N=S, using implicit objects and Non-local [60] for
message passing. ACRN [54] has the same graph struc-
ture, but uses Relational Networks [47] for message pass-
ing instead. LFB [61], in contrast, considers only a tem-
poral model, i.e.N=T, using Non-local as the mes-
sage passing function in a graph that is fully-connected in
time. However, [61] do not consider a spatial model to cap-
ture interactions between actors in the keyframe. Zhang et
al. [71], on the other hand, model both temporal and spa-
tial connections. However, they effectively model three
separate graphs: the ﬁrst models actors in short, 3-second
tubelets using GCN [29] for message passing. The other
two graphs model actor-actor and actor-object relations us-
ing a message passing method similar to GAT [57], but us-
ing a hand-deﬁned weighting function rather than a learned
one as in GAT (Eq. 9). Wang et al. [60] also model a spatio-
temporal graph, using GCN for message passing, and an ex-
plicit object representation in the spatial model. However,
their temporal connections are only among adjacent frames,
which does not allow information propagation between all
frames in long sequences, in contrast to our model which
is fully-connected temporally. With our uniﬁed framework,
we study the effect of graph modelling design choices and
show how we outperform previous work next in Sec. 4.
4. Experiments
We evaluate our method’s ability to model spatio-
temporal interactions with experiments on spatio-temporal
action detection and video scene graph classiﬁcation.
Table 1: Ablation study on Action Genome using a 3D ResNet 50 backbone. We report the effect of (a) different message passing functions,
(b) temporal connections in the graph ( c= 1;s= 1corresponds to only spatial message passing) and (c) iterations of message passing.
(a) Spatial message passing functions
SGCls PredCls
R@20 R@50 R@10 R@20
Baseline 48.9 51.3 78.7 93.8
Non-local in backbone 49.1 51.4 78.8 93.9
Non-local 50.4 52.6 79.3 94.2
GAT 51.1 53.2 79.7 94.4
GAT + Non-local 51.3 53.4 79.4 94.3(b) Temporal message passing
Temporal parameters SGCls PredCls
cs R@20 R@50 R@10 R@20
1 1 51.1 53.2 79.7 94.4
3 2 52.9 55.0 79.4 94.2
3 5 53.3 55.5 79.4 94.2
3 7 53.5 55.7 79.3 94.2
5 2 53.4 55.5 79.4 94.2
5 5 53.8 56.0 79.3 94.2
5 7 53.6 55.8 79.3 94.2(c) Message passing iterations
Iterations R@20
1 51.1
2 51.6
3 51.6
5 51.8
Spatio-temporal action recognition We evaluate on
A V A [17], the largest dataset for this task consisting of
15-minute video clips obtained from movies, and UCF101-
24 [53], the previous standard benchmark for this task. A V A
is labelled with atomic actions, where one person is typi-
cally performing multiple actions simultaneously, whilst ac-
tors perform only a single high-level action in UCF101-24.
We follow standard protocol and evaluate using the Frame
AP at an IoU threshold of 0.5 on both datasets. For A V A, we
use v2.2 annotations for ablations, and either v2.1 or v2.2
annotations for fair comparisons with prior work.
Video scene graph prediction We evaluate on the re-
cent Action Genome dataset [23], which adds scene graph
annotations to Charades [49], on two scene graph tasks:
scene graph classiﬁcation (SGCls) and predicate classiﬁca-
tion (PredCls). Both tasks are evaluated using the standard
Recall@K metric [39, 23] (R@K) which measures the frac-
tion of ground truth relationship triplets (subject-predicate-
object) that appear in the top K scoring predicted triplets. In
SGCls, ground truth bounding box co-ordinates are given,
and the aim is to predict their object classes, as well as rela-
tionship labels between pairs of objects. PredCls is simpler,
as both bounding box co-ordinates and object classes are
given, and only the relationship label must be predicted.
Implementation details We use the public implementa-
tion of SlowFast [11] as our baseline with the 3D ResNet
50 or ResNet 101 backbones [20], as it is the current state-
of-the-art. The network is similar to Fast-RCNN [15] as it
uses external region proposals to extract features from the
last feature map of res 5using ROI-Align [19]. These fea-
tures are then spatio-temporally pooled and classiﬁed.
In our graph model, we initialise the internal states, hi,
of our Foreground nodes using these ROI-Aligned res 5fea-
tures. For our action detection experiments, the Foreground
nodes in our graph correspond to bounding boxes of actors,
and we use the same person detector as [11, 61] for our ac-
tor region proposals. For scene graph experiments, we use
ground truth boxes of people and objects as our Foreground
nodes as we evaluate scene graph classiﬁcation (SGCls) and
predicate classiﬁcation (PredCls).Unless otherwise speciﬁed, we use SlowFast 88which
corresponds to 32 input frames where the video is subsam-
pled by a factor of 2. This means that each feature node,
hi, in the graph aggregates t= 2:1seconds of temporal
information on A V A as the videos are sampled at 30 frames
per second (fps). As keyframes in A V A are deﬁned at 1 fps,
it means that we need to set s2keyframes for the tem-
poral information captured by temporally-adjacent nodes in
the graph to not overlap. For Action Genome, the Charades
videos are sampled at 24 fps and the keyframes are on aver-
age 0.85 seconds, or 20.5 frames apart.
We train our network for 20 epochs using synchronous
SGD on 8 GPUs and a total batch size of 64, initialis-
ing from a Kinetics-400 [28] pretrained model. The base-
line model in all our experiments is SlowFast without any
graph module trained in an identical manner. When training
spatio-temporal graph models, we ﬁnetune for 10 epochs
from a model trained with only a spatial graph and reduce
the batch size by a factor of c. We also “freeze” batch
normalisation statistics as they have high variance for small
batches [19]. Full training details are in the appendix.
4.1. Experiments on video scene graph prediction
Spatial message passing We ﬁrst ablate only the spa-
tial component (Sec. 3.2) of our model in Tab. 1a. Graph
Attention (GAT) [57] performs slightly better than Non-
local [56, 59] as a message passing function, and we obtain
a further small improvement by combining the two methods
in parallel (Sec. 3.4). Overall, we improve upon our base-
line model, which is SlowFast based on ResNet 50 without
any graph modelling, by 2.4 points for R@20 on SGCls.
Another baseline is to insert a Non-Local layer [59] af-
ter the ﬁnal res 5layer and before the ROI-Align layer, since
Non-local can also be viewed as a graph network (Sec. 3.4).
In this case, the Foreground and Context nodes are not ex-
plicitly modelled as in our method and the overall perfor-
mance is 2.2 points less than our method for the R@20 of
SGCls. This suggests that explicit modelling of the Fore-
ground nodes which are subsequently classiﬁed is impor-
tant to performance. We note, however, that Non-Local [59]
is typically employed earlier in a network ( i.e. in res 3or
res4[59, 61, 11]) and can thus be seen as a complementary
Table 2: Comparison to existing methods on the Action Genome
dataset [23]. Previous methods reported by [23].
SGCls PredCls
R@20 R@50 R@10 R@20
MSDN [34] 44.0 47.2 – –
IMP [64] 44.1 47.4 – –
RelDN [69] 46.7 49.4 – –
SlowFast (ResNet 50) 48.9 51.3 78.7 93.8
Ours (ResNet 50) 53.8 56.0 79.3 94.2
personsofacouch
table
laptopnot looking atnot contacting
not contactingnot looking at
not contacting
in front of
looking aton the side of
in front of
person
sofacouchtelevision
laptop
table
not looking atnot contactingin front of
not contacting
behindlooking at
not looking atbeneathlooking at
in front of
sitting on
on the side ofabove
touching
Figure 4: Example scene graph predictions for consecutive
keyframes of Action Genome [23]. Classiﬁed bounding boxes are
on top, and the corresponding scene graph (objects are shown as
coloured rectangles, and relations with light-blue ovals) below.
method to improve features learned by the network.
Temporal message passing Table 1b adds temporal con-
nections (Sec. 3.3) to our model, using GAT for message
passing, as it outperformed Non-local in Tab. 1a. We ob-
serve consistent improvements for a wide range of temporal
contexts,c, and strides, s, showing the utility of modelling
temporal dynamics. We improve over a graph with only
spatial connections by as much as 2.7 points for R@20.
For a temporal model, another baseline to consider is
to simply increase the temporal information available to
the spatial-only graph model by increasing the number of
frames at the input. When tripling the number of input
frames, the R@20 is 52.4, less than all variants of our tem-
poral model with c= 3. Similarly, increasing input frames
by a factor of 5, the R@20 is 52.8, less than all our tempo-
ral models with c= 5. This improvement demonstrates the
beneﬁts of message passing on an explicit temporal graph.
Iterations of message passing Table 1c shows that the
model’s performance plateaus after 2 iterations of message
passing, where parameters are not tied across iterations.
Comparison to state-of-the-art Finally, we compare to
prior works in Tab. 2 which are single-image models evalu-Table 3: Analysis of message passing neighbourhood on A V A us-
ing SlowFast with a ResNet-50 backbone as the baseline. By vary-
ing the neighbourhood, we study the effect of object representa-
tions (implicit, explicit or none) and temporal connections. We
report the Frame mAP on the three types of action classes in A V A.
Message passing neighbourhood PoseHuman-
HumanHuman-
ObjectAll
SlowFast baseline (none) 43.1 25.2 17.4 24.8
Actors only 43.2 27.0 17.8 25.6
Implicit objects only 43.4 26.7 18.0 25.7
Explicit objects only 43.0 26.7 17.8 25.5
Actor + Implicit 43.4 26.8 18.3 25.9
Actor + Implicit + Explicit 43.7 27.0 18.4 26.1
Spatio-temporal 43.8 27.5 19.9 27.0
ated by [23]. Our SlowFast, ResNet 50 3D baseline out-
performs these, showing the importance of using spatio-
temporal features for this task. As Action Genome contains
videos of humans acting to scripts, there is temporal struc-
ture in the interactions between actors and objects in the
scene (Fig. 4). Our ﬁnal spatio-temporal graph structured
model improves substantially upon this baseline by 4.9 and
4.7 points for the R@20 and R@50 for SGCls respectively.
Our improvements over the baseline for PredCls are less, as
this task is easier and the performance is saturated.
4.2. Experiments on action recognition
Graph structure and object representation Table 3
compares the effect of changing the neighbourhood that
passes messages to each foreground node that is subse-
quently classiﬁed. We report the performance across the
three types of action categories in A V A [17] – Pose, Human-
Human and Human-Object – to show the effect that dif-
ferent object representations (implicit, explicit or neither as
described in Sec. 3.2) and also temporal connections have.
Our SlowFast baseline does not model any explicit graph
and hence effectively performs no message passing. For
our A V A experiments, Foreground nodes in the graph cor-
respond to bounding boxes of the actors in the scene, using
the same person detections as [61, 11]. Thus, in this section,
“Foreground” and “actor” nodes are used interchangeably.
We observe from the second row of Tab. 3 that pass-
ing messages only between actor nodes (and thus not mod-
elling object interactions) provides an overall improvement
of 0.8 points. The largest gain, as expected, are for Human-
Human action classes, as these are the interactions mod-
elled by passing messages between actor nodes in the graph.
When passing messages from implicit context nodes to
the actors (third row), we observe an improvement in all
types of action classes, but primarily in Human-Human and
Human-Object classes. This is because implicit context
nodes encompass the entire feature map, and thus capture
information about the whole scene (as also shown in Fig. 5).
Modelling objects explicitly with external regional pro-
Table 4: Comparison on A V A. We report the Mean AP using v2.1
and v2.2 annotations. All methods pretrained on Kinetics 400.
“Multiscale” refers to averaging results over three scales [61].
Method v2.1 v2.2
ACRN (S3D) [54] 17.4 –
Zhang et al. (R50) [71] 22.2 –
SlowFast baseline (R50) 24.5 24.8
Girdhar et al. (I3D) [14] 25.0 –
LFB (R50) [61] 25.8 –
Ours (R50) 26.5 27.0
Ours Multiscale (R50) 27.3 27.7
SlowFast baseline (R101) 26.3 26.7
LFB (R101) [61] 26.8 –
Ours (R101) 28.3 28.8
LFB Multiscale (R101) [61] 27.7 –
Ours Multiscale (R101) 29.5 30.0
posals (fourth row) improves on the same action types as the
implicit object model, but performs marginally worse. This
suggests that our RPN trained on OpenImages [31] is unable
to detect the objects that are most discriminative of A V A ac-
tions. As object annotations are not provided in A V A, it is
not possible to evaluate the recall of our external region pro-
posals on relevant objects. It is not clear if we should expect
more improvement from explicit objects, because although
previous works have considered implicit [14, 54, 71] and
explicit representations [3, 60], we are not aware of any that
have compared the two. Note that these explicit context
nodes also model Human-Human interactions as our RPN
is trained on people in OpenImages [31]. We obtain further
improvements by combining messages from actors, implicit
and explicit context nodes, as shown by the next two rows.
Finally, we evaluate our spatio-temporal model, which
passes messages from actors, implicit- and explicit-context
nodes withc= 3ands= 3, corresponding to an overall
temporal window of 8.5 seconds. This model performs the
best, improving upon the baseline by 2.2 points, or a relative
improvement of 8.9%. In particular, the temporal connec-
tions help to improve on Human-Object action classes.
A V A is a long-tailed dataset, and the most common fail-
ure modes of all variants of our model are the classes with
few training examples. However, our model improves on
both head and tail classes with respect to the baseline. This,
and detailed per-class results, are in the appendix.
State-of-the-art-comparison on A V A Table 4 compares
to recent, published work on A V A. Our method builds upon
SlowFast [11] as the base architecture, and our graph model
shows substantial improvements with either a 3D ResNet 50
or ResNet 101 backbone. As discussed in Sec. 3.6, Gird-
haret al. [14] and LFB [61] can be considered as special
cases of our graph model as they model only spatial andTable 5: Comparison to state-of-the-art on UCF101-24. We report
the Frame AP at 0.5 using the corrected annotations of [51].
Method Modality Mean AP
ACT [26] RGB + Flow 69.5
Song et al. [52] RGB + Flow 72.1
STEP [67] RGB + Flow 75.0
Guet al. [17] RGB + Flow 76.3
MOC [36] RGB + Flow 78.0
SlowFast (R50) RGB 76.6
SlowFast (R101) RGB 77.4
Ours (R50) RGB 78.6
Ours (R101) RGB 79.3
temporal edges respectively. Our method, which constructs
a spatio-temporal graph, outperforms both. Note that we
outperform LFB [61] across both ResNet 50 and ResNet
101 backbones. Zhang et al . [71] also model a spatio-
temporal interactions, but employ three separate graphs
with a hand-crafted aggregation function which we also out-
perform with our single, coherent spatio-temporal graph.
State-of-the-art comparison on UCF101-24 Table 5
shows that we outperform recent, published work on
UCF101-24 [53], using either a 3D ResNet 50 or ResNet
101 backbone. We do so without using optical ﬂow as an
additional input modality, showing that our network can
capture temporal information without it. Moreover, the im-
provements obtained from our graph model are consistent
with our results on the Action Genome and A V A datasets.
Qualitative results We visualise the spatial and tempo-
ral messages received by an actor node in the graph, on the
A V A dataset, when using GAT as the message passing func-
tion. In particular, we visualise the attention weights, ijin
(9), on each implicit object node (Fig. 5) and actor in neigh-
bouring keyframes (Fig. 6), for a given actor. We observe
that the network places more weight on graph nodes that are
intuitively consistent with the ﬁnal action prediction. For
example, in Fig. 5, the model focuses on the actor’s face
when his action is “talk to” and the other person’s face when
his action is “listen to”.
5. Conclusion and Future Work
We have presented a novel spatio-temporal graph neu-
ral network framework to explicitly model interactions be-
tween actors, objects and their environment. Our formula-
tion can model objects either implicitly or explicitly, and
generalises previous structured models for video under-
standing [14, 54, 61, 71]. Using our versatile approach, we
have achieved state-of-the-art results on two diverse tasks
across three datasets. Future work remains to harness ex-
plicit object representations more effectively on A V A.
(a)talk to (0.6), sit (0.92), watch person (0.81) (b)listen to person (0.9), watch person (0.89), sit (0.97) (c)carry/hold (1.0), sit(0.88)
Figure 5: Visualisation of spatial message passing when using implicit objects as detailed in Sec. 4.2. Note in (a) and (b) that the network
focuses on the selected actor (denoted by the green box on the left) when the action is “talk to”, and on the person speaking to him when
the action is “listen to person”. Attention weights, ijare colour-coded according to the bar on the far right. Best viewed in colour.
talk to (0.85), watch a person (0.82), sit(0.51)
Figure 6: Visualisation of temporal graph as detailed in Sec. 4.2. We show the neighbours for the actor depicted by the red box in the
centre keyframe, colour-coding the respective boxes with their attention weights ( ij). Predicted action scores of the red actor are below,
and selected keyframes are s= 2:1seconds apart. Notice that the model focuses on the people 2 keyframes (4.2 seconds) on either side
of the actor to recognise that she is “talking to” and “watching” them.
References
[1] Anurag Arnab, Chen Sun, Arsha Nagrani, and Cordelia
Schmid. Uncertainty-aware weakly supervised action detec-
tion from untrimmed videos. In ECCV , 2020. 2
[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.
Layer normalization. In arXiv preprint arXiv:1607.06450 ,
2016. 5
[3] Fabien Baradel, Natalia Neverova, Christian Wolf, Julien
Mille, and Greg Mori. Object level visual reasoning in
videos. In ECCV , 2018. 1, 2, 4, 8
[4] Peter Battaglia, Razvan Pascanu, Matthew Lai, and
Danilo Jimenez Rezende. Interaction networks for learning
about objects, relations and physics. In NeurIPS , 2016. 3, 4,
5
[5] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Al-
varo Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Ma-
linowski, Andrea Tacchetti, David Raposo, Adam San-
toro, Ryan Faulkner, et al. Relational inductive biases,
deep learning, and graph networks. In arXiv preprint
arXiv:1806.01261 , 2018. 2, 3, 5
[6] Joao Carreira and Andrew Zisserman. Quo vadis, action
recognition? a new model and the kinetics dataset. In CVPR ,
2017. 2
[7] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian
Schroff, and Hartwig Adam. Encoder-decoder with atrous
separable convolution for semantic image segmentation. In
ECCV , 2018. 1
[8] Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng
Yan, and Jiashi Feng. Aˆ 2-nets: Double attention networks.
InNeurIPS , 2018. 2
[9] Yunpeng Chen, Marcus Rohrbach, Zhicheng Yan, Yan
Shuicheng, Jiashi Feng, and Yannis Kalantidis. Graph-based
global reasoning networks. In CVPR , 2019. 2
[10] Bo Dai, Yuqi Zhang, and Dahua Lin. Detecting visual re-
lationships with deep relational networks. In CVPR , 2017.
3[11] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and
Kaiming He. Slowfast networks for video recognition. In
ICCV , 2019. 6, 7, 8, 12
[12] Christoph Feichtenhofer, Axel Pinz, and Richard Wildes.
Spatiotemporal residual networks for video action recogni-
tion. In NeurIPS , 2016. 2
[13] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol
Vinyals, and George E Dahl. Neural message passing for
quantum chemistry. In ICML , 2017. 1, 2, 3, 5
[14] Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zis-
serman. Video action transformer network. In CVPR , 2019.
1, 2, 3, 5, 8
[15] Ross Girshick. Fast r-cnn. In ICCV , 2015. 6
[16] Georgia Gkioxari, Ross Girshick, and Jitendra Malik. Con-
textual action recognition with r* cnn. In ICCV , 2015. 2
[17] Chunhui Gu, Chen Sun, David A Ross, Carl V ondrick, Car-
oline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan,
George Toderici, Susanna Ricco, Rahul Sukthankar, et al.
Ava: A video dataset of spatio-temporally localized atomic
visual actions. In CVPR , 2018. 1, 2, 4, 6, 7, 8, 12
[18] Abhinav Gupta, Aniruddha Kembhavi, and Larry S Davis.
Observing human-object interactions: Using spatial and
functional compatibility for recognition. PAMI , 31(10),
2009. 2
[19] Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir-
shick. Mask r-cnn. In ICCV , 2017. 4, 6
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition. In CVPR ,
2016. 1, 6
[21] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation net-
works. In CVPR , 2018. 2
[22] Ashesh Jain, Amir R Zamir, Silvio Savarese, and Ashutosh
Saxena. Structural-rnn: Deep learning on spatio-temporal
graphs. In CVPR , 2016. 2
[23] Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos
Niebles. Action genome: Actions as compositions of spatio-
temporal scene graphs. In CVPR , 2020. 2, 3, 4, 6, 7, 12
[24] Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Erik Learned-
Miller, and Xinlei Chen. In defense of grid features for visual
question answering. In CVPR , 2020. 4
[25] Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li,
David Shamma, Michael Bernstein, and Li Fei-Fei. Image
retrieval using scene graphs. In CVPR , 2015. 3
[26] Vicky Kalogeiton, Philippe Weinzaepfel, Vittorio Ferrari,
and Cordelia Schmid. Action tubelet detector for spatio-
temporal action localization. In ICCV , 2017. 2, 8
[27] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas
Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video
classiﬁcation with convolutional neural networks. In CVPR ,
2014. 2
[28] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang,
Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola,
Tim Green, Trevor Back, Paul Natsev, et al. The ki-
netics human action video dataset. In arXiv preprint
arXiv:1705.06950 , 2017. 6
[29] Thomas N Kipf and Max Welling. Semi-supervised classiﬁ-
cation with graph convolutional networks. In ICLR , 2017. 2,
3, 5
[30] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson,
Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan-
tidis, Li-Jia Li, David A Shamma, et al. Visual genome:
Connecting language and vision using crowdsourced dense
image annotations. IJCV , 123(1), 2017. 3
[31] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui-
jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan
Popov, Matteo Malloci, Tom Duerig, et al. The open im-
ages dataset v4: Uniﬁed image classiﬁcation, object detec-
tion, and visual relationship detection at scale. IJCV , 2020.
4, 8
[32] Ivan Laptev. On space-time interest points. IJCV , 64(2-3),
2005. 2
[33] Yikang Li, Wanli Ouyang, Xiaogang Wang, and Xiao’ou
Tang. Vip-cnn: Visual phrase guided convolutional neural
network. In CVPR , 2017. 3
[34] Yikang Li, Wanli Ouyang, Bolei Zhou, Kun Wang, and Xi-
aogang Wang. Scene graph generation from objects, phrases
and region captions. In ICCV , 2017. 7
[35] Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard
Zemel. Gated graph sequence neural networks. In ICLR ,
2016. 5
[36] Yixuan Li, Zixu Wang, Limin Wang, and Gangshan Wu. Ac-
tions as moving points. In ECCV , 2020. 8
[37] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays,
Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C Lawrence
Zitnick. Microsoft coco: Common objects in context. In
ECCV , 2014. 12
[38] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian
Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C
Berg. Ssd: Single shot multibox detector. In ECCV , 2016. 2
[39] Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-
Fei. Visual relationship detection with language priors. In
ECCV , 2016. 6
[40] Chih-Yao Ma, Asim Kadav, Iain Melvin, Zsolt Kira, Ghassan
AlRegib, and Hans Peter Graf. Attend and interact: Higher-order object interactions for video understanding. In CVPR ,
2018. 2
[41] Marcin Marszalek, Ivan Laptev, and Cordelia Schmid. Ac-
tions in Context. In CVPR , 2009. 2
[42] Joanna Materzynska, Tete Xiao, Roei Herzig, Huijuan Xu,
Xiaolong Wang, and Trevor Darrell. Something-else: Com-
positional action recognition with spatial-temporal interac-
tion networks. In CVPR , 2020. 2
[43] Aude Oliva and Antonio Torralba. The role of context in ob-
ject recognition. Trends in cognitive sciences , 11(12), 2007.
2
[44] Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen,
and Song-Chun Zhu. Learning human-object interactions by
graph parsing neural networks. In ECCV , 2018. 2
[45] Andrew Rabinovich, Andrea Vedaldi, Carolina Galleguillos,
Eric Wiewiora, and Serge Belongie. Objects in context. In
ICCV , 2007. 2
[46] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.
Faster r-cnn: Towards real-time object detection with region
proposal networks. In NeurIPS , 2015. 1, 2, 3, 4, 12
[47] Adam Santoro, David Raposo, David G Barrett, Mateusz
Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy
Lillicrap. A simple neural network module for relational rea-
soning. In NeurIPS , 2017. 5
[48] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Ha-
genbuchner, and Gabriele Monfardini. The graph neural net-
work model. IEEE Transactions on Neural Networks , 20(1),
2008. 2, 5
[49] Gunnar A. Sigurdsson, G. Varol, X. Wang, Ali Farhadi, I.
Laptev, and A. Gupta. Hollywood in homes: Crowdsourcing
data collection for activity understanding. In ECCV , 2016.
1, 3, 6
[50] Karen Simonyan and Andrew Zisserman. Two-stream con-
volutional networks for action recognition in videos. In
NeurIPS , 2014. 2
[51] Gurkirt Singh, Suman Saha, Michael Sapienza, Philip HS
Torr, and Fabio Cuzzolin. Online real-time multiple spa-
tiotemporal action localisation and prediction. In ICCV ,
2017. 2, 8
[52] Lin Song, Shiwei Zhang, Gang Yu, and Hongbin Sun. Tac-
net: Transition-aware context network for spatio-temporal
action detection. In CVPR , 2019. 8
[53] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.
Ucf101: A dataset of 101 human actions classes from videos
in the wild. In arXiv preprint arXiv:1212.0402 , 2012. 2, 6,
8, 12
[54] Chen Sun, Abhinav Shrivastava, Carl V ondrick, Kevin Mur-
phy, Rahul Sukthankar, and Cordelia Schmid. Actor-centric
relation network. In ECCV , 2018. 1, 2, 3, 4, 5, 8
[55] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann
LeCun, and Manohar Paluri. A closer look at spatiotemporal
convolutions for action recognition. In CVPR , 2018. 2
[56] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko-
reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia
Polosukhin. Attention is all you need. In NeurIPS , 2017. 2,
5, 6, 13
[57] Petar Veli ˇckovi ´c, Guillem Cucurull, Arantxa Casanova,
Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph at-
tention networks. In ICLR , 2018. 3, 5, 6, 12
[58] Heng Wang, Alexander Kl ¨aser, Cordelia Schmid, and
Cheng-Lin Liu. Dense trajectories and motion boundary de-
scriptors for action recognition. IJCV , 103(1), 2013. 2
[59] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaim-
ing He. Non-local neural networks. In CVPR , 2018. 2, 5, 6,
12
[60] Xiaolong Wang and Abhinav Gupta. Videos as space-time
region graphs. In ECCV , 2018. 1, 2, 3, 4, 5, 8
[61] Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaim-
ing He, Philipp Krahenbuhl, and Ross Girshick. Long-term
feature banks for detailed video understanding. In CVPR ,
2019. 1, 2, 3, 5, 6, 7, 8, 12
[62] Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin,
and Michael Auli. Pay less attention with lightweight and
dynamic convolutions. In ICLR , 2019. 2
[63] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen
Lo, and Ross Girshick. Detectron2. https://github.
com/facebookresearch/detectron2 , 2019. 12
[64] Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei.
Scene graph generation by iterative message passing. In
CVPR , 2017. 3, 7
[65] Huijuan Xu, Lizhi Yang, Stan Sclaroff, Kate Saenko, and
Trevor Darrell. Spatio-temporal action detection with multi-
object interaction. In arXiv preprint arXiv:2004.00180 ,
2020. 2
[66] Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi
Parikh. Graph r-cnn for scene graph generation. In ECCV ,
2018. 3
[67] Xitong Yang, Xiaodong Yang, Ming-Yu Liu, Fanyi Xiao,
Larry S Davis, and Jan Kautz. Step: Spatio-temporal pro-
gressive learning for video action detection. In CVPR , 2019.
8
[68] Bangpeng Yao and Li Fei-Fei. Modeling mutual context of
object and human pose in human-object interaction activi-
ties. In CVPR , 2010. 2
[69] Ji Zhang, Kevin J Shih, Ahmed Elgammal, Andrew Tao,
and Bryan Catanzaro. Graphical contrastive losses for scene
graph parsing. In CVPR , 2019. 7
[70] Li Zhang, Dan Xu, Anurag Arnab, and Philip HS Torr. Dy-
namic graph message passing networks. In CVPR , 2020. 2
[71] Yubo Zhang, Pavel Tokmakov, Martial Hebert, and Cordelia
Schmid. A structured model for action detection. In CVPR ,
2019. 1, 2, 3, 5, 8
[72] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Tor-
ralba. Temporal relational reasoning in videos. In ECCV ,
2018. 2
Appendix
In this appendix, we additional analysis on A V A (Sec. A)
and implementation details (Sec. B).
A. A V A per-class analysis
Figure 7 presents per-class results on A V A for our Slow-
Fast baseline, spatial and spatio-temporal graph models
when using a ResNet-50 backbone. We additionally show
the number of examples in the training set. Our proposed
graph models improve on both head and tail classes in A V A.
In fact, the largest absolute improvements are observed on
tail classes such as “cut”, “swim” and “sing to”. However,
note that the absolute accuracy of each class is still corre-
lated with the number of training examples. The classes
which all variants of our model perform the worst on, are
the classes with the least training examples such as “point
to (an object)”, “take a photo” and “turn (eg a screwdriver)”.
To emphasise the long-tailed distribution of A V A, note that
“point to (an object)” has the fewest training examples (just
97), whilst “watch (a person)” has the most with 168 148.
B. Additional Implementation details
Person detections For our spatio-temporal action detec-
tion experiments, we use external person detections as our
actor region proposals like [11, 61]. For our experiments
on A V A, we use the person detections publicly released
by [11, 61] during both training and testing. More speciﬁ-
cally, this is a Faster-RCNN [46] model, pretrained on Mi-
crosoft COCO [37] and then ﬁnetuned on the training set of
A V A [17], using Detectron [63].
For our experiments on UCF101-24 [53], we ﬁne-
tune a Faster-RCNN detector pretrained on COCO on the
UCF101-24 training set using Detectron.
When training on both A V A and UCF101-24 datasets, we
use both ground-truth and predicted person bounding boxes.
Predicted bounding boxes are assigned labels by matching
them to ground-truth boxes using an IoU threshold of 0.75.
Predicted bounding boxes which do not match any ground
truth boxes act as negative examples for all action classes.
OpenImages object detector For our “explicit object
representation” experiments on A V A, we use the publicly
available Faster-RCNN detector trained on OpenImages
v41.
Optimiser hyperparameters We train our models fol-
lowing the settings of the publicly released SlowFast
code [11]. We train for 20 epochs using synchronous
Stochastic Gradient Descent (SGD) and a momentum of 0.9
1https://tfhub.dev/google/faster_rcnn/
openimages_v4/inception_resnet_v2/1on 8 Nvidia V100 (16 GB) GPUs. The learning rate was set
to 0.1, and reduced by a factor of 10 after 10 and 15 epochs
respectively. We employed a linear-warmup schedule for
the learning rate, increasing linearly from 1:25104to0:1
in the ﬁrst 5 epochs. We also used a weight decay of 107.
An epoch is deﬁned as all the keyframes in the dataset.
Training loss functions For spatio-temporal action
recognition on A V A and UCF101-24, we use the binary
cross-entropy loss function. The loss function for a single
example is:
L(x;y) =1
CCX
iyilog ((xi))+(1yi) log (1(xi));
(10)
whereCis the number of classes, is the sigmoid activa-
tion function, x2RCdenotes the logits predicted by the
network and yi2f0;1gdenotes the binary ground truth for
theithclass.
For scene graph classiﬁcation, we use two loss functions:
One for predicting the object class of each bounding box
proposal, and another for predicting the relationship label
between each of theN(N1)
2pairs of object proposals. For
the former, we use the softmax cross-entropy, as each pro-
posal can only be assigned a single object class. And for
the latter, we use the sigmoid cross-entropy, as there can
be multiple relationship labels between any pair of objects.
Concretely, the loss is
L(x;w;y;z ) =Lobject(x;y) +Lrel(w;z) (11)
Lobject(x;y) =1
NNX
iCX
jyijlog (Softmax (x)ij)(12)
Lrel(w;z) =2
N(N1)RNX
ii1X
jRX
rzijrlog ((wijr))
+ (1zijr) log (1(wijr)): (13)
Here,Nis the number of object proposals in the video clip,
Rthe number of relationship classes and Cthe number of
object classes. yis the one-hot, ground-truth object class la-
bel,zthe binary ground-truth relationship label, and xand
wdenote the object- and relationship-logits respectively.
On Action Genome [23], there are R= 25 relationship
classes, and C= 35 object classes. We set = 0:5during
training on Action Genome to prevent the object class loss
from dominating the overall loss (the softmax cross entropy
has a higher loss than the binary sigmoid cross entropy at
the start of training).
Multi-headed attention For the GAT [57] and Non-
Local [59] message-passing functions used in the paper
Mean AP
Number of training examples
0.000.250.500.751.00
100100010000100000watch (a person)
stand
talk to (e.g. self a person a 
listen to (a person)
sit
carry/hold (an object)
walk
touch (an object)
bend/bow (at the waist)
lie/sleep
ride (e.g. a bike a car a horse)
run/jog
answer phone
dance
eat
smoke
fight/hit (a person)
drink
crouch/kneel
read
martial art
grab (a person)
watch (e.g. TV)
sing to (e.g. self a person a 
play musical instrument
open (e.g. a window a car 
drive (e.g. a car a truck)
hand clap
get up
hug (a person)
give/serve (an object) to (a 
close (e.g. a door a box)
write
sail boat
kiss (a person)
listen (e.g. to music)
hand shake
take (an object) from (a 
put down
lift/pick up
text on/look at a cellphone
lift (a person)
push (an object)
push (another person)
hand wave
pull (an object)
dress/put on clothing
fall down
climb (e.g. a mountain)
throw
jump/leap
enter
shoot
cut
take a photo
hit (an object)
work on a computer
turn (e.g. a screwdriver)
swim
point to (an object)
Baseline Spatial Spatio-temporal Training examplesFigure 7: Mean Average Precision for each action class on A V A, for the SlowFast baseline, spatial- and spatio-temporal graph models with
a ResNet-50 backbone. The number of examples in the training set are shown with the blue line on the right, vertical axis. Note that a
logarithmic scale is used, as A V A has a long-tailed distribution of labels.
Table 6: Ablation study of the number of heads when using GAT
as the message passing function. Mean AP reported on A V A using
a ResNet-50 backbone.
Number of heads 1 2 3 4 5
Mean AP 25.4 25.8 25.9 25.9 25.9
(Sec 3.4), it is common to compute “multi-headed” atten-
tion [56]. In all experiments reported in the main paper,
we used 4 heads, as motivated by the ablation study using
GAT on the A V A dataset in Tab. 6. To combine the mes-
sages from each head, we performed an attention-weighted
convex combination as done in Eq. 9.
Multiscale testing We use three scales when performing
multiscale testing (as done in Table 3 of the main paper).
Speciﬁcally, we resize input video-clips so that the shortest
side is 224, 256 and 320 respectively.