We introduce a method for simultaneously classifying, segmenting and tracking
object instances in a video sequence. Our method, named MaskProp, adapts the
popular Mask R-CNN to video by adding a mask propagation branch that propagates
frame-level object instance masks from each video frame to all the other frames
in a video clip. This allows our system to predict clip-level instance tracks
with respect to the object instances segmented in the middle frame of the clip.
Clip-level instance tracks generated densely for each frame in the sequence are
finally aggregated to produce video-level object instance segmentation and
classification. Our experiments demonstrate that our clip-level instance
segmentation makes our approach robust to motion blur and object occlusions in
video. MaskProp achieves the best reported accuracy on the YouTube-VIS dataset,
outperforming the ICCV 2019 video instance segmentation challenge winner
despite being much simpler and using orders of magnitude less labeled data
(1.3M vs 1B images and 860K vs 14M bounding boxes).
Gedas Bertasius Lorenzo Torresani
Classifying, Segmenting, and Tracking
Object Instances in Video with Mask Propagation
Gedas Bertasius, Lorenzo Torresani
Facebook AI
Abstract
We introduce a method for simultaneously classifying,
segmenting and tracking object instances in a video se-
quence. Our method, named MaskProp, adapts the popu-
lar Mask R-CNN to video by adding a mask propagation
branch that propagates frame-level object instance masks
from each video frame to all the other frames in a video clip.
This allows our system to predict clip-level instance tracks
with respect to the object instances segmented in the mid-
dle frame of the clip. Clip-level instance tracks generated
densely for each frame in the sequence are ﬁnally aggre-
gated to produce video-level object instance segmentation
and classiﬁcation. Our experiments demonstrate that our
clip-level instance segmentation makes our approach robust
to motion blur and object occlusions in video. MaskProp
achieves the best reported accuracy on the YouTube-VIS
dataset, outperforming the ICCV 2019 video instance seg-
mentation challenge winner despite being much simpler and
using orders of magnitude less labeled data (1.3M vs 1B im-
ages and 860K vs 14M bounding boxes).
1. Introduction
In this paper, we tackle the recently introduced video in-
stance segmentation problem [42]. This task requires seg-
menting all instances of a predeﬁned set of object classes
in each frame, classifying them, and linking individual in-
stances over the entire sequence.
In recent years, convolutional networks have obtained re-
markable results in still-image object detection [16, 33, 14,
15], and segmentation [27, 45, 8, 4]. However, extending
these models to video instance segmentation is challeng-
ing. In order to localize objects precisely, these methods
have to operate at very large spatial resolution. As a result,
detectors based on the popular ResNet-101 or ResNet-152
backbones [17] can rarely ﬁt more than one image per GPU
during training. In the context of video instance segmenta-
tion this is problematic because tracking objects over time
requires analyzing multiple video frames simultaneously.
To address this issue, one could reduce the spatial res-
Figure 1: In this paper, we tackle the problem of video in-
stance segmentation, which requires classifying, segment-
ing, and tracking object instances in a given video sequence.
Our proposed Mask Propagation framework (MaskProp)
provides a simple and effective way for solving this task.
olution of the input and ﬁt more video frames in a GPU.
However, doing so typically leads to a signiﬁcant drop in
segmentation or detection performance. Alternatively, one
could perform high-resolution instance segmentation on in-
dividual frames and then link segmentations temporally in
a separate post-processing stage. However, performing in-
stance segmentation and tracking in two disjoint steps of-
ten produces suboptimal results, because these two tasks
are closely intertwined. The key challenge then becomes
designing a uniﬁed model that can track objects in video
while maintaining strong detection accuracy.
Currently, the best method for video instance segmen-
tation is the ICCV 2019 challenge winner [28]. It tackles
video instance segmentation by dividing it into four prob-
lems: 1) detection, 2) classiﬁcation, 3) segmentation, and
4) tracking. These four problems are solved independently
using several off-the-shelf components and their respective
solutions are combined and adapted to the video instance
segmentation task. However, despite effective performance,
such an approach is disadvantageous because it requires de-
signing and tuning a separate model (or, in some cases, an
ensemble of models) for each of the four tasks. This ren-
ders the approach costly and cumbersome. On the other
end of the complexity spectrum, MaskTrack R-CNN [42] is
a simple uniﬁed approach trained end-to-end but it achieves
signiﬁcantly lower performance ( 30:3vs44:8video mAP).
To address the shortcomings of these prior methods wearXiv:1912.04573v4  [cs.CV]  9 Jul 2021
MaskTrack R-CNN [42] ICCV19 Challenge Winner [28] MaskProp
ModelClassiﬁcation cls head Mask R-CNN [16], ResNeXt-101 32x48d [30] cls head
Localization bbox head Mask R-CNN [16] bbox head
Segmentation mask head DeepLabv3 [9], Box2Seg [29] mask head
Tracking tracking head UnOVOST [47], ReID Net [18, 31] mask propagation head
Optical Flow - PWC-Net [35] -
Pre-training DatasetsImageNet [34] (1.3M images) X X X
COCO [25] (860K bboxes) X X X
Instagram [30] (1B images) - X -
OpenImages [23] (14M bboxes) - X -
Performancevideo mAP 30.3 44.8 46.6
video AP@75 32.6 48.9 51.2
Table 1: A table comparing our work to prior video instance segmentation methods [42, 28]. The ICCV 2019 Challenge
Winner [28] decomposes video instance segmentation into four different problems, solves each of them independently using
ensembles of different models, and then combines these solutions. In contrast, our approach relies on a single uniﬁed model
trained end-to-end. Despite being simpler, and using several orders of magnitude less pretraining data (1.3M vs 1B images
and 860K vs 14M bounding boxes) our model achieves higher accuracy. Furthermore, compared to MaskTrack R-CNN [42],
our work yields a 16.3% gain in mAP (46.6% vs 30.3%).
introduce MaskProp, a simple mask propagation framework
for simultaneously classifying, segmenting and tracking ob-
ject instances in video. Our method adapts the popular
Mask R-CNN [16] to video by adding a branch that prop-
agates frame-level instance masks from each video frame
to other frames within a temporal neighborhood (which we
refer to as a clip). This allows our method to compute clip-
level instance tracks centered at each individual frame of the
video. These densely estimated clip-level tracks are then
aggregated to form accurate and coherent object instance
sequences for the entire video, regardless of its length. This
renders our approach capable of handling challenging cases
of occlusions, disocclusions, and motion blur. Our method
achieves the best reported accuracy on the YouTube-VIS
dataset [42], outperforming the ICCV 2019 challenge win-
ner [28] despite being much simpler and using signiﬁcantly
less labeled data (1000x fewer images and 10x fewer bound-
ing boxes). In Table 1, we compare our approach vs these
prior methods in terms of accuracy and other characteristics.
2. Related Work
Instance Segmentation in Images. Compared to instance
segmentation in images [11, 16, 44, 1, 2, 22, 26], the prob-
lem considered in this paper requires not only to segment
object instances in individual frames, but also to determine
instance correspondences across multiple frames. We lever-
age the Mask R-CNN model [16] for still-image instance
segmentation and adapt it to track object instances in video.
Object Detection in Video. Object detection in video re-
quires classifying and localizing objects in every frame of
a given video. Most modern video object detection sys-
tems [46, 5, 40, 13] implement some form of spatiotem-
poral feature alignment for improving object detection ac-
curacy in individual video frames. However, these systems
are typically not designed for tracking object instances. In
contrast, our mask propagation produces clip-level instancesegmentations rather than frame-level bounding boxes.
Video Object Segmentation. The task of video object
segmentation requires segmenting foreground objects in a
class-agnostic fashion [43, 21, 36, 38], often by leveraging
ground truth masks available for the ﬁrst frame during in-
ference [6, 32, 10, 19, 37]. Instead, video instance segmen-
tation requires ﬁnding all instances of a predeﬁned set of
object classes in each frame, classifying them and linking
them over the entire sequence.
Video Instance Segmentation. The recently introduced
video instance segmentation task [42] requires classifying,
segmenting and tracking object instances in videos. This
is the task considered in this work. There are only a few
video instance segmentation methods we can compare our
approach to. The MaskTrack R-CNN [42] presents a uni-
ﬁed model for video instance segmentation. It augments the
original Mask R-CNN [16] with a tracking branch that es-
tablishes associations among object instances segmented in
separate frames. Furthermore, we include the ICCV 2019
video instance segmentation challenge winner [28] in our
comparison. This approach divides video instance segmen-
tation into four separate subproblems: classiﬁcation, de-
tection, segmentation, and tracking. A separate model (or
an ensemble of models) is used to solve each of these sub-
problems, and these solutions are then combined to produce
video instance segmentation results. For brevity, from now
on we refer to it as EnsembleVIS to indicate that it is an en-
semble approach designed for video instance segmentation.
Our MaskProp framework provides advantages over
both of these methods [42, 28]. Similarly to MaskTrack
R-CNN [42], our method is a uniﬁed and simple approach.
However, our mask propagation branch is much more effec-
tive than the tracking branch of MaskTrack R-CNN, achiev-
ing much higher accuracy relative to this baseline. Fur-
thermore, compared to EnsembleVIS [28], our method 1)
is much simpler, 2) uses signiﬁcantly less labeled data, and
     clip-level     instance tracktemporally propagated instance featuresfeature tensorinstance feature computation at time t    frame-level    instance mask time ttime t⌦<latexit sha1_base64="f9ZxFFopka18yU5lyTwVt2fcbbg=">AAAB7nicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNnJbDJkdmaZ6RXCko/w4kERr36PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+m/ntJ26s0OoRJykPEzpUIhaMopPaPY0i4bZfqfo1fw6ySoKCVKFAo1/56g00yxKukElqbTfwUwxzalAwyaflXmZ5StmYDnnXUUXdkjCfnzsl504ZkFgbVwrJXP09kdPE2kkSuc6E4sguezPxP6+bYXwT5kKlGXLFFoviTBLUZPY7GQjDGcqJI5QZ4W4lbEQNZegSKrsQguWXV0nrshb4teDhqlq/LeIowSmcwQUEcA11uIcGNIHBGJ7hFd681Hvx3r2PReuaV8ycwB94nz+F64+t</latexit><latexit sha1_base64="f9ZxFFopka18yU5lyTwVt2fcbbg=">AAAB7nicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNnJbDJkdmaZ6RXCko/w4kERr36PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+m/ntJ26s0OoRJykPEzpUIhaMopPaPY0i4bZfqfo1fw6ySoKCVKFAo1/56g00yxKukElqbTfwUwxzalAwyaflXmZ5StmYDnnXUUXdkjCfnzsl504ZkFgbVwrJXP09kdPE2kkSuc6E4sguezPxP6+bYXwT5kKlGXLFFoviTBLUZPY7GQjDGcqJI5QZ4W4lbEQNZegSKrsQguWXV0nrshb4teDhqlq/LeIowSmcwQUEcA11uIcGNIHBGJ7hFd681Hvx3r2PReuaV8ycwB94nz+F64+t</latexit><latexit sha1_base64="f9ZxFFopka18yU5lyTwVt2fcbbg=">AAAB7nicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNnJbDJkdmaZ6RXCko/w4kERr36PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+m/ntJ26s0OoRJykPEzpUIhaMopPaPY0i4bZfqfo1fw6ySoKCVKFAo1/56g00yxKukElqbTfwUwxzalAwyaflXmZ5StmYDnnXUUXdkjCfnzsl504ZkFgbVwrJXP09kdPE2kkSuc6E4sguezPxP6+bYXwT5kKlGXLFFoviTBLUZPY7GQjDGcqJI5QZ4W4lbEQNZegSKrsQguWXV0nrshb4teDhqlq/LeIowSmcwQUEcA11uIcGNIHBGJ7hFd681Hvx3r2PReuaV8ycwB94nz+F64+t</latexit><latexit sha1_base64="f9ZxFFopka18yU5lyTwVt2fcbbg=">AAAB7nicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNnJbDJkdmaZ6RXCko/w4kERr36PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+m/ntJ26s0OoRJykPEzpUIhaMopPaPY0i4bZfqfo1fw6ySoKCVKFAo1/56g00yxKukElqbTfwUwxzalAwyaflXmZ5StmYDnnXUUXdkjCfnzsl504ZkFgbVwrJXP09kdPE2kkSuc6E4sguezPxP6+bYXwT5kKlGXLFFoviTBLUZPY7GQjDGcqJI5QZ4W4lbEQNZegSKrsQguWXV0nrshb4teDhqlq/LeIowSmcwQUEcA11uIcGNIHBGJ7hFd681Hvx3r2PReuaV8ycwB94nz+F64+t</latexit>time t+5time t-5time t-10
time ttime t+10time t-5time t+5time t+10time ttime t-10propagated instance        segmentationfeature tensors for each frame             in a video clip
time t-5time t+5time t+10time ttime t-10
Figure 2: An illustration of our MaskProp system, which takes as input a video clip centered around frame t, and outputs
a clip-level instance track. Our mask propagation framework can be summarized in three high-level steps: 1) An instance-
speciﬁc feature map at time tis computed by masking the frame features at time twith the given instance segmentation
for framet(one for each instance detected in frame t). 2) Next, we use our mask propagation mechanism to temporally
propagate instance-speciﬁc features from frame tto all the other frames in the clip. 3) Lastly, our model predicts instance-
speciﬁc segmentations in every frame of the clip by implicitly matching the propagated instance features with the frame-level
features computed at each time step. This last step yields clip-level instance tracks centered around frame t.
3) produces higher accuracy on YouTube-VIS [42].
3. Video Instance Segmentation
Problem Deﬁnition. Let us denote with V2RL3HW
an input video consisting of LRGB frames of spatial size
HW. The aim of our system is to segment and temporally
link all object instances that are visible for at least one frame
inVand that belong to a predeﬁned set of categories C=
f1;:::;Kg. To achieve this goal, our model outputs a video-
level instance mask track Mi2RLHWwith a category
labelci2f1;:::;Kgand a conﬁdence score si2[0;1]for
each object instance idetected in the video.
Evaluation Metric. Video instance segmentation is evalu-
ated according to the metrics of average precision (AP) and
average recall (AR). Unlike in the image domain, these met-
rics are evaluated over the video sequence. Thus, to eval-
uate spatiotemporal consistency of the predicted mask se-
quences, the video Intersection over Union (IoU) between
a predicted object instance iand a ground truth object in-
stancejis computed as:
IoU(i;j) =PT
t=1jMi(t)\~Mj(t)j
PT
t=1jMi(t)[~Mj(t)j(1)
where ~Mj(t)is the ground-truth segmentation of object j
in framet. To achieve a large IoU, a model must not only
accurately classify and segment object instances at a frame-
level, but also reliably track them over the video sequence.
As in the COCO benchmark for image segmenta-
tion [25], the metrics of AP and AR are computed separatelyfor each object category, and then averaged over ten IoU
thresholds from 50% to95% at increments of 5%. Lastly,
the resulting AP and AR metrics are averaged over the cat-
egory set, which yields the ﬁnal evaluation metric.
4. Mask Propagation
MaskProp takes a video Vof arbitrary length Las in-
put and outputs video-level instance segmentation tracks
Mi, category labels ciand conﬁdence scores sifor all
objectsidetected in the video. In order to achieve this
goal, our method ﬁrst builds clip-level object instance tracks
Mi
tT:t+T2 R(2T+1)1HWfor each individual clip
VtT:t+T2R(2T+1)3HWof length (2T+ 1) in the
video, i.e., for t= 1;2;:::;L (clips at the beginning and
the end of the video will include fewer frames).
We want to use clips that are long enough to allow us
to jointly solve instance segmentation and tracking while
handling challenging cases of occlusion and motion blur.
At the same time, the clip should be short enough to allow
us to ﬁt it at high spatial resolution in the memory of a GPU.
The resulting clip-level instance masks Mi
tT:t+Tpro-
duced densely for all overlapping clips t= 1;:::;L are
then aggregated to produce video-level instance masks Mi.
Our approach for clip-level instance segmentation is de-
scribed in subsections 4.1 and 4.2. We also illustrate it in
Figure 2. The subsequent clip-level instance mask aggrega-
tion method is presented in subsection 4.3.
Frame t
Bbox...
ClsMask
Frame t+...     Mask  Propagation     CNN
CNN<latexit sha1_base64="AU/vq+DAgMm1x9RhUc20aA7BJwY=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3Q+5RDqo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB00oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4AkYyPHA==</latexit><latexit sha1_base64="AU/vq+DAgMm1x9RhUc20aA7BJwY=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3Q+5RDqo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB00oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4AkYyPHA==</latexit><latexit sha1_base64="AU/vq+DAgMm1x9RhUc20aA7BJwY=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3Q+5RDqo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB00oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4AkYyPHA==</latexit><latexit sha1_base64="AU/vq+DAgMm1x9RhUc20aA7BJwY=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3Q+5RDqo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB00oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4AkYyPHA==</latexit>
Figure 3: We adapt Mask R-CNN [16] to video by adding
a mask propagation branch, for tracking object segmenta-
tion instances in video. Given a video clip centered around
framet, our system outputs a clip-level instance segmenta-
tion track as well as a classiﬁcation score and a bounding
box for each object instanced detected in frame t. For com-
pactness, in this ﬁgure, we illustrate our system processing a
pair of frames but typically the propagation is applied from
the middle frame to all the other frames in the clip.
4.1. Video Mask R-CNN
Our video instance segmentation system is based on the
Mask R-CNN [16] model, which we adapt to video by
adding a mask propagation branch (See Figure 3). We train
our system with a multi-task loss Lt=Lcls
t+Lbox
t+
Lmask
t +Lprop
t wheretdenotes a time-step of a center
frame. We use identical loss terms Lcls
t;Lbox
t,Lmask
t as
in Mask R-CNN. The mask propagation loss is deﬁned as:
Lprop
t =~NtX
it+TX
t0=tT1sIoU (Mi
tT:t+T(t0);~Mi
tT:t+T(t0))(2)
whereMi
tT:t+T(t0)2[0;1]is the segmentation at time
t0for an instance ipredicted from a clip centered at tand
~Mi
tT:t+T(t0)is the corresponding ground truth mask at
timet0.~Ntis the number of ground truth object instances
in framet, andsIoU is deﬁned as:
sIoU (A;B) =P
pA(p)B(p)P
pA(p) +B(p)A(p)B(p)(3)
where the summations in numerator and denominator are
performed over every pixel location p. The loss above is a
soft IoU loss, which we observed to work slightly better
than the standard cross entropy loss for our task.
4.2. Mask Propagation Branch
Overview. Our main technical contribution is the design of
a mask propagation branch, that allows our method to track
object instances. Given a video clip VtT:t+Tcenteredat framet, our system outputs clip-level instance masks
Mi
tT:t+Tfor each predicted object instance iin framet.
Our mask propagation branch can be described in three
high-level steps: 1) instance-speciﬁc feature computation,
2) temporal propagation of instance features, and 3) propa-
gated instance segmentation. We will now describe each of
these steps in more detail. We introduce our mask propaga-
tion with the example of propagating object instance masks
from frametto framet+where2[T:T].
Computing Instance Speciﬁc Features. The mask
branch of our model predicts frame-level instance masks
Mi
t2R1H0W0from single frame inputs. We then
use these frame-level instance masks to compute instance-
speciﬁc features for frame t. Speciﬁcally, for each object in-
stancei, we compute an element-wise product between Mi
t
and the feature tensor from the backbone network ft. This
then yields a set of new feature tensors fi
t2RCH0W0,
wherei= 1;::;Nt, andNtis the number of object instances
detected in frame t. In other words, for each object instance
i, we are zeroing out the feature values in ftthat correspond
to pixels not belonging to that object instance.
Temporally Propagating Instance Features. Given
frame-level features ft;ft+2RCH0W0and instance-
speciﬁc feature tensor fi
t, our method generates a propa-
gated instance feature tensor gi
t;t+. Intuitively, gi
t;t+rep-
resents the features predicted by our model for object in-
stanceiin framet+from an instance-speciﬁc feature ten-
sorfi
t. The tensor gi
t;t+is generated by warping features
fi
tusing the alignment computed from frame-level features
ftandft+. We implement the propagation mechanism via
a deformable convolution [12], which has previously been
used for aligning features computed from separate frames
of a video [5, 3]. Speciﬁcally, we compute the element-
wise difference of tensors ft;ft+and feed it through a
simple residual block [17], which predicts motion offsets
ot;t+2R2k2H0W0. These offsets contain (x;y)sam-
pling locations for each entry of a kkdeformable convo-
lution kernel [12]. The propagation step takes as inputs 1)
the offsetsot;t+and 2) the instance feature tensor fi
t, and
then applies deformable convolution to output the propa-
gated instance feature tensor gi
t;t+for each instance i. We
use subscript t;t+to denote the propagated instance fea-
ture because, although gis obtained by propagating the fea-
ture tensorfi
t, the offset computation uses both frame tand
framet+. We stress that no explicit ground truth align-
ment is available between frames. The deformable convolu-
tional kernels are supervised implicitly by optimizing Eq. 2.
Segmenting Propagated Instances. Lastly, we use our
propagated feature map gi
t;t+for predicting a correspond-
ing object instance mask in frame t+. To do this we
ﬁrst, construct a new feature tensor i
t;t+=gi
t;t++ft+.
The addition effectively overimposes the tensor gi
t;t+pre-
dicted from time tfor object instance iin framet+, with
Tensor t
 <latexit sha1_base64="xbJVLRy5zV8jtLsjAis5DXwMT4U=">AAAB7nicbVDJSgNBEK1xjXGLevTSGARPYUYEPQa9eIxgFkiG0NPpSZr0MvQihCEf4cWDIl79Hm/+jZ1kDpr4oODxXhVV9ZKMM2PD8DtYW9/Y3Nou7ZR39/YPDitHxy2jnCa0SRRXupNgQzmTtGmZ5bSTaYpFwmk7Gd/N/PYT1YYp+WgnGY0FHkqWMoKtl9o9JZh0pl+phrVwDrRKooJUoUCjX/nqDRRxgkpLODamG4WZjXOsLSOcTss9Z2iGyRgPaddTiQU1cT4/d4rOvTJAqdK+pEVz9fdEjoUxE5H4ToHtyCx7M/E/r+tsehPnTGbOUkkWi1LHkVVo9jsaME2J5RNPMNHM34rICGtMrE+o7EOIll9eJa3LWhTWooerav22iKMEp3AGFxDBNdThHhrQBAJjeIZXeAuy4CV4Dz4WrWtBMXMCfxB8/gCVCY+3</latexit><latexit sha1_base64="xbJVLRy5zV8jtLsjAis5DXwMT4U=">AAAB7nicbVDJSgNBEK1xjXGLevTSGARPYUYEPQa9eIxgFkiG0NPpSZr0MvQihCEf4cWDIl79Hm/+jZ1kDpr4oODxXhVV9ZKMM2PD8DtYW9/Y3Nou7ZR39/YPDitHxy2jnCa0SRRXupNgQzmTtGmZ5bSTaYpFwmk7Gd/N/PYT1YYp+WgnGY0FHkqWMoKtl9o9JZh0pl+phrVwDrRKooJUoUCjX/nqDRRxgkpLODamG4WZjXOsLSOcTss9Z2iGyRgPaddTiQU1cT4/d4rOvTJAqdK+pEVz9fdEjoUxE5H4ToHtyCx7M/E/r+tsehPnTGbOUkkWi1LHkVVo9jsaME2J5RNPMNHM34rICGtMrE+o7EOIll9eJa3LWhTWooerav22iKMEp3AGFxDBNdThHhrQBAJjeIZXeAuy4CV4Dz4WrWtBMXMCfxB8/gCVCY+3</latexit><latexit sha1_base64="xbJVLRy5zV8jtLsjAis5DXwMT4U=">AAAB7nicbVDJSgNBEK1xjXGLevTSGARPYUYEPQa9eIxgFkiG0NPpSZr0MvQihCEf4cWDIl79Hm/+jZ1kDpr4oODxXhVV9ZKMM2PD8DtYW9/Y3Nou7ZR39/YPDitHxy2jnCa0SRRXupNgQzmTtGmZ5bSTaYpFwmk7Gd/N/PYT1YYp+WgnGY0FHkqWMoKtl9o9JZh0pl+phrVwDrRKooJUoUCjX/nqDRRxgkpLODamG4WZjXOsLSOcTss9Z2iGyRgPaddTiQU1cT4/d4rOvTJAqdK+pEVz9fdEjoUxE5H4ToHtyCx7M/E/r+tsehPnTGbOUkkWi1LHkVVo9jsaME2J5RNPMNHM34rICGtMrE+o7EOIll9eJa3LWhTWooerav22iKMEp3AGFxDBNdThHhrQBAJjeIZXeAuy4CV4Dz4WrWtBMXMCfxB8/gCVCY+3</latexit><latexit sha1_base64="xbJVLRy5zV8jtLsjAis5DXwMT4U=">AAAB7nicbVDJSgNBEK1xjXGLevTSGARPYUYEPQa9eIxgFkiG0NPpSZr0MvQihCEf4cWDIl79Hm/+jZ1kDpr4oODxXhVV9ZKMM2PD8DtYW9/Y3Nou7ZR39/YPDitHxy2jnCa0SRRXupNgQzmTtGmZ5bSTaYpFwmk7Gd/N/PYT1YYp+WgnGY0FHkqWMoKtl9o9JZh0pl+phrVwDrRKooJUoUCjX/nqDRRxgkpLODamG4WZjXOsLSOcTss9Z2iGyRgPaddTiQU1cT4/d4rOvTJAqdK+pEVz9fdEjoUxE5H4ToHtyCx7M/E/r+tsehPnTGbOUkkWi1LHkVVo9jsaME2J5RNPMNHM34rICGtMrE+o7EOIll9eJa3LWhTWooerav22iKMEp3AGFxDBNdThHhrQBAJjeIZXeAuy4CV4Dz4WrWtBMXMCfxB8/gCVCY+3</latexit>Tensor t+  <latexit sha1_base64="AU/vq+DAgMm1x9RhUc20aA7BJwY=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3Q+5RDqo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB00oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4AkYyPHA==</latexit><latexit sha1_base64="AU/vq+DAgMm1x9RhUc20aA7BJwY=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3Q+5RDqo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB00oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4AkYyPHA==</latexit><latexit sha1_base64="AU/vq+DAgMm1x9RhUc20aA7BJwY=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3Q+5RDqo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB00oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4AkYyPHA==</latexit><latexit sha1_base64="AU/vq+DAgMm1x9RhUc20aA7BJwY=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3Q+5RDqo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB00oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4AkYyPHA==</latexit> Residual      Block
⌦<latexit sha1_base64="f9ZxFFopka18yU5lyTwVt2fcbbg=">AAAB7nicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNnJbDJkdmaZ6RXCko/w4kERr36PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+m/ntJ26s0OoRJykPEzpUIhaMopPaPY0i4bZfqfo1fw6ySoKCVKFAo1/56g00yxKukElqbTfwUwxzalAwyaflXmZ5StmYDnnXUUXdkjCfnzsl504ZkFgbVwrJXP09kdPE2kkSuc6E4sguezPxP6+bYXwT5kKlGXLFFoviTBLUZPY7GQjDGcqJI5QZ4W4lbEQNZegSKrsQguWXV0nrshb4teDhqlq/LeIowSmcwQUEcA11uIcGNIHBGJ7hFd681Hvx3r2PReuaV8ycwB94nz+F64+t</latexit><latexit sha1_base64="f9ZxFFopka18yU5lyTwVt2fcbbg=">AAAB7nicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNnJbDJkdmaZ6RXCko/w4kERr36PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+m/ntJ26s0OoRJykPEzpUIhaMopPaPY0i4bZfqfo1fw6ySoKCVKFAo1/56g00yxKukElqbTfwUwxzalAwyaflXmZ5StmYDnnXUUXdkjCfnzsl504ZkFgbVwrJXP09kdPE2kkSuc6E4sguezPxP6+bYXwT5kKlGXLFFoviTBLUZPY7GQjDGcqJI5QZ4W4lbEQNZegSKrsQguWXV0nrshb4teDhqlq/LeIowSmcwQUEcA11uIcGNIHBGJ7hFd681Hvx3r2PReuaV8ycwB94nz+F64+t</latexit><latexit sha1_base64="f9ZxFFopka18yU5lyTwVt2fcbbg=">AAAB7nicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNnJbDJkdmaZ6RXCko/w4kERr36PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+m/ntJ26s0OoRJykPEzpUIhaMopPaPY0i4bZfqfo1fw6ySoKCVKFAo1/56g00yxKukElqbTfwUwxzalAwyaflXmZ5StmYDnnXUUXdkjCfnzsl504ZkFgbVwrJXP09kdPE2kkSuc6E4sguezPxP6+bYXwT5kKlGXLFFoviTBLUZPY7GQjDGcqJI5QZ4W4lbEQNZegSKrsQguWXV0nrshb4teDhqlq/LeIowSmcwQUEcA11uIcGNIHBGJ7hFd681Hvx3r2PReuaV8ycwB94nz+F64+t</latexit><latexit sha1_base64="f9ZxFFopka18yU5lyTwVt2fcbbg=">AAAB7nicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNnJbDJkdmaZ6RXCko/w4kERr36PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+m/ntJ26s0OoRJykPEzpUIhaMopPaPY0i4bZfqfo1fw6ySoKCVKFAo1/56g00yxKukElqbTfwUwxzalAwyaflXmZ5StmYDnnXUUXdkjCfnzsl504ZkFgbVwrJXP09kdPE2kkSuc6E4sguezPxP6+bYXwT5kKlGXLFFoviTBLUZPY7GQjDGcqJI5QZ4W4lbEQNZegSKrsQguWXV0nrshb4teDhqlq/LeIowSmcwQUEcA11uIcGNIHBGJ7hFd681Hvx3r2PReuaV8ycwB94nz+F64+t</latexit>⌦<latexit sha1_base64="f9ZxFFopka18yU5lyTwVt2fcbbg=">AAAB7nicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNnJbDJkdmaZ6RXCko/w4kERr36PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+m/ntJ26s0OoRJykPEzpUIhaMopPaPY0i4bZfqfo1fw6ySoKCVKFAo1/56g00yxKukElqbTfwUwxzalAwyaflXmZ5StmYDnnXUUXdkjCfnzsl504ZkFgbVwrJXP09kdPE2kkSuc6E4sguezPxP6+bYXwT5kKlGXLFFoviTBLUZPY7GQjDGcqJI5QZ4W4lbEQNZegSKrsQguWXV0nrshb4teDhqlq/LeIowSmcwQUEcA11uIcGNIHBGJ7hFd681Hvx3r2PReuaV8ycwB94nz+F64+t</latexit><latexit sha1_base64="f9ZxFFopka18yU5lyTwVt2fcbbg=">AAAB7nicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNnJbDJkdmaZ6RXCko/w4kERr36PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+m/ntJ26s0OoRJykPEzpUIhaMopPaPY0i4bZfqfo1fw6ySoKCVKFAo1/56g00yxKukElqbTfwUwxzalAwyaflXmZ5StmYDnnXUUXdkjCfnzsl504ZkFgbVwrJXP09kdPE2kkSuc6E4sguezPxP6+bYXwT5kKlGXLFFoviTBLUZPY7GQjDGcqJI5QZ4W4lbEQNZegSKrsQguWXV0nrshb4teDhqlq/LeIowSmcwQUEcA11uIcGNIHBGJ7hFd681Hvx3r2PReuaV8ycwB94nz+F64+t</latexit><latexit sha1_base64="f9ZxFFopka18yU5lyTwVt2fcbbg=">AAAB7nicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNnJbDJkdmaZ6RXCko/w4kERr36PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+m/ntJ26s0OoRJykPEzpUIhaMopPaPY0i4bZfqfo1fw6ySoKCVKFAo1/56g00yxKukElqbTfwUwxzalAwyaflXmZ5StmYDnnXUUXdkjCfnzsl504ZkFgbVwrJXP09kdPE2kkSuc6E4sguezPxP6+bYXwT5kKlGXLFFoviTBLUZPY7GQjDGcqJI5QZ4W4lbEQNZegSKrsQguWXV0nrshb4teDhqlq/LeIowSmcwQUEcA11uIcGNIHBGJ7hFd681Hvx3r2PReuaV8ycwB94nz+F64+t</latexit><latexit sha1_base64="f9ZxFFopka18yU5lyTwVt2fcbbg=">AAAB7nicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNnJbDJkdmaZ6RXCko/w4kERr36PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+m/ntJ26s0OoRJykPEzpUIhaMopPaPY0i4bZfqfo1fw6ySoKCVKFAo1/56g00yxKukElqbTfwUwxzalAwyaflXmZ5StmYDnnXUUXdkjCfnzsl504ZkFgbVwrJXP09kdPE2kkSuc6E4sguezPxP6+bYXwT5kKlGXLFFoviTBLUZPY7GQjDGcqJI5QZ4W4lbEQNZegSKrsQguWXV0nrshb4teDhqlq/LeIowSmcwQUEcA11uIcGNIHBGJ7hFd681Hvx3r2PReuaV8ycwB94nz+F64+t</latexit>   Deform. Conv. Offsets<latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit><latexit sha1_base64="N4ztu7rA6GcWPA6nADZ8tMOcvqE=">AAAB7XicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48RzAOSJcxOZpMxszPLTK8QQv7BiwdFvPo/3vwbJ8keNLGgoajqprsrSqWw6PvfXmFtfWNzq7hd2tnd2z8oHx41rc4M4w2mpTbtiFouheINFCh5OzWcJpHkrWh0O/NbT9xYodUDjlMeJnSgRCwYRSc1uzqVme2VK37Vn4OskiAnFchR75W/un3NsoQrZJJa2wn8FMMJNSiY5NNSN7M8pWxEB7zjqKIJt+Fkfu2UnDmlT2JtXCkkc/X3xIQm1o6TyHUmFId22ZuJ/3mdDOPrcCJUmiFXbLEoziRBTWavk74wnKEcO0KZEe5WwobUUIYuoJILIVh+eZU0L6qBXw3uLyu1mzyOIpzAKZxDAFdQgzuoQwMYPMIzvMKbp70X7937WLQWvHzmGP7A+/wBz/6PRQ==</latexit>
   2D Conv. 
   Softmax   Attention⌦<latexit sha1_base64="f9ZxFFopka18yU5lyTwVt2fcbbg=">AAAB7nicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNnJbDJkdmaZ6RXCko/w4kERr36PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+m/ntJ26s0OoRJykPEzpUIhaMopPaPY0i4bZfqfo1fw6ySoKCVKFAo1/56g00yxKukElqbTfwUwxzalAwyaflXmZ5StmYDnnXUUXdkjCfnzsl504ZkFgbVwrJXP09kdPE2kkSuc6E4sguezPxP6+bYXwT5kKlGXLFFoviTBLUZPY7GQjDGcqJI5QZ4W4lbEQNZegSKrsQguWXV0nrshb4teDhqlq/LeIowSmcwQUEcA11uIcGNIHBGJ7hFd681Hvx3r2PReuaV8ycwB94nz+F64+t</latexit><latexit sha1_base64="f9ZxFFopka18yU5lyTwVt2fcbbg=">AAAB7nicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNnJbDJkdmaZ6RXCko/w4kERr36PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+m/ntJ26s0OoRJykPEzpUIhaMopPaPY0i4bZfqfo1fw6ySoKCVKFAo1/56g00yxKukElqbTfwUwxzalAwyaflXmZ5StmYDnnXUUXdkjCfnzsl504ZkFgbVwrJXP09kdPE2kkSuc6E4sguezPxP6+bYXwT5kKlGXLFFoviTBLUZPY7GQjDGcqJI5QZ4W4lbEQNZegSKrsQguWXV0nrshb4teDhqlq/LeIowSmcwQUEcA11uIcGNIHBGJ7hFd681Hvx3r2PReuaV8ycwB94nz+F64+t</latexit><latexit sha1_base64="f9ZxFFopka18yU5lyTwVt2fcbbg=">AAAB7nicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNnJbDJkdmaZ6RXCko/w4kERr36PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+m/ntJ26s0OoRJykPEzpUIhaMopPaPY0i4bZfqfo1fw6ySoKCVKFAo1/56g00yxKukElqbTfwUwxzalAwyaflXmZ5StmYDnnXUUXdkjCfnzsl504ZkFgbVwrJXP09kdPE2kkSuc6E4sguezPxP6+bYXwT5kKlGXLFFoviTBLUZPY7GQjDGcqJI5QZ4W4lbEQNZegSKrsQguWXV0nrshb4teDhqlq/LeIowSmcwQUEcA11uIcGNIHBGJ7hFd681Hvx3r2PReuaV8ycwB94nz+F64+t</latexit><latexit sha1_base64="f9ZxFFopka18yU5lyTwVt2fcbbg=">AAAB7nicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNnJbDJkdmaZ6RXCko/w4kERr36PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+m/ntJ26s0OoRJykPEzpUIhaMopPaPY0i4bZfqfo1fw6ySoKCVKFAo1/56g00yxKukElqbTfwUwxzalAwyaflXmZ5StmYDnnXUUXdkjCfnzsl504ZkFgbVwrJXP09kdPE2kkSuc6E4sguezPxP6+bYXwT5kKlGXLFFoviTBLUZPY7GQjDGcqJI5QZ4W4lbEQNZegSKrsQguWXV0nrshb4teDhqlq/LeIowSmcwQUEcA11uIcGNIHBGJ7hFd681Hvx3r2PReuaV8ycwB94nz+F64+t</latexit>⌦<latexit sha1_base64="f9ZxFFopka18yU5lyTwVt2fcbbg=">AAAB7nicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNnJbDJkdmaZ6RXCko/w4kERr36PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+m/ntJ26s0OoRJykPEzpUIhaMopPaPY0i4bZfqfo1fw6ySoKCVKFAo1/56g00yxKukElqbTfwUwxzalAwyaflXmZ5StmYDnnXUUXdkjCfnzsl504ZkFgbVwrJXP09kdPE2kkSuc6E4sguezPxP6+bYXwT5kKlGXLFFoviTBLUZPY7GQjDGcqJI5QZ4W4lbEQNZegSKrsQguWXV0nrshb4teDhqlq/LeIowSmcwQUEcA11uIcGNIHBGJ7hFd681Hvx3r2PReuaV8ycwB94nz+F64+t</latexit><latexit sha1_base64="f9ZxFFopka18yU5lyTwVt2fcbbg=">AAAB7nicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNnJbDJkdmaZ6RXCko/w4kERr36PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+m/ntJ26s0OoRJykPEzpUIhaMopPaPY0i4bZfqfo1fw6ySoKCVKFAo1/56g00yxKukElqbTfwUwxzalAwyaflXmZ5StmYDnnXUUXdkjCfnzsl504ZkFgbVwrJXP09kdPE2kkSuc6E4sguezPxP6+bYXwT5kKlGXLFFoviTBLUZPY7GQjDGcqJI5QZ4W4lbEQNZegSKrsQguWXV0nrshb4teDhqlq/LeIowSmcwQUEcA11uIcGNIHBGJ7hFd681Hvx3r2PReuaV8ycwB94nz+F64+t</latexit><latexit sha1_base64="f9ZxFFopka18yU5lyTwVt2fcbbg=">AAAB7nicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNnJbDJkdmaZ6RXCko/w4kERr36PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+m/ntJ26s0OoRJykPEzpUIhaMopPaPY0i4bZfqfo1fw6ySoKCVKFAo1/56g00yxKukElqbTfwUwxzalAwyaflXmZ5StmYDnnXUUXdkjCfnzsl504ZkFgbVwrJXP09kdPE2kkSuc6E4sguezPxP6+bYXwT5kKlGXLFFoviTBLUZPY7GQjDGcqJI5QZ4W4lbEQNZegSKrsQguWXV0nrshb4teDhqlq/LeIowSmcwQUEcA11uIcGNIHBGJ7hFd681Hvx3r2PReuaV8ycwB94nz+F64+t</latexit><latexit sha1_base64="f9ZxFFopka18yU5lyTwVt2fcbbg=">AAAB7nicbVDLSgNBEOz1GeMr6tHLYBA8hV0R9Bj04jGCeUCyhNnJbDJkdmaZ6RXCko/w4kERr36PN//GSbIHTSxoKKq66e6KUiks+v63t7a+sbm1Xdop7+7tHxxWjo5bVmeG8SbTUptORC2XQvEmCpS8kxpOk0jydjS+m/ntJ26s0OoRJykPEzpUIhaMopPaPY0i4bZfqfo1fw6ySoKCVKFAo1/56g00yxKukElqbTfwUwxzalAwyaflXmZ5StmYDnnXUUXdkjCfnzsl504ZkFgbVwrJXP09kdPE2kkSuc6E4sguezPxP6+bYXwT5kKlGXLFFoviTBLUZPY7GQjDGcqJI5QZ4W4lbEQNZegSKrsQguWXV0nrshb4teDhqlq/LeIowSmcwQUEcA11uIcGNIHBGJ7hFd681Hvx3r2PReuaV8ycwB94nz+F64+t</latexit>3) Propagated Instance Segmentation (time t+  )<latexit sha1_base64="AU/vq+DAgMm1x9RhUc20aA7BJwY=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3Q+5RDqo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB00oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4AkYyPHA==</latexit><latexit sha1_base64="AU/vq+DAgMm1x9RhUc20aA7BJwY=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3Q+5RDqo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB00oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4AkYyPHA==</latexit><latexit sha1_base64="AU/vq+DAgMm1x9RhUc20aA7BJwY=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3Q+5RDqo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB00oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4AkYyPHA==</latexit><latexit sha1_base64="AU/vq+DAgMm1x9RhUc20aA7BJwY=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3Q+5RDqo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB00oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4AkYyPHA==</latexit>
2) Instance Feature Propagation (from time t to time t+  )<latexit sha1_base64="AU/vq+DAgMm1x9RhUc20aA7BJwY=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3Q+5RDqo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB00oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4AkYyPHA==</latexit><latexit sha1_base64="AU/vq+DAgMm1x9RhUc20aA7BJwY=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3Q+5RDqo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB00oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4AkYyPHA==</latexit><latexit sha1_base64="AU/vq+DAgMm1x9RhUc20aA7BJwY=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3Q+5RDqo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB00oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4AkYyPHA==</latexit><latexit sha1_base64="AU/vq+DAgMm1x9RhUc20aA7BJwY=">AAAB7XicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDabTbt2kw27E6GE/gcvHhTx6v/x5r9x2+agrQ8GHu/NMDMvSKUw6LrfTmltfWNzq7xd2dnd2z+oHh61jco04y2mpNLdgBouRcJbKFDybqo5jQPJO8H4duZ3nrg2QiUPOEm5H9NhIiLBKFqp3Q+5RDqo1ty6OwdZJV5BalCgOah+9UPFspgnyCQ1pue5Kfo51SiY5NNKPzM8pWxMh7xnaUJjbvx8fu2UnFklJJHSthIkc/X3RE5jYyZxYDtjiiOz7M3E/7xehtG1n4skzZAnbLEoyiRBRWavk1BozlBOLKFMC3srYSOqKUMbUMWG4C2/vEraF3XPrXv3l7XGTRFHGU7gFM7BgytowB00oQUMHuEZXuHNUc6L8+58LFpLTjFzDH/gfP4AkYyPHA==</latexit>1) Instance Feature Computation (time t)Instancej
<latexit sha1_base64="ocTLVc574NOAe/DoMycp9A9e/3g=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cW7Ae0oWy2k3bbzSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikpeNUMWyyWMSqE1CNgktsGm4EdhKFNAoEtoPJ/dxvP6HSPJaPZpqgH9Gh5CFn1FipMe6XK27VXYCsEy8nFchR75e/eoOYpRFKwwTVuuu5ifEzqgxnAmelXqoxoWxCh9i1VNIItZ8tDp2RC6sMSBgrW9KQhfp7IqOR1tMosJ0RNSO96s3F/7xuasJbP+MySQ1KtlwUpoKYmMy/JgOukBkxtYQyxe2thI2ooszYbEo2BG/15XXSuqp6btVrXFdqd3kcRTiDc7gED26gBg9QhyYwQHiGV3hzxs6L8+58LFsLTj5zCn/gfP4A0QGM7g==</latexit><latexit sha1_base64="ocTLVc574NOAe/DoMycp9A9e/3g=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cW7Ae0oWy2k3bbzSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikpeNUMWyyWMSqE1CNgktsGm4EdhKFNAoEtoPJ/dxvP6HSPJaPZpqgH9Gh5CFn1FipMe6XK27VXYCsEy8nFchR75e/eoOYpRFKwwTVuuu5ifEzqgxnAmelXqoxoWxCh9i1VNIItZ8tDp2RC6sMSBgrW9KQhfp7IqOR1tMosJ0RNSO96s3F/7xuasJbP+MySQ1KtlwUpoKYmMy/JgOukBkxtYQyxe2thI2ooszYbEo2BG/15XXSuqp6btVrXFdqd3kcRTiDc7gED26gBg9QhyYwQHiGV3hzxs6L8+58LFsLTj5zCn/gfP4A0QGM7g==</latexit><latexit sha1_base64="ocTLVc574NOAe/DoMycp9A9e/3g=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cW7Ae0oWy2k3bbzSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikpeNUMWyyWMSqE1CNgktsGm4EdhKFNAoEtoPJ/dxvP6HSPJaPZpqgH9Gh5CFn1FipMe6XK27VXYCsEy8nFchR75e/eoOYpRFKwwTVuuu5ifEzqgxnAmelXqoxoWxCh9i1VNIItZ8tDp2RC6sMSBgrW9KQhfp7IqOR1tMosJ0RNSO96s3F/7xuasJbP+MySQ1KtlwUpoKYmMy/JgOukBkxtYQyxe2thI2ooszYbEo2BG/15XXSuqp6btVrXFdqd3kcRTiDc7gED26gBg9QhyYwQHiGV3hzxs6L8+58LFsLTj5zCn/gfP4A0QGM7g==</latexit><latexit sha1_base64="ocTLVc574NOAe/DoMycp9A9e/3g=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cW7Ae0oWy2k3bbzSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikpeNUMWyyWMSqE1CNgktsGm4EdhKFNAoEtoPJ/dxvP6HSPJaPZpqgH9Gh5CFn1FipMe6XK27VXYCsEy8nFchR75e/eoOYpRFKwwTVuuu5ifEzqgxnAmelXqoxoWxCh9i1VNIItZ8tDp2RC6sMSBgrW9KQhfp7IqOR1tMosJ0RNSO96s3F/7xuasJbP+MySQ1KtlwUpoKYmMy/JgOukBkxtYQyxe2thI2ooszYbEo2BG/15XXSuqp6btVrXFdqd3kcRTiDc7gED26gBg9QhyYwQHiGV3hzxs6L8+58LFsLTj5zCn/gfP4A0QGM7g==</latexit>Instancei<latexit sha1_base64="9jtkmSnVkzQh6f031uh1XGFh3yI=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipyQflilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/PfYzt</latexit><latexit sha1_base64="9jtkmSnVkzQh6f031uh1XGFh3yI=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipyQflilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/PfYzt</latexit><latexit sha1_base64="9jtkmSnVkzQh6f031uh1XGFh3yI=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipyQflilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/PfYzt</latexit><latexit sha1_base64="9jtkmSnVkzQh6f031uh1XGFh3yI=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipyQflilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/PfYzt</latexit>Instancej
<latexit sha1_base64="ocTLVc574NOAe/DoMycp9A9e/3g=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cW7Ae0oWy2k3bbzSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikpeNUMWyyWMSqE1CNgktsGm4EdhKFNAoEtoPJ/dxvP6HSPJaPZpqgH9Gh5CFn1FipMe6XK27VXYCsEy8nFchR75e/eoOYpRFKwwTVuuu5ifEzqgxnAmelXqoxoWxCh9i1VNIItZ8tDp2RC6sMSBgrW9KQhfp7IqOR1tMosJ0RNSO96s3F/7xuasJbP+MySQ1KtlwUpoKYmMy/JgOukBkxtYQyxe2thI2ooszYbEo2BG/15XXSuqp6btVrXFdqd3kcRTiDc7gED26gBg9QhyYwQHiGV3hzxs6L8+58LFsLTj5zCn/gfP4A0QGM7g==</latexit><latexit sha1_base64="ocTLVc574NOAe/DoMycp9A9e/3g=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cW7Ae0oWy2k3bbzSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikpeNUMWyyWMSqE1CNgktsGm4EdhKFNAoEtoPJ/dxvP6HSPJaPZpqgH9Gh5CFn1FipMe6XK27VXYCsEy8nFchR75e/eoOYpRFKwwTVuuu5ifEzqgxnAmelXqoxoWxCh9i1VNIItZ8tDp2RC6sMSBgrW9KQhfp7IqOR1tMosJ0RNSO96s3F/7xuasJbP+MySQ1KtlwUpoKYmMy/JgOukBkxtYQyxe2thI2ooszYbEo2BG/15XXSuqp6btVrXFdqd3kcRTiDc7gED26gBg9QhyYwQHiGV3hzxs6L8+58LFsLTj5zCn/gfP4A0QGM7g==</latexit><latexit sha1_base64="ocTLVc574NOAe/DoMycp9A9e/3g=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cW7Ae0oWy2k3bbzSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikpeNUMWyyWMSqE1CNgktsGm4EdhKFNAoEtoPJ/dxvP6HSPJaPZpqgH9Gh5CFn1FipMe6XK27VXYCsEy8nFchR75e/eoOYpRFKwwTVuuu5ifEzqgxnAmelXqoxoWxCh9i1VNIItZ8tDp2RC6sMSBgrW9KQhfp7IqOR1tMosJ0RNSO96s3F/7xuasJbP+MySQ1KtlwUpoKYmMy/JgOukBkxtYQyxe2thI2ooszYbEo2BG/15XXSuqp6btVrXFdqd3kcRTiDc7gED26gBg9QhyYwQHiGV3hzxs6L8+58LFsLTj5zCn/gfP4A0QGM7g==</latexit><latexit sha1_base64="ocTLVc574NOAe/DoMycp9A9e/3g=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cW7Ae0oWy2k3bbzSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikpeNUMWyyWMSqE1CNgktsGm4EdhKFNAoEtoPJ/dxvP6HSPJaPZpqgH9Gh5CFn1FipMe6XK27VXYCsEy8nFchR75e/eoOYpRFKwwTVuuu5ifEzqgxnAmelXqoxoWxCh9i1VNIItZ8tDp2RC6sMSBgrW9KQhfp7IqOR1tMosJ0RNSO96s3F/7xuasJbP+MySQ1KtlwUpoKYmMy/JgOukBkxtYQyxe2thI2ooszYbEo2BG/15XXSuqp6btVrXFdqd3kcRTiDc7gED26gBg9QhyYwQHiGV3hzxs6L8+58LFsLTj5zCn/gfP4A0QGM7g==</latexit>Instancei<latexit sha1_base64="9jtkmSnVkzQh6f031uh1XGFh3yI=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipyQflilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/PfYzt</latexit><latexit sha1_base64="9jtkmSnVkzQh6f031uh1XGFh3yI=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipyQflilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/PfYzt</latexit><latexit sha1_base64="9jtkmSnVkzQh6f031uh1XGFh3yI=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipyQflilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/PfYzt</latexit><latexit sha1_base64="9jtkmSnVkzQh6f031uh1XGFh3yI=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipyQflilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/PfYzt</latexit>Figure 4: An illustration of the 3 steps implemented by our mask propagation branch: 1)For every detected instance in frame
t, we compute an instance-speciﬁc feature tensor via element-wise multiplication between tensor tand the given frame-level
instance mask at frame t.2)Next, the element-wise difference of the feature tensors associated with frames tandt+is
used to predict motion offsets between frames tandt+. The resulting offsets are used to propagate the instance-speciﬁc
tensors from time tto timet+via deformable convolution. The propagated tensors represent instance-speciﬁc features
predicted for time t+using the tensors computed at time t.3)Lastly, we add the propagated instance feature tensors to the
tensor effectively computed at t+. A convolutional layer applied to these tensors predicts instance masks in frame t+.
The pixels that do not belong to any object instances are zeroed out using an instance-agnostic attention map.
the tensorft+effectively computed from the frame at time
t+. If the object instance prediction is consistent with the
feature computation, the feature tensors will be aligned and
thus, they will reinforce each other in the predicted region.
Finally, the resulting feature tensors i
t;t+are fed into
a11convolution layer that outputs instance masks for
each object instance iin framet+. The masks are nor-
malized with the softmax nonlinearity across all Ntin-
stances. To zero-out pixels that do not belong to any object
instance, we use a single 33convolution that computes
an instance-agnostic attention map At+from feature ten-
sorft+. We then multiply At+with each of our predicted
instance masks. A detailed illustration of our mask propa-
gation branch is presented in Figure 4.
4.3. Video-Level Segmentation Instances
Given a video of length L, our goal is to produce video-
level segmentation instances Mi2 RLHW. Con-
ceptually, this requires linking clip-level instance tracks
Mi
tT:t+TandMj
t0T:t0+Twheniandjrepresent the same
object instance, i.e., when the instances are matching. We
achieve this by assigning a video-level instance ID to each
of our predicted clip-level instance tracks. Matching in-
stance tracks are assigned the same video-level instance ID.
Matching Clip-Level Instance Tracks. Consider a pair of
clip-level instance tracks Mi
tT:t+TandMj
t0T:t0+T, that
are centered around frames tandt0respectively. They over-
lap in time ifjtt0j<2T+ 1. Let us denote their over-
lapping time interval as \t;t0. Given two overlapping in-stance tracks, we can check if they match by comparing
their predicted instance masks in the overlapping frames.
If the masks of the two instance tracks align well, then they
are likely to encode the same object instance. Otherwise,
each track represents a different object instance. We com-
pute a matching score mi;j
t;t0between two clip-level instance
tracks using our previously deﬁned soft IoU metric as:
mi;j
t;t0=1
j\t;t0jX
~t2\t;t0sIoU (Mi
tT:t+T(~t); Mj
t0T:t0+T(~t))(4)
Video-Level Instance ID Assignment. We denote withY
the set of video-level instance IDs. The set Yis built incre-
mentally by matching clip-level instance tracks Mi
tT:t+T
in order from time t= 1 to timet=L. Initially, we set
Y=f1;2;:::;N 1gwhereN1is the number of object in-
stances detected at time t= 1 (i.e., in the ﬁrst clip of the
video). Let yi
t2Y denote the video-level instance ID as-
signed to clip-level instance track Mi
tT:t+T. As we move
forward in time t>1, the video-level instance ID yi
tis as-
signed by matching clip-level instance Mi
tT:t+Tto all pre-
viously processed instance tracks Mj
t0T:t0+Tthat overlap
with this clip, i.e., such that \t;t06=;. For each video-level
instance IDy2Y already in the ID set, we compute a score
qi
t(y)capturing how well Mi
tT:t+Tmatches the tracks that
have been already assigned video-level ID y:
qi
t(y) =P
t0s.t.\t;t06=;PNt0
j=11fyj
t0=ygmi;j
t;t0
P
t0s.t.\t;t06=;PNt0
j=11fyj
t0=yg(5)
time ttime t+5time t+10time t+15time t-15time t-10time t-5
Figure 5: An illustration of instance-speciﬁc features propagated from frame tto other frames in the given video clip. Here,
we visualize propagated activations from one randomly selected feature channel. The activations in the two rows correspond
to two different object instances detected at time t. Our visualizations suggest that MaskProp reliably propagates features
that are speciﬁc to each instance even when instances appear next to each other, and despite the changes in shape, pose and
the nuisances effects of deformation and occlusion.
where 1fyj
t0=ygis an indicator function that is equal
to1whenyj
t0=y, and 0otherwise.Nt0is the number
of detected instances at time t0, andmi;j
t;t0is the previously
deﬁned matching score between instance tracks Mi
tT:t+T
andMj
t0T:t0+T. The score qi
t(y)is effectively an average
of the soft IoU computed between instance track Mi
tT:t+T,
and all its overlapping instance tracks that have been as-
signed video-level instance ID y.
Letq= maxy2Yqi
t(y)be the maximum score obtained
by considering all possible video-level instance IDs y2Y.
Ifqis greater than a certain threshold, the current in-
stance track Mi
tT:t+Tis assigned the video-level instance
IDy= arg maxy2Yqi
t(y). Otherwise, the clip-level track
does not match any of the current video-level instances. In
such case we create a new video-level instance ID and as-
sign it to the the clip-level track while also expanding the
setY, i.e.,yi
t=jYj+ 1andY=Y[fjYj + 1g.
Finally, for each video-level instance ID y2 Y , we
generate the ﬁnal sequence of segmentation instance masks
My2RLHWas:
My(t) =(
Mi
tT:t+T(t)ifyi
t=y
0 otherwise .4.4. Implementation Details
Backbone Network. As our backbone we use a Spa-
tiotemporal Sampling Network [5] based on a Deformable
ResNeXt-101-64x4d [12, 41] with a feature pyramid net-
work (FPN) [24] attached on top of it.
Detection Network. For detection, we use a Hybrid Task
Cascade Network [7] with a 3stage cascade.
Mask Propagation Branch. The residual block in the
mask propagation branch consists of two 33convolu-
tions with 128output channels each. The instance feature
propagation is applied to the FPN feature map with the sec-
ond largest spatial resolution. To propagate instance fea-
tures from one frame to another, we predict 9 (x;y)offsets
for every pixel, which are then used as input to a 33de-
formable convolutional layer with 256channels. To capture
motion at different scales, we use three levels of dilated de-
formable convolutions with dilation rates 3;6;12as in [3].
High-Resolution Mask Reﬁnement. Predicting masks
from RoI features typically leads to low resolution predic-
tions. We address this issue via a high-resolution mask re-
ﬁnement step. Given the center of a detected bounding box,
we crop a 384384patch around the object, preserving
the original aspect ratio. We then feed the RGB patch and
the predicted low-resolution mask through 3residual blocks
each with 128channels to obtain a high-resolution mask.
Method Pre-training Data mAP AP@75 AR@1 AR@10
DeepSORTz[39] Imagenet [34], COCO [25] 26.1 26.1 27.8 31.3
FEELVOSz[37] Imagenet [34], COCO [25] 26.9 29.7 29.9 33.4
OSMNz[43] Imagenet [34], COCO [25] 27.5 29.1 28.6 33.1
MaskTrack R-CNNz[42] Imagenet [34], COCO [25] 30.3 32.6 31.0 35.5
MaskTrack R-CNNImagenet [34], COCO [25] 36.9 40.2 34.3 42.9
EnsembleVIS [28] Imagenet [34], COCO [25], Instagram [30], OpenImages [23] 44.8 48.9 42.7 51.7
MaskProp Imagenet [34], COCO [25] 46.6 51.2 44.0 52.6
MaskProp Imagenet [34], COCO [25], OpenImages [23] 50.0 55.9 44.6 54.5
Table 2: The results of video instance segmentation on the YouTube-VIS [42] validation dataset. We evaluate the performance
of each method according to mean average precision (mAP), average precision at 75% IoU threshold (AP@75), and average
recall given top 1(AR@1) and top 10(AR@10) detections. The baselines denoted withzwere implemented by the authors
in [42], whereas the methods marked withwere implemented by us, and use the same backbone and detection networks as
our approach. Despite its simplicity, MaskProp outperforms all prior video instance segmentation methods by a large margin.
Scoring Video-Level Sequences. Each video-level se-
quence contains a list of classiﬁcation scores and predicted
object labels. To assign a conﬁdence score to each video-
level sequence, we average classiﬁcation scores associated
with that sequence (separately for each object category).
Training. We use a similar training setup as in [7]. The loss
weights for each of the three cascade stages are set to 1;0:5;
and0:25respectively. The loss weight for the semantic seg-
mentation branch is set to 0:1. We train our model on pairs
of frames, where the second frame in a pair is randomly se-
lected with a time gap 2[25;25]relative to ﬁrst frame.
We use a multi-scale training approach implemented by re-
sizing the shorter side of the frame randomly between 400
and800pixels. Our model is trained in a distributed set-
ting using 64GPUs, each GPU holding a single clip. The
training is done for 20epochs with an initial learning rate
of0:008, which is decreased by 10at16and19epochs.
We initialize our model with a Mask R-CNN pretrained on
COCO for the instance segmentation. The hyperparameters
of RPN and FPN are the same as in [7].
Inference. During testing, we run the bounding box predic-
tion branch on 1000 proposals, apply non-maximum sup-
pression, and use boxes with a score higher than 0:1as in-
put to the mask prediction and mask propagation branches.
During inference, our MaskProp is applied to video clips
consisting of 13frames.
5. Experimental Results
In this section, we evaluate MaskProp for video instance
segmentation on YouTube-VIS [42], which contains 2;238
training, 302validation, and 343test videos. Each video
is annotated with per-pixel segmentation, category, and in-
stance labels. The dataset contains 40object categories.
Since the evaluation on the test set is currently closed, we
perform our evaluations on the validation set.
5.1. Quantitative Results
Video instance segmentation is a very recent task [42],
and thus, there are only a few established baselines that we
7 9 11 13 15 17 19 21
Instance Track Clip Length43.54444.54545.54646.547Accuracy (Video mAP)Figure 6: We plot video mAP as a function of an instance
track clip length (denoted as 2T+ 1in the paper). Based on
these results, we observe that optimal video instance seg-
mentation performance is achieved when we propagate in-
stance masks to T= 6previous and subsequent frames.
can compare our work to. We include in our comparison
MaskTrack R-CNN [42], and the EnsembleVIS method,
which won the ICCV 2019 video instance segmentation
challenge [28]. Additionally, to make the comparison with
MaskTrack R-CNN more fair, we reimplement it using the
same backbone and detection networks as our MaskProp
(see MaskTrack R-CNN* in Table 2).
We present our quantitative results in Table 2, where we
assess each method according to 1) mean video average pre-
cision (mAP), 2) video average precision at IoU threshold
of75%, and 3) average recall given 1and10highest scored
instances per video. From these results, we observe that
our MaskProp outperforms all the other baselines according
to all four evaluation metrics, thus achieving state-of-the-
art results in video instance segmentation on YouTube-VIS.
It can be noted that we outperform EnsembleVIS [28] by
1:8mAP even though our approach is much simpler and
even if it uses orders of magnitude less labeled data for
pre-training. Furthermore, compared to the MaskTrack R-
CNN, our method achieves a 16:3improvement in mAP. We
also note that our implementation of MaskTrack [42] sig-
niﬁcantly improves upon the original work, but it is still 9:7
Method mAP AP@75
FlowNet2 Propagation 31.4 33.6
MaskTrack R-CNN36.9 40.2
MaskProp 46.6 51.2
Table 3: Here, we study the effectiveness of our mask prop-
agation branch. If we replace it with the FlowNet2 propaga-
tion scheme, where masks are propagated using the optical
ﬂow predicted by a FlowNet2 network [20], the accuracy
drops from 46:6mAP to 31:4mAP. Similarly, if we replace
our mask propagation branch with the tracking branch from
MaskTrack R-CNN, the accuracy drops to 36:9mAP. Note
that all of these baselines are implemented using the same
backbone and detection networks.
worse in mAP compared to our model. Lastly, we note that
additionally pre-training MaskProp on the OpenImages [23]
dataset as was done in [28] further boosts our performance
to50:0mAP.
5.2. Ablation Experiments
Mask Propagation Branch. To investigate the effective-
ness of our mask propagation branch, we compare our
method with a FlowNet2 [20] propagation baseline. For
this baseline, we use exactly the same setup as for our
MaskProp, except that instance masks are propagated us-
ing the optical ﬂow predicted by a FlowNet2 network [20]
rather than our proposed mask propagation scheme. For a
more complete comparison, we also include the MaskTrack
R-CNNfrom Table 2, which uses the originally proposed
tracking branch [42], but is implemented using the same
backbone and detection networks as our MaskProp.
These baselines allow us to directly compare the effec-
tiveness of our mask propagation scheme versus the propa-
gation mechanisms employed by FlowNet2 and MaskTrack
R-CNN [42] methods. The results in Table 3 show that
MaskProp outperforms these baselines by a large margin.
Instance Track Clip Length. Due to occlusions, object in-
stances in video may not be visible in some frames. If there
are occlusions separated by 2T0+ 2time-steps, we can use
T >T0to predict longer clip-level instance tracks. In Fig-
ure 6, we study video instance segmentation performance as
a function of instance track clip length (denoted as 2T+ 1
in the paper). Our results indicate that the best accuracy is
achieved when we use a clip length of 13, meaning that we
propagate instances to 6previous and 6subsequent frames.
High-Resolution Mask Reﬁnement. We also study the im-
pact of our high-resolution mask reﬁnement, described in
Subsection 4.4. We report that removing this reﬁnement
causes a drop of 1:9%in video instance segmentation mAP.
Importance of Frame-Level Instance Masks. As de-
scribed in our main draft, we use frame-level instance masks
for instance-speciﬁc feature computation. To investigate theBackbone Network mAP AP@75
ResNet-50 40.0 42.9
ResNet-101 42.5 45.6
ResNeXt-101-64x4d 44.3 48.3
STSN [5]-ResNeXt-101-64x4d 46.6 51.2
Table 4: We study the effect of frame-level instance masks
to our system’s performance. We evaluate our method’s
accuracy when using instance masks obtained from Mask
R-CNN with several different backbones. Our results indi-
cate that frame-level instance masks obtained from stronger
models lead to better video instance segmentation results.
contribution of these masks to the performance of our sys-
tem, we experiment with masks obtained from several dif-
ferent Mask R-CNN models. In Table 4, we present our
results for this ablation. Our results indicate that frame-
level instance masks obtained from stronger models allow
us to achieve better video instance segmentation perfor-
mance. Thus, we expect that future improvements in image
instance segmentation will further beneﬁt our method.
5.3. Qualitative Results
In Figure 7, we compare our predicted clip-level instance
tracks (last row of predictions for each clip) with the Mask-
Track R-CNN predictions (ﬁrst row of predictions). We
use different colors to represent different object instances.
Our qualitative results suggest that our MaskProp produces
more robust and temporally coherent instance tracks than
MaskTrack R-CNN. Such differences in performance are
especially noticeable when a video contains large object
motion, occlusions, or overlapping objects.
In Figure 5, we also visualize instance-speciﬁc features
that are propagated from frame tto other frames in the
given video clip for two different object instances detected
in framet. Here, we show activations from a randomly se-
lected feature channel. Based on these results, we observe
that our MaskProp reliably propagates features that are spe-
ciﬁc to each instance despite motion blur, object deforma-
tions and large variations in object appearance.
6. Conclusion
In this work, we introduced MaskProp, a novel architec-
ture for video instance segmentation. Our method is con-
ceptually simple, it does not require large amounts of la-
beled data for pre-training, and it produces state-of-the-art
results on YouTube-VIS dataset. In future, we plan to ex-
tend MaskProp to scenarios where only bounding box anno-
tations are available. We are also interested in applying our
method to problems such as pose estimation and tracking.
Video ClipMaskPropMaskTrack      R-CNN
Video ClipMaskPropMaskTrack      R-CNN
Video Clip
MaskProp
MaskTrack      R-CNNVideo Clip
MaskPropMaskTrack      R-CNN
Figure 7: We compare our video instance segmentation results with MaskTrack R-CNN [42] predictions. Different object
instances are encoded with different colors. The ﬁrst row for each video shows the original frames. The second row illustrates
the mask predictions of MaskTrack R-CNN and the third row those obtained with our MaskProp. Compared to MaskTrack
R-CNN, our MaskProp tracks object instances more robustly even when they are occluded or overlap with each other.
References
[1] Anurag Arnab and Philip H. S. Torr. Pixelwise instance seg-
mentation with a dynamically instantiated network. In 2017
IEEE Conference on Computer Vision and Pattern Recog-
nition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 ,
pages 879–888, 2017. 2
[2] M. Bai and R. Urtasun. Deep watershed transform for in-
stance segmentation. In 2017 IEEE Conference on Computer
Vision and Pattern Recognition (CVPR) , pages 2858–2866,
July 2017. 2
[3] Gedas Bertasius, Christoph Feichtenhofer, Du Tran, Jianbo
Shi, and Lorenzo Torresani. Learning temporal pose esti-
mation from sparsely labeled videos. In Advances in Neural
Information Processing Systems 33 . 2019. 4, 6
[4] Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani. Se-
mantic segmentation with boundary neural ﬁelds. In The
IEEE Conference on Computer Vision and Pattern Recog-
nition (CVPR) , June 2016. 1
[5] Gedas Bertasius, Lorenzo Torresani, and Jianbo Shi. Object
detection in video with spatiotemporal sampling networks.
InECCV (12) , volume 11216 of Lecture Notes in Computer
Science , pages 342–357. Springer, 2018. 2, 4, 6, 8
[6] Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset,
Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool. One-
shot video object segmentation. In Computer Vision and Pat-
tern Recognition (CVPR) , 2017. 2
[7] Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao
Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi,
Wanli Ouyang, Chen Change Loy, and Dahua Lin. Hybrid
task cascade for instance segmentation. In IEEE Conference
on Computer Vision and Pattern Recognition, CVPR 2019,
Long Beach, CA, USA, June 16-20, 2019 , pages 4974–4983.
Computer Vision Foundation / IEEE, 2019. 6, 7
[8] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos,
Kevin Murphy, and Alan L. Yuille. Deeplab: Semantic im-
age segmentation with deep convolutional nets, atrous con-
volution, and fully connected crfs. IEEE Trans. Pattern Anal.
Mach. Intell. , 40(4):834–848, 2018. 1
[9] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian
Schroff, and Hartwig Adam. Encoder-decoder with atrous
separable convolution for semantic image segmentation. In
Computer Vision - ECCV 2018 - 15th European Conference,
Munich, Germany, September 8-14, 2018, Proceedings, Part
VII, pages 833–851, 2018. 2
[10] Yuhua Chen, Jordi Pont-Tuset, Alberto Montes, and Luc Van
Gool. Blazingly fast video object segmentation with pixel-
wise metric learning. In CVPR , pages 1189–1198. IEEE
Computer Society, 2018. 2
[11] Jifeng Dai, Kaiming He, and Jian Sun. Instance-aware se-
mantic segmentation via multi-task network cascades. In
CVPR , 2016. 2
[12] J. Dai, H. Qi, Y . Xiong, Y . Li, G. Zhang, H. Hu, and Y . Wei.
Deformable convolutional networks. In 2017 IEEE Interna-
tional Conference on Computer Vision (ICCV) , volume 00,
pages 764–773, Oct. 2017. 4, 6[13] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman.
Detect to track and track to detect. In International Confer-
ence on Computer Vision (ICCV) , 2017. 2
[14] Ross Girshick. Fast R-CNN. In Proceedings of the Interna-
tional Conference on Computer Vision (ICCV) , 2015. 1
[15] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra
Malik. Rich feature hierarchies for accurate object detec-
tion and semantic segmentation. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition
(CVPR) , 2014. 1
[16] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Gir-
shick. Mask R-CNN. In Proceedings of the International
Conference on Computer Vision (ICCV) , 2017. 1, 2, 4
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition. 2016 IEEE
Conference on Computer Vision and Pattern Recognition
(CVPR) , pages 770–778, 2016. 1, 4
[18] Alexander Hermans*, Lucas Beyer*, and Bastian Leibe. In
Defense of the Triplet Loss for Person Re-Identiﬁcation.
arXiv preprint arXiv:1703.07737 , 2017. 2
[19] Alba Herrera-Palacio, Carles Ventura, and Xavier Giro-i Ni-
eto. Video object linguistic grounding. In 1st International
Workshop on Multimodal Understanding and Learning for
Embodied Applications , MULEA ’19, pages 49–51, New
York, NY , USA, 2019. ACM. 2
[20] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper,
Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evo-
lution of optical ﬂow estimation with deep networks. CoRR ,
abs/1612.01925, 2016. 8
[21] Suyog Dutt Jain, Bo Xiong, and Kristen Grauman. Fusion-
seg: Learning to combine motion and appearance for fully
automatic segmentation of generic objects in videos. In 2017
IEEE Conference on Computer Vision and Pattern Recog-
nition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 ,
pages 2117–2126, 2017. 2
[22] Alexander Kirillov, Evgeny Levinkov, Bjoern Andres, Bog-
dan Savchynskyy, and Carsten Rother. InstanceCut: from
edges to instances with multicut. In CVPR , 2017. 2
[23] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Ui-
jlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan
Popov, Matteo Malloci, Tom Duerig, and Vittorio Ferrari.
The open images dataset v4: Uniﬁed image classiﬁcation,
object detection, and visual relationship detection at scale,
2018. 2, 7, 8
[24] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He,
Bharath Hariharan, and Serge Belongie. Feature pyramid
networks for object detection. In CVPR , 2017. 6
[25] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays,
Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence
Zitnick. Microsoft coco: Common objects in context. In
European Conference on Computer Vision (ECCV) , Zürich,
September 2014. 2, 3, 7
[26] S. Liu, J. Jia, S. Fidler, and R. Urtasun. Sgn: Sequential
grouping networks for instance segmentation. In 2017 IEEE
International Conference on Computer Vision (ICCV) , pages
3516–3524, Oct 2017. 2
[27] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully
convolutional networks for semantic segmentation. In
CVPR , 2015. 1
[28] Jonathon Luiten, Philip H.S. Torr, and Bastian Leibe. Video
instance segmentation 2019: A winning approach for com-
bined detection, segmentation, classiﬁcation and tracking.
In2019 IEEE International Conference on Computer Vision
Workshops, ICCV Workshops 2019, Seoul, South Korea, Oc-
tober 27-November 2, 2019 . IEEE, 2019. 1, 2, 7, 8
[29] Jonathon Luiten, Paul V oigtlaender, and Bastian Leibe. Pre-
mvos: Proposal-generation, reﬁnement and merging for
video object segmentation. In Asian Conference on Com-
puter Vision , 2018. 2
[30] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan,
Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe,
and Laurens van der Maaten. Exploring the limits of weakly
supervised pretraining. In ECCV , 2018. 2, 7
[31] Aljoša Ošep, Paul V oigtlaender, Jonathon Luiten, Stefan
Breuers, and Bastian Leibe. Large-scale object mining for
object discovery from unlabeled video. ICRA , 2019. 2
[32] F. Perazzi, A. Khoreva, R. Benenson, B. Schiele, and
A.Sorkine-Hornung. Learning video object segmentation
from static images. In Computer Vision and Pattern Recog-
nition , 2017. 2
[33] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.
Faster R-CNN: Towards real-time object detection with re-
gion proposal networks. In Neural Information Processing
Systems (NIPS) , 2015. 1
[34] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San-
jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy,
Aditya Khosla, Michael Bernstein, Alexander C. Berg, and
Li Fei-Fei. ImageNet Large Scale Visual Recognition Chal-
lenge. arXiv:1409.0575 , 2014. 2, 7
[35] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.
PWC-Net: CNNs for optical ﬂow using pyramid, warping,
and cost volume. 2018. 2
[36] P. Tokmakov, K. Alahari, and C. Schmid. Learning video
object segmentation with visual memory. In 2017 IEEE In-
ternational Conference on Computer Vision (ICCV) , pages
4491–4500, Oct 2017. 2
[37] Paul V oigtlaender, Yuning Chai, Florian Schroff, Hartwig
Adam, Bastian Leibe, and Liang-Chieh Chen. Feelvos: Fast
end-to-end embedding learning for video object segmenta-
tion. In CVPR , 2019. 2, 7
[38] Paul V oigtlaender, Michael Krause, Aljosa Osep, Jonathon
Luiten, Berin Balachandar Gnana Sekar, Andreas Geiger,
and Bastian Leibe. Mots: Multi-object tracking and segmen-
tation. In Proceedings IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR) , June 2019. 2
[39] Nicolai Wojke, Alex Bewley, and Dietrich Paulus. Simple
online and realtime tracking with a deep association metric.
CoRR , abs/1703.07402, 2017. 7
[40] Fanyi Xiao and Yong Jae Lee. Video object detection with an
aligned spatial-temporal memory. In European Conference
on Computer Vision (ECCV) , 2018. 2
[41] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and
Kaiming He. Aggregated residual transformations for deep
neural networks. In CVPR , 2017. 6[42] Linjie Yang, Yuchen Fan, and Ning Xu. Video instance seg-
mentation. In ICCV , 2019. 1, 2, 3, 7, 8, 9
[43] Linjie Yang, Yanran Wang, Xuehan Xiong, Jianchao Yang,
and Aggelos K. Katsaggelos. Efﬁcient video object segmen-
tation via network modulation. In 2018 IEEE Conference on
Computer Vision and Pattern Recognition, CVPR 2018, Salt
Lake City, UT, USA, June 18-22, 2018 , pages 6499–6507,
2018. 2, 7
[44] Jifeng Dai Xiangyang Ji Yi Li, Haozhi Qi and Yichen Wei.
Fully convolutional instance-aware semantic segmentation.
InCVPR , 2017. 2
[45] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang
Wang, and Jiaya Jia. Pyramid scene parsing network. In 2017
IEEE Conference on Computer Vision and Pattern Recog-
nition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 ,
pages 6230–6239, 2017. 1
[46] Xizhou Zhu, Yujie Wang, Jifeng Dai, Lu Yuan, and Yichen
Wei. Flow-guided feature aggregation for video object de-
tection. In International Conference on Computer Vision
(ICCV) , 2017. 2
[47] Idil Esen Zulﬁkar, Jonathon Luiten, and Bastian Leibe. Un-
ovost: Unsupervised ofﬂine video object segmentation and
tracking for the 2019 unsupervised davis challenge. 2