
DETR
Overview
The DETR model was proposed in End-to-End Object Detection with Transformers by
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov and Sergey Zagoruyko. DETR
consists of a convolutional backbone followed by an encoder-decoder Transformer which can be trained end-to-end for
object detection. It greatly simplifies a lot of the complexity of models like Faster-R-CNN and Mask-R-CNN, which use
things like region proposals, non-maximum suppression procedure and anchor generation. Moreover, DETR can also be
naturally extended to perform panoptic segmentation, by simply adding a mask head on top of the decoder outputs.
The abstract from the paper is the following:
We present a new method that views object detection as a direct set prediction problem. Our approach streamlines the
detection pipeline, effectively removing the need for many hand-designed components like a non-maximum suppression
procedure or anchor generation that explicitly encode our prior knowledge about the task. The main ingredients of the
new framework, called DEtection TRansformer or DETR, are a set-based global loss that forces unique predictions via
bipartite matching, and a transformer encoder-decoder architecture. Given a fixed small set of learned object queries,
DETR reasons about the relations of the objects and the global image context to directly output the final set of
predictions in parallel. The new model is conceptually simple and does not require a specialized library, unlike many
other modern detectors. DETR demonstrates accuracy and run-time performance on par with the well-established and
highly-optimized Faster RCNN baseline on the challenging COCO object detection dataset. Moreover, DETR can be easily
generalized to produce panoptic segmentation in a unified manner. We show that it significantly outperforms competitive
baselines.
This model was contributed by nielsr. The original code can be found here.
How DETR works
Here's a TLDR explaining how [~transformers.DetrForObjectDetection] works:
First, an image is sent through a pre-trained convolutional backbone (in the paper, the authors use
ResNet-50/ResNet-101). Let's assume we also add a batch dimension. This means that the input to the backbone is a
tensor of shape (batch_size, 3, height, width), assuming the image has 3 color channels (RGB). The CNN backbone
outputs a new lower-resolution feature map, typically of shape (batch_size, 2048, height/32, width/32). This is
then projected to match the hidden dimension of the Transformer of DETR, which is 256 by default, using a
nn.Conv2D layer. So now, we have a tensor of shape (batch_size, 256, height/32, width/32). Next, the
feature map is flattened and transposed to obtain a tensor of shape (batch_size, seq_len, d_model) =
(batch_size, width/32*height/32, 256). So a difference with NLP models is that the sequence length is actually
longer than usual, but with a smaller d_model (which in NLP is typically 768 or higher).
Next, this is sent through the encoder, outputting encoder_hidden_states of the same shape (you can consider
these as image features). Next, so-called object queries are sent through the decoder. This is a tensor of shape
(batch_size, num_queries, d_model), with num_queries typically set to 100 and initialized with zeros.
These input embeddings are learnt positional encodings that the authors refer to as object queries, and similarly to
the encoder, they are added to the input of each attention layer. Each object query will look for a particular object
in the image. The decoder updates these embeddings through multiple self-attention and encoder-decoder attention layers
to output decoder_hidden_states of the same shape: (batch_size, num_queries, d_model). Next, two heads
are added on top for object detection: a linear layer for classifying each object query into one of the objects or "no
object", and a MLP to predict bounding boxes for each query.
The model is trained using a bipartite matching loss: so what we actually do is compare the predicted classes +
bounding boxes of each of the N = 100 object queries to the ground truth annotations, padded up to the same length N
(so if an image only contains 4 objects, 96 annotations will just have a "no object" as class and "no bounding box" as
bounding box). The Hungarian matching algorithm is used to find
an optimal one-to-one mapping of each of the N queries to each of the N annotations. Next, standard cross-entropy (for
the classes) and a linear combination of the L1 and generalized IoU loss (for the
bounding boxes) are used to optimize the parameters of the model.
DETR can be naturally extended to perform panoptic segmentation (which unifies semantic segmentation and instance
segmentation). [~transformers.DetrForSegmentation] adds a segmentation mask head on top of
[~transformers.DetrForObjectDetection]. The mask head can be trained either jointly, or in a two steps process,
where one first trains a [~transformers.DetrForObjectDetection] model to detect bounding boxes around both
"things" (instances) and "stuff" (background things like trees, roads, sky), then freeze all the weights and train only
the mask head for 25 epochs. Experimentally, these two approaches give similar results. Note that predicting boxes is
required for the training to be possible, since the Hungarian matching is computed using distances between boxes.
Usage tips