

DETR uses so-called object queries to detect objects in an image. The number of queries determines the maximum
  number of objects that can be detected in a single image, and is set to 100 by default (see parameter
  num_queries of [~transformers.DetrConfig]). Note that it's good to have some slack (in COCO, the
  authors used 100, while the maximum number of objects in a COCO image is ~70).
The decoder of DETR updates the query embeddings in parallel. This is different from language models like GPT-2,
  which use autoregressive decoding instead of parallel. Hence, no causal attention mask is used.
DETR adds position embeddings to the hidden states at each self-attention and cross-attention layer before projecting
  to queries and keys. For the position embeddings of the image, one can choose between fixed sinusoidal or learned
  absolute position embeddings. By default, the parameter position_embedding_type of
  [~transformers.DetrConfig] is set to "sine".
During training, the authors of DETR did find it helpful to use auxiliary losses in the decoder, especially to help
  the model output the correct number of objects of each class. If you set the parameter auxiliary_loss of
  [~transformers.DetrConfig] to True, then prediction feedforward neural networks and Hungarian losses
  are added after each decoder layer (with the FFNs sharing parameters).
If you want to train the model in a distributed environment across multiple nodes, then one should update the
  num_boxes variable in the DetrLoss class of modeling_detr.py. When training on multiple nodes, this should be
  set to the average number of target boxes across all nodes, as can be seen in the original implementation here.
[~transformers.DetrForObjectDetection] and [~transformers.DetrForSegmentation] can be initialized with
  any convolutional backbone available in the timm library.
  Initializing with a MobileNet backbone for example can be done by setting the backbone attribute of
  [~transformers.DetrConfig] to "tf_mobilenetv3_small_075", and then initializing the model with that
  config.
DETR resizes the input images such that the shortest side is at least a certain amount of pixels while the longest is
  at most 1333 pixels. At training time, scale augmentation is used such that the shortest side is randomly set to at
  least 480 and at most 800 pixels. At inference time, the shortest side is set to 800. One can use
  [~transformers.DetrImageProcessor] to prepare images (and optional annotations in COCO format) for the
  model. Due to this resizing, images in a batch can have different sizes. DETR solves this by padding images up to the
  largest size in a batch, and by creating a pixel mask that indicates which pixels are real/which are padding.
  Alternatively, one can also define a custom collate_fn in order to batch images together, using
  [~transformers.DetrImageProcessor.pad_and_create_pixel_mask].
The size of the images will determine the amount of memory being used, and will thus determine the batch_size.
  It is advised to use a batch size of 2 per GPU. See this Github thread for more info.