

GPT-2 is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
  the left.
GPT-2 was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next
  token in a sequence. Leveraging this feature allows GPT-2 to generate syntactically coherent text as it can be
  observed in the run_generation.py example script.
The model can take the past_key_values (for PyTorch) or past (for TF) as input, which is the previously computed
  key/value attention pairs. Using this (past_key_values or past) value prevents the model from re-computing
  pre-computed values in the context of text generation. For PyTorch, see past_key_values argument of the
  [GPT2Model.forward] method, or for TF the past argument of the
  [TFGPT2Model.call] method for more information on its usage.
Enabling the scale_attn_by_inverse_layer_idx and reorder_and_upcast_attn flags will apply the training stability
  improvements from Mistral (for PyTorch only).