Although the embedding matrix has a size of 50400, only 50257 entries are used by the GPT-2 tokenizer. These extra
  tokens are added for the sake of efficiency on TPUs. To avoid the mismatch between embedding matrix size and vocab
  size, the tokenizer for GPT-J contains 143 extra tokens
  <|extratoken_1|> <|extratoken_143|>, so the vocab_size of tokenizer also becomes 50400.

Usage examples
The [~generation.GenerationMixin.generate] method can be used to generate text using GPT-J
model.
thon