This implementation is the same as [BertModel] with a tiny embeddings tweak as well as a setup
  for Roberta pretrained models.
RoBERTa has the same architecture as BERT, but uses a byte-level BPE as a tokenizer (same as GPT-2) and uses a
  different pretraining scheme.
RoBERTa doesn't have token_type_ids, you don't need to indicate which token belongs to which segment. Just
  separate your segments with the separation token tokenizer.sep_token (or </s>)

Same as BERT with better pretraining tricks: