Dropout was turned off in pre-training (quality win). Dropout should be re-enabled during fine-tuning.

Pre-trained on C4 only without mixing in the downstream tasks.

No parameter sharing between the embedding and classifier layer.

"xl" and "xxl" replace "3B" and "11B". The model shapes are a bit different - larger d_model and smaller
  num_heads and d_ff.