MLM: encoder input tokens are randomly replaced by a mask tokens and have to be predicted by the encoder (like in BERT)

GSG: whole encoder input sentences are replaced by a second mask token and fed to the decoder, but which has a causal mask to hide the future words like a regular auto-regressive transformer decoder.

FP16 is not supported (help/ideas on this appreciated!).

The adafactor optimizer is recommended for pegasus fine-tuning.