from transformers import create_optimizer, AdamWeightDecay
optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)

Then you can load DistilGPT2 with [TFAutoModelForCausalLM]:

from transformers import TFAutoModelForCausalLM
model = TFAutoModelForCausalLM.from_pretrained("distilbert/distilgpt2")

Convert your datasets to the tf.data.Dataset format with [~transformers.TFPreTrainedModel.prepare_tf_dataset]: