Then apply it over the entire dataset with [~datasets.Dataset.map]:

dataset = dataset.map(tokenize_dataset, batched=True)
   

A [DataCollatorWithPadding] to create a batch of examples from your dataset:

from transformers import DataCollatorWithPadding
data_collator = DataCollatorWithPadding(tokenizer=tokenizer)
   

Now gather all these classes in [Trainer]: