tokenized_squad = squad.map(preprocess_function, batched=True, remove_columns=squad["train"].column_names)

Now create a batch of examples using [DefaultDataCollator]. Unlike other data collators in 🤗 Transformers, the [DefaultDataCollator] does not apply any additional preprocessing such as padding.

from transformers import DefaultDataCollator
data_collator = DefaultDataCollator()
</pt>
<tf>py
from transformers import DefaultDataCollator
data_collator = DefaultDataCollator(return_tensors="tf")

Train