To apply the preprocessing function over the entire dataset, use 🤗 Datasets [~datasets.Dataset.with_transform] method. The transforms are applied on the fly when you load an element of the dataset:

food = food.with_transform(transforms)

Now create a batch of examples using [DefaultDataCollator]. Unlike other data collators in 🤗 Transformers, the DefaultDataCollator does not apply additional preprocessing such as padding.

from transformers import DefaultDataCollator
data_collator = DefaultDataCollator()