from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
   

Load a dataset:

from datasets import load_dataset
dataset = load_dataset("rotten_tomatoes")  # doctest: +IGNORE_RESULT
   

Create a function to tokenize the dataset:

def tokenize_dataset(dataset):
        return tokenizer(dataset["text"])
   

Then apply it over the entire dataset with [~datasets.Dataset.map]:

dataset = dataset.map(tokenize_dataset, batched=True)