

This approach works great for smaller datasets, but for larger datasets, you might find it starts to become a problem. Why?
Because the tokenized array and labels would have to be fully loaded into memory, and because NumPy doesn’t handle
“jagged” arrays, so every tokenized sample would have to be padded to the length of the longest sample in the whole
dataset. That’s going to make your array even bigger, and all those padding tokens will slow down training too!
Loading data as a tf.data.Dataset
If you want to avoid slowing down training, you can load your data as a tf.data.Dataset instead. Although you can write your own
tf.data pipeline if you want, we have two convenience methods for doing this: