Fastest way to tokenize millions of examples?
Currently I am using a pandas column of strings and tokenizing it by defining a function with the tokenization operation, and using that with pandasmapto transform my column of texts.It’s a slow process when I have millions of rows of texts, and I am wondering if there’s a faster way to tokenize all my training examples.
2022-07-05T03:44:06Z
bump, still looking for a solution
2022-09-26T05:38:15Z
I have a similar issue, using a pretrained WordPiece tokenizer on a large corpus of text takes several hours. I’m doing:tokenizer = AutoTokenizer.from_pretrained(“distilbert-base-uncased”)train_tokenized_encodings = tokenizer(df[df.split==‘train’].text.tolist(), truncation=True, padding=True, max_length=MAX_LENGTH)Any suggestions for speed up?
2022-09-27T20:52:05Z
Two options come to my mind:parallelization of the process itselfdetect similarities between the various rows, such that maybe previous tokenizations can be re-usedbut these are suggestions for which I have not practical experience myself, but maybe it helps somehow
2024-03-08T00:41:56Z
Can you share code please?