Tokenizer dataset is very slow
This is my tokenizer method. I found that no matter how much batch_size is set, the speed is the same. Tokenizer Spend time even longer than training. How cloud I do. Thanks very much.def tokenize_function(example):
    return tokenizer(example["sentence1"], truncation=True, max_length = 512)
tokenized_datasets = raw_datasets.map(tokenize_function, batched=True, batch_size = 8)
tokenized_datasets = tokenized_datasets.remove_columns(["sentence1"])
2022-06-28T18:14:48Z
Hi! What tokenizer are you using? What doestokenizer.is_fastreturn? If the returned value isFalse, you can setnum_proc> 1 to leverage multiprocessing inmap. Fast tokenizers use multithreading to process a batch in parallel on a single process by default, so it doesn’t make sense to usenum_procthere.
2024-03-02T06:19:01Z
I’ve checked mine, and I have a fast tokenizer. However, it’s still taking about 20 seconds per example for tokenization, which is too slow.Here’s the code,base_model_id = "google/gemma-7b"
tokenizer = AutoTokenizer.from_pretrained(
    base_model_id,
    padding_side="left",
    add_eos_token=True,
    add_bos_token=True,
)
tokenizer.pad_token = tokenizer.eos_tokenmax_length = 1026

def generate_and_tokenize_prompt(prompt):
    result = tokenizer(
        formatting_func(prompt),
        truncation=True,
        max_length=max_length,
        padding="max_length",
    )
    result["labels"] = result["input_ids"].copy()
    return result

train_dataset = dataset_split['train']
eval_dataset = dataset_split['test']
tokenized_train_dataset = train_dataset.map(generate_and_tokenize_prompt)
tokenized_val_dataset = eval_dataset.map(generate_and_tokenize_prompt)Can someone please help me figure out what I’m missing? Thanks.
2024-03-02T15:47:07Z
Feel free to report this issue in thetokenizersrepo.