NameError when tokenizing with num_proc
On my windows machine, running the multiprocessing code inThemap()method’s superpowersfails with"NameError: name 'slow_tokenizer' is not defined"Bindingslow_tokenizeras aslow_tokenize_functionparameter makes the code run …slow_tokenizer = AutoTokenizer.from_pretrained("bert-base-cased", use_fast=False)
def slow_tokenize_function(examples, slow_tokenizer=slow_tokenizer):
    return slow_tokenizer(examples["review"], truncation=True)
tokenized_dataset = drug_dataset.map(slow_tokenize_function, batched=True, num_proc=8)… but it’s much slower than withoutnum_proc- I think the python multiprocessing issues on Jupyter and Windows are pretty well know (o: