Get "using the `__call__` method is faster" warning with DataCollatorWithPadding
When I use the out-of-the-boxDataCollatorWithPaddingI get my output filled with the warning:You’re using a DebertaV2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the__call__method is faster than using a method to encode the text followed by a call to thepadmethod to get a padded encoding.If I change to use a different, custom collator, then the warning goes away.Would anyone know what I could be doing wrong that’s causing this warning?Or alternatively, if I can’t fix the problem that’s causing this warning, is there a way to hide it?I’ve tried a few different ways of turning off warnings, but so far I’ve had no luck and because it gets written out multiple times it starts to swamp the actual output from my training.
2022-10-05T18:38:03Z
I’m getting the same thing using aBertTokenizerFastwithDataCollatorWithPadding- the error appears once for each worker every time I loop over a DataLoader. I would prefer not to silence warnings in my training code, but here’s how I’m getting around it (based onthis linein thePretrainedTokenizerBaseclass, referencingthis section in the custom logger):import os
os.environ['TRANSFORMERS_NO_ADVISORY_WARNINGS'] = 'true'
2022-10-07T10:07:51Z
Same situation - happened with usage ofDistilBertTokenizerFasttokenizer on usingDataCollatorWithPadding. I don’t know what does this warning actually mean? And if I wanted to follow the suggestion as hinted in the warning message, what should I do?
2022-11-24T06:04:43Z
I am getting following suggestion while using NllbTokenizerFast tokenizeYou’re using a NllbTokenizerFast tokenizer. Please note that with a fast tokenizer, using the__call__method is faster than using a method to encode the text followed by a call to thepadmethod to get a padded encoding.Can anyone explain the given suggestion?
2023-05-09T15:29:37Z
I guess the problem most people in this thread have is not with thewarningsbut with the lack of knowledge (at least at my part) if there is a better way of doing what I am doing. I also use a preprocess function.
2023-05-13T14:08:26Z
It would be nice to have an answer from someone on this?So my guess is that doingtokenizer(input)is faster than wrapping thedatacollatoraround?
2023-09-21T08:45:26Z
Any news on this? Personally I found this warning more annoying than informative, so I ended up commenting it out in the source code, since I couldn’t find any information.
2024-02-01T19:19:21Z
Hello,the warning is confusing because it does not tell youhowto use thetokenizer(...).I think it just means, that one should use the padding feature of the tokenizer when tokenizing your dataset. So, instead of using a collator to pad your inputs, skip this step and usetokenizer(..., padding=True)to add padding while tokenizing. Apparently, this is faster forFastTokenizers(i.e. tokenizers written in Rust).Remark: In case you didn’t know when callingtokenizer(...)this internally callsTokenizer.__call__.  This is a magic python method:python-callable-instancesDisclaimer: That’s just my interpretation of what it could mean. I would be happy if someone could confirm this.
2024-06-03T21:21:47Z
You can turn off the warning withtokenizer.deprecation_warnings["Asking-to-pad-a-fast-tokenizer"] = TrueSeeWhen using transformers.DataCollatorWithPadding normally, always get annoying warning · Issue #22638 · huggingface/transformers · GitHub