Difference betweeen DistilBertTokenizerFast and DistilBertTokenizer?
Hi everyone,I’m trying to understand what is the difference between theDistilBertTokenizerFastand theDistilBertTokenizer.From the documentation it looks like that the DistilBertTokenizerFast “Construct a “fast” DistilBERT tokenizer (backed by HuggingFace’s tokenizers library).” whereas the DistilBertTokenizer “Construct a DistilBERT tokenizer.”, but I don’t understand what that means.This is what I found from the documentation:Tokenizer — transformers 4.5.0.dev0 documentationTo the best of my understanding the difference between a “fast” and a “non-fast” tokenizer is computation speed but there is no functional difference between them. Please correct me if I’m not the right direction.Any help will be greatly appreciated.Thank you,Ayala
2021-07-10T09:51:50Z
@ayalaallI also have the same thought as you.I also want to know about this. Are they the same in terms of function?
2021-07-10T10:27:46Z
FastTokenizers are implemented in Rust and are factors faster than the Python based tokenizers. Apart from that their encoding methods should behave the same. However, they are not functionally identical. From thedocs:When the tokenizer is a “Fast” tokenizer (i.e., backed by HuggingFacetokenizers library), this class provides in addition several advanced alignment methods which can be used to map between the original string (character and words) and the token space (e.g., getting the index of the token comprising a given character or the span of characters corresponding to a given token).Typically you want to use the fast tokenizer if it is available.