What does `tokenizers.normalizer.normalize` do?
Hey all! Loving the updatedtokenizerdocs and playing around withnormalizersat the moment. I’d like to updatemy article hereabout text preprocessing and using Datasets but I had a quick question:I know.normalize_strworks like so:normalizer.normalize_str("Héllò hôw are ü?")
# "Hello how are u?"Butnormalizer.normalizedoesn’t seem to be documented? Is this something that maybe I should be using, or is more for internal use?Just wondering ifnormalizer.normalize_stris the most efficient way to use anormalizerwithdatasets.mapor ifnormalizer.normalizecan do some magic? Is there a way to usebatched=Truewithindatasets.mapto make things even faster?Or if I addednormalizerto a pretrained tokenizer and then call thetokenizerwith datasets, will that also carry out the normalization before doing the tokenization?
2020-10-12T12:15:30Z
TheNormalizer.normalizefunction is for our internal use in the pipeline: it doesn’t take a string but a string with the offsets, as it adds some functionality to keep track of the original offsets wrt to the original text and works in place (so you can combine several normalizers easily).I don’t thinknormalize_stris less efficient. Also don’t think you can make things faster withbatched=Truehere as it will just iterate on the elements of a batch.Pinging@anthonyand@Narsilthat may have more insight
2020-10-12T12:54:12Z
morgan:Or if I addednormalizerto a pretrained tokenizer and then call thetokenizerwith datasets, will that also carry out the normalization before doing the tokenization?Yes that is exactly the way to go. To get maximum efficiency you want to avoid round trips between python and Rust as much as possible. We are in the process of getting a documentation out that will hopefully improve that.Just a gist here, the de-facto method to create a new Tokenizer in0.9.0is:from tokenizers import Tokenizer, models, normalizers, ....

tokenizer = Tokenizer(models.BPE())
tokenizer.normalizer = normalizers.NFKC()
tokenizer.pre_tokenizer = ... # some pre tokenizer
tokenizer.decoder = ...
tokenizer.processor = ...
etc...

# Optional training phase, not required 
# if you loaded from an existing model.

trainer = trainers.BpeTrainer(vocab_size=80000
tokenizer.train(trainer, ["myfile.txt"])

# Training

tokenizer.encode("my string").tokensThen everything will be run within rust, and should get you the best speedup possible.If you save yourtokenizerwithtokenizer.save('mytokenizer.json')it should save the whole tokenization pipeline. Its another way of checking the various options of existing tokenizers.
2020-10-12T15:27:25Z
Thanks@sgugger,@Narsilgreat to know.@Narsilcan I add a normalizer to a pretrained tokenizer and expect the same behaviour? Do pretrained tokenizers have their own normalizers that I might over-write by accident?Would something like the below work as expected or would I be interfering with what the pretrained tokenizer does?tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
tokenizer.normalizer = normalizers.NFKC()
tokenizer('Am I working?')
2020-10-12T15:44:02Z
It would not work as is.By defaultfrom_pretraineddoes not use (yet) the fast tokenizers (from thetokenizerslibrary).What you want to do istokenizer = AutoTokenizer.from_pretrained('bert-base-uncased', use_fast=True)

# The actual tokenizer from `tokenizers` is tokenizer._tokenizer as we still need a small python wrapper currently

tokenizer._tokenizer.normalizer = normalizers.NFKC()
tokenizer.tokenize("Am I working?")Keep in mind though that this works onmaster. On transformers 3.3.1 it would betokenizer._tokenizer._tokenizer.normalizer.If you want to add a new normalizer, you would have to recreate the actual sequence (bear in mind that normalizers order has importance, so you want to carefully add it at the correct place for your use case)
2020-10-12T16:04:23Z
Ok, so one would have to go to the source code and find the normalizers used and then replicate those? I guess this is a good starting point:tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased', use_fast=True)

tokenizer._tokenizer.normalizer

Output: <tokenizers.normalizers.BertNormalizer at 0x7fc7e939b2f0>Would it be worth exposing more information about the normalizer used with the pretrained tokenizers (e.g. with a.normalizeror.configattribute)?tokenizer.add_normalizerswould be ideal, but I guess its not going to be built for an audience of oneJust trying to think how to leverage the parallization power of the tokenizer with easy, off-the-shelf processing functions, without folks having to go digging into the source code(cc@anthony)