Training unigram on long sequences
This is a follow-up thread ofpyo3_runtime.PanicException: likelihood is NAN. Input sentence may be too long.BackgroundWe’re training a unigram tokenizer from scratch.This is how it’s instantiated:tokenizer = Tokenizer(Unigram())
tokenizer.normalizer = normalizers.Sequence(
    [normalizers.Replace("``", '"'), normalizers.Replace("''", '"')]
)
tokenizer.pre_tokenizer = pre_tokenizers.Metaspace()This is how the trainer is initialized:trainer = UnigramTrainer(
        unk_token="<unk>",
        special_tokens=["[CLS]", "[SEP]", "<unk>", "<pad>", "[MASK]"],
        vocab_size=10000,
)We’re training on a batch-by-batch basis:tokenizer.train_from_iterator(dataloader, trainer=trainer)We’re running into the following toward the end:huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
        - Avoid using `tokenizers` before the fork if possible
[04:52:53] Pre-processing sequences                 ██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ 50158000 /        0
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
        - Avoid using `tokenizers` before the fork if possible
[04:52:57] Pre-processing sequences                 ██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ 50164000 /        0
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
        - Avoid using `tokenizers` before the fork if possible
[04:53:25] Pre-processing sequences                 ██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ 0        /        0
[00:01:37] Suffix array seeds                       ██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ 18201510 / 18201510
[00:00:00] EM training                              ██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████ 18201510 /       32
thread '<unnamed>' panicked at 'likelihood is NAN. Input sentence may be too long.', /__w/tokenizers/tokenizers/tokenizers/src/models/unigram/trainer.rs:413:17
Traceback (most recent call last):
  File "train_unigram_tokenizer.py", line 106, in <module>
    main(**args)
  File "train_unigram_tokenizer.py", line 92, in main
    tokenizer.train_from_iterator(datagen, trainer=trainer)
pyo3_runtime.PanicException: likelihood is NAN. Input sentence may be too long.Here is the version info:transformers: 4.17.0tokenizers: 0.11.6Here’s what tried in the meanwhile:Used theenable_truncation()method that comes with a tokenizer. We set the length to a number for which we’re able to successfully train the tokenizer of another version of the dataset. But to our wonder, with the same sequence length and updated dataset version, the tokenizer training fails.To better understand the issue, we first sampled all the very long sequences (character sequence lengths having > 300000). Note that we used our updated dataset (which is causing the issue) to sample the sequences. We can successfully train the tokenizer on those sampled entries.We managed to train a BPE tokenizer on the same data on which the Unigram training is failing. Maybe there are some unhandled edge cases in the viterbi algorithm implementation?I’m also re-iterating the fact that we did manage to train the Unigram tokenizer with the top 20% longest sequences from the data we have. This means that individual sequence lengths do not lead to a training failure. It fails only when we provide the full data. We have also ensured that the data doesn’t have any non ASCII characters.
2022-06-22T07:40:25Z
Hi, I am facing the same error here and I wonder if you found a way to get around with it?
2022-06-22T13:22:57Z
Not yet
2022-06-22T14:11:21Z
So I changed the normalizer and the pretokenizer in their SentencePieceUnigramTokenizer implementation and it worked for me now. I didn’t try them separately but my guess is that the Metaspace pretokenizer caused the length error.tokenizer.normalizer = normalizers.NFKC()
tokenizer.pre_tokenizer = pre_tokenizers.Whitespace()
2022-06-23T04:12:25Z
Thanks for sharing. We can give it a try.