Tokenized sequence lengths
I have a strange situation where I’m trying to build a custom tokenizer for a custom “language” (encoded music). The language is designed to represent the data in a way that is compact and also reasonably human-readable. In most cases, the ideal tokenization would be essentially word-level, just separating by whitespace, but because I don’t necessarily know in advance all the possibilities, I want to avoid just forcing word-level tokenization.What I’m finding is that, if I train a tokenizer using the basic “intro” approach on the Tokenizers doc page, e.g.:tokenizer = Tokenizer(BPE())
corpus_root = './content'
paths = [str(x) for x in Path(corpus_root).glob("**/*.txt")]

tokenizer.pre_tokenizer = Whitespace()
from tokenizers.trainers import BpeTrainer

trainer = BpeTrainer(special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"], vocab_size=4000)
tokenizer.train(files=paths, trainer=trainer)then I get pretty much the ideal vocab, and the encoding works as expected. However, this tokenizer seems to work differently to those in the Transformers library (e.g., it doesn’t recognizetokenizer.vocab, and so on), so using a more “stock” Transformers tokenizer seems like the safer/easier option going forward.As a workaround, I tried saving the tokenizer files using:tokenizer.model.save('./tokenizer/roberta_tokenizer')
tokenizer.save('./tokenizer/roberta_tokenizer/config.json')and then loading this into a RobertaTokenizer, using the savedvocab.jsonandmerges.txtfiles. Although this does function, what I’m noticing is that the lengths of the encoded tokenizations are dramatically different—up to 3x longer for the sameinputwhen using RobertaTokenizer over theTokenizer(BPE.from_file())version, using the same files, e.g.:count = len(input.split(' '))
output = tokenizer.encode(input)
print(f'{output.tokens}, count = {count}, encoded count = {len(output.ids)}')
tokenizer.model.save('./tokenizer/roberta_tokenizer')
tokenizer.save('./tokenizer/roberta_tokenizer/config.json')
roberta_tokenizer = RobertaTokenizer('./tokenizer/roberta_tokenizer/vocab.json', './tokenizer/roberta_tokenizer/merges.txt')
roberta_tokenizer.save_pretrained('./tokenizers/roberta-tokenizer')

test_tok = AutoTokenizer.from_pretrained('./tokenizers/roberta-tokenizer')
test_tok2 = Tokenizer(BPE.from_file('./tokenizer/roberta_tokenizer/vocab.json', './tokenizer/gpt2_tokenizer/merges.txt'))

test = test_tok.encode(input)
print("Roberta test: ", test)

test2 = test_tok2.encode(input)
print("Tokenizer(BPE()) test: ", test2.ids)

------- 

"Roberta test:  [4000, 192, 357, 52, 5, 194, 661, 11, 33, 32, 9, 39, 9, 111, 51, 58, 6, 130, 53, 46, 51, 58, 6, 146, 6, 260, 58, 6, 1372, 7, 584, 7, 1876, 58, 6, 99, 101, 9, 37, 8, 59, 6, 151, 9, 77, 695, 11, 4001]"
"Tokenizer(BPE()) test:  [939, 711, 196, 115, 131, 120, 295, 165, 233, 309, 99, 395, 100, 335, 568]"I tried testing this with a small, natural-language dataset from the Datasets page, and I don’t see these dramatic differences—i.e., I only see differences of a few tokens, which could be explained by special/added tokens. So I’m assuming that it’s my “synthesized” language that is the difference. So is there a way to useTokenizer(BPE.from_file()), but still have all the integration features of using, for example,RobertaTokenizerfrom the Transformers library?I have been posting some of my saga on the Discord server, but I think this is probably a better place for something so detailed (and mysterious).Any help would be greatly appreciated.
2022-02-25T00:07:25Z
Or, alternately, does anyone know why:tokenizer = Tokenizer(BPE.from_file('./tokenizer/roberta_tokenizer/vocab.json', './tokenizer/roberta_tokenizer/merges.txt'))
print("vocab_size: ", tokenizer.model.vocab)Fails with an error that'tokenizers.models.BPE' object has no attribute 'vocab'. According to the docs, it should have:Input sequences — tokenizers documentationAccording totokenizers.__version__I’m running 0.11.0. These docs are for 0.10.0—is vocab removed in 0.11.0? Or is something just borked in my install?UPDATE: I gave 0.10.1 a try, just for kicks, but same error.
2022-02-25T02:05:49Z
Digging in further, it looks like the difference must be between BPE and ByteLevelBPETokenizer (i.e., RoBERTa’s tokenizer). With the former, I get the 4000 item vocab I want, but the latter only gives me a 1300 item vocab (despite indicating 4000 in thevocab_size).So to get what I’m after, I have to either;figure out how to get the BPE version into a tokenizer that plays nice with transformers ORfigure out how to get the ByteLevelBPETokenizer to learn a 4000 item vocab
2022-02-25T22:56:32Z
Okay, I’ve made some progress with this approach:from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace
from transformers import PreTrainedTokenizerFast

train = True

if train:
    tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
    trainer = BpeTrainer(special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"])

    tokenizer.pre_tokenizer = Whitespace()
    tokenizer.train(files, trainer)

    tokenizer.save('./tokenizer/bpe_tokenizer/tokenizer.json')


fast_tokenizer = PreTrainedTokenizerFast(tokenizer_file="./tokenizer/bpe_tokenizer/tokenizer.json")

test = fast_tokenizer.encode(input)
print("test tokenization: ", test)

decode_test = fast_tokenizer.decode(test)
print("test decode: ", decode_test)Then I modifiedrun_clm.pyto usetokenizer = PreTrainedTokenizerFast(tokenizer_file=model_args.model_name_or_path)instead of the original AutoTokenizer version.This now works as expected. Worth noting is that youcan’tusePreTrainedTokenizer(i.e., the slow version) or you’ll hit aNotImplementederror when trying to calltokenizer.encode(input).
2022-03-01T02:01:53Z
That’s right. The methods of the tokenizers from the tokenizers library are different from the methods of the tokenizers from the transformers library. You want to use PreTrainedTokenizerFast from the transformers library to access the functionality of the transformers tokenizers.I suspect you want to be working with a character-level BPE rather than a byte-level BPE tokenizer.I’m also working on encoded music. We should talk
2022-03-01T16:10:22Z
Yes, that’s what I finally realized, in a kind of roundabout way.And yes again, we should talk!
2022-03-10T19:12:02Z
btw, Huggingface people, I’m still wondering if there’s any way to force a larger vocabulary during training? Presumably this would just be more “merging”, no? Shouldn’t there be a parameter to force a larger vocab if you want it?EDIT: I notice I was apparently getting the 4000 word vocab when I posted this, but that’s not the case now… I requestvocab_size=4000and I get 2026. Hmm…