Find the eqivalent for word.index in BERT?
excuse me i need to get like a dictionary contains the word with its index like thisdef word_for_id(integer, tokenizer):
    for word, index in tokenizer.word_index.items():
        if index == integer:
            return word
    return Nonebut with using BERT i couldn’t find the equivalent as i gotberttokenizer' object has no attribute 'word_index'
2021-12-29T19:07:26Z
BERT has word-piece tokens, so if you are after the associated IDs for these word-piece tokens, you can find these.from transformers import AutoModel
checkpoint = 'bert-base-uncased'
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
tokenizer.vocab
2021-12-29T23:18:09Z
thanks a lot for replying …  excuse me what is the difference between tokenizer.get_vocab() and tokenizer. vocab ? are both work the same?
2021-12-31T16:25:36Z
excuse me do i need to load my sentences to get the vocab or the vocab here for the pre-trained bert ?
2021-12-31T21:52:20Z
iamholmes:from transformers import AutoModel
checkpoint = 'bert-base-uncased'
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
tokenizer.vocabFrom a very quick scan, tokenizer.vocab is an attribute. tokenizer.get_vocab() is a function. They both return a dictionary with the same number of items. I’m not sure why we need both.