Xlm tokenizer.lang2id is None
I was just followinghuggingface.coMultilingual models for inferenceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.to play with the model “xlm-clm-enfr-1024”. It turns out thatprint(tokenizer.lang2id)gives meNoneinstead of{'en': 0, 'fr': 1}. Any comment on this?
2021-06-04T19:33:34Z
I am also getting the same issue. I tried runningimport torch
from transformers import XLMTokenizer, XLMWithLMHeadModel

tokenizer = XLMTokenizer.from_pretrained("xlm-clm-enfr-1024")
model = XLMWithLMHeadModel.from_pretrained("xlm-clm-enfr-1024")

language_id = tokenizer.lang2id['en']And I am seeing NoneType error