Does T5Tokenizer support the Greek language?
Does T5Tokenizer support the Greek language?When I run the 3 lines of code below, then the input_ids are just 2 and 3 which correspond to the unknown token and the underscore respectively. This is the same for any input text of Greek letters.from transformers import T5Tokenizertokenizer = T5Tokenizer.from_pretrained(“t5-small”)input_ids = tokenizer(‘Γειά σου Κόσμε’, return_tensors=‘pt’).input_ids
2021-11-24T10:23:29Z
Hi,T5 itself was trained on English data only. However, there’s a multilingual variant calledmT5which supports Greek.