Finetune t5 for English-Vietnamese translation
I finetuned t5-small model followingthis official example script.After training, I generated the translation using the following code:from transformers import AutoTokenizer, AutoModelForSeq2SeqLMmax_input_length = 128max_target_length = 128model_name = “models”model = AutoModelForSeq2SeqLM.from_pretrained(model_name)tokenizer = AutoTokenizer.from_pretrained(model_name)inputs = [“translate English to Vietnamese: I am so happy today”]model_inputs = tokenizer(inputs, max_length=max_input_length, truncation=True, return_tensors=‘pt’, padding=True)translation = model.generate(**model_inputs)tokenizer.batch_decode(translation, skip_special_tokens=True)[‘Tôi rt hi ngày nay’]tokenizer.batch_decode(translation, skip_special_tokens=False)“[‘ Tôi rt hi ngày nay’]”As you can see, many utf-8 characters are not recognized. The true translation would be: “Tôi rất hạnh phúc ngày hôm nay”.What has gone wrong? Do I need to train from the scratch given that Vietnamese is not used for pre-training of t5?
2022-05-13T14:58:45Z
Since this site does not allow special characters, I’m adding the following figure to illustrate the output whenskip_special_tokens = False.2022-05-12_16-38-49776×116 102 KB
2022-05-28T20:58:38Z
I am facing the same problem, someone help please!!