Select Source and Target Langauge in multi-language translation models
I want to fine tune a Facebook NLLB model for translation, my question is, how can I specify the source and target language for the tokenizer and the trainer?Is there any other information I need to know too? I’m a beginner.
2024-08-14T11:35:35Z
You just need to load the right tokenizer  when you first load the tokenizer. Rest is similar to finetuning of translation model.from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

MODEL_REPO = "facebook/nllb-200-1.3B"
tokenizer = AutoTokenizer.from_pretrained(MODEL_REPO, src_lang="jpn_Jpan", tgt_lang="eng_Latn")
model = AutoModelForSeq2SeqLM.from_pretrained(MODEL_REPO)See more detail on using NLLBtokenizer:NLLB200 - Hugging face docs