Fine-tuning for translation with facebook mbart-large-50
I am trying to use the facebook mbart-large-50 model to fine-tune for en-ro translation task.raw_datasets = load_dataset(“wmt16”, “ro-en”)Referring to thenotebook, I have modified the code as follows.tokenizer = MBart50TokenizerFast.from_pretrained(‘facebook/mbart-large-50’, src_lang=“en_XX”, tgt_lang=“ro_RO”)max_input_length = 128max_target_length = 128source_lang = “en_XX”target_lang = “ro_RO”I am trying to use the facebook mbart-large-50 model to fine-tune for en-ro translation task.raw_datasets = load_dataset(“wmt16”, “ro-en”)Referring to thenotebook, I have modified the code as follows.Aloka:tokenizer = MBart50TokenizerFast.from_pretrained(‘facebook/mbart-large-50’, src_lang=“en_XX”, tgt_lang=“ro_RO”)max_input_length = 128max_target_length = 128source_lang = “en_XX”target_lang = “ro_RO”def preprocess_function(examples):inputs = [source_lang+’ ‘+ex[‘en’] for ex in examples[“translation”]]targets = [target_lang+’ '+ex[‘ro’] for ex in examples[“translation”]]model_inputs = tokenizer(inputs, max_length=max_input_length, padding=True, truncation=True, return_tensors="pt")


# Setup the tokenizer for targets
with tokenizer.as_target_tokenizer():
    labels = tokenizer(targets, max_length=max_target_length, padding=True, truncation=True, return_tensors="pt").input_ids


return model_inputsPlease let me know the following.tokenized_datasets = raw_datasets.map(preprocess_function, batched=True)Is the above step necessary? I am unable to run it as get the following error.TypeError: Providedfunctionwhich is applied to all elements of table returns adictof types [<class ‘torch.Tensor’>, <class ‘torch.Tensor’>]. When usingbatched=True, make sure providedfunctionreturns adictof types like(<class 'list'>, <class 'numpy.ndarray'>).If the above step is committed the below error is given during training. ie. trainer.train()We have to pad the labels before callingtokenizer.padas this method won’t pad them and needs them of the276         # same length to return tensors.*AttributeError: ‘tokenizers.Encoding’ object has no attribute ‘keys’Please let me know the correct way of passing the values.Thank you.Please let me know the following.tokenized_datasets = raw_datasets.map(preprocess_function, batched=True)Is the above step necessary? I am unable to run it as get the following error.TypeError: Providedfunctionwhich is applied to all elements of table returns adictof types [<class ‘torch.Tensor’>, <class ‘torch.Tensor’>]. When usingbatched=True, make sure providedfunctionreturns adictof types like(<class 'list'>, <class 'numpy.ndarray'>).If the above step is bypassed the below error is given during training. ie. trainer.train()We have to pad the labels before callingtokenizer.padas this method won’t pad them and needs them of the276         # same length to return tensors.*AttributeError: ‘tokenizers.Encoding’ object has no attribute ‘keys’Please let me know the correct way of passing the values.Thank you.
2024-03-16T12:22:45Z
Hi ,I am getting the same errorAttributeError: ‘tokenizers.Encoding’ object has no attribute ‘keys’. Did you solve it. Please help me with this…