Help with finetuning mBART on an unseen language
Hi everyone,I wanted to know how would we finetune mBART on a summarization task on a different language than that of English. Also, how can we finetune mBART on a translation task where one of the languages is not present in the language code list that mBART has been trained on.Appreciate any help!! Thank you.
2020-10-13T15:57:32Z
Hi@LaibaMehnazDISCLAIMER: I haven’t tried this myself , and as Sam found in his experiments mBART doesn’t always give good results.for mBart source seq ends with src lang id and tgt seq starts with the tgt seq id. so for summ you can pass the same lang id for both source and tgt lang and then finetune it the same way you finetune any other seq2seq model.For translation, if the lang is not present, you can try without using any lang id in the sequences.
2020-10-14T14:01:57Z
Hi@valhalla,I did try using mBart without any lang id, but it gives me this error:self.cur_lang_code = self.lang_code_to_id[src_lang]KeyError: ‘’
2020-10-14T16:01:06Z
Also, when I am using the same language code on both the sides, the generations are in a totally different script.
2020-10-16T17:19:55Z
For this, you’ll need tokenize input and output seq without usingprepare_seq2seq_batchmethod or overrideprepare_seq2seq_batchand modify it to not use lang id
2020-10-19T13:02:10Z
I modified the tokenizer to not use the lang id as you suggested, but still the same problem. ROUGE is 0.0, as the generations are in another script.
2020-10-20T06:16:03Z
Also, is it possible because I am using tiny-mbart and not mbart-large-cc25. I was trying out tiny-mbart due to memory constraints.
2020-10-20T06:45:03Z
hi@LaibaMehnaztiny-mbartis just meant for testing, it’s randomly initialised model.You could try to create a smaller student usingmake_student.pyscript.
2020-10-20T07:55:55Z
Oh, thanks a lot. I will proceed this way and let you know. Thanks again:)
2020-10-20T08:15:01Z
Also, how many encoder and decoder layers would you suggest?
2020-10-20T11:38:38Z
hard to say, depends on problem. But you could start with same number of encoder layers and 6 decoder layers distillbart-12-6 performs really well on summarization
2020-10-20T11:54:31Z
Alright, thank you so much.
2020-10-23T00:15:01Z
Hi, I am also interested in topic and I am trying to add mBart functionality to another library, but I have encountered strange error:https://huggingface.co/transformers/model_doc/mbart.htmlstates thatprepare_seq2seq_batchshould give me dict with this keys:[input_ids, attention_mask, decoder_input_ids,  decoder_attention_mask], but actually it gives me[input_ids, attention_mask, labels]. I am a bit confusedIs it a bug or me doing something wrong?
2020-10-24T07:28:42Z
Hi@ZhylkaaaThe doc is incorrect. Theprepare_seq2seq_batchreturns[input_ids, attention_mask, labels], and it’s not a bugcc.@sshleifer
2020-10-24T19:33:02Z
Hi@valhallathanks for your respond, but how I am supposed to create decoder inputs? because there is difference in lang_id positionshould I use something like:[lang_id] + prepare_seq2seq_batch(decoder_input)['input_ids'][:-1]+ padding if required?Or should I just modifyprepare_seq2seq_batchthrowing away lang_id for summarisation task? (I am not sure about this modification because my intuition tells me that lang_id is something like language conditioned [CLS] token, or my intuition is wrong again😁?)Thanks!
2020-10-27T03:42:33Z
I have read that I can tag@sshleiferfor summarisation and BART problems/questions. Sorry if I am wrong.
2020-10-29T17:27:26Z
You can keep lang id for summarisation, you could pass the same lang id assrc_langandtgt_langtoprepare_seq2seq_batchmethod
2020-10-29T21:05:37Z
Thank you@valhalla,but what aboutdecoder_input_ids? because I doesn’t receive this value after I useprepare_seq2seq_batch
2020-10-30T05:45:33Z
finetune.pyandfinetune_trainer.pywill make the right rightdeocder_input_ids, you won’t need to pass them
2020-10-30T20:41:27Z
thanks, actually I’ve been digging through source code and found thatforwardmethod actually generatesdecoder_input_idsfromlabelsthroughshift_tokens_right. Thank you for help, and sorry for being annoying, should have checked source code first