Translation - MBART, translation with identical source and target language, for text normalization
Hi,This is rather a general question about translation and I am aware that I don’t follow exactly your guidelines, so I am sorry for that.(We could run the examples mentioned in your readme, great tool!)We try to conceive normalization for Dutch, as a ‘translation’ task.So, is it for instance possible to use source + target language, defined as the same language, for instance--source_lang nl_XX \
--target_lang nl_XX \{“translation”: {“nl_XX”: “liefst geen energie vandaag . waar ism’noplaadstation ?”, “nl_XX”: “liefst geen energie vandaag . waar ismijnoplaadstation ?”}}or--source_lang source\
--target_lang target \{“translation”: {“source”: “liefst geen energie vandaag . waar ism’noplaadstation ?”, “target”: “liefst geen energie vandaag . waar ismijnoplaadstation ?”}}Environment infogithub.comhuggingface/transformersmaster/examples/pytorch/translation🤗 Transformers: State-of-the-art Natural Language Processing for Pytorch, TensorFlow, and JAX.–model_name_or_path facebook/mbart-large-50-many-to-many-mmtThanks for your answer!
2021-07-14T15:11:21Z
You should change the data processing in the script as thesource_langandtarget_langare also used to set the langauges on multilingual tokenizers (if you’re using mBart for instance, this approach wouldn’t work).
2021-07-14T19:03:56Z
Hi,thanks for your answer.So as I understand it, using MBart with identical source and target language is not feasible with the current architecture?Thanks
2021-07-14T19:08:38Z
It is, but you need to change a bit the way the data is processed in the script since you can’t have a dictionary with the same key twice.