Model to translate between Norwegian Bokmål and Norwegian Nynorsk
Norwegian has two different written standards or variants, Bokmål and Nynorsk. Some mean they should be considered two different languages, but with extensive lexical as well as grammatical overlap. In many cases a word-by-word translation suffices, but there are for instance also some auxiliary and voice constructions that are different in addition to differing pronoun systems. Students and civil servants are expected to be fluent in both languages and you have as a Norwegian citizen the right to receive public documents in your preferred language. Nynorsk is by far the minority, used in less than 20% of the cases.Translation between Bokmål and Nynorsk seems like a low hanging machine translation task, especially since the loss can be calculated more effectively. Even not translating at all, will probably give you a BLEU score in the area of 20-30 because of the overlaps.There are no recent machine translation-models trying to do this translation task, most likely due to the lack of available corpora. We are however able to make such a dataset available for this workshop.LanguageWe aim at using as much of the available Flax code as possible.ModelTo be decided by the team. To our knowledge, both BERT-infused NML-models, T5-like models and BART might be suitable for such a task. Model choice might be influenced based on what modules are available in FLAX and performance.DatasetsFor this workshop we are able to make publicly available two pre-prepared datasets collected though other projects:A balanced (5+5=10GB) corpus in Bokmål and Nynorsk. Collected through various sources, MC4, Oscar, Wikipedia, public documents, pdf-scans. There is even more text available in Bokmål here, but we feel that a balanced set would be best for this task.A high quality parallel corpus of 100.000 sentence-pairs in Bokmål and Nynorsk.The sets are already thoroughly cleaned, and we will make sure they are ready to import by the start of the workshop.Training scriptsThe goal is to use the current scripts as much as possible due to the short timespan.ChallengesThe first challenge is to find a model architecture that can give a reasonable result in this timeframe. It is however reasonable to assume that this will be a much easier task than translating from for instance English to German.For the project to be realistic, we need team members with previous experience with machine translation. The proposers have solid experience in training large language models, but lack experience with translation. Since the timeframe for the workshop is short, we should find suitable models already implemented in Flax.No knowledge of Norwegian is necessary.Desired project outcomeIt is a tech demo, hoping to show that it is doable to build a mainly unsupervised machine learning algorithm with Flax in a week.
2021-07-01T09:33:19Z
This is an interesting idea, but I wonder if we’d had enough data to test a simple fine-tuning first.
2021-07-01T10:50:19Z
We might get a decent result here by just finetuning mBART on the Bokmål-Nynorsk parallell corpus. Unfortunately Nynorsk is a minority language (really a minority subset of a minority language).
2021-07-01T16:46:40Z
According to todays presentations. Pretraining scripts for BART is not yet available. A T5 model would probably do fine here, and both pretraining and finetuning scripts are available.
2021-07-02T00:04:18Z
Awesome finalizing this project then