Train Dutch FlaxBigBird
FlaxBigBird for Dutch languageCurrently, there are only a very limited amount of long-range sequence models for other languages than English on the hub. The goal of this project is to create a strong Dutch FlaxBigBird model.ModelA randomly initialized  FlaxBigBird model.DatasetsCC-100mC4Available training scriptsScripts to pre-train Flax models are here:transformers/examples/flax/language-modeling at master · huggingface/transformers · GitHubScripts for fine-tuning and evaluation for FlaxBigBird just got released here:transformers/examples/research_projects/jax-projects/big_bird at master · huggingface/transformers · GitHubThanks to@vasudevgupta(Optional) Desired project outcomeThe desired project output is a strong Dutch FlaxBigBird model in Dutch. For downstream tasks that require long sequences (e.g. long text classification).(Optional) Challenges
2021-06-25T17:31:23Z
Agree it would be great to have more long-range sequence models for Dutch! We don’t have Longformer implemented in Jax/Flax - maybe FlaxBigBird could be an option?
2021-06-25T19:32:09Z
Yes even better! Thanks for the suggestion, I will change it.
2021-06-29T14:40:32Z
Let’s try to get some participants for this project - think it’s a great idea
2021-07-02T08:10:39Z
I could help a bit here and there. I have experience with getting and preprocessing texts, comfortable in the shell and python. Knowledge of transformer architectures from a users perspective. Unfortunately missed the flax/jax intro’s.
2021-07-02T15:26:58Z
Wuhu, let’s create it!
2021-07-02T16:08:25Z
I’ll join! Sounds like a fun project.