Pretrain and Fine Tune Byte-level model for multilingual extractive QA (Like ByT5)
I am really fascinated by Byte-level models which have been proposed currently such as ByT5 and Canine that get rid of tokenizers. I have tried probing the ByT5 model to check whether it learns morphological and syntactic features.Project Proposal: I intend to pre-train the ByT5 model from scratch for a few languages(currently it is trained on mC4 corpus which has 100+ languages) and then fine-tune the model for zero extractive QA. More information can be foundhereGoal: At the end, I would like to have insights into how does the ByT5 model performs when random syenitic noise is added to the data (See the last part of the experimental section of the paper for more information).Languages: As this is a multilingual project, I am considering a set of languages that belong to a family(Like the Indian Languages or European Languages) and the dataset used would be part of mC4 data set(this can be changed if we get more data)Dataset for Multilingual QA:MLQAor XPretraining script; The fine-tuning script is available in the Transformer’s library,  pretarning script is available in the officialrepo of the modelChallenges: Getting the pretraining script running with JAX may be a challenge, though I am not sure how challenging it would be.
2021-06-24T12:07:53Z
Hey there,@sidicity,The project idea soundsinterestingand I have been wanting to work on training Multi-lingual models myself.Count me in!
2021-06-27T05:28:21Z
Count me in.
2021-06-28T03:57:32Z
I’m interested! Hope I can also contribute a bit.
2021-06-28T05:30:00Z
Effects of tokenization on extractive QA is one of my past research interests. I would be very interested in seeing the results from this experiment. Even though I think I will participate in another topic for this event.
2021-06-28T07:49:46Z
@sidicityLove the idea. As a highly agglutinative language user, dealing with current count-based tokenizers is troublesome and doesn’t even seem right. I would love to join this project.
2021-06-28T09:55:08Z
Transfer learning on the encoder side has been explored to great extent in various papers, I am not sure if there are many papers that do transfer learning on the decoder side especially when the decoder is generative. Like if we had a sequence labeling problem, then transfer of labels is possible from one language to another, but  in case of generative decoder, that seems a difficult problem, any pointers here@admins?
2021-06-28T16:59:54Z
Really like to see this project! Think most of the code exists (the T5-like pretraining code will be merged today/tomorrow for JAX, see:https://github.com/huggingface/transformers/pull/12355).It will be important to decide on which languages exactly ByT5 should be pretrained on! Think it’s a good idea to make sure that the languages are related/close to each other.Putting everybody that is interested in an official project for now
2021-06-28T17:04:55Z
So, maybe it makes sense to limit the project to:PreTrain ByT5-base to a family of languages (to be defined)Fine-tune the pretrained model on extractive QAEvaluate on corrupted input(maybe) compare to officially released multi-lingual checkpoints
2021-06-28T18:33:24Z
Would love to contribute in this project.
2021-06-30T14:48:09Z
I’m also interested in this one. I have trained multilingual models for indian languages. But we  transliterated to a common script and then we used sentencepiece for  tokenization. It would be great to use a model that can automatically figure out how to tokenize.
2021-07-01T10:25:26Z
Great added both of you
2021-07-02T00:26:28Z
Did you guys set up a discord channel yet?Giving you guys directly two TPUs tomorrow! Split the team randomly into two in the official google sheet, but this shouldn’t change anything - just that you have access to 2 TPU v3-8sMight make organization a bit easier to split work on two VMs!
2021-07-02T21:53:53Z
Yes, we had setup a Discord channelFlax-HuggingFace-Community-Week.   On the TPU, we only received 1 TPU name for our team (which I believe should correspondence to only 1 TPU v3-8 , is the second TPU coming soon or we need to get a second person to put his name in the second slot of the TPU request signing form?