Multilingual Visual Question Answering
Multilingual Visual Question AnsweringWe are currently planning to use ViT + mBert for two modalities namely images and text for Multilingual Visual Question Answering taskModelPre-trained modelsViT, mBERT , can be used for this task.4. DatasetsCurrently we came across one datasetWITwhich is a large multimodal multilingual dataset , We can update the datasets if we encounter more suitable dataset for this task.5. Training scriptsSince this is a Seq2Seq model, therun_summarization_flax.pyscript can be used for training this model with some modifications.6. (Optional) ChallengesHow to combine the two modalities i.e. image and text, some of the ways to this is likeVisualBert,Here we can use ViT for images and use the embeddings from earlier stage for mBERT along with word embeddings. One other way for interaction of two modalities can be done through coattentional transformer layer like inViLBERT, so we need to decide which would be better suitable for our task.7. (Optional) Desired project outcomeFinal Goal is to have an end to end model where we can perform Visual QA Task in multiple languages.We are also planning to benchmark the few-Shot/zero-shot performance onVQA/VizWiz/GQA
2021-06-30T07:25:56Z
Count me in!
2021-06-30T11:55:24Z
Awesome let’s finalize this project
2021-07-01T12:18:57Z
Great project. Looking to forward to working with you.
2021-07-01T18:32:54Z
I want to join this project. I hope I’m not late.
2021-07-01T23:52:43Z
Added you guys!
2021-07-02T11:26:38Z
Hey allI am a beginner with Transformers and FLAXand want to get into Transformers via a project. Basically what I would like to take away is - knowledge about what it means  to get a pre-trained model tuned to another task using HuggingFace’s infrastructure/pipeline and learn some  FLAX basicsI have sufficient background in CV and Deep Learning (Object detection.Semantic Seg etc via SSD, U-Net) in the autonomous driving space using PyTorchI do like this idea and would like to sign up. Do you think I can help out in some way?
2021-07-02T17:47:10Z
Hi guys, I would also like to be a part of this project.
2021-07-02T19:31:09Z
Hi@patrickvonplaten, can you add me here in this Multilingual Visual QA ?