Multilingual Image Captioning
We’re planning to use ViT encoder, mBART decoder and train them end-to-end for image captioning in different languages.ModelPre-trainedViT,mBART(will be merged soon) can be leveraged for our task.DatasetsCurrently we’re thinking of usingWIT.COCOdataset can also be used but it has caption only in English language. Dataset can be changed later if we come across more suitable dataset for this task.Available training scriptsSince this is a Seq2Seq model, therun_summarization_flax.pyscript can be used for training this model with some modifications.(Optional) Desired project outcomeOur end use case would be to run this model on a video clip or movie and make it like an accessibility tool for visually challenged people. This can be showcased with a streamlit or gradio app.We’re also planning to benchmark its zero-shot performance onVizWiz(Optional) Challenges-Training encoder-decoder model end-to-end in FLAX/JAX will require some effort-Data Processing incase new datasets are available only in English
2021-06-29T18:07:52Z
I am in@abheesht@Vaibhavbrkn@nilshoehinghave also shown interest in this sheet:JAX/Flax Vision-and-Language Projects - Google SheetsOthers are welcome to join!
2021-06-29T18:19:44Z
Yes, I am also in!
2021-06-30T03:45:55Z
I am also in
2021-06-30T04:10:35Z
consider me on this
2021-06-30T04:48:08Z
I am interested in this task.
2021-06-30T04:50:27Z
We can also use this  datasetGitHub - bharathichezhiyan/multimodalmachinetranslation-Tamil: Multilingual Multimodal Machine Translation for Dravidian Languages utilizing Phonetic Transcription
2021-06-30T08:45:25Z
Yep, I’m in,@bhavitvyamalik,@gchhablani
2021-06-30T10:13:19Z
Great Idea!One important thing is to define the scope of the project, specifically which and how many languages to use. We should decide this based on the time and compute constraint so that the project can be finished in timeIt’s also possible to use other multilingual models like mBERT of XLM-ROBERTa, but as those are encoder-only models we will need to add a cross-attention layer in it.I’m officially defining this project!Added you the team@bhavitvyamalik,@gchhablani,@nilshoehing,@knilakshan20,@Monda,@rays2pix,@abheesht,@VaibhavbrknLet me know if you have any comments
2021-07-02T11:13:05Z
Hey allI am a beginner with Transformers and FLAXand want to get into  Transformers via a project. Basically I would like to be able to work on getting a minimal training pipeline built and a model trained using HuggingFace’s infraI have sufficient background in CV and Deep Learning (Object detection.Semantic Seg etc via SSD, U-Net) in the autonomous driving spaceI do like this idea and would like to sign up. Do you think I can help out in some way?
2021-07-06T12:01:53Z
I would like to work on this Project too.