Image captioning for Spanish with pre-trained vision and text model
Image captioning for Spanish with pre-trained vision and text modelFor this project, a pre-trained image model likeViTcan be used as an encoder, and a pre-trained text model like BERT and/or GPT2 can be used as a decoder.ModelPre-trainedViT,BERTmodels can be found on the model hub. We could use multi-lingual BERT/ROERTa models for the Spanish language.DatasetsTheWITdataset can be used for this task. It has almost over 1M image-text pairs for Spanish.Available training scriptsAs this will be a Seq2Seq model, therun_summarization_flax.pyscript can be used for training this model with some modifications.(Optional) Desired project outcomeThe desired outcome is to see if pre-trained vision and text models can be leveraged for image captioning and also train captioning models in for the Spanish language.  This can be showcased with a streamlit  or gradio app.(Optional) ChallengesThis model will require some modifications to the existing text models. Specifically, as this will be a seq2seq model, we’ll need to add a randomly initialized cross-attention layer in BERT or GPT2 to use it as a decoder in the encoder-decoder setting.(Optional) Links to read uponThis keras example presents an excellent example of how image encoder and transformer can be used for image captioning.keras.ioKeras documentation: Image CaptioningKeras documentation
2021-06-23T15:48:56Z
Hi,I would like to join this project. I am not experienced enough with NLP to create a project on my own. But I do have experience with Computer Vision overall, and would like to try and see how well spanish text is generated for the image-captioning.I am David and I am in  the Amsterdam timezone(CET). I have some experience with other frameworks, and training/evaluating object detection networks, but not with NLP. I can contribute by putting in my previous knowledge and discipline towards achieving this goal
2021-06-23T15:54:43Z
So you want to create a CLIP like (dual encoder) model?@valhalla
2021-06-23T17:13:55Z
Hey@dmatos2012, don’t worry about experience. We always try to make things easier for everyone and we have a super cool speaker lineup for getting familiar with JAX/Flax/Transformers. And we will try to answer all questions:)@mrm8488For image captioning it’ll be more like an encoder-decoder model. The encoder will be an image model and the decoder can be any transformer model with cross-attention which will takehidden_statesfrom image model and will generate text auto-regressively
2021-06-23T21:29:57Z
Awesome@valhalla! thanks
2021-06-27T00:44:24Z
Hey there, this is very interesting, I have some experience with NLP and computer vision, and always wanted to get more experience with multi-modal models (text + vision), also since I saw the WIT dataset for the first time, I wanted to use it for some project, this seems a good opportunity.If you want to know a little more about my background, check out myGitHub.
2021-06-29T15:24:46Z
Great, let’s define the project!@mrm8488let me know if you want to be added here as well
2021-06-30T03:26:38Z
Hello@valhalla@patrickvonplaten,Hope you are doing well…I am interested in this project as well… So much to learn from doing this project…It is similar to CLIP. But in Clip, we use encoders for both image & text… In this project, we are gonna use encoder for image & decoder for text.Also, I have a suggestion that we could create a separate thread indexed with all JAX/FLAX learning resources together in 1 place…It will be useful for all of us to learn…Cheers,Sri Lakshmi
2021-06-30T08:00:56Z
Hey@srisweet!Added you the teamGreat idea, feel free to create a thread to post JAX/Flax resources!
2021-06-30T08:41:55Z
Hi@valhalla. could you add me as well please? I replied at the top of the post :). Thank you!
2021-06-30T08:58:21Z
hi@dmatos2012you are already part of the teamHere’s thesheet
2021-06-30T09:00:21Z
oh@valhallaapologies, did not see that. Sorry!
2021-06-30T09:54:10Z
Hi@valhalla,Just created this thread with JAX/Flax resources…FLAX/JAX Learning ResourcesFlax/JAX ProjectsHello all, 
I have tried my best to organize FLAX/JAX learning resources. I hope it will be useful for all of us. 
If you have more learning resources, drop a comment & I will add them here…JAXis a numerical computation library that exposes a NumPy-like API with tracing capabilities. 
Flax is the most widely used JAX library, with129 dependent projectsas of May 2021.Flaxbuilds on top of JAX with an ergonomic module abstraction using Python dataclasses that leads to concise and explicit c…
2021-07-19T08:13:49Z
Hi Suraj,Me and@gchhablaniare working on spanish image captioning currently with CLIP + Marian