CLIP like contrastive vision-language models for Spanish with pre-trained text and vision models
CLIP like contrastive vision-language models for Spanish with pre-trained text and vision modelsFor this project, a pre-trained image model likeViTand a pre-trained text model likeBERTcan be used as an image encoder and text encoder respectively.ModelPre-trainedViT,BERTmodels can be found on the model hub. We could also use multi-lingual BERT/ROBERTa models for the Spanish language.DatasetsTheWITdataset can be used for this task.Available training scriptsA training script for this will be provided soon. (seePR)(Optional) Desired project outcomeThe desired outcome is to train a CLIP-like model for Spanish language. This can be showcased with a streamlit or gradio app.(Optional) ChallengesThis model will require some modifications to the existing models. Specifically, we will need to add projection layers in both the text and image encoder models.
2021-06-28T12:57:48Z
Count me in!
2021-06-28T13:07:07Z
We will do it!
2021-06-28T15:56:07Z
I would love to participate too!
2021-06-29T15:22:58Z
Awesome! Let’s define the project