CLIP like contrastive vision-language models for German with pre-traind text and vision models
CLIP like contrastive vision-language models for German with pre-trained text and vision modelsFor this project, a pre-trained image model likeViTand a pre-trained text model likeBERTcan be used as an image encoder and text encoder respectively.ModelPre-trainedViT,BERTmodels can be found on the model hub. We could also use multi-lingual BERT/ROBERTa models for the German language.DatasetsTheWITdataset can be used for this task.Available training scriptsA training script for this will be provided soon. (seePR)(Optional) Desired project outcomeThe desired outcome is to train a CLIP-like model for German language.  This can be showcased with a streamlit  or gradio app.(Optional) ChallengesThis model will require some modifications to the existing models. Specifically, we will need to add projection layers in both the text and image encoder models.(Optional) Links to read uponSachin’s Blog – 7 Mar 21Multilingual CLIP with Huggingface + PyTorch Lightning 🤗 ⚡Training OpenAI’s CLIP on google colab
2021-06-25T12:38:25Z
It’s quite an interesting project ,count me in
2021-06-29T13:19:44Z
Hi guys!Sounds like an interesting topic to get started with Jax/Flax and CLIP.My main background is from recommender systems and NLP but I am interested in overlaps with CV, which makes this a great place to start!Count me in!
2021-06-29T15:12:59Z
Great defining this project! cc@valhalla
2021-07-01T21:00:14Z
This project sounds interesting! I would love to join it. I have experience with GANs, Computer Vision, Pytorch Lightning, and at the same time studying German (B1). It would be a great learning experience for me. Please let me know how can I join?
2021-07-04T05:50:41Z
Where can I find the project in discord? Whats the channel name?