Use custom text encoder in CLIP
Hi guys!I want to train a CLIP model for Ukrainian and would like to replace a text encoder with pretrained Ukrainian text encoder. I found a couple of discussions about CLIP implemented in Flax for theSpanishandKoreanlanguages and a research papers about replacing CLIP text encoder for other languages (AltCLIP), but they all kinda reimplement CLIP for that.Is there an easy way to initialize a CLIPModel but with custom text encoder (available on hf hub)? Same question for replacing the tokenizer in the CLIPProcessor.I would like something like this:model = CLIPModel(text_encoder=my_text_encoder, image_encoder=my_image_encoder)
processor = CLIPProcessor(tokenizer=my_tokenizer, image_processor=my_image_processor)
2024-01-22T13:57:29Z
Hi, the solution i found is to use VisionTextDualEncoder class rather than CLIPModel class of transformers library where you can customize the vision and text encoders : " This model can be used to align the vision-text embeddings using CLIP like contrastive image-text training and then can be used for zero-shot vision tasks such image-classification or retrieval."Not sure all text encoders architectures are compatible but worth a try !
2024-01-22T14:07:00Z
Thank you so much! This looks like exactly what I need! Another thing I didn’t mention was that it’s actually for SigLIP model and as I seeVisionTextDualEncoderonly uses CLIP loss for now. I will create a PR to make it work with SigLIP loss as well
2024-01-23T02:07:04Z
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.