Similarity search with combined image and text?
How can I do similarity match by combining both image and text?Lets stay:Product1 = Image1, Text1Product2 = Image2, Text2I want to do contrastive learning by combining both the image and text.Is there such a model?Can anyone please suggest a model?
2022-06-20T06:46:37Z
TheSentenceTransformercan encode images and text into a single vector space. You could combine both to create a new vector space for products, and then implement contrastive learning for this vector space.Seesentence-transformers/Image_Search.ipynb at master · UKPLab/sentence-transformers · GitHub
2022-06-20T08:17:38Z
Like in the notebook referenced by@raphaelmerx, I also used a pre-trained CLIP model to embed images and text in the same vector space, so you can perform semantic search:Weights & Biases.
2022-06-21T19:17:36Z
@raphaelmerxDo you have a sample code for contrastive learning using SentenceTransformer?
2022-06-21T19:46:12Z
@raphaelmerxI understand the idea of combining the text and image into a single vector space and then implement contrastive learning.But wondering are you aware of an open source implementation for doing contrastive learning? Or code that I could adapt for this purpose.
2022-06-24T00:34:04Z
@raphaelmerxin the given example, you have shownmodel.encodeto encode images and text. Do you have any example how to apply that for contrastive learning?
2022-06-24T04:35:53Z
I don’t have any code sample of contrastive learning no