text
we are very proud to introduce 
we are very proud to introduce 
resource
https://huggingface.co/jinaai/jina-clip-v1
model
jinaai/jina-clip-v1
https://huggingface.co/jinaai/jina-clip-v1
text
, aka "jina-embeddings-multimodal".
, aka "jina-embeddings-multimodal".
new_line
new_line
text
The OpenAI CLIP 
The OpenAI CLIP 
resource
https://huggingface.co/openai/clip-vit-base-patch32
model
openai/clip-vit-base-patch32
https://huggingface.co/openai/clip-vit-base-patch32
text
 have nice performance to align text and image modality, that user can perform cross-modal text image retrieval or image classification on top of it.  However, due to the training data and recipe, it can not:
 have nice performance to align text and image modality, that user can perform cross-modal text image retrieval or image classification on top of it.  However, due to the training data and recipe, it can not:
new_line
new_line
text
1. model longer sequence of text inputs (77 token constraint).
1. model longer sequence of text inputs (77 token constraint).
new_line
text
2. align text representations (CLIP Text Tower is weak for text search).
2. align text representations (CLIP Text Tower is weak for text search).
new_line
new_line
text
In our latest publication, 
In our latest publication, 
resource
https://huggingface.co/papers/2405.20204
paper
2405.20204
https://huggingface.co/papers/2405.20204
Jina CLIP: Your CLIP Model Is Also Your Text Retriever (2405.20204)
text
 , we proposed a multi-task, multi-objective learning scheme. The produced CLIP model shows:
 , we proposed a multi-task, multi-objective learning scheme. The produced CLIP model shows:
new_line
new_line
text
1. Stronger cross-modal performance against OpenAI sets, 2% and 6% improvement on cross-modal retrieval recall@5.
1. Stronger cross-modal performance against OpenAI sets, 2% and 6% improvement on cross-modal retrieval recall@5.
new_line
text
2. Text tower of the JinaCLIP is a strong text encoder, reach the same performance as 
2. Text tower of the JinaCLIP is a strong text encoder, reach the same performance as 
resource
https://huggingface.co/jinaai/jina-embeddings-v2-base-en
model
jinaai/jina-embeddings-v2-base-en
https://huggingface.co/jinaai/jina-embeddings-v2-base-en
text
, 165% improvement on MTEB[BEIR] recall@5.
, 165% improvement on MTEB[BEIR] recall@5.
new_line
text
3. Image tower of the JinaCLIP also shows strong performance in image-image search (CBIR), 12% recall improvement on Cifar100 test set.
3. Image tower of the JinaCLIP also shows strong performance in image-image search (CBIR), 12% recall improvement on Cifar100 test set.
new_line
new_line
text
If you are working on MuRAG (multimodal-retrieval argumented generation), try it out!
If you are working on MuRAG (multimodal-retrieval argumented generation), try it out!
new_line
new_line
new_line
we are very proud to introduce https://huggingface.co/jinaai/jina-clip-v1, aka "jina-embeddings-multimodal".

The OpenAI CLIP https://huggingface.co/openai/clip-vit-base-patch32 have nice performance to align text and image modality, that user can perform cross-modal text image retrieval or image classification on top of it.  However, due to the training data and recipe, it can not:

1. model longer sequence of text inputs (77 token constraint).
2. align text representations (CLIP Text Tower is weak for text search).

In our latest publication, https://huggingface.co/papers/2405.20204 , we proposed a multi-task, multi-objective learning scheme. The produced CLIP model shows:

1. Stronger cross-modal performance against OpenAI sets, 2% and 6% improvement on cross-modal retrieval recall@5.
2. Text tower of the JinaCLIP is a strong text encoder, reach the same performance as https://huggingface.co/jinaai/jina-embeddings-v2-base-en, 165% improvement on MTEB[BEIR] recall@5.
3. Image tower of the JinaCLIP also shows strong performance in image-image search (CBIR), 12% recall improvement on Cifar100 test set.

If you are working on MuRAG (multimodal-retrieval argumented generation), try it out!



🚀
florianhoenicke florian-hoenicke zhenwang23 mwerk numb3r3 alaeddineabdessalem osanseviero alexcg2 nan den0620 capricareloaded Jofthomas
❤️
florianhoenicke florian-hoenicke zhenwang23 alaeddineabdessalem osanseviero alexcg2 nan capricareloaded Jofthomas
🔥
florianhoenicke florian-hoenicke alaeddineabdessalem osanseviero alexcg2 nan capricareloaded jaisanrobert
👍
zhenwang23 alaeddineabdessalem osanseviero alexcg2 sbarman25 nan capricareloaded florianhoenicke