text
If you have documents that do not only have text and you're doing retrieval or RAG (using OCR and LLMs), give it up and give ColPali and vision language models a try 🤗
If you have documents that do not only have text and you're doing retrieval or RAG (using OCR and LLMs), give it up and give ColPali and vision language models a try 🤗
new_line
new_line
text
Why? Documents consist of multiple modalities: layout, table, text, chart, images. Document processing pipelines often consist of multiple models and they're immensely brittle and slow. 🥲
Why? Documents consist of multiple modalities: layout, table, text, chart, images. Document processing pipelines often consist of multiple models and they're immensely brittle and slow. 🥲
new_line
new_line
text
How? ColPali is a ColBERT-like document retrieval model built on PaliGemma, it operates over image patches directly, and indexing takes far less time with more accuracy. You can use it for retrieval, and if you want to do retrieval augmented generation, find the closest document, and do not process it, give it directly to a VLM like Qwen2-VL (as image input) and give your text query. 🤝
How? ColPali is a ColBERT-like document retrieval model built on PaliGemma, it operates over image patches directly, and indexing takes far less time with more accuracy. You can use it for retrieval, and if you want to do retrieval augmented generation, find the closest document, and do not process it, give it directly to a VLM like Qwen2-VL (as image input) and give your text query. 🤝
new_line
new_line
text
This is much faster + you do not lose out on any information + much easier to maintain too! 🥳
This is much faster + you do not lose out on any information + much easier to maintain too! 🥳
new_line
new_line
text
Multimodal RAG 
Multimodal RAG 
resource
https://huggingface.co/collections/merve/multimodal-rag-66d97602e781122aae0a5139
collection
merve/multimodal-rag-66d97602e781122aae0a5139
https://huggingface.co/collections/merve/multimodal-rag-66d97602e781122aae0a5139
text
 💬
 💬
new_line
text
Document AI (made it way before, for folks who want structured input/output and can fine-tune a model) 
Document AI (made it way before, for folks who want structured input/output and can fine-tune a model) 
resource
https://huggingface.co/collections/merve/awesome-document-ai-65ef1cdc2e97ef9cc85c898e
collection
merve/awesome-document-ai-65ef1cdc2e97ef9cc85c898e
https://huggingface.co/collections/merve/awesome-document-ai-65ef1cdc2e97ef9cc85c898e
text
 📖
 📖
If you have documents that do not only have text and you're doing retrieval or RAG (using OCR and LLMs), give it up and give ColPali and vision language models a try 🤗

Why? Documents consist of multiple modalities: layout, table, text, chart, images. Document processing pipelines often consist of multiple models and they're immensely brittle and slow. 🥲

How? ColPali is a ColBERT-like document retrieval model built on PaliGemma, it operates over image patches directly, and indexing takes far less time with more accuracy. You can use it for retrieval, and if you want to do retrieval augmented generation, find the closest document, and do not process it, give it directly to a VLM like Qwen2-VL (as image input) and give your text query. 🤝

This is much faster + you do not lose out on any information + much easier to maintain too! 🥳

Multimodal RAG https://huggingface.co/collections/merve/multimodal-rag-66d97602e781122aae0a5139 💬
Document AI (made it way before, for folks who want structured input/output and can fine-tune a model) https://huggingface.co/collections/merve/awesome-document-ai-65ef1cdc2e97ef9cc85c898e 📖
👍
adorkin web3builder John6666 Percifal jrmasiero rwightman seek007 abishekcodes zliu AI4Industry louisbrulenaudet byteprobe muhtasham rumbleFTW
🔥
umair894 abishekcodes fsommers jithinrocs rumbleFTW
❤️
Csplk rumbleFTW madstuntman11
https://cdn-avatars.huggingface.co/v1/production/uploads/6444b3135af87c73bbbd7447/-WLquJY3E1KZSJbnYUkwD.jpeg
Frank Sommers
fsommers
user