text
Forget any document retrievers, use ColPali 💥💥
Forget any document retrievers, use ColPali 💥💥
new_line
new_line
text
Document retrieval is done through OCR + layout detection, but you are losing a lot of information in between, stop doing that! 🤓
Document retrieval is done through OCR + layout detection, but you are losing a lot of information in between, stop doing that! 🤓
new_line
new_line
text
ColPali uses a vision language model, which is better in doc understanding 📑 
ColPali uses a vision language model, which is better in doc understanding 📑 
new_line
text
ColPali: 
ColPali: 
resource
https://huggingface.co/vidore/colpali
model
vidore/colpali
https://huggingface.co/vidore/colpali
text
 (mit license!)
 (mit license!)
new_line
text
Blog post: 
Blog post: 
link
https://huggingface.co/blog/manu/colpali
https://huggingface.co/blog/manu/colpali
new_line
text
The authors also released a new benchmark for document retrieval: 
The authors also released a new benchmark for document retrieval: 
new_line
text
ViDoRe Benchmark: 
ViDoRe Benchmark: 
resource
https://huggingface.co/collections/vidore/vidore-benchmark-667173f98e70a1c0fa4db00d
collection
vidore/vidore-benchmark-667173f98e70a1c0fa4db00d
https://huggingface.co/collections/vidore/vidore-benchmark-667173f98e70a1c0fa4db00d
new_line
text
ViDoRe Leaderboard: 
ViDoRe Leaderboard: 
resource
https://huggingface.co/spaces/vidore/vidore-leaderboard
space
vidore/vidore-leaderboard
https://huggingface.co/spaces/vidore/vidore-leaderboard
new_line
new_line
text
ColPali marries the idea of modern vision language models with retrieval 🤝
ColPali marries the idea of modern vision language models with retrieval 🤝
new_line
new_line
text
The authors apply contrastive fine-tuning to SigLIP on documents, and pool the outputs (they call it BiSigLip). Then they feed the patch embedding outputs to PaliGemma and create BiPali 🖇️
The authors apply contrastive fine-tuning to SigLIP on documents, and pool the outputs (they call it BiSigLip). Then they feed the patch embedding outputs to PaliGemma and create BiPali 🖇️
new_line
text
BiPali natively supports image patch embeddings to an LLM, which enables leveraging the ColBERT-like late interaction computations between text tokens and image patches (hence the name ColPali!) 🤩
BiPali natively supports image patch embeddings to an LLM, which enables leveraging the ColBERT-like late interaction computations between text tokens and image patches (hence the name ColPali!) 🤩
new_line
new_line
text
The authors created the ViDoRe benchmark by collecting PDF documents and generate queries from Claude-3 Sonnet. 
The authors created the ViDoRe benchmark by collecting PDF documents and generate queries from Claude-3 Sonnet. 
new_line
text
ColPali seems to be the most performant model on ViDoRe. Not only this, but is way faster than traditional PDF parsers too!
ColPali seems to be the most performant model on ViDoRe. Not only this, but is way faster than traditional PDF parsers too!
Forget any document retrievers, use ColPali 💥💥

Document retrieval is done through OCR + layout detection, but you are losing a lot of information in between, stop doing that! 🤓

ColPali uses a vision language model, which is better in doc understanding 📑 
ColPali: https://huggingface.co/vidore/colpali (mit license!)
Blog post: https://huggingface.co/blog/manu/colpali
The authors also released a new benchmark for document retrieval: 
ViDoRe Benchmark: https://huggingface.co/collections/vidore/vidore-benchmark-667173f98e70a1c0fa4db00d
ViDoRe Leaderboard: https://huggingface.co/spaces/vidore/vidore-leaderboard

ColPali marries the idea of modern vision language models with retrieval 🤝

The authors apply contrastive fine-tuning to SigLIP on documents, and pool the outputs (they call it BiSigLip). Then they feed the patch embedding outputs to PaliGemma and create BiPali 🖇️
BiPali natively supports image patch embeddings to an LLM, which enables leveraging the ColBERT-like late interaction computations between text tokens and image patches (hence the name ColPali!) 🤩

The authors created the ViDoRe benchmark by collecting PDF documents and generate queries from Claude-3 Sonnet. 
ColPali seems to be the most performant model on ViDoRe. Not only this, but is way faster than traditional PDF parsers too!
🚀
ucsahin louisbrulenaudet Jebadiah diasbalmash osanseviero Zhofang
🤗
Cuiunbo
👍
mohammedbriman