text
We release Idefics2-8B, a foundation vision language model with SOTA results for its size on many benchmarks.
We release Idefics2-8B, a foundation vision language model with SOTA results for its size on many benchmarks.
new_line
new_line
text
For Idefics2, we adopted a simple architecture:
For Idefics2, we adopted a simple architecture:
new_line
text
-Images are fed to a vision encoder, then to a modality projection to match the input dimension of the LLM, and finally to a perceiver resampler for efficient pooling.
-Images are fed to a vision encoder, then to a modality projection to match the input dimension of the LLM, and finally to a perceiver resampler for efficient pooling.
new_line
text
-Interleaved image-text data are then passed to the LLM.
-Interleaved image-text data are then passed to the LLM.
new_line
new_line
text
During the pre-training:
During the pre-training:
new_line
text
-The modality projection and perceiver resampler weights are newly initialized.
-The modality projection and perceiver resampler weights are newly initialized.
new_line
text
-We start with pre-trained models for the vision encoder and the LLM, and continue the training with LoRA.
-We start with pre-trained models for the vision encoder and the LLM, and continue the training with LoRA.
new_line
text
-In total, we see 1.5T images!
-In total, we see 1.5T images!
new_line
new_line
text
We pre-train on 3 types of data, all publicly available:
We pre-train on 3 types of data, all publicly available:
new_line
text
-Interleaved image-text documents: our dataset OBELICS 
-Interleaved image-text documents: our dataset OBELICS 
resource
https://huggingface.co/datasets/HuggingFaceM4/OBELICS
dataset
HuggingFaceM4/OBELICS
https://huggingface.co/datasets/HuggingFaceM4/OBELICS
new_line
text
-Image caption pairs: only synthetic captions!
-Image caption pairs: only synthetic captions!
new_line
text
-PDF documents: IDL and PDFA
-PDF documents: IDL and PDFA
new_line
new_line
text
We kept the aspect ratio of the images with the Patch n' Pack strategy, with a resolution of up to 980x980.
We kept the aspect ratio of the images with the Patch n' Pack strategy, with a resolution of up to 980x980.
new_line
text
At inference, it's also more efficient for lower-resolution images.
At inference, it's also more efficient for lower-resolution images.
new_line
new_line
text
For the SFT, we build The Cauldron, a collection of 50 high-quality datasets in the user/assistant format.
For the SFT, we build The Cauldron, a collection of 50 high-quality datasets in the user/assistant format.
new_line
text
It is a ready-to-use dataset for the fine-tuning of any VLM.
It is a ready-to-use dataset for the fine-tuning of any VLM.
new_line
resource
https://huggingface.co/datasets/HuggingFaceM4/the_cauldron
dataset
HuggingFaceM4/the_cauldron
https://huggingface.co/datasets/HuggingFaceM4/the_cauldron
new_line
new_line
text
Most current models, like LLaVA-NeXT, encode images with an excessive number of tokens, like 2880.
Most current models, like LLaVA-NeXT, encode images with an excessive number of tokens, like 2880.
new_line
text
Instead, we put a focus on being efficient at inference by training on a mix of images encoded with 64 tokens, and 320 tokens.
Instead, we put a focus on being efficient at inference by training on a mix of images encoded with 64 tokens, and 320 tokens.
new_line
text
The result is that we perform favorably compared to the best models in our size class, while being efficient at inference.
The result is that we perform favorably compared to the best models in our size class, while being efficient at inference.
We release Idefics2-8B, a foundation vision language model with SOTA results for its size on many benchmarks.

For Idefics2, we adopted a simple architecture:
-Images are fed to a vision encoder, then to a modality projection to match the input dimension of the LLM, and finally to a perceiver resampler for efficient pooling.
-Interleaved image-text data are then passed to the LLM.

During the pre-training:
-The modality projection and perceiver resampler weights are newly initialized.
-We start with pre-trained models for the vision encoder and the LLM, and continue the training with LoRA.
-In total, we see 1.5T images!

We pre-train on 3 types of data, all publicly available:
-Interleaved image-text documents: our dataset OBELICS https://huggingface.co/datasets/HuggingFaceM4/OBELICS
-Image caption pairs: only synthetic captions!
-PDF documents: IDL and PDFA

We kept the aspect ratio of the images with the Patch n' Pack strategy, with a resolution of up to 980x980.
At inference, it's also more efficient for lower-resolution images.

For the SFT, we build The Cauldron, a collection of 50 high-quality datasets in the user/assistant format.
It is a ready-to-use dataset for the fine-tuning of any VLM.
https://huggingface.co/datasets/HuggingFaceM4/the_cauldron

Most current models, like LLaVA-NeXT, encode images with an excessive number of tokens, like 2880.
Instead, we put a focus on being efficient at inference by training on a mix of images encoded with 64 tokens, and 320 tokens.
The result is that we perform favorably compared to the best models in our size class, while being efficient at inference.
🤯
VictorSanh victor danaaubakirova sbarman25 m-ric clem lunarflu Violette fdaudens sbrandeis
🚀
m-ric clem samusenps Sylvestre Molbap NePe lunarflu fynnkroeger Violette
🧠
VictorSanh m-ric clem lunarflu Violette
🤝
WenchuanYang clem samusenps lunarflu Violette
👍
Violette