text
Good folks at 
Good folks at 
mention
@nvidia
nvidia
text
 have just released NVLM 1.0, a family of frontier-class multimodal large language models that achieve state-of-the-art results across vision-language tasks.
 have just released NVLM 1.0, a family of frontier-class multimodal large language models that achieve state-of-the-art results across vision-language tasks.
new_line
new_line
text
Here is how they did it:
Here is how they did it:
new_line
new_line
text
1. Model Architecture Design:
1. Model Architecture Design:
new_line
text
- Developed three model architectures:
- Developed three model architectures:
new_line
text
a) NVLM-D: Decoder-only architecture
a) NVLM-D: Decoder-only architecture
new_line
text
b) NVLM-X: Cross-attention-based architecture
b) NVLM-X: Cross-attention-based architecture
new_line
text
c) NVLM-H: Novel hybrid architecture
c) NVLM-H: Novel hybrid architecture
new_line
new_line
text
2. Vision Encoder:
2. Vision Encoder:
new_line
text
- Used InternViT-6B-448px-V1-5 as the vision encoder
- Used InternViT-6B-448px-V1-5 as the vision encoder
new_line
text
- Implemented dynamic high-resolution (DHR) input handling
- Implemented dynamic high-resolution (DHR) input handling
new_line
new_line
text
3. Language Model:
3. Language Model:
new_line
text
- Used Qwen2-72B-Instruct as the base LLM
- Used Qwen2-72B-Instruct as the base LLM
new_line
new_line
text
4. Training Data Curation:
4. Training Data Curation:
new_line
text
- Carefully curated high-quality pretraining and supervised fine-tuning datasets
- Carefully curated high-quality pretraining and supervised fine-tuning datasets
new_line
text
- Included diverse task-oriented datasets for various capabilities
- Included diverse task-oriented datasets for various capabilities
new_line
new_line
text
5. Pretraining:
5. Pretraining:
new_line
text
- Froze LLM and vision encoder
- Froze LLM and vision encoder
new_line
text
- Trained only modality-alignment modules (e.g., MLP projector, cross-attention layers)
- Trained only modality-alignment modules (e.g., MLP projector, cross-attention layers)
new_line
text
- Used a large batch size of 2048
- Used a large batch size of 2048
new_line
new_line
text
6. Supervised Fine-Tuning (SFT):
6. Supervised Fine-Tuning (SFT):
new_line
text
- Unfroze LLM while keeping the vision encoder frozen
- Unfroze LLM while keeping the vision encoder frozen
new_line
text
- Trained on multimodal SFT datasets and high-quality text-only SFT data
- Trained on multimodal SFT datasets and high-quality text-only SFT data
new_line
text
- Implemented 1-D tile tagging for dynamic high-resolution inputs
- Implemented 1-D tile tagging for dynamic high-resolution inputs
new_line
new_line
text
7. Evaluation:
7. Evaluation:
new_line
text
- Evaluated on multiple vision-language benchmarks
- Evaluated on multiple vision-language benchmarks
new_line
text
- Compared performance to leading proprietary and open-source models
- Compared performance to leading proprietary and open-source models
new_line
new_line
text
8. Optimization:
8. Optimization:
new_line
text
- Iterated on model designs and training approaches
- Iterated on model designs and training approaches
new_line
text
- Used smaller 34B models for faster experimentation before scaling to 72B
- Used smaller 34B models for faster experimentation before scaling to 72B
new_line
new_line
text
9. Now comes the best part...Open-Sourcing:
9. Now comes the best part...Open-Sourcing:
new_line
text
- Released model weights and full technical details to the research community
- Released model weights and full technical details to the research community
new_line
new_line
text
The paper provides fascinating insights into architecture design, training data curation, and achieving production-grade multimodality. A must-read for anyone working on multimodal AI!
The paper provides fascinating insights into architecture design, training data curation, and achieving production-grade multimodality. A must-read for anyone working on multimodal AI!
Good folks at @nvidia have just released NVLM 1.0, a family of frontier-class multimodal large language models that achieve state-of-the-art results across vision-language tasks.

Here is how they did it:

1. Model Architecture Design:
- Developed three model architectures:
a) NVLM-D: Decoder-only architecture
b) NVLM-X: Cross-attention-based architecture
c) NVLM-H: Novel hybrid architecture

2. Vision Encoder:
- Used InternViT-6B-448px-V1-5 as the vision encoder
- Implemented dynamic high-resolution (DHR) input handling

3. Language Model:
- Used Qwen2-72B-Instruct as the base LLM

4. Training Data Curation:
- Carefully curated high-quality pretraining and supervised fine-tuning datasets
- Included diverse task-oriented datasets for various capabilities

5. Pretraining:
- Froze LLM and vision encoder
- Trained only modality-alignment modules (e.g., MLP projector, cross-attention layers)
- Used a large batch size of 2048

6. Supervised Fine-Tuning (SFT):
- Unfroze LLM while keeping the vision encoder frozen
- Trained on multimodal SFT datasets and high-quality text-only SFT data
- Implemented 1-D tile tagging for dynamic high-resolution inputs

7. Evaluation:
- Evaluated on multiple vision-language benchmarks
- Compared performance to leading proprietary and open-source models

8. Optimization:
- Iterated on model designs and training approaches
- Used smaller 34B models for faster experimentation before scaling to 72B

9. Now comes the best part...Open-Sourcing:
- Released model weights and full technical details to the research community

The paper provides fascinating insights into architecture design, training data curation, and achieving production-grade multimodality. A must-read for anyone working on multimodal AI!
👍
Thunderbird2410 TacImpulse osanseviero AINovice2005
👀
John6666 louisbrulenaudet