text
Thrilled to share some of our recent work in the field of Multimodal Large Language Models (MLLMs). 
Thrilled to share some of our recent work in the field of Multimodal Large Language Models (MLLMs). 
new_line
new_line
text
1️⃣ A Survey on Multimodal Reasoning 📚
1️⃣ A Survey on Multimodal Reasoning 📚
new_line
text
Are you curious about the reasoning abilities of MLLMs? In our latest survey, we delve into the world of multimodal reasoning. We comprehensively review existing evaluation protocols, categorize the frontiers of MLLMs, explore recent trends in their applications for reasoning-intensive tasks, and discuss current practices and future directions. For an in-depth exploration, check out our paper: 
Are you curious about the reasoning abilities of MLLMs? In our latest survey, we delve into the world of multimodal reasoning. We comprehensively review existing evaluation protocols, categorize the frontiers of MLLMs, explore recent trends in their applications for reasoning-intensive tasks, and discuss current practices and future directions. For an in-depth exploration, check out our paper: 
resource
https://huggingface.co/papers/2401.06805
paper
2401.06805
https://huggingface.co/papers/2401.06805
Exploring the Reasoning Abilities of Multimodal Large Language Models
  (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning (2401.06805)
new_line
new_line
text
2️⃣ Advancing Flamingo with InfiMM 🔥
2️⃣ Advancing Flamingo with InfiMM 🔥
new_line
text
Building upon the foundation of Flamingo, we introduce the InfiMM model series. InfiMM is a reproduction of Flamingo, enhanced with stronger Large Language Models (LLMs) such as LLaMA2-13B, Vicuna-13B, and Zephyr7B. We've meticulously filtered pre-training data and fine-tuned instructions, resulting in superior performance on recent benchmarks like MMMU, InfiMM-Eval, MM-Vet, and more. Explore the power of InfiMM on Huggingface: 
Building upon the foundation of Flamingo, we introduce the InfiMM model series. InfiMM is a reproduction of Flamingo, enhanced with stronger Large Language Models (LLMs) such as LLaMA2-13B, Vicuna-13B, and Zephyr7B. We've meticulously filtered pre-training data and fine-tuned instructions, resulting in superior performance on recent benchmarks like MMMU, InfiMM-Eval, MM-Vet, and more. Explore the power of InfiMM on Huggingface: 
resource
https://huggingface.co/Infi-MM/infimm-zephyr
model
Infi-MM/infimm-zephyr
https://huggingface.co/Infi-MM/infimm-zephyr
new_line
new_line
text
3️⃣ Exploring Multimodal Instruction Fine-tuning 🖼️
3️⃣ Exploring Multimodal Instruction Fine-tuning 🖼️
new_line
text
Visual Instruction Fine-tuning (IFT) is crucial for aligning MLLMs' output with user intentions. Our research identified challenges with models trained on the LLaVA-mix-665k dataset, particularly in multi-round dialog settings. To address this, we've created a new IFT dataset with high-quality, diverse instruction annotations and images sourced exclusively from the COCO dataset. Our experiments demonstrate that when fine-tuned with this dataset, MLLMs excel in open-ended evaluation benchmarks for both single-round and multi-round dialog settings. Dive into the details in our paper: 
Visual Instruction Fine-tuning (IFT) is crucial for aligning MLLMs' output with user intentions. Our research identified challenges with models trained on the LLaVA-mix-665k dataset, particularly in multi-round dialog settings. To address this, we've created a new IFT dataset with high-quality, diverse instruction annotations and images sourced exclusively from the COCO dataset. Our experiments demonstrate that when fine-tuned with this dataset, MLLMs excel in open-ended evaluation benchmarks for both single-round and multi-round dialog settings. Dive into the details in our paper: 
resource
https://huggingface.co/papers/2401.08968
paper
2401.08968
https://huggingface.co/papers/2401.08968
COCO is "ALL'' You Need for Visual Instruction Fine-tuning (2401.08968)
new_line
new_line
text
Stay tuned for more exciting developments. 
Stay tuned for more exciting developments. 
new_line
text
Special thanks to all our collaborators: 
Special thanks to all our collaborators: 
mention
@Ye27
Ye27
text
mention
@wwyssh
wwyssh
text
mention
@Yongfei
Yongfei
text
mention
@Yi-Qi638
Yi-Qi638
text
mention
@xudonglin
xudonglin
text
mention
@KhalilMrini
KhalilMrini
text
mention
@lllliuhhhhggg
lllliuhhhhggg
text
mention
@Borise
Borise
text
mention
@Hongxia
Hongxia
text
Thrilled to share some of our recent work in the field of Multimodal Large Language Models (MLLMs). 

1️⃣ A Survey on Multimodal Reasoning 📚
Are you curious about the reasoning abilities of MLLMs? In our latest survey, we delve into the world of multimodal reasoning. We comprehensively review existing evaluation protocols, categorize the frontiers of MLLMs, explore recent trends in their applications for reasoning-intensive tasks, and discuss current practices and future directions. For an in-depth exploration, check out our paper: https://huggingface.co/papers/2401.06805

2️⃣ Advancing Flamingo with InfiMM 🔥
Building upon the foundation of Flamingo, we introduce the InfiMM model series. InfiMM is a reproduction of Flamingo, enhanced with stronger Large Language Models (LLMs) such as LLaMA2-13B, Vicuna-13B, and Zephyr7B. We've meticulously filtered pre-training data and fine-tuned instructions, resulting in superior performance on recent benchmarks like MMMU, InfiMM-Eval, MM-Vet, and more. Explore the power of InfiMM on Huggingface: https://huggingface.co/Infi-MM/infimm-zephyr

3️⃣ Exploring Multimodal Instruction Fine-tuning 🖼️
Visual Instruction Fine-tuning (IFT) is crucial for aligning MLLMs' output with user intentions. Our research identified challenges with models trained on the LLaVA-mix-665k dataset, particularly in multi-round dialog settings. To address this, we've created a new IFT dataset with high-quality, diverse instruction annotations and images sourced exclusively from the COCO dataset. Our experiments demonstrate that when fine-tuned with this dataset, MLLMs excel in open-ended evaluation benchmarks for both single-round and multi-round dialog settings. Dive into the details in our paper: https://huggingface.co/papers/2401.08968

Stay tuned for more exciting developments. 
Special thanks to all our collaborators: @Ye27 @wwyssh @Yongfei @Yi-Qi638 @xudonglin @KhalilMrini @lllliuhhhhggg @Borise @Hongxia 
🤗
Ye27 wwyssh samusenps lllliuhhhhggg osanseviero victor xudonglin xiaotianhan Tonic yarth69 clem lunarflu
👍
abacaj Tonic clem lunarflu xiaotianhan Yi-Qi638
❤️
XaevrM lunarflu xiaotianhan