text
Do we fully leverage ViT encoders in vision language models? 
Do we fully leverage ViT encoders in vision language models? 
new_line
new_line
text
A new paper (by 
A new paper (by 
mention
@HuanjinYao
HuanjinYao
text
 et al) built a dense connector that does it better! 
 et al) built a dense connector that does it better! 
resource
https://huggingface.co/spaces/HuanjinYao/DenseConnector-v1.5-8B
space
HuanjinYao/DenseConnector-v1.5-8B
https://huggingface.co/spaces/HuanjinYao/DenseConnector-v1.5-8B
new_line
resource
https://huggingface.co/collections/HuanjinYao/denseconnector-66500e173fc8c9f05dc98dea
collection
HuanjinYao/denseconnector-66500e173fc8c9f05dc98dea
https://huggingface.co/collections/HuanjinYao/denseconnector-66500e173fc8c9f05dc98dea
text
new_line
new_line
text
VLMs consist of an image encoder block, a projection layer that projects image embeddings to text embedding space and then a text decoder sequentially connected 📖
VLMs consist of an image encoder block, a projection layer that projects image embeddings to text embedding space and then a text decoder sequentially connected 📖
new_line
text
This paper explores using intermediate states of image encoder and not a single output 🤩
This paper explores using intermediate states of image encoder and not a single output 🤩
new_line
text
The authors explore three different ways of instantiating dense connector: sparse token integration, sparse channel integration and dense channel integration. (see paper on how they do it 
The authors explore three different ways of instantiating dense connector: sparse token integration, sparse channel integration and dense channel integration. (see paper on how they do it 
resource
https://huggingface.co/papers/2405.13800
paper
2405.13800
https://huggingface.co/papers/2405.13800
Dense Connector for MLLMs (2405.13800)
text
)
)
new_line
new_line
text
They explore all three of them integrated to LLaVA 1.5 and found out each of the new models are superior to the original LLaVA 1.5 🥹 I tried the model and it seems to work very well. As part of the release, the authors have released various ckpts based on different decoders (Vicuna 7/13B and Llama 3-8B) that you can find in the collection 🤗 
They explore all three of them integrated to LLaVA 1.5 and found out each of the new models are superior to the original LLaVA 1.5 🥹 I tried the model and it seems to work very well. As part of the release, the authors have released various ckpts based on different decoders (Vicuna 7/13B and Llama 3-8B) that you can find in the collection 🤗 
new_line
new_line
Do we fully leverage ViT encoders in vision language models? 

A new paper (by @HuanjinYao et al) built a dense connector that does it better! https://huggingface.co/spaces/HuanjinYao/DenseConnector-v1.5-8B
https://huggingface.co/collections/HuanjinYao/denseconnector-66500e173fc8c9f05dc98dea 

VLMs consist of an image encoder block, a projection layer that projects image embeddings to text embedding space and then a text decoder sequentially connected 📖
This paper explores using intermediate states of image encoder and not a single output 🤩
The authors explore three different ways of instantiating dense connector: sparse token integration, sparse channel integration and dense channel integration. (see paper on how they do it https://huggingface.co/papers/2405.13800)

They explore all three of them integrated to LLaVA 1.5 and found out each of the new models are superior to the original LLaVA 1.5 🥹 I tried the model and it seems to work very well. As part of the release, the authors have released various ckpts based on different decoders (Vicuna 7/13B and Llama 3-8B) that you can find in the collection 🤗 


👀
lunarflu HuanjinYao Norod78 osanseviero Iheb404notfound Abhinay45 arafat55 luancloud
🚀
ceofast