text
Chameleon 🦎 by Meta is now available in Hugging Face transformers 😍
Chameleon 🦎 by Meta is now available in Hugging Face transformers 😍
new_line
text
A vision language model that comes in 7B and 34B sizes 🤩
A vision language model that comes in 7B and 34B sizes 🤩
new_line
text
But what makes this model so special?  
But what makes this model so special?  
new_line
new_line
text
Demo: 
Demo: 
resource
https://huggingface.co/spaces/merve/chameleon-7b
space
merve/chameleon-7b
https://huggingface.co/spaces/merve/chameleon-7b
new_line
text
Models: 
Models: 
resource
https://huggingface.co/collections/facebook/chameleon-668da9663f80d483b4c61f58
collection
facebook/chameleon-668da9663f80d483b4c61f58
https://huggingface.co/collections/facebook/chameleon-668da9663f80d483b4c61f58
new_line
new_line
text
keep reading ⥥
keep reading ⥥
new_line
new_line
text
Chameleon is a unique model: it attempts to scale early fusion 🤨
Chameleon is a unique model: it attempts to scale early fusion 🤨
new_line
text
But what is early fusion?
But what is early fusion?
new_line
text
Modern vision language models use a vision encoder with a projection layer to project image embeddings so it can be promptable to text decoder (LLM)
Modern vision language models use a vision encoder with a projection layer to project image embeddings so it can be promptable to text decoder (LLM)
new_line
new_line
text
Early fusion on the other hand attempts to fuse all features together (image patches and text) by using an image tokenizer and all tokens are projected into a shared space, which enables seamless generation 😏 
Early fusion on the other hand attempts to fuse all features together (image patches and text) by using an image tokenizer and all tokens are projected into a shared space, which enables seamless generation 😏 
new_line
new_line
text
Authors have also introduced different architectural improvements (QK norm and revise placement of layer norms) for scalable and stable training and they were able to increase the token count (5x tokens compared to Llama 3 which is a must with early-fusion IMO)
Authors have also introduced different architectural improvements (QK norm and revise placement of layer norms) for scalable and stable training and they were able to increase the token count (5x tokens compared to Llama 3 which is a must with early-fusion IMO)
new_line
new_line
text
This model is an any-to-any model thanks to early fusion: it can take image and text input and output image and text, but image generation are disabled to prevent malicious use. 
This model is an any-to-any model thanks to early fusion: it can take image and text input and output image and text, but image generation are disabled to prevent malicious use. 
new_line
new_line
text
One can also do text-only prompting, authors noted the model catches up with larger LLMs (like Mixtral 8x7B or larger Llama-2 70B) and also image-pair prompting with larger VLMs like IDEFICS2-80B (see paper for the benchmarks 
One can also do text-only prompting, authors noted the model catches up with larger LLMs (like Mixtral 8x7B or larger Llama-2 70B) and also image-pair prompting with larger VLMs like IDEFICS2-80B (see paper for the benchmarks 
resource
https://huggingface.co/papers/2405.09818
paper
2405.09818
https://huggingface.co/papers/2405.09818
Chameleon: Mixed-Modal Early-Fusion Foundation Models (2405.09818)
text
)
)
new_line
text
Thanks for reading!
Thanks for reading!
Chameleon 🦎 by Meta is now available in Hugging Face transformers 😍
A vision language model that comes in 7B and 34B sizes 🤩
But what makes this model so special?  

Demo: https://huggingface.co/spaces/merve/chameleon-7b
Models: https://huggingface.co/collections/facebook/chameleon-668da9663f80d483b4c61f58

keep reading ⥥

Chameleon is a unique model: it attempts to scale early fusion 🤨
But what is early fusion?
Modern vision language models use a vision encoder with a projection layer to project image embeddings so it can be promptable to text decoder (LLM)

Early fusion on the other hand attempts to fuse all features together (image patches and text) by using an image tokenizer and all tokens are projected into a shared space, which enables seamless generation 😏 

Authors have also introduced different architectural improvements (QK norm and revise placement of layer norms) for scalable and stable training and they were able to increase the token count (5x tokens compared to Llama 3 which is a must with early-fusion IMO)

This model is an any-to-any model thanks to early fusion: it can take image and text input and output image and text, but image generation are disabled to prevent malicious use. 

One can also do text-only prompting, authors noted the model catches up with larger LLMs (like Mixtral 8x7B or larger Llama-2 70B) and also image-pair prompting with larger VLMs like IDEFICS2-80B (see paper for the benchmarks https://huggingface.co/papers/2405.09818)
Thanks for reading!
🔥
sumandas atahanuz Nymbo ucsahin GPT007 fdaudens osanseviero alkzar90 dblasko Svngoku gokaygokay mikestaub abdulbasit-nubytes
🤗
alanonbing
https://cdn-avatars.huggingface.co/v1/production/uploads/1648113222875-6141a88b3a0ec78603c9e784.png
Merve Noyan
merve
user
/avatars/cfc399a521b05df08b799595b0390d13.svg
Prasanna Iyer
prasiyer
user
https://cdn-avatars.huggingface.co/v1/production/uploads/65d883893a52cd9bcd8ab7cf/tRsCJlHNZo1D02kBTmfy9.jpeg
leroy Samuel Dyer
LeroyDyer
user
https://cdn-avatars.huggingface.co/v1/production/uploads/667e763c6f5dee59721f016d/V2e6Z0BjzXE3t_obD3JXS.jpeg
Anastasia
Ana111op
user