text
Emu3: Next-token prediction conquers multimodal tasks 🔥
Emu3: Next-token prediction conquers multimodal tasks 🔥
new_line
new_line
text
This is the most important research in months: we’re now very close to having a single architecture to handle all modalities. The folks at Beijing Academy of Artificial Intelligence (BAAI) just released Emu3, a single model that handles text, images, and videos all at once.
This is the most important research in months: we’re now very close to having a single architecture to handle all modalities. The folks at Beijing Academy of Artificial Intelligence (BAAI) just released Emu3, a single model that handles text, images, and videos all at once.
new_line
new_line
text
𝗪𝗵𝗮𝘁'𝘀 𝘁𝗵𝗲 𝗯𝗶𝗴 𝗱𝗲𝗮𝗹?
𝗪𝗵𝗮𝘁'𝘀 𝘁𝗵𝗲 𝗯𝗶𝗴 𝗱𝗲𝗮𝗹?
new_line
text
🌟 Emu3 is the first model to truly unify all these different types of data (text, images, video) using just one simple trick: predicting the next token.
🌟 Emu3 is the first model to truly unify all these different types of data (text, images, video) using just one simple trick: predicting the next token.
new_line
text
And it’s only 8B, but really strong:
And it’s only 8B, but really strong:
new_line
text
🖼️ For image generation, it's matching the best specialized models out there, like SDXL.
🖼️ For image generation, it's matching the best specialized models out there, like SDXL.
new_line
text
👁️ In vision tasks, it's outperforming top models like LLaVA-1.6-7B, which is a big deal for a model that wasn't specifically designed for this.
👁️ In vision tasks, it's outperforming top models like LLaVA-1.6-7B, which is a big deal for a model that wasn't specifically designed for this.
new_line
text
🎬 It's the first to nail video generation without using complicated diffusion techniques.
🎬 It's the first to nail video generation without using complicated diffusion techniques.
new_line
new_line
text
𝗛𝗼𝘄 𝗱𝗼𝗲𝘀 𝗶𝘁 𝘄𝗼𝗿𝗸?
𝗛𝗼𝘄 𝗱𝗼𝗲𝘀 𝗶𝘁 𝘄𝗼𝗿𝗸?
new_line
text
🧩 Emu3 uses a special tokenizer (SBER-MoVQGAN) to turn images and video clips into sequences of 4,096 tokens.
🧩 Emu3 uses a special tokenizer (SBER-MoVQGAN) to turn images and video clips into sequences of 4,096 tokens.
new_line
text
🔗 Then, it treats everything - text, images, and videos - as one long series of tokens to predict.
🔗 Then, it treats everything - text, images, and videos - as one long series of tokens to predict.
new_line
text
🔮 During training, it just tries to guess the next token, whether that's a word, part of an image, or a video frame.
🔮 During training, it just tries to guess the next token, whether that's a word, part of an image, or a video frame.
new_line
new_line
text
𝗖𝗮𝘃𝗲𝗮𝘁𝘀 𝗼𝗻 𝘁𝗵𝗲 𝗿𝗲𝘀𝘂𝗹𝘁𝘀:
𝗖𝗮𝘃𝗲𝗮𝘁𝘀 𝗼𝗻 𝘁𝗵𝗲 𝗿𝗲𝘀𝘂𝗹𝘁𝘀:
new_line
text
👉 In image generation, Emu3 beats SDXL, but it’s also much bigger (8B vs 3.5B). It would be more difficult to beat the real diffusion GOAT FLUX-dev.
👉 In image generation, Emu3 beats SDXL, but it’s also much bigger (8B vs 3.5B). It would be more difficult to beat the real diffusion GOAT FLUX-dev.
new_line
text
👉 In vision, authors also don’t show a comparison against all the current SOTA models like Qwen-VL or Pixtral.
👉 In vision, authors also don’t show a comparison against all the current SOTA models like Qwen-VL or Pixtral.
new_line
new_line
text
This approach is exciting because it's simple (next token prediction) and scalable(handles all sorts of data)!
This approach is exciting because it's simple (next token prediction) and scalable(handles all sorts of data)!
new_line
new_line
text
Read the paper 👉 
Read the paper 👉 
resource
https://huggingface.co/papers/2409.18869
paper
2409.18869
https://huggingface.co/papers/2409.18869
Emu3: Next-Token Prediction is All You Need (2409.18869)
Emu3: Next-token prediction conquers multimodal tasks 🔥

This is the most important research in months: we’re now very close to having a single architecture to handle all modalities. The folks at Beijing Academy of Artificial Intelligence (BAAI) just released Emu3, a single model that handles text, images, and videos all at once.

𝗪𝗵𝗮𝘁'𝘀 𝘁𝗵𝗲 𝗯𝗶𝗴 𝗱𝗲𝗮𝗹?
🌟 Emu3 is the first model to truly unify all these different types of data (text, images, video) using just one simple trick: predicting the next token.
And it’s only 8B, but really strong:
🖼️ For image generation, it's matching the best specialized models out there, like SDXL.
👁️ In vision tasks, it's outperforming top models like LLaVA-1.6-7B, which is a big deal for a model that wasn't specifically designed for this.
🎬 It's the first to nail video generation without using complicated diffusion techniques.

𝗛𝗼𝘄 𝗱𝗼𝗲𝘀 𝗶𝘁 𝘄𝗼𝗿𝗸?
🧩 Emu3 uses a special tokenizer (SBER-MoVQGAN) to turn images and video clips into sequences of 4,096 tokens.
🔗 Then, it treats everything - text, images, and videos - as one long series of tokens to predict.
🔮 During training, it just tries to guess the next token, whether that's a word, part of an image, or a video frame.

𝗖𝗮𝘃𝗲𝗮𝘁𝘀 𝗼𝗻 𝘁𝗵𝗲 𝗿𝗲𝘀𝘂𝗹𝘁𝘀:
👉 In image generation, Emu3 beats SDXL, but it’s also much bigger (8B vs 3.5B). It would be more difficult to beat the real diffusion GOAT FLUX-dev.
👉 In vision, authors also don’t show a comparison against all the current SOTA models like Qwen-VL or Pixtral.

This approach is exciting because it's simple (next token prediction) and scalable(handles all sorts of data)!

Read the paper 👉 https://huggingface.co/papers/2409.18869
🔥
umair894 acamilogg88 lamhieu Donutanti
👀
John6666