text
Genie is a new method from Google DeepMind that generates interactive, action-controllable virtual worlds from unlabelled internet videos using. 
Genie is a new method from Google DeepMind that generates interactive, action-controllable virtual worlds from unlabelled internet videos using. 
new_line
new_line
text
Keypoints:
Keypoints:
new_line
text
* Genie leverages a spatiotemporal video tokenizer, an autoregressive dynamics model, and a latent action model to generate controllable video environments.
* Genie leverages a spatiotemporal video tokenizer, an autoregressive dynamics model, and a latent action model to generate controllable video environments.
new_line
text
* The model is trained on video data alone, without requiring action labels, using unsupervised learning to infer latent actions between frames.
* The model is trained on video data alone, without requiring action labels, using unsupervised learning to infer latent actions between frames.
new_line
text
* The method restricts the size of the action vocabulary to 8 to ensure that the number of possible latent actions remains small.
* The method restricts the size of the action vocabulary to 8 to ensure that the number of possible latent actions remains small.
new_line
text
* The dataset used for training is generated by filtering publicly available internet videos with specific criteria related to 2D platformer games for a total of 6.8M videos used for training.
* The dataset used for training is generated by filtering publicly available internet videos with specific criteria related to 2D platformer games for a total of 6.8M videos used for training.
new_line
new_line
text
Paper: 
Paper: 
resource
https://huggingface.co/papers/2402.15391
paper
2402.15391
https://huggingface.co/papers/2402.15391
Genie: Generative Interactive Environments (2402.15391)
new_line
text
Project page: 
Project page: 
link
https://sites.google.com/view/genie-2024/
https://sites.google.com/view/genie-2024/
new_line
text
More detailed overview in my blog: 
More detailed overview in my blog: 
link
https://huggingface.co/blog/vladbogo/genie-generative-interactive-environments
https://huggingface.co/blog/vladbogo/genie-generative-interactive-environments
new_line
new_line
text
Congrats to the authors for their work!
Congrats to the authors for their work!
Genie is a new method from Google DeepMind that generates interactive, action-controllable virtual worlds from unlabelled internet videos using. 

Keypoints:
* Genie leverages a spatiotemporal video tokenizer, an autoregressive dynamics model, and a latent action model to generate controllable video environments.
* The model is trained on video data alone, without requiring action labels, using unsupervised learning to infer latent actions between frames.
* The method restricts the size of the action vocabulary to 8 to ensure that the number of possible latent actions remains small.
* The dataset used for training is generated by filtering publicly available internet videos with specific criteria related to 2D platformer games for a total of 6.8M videos used for training.

Paper: https://huggingface.co/papers/2402.15391
Project page: https://sites.google.com/view/genie-2024/
More detailed overview in my blog: https://huggingface.co/blog/vladbogo/genie-generative-interactive-environments

Congrats to the authors for their work!
👍
jsunster merve osanseviero kramp terpjwu1 programORdie2 mvaloatto danielhanchen Concor 2dts notune thomwolf julien-c
🤯
jeffboudier thomwolf
https://cdn-avatars.huggingface.co/v1/production/uploads/1648113222875-6141a88b3a0ec78603c9e784.png
Merve Noyan
merve
user
https://cdn-avatars.huggingface.co/v1/production/uploads/6032802e1f993496bc14d9e3/w6hr-DEQot4VVkoyRIBiy.png
Omar Sanseviero
osanseviero
user
https://cdn-avatars.huggingface.co/v1/production/uploads/657217faabb25ed8aedd5e48/UUHAXeGtOnQBXFD3nYtf2.jpeg
Vlad Bogolin
vladbogo
user