text
🎮 𝗔 𝗻𝗲𝘂𝗿𝗮𝗹 𝗻𝗲𝘁𝘄𝗼𝗿𝗸 𝘀𝗶𝗺𝘂𝗹𝗮𝘁𝗲𝘀 𝗗𝗢𝗢𝗠: 𝗚𝗼𝗼𝗴𝗹𝗲 𝗿𝗲𝘀𝗲𝗮𝗿𝗰𝗵𝗲𝗿𝘀 𝗼𝗽𝗲𝗻 𝘁𝗵𝗲 𝘄𝗮𝘆 𝗳𝗼𝗿 𝗰𝗼𝗺𝗽𝗹𝗲𝘁𝗲𝗹𝘆-𝗔𝗜-𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗲𝗱 𝗴𝗮𝗺𝗲𝘀!
🎮 𝗔 𝗻𝗲𝘂𝗿𝗮𝗹 𝗻𝗲𝘁𝘄𝗼𝗿𝗸 𝘀𝗶𝗺𝘂𝗹𝗮𝘁𝗲𝘀 𝗗𝗢𝗢𝗠: 𝗚𝗼𝗼𝗴𝗹𝗲 𝗿𝗲𝘀𝗲𝗮𝗿𝗰𝗵𝗲𝗿𝘀 𝗼𝗽𝗲𝗻 𝘁𝗵𝗲 𝘄𝗮𝘆 𝗳𝗼𝗿 𝗰𝗼𝗺𝗽𝗹𝗲𝘁𝗲𝗹𝘆-𝗔𝗜-𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗲𝗱 𝗴𝗮𝗺𝗲𝘀!
new_line
new_line
text
Imagine if games were completely live-generated by an AI model : the NPCs and their dialogues, the storyline, and even the game environment. The player’s in-game actions would have a real, lasting impact on the game story.
Imagine if games were completely live-generated by an AI model : the NPCs and their dialogues, the storyline, and even the game environment. The player’s in-game actions would have a real, lasting impact on the game story.
new_line
new_line
text
In a very exciting paper, Google researchers just gave us the first credible glimpse of this future.
In a very exciting paper, Google researchers just gave us the first credible glimpse of this future.
new_line
new_line
text
➡️ They created GameNGen, the first neural model that can simulate a complex 3D game in real-time. They use it to simulate the classic game DOOM running at over 20 frames per second on a single TPU, with image quality comparable to lossy JPEG compression. And it feels just like the true game!
➡️ They created GameNGen, the first neural model that can simulate a complex 3D game in real-time. They use it to simulate the classic game DOOM running at over 20 frames per second on a single TPU, with image quality comparable to lossy JPEG compression. And it feels just like the true game!
new_line
new_line
text
Here's how they did it:
Here's how they did it:
new_line
text
1. They trained an RL agent to play DOOM and recorded its gameplay sessions.
1. They trained an RL agent to play DOOM and recorded its gameplay sessions.
new_line
text
2. They then used these recordings to train a diffusion model to predict the next frame, based on past frames and player actions.
2. They then used these recordings to train a diffusion model to predict the next frame, based on past frames and player actions.
new_line
text
3. During inference, they use only 4 denoising steps (instead of the usual dozens) to generate each frame quickly.
3. During inference, they use only 4 denoising steps (instead of the usual dozens) to generate each frame quickly.
new_line
new_line
text
𝗞𝗲𝘆 𝗶𝗻𝘀𝗶𝗴𝗵𝘁𝘀:
𝗞𝗲𝘆 𝗶𝗻𝘀𝗶𝗴𝗵𝘁𝘀:
new_line
text
🎮🤔 Human players can barely tell the difference between short clips (3 seconds) of the real game or the simulation
🎮🤔 Human players can barely tell the difference between short clips (3 seconds) of the real game or the simulation
new_line
text
🧠 The model maintains game state (health, ammo, etc.) over long periods despite having only 3 seconds of effective context length
🧠 The model maintains game state (health, ammo, etc.) over long periods despite having only 3 seconds of effective context length
new_line
text
🔄 They use "noise augmentation" during training to prevent quality degradation in long play sessions
🔄 They use "noise augmentation" during training to prevent quality degradation in long play sessions
new_line
text
🚀 The game runs on one TPU at 20 FPS with 4 denoising steps, or 50 FPS with model distillation (with some quality loss)
🚀 The game runs on one TPU at 20 FPS with 4 denoising steps, or 50 FPS with model distillation (with some quality loss)
new_line
new_line
text
The researchers did not open source the code, but I feel like we’ve just seen a part of the future being written!
The researchers did not open source the code, but I feel like we’ve just seen a part of the future being written!
new_line
new_line
text
Their paper (exploding the upvote counter) 👉 
Their paper (exploding the upvote counter) 👉 
resource
https://huggingface.co/papers/2408.14837
paper
2408.14837
https://huggingface.co/papers/2408.14837
Diffusion Models Are Real-Time Game Engines (2408.14837)
new_line
text
In a similar vein, play 
In a similar vein, play 
mention
@Jofthomas
Jofthomas
text
's 'Everchanging Quest' 🎮 
's 'Everchanging Quest' 🎮 
resource
https://huggingface.co/spaces/Jofthomas/Everchanging-Quest
space
Jofthomas/Everchanging-Quest
https://huggingface.co/spaces/Jofthomas/Everchanging-Quest
🎮 𝗔 𝗻𝗲𝘂𝗿𝗮𝗹 𝗻𝗲𝘁𝘄𝗼𝗿𝗸 𝘀𝗶𝗺𝘂𝗹𝗮𝘁𝗲𝘀 𝗗𝗢𝗢𝗠: 𝗚𝗼𝗼𝗴𝗹𝗲 𝗿𝗲𝘀𝗲𝗮𝗿𝗰𝗵𝗲𝗿𝘀 𝗼𝗽𝗲𝗻 𝘁𝗵𝗲 𝘄𝗮𝘆 𝗳𝗼𝗿 𝗰𝗼𝗺𝗽𝗹𝗲𝘁𝗲𝗹𝘆-𝗔𝗜-𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗲𝗱 𝗴𝗮𝗺𝗲𝘀!

Imagine if games were completely live-generated by an AI model : the NPCs and their dialogues, the storyline, and even the game environment. The player’s in-game actions would have a real, lasting impact on the game story.

In a very exciting paper, Google researchers just gave us the first credible glimpse of this future.

➡️ They created GameNGen, the first neural model that can simulate a complex 3D game in real-time. They use it to simulate the classic game DOOM running at over 20 frames per second on a single TPU, with image quality comparable to lossy JPEG compression. And it feels just like the true game!

Here's how they did it:
1. They trained an RL agent to play DOOM and recorded its gameplay sessions.
2. They then used these recordings to train a diffusion model to predict the next frame, based on past frames and player actions.
3. During inference, they use only 4 denoising steps (instead of the usual dozens) to generate each frame quickly.

𝗞𝗲𝘆 𝗶𝗻𝘀𝗶𝗴𝗵𝘁𝘀:
🎮🤔 Human players can barely tell the difference between short clips (3 seconds) of the real game or the simulation
🧠 The model maintains game state (health, ammo, etc.) over long periods despite having only 3 seconds of effective context length
🔄 They use "noise augmentation" during training to prevent quality degradation in long play sessions
🚀 The game runs on one TPU at 20 FPS with 4 denoising steps, or 50 FPS with model distillation (with some quality loss)

The researchers did not open source the code, but I feel like we’ve just seen a part of the future being written!

Their paper (exploding the upvote counter) 👉 https://huggingface.co/papers/2408.14837
In a similar vein, play @Jofthomas's 'Everchanging Quest' 🎮 https://huggingface.co/spaces/Jofthomas/Everchanging-Quest
🔥
alielfilali01 John6666 Jofthomas Joseph717171 prithivMLmods Hrushi bmorphism victor BoothyBoothyBoothy Pomni zhangsibo1129
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/xrJ30F3IA11cBe3BYCLMB.jpeg
dixydomangrimes
rixy1
user