text
Less than two days ago Kyutai Labs open sourced Moshi - an ~7.6B on-device Speech to Speech foundation model and Mimi - SoTA streaming speech codec! 🔥
Less than two days ago Kyutai Labs open sourced Moshi - an ~7.6B on-device Speech to Speech foundation model and Mimi - SoTA streaming speech codec! 🔥
new_line
new_line
text
The release includes:
The release includes:
new_line
new_line
text
1. Moshiko & Moshika - Moshi finetuned on synthetic data (CC-BY license) (
1. Moshiko & Moshika - Moshi finetuned on synthetic data (CC-BY license) (
resource
https://huggingface.co/collections/kyutai/moshi-v01-release-66eaeaf3302bef6bd9ad7acd
collection
kyutai/moshi-v01-release-66eaeaf3302bef6bd9ad7acd
https://huggingface.co/collections/kyutai/moshi-v01-release-66eaeaf3302bef6bd9ad7acd
text
)
)
new_line
text
2. Mimi - Streaiming Audio Codec, processes 24 kHz audio, down to a 12.5 Hz representation with a bandwidth of 1.1 kbps (CC-BY license) (
2. Mimi - Streaiming Audio Codec, processes 24 kHz audio, down to a 12.5 Hz representation with a bandwidth of 1.1 kbps (CC-BY license) (
resource
https://huggingface.co/kyutai/mimi
model
kyutai/mimi
https://huggingface.co/kyutai/mimi
text
)
)
new_line
text
3. Model checkpoints & Inference codebase written in Rust (Candle), PyTorch & MLX (Apache license) (
3. Model checkpoints & Inference codebase written in Rust (Candle), PyTorch & MLX (Apache license) (
link
https://github.com/kyutai-labs/moshi
https://github.com/kyutai-labs/moshi
text
)
)
new_line
new_line
text
How does Moshi work?
How does Moshi work?
new_line
new_line
text
1. Moshi processes two audio streams: one for itself and one for the user, with the user's stream coming from audio input and Moshi's stream generated by the model.
1. Moshi processes two audio streams: one for itself and one for the user, with the user's stream coming from audio input and Moshi's stream generated by the model.
new_line
new_line
text
2. Along with these audio streams, Moshi predicts text tokens for its speech, enhancing its generation quality.
2. Along with these audio streams, Moshi predicts text tokens for its speech, enhancing its generation quality.
new_line
new_line
text
3. The model uses a small Depth Transformer for codebook dependencies and a large 7B parameter Temporal Transformer for temporal dependencies.
3. The model uses a small Depth Transformer for codebook dependencies and a large 7B parameter Temporal Transformer for temporal dependencies.
new_line
new_line
text
4. The theoretical latency is 160ms, with a practical latency of around 200ms on an L4 GPU.
4. The theoretical latency is 160ms, with a practical latency of around 200ms on an L4 GPU.
new_line
new_line
text
Model size & inference:
Model size & inference:
new_line
new_line
text
Moshiko/ka are 7.69B param models
Moshiko/ka are 7.69B param models
new_line
new_line
text
bf16 ~16GB VRAM
bf16 ~16GB VRAM
new_line
text
8-bit ~8GB VRAM
8-bit ~8GB VRAM
new_line
text
4-bit ~4GB VRAM
4-bit ~4GB VRAM
new_line
new_line
text
You can run inference via Candle 🦀, PyTorch and MLX - based on your hardware.
You can run inference via Candle 🦀, PyTorch and MLX - based on your hardware.
new_line
new_line
text
The Kyutai team,  
The Kyutai team,  
mention
@adefossez
adefossez
text
mention
@lmz
lmz
text
  and team are cracked AF, they're bringing some serious firepower to the open source/ science AI scene, looking forward to what's next! 🐐
  and team are cracked AF, they're bringing some serious firepower to the open source/ science AI scene, looking forward to what's next! 🐐
Less than two days ago Kyutai Labs open sourced Moshi - an ~7.6B on-device Speech to Speech foundation model and Mimi - SoTA streaming speech codec! 🔥

The release includes:

1. Moshiko & Moshika - Moshi finetuned on synthetic data (CC-BY license) (https://huggingface.co/collections/kyutai/moshi-v01-release-66eaeaf3302bef6bd9ad7acd)
2. Mimi - Streaiming Audio Codec, processes 24 kHz audio, down to a 12.5 Hz representation with a bandwidth of 1.1 kbps (CC-BY license) (https://huggingface.co/kyutai/mimi)
3. Model checkpoints & Inference codebase written in Rust (Candle), PyTorch & MLX (Apache license) (https://github.com/kyutai-labs/moshi)

How does Moshi work?

1. Moshi processes two audio streams: one for itself and one for the user, with the user's stream coming from audio input and Moshi's stream generated by the model.

2. Along with these audio streams, Moshi predicts text tokens for its speech, enhancing its generation quality.

3. The model uses a small Depth Transformer for codebook dependencies and a large 7B parameter Temporal Transformer for temporal dependencies.

4. The theoretical latency is 160ms, with a practical latency of around 200ms on an L4 GPU.

Model size & inference:

Moshiko/ka are 7.69B param models

bf16 ~16GB VRAM
8-bit ~8GB VRAM
4-bit ~4GB VRAM

You can run inference via Candle 🦀, PyTorch and MLX - based on your hardware.

The Kyutai team,  @adefossez @lmz  and team are cracked AF, they're bringing some serious firepower to the open source/ science AI scene, looking forward to what's next! 🐐
🔥
lmz AtAndDev damerajee den0620 adorkin Forbu14 abdullah Nelathan osanseviero Dunateo bitsoko
🧠
MohamedRashad John6666 fdaudens AtAndDev damerajee bitsoko
https://cdn-avatars.huggingface.co/v1/production/uploads/6487239cca30096ea9f52115/HMte9wjKJgfcxsO-5vb_Q.jpeg
dame rajee
damerajee
user