text
I just added 5 more models to my open source TTS model benchmark, 
I just added 5 more models to my open source TTS model benchmark, 
resource
https://huggingface.co/spaces/ttsds/benchmark
space
ttsds/benchmark
https://huggingface.co/spaces/ttsds/benchmark
text
.
.
new_line
text
Let's talk about the results!
Let's talk about the results!
new_line
new_line
text
Over the last couple days, I added 
Over the last couple days, I added 
resource
https://huggingface.co/jbetker/tortoise-tts-v2
model
jbetker/tortoise-tts-v2
https://huggingface.co/jbetker/tortoise-tts-v2
text
, 
, 
resource
https://huggingface.co/metavoiceio/metavoice-1B-v0.1
model
metavoiceio/metavoice-1B-v0.1
https://huggingface.co/metavoiceio/metavoice-1B-v0.1
text
, 
, 
resource
https://huggingface.co/audo/HierSpeechpp
model
audo/HierSpeechpp
https://huggingface.co/audo/HierSpeechpp
text
, and the unofficial implementations of 
, and the unofficial implementations of 
resource
https://huggingface.co/spaces/amphion/NaturalSpeech2
space
amphion/NaturalSpeech2
https://huggingface.co/spaces/amphion/NaturalSpeech2
text
 and 
 and 
resource
https://huggingface.co/amphion/valle
model
amphion/valle
https://huggingface.co/amphion/valle
text
 by 
 by 
link
https://huggingface.co/amphion
https://huggingface.co/amphion
new_line
new_line
text
Takeaways:
Takeaways:
new_line
text
- TorToiSe does very well, falling into second place after StyleTTS 2, which is also ranked first in the human evaluation at 
- TorToiSe does very well, falling into second place after StyleTTS 2, which is also ranked first in the human evaluation at 
resource
https://huggingface.co/spaces/TTS-AGI/TTS-Arena
space
TTS-AGI/TTS-Arena
https://huggingface.co/spaces/TTS-AGI/TTS-Arena
text
.
.
new_line
text
- MetaVoice-1B's overall score is dragged down by its Intelligibility Score (probably due to utterances being cut short), it achieves #3 in Speaker Score, which indicates good voice cloning ability. 
- MetaVoice-1B's overall score is dragged down by its Intelligibility Score (probably due to utterances being cut short), it achieves #3 in Speaker Score, which indicates good voice cloning ability. 
new_line
text
- HierSpeech++ lands in the middle of the road in terms of performance, but excels at the Environment Score, achieving #2 - this means the model is especially good at modeling recording conditions such as microphone and background noise.
- HierSpeech++ lands in the middle of the road in terms of performance, but excels at the Environment Score, achieving #2 - this means the model is especially good at modeling recording conditions such as microphone and background noise.
new_line
text
- The Amphion models, possibly due to not being trained for the same amount as in the papers, achieve relatively low scores. However, they seem to struggle for different reasons. The autoregressive VALLE models have low Intelligibility Scores (possibly due to "babbling" or early stop tokens) while NaturalSpeech2 has low Speaker and Prosody scores.
- The Amphion models, possibly due to not being trained for the same amount as in the papers, achieve relatively low scores. However, they seem to struggle for different reasons. The autoregressive VALLE models have low Intelligibility Scores (possibly due to "babbling" or early stop tokens) while NaturalSpeech2 has low Speaker and Prosody scores.
new_line
new_line
text
What's next? 
What's next? 
new_line
text
I'm planning to add more open source TTS models like 
I'm planning to add more open source TTS models like 
resource
https://huggingface.co/suno/bark
model
suno/bark
https://huggingface.co/suno/bark
text
, 
, 
resource
https://huggingface.co/CAMB-AI/MARS5-TTS
model
CAMB-AI/MARS5-TTS
https://huggingface.co/CAMB-AI/MARS5-TTS
text
 and 
 and 
resource
https://huggingface.co/fishaudio/fish-speech-1.2
model
fishaudio/fish-speech-1.2
https://huggingface.co/fishaudio/fish-speech-1.2
text
. I'll also write an article on these and all the other results soon, since our paper, 
. I'll also write an article on these and all the other results soon, since our paper, 
resource
https://huggingface.co/papers/2407.12707
paper
2407.12707
https://huggingface.co/papers/2407.12707
TTSDS -- Text-to-Speech Distribution Score (2407.12707)
text
, mostly focused on establishing the benchmark itself rather than the indiviual TTS systems.
, mostly focused on establishing the benchmark itself rather than the indiviual TTS systems.
I just added 5 more models to my open source TTS model benchmark, https://huggingface.co/spaces/ttsds/benchmark.
Let's talk about the results!

Over the last couple days, I added https://huggingface.co/jbetker/tortoise-tts-v2, https://huggingface.co/metavoiceio/metavoice-1B-v0.1, https://huggingface.co/audo/HierSpeechpp, and the unofficial implementations of https://huggingface.co/spaces/amphion/NaturalSpeech2 and https://huggingface.co/amphion/valle by https://huggingface.co/amphion

Takeaways:
- TorToiSe does very well, falling into second place after StyleTTS 2, which is also ranked first in the human evaluation at https://huggingface.co/spaces/TTS-AGI/TTS-Arena.
- MetaVoice-1B's overall score is dragged down by its Intelligibility Score (probably due to utterances being cut short), it achieves #3 in Speaker Score, which indicates good voice cloning ability. 
- HierSpeech++ lands in the middle of the road in terms of performance, but excels at the Environment Score, achieving #2 - this means the model is especially good at modeling recording conditions such as microphone and background noise.
- The Amphion models, possibly due to not being trained for the same amount as in the papers, achieve relatively low scores. However, they seem to struggle for different reasons. The autoregressive VALLE models have low Intelligibility Scores (possibly due to "babbling" or early stop tokens) while NaturalSpeech2 has low Speaker and Prosody scores.

What's next? 
I'm planning to add more open source TTS models like https://huggingface.co/suno/bark, https://huggingface.co/CAMB-AI/MARS5-TTS and https://huggingface.co/fishaudio/fish-speech-1.2. I'll also write an article on these and all the other results soon, since our paper, https://huggingface.co/papers/2407.12707, mostly focused on establishing the benchmark itself rather than the indiviual TTS systems.
🔥
Gatozu35 SicariusSicariiStuff
/avatars/222ab1f6f7cf249b6c13b6f1a81ec319.svg
Full Name
Gatozu35
user
https://cdn-avatars.huggingface.co/v1/production/uploads/6569216f9c96f1a47bf45788/mCLqmAs4dOjKdxNQVAp1w.png
Sica Rius
SicariusSicariiStuff
user
/avatars/fa1151e05cc22698a75a48134aa38dbc.svg
Christoph Minixhofer
cdminix
user