[News] Facebook's Real-Time TTS system runs on CPUs only!
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper facebook fair tts text-to-speech real-time wavenet rnn spectrogram mel frequency vocoder linguistic features speaker soft style tone phonemes neural recurrent human assistant
Facebook AI's new Text-To-Speech system is able to create 1 second of speech in as little as 500ms, making it real-time. What's even more impressive is the fact that this does not require a rack of GPUs, but runs on merely 4 CPUs.

OUTLINE:
0:00 - Intro
1:00 - Problem Formulation
3:20 - System Explanation
15:00 - Speeding up the computation

https://ai.facebook.com/blog/a-highly-efficient-real-time-text-to-speech-system-deployed-on-cpus/

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there, check this out. Modern text-to-speech systems have come a long way in using neural networks to mimic the nuances of human voice. To generate human-like audio, one second of speech can require a TTS system to output as many as 24,000 samples, sometimes even more. The size and complexity of state-of-the-art models require massive computation, which often needs to run on GPUs or other specialized hardware. This is generated by a system that Facebook AI has built. We're going to look at it. It's called a highly efficient real-time text-to-speech system deployed on CPUs. There's a lot to unpack here, but this is not a paper. This is basically a technical blog post. I think they built this into their products and it's mainly explaining on a high level what they did. They have a real-time text-to-speech system, which means that you have a text like this sentence here, share on Facebook. You give it to the system and the system comes up with a sound wave that says this sentence. If you listen to it, you'll hear share on Facebook. It has to do that in a credible way, such that it is a human-like voice, because people like hearing that. Not these kind of robot, old-school telephone robot voices where they just chunk together words. It has to flow naturally. What you want to do for this is you want to have some sort of recurrent neural network or any sort of autoregressive network that outputs basically these samples here. One at a time. These points you're going to output one at a time. For one second of audio, it can require you to output 24,000 of these data points. 24,000 forward propagations of your autoregressive model. That's massive. If you want to do it in real-time, and that's why real-time is so impressive, you have to do this in less than one second. You have to do these 24,000 forward passes in less than a second. Even more so, this was already possible, I think, but it required a big data center with many, many, many, many GPUs in it. You basically would send your text to this and it would stream back the audio. They can do this just on CPUs. In fact, they can do this on a quad-core CPU in real-time. They can generate this many samples in half a second. Pretty impressive. Let's dive into how they do it. They say, It's deployed in Portal, our video calling service, and available for use across a range of other Facebook applications. From reading support for the visually impaired to visual reality experiences. First, they show this graph right here. What are they doing? This is their entire system. It is chunked in multiple parts. If you're a deep learning practitioner, you're very keen on taking this text and just like a giant neural network and just run it through and generate audio end-to-end. This doesn't really work in this case, first of all, because it would be too many parameters to evaluate this many times. But also, especially text and audio are such different modalities that you'll have to basically chunk this into individual parts. That's what they do. They have this linguistic front-end. The linguistic front-end generates two different things. It generates what needs to be said and how does it need to be said. What needs to be said, they call this linguistic features. The linguistic features are things like phonemes and so on. They don't even have a title for this one. The linguistic front-end converts the input text, probably it's a sentence by sentence thing. This is one sentence of text into a sequence of linguistic features such as phonemes and sentence type. These linguistic features, if it's like share on Facebook, it would be like, okay, is one and then a and then r. These are the phonemes of share, right? It would kind of chunk it into that. That is much closer to what we think of an audio signal. This will make one sound, this will make a sound and this will make a sound. We chunk it into that and then here the how is it said. This would be, for example, the fact that share on Facebook is sort of an instruction. This is different from if this would be a question. If it said, do you want to share this on Facebook? Then the what would output much the same phonemes right here, except it's a different word now. But the how would output, this is a question. As you can see, the information flow right here informs the later stages. This would then cause at the end of the sentence the voice to go up because it's a question. This linguistic front end, as you can see, it still deals with text. It deals with text and it outputs these how features and these linguistic, these what features. The what features now go into an acoustic model. What does the acoustic model do? The acoustic model is meant to generate a spectrum, a spectrogram of the sound. We'll skip this for the moment and go to the neural vocoder. The neural vocoder is a kind of a standard thing in text in any speech producing. It takes a spectrogram of the sound and turns it into actual audio. So this here, I think they achieve it with, they say something similar like a wave RNN based on plus like a CNN. So we'll look into that quickly. So the spectrum, the spectrogram of the sound is going to be a bit of an image. And the image has time on this axis and frequency on this axis. And then there is a, it's usually somewhat like color coded, but it's just intensity. So whenever there is, whenever a frequency at a given time is expressed strongly, it will light up. So it could be something like this. So over time, the mid frequency is always there, but this frequency right here is not there. And this is here at the beginning. So there is a sort of a way to read the spectrogram. But this represents maybe something like this represents the not even sure how much sound this represents, but this can represent maybe something like 200 milliseconds of audio. And you have to basically perform a Fourier transform to transform that into 200 milliseconds of actual wave audio. But since the audio has to be output at what this 24,000 samples per second, what's coming in here is not that much. The acoustic model outputs not 24,000 spectrums per second. So first of all, this has to be up sampled. So this time dimension here has just too few samples. So we use the CNN because this is basically an image in order to up sample this in order to make a long image out of this. And basically, the CNN will have to impute. This is learned, right? This is learned how this spectrogram would look if it were sampled much more densely. And now it has basically the correct number of samples here. And now this wave RNN can step through it and look at these slices and look at the last slices or actually also can look at the last spectrograms, maybe. I don't know. They don't really say what the RNN exactly goes over. But this is how I imagine it. And the wave RNN is based on a wave net architecture. And that means wave net is sort of like if you look to produce this thing right here, you can look back all the way to the beginning. But this would be too many connections, right? It would be too much memory. So what you do is you can look back at the directions at the things right before you in very great detail. But then as you go back further and further, you basically lose detail. See, there's only two connections here in this long stretch where there is many, many connections here at the very beginning. So right before you. So that means you look in more detail what you've produced recently. But you also sort of look back in a more blurry way at what you've produced a long time ago. So this is a wave net architecture. And that's they say they use something like this in order to then actually produce the final audio from the spectrograms. So actually, neural vocoder, you can you can train this thing. You can train it by itself. Right. You simply feed spectrums and you make it go audio. And you know, there's a lot of audio on the Internet. You can simply produce spectrograms from that and then train the vocoder to produce the audio. So the good thing about this pipeline here is you can train a lot of these things independently from each other. I don't actually know whether they do that or not, but you can. You see this box up here, this prosody model, and that will take in. These how features. So how does something need to be said? How? Come on. Well, this is an H. How does something need to be said? And it will transform it into features that the sort of neural network can understand. These these neural networks, they need features like they need embeddings. And as you can see here, it also takes into account the speaker embedding, which is not only how what the sentence is, the fact that the sentence is a question. You would also get the information that the speaker should be, you know, kind of calm. Sorry, that would be the style. The speaker should be maybe a woman voice. And then the language should be it should have like an English sound or a German sound and so on. So this model here will take in all of that and emit features that these neural networks here can understand. So as you see here, the neural vocoder, not only does it transform spectrograms to audio, but it takes into account how you want the audio to sound. And so does this acoustic model. So the acoustic model is sort of along with the prosody model is a bit of the heart of the thing here. It takes in these linguistic features. So what needs to be said, right? The acoustic model, it takes into account the again, the speaker embedding, language embedding and so on. It takes into account the output from the prosody model of the how you would like it to be said. This includes what type of sentence this is. And this it synthesizes it all in order to come up with the spectrum right here in order to come up with these spectrograms. So this is sort of the heart of the of the thing. OK, so about the prosody model, they say we use style embeddings that allow us to create new voice styles, including assistant soft fast project and formal using only a small amount of additional data with the existing data sets. We don't have to create a separate model for each style. We need only 30 to 60 minutes of training data for each voice style. So that happens because you you take into you take in actually an embedding of the feet of the speakers and you train these things independently. So that means you can sort of generalize to a new style really quickly. Here the they describe in essence how their acoustic model works. The fact that they output 13 dimensional MFCC features concatenated with the fundamental frequency and a five dimensional periodicity feature, which is much easier for the acoustic model to generate. So that's what the acoustic model generates. OK, and then here are conditional neural vocoders is the final part of the pipeline, consists of two components, a convolutional neural network that up samples or expands the input feature vectors from the frame rate 200 predictions per second. So that's not it's not 200 milliseconds. It would be 200 spectrograms a second to the sample rate 24000 predictions per second. And this similar to wave RNN, which synthesizes audio samples autoregressively. That means one sample at a time at 24000 samples per second. Crazy, right? So this all seems like it's a lot of computation. And now they describe how they get this to run faster than real time. And they list their their individual contributions here. So if they just run this on one CPU core, you see it takes 80 seconds to produce one second of audio. Then they do optimized inference operators, which means they basically use a pie torch JIT along with some. So these pie torches, these pie torch JIT, which is kind of where you can sort of compile your deep learning model to an optimized to an optimized form that runs faster. And they also customize this. So they get to 20 seconds per one second. Then they do parameter reduction by sparsification. And they are able to abuse the sparse matrix compute operator. I think they implemented a custom one to do that. So this here is very much like if you have a neural network and. That's somehow connected, connected, connected. You want to train it in such a way that it's sparse, meaning that only very few of these connections have non zero weights. And because of that, you don't have to store the non zero weights and you don't also have to compute because something multiplied by zero is going to be zero. So you don't have to compute that. So they're they achieve sparsity 60, 96 percent. And this basically you do by some sort of teacher, teacher, student model. Or there are many ways to do this, but you can you can sparsity regularize a neural network and basically force most of the connections to be sparse while still maintaining a good training error or a good generalization error. So they supercharge this. And with this sparsity, they bring this down to five seconds per second. And then they go further and do blockwise sparsification and distillation. So they distill it to an entirely smaller model that then is also blockwise sparse. So they also enforce this blockwise sparsification. And that not only can you then have a better operator, they implement this block sparse matrix compute operator that is specifically designed to multiply block sparse matrices. They also in the text, they describe that it also optimizes cache access. So if you know about CPUs, they have these level one, level two, level three caches and you can optimize your computations. That's what libraries like LAPOC and BLAS do. You can optimize your computations such that your cache access is optimized. And therefore you can speed up a lot your computations because the amount of times that you actually have to go to your RAM and retrieve something is minimized. And that tends to be the slow part of the process when your cache misses. So again, they achieve 94 percent block sparsity and that almost gets them to real time. And it's still one CPU. So now they parallelize the operators that are doing the heavy heavy lifting to four CPU cores. And that doesn't divide it by four, of course, because there is an overhead in parallelization and synchronization. But that gets you to this one half a second needed to produce one second of audio. So there you're at real time. And that is pretty impressive. And they go on to describe so in detail how they did it. But they also give some examples of what they can do and would like to achieve even better in the future. So here is an example where they can adapt their model to a given style. And here they have a British accent. Recently, we successfully applied our new approach to create a British accented voice. This is the first of more accents and languages to come. And also you can adapt it. Their idea is that you have sort of an assistant and this assistant will be able to adapt to, let's say, your mood. We're also exploring features to make our voice respond intelligently with different styles of speaking based on the context. For example, when you're rushing out the door in the morning and need to know the time, your assistant would match your hurried pace. When you're in a quiet place and you are speaking softly, your assistant would reply to you in a quiet voice. And later, when it gets noisy in the kitchen, your assistant would switch to a projected voice so you can hear the call from your mom. Right. So this ties in very much with sort of conversational AI, so assistance and so on. But it also ties into wearables, I think. So the fact that you are now smaller than real time means you can run this potentially directly on your smartphone. You could run this on your fridge or something, on your stove, in your car without having to stream, basically. So you'll get much more real time on device assistance, maybe even in your watch. And I'm excited by this technology. So far, it seems you can get it in Facebook products, but I'm sure this will come to places. All right. If you enjoyed this, please consider subscribing. Thank you for listening and watching. Leave a like if you liked it and leave a comment if you have something to comment with that. Bye bye.
 Hi there, check this out.
 Modern text-to-speech systems have come a long way in using neural networks to mimic the nuances of human voice.
 To generate human-like audio, one second of speech can require a TTS system to output as many as 24,000 samples, sometimes even more.
 The size and complexity of state-of-the-art models require massive computation, which often needs to run on GPUs or other specialized hardware.
 This is generated by a system that Facebook AI has built.
 We're going to look at it. It's called a highly efficient real-time text-to-speech system deployed on CPUs.
 There's a lot to unpack here, but this is not a paper. This is basically a technical blog post.
 I think they built this into their products and it's mainly explaining on a high level what they did.
 They have a real-time text-to-speech system, which means that you have a text like this sentence here, share on Facebook.
 You give it to the system and the system comes up with a sound wave that says this sentence.
 If you listen to it, you'll hear share on Facebook.
 It has to do that in a credible way, such that it is a human-like voice, because people like hearing that.
 Not these kind of robot, old-school telephone robot voices where they just chunk together words. It has to flow naturally.
 What you want to do for this is you want to have some sort of recurrent neural network or any sort of autoregressive network that outputs basically these samples here.
 One at a time. These points you're going to output one at a time.
 For one second of audio, it can require you to output 24,000 of these data points.
 24,000 forward propagations of your autoregressive model. That's massive.
 If you want to do it in real-time, and that's why real-time is so impressive, you have to do this in less than one second.
 You have to do these 24,000 forward passes in less than a second.
 Even more so, this was already possible, I think, but it required a big data center with many, many, many, many GPUs in it.
 You basically would send your text to this and it would stream back the audio.
 They can do this just on CPUs. In fact, they can do this on a quad-core CPU in real-time.
 They can generate this many samples in half a second. Pretty impressive.
 Let's dive into how they do it. They say,
 It's deployed in Portal, our video calling service, and available for use across a range of other Facebook applications.
 From reading support for the visually impaired to visual reality experiences.
 First, they show this graph right here. What are they doing?
 This is their entire system. It is chunked in multiple parts.
 If you're a deep learning practitioner, you're very keen on taking this text and just like a giant neural network and just run it through and generate audio end-to-end.
 This doesn't really work in this case, first of all, because it would be too many parameters to evaluate this many times.
 But also, especially text and audio are such different modalities that you'll have to basically chunk this into individual parts.
 That's what they do. They have this linguistic front-end.
 The linguistic front-end generates two different things.
 It generates what needs to be said and how does it need to be said.
 What needs to be said, they call this linguistic features.
 The linguistic features are things like phonemes and so on.
 They don't even have a title for this one.
 The linguistic front-end converts the input text, probably it's a sentence by sentence thing.
 This is one sentence of text into a sequence of linguistic features such as phonemes and sentence type.
 These linguistic features, if it's like share on Facebook, it would be like, okay, is one and then a and then r.
 These are the phonemes of share, right? It would kind of chunk it into that.
 That is much closer to what we think of an audio signal.
 This will make one sound, this will make a sound and this will make a sound.
 We chunk it into that and then here the how is it said.
 This would be, for example, the fact that share on Facebook is sort of an instruction.
 This is different from if this would be a question.
 If it said, do you want to share this on Facebook?
 Then the what would output much the same phonemes right here, except it's a different word now.
 But the how would output, this is a question.
 As you can see, the information flow right here informs the later stages.
 This would then cause at the end of the sentence the voice to go up because it's a question.
 This linguistic front end, as you can see, it still deals with text.
 It deals with text and it outputs these how features and these linguistic, these what features.
 The what features now go into an acoustic model.
 What does the acoustic model do?
 The acoustic model is meant to generate a spectrum, a spectrogram of the sound.
 We'll skip this for the moment and go to the neural vocoder.
 The neural vocoder is a kind of a standard thing in text in any speech producing.
 It takes a spectrogram of the sound and turns it into actual audio.
 So this here, I think they achieve it with, they say something similar like a wave RNN based on plus like a CNN.
 So we'll look into that quickly.
 So the spectrum, the spectrogram of the sound is going to be a bit of an image.
 And the image has time on this axis and frequency on this axis.
 And then there is a, it's usually somewhat like color coded, but it's just intensity.
 So whenever there is, whenever a frequency at a given time is expressed strongly, it will light up.
 So it could be something like this.
 So over time, the mid frequency is always there, but this frequency right here is not there.
 And this is here at the beginning.
 So there is a sort of a way to read the spectrogram.
 But this represents maybe something like this represents the not even sure how much sound this represents,
 but this can represent maybe something like 200 milliseconds of audio.
 And you have to basically perform a Fourier transform to transform that into 200 milliseconds of actual wave audio.
 But since the audio has to be output at what this 24,000 samples per second, what's coming in here is not that much.
 The acoustic model outputs not 24,000 spectrums per second.
 So first of all, this has to be up sampled.
 So this time dimension here has just too few samples.
 So we use the CNN because this is basically an image in order to up sample this in order to make a long image out of this.
 And basically, the CNN will have to impute.
 This is learned, right?
 This is learned how this spectrogram would look if it were sampled much more densely.
 And now it has basically the correct number of samples here.
 And now this wave RNN can step through it and look at these slices and look at the last slices
 or actually also can look at the last spectrograms, maybe.
 I don't know. They don't really say what the RNN exactly goes over.
 But this is how I imagine it.
 And the wave RNN is based on a wave net architecture.
 And that means wave net is sort of like if you look to produce this thing right here, you can look back all the way to the beginning.
 But this would be too many connections, right?
 It would be too much memory.
 So what you do is you can look back at the directions at the things right before you in very great detail.
 But then as you go back further and further, you basically lose detail.
 See, there's only two connections here in this long stretch where there is many, many connections here at the very beginning.
 So right before you.
 So that means you look in more detail what you've produced recently.
 But you also sort of look back in a more blurry way at what you've produced a long time ago.
 So this is a wave net architecture.
 And that's they say they use something like this in order to then actually produce the final audio from the spectrograms.
 So actually, neural vocoder, you can you can train this thing.
 You can train it by itself.
 Right. You simply feed spectrums and you make it go audio.
 And you know, there's a lot of audio on the Internet.
 You can simply produce spectrograms from that and then train the vocoder to produce the audio.
 So the good thing about this pipeline here is you can train a lot of these things independently from each other.
 I don't actually know whether they do that or not, but you can.
 You see this box up here, this prosody model, and that will take in.
 These how features. So how does something need to be said?
 How? Come on.
 Well, this is an H. How does something need to be said?
 And it will transform it into features that the sort of neural network can understand.
 These these neural networks, they need features like they need embeddings.
 And as you can see here, it also takes into account the speaker embedding, which is not only how what the sentence is, the fact that the sentence is a question.
 You would also get the information that the speaker should be, you know, kind of calm.
 Sorry, that would be the style. The speaker should be maybe a woman voice.
 And then the language should be it should have like an English sound or a German sound and so on.
 So this model here will take in all of that and emit features that these neural networks here can understand.
 So as you see here, the neural vocoder, not only does it transform spectrograms to audio, but it takes into account how you want the audio to sound.
 And so does this acoustic model.
 So the acoustic model is sort of along with the prosody model is a bit of the heart of the thing here.
 It takes in these linguistic features.
 So what needs to be said, right?
 The acoustic model, it takes into account the again, the speaker embedding, language embedding and so on.
 It takes into account the output from the prosody model of the how you would like it to be said.
 This includes what type of sentence this is.
 And this it synthesizes it all in order to come up with the spectrum right here in order to come up with these spectrograms.
 So this is sort of the heart of the of the thing.
 OK, so about the prosody model, they say we use style embeddings that allow us to create new voice styles,
 including assistant soft fast project and formal using only a small amount of additional data with the existing data sets.
 We don't have to create a separate model for each style.
 We need only 30 to 60 minutes of training data for each voice style.
 So that happens because you you take into you take in actually an embedding of the feet of the speakers and you train these things independently.
 So that means you can sort of generalize to a new style really quickly.
 Here the they describe in essence how their acoustic model works.
 The fact that they output 13 dimensional MFCC features concatenated with the fundamental frequency and a five dimensional periodicity feature,
 which is much easier for the acoustic model to generate.
 So that's what the acoustic model generates.
 OK, and then here are conditional neural vocoders is the final part of the pipeline,
 consists of two components, a convolutional neural network that up samples or expands the input feature vectors from the frame rate 200 predictions per second.
 So that's not it's not 200 milliseconds.
 It would be 200 spectrograms a second to the sample rate 24000 predictions per second. And this similar to wave RNN,
 which synthesizes audio samples autoregressively.
 That means one sample at a time at 24000 samples per second.
 Crazy, right?
 So this all seems like it's a lot of computation.
 And now they describe how they get this to run faster than real time.
 And they list their their individual contributions here.
 So if they just run this on one CPU core, you see it takes 80 seconds to produce one second of audio.
 Then they do optimized inference operators, which means they basically use a pie torch JIT along with some.
 So these pie torches, these pie torch JIT, which is kind of where you can sort of compile your deep learning model to an optimized to an optimized form that runs faster.
 And they also customize this.
 So they get to 20 seconds per one second.
 Then they do parameter reduction by sparsification.
 And they are able to abuse the sparse matrix compute operator.
 I think they implemented a custom one to do that.
 So this here is very much like if you have a neural network and.
 That's somehow connected, connected, connected.
 You want to train it in such a way that it's sparse, meaning that only very few of these connections have non zero weights.
 And because of that, you don't have to store the non zero weights and you don't also have to compute because something multiplied by zero is going to be zero.
 So you don't have to compute that.
 So they're they achieve sparsity 60, 96 percent.
 And this basically you do by some sort of teacher, teacher, student model.
 Or there are many ways to do this, but you can you can sparsity regularize a neural network and basically force most of the connections to be sparse while still maintaining a good training error or a good generalization error.
 So they supercharge this.
 And with this sparsity, they bring this down to five seconds per second.
 And then they go further and do blockwise sparsification and distillation.
 So they distill it to an entirely smaller model that then is also blockwise sparse.
 So they also enforce this blockwise sparsification.
 And that not only can you then have a better operator, they implement this block sparse matrix compute operator that is specifically designed to multiply block sparse matrices.
 They also in the text, they describe that it also optimizes cache access.
 So if you know about CPUs, they have these level one, level two, level three caches and you can optimize your computations.
 That's what libraries like LAPOC and BLAS do.
 You can optimize your computations such that your cache access is optimized.
 And therefore you can speed up a lot your computations because the amount of times that you actually have to go to your RAM and retrieve something is minimized.
 And that tends to be the slow part of the process when your cache misses.
 So again, they achieve 94 percent block sparsity and that almost gets them to real time.
 And it's still one CPU. So now they parallelize the operators that are doing the heavy heavy lifting to four CPU cores.
 And that doesn't divide it by four, of course, because there is an overhead in parallelization and synchronization.
 But that gets you to this one half a second needed to produce one second of audio.
 So there you're at real time. And that is pretty impressive.
 And they go on to describe so in detail how they did it.
 But they also give some examples of what they can do and would like to achieve even better in the future.
 So here is an example where they can adapt their model to a given style.
 And here they have a British accent.
 Recently, we successfully applied our new approach to create a British accented voice.
 This is the first of more accents and languages to come.
 And also you can adapt it.
 Their idea is that you have sort of an assistant and this assistant will be able to adapt to, let's say, your mood.
 We're also exploring features to make our voice respond intelligently with different styles of speaking based on the context.
 For example, when you're rushing out the door in the morning and need to know the time, your assistant would match your hurried pace.
 When you're in a quiet place and you are speaking softly, your assistant would reply to you in a quiet voice.
 And later, when it gets noisy in the kitchen, your assistant would switch to a projected voice so you can hear the call from your mom.
 Right. So this ties in very much with sort of conversational AI, so assistance and so on.
 But it also ties into wearables, I think.
 So the fact that you are now smaller than real time means you can run this potentially directly on your smartphone.
 You could run this on your fridge or something, on your stove, in your car without having to stream, basically.
 So you'll get much more real time on device assistance, maybe even in your watch.
 And I'm excited by this technology.
 So far, it seems you can get it in Facebook products, but I'm sure this will come to places.
 All right. If you enjoyed this, please consider subscribing.
 Thank you for listening and watching.
 Leave a like if you liked it and leave a comment if you have something to comment with that. Bye bye.