End-to-End Adversarial Text-to-Speech (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper tts text-to-speech aligner convolutions spectrogram mel alignment phonemes deepmind deep mind dynamic time warping gaussian kernel adversarial gan discriminator tokens sound wave speech
Text-to-speech engines are usually multi-stage pipelines that transform the signal into many intermediate representations and require supervision at each step. When trying to train TTS end-to-end, the alignment problem arises: Which text corresponds to which piece of sound? This paper uses an alignment module to tackle this problem and produces astonishingly good sound.

OUTLINE:
0:00 - Intro & Overview
1:55 - Problems with Text-to-Speech
3:55 - Adversarial Training
5:20 - End-to-End Training
7:20 - Discriminator Architecture
10:40 - Generator Architecture
12:20 - The Alignment Problem
14:40 - Aligner Architecture
24:00 - Spectrogram Prediction Loss
32:30 - Dynamic Time Warping
38:30 - Conclusion

Paper: https://arxiv.org/abs/2006.03575
Website: https://deepmind.com/research/publications/End-to-End-Adversarial-Text-to-Speech

Abstract:
Modern text-to-speech synthesis pipelines typically involve multiple processing stages, each of which is designed or learnt independently from the rest. In this work, we take on the challenging task of learning to synthesise speech from normalised text or phonemes in an end-to-end manner, resulting in models which operate directly on character or phoneme input sequences and produce raw speech audio outputs. Our proposed generator is feed-forward and thus efficient for both training and inference, using a differentiable monotonic interpolation scheme to predict the duration of each input token. It learns to produce high fidelity audio through a combination of adversarial feedback and prediction losses constraining the generated audio to roughly match the ground truth in terms of its total duration and mel-spectrogram. To allow the model to capture temporal variation in the generated audio, we employ soft dynamic time warping in the spectrogram-based prediction loss. The resulting model achieves a mean opinion score exceeding 4 on a 5 point scale, which is comparable to the state-of-the-art models relying on multi-stage training and additional supervision.

Authors: Jeff Donahue, Sander Dieleman, Mikołaj Bińkowski, Erich Elsen, Karen Simonyan

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 In this work, we take on the challenging task of learning to synthesize speech from normalized text or phonemes in an entwined manner, resulting in models which operate directly on character or phoneme input sequences and produce raw speech audio outputs. Okay, that wasn't the real model. I just thought it sounded really funny. This is a text to speech model, and it actually sounds like this. In this work, we take on the challenging task of learning to synthesize speech from normalized text or phonemes in an entwined manner, resulting in models which operate directly on character or phoneme input sequences and produce raw speech audio outputs. Okay, now you've probably, if you have listened to the text and not just how the text sounds, you have gotten what this paper is about. So the paper is called End to End Adversarial Text to Speech by Jeff Donahue, Sander Dieleman, Mikolai Binkowski, Eric Elson and Karen Simonian of, I believe, of mostly deep mind. And this paper on a high level, it produces speech, so the sound waves of speech directly from text or from what they call normalized text or phoneme text. And it does so without any intermediate supervised representations. And that's a challenging task. And the main problems here are the alignment problem that they have to solve and actually making this work in an adversarial manner. So we're going to look at this paper. As always, if you like work like this, consider subscribing and sharing it out. And if you have any comments, leave them in the comment section. Okay, so what's the problem with text to speech? Text to speech is basically you take a piece of text like this one, modern text to speech synthesis pipelines typically involve blah, blah, blah. And you want to make a model that takes this and outputs sound waves, as if a human would say it right so you do modern text to speech and so on. Now you have multiple problems when doing this. First of all, the text here is words. Let's say we can tokenize the text into words. So you have modern text to speech. Those are four tokens. However, sound waves are, of course, much, much densely sampled. So these sound waves, they are typically in the order of something like 24 kilohertz sampled. So that's the the ratio of one token to output samples is super high. So one token will produce many, many thousand samples in the speech. So that's the first problem. The second problem is that if you have training data, so you have data that has a piece of text and you have the sound wave that a human you know the human read that particular piece of text, you still don't know which word exactly corresponds to which portion of that text. You simply know the entire text corresponds to the entire sound wave. You don't know this word text right here. You don't know. You don't know where it starts and where it ends in this sound wave. And the last problem you obviously have is that you want to make this in a way that it generalizes, that it sounds like a human, but also generalizes to some other text. And this paper here solves all of these problems jointly by doing an adversarial approach to learning. And it does it end to end. Now adversarial simply means that you have a generator that takes in the piece of text right here and generates this sound wave. And then you have a discriminator that looks at this sound wave and it looks at the real sound wave. So the real, okay, the real sense. So let's say this over here is real. And this is what the generator has produced. The discriminator tries to discriminate between the two. Now this is not entirely the same thing as a supervised loss. Usually in a GAN, you do not have the corresponding samples, right? You simply input a real sample here and the generator produces a generated sample here. You do not necessarily have in a classic GAN the corresponding sample. Here you assume that you have the corresponding real sample, but it's still different than supervised learning in that both go through a discriminator and the discriminator tries to tell, the discriminator is a neural network, tries to tell which one is real and which one is generated. In fact, the discriminator is a set of neural networks. We're going to go into that shortly. So it's adversarial in the sense that there is a generator and a discriminator. And it is end to end in the sense that usually what these pipelines do is they take the text. And we've looked at this, for example, in the video about this Facebook's text to speech system. So they take the text. And the first thing they do is they would produce a set of whatever they would call features like textual features. So these are these are sort of intermediate features for the text to be produced. And then another model would take these and it would produce something like spectrograms spectrograms. And then another model would take the spectrograms and finally produce sound or speech. So you have usually in these systems, you have intermediate representation. And each of these models right here can be trained by itself. So that's an advantage that you can train. For example, you can train a model that goes from a spectrogram to a sound wave by itself. And you don't need you simply need sound for that. The computation from sound to spectrogram is super easy. So you can generate your own training data. So you can go from spectrogram to sound. You can train a model like this. So in these pipelines, usually there are multiple stages and each of these models has to be trained by itself. This paper tries to do this end to end. That means you input the text and you get out the you get out the sound wave. And and there is nothing in between that the I mean, of course, there are latent representations, but you train it ends to end in one go. Okay, so let's look at the different systems they employ. First of all, let's look at the discriminators, because that's the easiest. So they have these discriminators and these are adopted from this GAN TTS paper. Now, as we already said, the discriminator try to to differentiate between real and fake sound. And they do it in a sort of. So if they were to just look at the entire sound wave, then it would just basically reduce to comparing the two. But instead, the discriminators, they operate on very small windows. So in specific, they have five different discriminators and each of the five different discriminators take a different size window length. But all of these windows are super short. So one discriminator might take this long windows. Another discriminator might take a bit longer windows and another discriminator might take a bit shorter windows. And that, of course, from the real and from the fake and the discriminators simply try to discriminate only in these windows, whether it's real or fake. And now we're a bit more into the GAN setting where, you know, you simply have one data point of the real and one data point of the fake. And you have to compare them. And this here, I believe, is one of the keys why this model generalizes, because the discriminators, basically, they try to assess whether a short sequence sounds like real or sounds like fake and whether the two samples sound alike in different scales of time. And by only doing this on these short scales, you can you can the loss sort of generalizes. Otherwise, if you compare it on the entire sound wave, it would just reduce to comparing point like point by point right here. And if the generator produces something that's not exactly aligned, then, of course, every point would be wrong. And you'd run into all sorts of questions. So this is a set of five discriminators that all try to take each takes a different length of sub sub sound of this wave tries to discriminate real from fake. That's the discriminator loss. They have an additional discriminator loss where they compute spectrograms of these things. So spectrograms and compute spectrogram of this. And they have a discriminator, another neural network here that it tries to distinguish which one is real and which one is fake. Note that this is not the same as down here where the spectrogram is an intermediate representation. Here, the sound is the output and from the sound you compute the spectrogram and then you compare the two. So this is simply a different spectrogram is a different feature space for the discriminator to compute the loss. It is not an intermediate representation on the way to produce the sound itself so that the difference. That's the difference here between the classic approach and this approach. So it's end to end adversarial. So we got the discriminator. The discriminators simply try to differentiate the sound waves and short scales as well as the spectrograms. Now, the second part is, of course, the generator. How do we even produce sound? And that's this diagram right here. So you have this Gantt generator. This is a generator that takes in a hidden representation. It takes in tokens. Let's say token one, token two. Let's go from the one before. Think of a sentence. Hello there. There. OK, so it takes these tokens and of course it takes like hidden representations of the tokens. And it will output. It will output for each one or for this joint sequence. One hidden, two. It would output the sound wave. OK. And this has been a paper before, the Gantt TTS. And you also you condition it on the speaker and on latent variables like how you want the pitch to be and so on. That's not really important for us right here. The generator can simply take these token embeddings and produce sound. The problem is in the original paper, you had an alignment. You knew which token corresponded to which piece of sound. And therefore you sort of knew. So after that, you need to compare this to the generator thing. And you knew which token corresponded to which piece of sound right here. So the generator knew what it had to produce from each token, how long it should be and so on. So this is the generally the alignment problem or what I call the alignment problem. So if you take a piece of text like this entire paragraph right here. Let's look at this paragraph. This paragraph to read it out takes like 30 to 60 seconds. You can't train really models that output this long of sound. It would be too big of a sample. You want to train ideally on segments. They train segments that are, I believe here, two second windows from each examples. So because they say if we train on 20 seconds, that would just be wasteful and prohibitively expensive. Now, the problem, of course, is if I simply take a window here like this one of two seconds and I have my human that has read this entire paragraph in one go, I have no clue again which part of the entire sound wave of this paragraph this subsequence corresponds to. A good guess would be to go like, well, this is about 50 percent in. So maybe here to here. Maybe. Who knows. And even within that, and that's what we discussed before, you have no clue how long this word here is going to take up within this piece of sound. And that's the general alignment problem right here. So in this entire sound wave, where is the piece and how do these words distribute across the wave of sound? The original model had, as I understand it, such alignments and therefore this generator could work really well because you had these alignments without these alignments. It doesn't work as well. And you can on their website that I've shown you initially, you can listen to samples where they disable each of these things. So this generator is really good at producing sound when it has these alignments. So the challenging task here is how do you compute these alignments? How do you compute this thing? If you don't have it, if you're if you don't have it in your training data, so it needs to be part of the loss. So that's what this entire architecture down here is. So the text is down here. It goes in. And the first thing they do is they normalize the text and they transform it into phonemes, which is you can do this in a deterministic fashion. There are scripts that do this. This is the only preprocessing they do. And they can also leave it away in the heaven ablation on their website. So this is like phoneme text. Cat sat on the mat. Now, this phoneme text goes through these big block of convolutions and of dilated convolutions. And this outputs a 200 hertz representation token length. Alignment. OK, I should specify this. So for each token here, it outputs a length. So this thing predicts the length of each of the tokens. These all of this all of this thing here is to to embed the tokens in hidden space and then predict its length. You can see that right here. OK, so first we use F to to take X. So F is a stack of dilated convolutions and it takes X and outputs a hidden representation. So H. So first X goes to H and then H is used to predict the length of each of the tokens. H is used to predict L and L is the length of that token. So we embed this into a hidden representation with this right here. And then we use this stack to predict the length of each token. So this could be this could say something like this cat token right here is 20 milliseconds long. Or instead of milliseconds, you would use something like frames or data points. Maybe this is 200 data points long. And then SAT is a bit shorter. So this is 100 long. And ON is really short. So this is 50 long. So for each token, it predicts the length. All right. So now if we have the length of each, we can sort of calculate where the starting point is. So if we want to know if we know that here is the beginning and the beginning of the sentence, we we conservatively assume that there is so they give some silence buffer here. But roughly, you can assume that the beginning of the speech corresponds to the first the first token. Right. You can simply trace the waveform. And whenever it goes up, that's where the first token starts. And then since we know that's where the first token starts, and if we could predict the length of each one correctly, we could simply sum those up to figure out where our word starts. So if we want to know where on starts, we simply go from the beginning and go 200 plus 100 milliseconds. So our data points 200 plus 100. Here is where on starts. OK. And if we want to figure out the middle of on, we simply add the half of this number. So plus 25 gets you to the middle. So this is this here is the center of the token on. So for each token, we predict the length like this. And thereby, we can just calculate for each one by summing up from the beginning and then adding half of its own length where the center of that token in the entire sequence is. And now we do this. We said we take random two second audio, but we do this procedure for the entire for the entire text. OK, for the for every single token in the text that we look at in the 20 second text, we do this because then for each token, we'll get a token center. And now the aligners job here is to align that to the actual sound. So what we also give the generators here, the offset. So let's say we have this 20 second of speech and we randomly sampled these two seconds. And that's maybe five seconds from the beginning. We also tell it this is five seconds right here. So what we can now do is we can calculate back sort of and say, OK, here I have I first need to discard five seconds of my signal and I have a prediction how long each token is. So I can just cross out tokens until I have basically wasted five seconds. And then I know, OK, from here to wherever these things sum up to two seconds from here to that. Those are my two seconds that I want to look at. Now, this is how I figure out where in the big sound wave my fragment is. Because I have this offset where I sampled it and I simply add use this and the predicted lengths to figure it out. I still need to figure out these tokens that are actually in the span. How do they distribute? And that's what this aligner here does. Since we've already predicted the token centers, we simply assume that if these are correct, right, then if this is, let's say if this is one second long, I assume that the middle is after point five seconds. So this is one second. The middle is point five seconds. So I think that this token is aligned right here. This is the center of the token. Now, we want to be a little bit a little bit fuzzy with respect to that. So what they do is they sort of use a Gaussian kernel right here. So for each token, as you can see here, each token has a center, which is here. So the y axis is the time in sound and the x axis is the token. And for each token, we say, well, it doesn't have to be exactly there. It can be so they put a Gaussian kernel like this. OK, if you imagine this kernel popping out of the frame, they say this is about where the center is. And for this token, right for this token right here, they say, well, it's it's probably here in the middle, but it could also be here or here or here or here. And we weigh this like this. So these are these are the weights. And then you simply sum up the weights with these embeddings. So for each token out of this dilated convolution block, you get a hidden embedding. And by using this alignment matrix that you computed by predicting the lengths and therefore predicting the centers of the tokens, you can then sort of shift. So first, you assume that h1, h2, h3, if you were to do nothing, these would just all take up like a third of the time. And now by multiplying with this matrix, you have the opportunity because you predicted a longer length for the first token. You have the opportunity to shift that a bit to the right and maybe shorten the second token a bit. And then the third token goes until the end. OK, that's what this aligner thing is. This is not a model by itself. All that this takes in is the computation right here of the token lengths. This estimates these token lengths for each of the tokens. And the rest is deterministic. It's simply saying, OK, how much is the offset? Cool. That's how we know where in the sound wave we are. And then where is each of the centers? And we simply do that by summing up the predicted token lengths. And then we use a Gaussian kernel with like a set hyperparameter to be a little bit fuzzy with respect to these lengths right here. So to be differentiable, basically. And that will that will ultimately train this loss, this model right here that computes the token lengths. Right. So we sum up in a weighted fashion these embeddings right here. And that's what goes into the generator. So now we have embeddings and we have the alignments for the embeddings, which are these pieces of where in the sound wave these are. And from that, the generator can now produce the sound wave itself. OK. And that's basically that's just an up sampling here. I think that's just an up convolution up sampling from 200 hertz signal to a 24 kilohertz signal. Cool. So that's that. Now they discover this doesn't work. And why doesn't it work? It's because at the beginning of training, these token length predictions here are pretty crappy. And so that means that I guess especially this part, even where you say, well, where where in the sound wave of my 20 seconds do I even need to cut out to compare with the discriminator? Right. So if you give if you sample this piece here and that's what you give to the discriminator, but your length predictions are so far off that the generator is trying to produce this particular piece because it thinks it thinks, oh, instead of producing this token here, which is what the discriminator looks at, it produces these tokens here. Of course, you have no chance, no matter how good your adversarial loss is. Remember, the this is these length predictions are used to see basically to see which of these tokens the generator needs to produce the sound for and how they're aligned. So they have an additional loss right here. What they do is they produce from the again, they go via the spectrograms within this spectrogram prediction loss. So they say we discovered that adversarial feedback is insufficient to learn alignment. At the start of training, the aligner does not produce an accurate alignment. So the information in the input tokens is incorrectly temporally distributed. This encourages the decoder to ignore the aligner output. The unconditional discriminators provide no useful signal to correct this. Oh, yeah, I should have mentioned this. The discriminators here, since you don't know, you don't know which tokens you should produce. The discriminators are unconditional. They don't know which text is produced. You don't give them the tokens. You simply give them the sound waves. That's something I find particularly interesting here. Now, you of course, this wouldn't work in a like a traditional again, because you simply have a data sample here and a data sample right here. But in this case, you of course have the corresponding sound samples. But still, they are, you know, they are cut down to a subsequence. So you don't know which text you're producing. So you have to make the discriminators unconditional. And therefore, they are going to discriminate, as we said, between potentially between two completely non overlapping pieces of the sound wave, which, of course, doesn't help you. And then the aligner can also not learn anything because there is no learning signal because everything just says this is not the same. OK. And that's what they say here. We face a different problem. We do not have aligned ground truth. Conditional discriminators, which they don't have, need an aligner module, which cannot function correctly at the start of training, effectively turning them into unconditional discriminators. So even if they were to input the text, it would still be the wrong text because their aligner is wrong at the beginning. Although it should be possible in theory to train the discriminators aligner module adversarially, we find that this does not work in practice and training gets stuck. So what do they do? They say instead we propose to guide learning by using an explicit prediction loss in the spectrogram domain. We minimize the L one loss between the log scale male spectrograms of the generator output and the corresponding ground truth training window. This helps learning to take off and renders conditional discriminators unnecessary, simplifying the model. So they take the they take the spectrogram of the generator output and the corresponding ground truth training window and they simply calculate the L one difference of the spectrograms. Now this, as I understand it, this is different from this is different from because we said they also have a discriminator on the spectrograms. This is different from that. This is even in addition to that. So here somewhere we had this. This was the discriminator on the spectrograms. And I think this is even different. So what they're doing is they also the discriminator simply decides do the spectrograms look real or fake? Does the spectrogram look real or fake? Now they also take the spectrograms and compare them with the L one loss. So this is exactly what they said they wouldn't do right here. Now it's still the case, right? It's still the case that they don't use spectrograms as intermediate representations, but they now do have a supervised loss on the spectrograms. And one of the motivations to do this end to end is saying, you know, maybe these auxiliary losses and supervised losses, they sort of distract. They're good to guide the training, but they sort of distract. And now they see, OK, maybe we have to introduce this one right here in order to make the training start, because this is a real signal. But again, you run into a problem, namely, if you produce something with the generator. And so first of all, this is not a discriminator anymore. This is a true L one loss. So we potentially run into this problem, right? Of the of the generator simply copying the input because you always tell it what the correct input is. This is now a supervised loss that we guide the training with. And what was I going to say? Yeah, so you take the generator output, you transform it into a spectrogram. You take the real output, transform it into a spectrogram, compare the L one loss. Now, you sort of run into the same problem in that if these are completely not aligned, then this is not going to work. But since you have a supervised loss, this it can it gives you a much stronger learning signal of what the generator should produce. So you're kind of counting at the beginning of training, you're counting on sort of a reverse reverse learning process in that the real the real sound will go into a spectrogram. And the generator will go here. And then that learning signal will sort of travel to make the generator produce more of whatever the real sound is. And that almost like if you think that the aligner is so bad that we have even non overlapping fragments, basically you teach the generator to ignore the inputs that it gets from down here, that it gets from its entire backbone. You teach it to sort of ignore all of that. If if that makes any sense. It simply produces the sound according to this supervised loss. Now, of course, it doesn't ignore it. It still takes the features, but it ignores the this whole alignment thing. And now once the generator gets a better signal of what it should produce, that signal can travel back to the aligner module to this length estimation module. And guide that one to make better predictions about the lengths. Okay, so that's how you at the beginning of training, you sort of rely on this path of learning to make to initialize this module of the aligner. And then once these length predictors are better, then the the loss can travel in its intended path where you forward produce these aligned sound waves. And then these discriminators take over. I don't exactly know if they trade this off during training or they simply set it to a number such that it helps them at the beginning. But it's a it's a good idea. And it's a it's a good trick to introduce here a supervised portion to make the beginning easier. But of course, you'd run into the same problem as I said, and that the fact that if you have two spectrograms, they not don't necessarily align again. And here they use this dynamic time warping loss. Now, this looks very, very similar to the aligner, but it is something different. Because now you have to the difference here is you have two things that you know should match. Right. You have this thing and you have this thing and they both have the same amount of entries. So they both have a, b, c, d, e. This has an a, a b, a c, a d and an e slot. And this also has an a, a b, a c, a d and an e slot. And you know that you assume so here is something you assume you assume that the beginning and the ends match. This is not true, of course, because we could have completely unaligned. But they say in practice, this works. So you assume that sort of at least a little bit. These are aligned. Right. So they have, by the way, there's so much to this paper, by the way, they have an auxiliary loss where the produced lengths, all the lengths that the this length prediction module produces, they I don't remember where that is, but they have an auxiliary loss where all the lengths must add up right here. All the lengths that these length predictors must add up to the total length of the sound, which in our case, I guess, is the two seconds. OK, so that's how they if so really quickly, these length predictions will sort of at least the least thing they can do is they can all predict like L over N. And that will give you a sort of a rough alignment such that it it kind of makes sense to to do this dynamic time warping to assume that the beginnings and the endings align. All right, so we have two things with they have the same amount of of slots. We know the beginnings and ends align or we assume that. How do we make it? How do we find out which slots align to which? And this is a dynamic programming. They formulate this as a dynamic programming problem that you might, you know, from you might know from from like these are often taught in algorithms and data structure courses and so on, where you you can figure out which of these align. So if you go a step here, that means that you go one step in each in each of the sequences. And then if you go a step here, that means only this one advances and this one still corresponds to this one right here. And OK, I formulated wrong at the beginning. You don't have ABCDE. I guess you would actually have all of these slots and you would figure out which ones correspond to which. And we have the same problem here. And we have the same problem again, where we have a different selection. Yeah, but I hope you recognize these sort of problems where and the here you align them again. So these are classic dynamic programming alignment problems and they align it like this. And this is a larger penalty we give. So they give a penalty with respect to how much this path deviates. So here you can see how much the spectrogram of the generated the generated sound aligns with the spectrogram of the ground truth. And here is a penalty for each time that the two spectrograms don't align correctly. So they align in a soft way. So they do every single possible path right here. And you can again do this using dynamic programming. And the entire catch here is that the alignment must be monotonic because no matter how long or short the sequences are, they always follow one after another in both of the spectrograms and both of the sounds. So that's why you can optimize it in a way. So over all the possible paths that you can align them, you weigh these paths by their score that you give them here. And then you calculate the loss across all these different paths. And that will give you that is sort of a fuzzy loss. So you don't compare the spectrograms directly, but you compare them and you sort of forgive them for not aligning too well. But the more they don't align, you give a penalty. And that's how you sort of force the generator. Again, you force the generator to produce things that are aligned. You produce these length predictions that make the spectrograms closer to each other. So that's how you calculate the spectrogram loss. This is entirely deterministic. There's no learned weights right here. Okay, cool. Last thing they say is that they use this phony miser. That's the very beginning, but they also ablate that. So in the results, they do a lot, lot of ablation studies, which I don't want to go into right now. I've already shown you some. And they do a even I think they do a human evaluation. Do they do a human evaluation? I know this might have been in another paper. But as you have heard from the examples, this sounds extremely realistic. I'll link the website to the samples in the in in the video description for sure. So I think we've gone over everything. The generator starts off with text, puts that into normalized text, calculates hidden features right here. These hidden features on one hand are used to predict the lengths of each of the tokens in the sound and are also used to as an input to the generator here. Now, they can only be used as an input to the generator if the generator knows how to align them in time and how to align them in time is predicted from these predicted lengths right here via this aligner algorithm. This is the lengths are the only thing that is predicted. Everything then is deterministic. The aligner is simply a Gaussian kernel over the predicted locations on the on the time axis. It is so the Gaussian kernel is to make it to make this alignment a bit fuzzy to make this prediction fuzzy. You perform a weighted sum with these features and then the generator knows where to put the feet where to put the tokens. Finally, the generator can up sample the token, the now aligned tokens into sound. This goes into the discriminator. The discriminator is actually five different discriminators, which try each try to discriminate the original from the real. Sorry, the generated from the real at different time scales. In addition to that, you have a discriminator on the spectrograms and you also have an L1 loss on the spectrograms, which helps especially at the beginning of training for the L1 loss of the spectrograms. You have to again compute an alignment, but you do this in a deterministic way by this thing down here. This dynamic time warping where you simply assume that they are aligned and forgive them for not being aligned with a with a a soft penalty and not a hard hard zero score. All right, this was the paper. Again, if you like this, leave a like a comment, share it out, subscribe and have a good day. Bye bye.
 In this work, we take on the challenging task of learning to synthesize speech from normalized text or phonemes in an entwined manner, resulting in models which operate directly on character or phoneme input sequences and produce raw speech audio outputs.
 Okay, that wasn't the real model. I just thought it sounded really funny. This is a text to speech model, and it actually sounds like this.
 In this work, we take on the challenging task of learning to synthesize speech from normalized text or phonemes in an entwined manner, resulting in models which operate directly on character or phoneme input sequences and produce raw speech audio outputs.
 Okay, now you've probably, if you have listened to the text and not just how the text sounds, you have gotten what this paper is about. So the paper is called End to End Adversarial Text to Speech by Jeff Donahue, Sander Dieleman, Mikolai Binkowski, Eric Elson and Karen Simonian of, I believe, of mostly deep mind.
 And this paper on a high level, it produces speech, so the sound waves of speech directly from text or from what they call normalized text or phoneme text.
 And it does so without any intermediate supervised representations. And that's a challenging task. And the main problems here are the alignment problem that they have to solve and actually making this work in an adversarial manner.
 So we're going to look at this paper. As always, if you like work like this, consider subscribing and sharing it out. And if you have any comments, leave them in the comment section.
 Okay, so what's the problem with text to speech? Text to speech is basically you take a piece of text like this one, modern text to speech synthesis pipelines typically involve blah, blah, blah.
 And you want to make a model that takes this and outputs sound waves, as if a human would say it right so you do modern text to speech and so on.
 Now you have multiple problems when doing this. First of all, the text here is words. Let's say we can tokenize the text into words.
 So you have modern text to speech. Those are four tokens. However, sound waves are, of course, much, much densely sampled.
 So these sound waves, they are typically in the order of something like 24 kilohertz sampled. So that's the the ratio of one token to output samples is super high.
 So one token will produce many, many thousand samples in the speech. So that's the first problem.
 The second problem is that if you have training data, so you have data that has a piece of text and you have the sound wave that a human you know the human read that particular piece of text,
 you still don't know which word exactly corresponds to which portion of that text. You simply know the entire text corresponds to the entire sound wave.
 You don't know this word text right here. You don't know. You don't know where it starts and where it ends in this sound wave.
 And the last problem you obviously have is that you want to make this in a way that it generalizes, that it sounds like a human, but also generalizes to some other text.
 And this paper here solves all of these problems jointly by doing an adversarial approach to learning.
 And it does it end to end. Now adversarial simply means that you have a generator that takes in the piece of text right here and generates this sound wave.
 And then you have a discriminator that looks at this sound wave and it looks at the real sound wave. So the real, okay, the real sense.
 So let's say this over here is real. And this is what the generator has produced. The discriminator tries to discriminate between the two.
 Now this is not entirely the same thing as a supervised loss. Usually in a GAN, you do not have the corresponding samples, right?
 You simply input a real sample here and the generator produces a generated sample here.
 You do not necessarily have in a classic GAN the corresponding sample. Here you assume that you have the corresponding real sample,
 but it's still different than supervised learning in that both go through a discriminator and the discriminator tries to tell, the discriminator is a neural network,
 tries to tell which one is real and which one is generated. In fact, the discriminator is a set of neural networks. We're going to go into that shortly.
 So it's adversarial in the sense that there is a generator and a discriminator. And it is end to end in the sense that usually what these pipelines do is they take the text.
 And we've looked at this, for example, in the video about this Facebook's text to speech system. So they take the text.
 And the first thing they do is they would produce a set of whatever they would call features like textual features.
 So these are these are sort of intermediate features for the text to be produced.
 And then another model would take these and it would produce something like spectrograms spectrograms.
 And then another model would take the spectrograms and finally produce sound or speech.
 So you have usually in these systems, you have intermediate representation.
 And each of these models right here can be trained by itself. So that's an advantage that you can train.
 For example, you can train a model that goes from a spectrogram to a sound wave by itself. And you don't need you simply need sound for that.
 The computation from sound to spectrogram is super easy. So you can generate your own training data.
 So you can go from spectrogram to sound. You can train a model like this.
 So in these pipelines, usually there are multiple stages and each of these models has to be trained by itself.
 This paper tries to do this end to end. That means you input the text and you get out the you get out the sound wave.
 And and there is nothing in between that the I mean, of course, there are latent representations, but you train it ends to end in one go.
 Okay, so let's look at the different systems they employ.
 First of all, let's look at the discriminators, because that's the easiest.
 So they have these discriminators and these are adopted from this GAN TTS paper.
 Now, as we already said, the discriminator try to to differentiate between real and fake sound.
 And they do it in a sort of. So if they were to just look at the entire sound wave, then it would just basically reduce to comparing the two.
 But instead, the discriminators, they operate on very small windows.
 So in specific, they have five different discriminators and each of the five different discriminators take a different size window length.
 But all of these windows are super short. So one discriminator might take this long windows.
 Another discriminator might take a bit longer windows and another discriminator might take a bit shorter windows.
 And that, of course, from the real and from the fake and the discriminators simply try to discriminate only in these windows, whether it's real or fake.
 And now we're a bit more into the GAN setting where, you know, you simply have one data point of the real and one data point of the fake.
 And you have to compare them. And this here, I believe, is one of the keys why this model generalizes, because the discriminators, basically, they try to assess
 whether a short sequence sounds like real or sounds like fake and whether the two samples sound alike in different scales of time.
 And by only doing this on these short scales, you can you can the loss sort of generalizes.
 Otherwise, if you compare it on the entire sound wave, it would just reduce to comparing point like point by point right here.
 And if the generator produces something that's not exactly aligned, then, of course, every point would be wrong.
 And you'd run into all sorts of questions. So this is a set of five discriminators that all try to take each takes a different length of sub sub sound of this wave tries to discriminate real from fake.
 That's the discriminator loss. They have an additional discriminator loss where they compute spectrograms of these things.
 So spectrograms and compute spectrogram of this.
 And they have a discriminator, another neural network here that it tries to distinguish which one is real and which one is fake.
 Note that this is not the same as down here where the spectrogram is an intermediate representation.
 Here, the sound is the output and from the sound you compute the spectrogram and then you compare the two.
 So this is simply a different spectrogram is a different feature space for the discriminator to compute the loss.
 It is not an intermediate representation on the way to produce the sound itself so that the difference.
 That's the difference here between the classic approach and this approach.
 So it's end to end adversarial. So we got the discriminator.
 The discriminators simply try to differentiate the sound waves and short scales as well as the spectrograms.
 Now, the second part is, of course, the generator. How do we even produce sound?
 And that's this diagram right here.
 So you have this Gantt generator. This is a generator that takes in a hidden representation.
 It takes in tokens. Let's say token one, token two.
 Let's go from the one before. Think of a sentence. Hello there.
 There. OK, so it takes these tokens and of course it takes like hidden representations of the tokens.
 And it will output. It will output for each one or for this joint sequence.
 One hidden, two. It would output the sound wave. OK.
 And this has been a paper before, the Gantt TTS. And you also you condition it on the speaker and on latent variables like how you want the pitch to be and so on.
 That's not really important for us right here. The generator can simply take these token embeddings and produce sound.
 The problem is in the original paper, you had an alignment. You knew which token corresponded to which piece of sound.
 And therefore you sort of knew. So after that, you need to compare this to the generator thing.
 And you knew which token corresponded to which piece of sound right here.
 So the generator knew what it had to produce from each token, how long it should be and so on.
 So this is the generally the alignment problem or what I call the alignment problem.
 So if you take a piece of text like this entire paragraph right here.
 Let's look at this paragraph. This paragraph to read it out takes like 30 to 60 seconds.
 You can't train really models that output this long of sound. It would be too big of a sample.
 You want to train ideally on segments. They train segments that are, I believe here, two second windows from each examples.
 So because they say if we train on 20 seconds, that would just be wasteful and prohibitively expensive.
 Now, the problem, of course, is if I simply take a window here like this one of two seconds and I have my human that has read this entire paragraph in one go,
 I have no clue again which part of the entire sound wave of this paragraph this subsequence corresponds to.
 A good guess would be to go like, well, this is about 50 percent in. So maybe here to here. Maybe. Who knows.
 And even within that, and that's what we discussed before, you have no clue how long this word here is going to take up within this piece of sound.
 And that's the general alignment problem right here. So in this entire sound wave, where is the piece and how do these words distribute across the wave of sound?
 The original model had, as I understand it, such alignments and therefore this generator could work really well because you had these alignments without these alignments.
 It doesn't work as well. And you can on their website that I've shown you initially, you can listen to samples where they disable each of these things.
 So this generator is really good at producing sound when it has these alignments.
 So the challenging task here is how do you compute these alignments?
 How do you compute this thing?
 If you don't have it, if you're if you don't have it in your training data, so it needs to be part of the loss.
 So that's what this entire architecture down here is.
 So the text is down here.
 It goes in. And the first thing they do is they normalize the text and they transform it into phonemes, which is you can do this in a deterministic fashion.
 There are scripts that do this.
 This is the only preprocessing they do.
 And they can also leave it away in the heaven ablation on their website.
 So this is like phoneme text. Cat sat on the mat.
 Now, this phoneme text goes through these big block of convolutions and of dilated convolutions.
 And this outputs a 200 hertz representation token length.
 Alignment. OK, I should specify this.
 So for each token here, it outputs a length.
 So this thing predicts the length of each of the tokens.
 These all of this all of this thing here is to to embed the tokens in hidden space and then predict its length.
 You can see that right here.
 OK, so first we use F to to take X.
 So F is a stack of dilated convolutions and it takes X and outputs a hidden representation.
 So H. So first X goes to H and then H is used to predict the length of each of the tokens.
 H is used to predict L and L is the length of that token.
 So we embed this into a hidden representation with this right here.
 And then we use this stack to predict the length of each token.
 So this could be this could say something like this cat token right here is 20 milliseconds long.
 Or instead of milliseconds, you would use something like frames or data points.
 Maybe this is 200 data points long.
 And then SAT is a bit shorter. So this is 100 long.
 And ON is really short. So this is 50 long.
 So for each token, it predicts the length.
 All right. So now if we have the length of each, we can sort of calculate where the starting point is.
 So if we want to know if we know that here is the beginning and the beginning of the sentence,
 we we conservatively assume that there is so they give some silence buffer here.
 But roughly, you can assume that the beginning of the speech corresponds to the first the first token.
 Right. You can simply trace the waveform.
 And whenever it goes up, that's where the first token starts.
 And then since we know that's where the first token starts,
 and if we could predict the length of each one correctly,
 we could simply sum those up to figure out where our word starts.
 So if we want to know where on starts, we simply go from the beginning and go 200 plus 100 milliseconds.
 So our data points 200 plus 100. Here is where on starts.
 OK. And if we want to figure out the middle of on, we simply add the half of this number.
 So plus 25 gets you to the middle.
 So this is this here is the center of the token on.
 So for each token, we predict the length like this.
 And thereby, we can just calculate for each one by summing up from the beginning
 and then adding half of its own length where the center of that token in the entire sequence is.
 And now we do this. We said we take random two second audio,
 but we do this procedure for the entire for the entire text.
 OK, for the for every single token in the text that we look at in the 20 second text,
 we do this because then for each token, we'll get a token center.
 And now the aligners job here is to align that to the actual sound.
 So what we also give the generators here, the offset.
 So let's say we have this 20 second of speech and we randomly sampled these two seconds.
 And that's maybe five seconds from the beginning.
 We also tell it this is five seconds right here.
 So what we can now do is we can calculate back sort of and say, OK, here I have
 I first need to discard five seconds of my signal and I have a prediction how long each token is.
 So I can just cross out tokens until I have basically wasted five seconds.
 And then I know, OK, from here to wherever these things sum up to two seconds from here to that.
 Those are my two seconds that I want to look at.
 Now, this is how I figure out where in the big sound wave my fragment is.
 Because I have this offset where I sampled it and I simply add use this and the predicted lengths to figure it out.
 I still need to figure out these tokens that are actually in the span.
 How do they distribute? And that's what this aligner here does.
 Since we've already predicted the token centers, we simply assume that if these are correct, right,
 then if this is, let's say if this is one second long, I assume that the middle is after point five seconds.
 So this is one second. The middle is point five seconds.
 So I think that this token is aligned right here.
 This is the center of the token.
 Now, we want to be a little bit a little bit fuzzy with respect to that.
 So what they do is they sort of use a Gaussian kernel right here.
 So for each token, as you can see here, each token has a center, which is here.
 So the y axis is the time in sound and the x axis is the token.
 And for each token, we say, well, it doesn't have to be exactly there.
 It can be so they put a Gaussian kernel like this.
 OK, if you imagine this kernel popping out of the frame, they say this is about where the center is.
 And for this token, right for this token right here, they say, well, it's it's probably here in the middle,
 but it could also be here or here or here or here.
 And we weigh this like this.
 So these are these are the weights.
 And then you simply sum up the weights with these embeddings.
 So for each token out of this dilated convolution block, you get a hidden embedding.
 And by using this alignment matrix that you computed by predicting the lengths
 and therefore predicting the centers of the tokens, you can then sort of shift.
 So first, you assume that h1, h2, h3, if you were to do nothing,
 these would just all take up like a third of the time.
 And now by multiplying with this matrix, you have the opportunity because you predicted a longer length for the first token.
 You have the opportunity to shift that a bit to the right and maybe shorten the second token a bit.
 And then the third token goes until the end.
 OK, that's what this aligner thing is.
 This is not a model by itself.
 All that this takes in is the computation right here of the token lengths.
 This estimates these token lengths for each of the tokens.
 And the rest is deterministic.
 It's simply saying, OK, how much is the offset?
 Cool.
 That's how we know where in the sound wave we are.
 And then where is each of the centers?
 And we simply do that by summing up the predicted token lengths.
 And then we use a Gaussian kernel with like a set hyperparameter to be a little bit fuzzy with respect to these lengths right here.
 So to be differentiable, basically.
 And that will that will ultimately train this loss, this model right here that computes the token lengths.
 Right. So we sum up in a weighted fashion these embeddings right here.
 And that's what goes into the generator.
 So now we have embeddings and we have the alignments for the embeddings, which are these pieces of where in the sound wave these are.
 And from that, the generator can now produce the sound wave itself.
 OK.
 And that's basically that's just an up sampling here.
 I think that's just an up convolution up sampling from 200 hertz signal to a 24 kilohertz signal.
 Cool.
 So that's that.
 Now they discover this doesn't work.
 And why doesn't it work?
 It's because at the beginning of training, these token length predictions here are pretty crappy.
 And so that means that I guess especially this part, even where you say, well, where where in the sound wave of my 20 seconds do I even need to cut out to compare with the discriminator?
 Right.
 So if you give if you sample this piece here and that's what you give to the discriminator, but your length predictions are so far off that the generator is trying to produce this particular piece because it thinks it thinks, oh, instead of producing this token here, which is what the discriminator looks at, it produces these tokens here.
 Of course, you have no chance, no matter how good your adversarial loss is.
 Remember, the this is these length predictions are used to see basically to see which of these tokens the generator needs to produce the sound for and how they're aligned.
 So they have an additional loss right here.
 What they do is they produce from the again, they go via the spectrograms within this spectrogram prediction loss.
 So they say we discovered that adversarial feedback is insufficient to learn alignment.
 At the start of training, the aligner does not produce an accurate alignment.
 So the information in the input tokens is incorrectly temporally distributed.
 This encourages the decoder to ignore the aligner output.
 The unconditional discriminators provide no useful signal to correct this.
 Oh, yeah, I should have mentioned this.
 The discriminators here, since you don't know, you don't know which tokens you should produce.
 The discriminators are unconditional.
 They don't know which text is produced.
 You don't give them the tokens.
 You simply give them the sound waves.
 That's something I find particularly interesting here.
 Now, you of course, this wouldn't work in a like a traditional again, because you simply have a data sample here and a data sample right here.
 But in this case, you of course have the corresponding sound samples.
 But still, they are, you know, they are cut down to a subsequence.
 So you don't know which text you're producing.
 So you have to make the discriminators unconditional.
 And therefore, they are going to discriminate, as we said, between potentially between two completely non overlapping pieces of the sound wave, which, of course, doesn't help you.
 And then the aligner can also not learn anything because there is no learning signal because everything just says this is not the same.
 OK.
 And that's what they say here.
 We face a different problem.
 We do not have aligned ground truth.
 Conditional discriminators, which they don't have, need an aligner module, which cannot function correctly at the start of training, effectively turning them into unconditional discriminators.
 So even if they were to input the text, it would still be the wrong text because their aligner is wrong at the beginning.
 Although it should be possible in theory to train the discriminators aligner module adversarially, we find that this does not work in practice and training gets stuck.
 So what do they do?
 They say instead we propose to guide learning by using an explicit prediction loss in the spectrogram domain.
 We minimize the L one loss between the log scale male spectrograms of the generator output and the corresponding ground truth training window.
 This helps learning to take off and renders conditional discriminators unnecessary, simplifying the model.
 So they take the they take the spectrogram of the generator output and the corresponding ground truth training window and they simply calculate the L one difference of the spectrograms.
 Now this, as I understand it, this is different from this is different from because we said they also have a discriminator on the spectrograms.
 This is different from that.
 This is even in addition to that.
 So here somewhere we had this.
 This was the discriminator on the spectrograms.
 And I think this is even different.
 So what they're doing is they also the discriminator simply decides do the spectrograms look real or fake?
 Does the spectrogram look real or fake?
 Now they also take the spectrograms and compare them with the L one loss.
 So this is exactly what they said they wouldn't do right here.
 Now it's still the case, right?
 It's still the case that they don't use spectrograms as intermediate representations, but they now do have a supervised loss on the spectrograms.
 And one of the motivations to do this end to end is saying, you know, maybe these auxiliary losses and supervised losses, they sort of distract.
 They're good to guide the training, but they sort of distract.
 And now they see, OK, maybe we have to introduce this one right here in order to make the training start, because this is a real signal.
 But again, you run into a problem, namely, if you produce something with the generator.
 And so first of all, this is not a discriminator anymore.
 This is a true L one loss.
 So we potentially run into this problem, right?
 Of the of the generator simply copying the input because you always tell it what the correct input is.
 This is now a supervised loss that we guide the training with.
 And what was I going to say?
 Yeah, so you take the generator output, you transform it into a spectrogram.
 You take the real output, transform it into a spectrogram, compare the L one loss.
 Now, you sort of run into the same problem in that if these are completely not aligned, then this is not going to work.
 But since you have a supervised loss, this it can it gives you a much stronger learning signal of what the generator should produce.
 So you're kind of counting at the beginning of training, you're counting on sort of a reverse reverse learning process in that the real the real sound will go into a spectrogram.
 And the generator will go here. And then that learning signal will sort of travel to make the generator produce more of whatever the real sound is.
 And that almost like if you think that the aligner is so bad that we have even non overlapping fragments, basically you teach the generator to ignore the inputs that it gets from down here, that it gets from its entire backbone.
 You teach it to sort of ignore all of that. If if that makes any sense.
 It simply produces the sound according to this supervised loss.
 Now, of course, it doesn't ignore it. It still takes the features, but it ignores the this whole alignment thing.
 And now once the generator gets a better signal of what it should produce, that signal can travel back to the aligner module to this length estimation module.
 And guide that one to make better predictions about the lengths.
 Okay, so that's how you at the beginning of training, you sort of rely on this path of learning to make to initialize this module of the aligner.
 And then once these length predictors are better, then the the loss can travel in its intended path where you forward produce these aligned sound waves.
 And then these discriminators take over. I don't exactly know if they trade this off during training or they simply set it to a number such that it helps them at the beginning.
 But it's a it's a good idea. And it's a it's a good trick to introduce here a supervised portion to make the beginning easier.
 But of course, you'd run into the same problem as I said, and that the fact that if you have two spectrograms, they not don't necessarily align again.
 And here they use this dynamic time warping loss. Now, this looks very, very similar to the aligner, but it is something different.
 Because now you have to the difference here is you have two things that you know should match.
 Right. You have this thing and you have this thing and they both have the same amount of entries.
 So they both have a, b, c, d, e. This has an a, a b, a c, a d and an e slot.
 And this also has an a, a b, a c, a d and an e slot.
 And you know that you assume so here is something you assume you assume that the beginning and the ends match.
 This is not true, of course, because we could have completely unaligned. But they say in practice, this works.
 So you assume that sort of at least a little bit. These are aligned.
 Right. So they have, by the way, there's so much to this paper, by the way, they have an auxiliary loss where the produced lengths,
 all the lengths that the this length prediction module produces, they I don't remember where that is,
 but they have an auxiliary loss where all the lengths must add up right here.
 All the lengths that these length predictors must add up to the total length of the sound, which in our case, I guess, is the two seconds.
 OK, so that's how they if so really quickly, these length predictions will sort of at least the least thing they can do is they can all predict like L over N.
 And that will give you a sort of a rough alignment such that it it kind of makes sense to to do this dynamic time warping to assume that the beginnings and the endings align.
 All right, so we have two things with they have the same amount of of slots.
 We know the beginnings and ends align or we assume that.
 How do we make it? How do we find out which slots align to which?
 And this is a dynamic programming.
 They formulate this as a dynamic programming problem that you might, you know, from you might know from from like these are often taught in algorithms and data structure courses and so on,
 where you you can figure out which of these align.
 So if you go a step here, that means that you go one step in each in each of the sequences. And then if you go a step here, that means only this one advances and this one still corresponds to this one right here.
 And OK, I formulated wrong at the beginning.
 You don't have ABCDE.
 I guess you would actually have all of these slots and you would figure out which ones correspond to which.
 And we have the same problem here.
 And we have the same problem again, where we have a different selection.
 Yeah, but I hope you recognize these sort of problems where and the here you align them again.
 So these are classic dynamic programming alignment problems and they align it like this.
 And this is a larger penalty we give. So they give a penalty with respect to how much this path deviates.
 So here you can see how much the spectrogram of the generated the generated sound aligns with the spectrogram of the ground truth.
 And here is a penalty for each time that the two spectrograms don't align correctly.
 So they align in a soft way. So they do every single possible path right here.
 And you can again do this using dynamic programming.
 And the entire catch here is that the alignment must be monotonic because no matter how long or short the sequences are,
 they always follow one after another in both of the spectrograms and both of the sounds.
 So that's why you can optimize it in a way.
 So over all the possible paths that you can align them, you weigh these paths by their score that you give them here.
 And then you calculate the loss across all these different paths.
 And that will give you that is sort of a fuzzy loss.
 So you don't compare the spectrograms directly, but you compare them and you sort of forgive them for not aligning too well.
 But the more they don't align, you give a penalty.
 And that's how you sort of force the generator.
 Again, you force the generator to produce things that are aligned.
 You produce these length predictions that make the spectrograms closer to each other.
 So that's how you calculate the spectrogram loss.
 This is entirely deterministic.
 There's no learned weights right here.
 Okay, cool.
 Last thing they say is that they use this phony miser.
 That's the very beginning, but they also ablate that.
 So in the results, they do a lot, lot of ablation studies, which I don't want to go into right now.
 I've already shown you some.
 And they do a even I think they do a human evaluation.
 Do they do a human evaluation?
 I know this might have been in another paper.
 But as you have heard from the examples, this sounds extremely realistic.
 I'll link the website to the samples in the in in the video description for sure.
 So I think we've gone over everything.
 The generator starts off with text, puts that into normalized text, calculates hidden features right here.
 These hidden features on one hand are used to predict the lengths of each of the tokens in the sound
 and are also used to as an input to the generator here.
 Now, they can only be used as an input to the generator if the generator knows how to align them in time
 and how to align them in time is predicted from these predicted lengths right here via this aligner algorithm.
 This is the lengths are the only thing that is predicted.
 Everything then is deterministic.
 The aligner is simply a Gaussian kernel over the predicted locations on the on the time axis.
 It is so the Gaussian kernel is to make it to make this alignment a bit fuzzy to make this prediction fuzzy.
 You perform a weighted sum with these features and then the generator knows where to put the feet where to put the tokens.
 Finally, the generator can up sample the token, the now aligned tokens into sound.
 This goes into the discriminator.
 The discriminator is actually five different discriminators, which try each try to discriminate the original from the real.
 Sorry, the generated from the real at different time scales.
 In addition to that, you have a discriminator on the spectrograms and you also have an L1 loss on the spectrograms,
 which helps especially at the beginning of training for the L1 loss of the spectrograms.
 You have to again compute an alignment, but you do this in a deterministic way by this thing down here.
 This dynamic time warping where you simply assume that they are aligned and forgive them for not being aligned with a with a
 a soft penalty and not a hard hard zero score.
 All right, this was the paper.
 Again, if you like this, leave a like a comment, share it out, subscribe and have a good day.
 Bye bye.