Blockwise Parallel Decoding for Deep Autoregressive Models
machine learning deep learning transformers nlp natural language processing ai artificial intelligence google brain autoregressive greedy decoding inference language model speedup
https://arxiv.org/abs/1811.03115

Abstract:
Deep autoregressive sequence-to-sequence models have demonstrated impressive performance across a wide variety of tasks in recent years. While common architecture classes such as recurrent, convolutional, and self-attention networks make different trade-offs between the amount of computation needed per layer and the length of the critical path at training time, generation still remains an inherently sequential process. To overcome this limitation, we propose a novel blockwise parallel decoding scheme in which we make predictions for multiple time steps in parallel then back off to the longest prefix validated by a scoring model. This allows for substantial theoretical improvements in generation speed when applied to architectures that can process output sequences in parallel. We verify our approach empirically through a series of experiments using state-of-the-art self-attention models for machine translation and image super-resolution, achieving iteration reductions of up to 2x over a baseline greedy decoder with no loss in quality, or up to 7x in exchange for a slight decrease in performance. In terms of wall-clock time, our fastest models exhibit real-time speedups of up to 4x over standard greedy decoding.

Authors: Mitchell Stern, Noam Shazeer, Jakob Uszkoreit
 Hi there, today we'll look at blockwise parallel decoding for deep autoregressive models by Mitchell Stern, Noam Shazir and Jakob Uschkordei of UC Berkeley and Google Brain. So this is a bit more of an engineering paper than usual, which I find cool. It's basically an engineering trick to get these autoregressive models to decode faster, while you can either preserve fully their performance or suffer a bit of a drop in performance, while even speeding them up more. Alright, so let's dive in actually. The paper starts out with a description of what autoregressive models are and what decoding is in them. So let me try to quickly explain this. So what is an autoregressive model? So basically we're talking about, let's say, language models. So language models are the classic examples of these models, where you have a language model is a model that simply predicts the next word in a sequence. So you could have something like a cat sits on the, and then here is blank. So the language model is asked to predict which word is the word that follows. The language model basically does this by predicting the probability distribution over the next word. So w t plus one, if this is t here, this is t minus one and so on, given all the w's smaller or equal than t. So all the words that come before should lead to the next word being predicted. So the language model is tasked to ask what is the next word in the sequence, or what's the probability distribution over the next word. And then you can simply, you know, pick the maximum probability word or something like this. So that's that's pretty standard so far. So what is the autoregressive part in here? So basically the autoregressive part means that in order for me to find this word here, this next word, I will look at all of these words here. And what does it mean then when I want to use this language model for generating generating a sentence, let's say, so I'm now I've trained the language model, it's really good at predicting the next word, I want to actually use it to do something more interesting. So I, I want it to generate a full sentence, what I do, let's say I pick the first word, the right, I pick the first word, and I simply ask the language model, why what's the next word? Right? And the language model can do this, it can assess what's the probability distribution here over words, and it will, for example, give me some some distribution over words, and I pick the maximum one, I say, okay, the maximum one here is house. Okay, the house. The house. And then I go back and I ask the language model, well, what's the next word then? See, clearly, you're a language model. So you can give me based on the two previous words, you can give me the next word, what's the next word, and the language model will maybe say the house is, and so on. So you can see how you can generate a sentence by simply basically feeding the answer that the language model gives feeding it into the next step of predicting. So all of these now go into the next step, and once you've predicted the next step, the house is on. Once you've predicted that, then you can take that and in conjunction with everything you've predicted so far, to predict the next step. So you can use the language model that is trained to predict the next word to predict an entire sentence and the autoregressive part basically means that its own predictions will serve as the basis for the next predictions, and this introduces a fundamental problem, namely that I have to basically wait for one prediction, so I have to wait here for is before I can predict on, and this means if I have a, I basically can't help but, so if this is my language model, it's a box, I can't help but go to the language model, wait for a response, okay, then go to the language model again, wait for a response again. This is inherently sequential nature here where I have to do like M steps if M is the length of the sentence that I want, and we can't make use of batching normally, so usually what you do during training, during training you have a whole bunch of data, right, you have the cat sits on the mat, you have the house, the house is blue, so I can generate, just from these two sentences I can generate a bunch of training examples, I can ask, this is a training example where the input is the cat and it's meaning to predict sits, then this is a training example where the input is the cat sits and the language model has to predict on, this here is a training example, this, this is a training example, so I can chunk this up into a whole bunch of training examples and all of those I can write, I can feed in parallel into a big matrix, I can all put them here and then run this thing through my language model in training mode because each of them is already like is in the corpus, I can batch the training but I can't batch the prediction because of what we've seen before because inherently the next predicting the next word depends on the last word that the model itself has output, so there is no training corpus around since we're not training, yeah, so this is the fundamental problem and these authors tackle this problem, they say how can we make this faster, this decoding, so they introduce greedy decoding here where they say okay, this is what we've just seen, the probability of the next word is like the maximum, the maximum log probability here in that case if the model predicts a log probability over the words that we've input so far, right, and this X here is, so this is for example a translation task, a machine translation task, so the X would be the source language sentence, so maybe like a French sentence and the Y smaller equal to J would be the so far decoded English sentence if we're trying to translate to English and the Y J plus one would be the next word that we're trying to predict in the English sentence given the English sentence so far and the French sentence, the total French sentence, so greedy decoding just does this one step after another and we try to go to what they call blockwise parallel decoding. So we can just jump to the graphics straight away because what they do is pretty straightforward and is best illustrated in this graphic actually, so they go from this situation where they already have this here, they have a saw a dog ride, this is the sentence that has been decoded so far and we have to try to complete it, naturally we'll ask what's the next word, but they say okay what if we could predict not only the next word from this but the word two positions away or three positions away, we could do this all at the same time, right, I mean I can certainly build a model, a language model that doesn't only predict the next word but predicts the word after that as well, though of course if then this word, the predictor for this word still only gets this as an input so this is the important thing here, so the part of the model that predicts the is two words away isn't being informed that this word is being produced here, so naturally you would expect the quality to be worse because the word one position away, two positions away and three positions away are each predicted basically independently of each other just from the source context, so there is no, you can't expect like a coherency between the words or not a lot, so this is the fundamental trade-off with such a model, you can predict farther into the future at the same time but then these predictions can't basically depend on each other and this degrades your performance quite a bit, so what these authors do is to remedy that, they say well these things here we can, I mean we can produce a bunch of them, right, since all that's required as an input is this, we can actually produce like, we can produce a batch of them at the same time, so we can produce one, two and three words into the future and we can do this like a hundred times in parallel, no problem, alright, and we can sample this, we don't have to always take the most likely word, we can actually sample a bunch into the future and this now gets smarter because now I have a list of one hundred basically suggestions of what the continuation here could be, right, I have, I take this not as a given but I take these outputs as suggestions, alright, and then I can have another model that, this is called verify here, I can have another model that scores all of these different, all of these different decodings in parallel, both of these can be done by the same model, we saw the language model can be either used to predict or to score something, since it inherently predicts the probability of sequences or of following words, we can, we can let it output this probability all in parallel, so this also can count as a score, what I'm trying to say is you can, since the language model is constructed as a, as outputting probabilities anyway, like such, we can use it both to predict the next word and also if we have a suggestion we can use it to score that and to say okay how likely is that, right, and then what we can make sure is that the suggestion, we are looking for the suggestion basically that has the highest score and if you want to be really true to your original model you say I want to look for the suggestion that has the maximum, that would have had the maximum score had I decoded one by one, so then basically you retain the original performance and you gain a speed up as long as the, what the greedy decoding would have produced is in your suggestion, in your box of suggestions that you produce, as long as that's in there you gain a speed up, if that's not in there then you can always, you always have the one word ahead model because that's, you have that anyway, you predict the next word anyway, so in case none of these suggestions work out you still have this one word prediction basically which is the model you started with, so at worst case you're as fast as the greedy model and in best case you always, your suggestions are so good that they are always the one that would have been decoded anyway, so you can basically in this case do three steps at once. Alright, so this verify step here is shown here and you see it will decode, now this is just one suggestion keep in mind, they can produce many suggestions at the same time if there's memory or and they can actually, they can score each of this, so they can score this, they can score this and they can score this also independently as a batch, so they can do this in parallel and here you see, yeah here is executed in parallel, so the model will go and will score this word in and say ah this would have been, this is the argmax of the greedy decoding anyway and it can also score this step and say aha given that there is an in that this the is the argmax anyway, right and you can score this step and say ah given that there's in the, the argmax would have been car, so that's not bus, so we reject this suggestion but we keep that part of the suggestion and say okay the in the is basically what would have been decoded anyway according to the greedy decoding, so we can basically accept this here and continue from there, this is the accept step here, so this basically, so you can see in this one step which yeah we'll call one decoding step, we have basically done two of the greedy decoding steps in one go, so by predicting into the future and then selecting the one that agrees with the original model because we can, the fundamental thing is we can score in parallel but we can greedily produce not in parallel, alright so they actually push this further by also eliminating one of the, one of the evaluations here by combining basically the next predict step with the previous verify step and it's pretty cool to look at that, so we're in the same situation, you have this and you suggest this continuation and then the score model again will go here but while you verify you also do the next predict at the same time, since you've built your model, since it's the same model and this model every time you execute it, it outputs a distribution over the next set of positions, you might as well take the outputs of it, right, so when you then decide to accept this here, you will already have the outputs computed for the next three positions, so this you can feed directly into this next predict step, you basically don't have to execute it, you simply go to the one you've accepted and then you look at the outputs that you get anyway from this model and use them, so you might ask, okay which, how does a model look that like scores and predicts into the future and this, the answer is here, it's a bit out of order, I would have maybe liked this more previously but in any case this is what they do, so they use a transformer architecture and you have to imagine it starts down here and actually there is a huge network down here, right, this is just the output layer, so there's a giant transformer network down below and it produces this output representation, now normally from this representation you would go to this what's called p layer here, this is a output vocabulary projection, so this has one entry for each of the words in your vocabulary, so the, a, cat and so on and you would then for each one predict a probability, so with this representation you basically project it onto this vocabulary and predict the probability distribution over the next word, but what they do is they say no no no we not only need the next word, we need the next three words, so let's actually split this output signal into three output signals and they do this by introducing this hidden feed forward layer here or a hidden transformer layer, it's a hidden layer, yeah we insert a single feed forward layer with hidden size, okay, so they insert a hidden layer and then they also add these skip connections here, right, they add the skip connections which basically just means they feed through this output directly to here and add it to that, so basically the feed forward layer needs to transform this output here into the vocabulary input, one step ahead, two steps ahead and three steps ahead and you can see here that those are independent, right, they don't depend on each other, there's nothing feeding back p1 here into the decision of p2 so they can be executed in parallel, but they lose the dependence on each other, alright, so that's the architecture and you can clearly see here it's able to predict three steps into the future at the same time, so yeah, alright so they also do different adjustments where they say now yeah we can also kind of sacrifice a bit of the fidelity to the original model by not requiring that the basically we don't only accept when the suggestion is the perfect best suggestion that would have been decoded by the greedy model, but what we could do is we could just if it's in the top k we could accept it, if it's in the if it's good enough basically one of the suggestions that we have is good enough then we'll accept it or when you have like some sort of distance metric they say here so the distance between our suggestion and the maximum so the what would have been best by the greedy should be smaller than some constant epsilon and that way you can sacrifice a bit of performance but your suggestions will be accepted much more often and thereby your speedup will be much higher and they also experiment with whether or not they should fine tune the original model along with their model and also the experiment with knowledge distillation where they basically have like some some teacher model and you train the your model on the output of the teacher model don't want to go too far into this since these are mostly kind of things to make it work even better and you can see here that this is for example a machine translation task so this is the WMT 2014 English German translation and there's a regular they get a blow score of 26 and here higher is better and if you can see they get a fairly sizable speedups by keeping the blow scores fairly constant so they they almost speed up by 2x but if they allow the blow scores to go down a bit they get a much higher speedup of like 3 and then if they do like distillation and fine tuning they actually manage to keep up the performance even though they get very very high speedups so they get speedups until like 5x by not dropping the blow scores very much so that's that's pretty impressive another experiment they do is image super resolution where you can see here with regular they try to really keep exactly the original model output and it doesn't it doesn't speed it up too much but when they allow for a bit of a mistake to be made so here this is image super resolution so values are between zero and 255 and they allow epsilon equals to two of that so that's that's kind of less than 1% error on the individual pixel then they get a speed ups of 7x or something like this and you can see in this region here that when the K is for in case the number of steps that you decode ahead so and the mini mean block size is 3.75 that means on average 3.75 steps ahead or accepted which means basically there their suggestions are almost always good enough to be accepted so they get this massive speed up by basically being able to jump these decoding steps yeah so they have a bunch of other results here there show their wall clock time speed up since iteration speed up as well but if you have to pay in huge computational cost it's not so good but they also show that they have a big kind of wall clock speed up up to up to 4x here in super resolution and over 3x in translation so it's a pretty cool paper they give some examples here a bunch of more tables some examples of their super resolution and yeah if this might be something for you then use it it's I think it's a pretty neat trick and yeah especially for production systems all right that was it bye bye.
 Hi there, today we'll look at blockwise parallel decoding for deep autoregressive models by
 Mitchell Stern, Noam Shazir and Jakob Uschkordei of UC Berkeley and Google Brain.
 So this is a bit more of an engineering paper than usual, which I find cool.
 It's basically an engineering trick to get these autoregressive models to decode faster,
 while you can either preserve fully their performance or suffer a bit of a drop in performance,
 while even speeding them up more.
 Alright, so let's dive in actually.
 The paper starts out with a description of what autoregressive models are and what decoding
 is in them.
 So let me try to quickly explain this.
 So what is an autoregressive model?
 So basically we're talking about, let's say, language models.
 So language models are the classic examples of these models, where you have a language
 model is a model that simply predicts the next word in a sequence.
 So you could have something like a cat sits on the, and then here is blank.
 So the language model is asked to predict which word is the word that follows.
 The language model basically does this by predicting the probability distribution over
 the next word.
 So w t plus one, if this is t here, this is t minus one and so on, given all the w's smaller
 or equal than t.
 So all the words that come before should lead to the next word being predicted.
 So the language model is tasked to ask what is the next word in the sequence, or what's
 the probability distribution over the next word.
 And then you can simply, you know, pick the maximum probability word or something like
 this.
 So that's that's pretty standard so far.
 So what is the autoregressive part in here?
 So basically the autoregressive part means that in order for me to find this word here,
 this next word, I will look at all of these words here.
 And what does it mean then when I want to use this language model for generating generating
 a sentence, let's say, so I'm now I've trained the language model, it's really good at predicting
 the next word, I want to actually use it to do something more interesting.
 So I, I want it to generate a full sentence, what I do, let's say I pick the first word,
 the right, I pick the first word, and I simply ask the language model, why what's the next
 word?
 Right?
 And the language model can do this, it can assess what's the probability distribution
 here over words, and it will, for example, give me some some distribution over words,
 and I pick the maximum one, I say, okay, the maximum one here is house.
 Okay, the house.
 The house.
 And then I go back and I ask the language model, well, what's the next word then?
 See, clearly, you're a language model.
 So you can give me based on the two previous words, you can give me the next word, what's
 the next word, and the language model will maybe say the house is, and so on.
 So you can see how you can generate a sentence by simply basically feeding the answer that
 the language model gives feeding it into the next step of predicting.
 So all of these now go into the next step, and once you've predicted the next step, the
 house is on.
 Once you've predicted that, then you can take that and in conjunction with everything you've
 predicted so far, to predict the next step.
 So you can use the language model that is trained to predict the next word to predict
 an entire sentence and the autoregressive part basically means that its own predictions
 will serve as the basis for the next predictions, and this introduces a fundamental problem,
 namely that I have to basically wait for one prediction, so I have to wait here for is
 before I can predict on, and this means if I have a, I basically can't help but, so if
 this is my language model, it's a box, I can't help but go to the language model, wait for
 a response, okay, then go to the language model again, wait for a response again.
 This is inherently sequential nature here where I have to do like M steps if M is the
 length of the sentence that I want, and we can't make use of batching normally, so usually
 what you do during training, during training you have a whole bunch of data, right, you
 have the cat sits on the mat, you have the house, the house is blue, so I can generate,
 just from these two sentences I can generate a bunch of training examples, I can ask, this
 is a training example where the input is the cat and it's meaning to predict sits, then
 this is a training example where the input is the cat sits and the language model has
 to predict on, this here is a training example, this, this is a training example, so I can
 chunk this up into a whole bunch of training examples and all of those I can write, I can
 feed in parallel into a big matrix, I can all put them here and then run this thing
 through my language model in training mode because each of them is already like is in
 the corpus, I can batch the training but I can't batch the prediction because of what
 we've seen before because inherently the next predicting the next word depends on the last
 word that the model itself has output, so there is no training corpus around since we're
 not training, yeah, so this is the fundamental problem and these authors tackle this problem,
 they say how can we make this faster, this decoding, so they introduce greedy decoding
 here where they say okay, this is what we've just seen, the probability of the next word
 is like the maximum, the maximum log probability here in that case if the model predicts a
 log probability over the words that we've input so far, right, and this X here is, so
 this is for example a translation task, a machine translation task, so the X would be
 the source language sentence, so maybe like a French sentence and the Y smaller equal
 to J would be the so far decoded English sentence if we're trying to translate to English and
 the Y J plus one would be the next word that we're trying to predict in the English sentence
 given the English sentence so far and the French sentence, the total French sentence,
 so greedy decoding just does this one step after another and we try to go to what they
 call blockwise parallel decoding.
 So we can just jump to the graphics straight away because what they do is pretty straightforward
 and is best illustrated in this graphic actually, so they go from this situation where they
 already have this here, they have a saw a dog ride, this is the sentence that has been
 decoded so far and we have to try to complete it, naturally we'll ask what's the next word,
 but they say okay what if we could predict not only the next word from this but the word
 two positions away or three positions away, we could do this all at the same time, right,
 I mean I can certainly build a model, a language model that doesn't only predict the next word
 but predicts the word after that as well, though of course if then this word, the predictor
 for this word still only gets this as an input so this is the important thing here, so the
 part of the model that predicts the is two words away isn't being informed that this
 word is being produced here, so naturally you would expect the quality to be worse because
 the word one position away, two positions away and three positions away are each predicted
 basically independently of each other just from the source context, so there is no, you
 can't expect like a coherency between the words or not a lot, so this is the fundamental
 trade-off with such a model, you can predict farther into the future at the same time but
 then these predictions can't basically depend on each other and this degrades your performance
 quite a bit, so what these authors do is to remedy that, they say well these things here
 we can, I mean we can produce a bunch of them, right, since all that's required as an input
 is this, we can actually produce like, we can produce a batch of them at the same time,
 so we can produce one, two and three words into the future and we can do this like a
 hundred times in parallel, no problem, alright, and we can sample this, we don't have to always
 take the most likely word, we can actually sample a bunch into the future and this now
 gets smarter because now I have a list of one hundred basically suggestions of what
 the continuation here could be, right, I have, I take this not as a given but I take these
 outputs as suggestions, alright, and then I can have another model that, this is called
 verify here, I can have another model that scores all of these different, all of these
 different decodings in parallel, both of these can be done by the same model, we saw the
 language model can be either used to predict or to score something, since it inherently
 predicts the probability of sequences or of following words, we can, we can let it output
 this probability all in parallel, so this also can count as a score, what I'm trying
 to say is you can, since the language model is constructed as a, as outputting probabilities
 anyway, like such, we can use it both to predict the next word and also if we have a suggestion
 we can use it to score that and to say okay how likely is that, right, and then what we
 can make sure is that the suggestion, we are looking for the suggestion basically that
 has the highest score and if you want to be really true to your original model you say
 I want to look for the suggestion that has the maximum, that would have had the maximum
 score had I decoded one by one, so then basically you retain the original performance and you
 gain a speed up as long as the, what the greedy decoding would have produced is in your suggestion,
 in your box of suggestions that you produce, as long as that's in there you gain a speed
 up, if that's not in there then you can always, you always have the one word ahead model because
 that's, you have that anyway, you predict the next word anyway, so in case none of these
 suggestions work out you still have this one word prediction basically which is the model
 you started with, so at worst case you're as fast as the greedy model and in best case
 you always, your suggestions are so good that they are always the one that would have been
 decoded anyway, so you can basically in this case do three steps at once. Alright, so this
 verify step here is shown here and you see it will decode, now this is just one suggestion
 keep in mind, they can produce many suggestions at the same time if there's memory or and
 they can actually, they can score each of this, so they can score this, they can score
 this and they can score this also independently as a batch, so they can do this in parallel
 and here you see, yeah here is executed in parallel, so the model will go and will score
 this word in and say ah this would have been, this is the argmax of the greedy decoding
 anyway and it can also score this step and say aha given that there is an in that this
 the is the argmax anyway, right and you can score this step and say ah given that there's
 in the, the argmax would have been car, so that's not bus, so we reject this suggestion
 but we keep that part of the suggestion and say okay the in the is basically what would
 have been decoded anyway according to the greedy decoding, so we can basically accept
 this here and continue from there, this is the accept step here, so this basically, so
 you can see in this one step which yeah we'll call one decoding step, we have basically
 done two of the greedy decoding steps in one go, so by predicting into the future and then
 selecting the one that agrees with the original model because we can, the fundamental thing
 is we can score in parallel but we can greedily produce not in parallel, alright so they actually
 push this further by also eliminating one of the, one of the evaluations here by combining
 basically the next predict step with the previous verify step and it's pretty cool to look at
 that, so we're in the same situation, you have this and you suggest this continuation
 and then the score model again will go here but while you verify you also do the next
 predict at the same time, since you've built your model, since it's the same model and
 this model every time you execute it, it outputs a distribution over the next set of positions,
 you might as well take the outputs of it, right, so when you then decide to accept this
 here, you will already have the outputs computed for the next three positions, so this you
 can feed directly into this next predict step, you basically don't have to execute it, you
 simply go to the one you've accepted and then you look at the outputs that you get anyway
 from this model and use them, so you might ask, okay which, how does a model look that
 like scores and predicts into the future and this, the answer is here, it's a bit out of
 order, I would have maybe liked this more previously but in any case this is what they
 do, so they use a transformer architecture and you have to imagine it starts down here
 and actually there is a huge network down here, right, this is just the output layer,
 so there's a giant transformer network down below and it produces this output representation,
 now normally from this representation you would go to this what's called p layer here,
 this is a output vocabulary projection, so this has one entry for each of the words in
 your vocabulary, so the, a, cat and so on and you would then for each one predict a
 probability, so with this representation you basically project it onto this vocabulary
 and predict the probability distribution over the next word, but what they do is they say
 no no no we not only need the next word, we need the next three words, so let's actually
 split this output signal into three output signals and they do this by introducing this
 hidden feed forward layer here or a hidden transformer layer, it's a hidden layer, yeah
 we insert a single feed forward layer with hidden size, okay, so they insert a hidden
 layer and then they also add these skip connections here, right, they add the skip connections
 which basically just means they feed through this output directly to here and add it to
 that, so basically the feed forward layer needs to transform this output here into the
 vocabulary input, one step ahead, two steps ahead and three steps ahead and you can see
 here that those are independent, right, they don't depend on each other, there's nothing
 feeding back p1 here into the decision of p2 so they can be executed in parallel, but
 they lose the dependence on each other, alright, so that's the architecture and you can clearly
 see here it's able to predict three steps into the future at the same time, so yeah,
 alright so they also do different adjustments where they say now yeah we can also kind of
 sacrifice a bit of the fidelity to the original model by not requiring that the basically
 we don't only accept when the suggestion is the perfect best suggestion that would have
 been decoded by the greedy model, but what we could do is we could just if it's in the
 top k we could accept it, if it's in the if it's good enough basically one of the suggestions
 that we have is good enough then we'll accept it or when you have like some sort of distance
 metric they say here so the distance between our suggestion and the maximum so the what
 would have been best by the greedy should be smaller than some constant epsilon and
 that way you can sacrifice a bit of performance but your suggestions will be accepted much
 more often and thereby your speedup will be much higher and they also experiment with
 whether or not they should fine tune the original model along with their model and also the
 experiment with knowledge distillation where they basically have like some some teacher
 model and you train the your model on the output of the teacher model don't want to
 go too far into this since these are mostly kind of things to make it work even better
 and you can see here that this is for example a machine translation task so this is the
 WMT 2014 English German translation and there's a regular they get a blow score of 26 and
 here higher is better and if you can see they get a fairly sizable speedups by keeping the
 blow scores fairly constant so they they almost speed up by 2x but if they allow the blow
 scores to go down a bit they get a much higher speedup of like 3 and then if they do like
 distillation and fine tuning they actually manage to keep up the performance even though
 they get very very high speedups so they get speedups until like 5x by not dropping the
 blow scores very much so that's that's pretty impressive another experiment they do is image
 super resolution where you can see here with regular they try to really keep exactly the
 original model output and it doesn't it doesn't speed it up too much but when they allow for
 a bit of a mistake to be made so here this is image super resolution so values are between
 zero and 255 and they allow epsilon equals to two of that so that's that's kind of less
 than 1% error on the individual pixel then they get a speed ups of 7x or something like
 this and you can see in this region here that when the K is for in case the number of steps
 that you decode ahead so and the mini mean block size is 3.75 that means on average 3.75
 steps ahead or accepted which means basically there their suggestions are almost always
 good enough to be accepted so they get this massive speed up by basically being able to
 jump these decoding steps yeah so they have a bunch of other results here there show their
 wall clock time speed up since iteration speed up as well but if you have to pay in huge
 computational cost it's not so good but they also show that they have a big kind of wall
 clock speed up up to up to 4x here in super resolution and over 3x in translation so it's
 a pretty cool paper they give some examples here a bunch of more tables some examples
 of their super resolution and yeah if this might be something for you then use it it's
 I think it's a pretty neat trick and yeah especially for production systems all right
 that was it bye bye.