Faster Neural Network Training with Data Echoing (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper google brain pipeline bottleneck speed gpu tpu idle network distributed preprocessing augmentation
CPUs are often bottlenecks in Machine Learning pipelines. Data fetching, loading, preprocessing and augmentation can be slow to a point where the GPUs are mostly idle. Data Echoing is a technique to re-use data that is already in the pipeline to reclaim this idle time and keep the GPUs busy at all times.

https://arxiv.org/abs/1907.05550

Abstract:
In the twilight of Moore's law, GPUs and other specialized hardware accelerators have dramatically sped up neural network training. However, earlier stages of the training pipeline, such as disk I/O and data preprocessing, do not run on accelerators. As accelerators continue to improve, these earlier stages will increasingly become the bottleneck. In this paper, we introduce "data echoing," which reduces the total computation used by earlier pipeline stages and speeds up training whenever computation upstream from accelerators dominates the training time. Data echoing reuses (or "echoes") intermediate outputs from earlier pipeline stages in order to reclaim idle capacity. We investigate the behavior of different data echoing algorithms on various workloads, for various amounts of echoing, and for various batch sizes. We find that in all settings, at least one data echoing algorithm can match the baseline's predictive performance using less upstream computation. We measured a factor of 3.25 decrease in wall-clock time for ResNet-50 on ImageNet when reading training data over a network.

Authors: Dami Choi, Alexandre Passos, Christopher J. Shallue, George E. Dahl

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there! Today we're looking at faster neural network training with data echoing by Damme Choi, Alexander Passo, Christopher J. Shalu and George E. Dahl. So on a high level this paper basically says you should repeat data that's already in memory in order to speed up the entire process of neural network training. And it also says that this can speed up your wall time without hurting your performance too much. And I have mixed feelings about it. So let's jump in. So they basically make a point of saying that machine learning doesn't happen in just one thing. It's not like sklearn.fit anymore. It is more of a pipeline. So what do we mean by this? If you think of something like you want to train an ImageNet model, what you want to do is you have like your data set somewhere. And that could be in a database, it could be in the network somewhere. So if you have even something larger than ImageNet, you'll probably store it on a central server on an Amazon bucket somewhere. So this is in AWS. And the first thing you actually need to do is you need to read that data set. Now usually you're not gonna have enough memory on a machine to just load in the entire data set into memory. So that means this process here is streaming. So this is continuously streaming data points. And once you have used the data points, you're gonna throw it away because you need space for the next one, right? And so the streaming is done continuously. The next process is read and decode. That means you have to read it from the network and actually bring it into a format where you can use it. Usually something like an umpire array or a tensorflow tensor. You need to apply some shuffling because usually the order, you can't really trust the order that it is stored in. Oftentimes there is like a bias in the ordering, so you need some sort of a shuffle buffer here. Then often you want to apply some data augmentation to it. That means that you have one image. And we know for these models, if this is your cat, we know for these models that what can help is to basically make many many different images from one image. So this could be by cropping part of it and saying, well, if it was a cat before, the small upper right part here is still a cat. So this is one update. It's called data augmentation. And you're gonna apply a whole bunch of these things. So you can crop, you can rotate the image a bit. It's still a cat, right? And you can also change its luminance and a bit of its colors. You can jitter the colors, you can horizontally flip it and it'll still be a cat. And that's basically how you make many data points from one data point. And we know that helps. Then what you want to do is you want to batch this data. So you want to put it into mini batches. Since you've shuffled here, that means when the next time the same data point comes along, it's going to be batched with a different group of images and of course augmented differently with a different group of images. And that basically means it's a different training batch for the model. So this entire pipeline is basically a way that we take data points that we have and we make a whole bunch of variations and various groupings and batchings of it. And we know that helps enormously with the generalization capability of your final models. And then here you do your apply your SGD update. So that's usually where you forward propagate your data through your network, which here is F. You'll get like some Y hat as an output. And then you have your labels that also come through the pipeline. And you have some sort of loss function L that takes both as an input and gives you an output. And then you do your back propagation. So the back propagation would go through your loss function through your network and update the network parameters such that your network learned something right now this step to the right here is usually what what we focus on when we do deep learning. The step on the right here, all of this, this can be done on a GPU or is usually done on something like a GPU or a TPU. Right. But in these things are getting faster and faster. The point the paper makes is that the TPUs and GPUs of the world are getting faster. But this entire other thing right here, this is basically CPU land. Now I know there is some data augmentation now happening on the GPU and so on. But in essence, you can think of a pipeline where the thing to the left is happening on CPU and the thing to the right is happening on TPU. And even even worse, let's say the speed is continuously is continuously increasing. So in your pipeline, the kind of speed would be so here is the network reading. And here over here is the GPU SGD step. And this is speed. Basically, the further to the right in your pipeline, you go the faster your the faster your hardware gets. And that means that if if this since this is a continuous pipeline, right, that basically means that if I input something here, it goes through the pipeline. And even if this is all running in parallel, at this thing over here is going to idle this since this is the fastest part of the pipeline, it is going to just idle a lot. Right, it because because it can only consume things as fast as this thing can produce. Now if you if you have some sort of a multi GPU machine and just train image net, like you just run the code, usually your this is not the bottleneck. Usually, your GPU is here are at 100% capacity. So this paper is not for you. But if you are, let's say a big company have this network storage, have a big data set, have very expensive data augmentation. This happens, for example, this can happen in NLP and so on. This can be quite your situation, where the earlier in the pipeline, the slower it is. And don't you just love these graphics? So here's here's time. And apparently, it goes in both directions. And so does it go like this? I think what they mean is just time goes in this direction. And you're here. And you're upstream. So your upstream is your network. This is your network reading or your pre processing and the downstream, this is the GPU. So as you are pre processing things, you you and this should be this should be different. It should mean okay, to correct this right here, this is idle. And this is running. So as you're upstream processes images, right at the beginning, your GPU is idle. But then as soon as it ships off the first batch of images, your GPU can run now it's running. While you're doing that, your upstream your network is still reading new images, pre processing them and so on, but it cannot is too slow to insert a batch at the time that the GPU is done. The time the GPU is done, it's still processing this batch. So the GPU is idle until here where it finally manages to process that batch and then the GPU is running again. I think that would have been a much better graphic. But you know, so their goal basically is that what you'll have is right here, for example, after the batch, what you'll do is you scrap this connection, you take this and you put it into a smaller buffer. And the buffer is a repeat buffer. So what it does is it simply will repeat the whatever you have in the buffer until something new comes in, right? So new data point comes in, you just output that data point again, again, again, again, the for the GPU, it's gonna feel like these are all new batches and they continuously come in. But it's always the same until the next data point comes in. And then you output that one again and again and again and again. Now the the actual factor here you can, of course, tune by hand or you can just say repeat until something else comes in. In this paper, they have an explicit factor where they say we repeat each data point four times or three times or so on. So this is data echoing, you basically echo the data point multiple times. And this can be done in various places. So they experiment with echoing in any of these places right here. So the egg they experiment with it right here, after reading and decoding, after shuffling. No, I think always before shuffling. Because if you if you have a shuffle buffer anyway, they say it makes sense that if you do the echoing, you you do your shuffle buffer after you're echoing. So here, then after augmentation and after batching. So they experiment with these three locations in in echoing. Now what could be the downturn of something like this, the downturn, of course, is that this SGD procedure right here, basically, it relies on the data incoming being an IID sample from your data distribution, right? That's that's how we formulate SGD is that there's always new data incoming. Now, if you just output the same data point all the time, that could that is like no new information, first of all, and second of all, it could bias the SGD update, such that you it because it sees the same data, it doesn't it sees the same information over and over, is going to think that's the whole data set, right? So potentially, it can make too many steps into the wrong direction. That just happens to be the bias in this particular data point. So the IID assumption is is is invalid. Now, why do you experiment with this in different locations? Because what you expect is that it hurts more or it hurts less the earlier you introduce this. So if you introduce echoing right here, so if you echo your data until new data from the network comes in, it's still going to be shuffled differently, right? It's and it's still going to be augmented differently. So each time the data point comes out of the echo buffer, it is going to be shuffled. And it is going to be augmented in a different way than the last time the same data point came out. And this is going to be batched together because you've shuffled differently, it's going to be batched together with a different bunch of data points. And that means SGD gets new information. But if you go on to the very last thing, where you just after the batch right here, where you input the echo, that means SGD just gets to see the same batch of data augmented in the same way all the time, right? So the of course, where you exactly have to echo, you have to trade this off. So you have to trade off the how much you basically violate the IID fresh data assumption against where in your data pipeline is the bottleneck. So if your bottleneck is in the data augmentation, it may make little sense to echo before that because your bottleneck is the data augmentation. And that being said, if the bottleneck is that you don't have enough GPUs, then it probably doesn't make sense to data echo at all, though their experiments are somehow wonky on this. But so let's dive in, they make the following claims. Let's just go through them really quick. Data echo reduces the amount of upstream that think of network reading or augmentation computation needed to reach a competitive out of sample error rate on various data sets and model architectures. Second, data echoing can provide a wall time speed up in practice. Third, data echoing can support a wide range of echoing factors. And that's the echoing factor is how often you repeat the data. Fourth, the effectiveness of data echoing depends on the intersection point in the training pipeline, sorry, in the insertion point. That's what our hypothesis was, right? Fifth, data echoing can benefit from additional shuffling after echoing, but does not require it. And six, countering expectations, data echoing reaches the same final error rate as well tuned baselines. So I am can absolutely accept one through five, especially in like an actual practical in the wild setting. But six, we'll see about six. So let's jump into their models. They, sorry about that, they train the following four models. So they train a transformer on these two data sets LM1B and common crawl. So I guess technically it's five models on language modeling. They train the ResNet 32 on CIFAR 10. They train the ResNet 50 on ImageNet and they train SD on Coco. Now here is the accuracies they get and here is, sorry, this is the target. So what they do is they train these models and then they say, okay, what's the accuracy we reach? And then they set a target value. So on ResNet 50 on ImageNet, a very common number to reach is something like 76.5. If you look at, for example, torch vision models, they reach something like this. And so they say, well, our target accuracy here is just a little bit below that. So and then we just measure how many steps or how many their measurement here is fresh data points. So how many actual fresh training samples do we need to reach this target? And this is where it gets wonky because, for example, take the 91% here on CIFAR 10. That is quite, quite low. And also the ResNet 50 is, I mean, this is standard, but still ImageNet is much further nowadays. And I think the effectiveness of something like this has a lot to do with how competitive you want to get. Maybe this is all just an effect of how much under par your, this target performance really is. And I would expect that even though they say it doesn't hurt their performance in their experiments, I would at least expect it will hurt your performance in general if you try to get competitive. Because these things aren't, as of now, at least the ones I know, like the ResNets, aren't really competitive. But so what do they do? They measure data echoing with an echoing factor of 2. So that means data that's incoming is output twice in a row. And every data point that's coming in is just emitted twice from the buffer. And then the next data point is emitted twice and so on. And what they measure, again, is the fresh examples read. So how many fresh data points do you need to achieve something? This is a good measurement because this is kind of independent of hardware. So if you're really in the situation where your GPU is twice as fast as the rest of your pipeline, then an echoing factor of 2 will speed up at most your training procedure by a factor of 2. All right, so you have the baseline in red. And then you have batch echoing, which is where you echo what we said at the worst possible time right after batching. So this might hurt your performance the most, but also it has the potential to be the fastest if maybe your augmentation is very expensive. Then, sorry, or your batching. You have example echoing after augmentation. So that would mean the augmentation is very expensive. So you save the augmented data point. And then you emit it multiple times, but each time it is batched differently. So it is shuffled and then batched with different other data points. So you have a shuffle buffer after it. And then you have example echoing before data augmentation. So that means the same data point emitted multiple times will be augmented in different ways and basically will lead to slightly different data points. So the results here are pretty much what you could expect in that the earlier you do the echoing, as you can see here, the more this echoing helps. So the number, if you, for example, this is the object segmentation task, the baseline needs this many fresh examples to reach this target accuracy. With batch echoing, not only do you, sorry, with batch echoing, you need less fresh training examples. So that means even though you kind of train on the same data twice, this helps you more, or this helps you. It doesn't help you fully because the dashed line here is the, if it would help you as much as a fresh data point, you'd be at the dashed line, right? This is exactly half of this because the echoing factor is two. So if a repeated data point was as useful as a fresh data point, you'd be at the dashed line. As you can see right here, you're not at the dashed line, but at least it doesn't hurt. You might expect that it hurts, but it doesn't hurt. It actually speeds up. So the repeated data points at least have some utility. Again, this is only useful if you have this asymmetry in your pipeline. If your pipeline is actually symmetric and you do an echoing factor of two, the wall time here, the wall time plot would look this for the baseline and then almost twice as high for the batch echoing. Because even though it needs the same amount of fresh, or almost the same amount of fresh example, you echo each one twice. So it needs to process it twice so it'll take much longer. So again, this is useful if you have this asymmetry and if the echoing factor is kind of smaller than your asymmetry. Otherwise you're simply wasting time repeating data points. Then if you do example echoing here after augmentation, you use even less fresh data points. And if you do it before augmentation, this is really surprising. You almost get the benefit of fresh data points, which is something you might expect, right? Because an augmented newly shuffled data point is kind of almost a new data point. But still, it's quite surprising that you almost get to the level of the of the theoretical possible. And also here on the image net task. Now here is actually an example where you can see that it hurts to do this batch echoing. Because the reasons why it could hurt is just that you have you violate this IID assumption, you basically have correlated data points. This is a big, big problem, for example, in reinforcement learning, where already by nature of you running episodes and then feeding the episodes back into the training procedure, you have correlated data points. And that hurts your performance here actually compared to the to the baseline. But then if you go to example echoing, and the example echoing before augmentation, again, you get a speed up, which is pretty cool. Okay, so they do a bunch of other experiments. And I appreciate these experiments here to really show what's going on. And until when can you push this? So here they have a plot of example echoing before augmentation can reduce training time for ResNet 50 on image net. So this is before augmentation. And the echoing factor describes how often you repeat each data point. So this goes from two to five. And you can see that basically you you get the speed up, you just sort of get it for free. As you can see, the dashed line again is as if if at repeated data point were as useful as a fresh data point, you'd be at the dashed line. And you can see right here that you are just above this dashed line. So this can help a lot. And so this is the fresh examples read and this is the wall time in their particular situation. In this case, it doesn't help as much. But again, it if that very much depends on how the asymmetry in your pipeline is. Now, in these experiments, I would actually appreciate something like they do down here, where I would always like to see where it breaks. So how far can you go with the echoing factor until it doesn't help anymore? Because this sort of tells me pretty much nothing. I want to see where is the low point? Where's kind of the optimal echoing factor? And what can you tell me about this optimal echoing factor? How can we determine it sort of beforehand? Or how can you reason how does it connect to the different parts of your architecture? So if I had to point out a flaw in this paper, it would be that right here, I would expect the them to continue this echoing factor increase until it breaks, sort of like they do down here. This is for I believe this is for the transformer on LM 1B. Now here they have a batch size of 1024. And you can see, and this is the this is their standard setting for the transformer, the 1024 batch size, you can see that the baseline uses this many 1.5 times 10 to the seventh fresh examples to train until their target. If you increase the echoing factor by two, you basically need half as many fresh examples, as long as you echo each one twice. Again, very surprising the fact how close you can get to the as if each batch were a a perfect fresh data point. But you can see as you increase this echoing factor, and here is exactly what I said, right, you at some point, this hurts at some point, you get to the point where the non IID ness, the correlation of date of successive data points will actually hurt you. And they make a point here of saying that this is, for example, dependent on batch size. Now in this experiment over here, they have a larger batch size. And here is again the the baseline number of data points to reach the target. And you can see again it goes down. But now with the echoing factor where before you had a you had an increase again, now it continues to decrease. Again, it will be interesting to see where it goes up here and how the number at the lowest like here the four and here the I don't know what it's gonna be maybe the 16, how this will kind of depend on your batch size. And here is another problem. And that's what I alluded to at the beginning, this this performance dependence. Now, I have not read anything differently in the paper. So I had to assume that they trained this here, this number of fresh examples to reach the target is still the target that they determined at the beginning. So it's that 3.9 in the table, that 3.9 was achieved with this batch size with 1024. And we know especially in language models that larger batch sizes will lead to a better performance, even if you need let's say more samples. So here you can see that the samples here is 1.5 and here it's actually four, because you increase that batch size. So that will tell you something 1.5 and four, that is that is like a times. Okay, that's like an times 2.5. So you go with the batch size of times four, and you need 2.5 more more fresh training samples to reach the same target accuracy. First of all, we know that the larger batch sizes can reach higher target accuracies. So again, these results, the dependence of them on the actual performance to the maximum achievable value, to me that's kind of a shady world here to always say, okay, how long does it take to reach that particular target? Because we know that this model right here can reach a much higher target, but we don't know this about these models here. What is their kind of performance in the limit? And they try to make these experiments, but I don't really believe them. Maybe yeah. And yes, and the second, right, that will that will be that will that is already interesting. So this ratio right here, this 2.5 to 4, this ratio must mean something, right? It's it's I go to a higher batch size, four times higher batch size, and I need 2.5 many more fresh training samples to reach the same target. That must somehow tell you something about the usefulness of a single data point versus a succession of data points, right? So it doesn't seem because I would expect if each data point was valuable, I would expect this to be times four. And if it were if it were times one, so if it were no, no speed up at all, sorry, not times four, if it would be times one, it would mean I'd need the same number of fresh training samples, right? No matter how I batch them. But it were times four. That means basically that it doesn't matter really how many training points I have in a batch as long as I have enough and the 1024 seems to be enough. It just it just matters how many you know SGD steps I do. So basically, what we're saying SGD isn't getting the most out of these data points. And this ratio this 2.5, this this tells you something about the information content of a of an additional data point versus the usefulness content of an additional step of SGD. And I would expect that to depend to intrinsically be connected to the where the low point of this echoing factor is because that's exactly what the echoing does. It trades off freshness of data point versus doing more steps on the on the on the same information. And for a paper, especially paper by Google brain, I this this is a connection that I would love to see investigated. But enough of the ranting, they do investigate other things they do investigate, for example, what happens if we just up the batch size. And you can see here, yeah, this is interesting in the baseline needs more fresh samples as you up the batch size. But and at the beginning, this batch echoing, for example, doesn't help doesn't hurt, but doesn't help. But as you go to higher and higher batch sizes, this batch echoing starts to help more and more. Again, I believe this is connected to the usefulness of the single data point, at some point, your batch size is just too large for the problem, you'd rather do more steps. And that's why this helps. But also, this model right here might have a higher ceiling accuracy. So indeed is the question whether this model right here has the same or whether this model right here, the batch echoing model would actually fall back to the ceiling accuracy of one of these models over here. Yeah, in any case, their point is basically that as you increase the batch size, this echoing tends to help more relatively. Because maybe it's because what I said, right, they say as batch size increases, the performance of batch echoing relative to the baseline stays either stays the same or improves. While for example, echoing it either stays the same or it gets sorry, while for example, echoing it either stays the same or gets worse. Dashed lines indicate the expected values if repeated examples were as useful as fresh examples. Yeah, so I believe there is an intrinsic connection here between the usefulness of more data and usefulness of doing additional steps. And here the example echoing you can almost see it as more data because especially here you're going to do augmentation on top of it and you see the non augmented versus the augmented ratio changes dramatically from here to here. Okay, final set of experiments. As you can tell, this is more mostly an experimental paper. And it is always easy to criticize experimental papers and rightfully so because I would not trust this very much. But given that it comes from a big institution, and it is a very well written paper, I would trust it more than I would a regular paper. And I would say if you're in practice, this is certainly worth trying. Absolutely. I'm just I just think that some of the things aren't aren't researched, like some of my questions aren't answered of this. So they investigate sizes. So they now build shuffle buffers. So we have batch echoing, but they say ah, but we can do batch echoing with shuffle buffers. So after the batch echoing, right, we have this state where we have the batching. And then we have the echoing. So this is our echo buffer, where we output the each data point multiple times. And then we have another buffer, which is a shuffle buffer, that a shuffle buffer just collects data points and then shuffles them around before outputting them. And that means even though we you know, output this five times, it might not come out five times after each other, it might be that it comes out once, and then another data point that was already in the shuffle buffer comes out. And then it will just say that in total, it comes out five times. But it is first shuffled together with a bunch of other data points. Of course, this uses more memory, but it returns to that more IID setting. And you can see here as the buffer size increases, then the performance gets more and more to the performance that you would have with completely fresh data. Right. So again, trading off freshness and freshness and doing multiple steps with by by basically repeating, repeating data points straight out versus repeating data points shuffled. And also here you have the same with example echoing. So if you apply the shuffle buffer to example echoing and you increase its size, you can get very, very, very close to the performance that you would get with fresh data, which of course, if you increase the shuffle buffer to the size of the data set, you are at the situation that's the limit you are at the situation of fresh data, right? If you do example echoing. Right. So here is where it gets into the funky part, where they say we actually measure the validation cross entropy and the validation accuracy versus the number of fresh examples read. And here I want to concentrate on the ResNet 50 on ImageNet. And as you can see, most of these models, they pretty much end up in the same place here. It's just that the echoing models end up there faster. Right. And this this is, I mean, this is where it gets a bit confusing, honestly, because why do you have this super sharp thing here? Because usually and here it sort of speeds up in the middle, you see you see that and then it kind of sharply declines. Is this maybe because they drop the learning rate or something. Now, my main thing is that the performance here even though this target thing is lower than than the even though this target thing is the same for everyone, it is lower than the best reachable accuracy. And I'm I'm just this this is just confusing. If this is really true. Whoa. If this is really true, I think we have a lot to learn about SGD yet and how we're not actually doing SGD correctly. And because it seems like almost the the echo versions are better or reach a better accuracy than the baseline. I don't know, do they just cap it at the performance? I don't think so. I think they say they let it reach. They also have these things right here. These these curves where they say this is the best we reach. And this is the ResNet 32 on C410. Again, 91% on C410 is just very, very low. And I'm almost thinking that, okay, this might help if you just throw something that we know is kind of overpowered because we can reach 99%. Or at least you can reach something like 94% on C410 easily, easily with a network smaller than ResNet 32. Maybe this effect manifests if you if you have actually something that could reach higher, but for some reason you only reach this low. I'm not sure. But this is confusing. And if this is really true. Yeah, I would just if it's true, which I believe I believe this paper, it might be just an effect of not reaching the actual ceiling. And again, look at this. This is just the curves are just strange, right? You have the echoing before augmentation, like it seems like it's outperforming the the fresh data points. I don't know, there's a little bell in my head that doesn't like this. If it's actually true, then you know, that's cool. But yeah, so my main criticisms are a bit with the experimental methodology, for example, where they increase the batch size, but still reach the same target accuracy, even though we know that there is a higher ceiling if you increase the batch size for language models. My other criticism is the non investigation of this connection. This connection right here, maybe, but all in all, it's a pretty cool paper. If I had a big company with these pipeline issues, I would absolutely implement this. This seems like a no brainer to do this and can help you tremendously. Alright, that was it. Thank you for listening. If you're still here, subscribe, like, tell a friend. Bye bye.
 Hi there! Today we're looking at faster neural network training with data
 echoing by Damme Choi, Alexander Passo, Christopher J. Shalu and George E. Dahl.
 So on a high level this paper basically says you should repeat data that's
 already in memory in order to speed up the entire process of neural network
 training. And it also says that this can speed up your wall time without hurting
 your performance too much. And I have mixed feelings about it. So let's jump in.
 So they basically make a point of saying that machine learning doesn't happen in
 just one thing. It's not like sklearn.fit anymore. It is more of a pipeline. So
 what do we mean by this? If you think of something like you want to
 train an ImageNet model, what you want to do is you have like your data set
 somewhere. And that could be in a database, it could be in the network
 somewhere. So if you have even something larger than ImageNet, you'll probably
 store it on a central server on an Amazon bucket somewhere. So this is in
 AWS. And the first thing you actually need to do is you need to read that
 data set. Now usually you're not gonna have enough memory on a machine to just
 load in the entire data set into memory. So that means this process here is
 streaming. So this is continuously streaming data points. And once you have
 used the data points, you're gonna throw it away because you need space for the
 next one, right? And so the streaming is done continuously. The next process is
 read and decode. That means you have to read it from the network and
 actually bring it into a format where you can use it. Usually something like an
 umpire array or a tensorflow tensor. You need to apply some shuffling because
 usually the order, you can't really trust the order that it is stored in.
 Oftentimes there is like a bias in the ordering, so you need some sort of a
 shuffle buffer here. Then often you want to apply some data augmentation to it.
 That means that you have one image. And we know for these models, if this is your
 cat, we know for these models that what can help is to basically make many many
 different images from one image. So this could be by cropping part of it and
 saying, well, if it was a cat before, the small upper right part here is still a
 cat. So this is one update. It's called data augmentation. And you're gonna apply
 a whole bunch of these things. So you can crop, you can rotate the image a bit. It's
 still a cat, right? And you can also change its luminance and a bit of its
 colors. You can jitter the colors, you can horizontally flip it and it'll still be a
 cat. And that's basically how you make many data points from one data point. And
 we know that helps. Then what you want to do is you want to batch this data. So you
 want to put it into mini batches. Since you've shuffled here, that means when the
 next time the same data point comes along, it's going to be batched with a
 different group of images and of course augmented differently with a
 different group of images. And that basically means it's a different training
 batch for the model. So this entire pipeline is basically a way that we take
 data points that we have and we make a whole bunch of variations and various
 groupings and batchings of it. And we know that helps enormously with the
 generalization capability of your final models. And then here you do your apply
 your SGD update. So that's usually where you forward propagate your data through
 your network, which here is F. You'll get like some Y hat as an output. And then
 you have your labels that also come through the pipeline. And you have some
 sort of loss function L that takes both as an input and gives you an output. And
 then you do your back propagation. So the back propagation would go through your
 loss function through your network and update the network parameters such that
 your network learned something right now this step to the right here is usually
 what what we focus on when we do deep learning. The step on the right here, all
 of this, this can be done on a GPU or is usually done on something like a GPU or a
 TPU. Right. But in these things are getting faster and faster. The point the
 paper makes is that the TPUs and GPUs of the world are getting faster. But this
 entire other thing right here, this is basically CPU land. Now I know there is
 some data augmentation now happening on the GPU and so on. But in essence, you can
 think of a pipeline where the thing to the left is happening on CPU and the
 thing to the right is happening on TPU. And even even worse, let's say the speed
 is continuously is continuously increasing. So in your pipeline, the kind
 of speed would be so here is the network reading. And here over here is the GPU SGD
 step. And this is speed. Basically, the further to the right in your pipeline,
 you go the faster your the faster your hardware gets. And that means that if if
 this since this is a continuous pipeline, right, that basically means that if I
 input something here, it goes through the pipeline. And even if this is all running
 in parallel, at this thing over here is going to idle this since this is the
 fastest part of the pipeline, it is going to just idle a lot. Right, it because
 because it can only consume things as fast as this thing can produce. Now if
 you if you have some sort of a multi GPU machine and just train image net, like
 you just run the code, usually your this is not the bottleneck. Usually, your GPU
 is here are at 100% capacity. So this paper is not for you. But if you are,
 let's say a big company have this network storage, have a big data set, have
 very expensive data augmentation. This happens, for example, this can happen in
 NLP and so on. This can be quite your situation, where the earlier in the
 pipeline, the slower it is. And don't you just love these graphics? So here's
 here's time. And apparently, it goes in both directions. And so does it go like
 this? I think what they mean is just time goes in this direction. And you're
 here. And you're upstream. So your upstream is your network. This is your
 network reading or your pre processing and the downstream, this is the GPU. So as
 you are pre processing things, you you and this should be this should be
 different. It should mean okay, to correct this right here, this is idle.
 And this is running. So as you're upstream processes images, right at the
 beginning, your GPU is idle. But then as soon as it ships off the first batch of
 images, your GPU can run now it's running. While you're doing that, your
 upstream your network is still reading new images, pre processing them and so
 on, but it cannot is too slow to insert a batch at the time that the GPU is done.
 The time the GPU is done, it's still processing this batch. So the GPU is idle
 until here where it finally manages to process that batch and then the GPU is
 running again. I think that would have been a much better graphic. But you know,
 so their goal basically is that what you'll have is right here, for example,
 after the batch, what you'll do is you scrap this connection, you take this
 and you put it into a smaller buffer. And the buffer is a repeat buffer. So what it
 does is it simply will repeat the whatever you have in the buffer until
 something new comes in, right? So new data point comes in, you just output that
 data point again, again, again, again, the for the GPU, it's gonna feel like these
 are all new batches and they continuously come in. But it's always the same until
 the next data point comes in. And then you output that one again and again and
 again and again. Now the the actual factor here you can, of course, tune by
 hand or you can just say repeat until something else comes in. In this paper,
 they have an explicit factor where they say we repeat each data point four times
 or three times or so on. So this is data echoing, you basically echo the data
 point multiple times. And this can be done in various places. So they
 experiment with echoing in any of these places right here. So the egg they
 experiment with it right here, after reading and decoding, after shuffling. No,
 I think always before shuffling. Because if you if you have a shuffle buffer
 anyway, they say it makes sense that if you do the echoing, you you do your
 shuffle buffer after you're echoing. So here, then after augmentation and after
 batching. So they experiment with these three locations in in echoing. Now what
 could be the downturn of something like this, the downturn, of course, is that
 this SGD procedure right here, basically, it relies on the data incoming being an
 IID sample from your data distribution, right? That's that's how we formulate SGD
 is that there's always new data incoming. Now, if you just output the same data
 point all the time, that could that is like no new information, first of all,
 and second of all, it could bias the SGD update, such that you it because it sees
 the same data, it doesn't it sees the same information over and over, is going
 to think that's the whole data set, right? So potentially, it can make too many
 steps into the wrong direction. That just happens to be the bias in this particular
 data point. So the IID assumption is is is invalid. Now, why do you experiment
 with this in different locations? Because what you expect is that it hurts more or
 it hurts less the earlier you introduce this. So if you introduce echoing right
 here, so if you echo your data until new data from the network comes in, it's
 still going to be shuffled differently, right? It's and it's still going to be
 augmented differently. So each time the data point comes out of the echo buffer,
 it is going to be shuffled. And it is going to be augmented in a different way
 than the last time the same data point came out. And this is going to be batched
 together because you've shuffled differently, it's going to be batched
 together with a different bunch of data points. And that means SGD gets new
 information. But if you go on to the very last thing, where you just after the
 batch right here, where you input the echo, that means SGD just gets to see
 the same batch of data augmented in the same way all the time, right? So the of
 course, where you exactly have to echo, you have to trade this off. So you have
 to trade off the how much you basically violate the IID fresh data assumption
 against where in your data pipeline is the bottleneck. So if your bottleneck is
 in the data augmentation, it may make little sense to echo before that because
 your bottleneck is the data augmentation. And that being said, if the bottleneck is
 that you don't have enough GPUs, then it probably doesn't make sense to data
 echo at all, though their experiments are somehow wonky on this. But so let's dive
 in, they make the following claims. Let's just go through them really quick. Data
 echo reduces the amount of upstream that think of network reading or
 augmentation computation needed to reach a competitive out of sample error rate
 on various data sets and model architectures. Second, data echoing can
 provide a wall time speed up in practice. Third, data echoing can support a wide
 range of echoing factors. And that's the echoing factor is how often you repeat
 the data. Fourth, the effectiveness of data echoing depends on the intersection
 point in the training pipeline, sorry, in the insertion point. That's what
 our hypothesis was, right? Fifth, data echoing can benefit from additional
 shuffling after echoing, but does not require it. And six, countering
 expectations, data echoing reaches the same final error rate as well tuned
 baselines. So I am can absolutely accept one through five, especially in like an
 actual practical in the wild setting. But six, we'll see about six. So let's jump
 into their models. They, sorry about that, they train the following four models. So
 they train a transformer on these two data sets LM1B and common crawl. So I
 guess technically it's five models on language modeling. They train the ResNet
 32 on CIFAR 10. They train the ResNet 50 on ImageNet and they train SD on Coco.
 Now here is the accuracies they get and here is, sorry, this is the target. So
 what they do is they train these models and then they say, okay, what's the
 accuracy we reach? And then they set a target value. So on ResNet 50 on ImageNet,
 a very common number to reach is something like 76.5. If you look at, for
 example, torch vision models, they reach something like this. And so they say,
 well, our target accuracy here is just a little bit below that. So and then we
 just measure how many steps or how many their measurement here is fresh data
 points. So how many actual fresh training samples do we need to reach this target?
 And this is where it gets wonky because, for example, take the 91% here on CIFAR 10.
 That is quite, quite low. And also the ResNet 50 is, I mean, this is standard,
 but still ImageNet is much further nowadays. And I think the effectiveness of
 something like this has a lot to do with how competitive you want to get. Maybe
 this is all just an effect of how much under par your, this target performance
 really is. And I would expect that even though they say it doesn't hurt their
 performance in their experiments, I would at least expect it will hurt your
 performance in general if you try to get competitive. Because these things aren't,
 as of now, at least the ones I know, like the ResNets, aren't really
 competitive. But so what do they do? They measure data echoing with an echoing
 factor of 2. So that means data that's incoming is output twice in a row. And
 every data point that's coming in is just emitted twice from the buffer. And
 then the next data point is emitted twice and so on. And what they measure,
 again, is the fresh examples read. So how many fresh data points do you need to
 achieve something? This is a good measurement because this is kind of
 independent of hardware. So if you're really in the situation where your GPU is
 twice as fast as the rest of your pipeline, then an echoing factor of 2 will
 speed up at most your training procedure by a factor of 2. All right, so you have
 the baseline in red. And then you have batch echoing, which is where you echo
 what we said at the worst possible time right after batching. So this might hurt
 your performance the most, but also it has the potential to be the fastest if
 maybe your augmentation is very expensive. Then, sorry, or your batching.
 You have example echoing after augmentation. So that would mean the
 augmentation is very expensive. So you save the augmented data point. And then
 you emit it multiple times, but each time it is batched differently. So it is
 shuffled and then batched with different other data points. So you have a shuffle
 buffer after it. And then you have example echoing before data augmentation.
 So that means the same data point emitted multiple times will be augmented in
 different ways and basically will lead to slightly different data points. So the
 results here are pretty much what you could expect in that the earlier you do
 the echoing, as you can see here, the more this echoing helps. So the number, if
 you, for example, this is the object segmentation task, the baseline needs
 this many fresh examples to reach this target accuracy. With batch echoing, not
 only do you, sorry, with batch echoing, you need less fresh training examples. So
 that means even though you kind of train on the same data twice, this
 helps you more, or this helps you. It doesn't help you fully because the dashed
 line here is the, if it would help you as much as a fresh data point, you'd be at
 the dashed line, right? This is exactly half of this because the echoing factor
 is two. So if a repeated data point was as useful as a fresh data point,
 you'd be at the dashed line. As you can see right here, you're not at the dashed
 line, but at least it doesn't hurt. You might expect that it hurts, but it
 doesn't hurt. It actually speeds up. So the repeated data points at least have
 some utility. Again, this is only useful if you have this asymmetry
 in your pipeline. If your pipeline is actually symmetric and you do an echoing
 factor of two, the wall time here, the wall time plot would look this for the
 baseline and then almost twice as high for the batch echoing. Because even
 though it needs the same amount of fresh, or almost the same amount of fresh
 example, you echo each one twice. So it needs to process it twice so it'll
 take much longer. So again, this is useful if you have this asymmetry and if
 the echoing factor is kind of smaller than your asymmetry. Otherwise you're
 simply wasting time repeating data points. Then if you do example echoing
 here after augmentation, you use even less fresh data points. And if you do it
 before augmentation, this is really surprising. You almost get the benefit of
 fresh data points, which is something you might expect, right? Because an
 augmented newly shuffled data point is kind of almost a new data point. But
 still, it's quite surprising that you almost get to the level of the of the
 theoretical possible. And also here on the image net task. Now here is actually
 an example where you can see that it hurts to do this batch echoing. Because
 the reasons why it could hurt is just that you have you violate this IID
 assumption, you basically have correlated data points. This is a big, big problem,
 for example, in reinforcement learning, where already by nature of you running
 episodes and then feeding the episodes back into the training procedure, you
 have correlated data points. And that hurts your performance here actually
 compared to the to the baseline. But then if you go to example echoing, and the
 example echoing before augmentation, again, you get a speed up, which is pretty
 cool. Okay, so they do a bunch of other experiments. And I appreciate these
 experiments here to really show what's going on. And until when can you push
 this? So here they have a plot of example echoing before augmentation can reduce
 training time for ResNet 50 on image net. So this is before augmentation. And the
 echoing factor describes how often you repeat each data point. So this goes from
 two to five. And you can see that basically you you get the speed up, you
 just sort of get it for free. As you can see, the dashed line again is as if if at
 repeated data point were as useful as a fresh data point, you'd be at the dashed
 line. And you can see right here that you are just above this dashed line. So this
 can help a lot. And so this is the fresh examples read and this is the wall time
 in their particular situation. In this case, it doesn't help as much. But again,
 it if that very much depends on how the asymmetry in your pipeline is. Now, in
 these experiments, I would actually appreciate something like they do down
 here, where I would always like to see where it breaks. So how far can you go
 with the echoing factor until it doesn't help anymore? Because this sort of tells
 me pretty much nothing. I want to see where is the low point? Where's kind of
 the optimal echoing factor? And what can you tell me about this optimal echoing
 factor? How can we determine it sort of beforehand? Or how can you reason how
 does it connect to the different parts of your architecture? So if I had to point
 out a flaw in this paper, it would be that right here, I would expect the them
 to continue this echoing factor increase until it breaks, sort of like they do
 down here. This is for I believe this is for the transformer on LM 1B. Now here
 they have a batch size of 1024. And you can see, and this is the this is their
 standard setting for the transformer, the 1024 batch size, you can see that the
 baseline uses this many 1.5 times 10 to the seventh fresh examples to train
 until their target. If you increase the echoing factor by two, you basically need
 half as many fresh examples, as long as you echo each one twice. Again, very
 surprising the fact how close you can get to the as if each batch were a a
 perfect fresh data point. But you can see as you increase this echoing factor, and
 here is exactly what I said, right, you at some point, this hurts at some point,
 you get to the point where the non IID ness, the correlation of date of
 successive data points will actually hurt you. And they make a point here of
 saying that this is, for example, dependent on batch size. Now in this
 experiment over here, they have a larger batch size. And here is again the the
 baseline number of data points to reach the target. And you can see again it goes
 down. But now with the echoing factor where before you had a you had an
 increase again, now it continues to decrease. Again, it will be interesting to
 see where it goes up here and how the number at the lowest like here the four
 and here the I don't know what it's gonna be maybe the 16, how this will
 kind of depend on your batch size. And here is another problem. And that's what
 I alluded to at the beginning, this this performance dependence. Now, I have not
 read anything differently in the paper. So I had to assume that they trained
 this here, this number of fresh examples to reach the target is still the target
 that they determined at the beginning. So it's that 3.9 in the table, that 3.9
 was achieved with this batch size with 1024. And we know especially in language
 models that larger batch sizes will lead to a better performance, even if you need
 let's say more samples. So here you can see that the samples here is 1.5 and
 here it's actually four, because you increase that batch size. So that will
 tell you something 1.5 and four, that is that is like a times. Okay, that's like
 an times 2.5. So you go with the batch size of times four, and you need 2.5 more
 more fresh training samples to reach the same target accuracy. First of all, we
 know that the larger batch sizes can reach higher target accuracies. So again,
 these results, the dependence of them on the actual performance to the
 maximum achievable value, to me that's kind of a shady world here to always
 say, okay, how long does it take to reach that particular target? Because we
 know that this model right here can reach a much higher target, but we don't
 know this about these models here. What is their kind of performance in the limit?
 And they try to make these experiments, but I don't really believe them. Maybe
 yeah. And yes, and the second, right, that will that will be that will that is
 already interesting. So this ratio right here, this 2.5 to 4, this ratio must mean
 something, right? It's it's I go to a higher batch size, four times higher
 batch size, and I need 2.5 many more fresh training samples to reach the same
 target. That must somehow tell you something about the usefulness of a
 single data point versus a succession of data points, right? So it doesn't seem
 because I would expect if each data point was valuable, I would expect this
 to be times four. And if it were if it were times one, so if it were no, no speed
 up at all, sorry, not times four, if it would be times one, it would mean I'd
 need the same number of fresh training samples, right? No matter how I batch
 them. But it were times four. That means basically that it doesn't matter really
 how many training points I have in a batch as long as I have enough and the
 1024 seems to be enough. It just it just matters how many you know SGD steps I do.
 So basically, what we're saying SGD isn't getting the most out of these data
 points. And this ratio this 2.5, this this tells you something about the
 information content of a of an additional data point versus the
 usefulness content of an additional step of SGD. And I would expect that to
 depend to intrinsically be connected to the where the low point of this echoing
 factor is because that's exactly what the echoing does. It trades off
 freshness of data point versus doing more steps on the on the on the same
 information. And for a paper, especially paper by Google brain, I this this is a
 connection that I would love to see investigated. But enough of the ranting,
 they do investigate other things they do investigate, for example, what happens if
 we just up the batch size. And you can see here, yeah, this is interesting in
 the baseline needs more fresh samples as you up the batch size. But and at the
 beginning, this batch echoing, for example, doesn't help doesn't hurt, but
 doesn't help. But as you go to higher and higher batch sizes, this batch echoing
 starts to help more and more. Again, I believe this is connected to the
 usefulness of the single data point, at some point, your batch size is just too
 large for the problem, you'd rather do more steps. And that's why this helps. But
 also, this model right here might have a higher ceiling accuracy. So indeed is
 the question whether this model right here has the same or whether this model
 right here, the batch echoing model would actually fall back to the ceiling
 accuracy of one of these models over here. Yeah, in any case, their point is
 basically that as you increase the batch size, this echoing tends to help more
 relatively. Because maybe it's because what I said, right, they say as batch
 size increases, the performance of batch echoing relative to the baseline stays
 either stays the same or improves. While for example, echoing it either stays the
 same or it gets sorry, while for example, echoing it either stays the same or gets
 worse. Dashed lines indicate the expected values if repeated examples were as
 useful as fresh examples. Yeah, so I believe there is an intrinsic connection
 here between the usefulness of more data and usefulness of doing additional steps.
 And here the example echoing you can almost see it as more data because
 especially here you're going to do augmentation on top of it and you see
 the non augmented versus the augmented ratio changes dramatically from here to
 here. Okay, final set of experiments. As you can tell, this is more mostly an
 experimental paper. And it is always easy to criticize experimental papers and
 rightfully so because I would not trust this very much. But given that it comes
 from a big institution, and it is a very well written paper, I would trust it
 more than I would a regular paper. And I would say if you're in practice, this is
 certainly worth trying. Absolutely. I'm just I just think that some of the
 things aren't aren't researched, like some of my questions aren't answered of
 this. So they investigate sizes. So they now build shuffle buffers. So we have
 batch echoing, but they say ah, but we can do batch echoing with shuffle buffers.
 So after the batch echoing, right, we have this state where we have the
 batching. And then we have the echoing. So this is our echo buffer, where we
 output the each data point multiple times. And then we have another buffer,
 which is a shuffle buffer, that a shuffle buffer just collects data points and
 then shuffles them around before outputting them. And that means even
 though we you know, output this five times, it might not come out five times
 after each other, it might be that it comes out once, and then another data
 point that was already in the shuffle buffer comes out. And then it will just
 say that in total, it comes out five times. But it is first shuffled together
 with a bunch of other data points. Of course, this uses more memory, but it
 returns to that more IID setting. And you can see here as the buffer size
 increases, then the performance gets more and more to the performance that you
 would have with completely fresh data. Right. So again, trading off freshness
 and freshness and doing multiple steps with by by basically repeating,
 repeating data points straight out versus repeating data points shuffled.
 And also here you have the same with example echoing. So if you apply the
 shuffle buffer to example echoing and you increase its size, you can get very,
 very, very close to the performance that you would get with fresh data, which of
 course, if you increase the shuffle buffer to the size of the data set, you
 are at the situation that's the limit you are at the situation of fresh data,
 right? If you do example echoing. Right. So here is where it gets into the funky
 part, where they say we actually measure the validation cross entropy and the
 validation accuracy versus the number of fresh examples read. And here I want to
 concentrate on the ResNet 50 on ImageNet. And as you can see, most of these
 models, they pretty much end up in the same place here. It's just that the
 echoing models end up there faster. Right. And this this is, I mean, this is
 where it gets a bit confusing, honestly, because why do you have this super sharp
 thing here? Because usually and here it sort of speeds up in the middle, you see
 you see that and then it kind of sharply declines. Is this maybe because they
 drop the learning rate or something. Now, my main thing is that the performance
 here even though this target thing is lower than than the even though this
 target thing is the same for everyone, it is lower than the best reachable
 accuracy. And I'm I'm just this this is just confusing. If this is really true.
 Whoa. If this is really true, I think we have a lot to learn about SGD yet and
 how we're not actually doing SGD correctly. And because it seems like
 almost the the echo versions are better or reach a better accuracy than the
 baseline. I don't know, do they just cap it at the performance? I don't think so.
 I think they say they let it reach. They also have these things right here.
 These these curves where they say this is the best we reach. And this is the
 ResNet 32 on C410. Again, 91% on C410 is just very, very low. And I'm almost
 thinking that, okay, this might help if you just throw something that we know is
 kind of overpowered because we can reach 99%. Or at least you can reach
 something like 94% on C410 easily, easily with a network smaller than ResNet 32.
 Maybe this effect manifests if you if you have actually something that could
 reach higher, but for some reason you only reach this low. I'm not sure. But
 this is confusing. And if this is really true. Yeah, I would just if it's true,
 which I believe I believe this paper, it might be just an effect of not reaching
 the actual ceiling. And again, look at this. This is just the curves are just
 strange, right? You have the echoing before augmentation, like it seems like
 it's outperforming the the fresh data points. I don't know, there's a little
 bell in my head that doesn't like this. If it's actually true, then you know,
 that's cool. But yeah, so my main criticisms are a bit with the
 experimental methodology, for example, where they increase the batch size, but
 still reach the same target accuracy, even though we know that there is a
 higher ceiling if you increase the batch size for language models. My other
 criticism is the non investigation of this connection. This connection right
 here, maybe, but all in all, it's a pretty cool paper. If I had a big company
 with these pipeline issues, I would absolutely implement this. This seems
 like a no brainer to do this and can help you tremendously. Alright, that was
 it. Thank you for listening. If you're still here, subscribe, like, tell a
 friend. Bye bye.