Accelerating Deep Learning by Focusing on the Biggest Losers
machine learning deep learning dl neural network training convergence loss importance speed-up faster ai dnn deep neural network backprop backpropagation cifar10 svhn classifier
What if you could reduce the time your network trains by only training on the hard examples? This paper proposes to select samples with high loss and only train on those in order to speed up training.

Abstract:
This paper introduces Selective-Backprop, a technique that accelerates the training of deep neural networks (DNNs) by prioritizing examples with high loss at each iteration. Selective-Backprop uses the output of a training example's forward pass to decide whether to use that example to compute gradients and update parameters, or to skip immediately to the next example. By reducing the number of computationally-expensive backpropagation steps performed, Selective-Backprop accelerates training. Evaluation on CIFAR10, CIFAR100, and SVHN, across a variety of modern image models, shows that Selective-Backprop converges to target error rates up to 3.5x faster than with standard SGD and between 1.02--1.8x faster than a state-of-the-art importance sampling approach. Further acceleration of 26% can be achieved by using stale forward pass results for selection, thus also skipping forward passes of low priority examples.

Authors: Angela H. Jiang, Daniel L.-K. Wong, Giulio Zhou, David G. Andersen, Jeffrey Dean, Gregory R. Ganger, Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary C. Lipton, Padmanabhan Pillai

https://arxiv.org/abs/1910.00762
 Hi there! Today we're looking at accelerating deep learning by focusing on the biggest losers by Angela Jiang et al. This paper is pretty simple, pretty short in idea and is a pretty much an engineering paper. So we'll go over this idea and give it a good look and discuss advantages, disadvantages and so on. What's the basic idea? The basic idea is the following. If you train a neural network, what do you do? Usually you have a training data set, which I represent here. Each line is a sample and usually your network has a bunch of layers. Each line here is a layer of weights. What you do is you group your training data set into mini batches. Let's say that's a mini batch, four samples and you pass it through the network. This is called the forward propagation. You then calculate the loss of your forward propagated signal and then you back propagate this loss. When back propagating, you want to back propagate the loss such that it reaches each of the layers and it tells each layer how to update itself. What you want to do is for each layer you actually need to back prop the loss once towards the layer below it and once towards itself in order for the layer below it to continue the back prop and for the layer itself to update its weights. Each time you back prop basically once towards the lower layer and once towards yourself. That's a lot of work. You see whatever work you have passing your samples through the network here, you basically double the work going back. The core idea of this paper is if you look at the following. In a traditional training neural network you'll have some overhead in each training step, some overhead of maybe putting the data to the GPU or something like this. Then you have a time that you require for a forward pass and then you have a big chunk that you require for the backward pass. You see it's about double the size of this forward pass. This paper asks how can we reduce this backward pass time. What they propose is the following. They propose if the backward pass is expensive and we do it here for each data point in these mini batches, why don't we stop doing this and only try to select examples that are important. Once we only have selected the important examples, only those examples get to do the backward pass. Thereby let's say if we can only select one third of the examples to do the backward pass, we can reduce the amount that's required in the backward pass, the amount of work, by one third or sorry by two thirds. The way they select the important examples is by looking at the loss. They basically say whichever examples have a high loss, these must be the important examples, these are the hard examples. If we only train on the hard examples or if we train on the hard examples more, then the network will learn on these hard examples faster. Of course there is an implication there that if your network is good on the hard examples, it's also going to be good on the easy examples. That's like the definition of hard and easy examples. Of course that's a kind of a simplifying assumption. The idea is only select the hard examples and only by how much loss they have and only then backprop these hard examples. That's how they can reduce this by a lot. There's several intricacies here. The setup time of course is the same. What they do next is they forward propagate the entire mini batch here, because they need the loss of each example and then therefore they need to forward propagate the entire mini batch. At the end of this they select the examples with the highest loss and they only use those in training. Training consists of another forward pass, but this one is much smaller because you only forward pass the examples that you're actually training on. Then the backward pass accordingly will also be much much smaller because now again you have less samples to actually train on. The reason that you even need this second forward pass is the following. When you do backprop you can't simply start with a signal back here and then backprop that through the network. That doesn't work usually with most network architectures. Namely what you need to do is actually while you forward pass you need to remember information at each layer. A good example of this is the MaxPool operation. In MaxPool what you do is you maybe have four pixels that are next to each other and you select one of them. Now you need to remember during the forward pass which one you selected. Otherwise the backward pass won't work. You need to know which pixel to back prop through. That's why at each point you need to remember information to inform the backward pass. That's why basically you need a second forward pass with only the examples that you want to train on. You forward pass once, calculate this loss, select the ones with the high loss, then forward pass these again and then backprop only these examples. That's the main gist of it. This is exactly what you see here. Forward pass everything, then forward pass those again that have high loss and then backprop them. There is actually an interesting thing in this graphic in that you see that this forward pass here also is shorter than this forward pass. I assume that's because this forward pass here actually needs to do those additional saving of information while this forward pass here is simply a forward pass without intention of backward passing. You can instruct the deep learning frameworks to then not remember this information. They have another improvement over their algorithm called stale selective backprop. This is called selective backprop. They have stale selective backprop. What stale selective backprop does is it says well we might not always need this forward pass. What we might be able to do is actually, first we take the entire data set, let's use a different color here, let's use this. We take the entire data set forward properly through the network and then save this save into some database the losses. Then we use these losses to select the individual points here for a while. We perform maybe this is training here. You start here, you do this loss calculation and then you run your training until a couple of epochs and then you say okay now this information here is really outdated, I should really update it. Then you do this entire thing again and then you run training some more until you again stop and say okay now this information is stale again. Thereby you can amortize the cost of these forward passes. You pass your entire training set once in a while and then use those losses to select the hard examples. That's amortized. You can then reduce this forward pass that's used for selecting again by a lot. Of course the paper shows that this doesn't hurt your performance too much if you have this stale information. This is the entire idea of the algorithm and the algorithm is detailed here. Very briefly you have this buffer and you go through the data, you forward pass every example. For each loss you calculate the probability that you should retain it. It's a probabilistic framework, it's not absolute cutoff. If you decide to choose it probabilistically you append it to this buffer. If this buffer is of a certain size then you do the back prop only on this buffer. This buffer now only has the high loss examples with higher probability. Don't forget within this backward here there is also an implicit forward pass. Then you clear the buffer and go on. There are lots of forward passes here to compute the losses and then every now and then there is a backward pass whenever the buffer is of certain size. The probabilistic calculation of how and when to retain a sample is very simple. You have a deck of recent losses, a history of recent losses. You simply calculate the percentile that a given loss has in this history and that percentile will then decide on the probability. If you raise it to a power and that looks something like this. What's often used in this paper is this 33% selection. That would be the blue curve and you see the median example here. If you are in the median then you have about a 33% chance of being retained. If you have a higher loss than that then your probability rises. The first interesting thing actually is this graphic here where they show what the algorithm deems the hardest and easiest examples. Examples chosen least frequently and this is the CIFAR-10 dataset which is images 32 by 32 in color and you need to classify them into 10 categories. You see the easiest images, the ones chosen least frequently, are almost all automobiles. Of the automobiles they're almost all where you see the full car with the wheels and whatnot like this one here. These are what the algorithm deems easy samples and if you look at the hard samples, examples chosen most frequently by selective backprop, it's these here. For example bird and bird in this case is just kind of a smear. They're just kind of smears on a blue background. It's understandably that this resolution is pretty hard to make out that this is a bird. Airplane and automobile here you see that it's only a partial view of the thing. It's not like the full car like we saw in the easy pictures. It's only partial and this seems to be pretty hard and it's understandable. This cat here to me it's even unclear if this is a cat or a dog and I think dog is also a category in CIFAR-10 so the algorithm is certainly understandably confused by this example and deems it a hard example. And here even more you see truck and this isn't a truck as far as I can make out. These are two humans on the picture with no truck anywhere visible. So this seems to be a mislabeled example and of course mislabeled examples are going to be of high loss to the algorithm. This is the first criticism or thing and the authors recognize this that if you up weigh your examples with high loss you are going to up weigh all the mislabeled examples as well and thereby you're going to train more on the mislabeled examples and thereby you're going to possibly degrade your test performance much more than had you given every sample the same weight. And the authors address this actually nicely by doing an experiment and the experiment is what if we artificially mislabel examples how much can these algorithms tolerate. And so they often have these graphics here where they show test error over time. So test error and the x-axis here is number of back propped images which is kind of a time dimension in training these algorithms. You see the blue is a traditional model and the pink is a selective back prop with a 33% retain rate. So you see the selective back prop is much faster in reaching a low error and this first thing is simply with 1% of the labels shuffled. So 1% of the images are mislabeled. Selective back prop is still much faster than the traditional trajectory. If you go to 10% shuffled still you can see selective back prop is faster reaching a low error. Of course the error now generally is higher. But if you go to 20% here what you can see 20% shuffled labels what you can see it starts to become clear that these selective back prop it retains 33% of the hardest examples right. So and 20% of the examples have a wrong label. That means most of what it upweighs are wrongly labeled examples. Almost let's say that there's still a lot of correctly labeled examples. But you see it gets to a low error but then it gets up again as it kind of massively overfits on these wrongly labeled examples because it upweighs them so much. Because here in still every example is hard right. So these wrongly labeled examples they'll get about the same weight as correctly labeled examples because the network isn't trained yet. But as you go lower it starts to massively overfit. So compared to the traditional model kind of just reaches this low error that okay is now corrupted by these wrong labels but it doesn't it doesn't hurt as much. So that's kind of my first criticism. If you have a lot of noisy labels or if you have a lot of mislabeled examples then this method might actually hurt more than it helps. But the level is interesting that it can kind of tolerate 10% but it gets kind of into trouble at 20 or so more percent. So this is the first criticism and that's how the authors address it. I really like this ablation study that they do. Here this is kind of the meat of the experiment. So what they show here these curves on the bottom and let's look at this curve is on the x-axis you actually have wall clock time now. So how much time do you need in order to reach a kind of low error. Here is test set error. You see the traditional model in blue has a certain trajectory. Now cath 18 is a baseline don't worry about it. What we're interested in is the selective backprop which is the pink which you can see outperforms this traditional training. And what we're also interested in is the stale SB. So stale meaning it has this buffer of information that reduces it's supposed to reduce the time again. And you see that is even that even more outperforms the traditional approach. You can also see that the staleness here apparently doesn't hurt the performance too much. You see the error is fairly close and it reaches this error in a much faster time. This on CIFAR 10. They have this nice table up here where they show the speed up to reach a given error. So what they do is they take this error of the traditional model this test set error and they ask how fast are these methods in reaching this error times a constant. So times 1.1 times 1.2 times 1.4 now. Of course the reaching 1.4 times the final error is much is easier and thereby but it's also easier for the traditional model of course. So that's the catch but these are kind of benchmarks they chose to how fast are these models in reaching 1.1 1.2 1.4 times the error of a traditionally trained model. You can see here on CIFAR 10 for example actually it's go to SVHN. SVHN is the easiest of the of the data sets and it shows the most clear thing. So the traditional error is 1.7% and you see that the speed up is so this selective back prop is 3.4 times faster in reaching this 1.1 times the error of this traditional model and it's also 3.4 times faster reaching 1.2 times and it's 3.5 times faster in reaching it 1.4 times. The stale selective back prop is even faster so 4.3 4.9 5 times faster in reaching 1.4 times this reaching 1.4 times the the error and so what you can what you can see here is that these methods really make it faster but also there's kind of two things two important things to note in this table. First of all you can see as you go to the right in the table the speed ups get higher and what it means is that as you need to reach as you make the problem easier so as you need to reach a higher error which is as you need to reach a higher loss value these methods are there faster what that means is they're really fast at reaching a somewhat decent point which is represented here they're really fast but if they need them to reach a more and more accurate performance they themselves get slower and slower so this this is of course clear because what you're doing is you're no longer treating every day to point the same you are introducing a bias into your training by only training on the hard examples so you're introducing a bias and this bias will give you a speed up but also hurt your performance and thereby if you have to get more and more accurate you will you will lose much of that speed up because you need to reduce that bias at the end that you introduced so that's the first caveat as you want to get to a higher and higher performance these methods will help less and less because they basically introduce the bias to gain speed at the beginning of training or to reach less accurate points the second thing is as you look at these problems here so SVH n 1.7 percent error C for 10 is a slightly harder problem 2.9 percent error and C for 100 is really a harder problem where a traditional model has 18 percent error if you look at the speed ups now then you can see even at this right most end here you have the 3.5 and 5x speed up here we have a 1.5 2x speed up here we have a 1.2 1.6x speed up so as the problems get harder and as the kind of models get get fancier as the classes get more then the the speed up is much lower and I believe that's because the the bias you introduce by reweighing the samples the bias you introduce will hurt you much more on a difficult and large problem with a large network then it will hurt you on an easy problem right easy problem you were fine introducing some bias but if you have a hard noisy problem then this bias you introduce will hurt you much more and thereby this the speed up that these methods give you is much much less and so this means that the performance of these models is directly anti correlated with the hardness of the problem and that tells me it kind of makes it almost unusable or it goes towards if I look at the numbers if I look at the numbers over here and extrapolate that to something like image net it tells me that these methods are going to be almost useless on a data set of the size and complexity as image net and the interesting problems nowadays are very much in the domain of more hard more complex problems so the the kind of usefulness of this method in practice is something that I wouldn't bet on just from reading this paper I'm open to be convinced otherwise but just from reading this papers it seems like the harder you make the problem the less these methods help and that's exactly not what you want you want exactly the opposite you want to say oh if I scale this up it'll it'll you know give me even more of a speed up and that's going to be even better but this is the opposite so and given that they have no basically no theoretical analysis of how much this bias hurts you or how you can still make it kind of good in expectation how you would need to correct at the end and so on I would I would I would first of course test it I'm very interested to see tests on larger more complex problems but from this I'm a bit skeptical I'm sorry yeah so they they show I mean they show that on these states that it clearly helps clearly speeds up the training and that's of course that's already a good good thing and they do the required experiments they do the ablation studies on these data sets and so on so you can see here for example on these first graphics on all the data sets see clearly goes down as you introduce the more sophisticated algorithms but again you can see on the hard data set it doesn't go down as much all right but they do discuss this they're really fair to themselves they do risk they discuss this in their paper of how you know how practical this is and so on and what they what else they tried and didn't work and and that's a I think that it's a really good paper in itself and it's a really good investigation all right so that was it for me have a fun day bye bye
 Hi there! Today we're looking at accelerating deep learning by focusing on
 the biggest losers by Angela Jiang et al. This paper is pretty simple, pretty short
 in idea and is a pretty much an engineering paper. So we'll go over this
 idea and give it a good look and discuss advantages, disadvantages and so on.
 What's the basic idea? The basic idea is the following. If you train a neural
 network, what do you do? Usually you have a training data set, which I represent
 here. Each line is a sample and usually your network has a bunch of
 layers. Each line here is a layer of weights. What you do is you group your
 training data set into mini batches. Let's say that's a mini batch, four
 samples and you pass it through the network. This is called the forward
 propagation. You then calculate the loss of your forward propagated
 signal and then you back propagate this loss. When back propagating, you
 want to back propagate the loss such that it reaches each of the layers and
 it tells each layer how to update itself. What you want to do is for each
 layer you actually need to back prop the loss once towards the layer below it and
 once towards itself in order for the layer below it to continue the back prop
 and for the layer itself to update its weights. Each time you back prop
 basically once towards the lower layer and once towards yourself. That's a
 lot of work. You see whatever work you have passing your samples through the
 network here, you basically double the work going back.
 The core idea of this paper is if you look at the following. In a
 traditional training neural network you'll have some overhead in each
 training step, some overhead of maybe putting the data to the GPU or something
 like this. Then you have a time that you require for a forward pass and then you
 have a big chunk that you require for the backward pass. You see it's about
 double the size of this forward pass. This paper asks how can we reduce this
 backward pass time. What they propose is the following. They propose if
 the backward pass is expensive and we do it here for each data point in these
 mini batches, why don't we stop doing this and only try to select
 examples that are important. Once we only have selected the important
 examples, only those examples get to do the backward pass. Thereby let's say if
 we can only select one third of the examples to do the backward pass, we can
 reduce the amount that's required in the backward pass, the amount of work, by
 one third or sorry by two thirds. The way they select the important examples
 is by looking at the loss. They basically say whichever examples have a
 high loss, these must be the important examples, these are the hard examples.
 If we only train on the hard examples or if we train on the hard
 examples more, then the network will learn on these hard examples faster.
 Of course there is an implication there that if your network is good on the hard
 examples, it's also going to be good on the easy examples. That's like the
 definition of hard and easy examples. Of course that's a kind of a simplifying
 assumption. The idea is only select the hard examples and only by how much
 loss they have and only then backprop these hard examples. That's how
 they can reduce this by a lot. There's several intricacies here.
 The setup time of course is the same. What they do next is they forward
 propagate the entire mini batch here, because they need the loss of each
 example and then therefore they need to forward propagate the entire mini batch.
 At the end of this they select the examples with the highest loss and they
 only use those in training. Training consists of another forward
 pass, but this one is much smaller because you only forward pass the
 examples that you're actually training on. Then the backward pass accordingly
 will also be much much smaller because now again you have less samples to
 actually train on. The reason that you even need this second forward
 pass is the following. When you do backprop you can't simply start with a
 signal back here and then backprop that through the network. That doesn't work
 usually with most network architectures. Namely what you need to do is actually
 while you forward pass you need to remember information at each layer. A good
 example of this is the MaxPool operation. In MaxPool what you do is you
 maybe have four pixels that are next to each other and you select one of them.
 Now you need to remember during the forward pass which one you selected.
 Otherwise the backward pass won't work. You need to know which pixel to back
 prop through. That's why at each point you need to remember information to
 inform the backward pass. That's why basically you need a second forward
 pass with only the examples that you want to train on.
 You forward pass once, calculate this loss, select the ones with the
 high loss, then forward pass these again and then backprop only these examples.
 That's the main gist of it. This is exactly what you see here.
 Forward pass everything, then forward pass those again that have high loss
 and then backprop them. There is actually an interesting thing in this
 graphic in that you see that this forward pass here also is shorter than
 this forward pass. I assume that's because this forward pass here actually
 needs to do those additional saving of information while this forward pass here
 is simply a forward pass without intention of backward passing. You can
 instruct the deep learning frameworks to then not remember this information.
 They have another improvement over their algorithm called stale
 selective backprop. This is called selective backprop. They have stale
 selective backprop. What stale selective backprop does is it says well we might
 not always need this forward pass. What we might be able to do is
 actually, first we take the entire data set,
 let's use a different color here, let's use this. We take the
 entire data set forward properly through the network and then save this
 save into some database the losses. Then we use these losses to select
 the individual points here for a while. We perform maybe
 this is training here. You start here, you do this loss calculation and then
 you run your training until a couple of epochs and then you say okay now this
 information here is really outdated, I should really update it. Then you do this
 entire thing again and then you run training some more until you again stop
 and say okay now this information is stale again. Thereby you can amortize
 the cost of these forward passes. You pass your entire training set once
 in a while and then use those losses to select the hard examples. That's
 amortized. You can then reduce this forward pass that's used for selecting
 again by a lot. Of course the paper shows that this doesn't hurt your
 performance too much if you have this stale information. This is the
 entire idea of the algorithm and the algorithm is detailed here. Very
 briefly you have this buffer and you go through the data, you forward pass every
 example. For each loss you calculate the probability that you should retain it.
 It's a probabilistic framework, it's not absolute cutoff. If you decide to
 choose it probabilistically you append it to this buffer. If this buffer is of a
 certain size then you do the back prop only on this buffer. This buffer
 now only has the high loss examples with higher
 probability. Don't forget within this backward here there is also an implicit
 forward pass. Then you clear the buffer and go on. There are lots of
 forward passes here to compute the losses and then every now
 and then there is a backward pass whenever the buffer is of certain size.
 The probabilistic calculation of how and when to retain a sample is very
 simple. You have a deck of recent losses, a history of recent losses. You simply
 calculate the percentile that a given loss has in this history and that
 percentile will then decide on the probability. If you raise it to a power
 and that looks something like this. What's often used in this paper is this
 33% selection. That would be the blue curve and you see the median example
 here. If you are in the median then you have about a 33% chance of being
 retained. If you have a higher loss than that then your probability rises.
 The first interesting thing actually is this graphic here where they show
 what the algorithm deems the hardest and easiest examples. Examples chosen
 least frequently and this is the CIFAR-10 dataset which is images 32 by 32 in
 color and you need to classify them into 10 categories. You see the easiest
 images, the ones chosen least frequently, are almost all automobiles.
 Of the automobiles they're almost all where you see the full car with the
 wheels and whatnot like this one here. These are what the
 algorithm deems easy samples and if you look at the hard samples, examples
 chosen most frequently by selective backprop, it's these here. For example
 bird and bird in this case is just kind of a smear. They're just kind of smears
 on a blue background. It's understandably that this resolution is pretty hard to
 make out that this is a bird. Airplane and automobile here you see that it's
 only a partial view of the thing. It's not like the full car like we saw
 in the easy pictures. It's only partial and this seems to be pretty hard and
 it's understandable. This cat here to me it's even unclear if this is a cat or a
 dog and I think dog is also a category in CIFAR-10 so the algorithm is
 certainly understandably confused by this example and deems it a hard example.
 And here even more you see truck and this isn't a truck as far as I can make
 out. These are two humans on the picture with no truck anywhere visible. So this
 seems to be a mislabeled example and of course mislabeled examples are going to
 be of high loss to the algorithm. This is the first criticism or thing and the
 authors recognize this that if you up weigh your examples with high loss you
 are going to up weigh all the mislabeled examples as well and thereby you're going
 to train more on the mislabeled examples and thereby you're going to possibly
 degrade your test performance much more than had you given every sample the same
 weight. And the authors address this actually nicely by doing an experiment
 and the experiment is what if we artificially mislabel examples how much
 can these algorithms tolerate. And so they often have these graphics here
 where they show test error over time. So test error and the x-axis here is number
 of back propped images which is kind of a time dimension in training these
 algorithms. You see the blue is a traditional model and the pink is a
 selective back prop with a 33% retain rate. So you see the selective back prop
 is much faster in reaching a low error and this first thing is simply with 1%
 of the labels shuffled. So 1% of the images are mislabeled. Selective back
 prop is still much faster than the traditional trajectory. If you go to 10%
 shuffled still you can see selective back prop is faster reaching a low error.
 Of course the error now generally is higher. But if you go to 20% here what
 you can see 20% shuffled labels what you can see it starts to become clear that
 these selective back prop it retains 33% of the hardest examples right. So and
 20% of the examples have a wrong label. That means most of what it upweighs are
 wrongly labeled examples. Almost let's say that there's still a lot of
 correctly labeled examples. But you see it gets to a low error but then it gets
 up again as it kind of massively overfits on these wrongly labeled examples
 because it upweighs them so much. Because here in still every
 example is hard right. So these wrongly labeled examples they'll get about the
 same weight as correctly labeled examples because the network isn't
 trained yet. But as you go lower it starts to massively overfit. So compared
 to the traditional model kind of just reaches this low error that okay is now
 corrupted by these wrong labels but it doesn't it doesn't hurt as much. So
 that's kind of my first criticism. If you have a lot of noisy labels or if you
 have a lot of mislabeled examples then this method might actually hurt more
 than it helps. But the level is interesting that it can kind of tolerate
 10% but it gets kind of into trouble at 20 or so more percent. So this is the
 first criticism and that's how the authors address it. I really like this
 ablation study that they do. Here this is kind of the meat of the experiment.
 So what they show here these curves on the bottom and let's look at this curve
 is on the x-axis you actually have wall clock time now. So how much time do you
 need in order to reach a kind of low error. Here is test set error. You see the
 traditional model in blue has a certain trajectory. Now cath 18 is a baseline
 don't worry about it. What we're interested in is the selective
 backprop which is the pink which you can see outperforms this traditional
 training. And what we're also interested in is the stale SB. So stale meaning it
 has this buffer of information that reduces it's supposed to reduce the time
 again. And you see that is even that even more outperforms the traditional
 approach. You can also see that the staleness here apparently doesn't hurt
 the performance too much. You see the error is fairly close and it reaches
 this error in a much faster time. This on CIFAR 10. They have this nice table up here
 where they show the speed up to reach a given error. So what they do is they take
 this error of the traditional model this test set error and they ask how fast are
 these methods in reaching this error times a constant. So times 1.1 times 1.2
 times 1.4 now. Of course the reaching 1.4 times the final error is much is
 easier and thereby but it's also easier for the traditional model of course. So
 that's the catch but these are kind of benchmarks they chose to how fast are
 these models in reaching 1.1 1.2 1.4 times the error of a traditionally
 trained model. You can see here on CIFAR 10 for example actually it's go to SVHN.
 SVHN is the easiest of the of the data sets and it shows the most clear thing.
 So the traditional error is 1.7% and you see that the speed up is so this
 selective back prop is 3.4 times faster in reaching this 1.1 times the
 error of this traditional model and it's also 3.4 times faster reaching 1.2
 times and it's 3.5 times faster in reaching it 1.4 times. The stale
 selective back prop is even faster so 4.3 4.9 5 times faster in reaching 1.4
 times this reaching 1.4 times the the error and so what you can what you can
 see here is that these methods really make it faster but also there's kind of
 two things two important things to note in this table. First of all you can see
 as you go to the right in the table the speed ups get higher and what it means
 is that as you need to reach as you make the problem easier so as you need to
 reach a higher error which is as you need to reach a higher loss value these
 methods are there faster what that means is they're really fast at reaching a
 somewhat decent point which is represented here they're really fast but
 if they need them to reach a more and more accurate performance they
 themselves get slower and slower so this this is of course clear because what
 you're doing is you're no longer treating every day to point the same you
 are introducing a bias into your training by only training on the hard
 examples so you're introducing a bias and this bias will give you a speed up
 but also hurt your performance and thereby if you have to get more and more
 accurate you will you will lose much of that speed up because you need to reduce
 that bias at the end that you introduced so that's the first caveat as you want
 to get to a higher and higher performance these methods will help less
 and less because they basically introduce the bias to gain speed at the
 beginning of training or to reach less accurate points the second thing is as
 you look at these problems here so SVH n 1.7 percent error C for 10 is a
 slightly harder problem 2.9 percent error and C for 100 is really a harder
 problem where a traditional model has 18 percent error if you look at the speed
 ups now then you can see even at this right most end here you have the 3.5 and
 5x speed up here we have a 1.5 2x speed up here we have a 1.2 1.6x speed up so
 as the problems get harder and as the kind of models get get fancier as the
 classes get more then the the speed up is much lower and I believe that's
 because the the bias you introduce by reweighing the samples the bias you
 introduce will hurt you much more on a difficult and large problem with a large
 network then it will hurt you on an easy problem right easy problem you were fine
 introducing some bias but if you have a hard noisy problem then this bias you
 introduce will hurt you much more and thereby this the speed up that these
 methods give you is much much less and so this means that the performance of
 these models is directly anti correlated with the hardness of the problem and
 that tells me it kind of makes it almost unusable or it goes towards if I look at
 the numbers if I look at the numbers over here and extrapolate that to
 something like image net it tells me that these methods are going to be
 almost useless on a data set of the size and complexity as image net and the
 interesting problems nowadays are very much in the domain of more hard more
 complex problems so the the kind of usefulness of this method in practice
 is something that I wouldn't bet on just from reading this paper I'm open to be
 convinced otherwise but just from reading this papers it seems like the
 harder you make the problem the less these methods help and that's exactly
 not what you want you want exactly the opposite you want to say oh if I scale
 this up it'll it'll you know give me even more of a speed up and that's going
 to be even better but this is the opposite so and given that they have no
 basically no theoretical analysis of how much this bias hurts you or how you can
 still make it kind of good in expectation how you would need to correct
 at the end and so on I would I would I would first of course test it I'm very
 interested to see tests on larger more complex problems but from this I'm a bit
 skeptical I'm sorry yeah so they they show I mean they show that on these
 states that it clearly helps clearly speeds up the training and that's of
 course that's already a good good thing and they do the required experiments
 they do the ablation studies on these data sets and so on so you can see here
 for example on these first graphics on all the data sets see clearly goes down
 as you introduce the more sophisticated algorithms but again you can see on the
 hard data set it doesn't go down as much all right but they do discuss this
 they're really fair to themselves they do risk they discuss this in their paper
 of how you know how practical this is and so on and what they what else they
 tried and didn't work and and that's a I think that it's a really good paper in
 itself and it's a really good investigation all right so that was it
 for me have a fun day bye bye