Processing Megapixel Images with Deep Attention-Sampling Models
machine learning deep learning research attention attention sampling attention model attention distribution megapixel images large images artificial intelligence megapixel mnist street sign dataset monte carlo speed memory cnn convolutional neural networks limited resources ai image recognition image classifier
Current CNNs have to downsample large images before processing them, which can lose a lot of detail information. This paper proposes attention sampling, which learns to selectively process parts of any large image in full resolution, while discarding uninteresting bits. This leads to enormous gains in speed and memory consumption.

https://arxiv.org/abs/1905.03711

Abstract:
Existing deep architectures cannot operate on very large signals such as megapixel images due to computational and memory constraints. To tackle this limitation, we propose a fully differentiable end-to-end trainable model that samples and processes only a fraction of the full resolution input image. The locations to process are sampled from an attention distribution computed from a low resolution view of the input. We refer to our method as attention sampling and it can process images of several megapixels with a standard single GPU setup. We show that sampling from the attention distribution results in an unbiased estimator of the full model with minimal variance, and we derive an unbiased estimator of the gradient that we use to train our model end-to-end with a normal SGD procedure. This new method is evaluated on three classification tasks, where we show that it allows to reduce computation and memory footprint by an order of magnitude for the same accuracy as classical architectures. We also show the consistency of the sampling that indeed focuses on informative parts of the input images.

Authors: Angelos Katharopoulos, François Fleuret
 Hi there, today we're looking at processing megapixel images with deep attention sampling models by Angelos Kateropoulos and François Fleuret. This is another paper that I saw the talk of at ICML and it's a pretty cool idea, it's pretty simple and apparently it works very well. So consider the following image here of a street situation and ask yourself if a self-driving car sees this, what are the kind of things it needs to be aware of? So of course one of the things it needs to be aware of is like the road, the cars and so on but also what's encircled in red here, the street sign and the street sign especially is important because there's a number on it and you want to see what the number is otherwise you won't be able to adjust your speed. So if this is now a really large image, so if the camera is really good and the dimensions of this image are really large, then current machine learning methods have a problem because current machine learning methods kind of go up to maybe something like 200 by 200 pixels or the current image net models, some down sample and so on. So if this is much larger than this, what current machine learning models would do is they would simply down sample, like compress the size, just compress it a bit and so on. And by that, as you see here on the right, if the original patch in the image you could cut it out and enlarge it, it would look like this. If you compress the whole image, the same patch would now look like this, blurred. So in the bottom half you'd be able to recognize the number, in the top half you wouldn't. So a standard CNN might be able to recognize the road and the car still at the lower resolution but not the speed sign. What we want is a method that can selectively pay attention to parts of the image that it finds interesting and then look at those parts in full detail while basically deciding to discard other parts completely such as the sky here. So this paper is one that does this and does so in a very efficient manner. So the basic premise is very simple. All right, I'm going to show you on this on the same image. So what you do is first you actually compress the image. So this image will become a smaller image, right? So here maybe this is 1000 by 2000, you compress it down to maybe 100 by 200. Still the same image but compressed. Here's the road, here's a bunch of trees. I'm very good at drawing trees. And here's this street sign and here is a car and here is another car. All right, so and there is a sky up here. So now what you do is on this smaller version you classify every location. I guess you could classify, you could subsample but you want to classify every single location on it on how interesting is it. And what they do is they take this and just put it through what they call an attention network which is just this it just a neural network. In their case it's a CNN that for each location here for each blue location outputs a function a of a and let's call it a x y at coordinates x and y of this image x. Okay, this is stupid notation. That's a of x so the image is x at coordinates i, j. Right, so all of these blue things here are i's and j's. Different i's and j's. And then what does this gives you now if you normalize correctly, so if you normalize over all the a's and i, j, a, i, j. If you normalize this gives you a distribution over this image. So if we look at it in like 1D this gives you like a distribution not a continuous one in this case a discrete one. How interesting is each patch and at the end if you have this distribution, so let's finish here, what you want to do is you want to say which are the most interesting locations. So this one's pretty high and these are very high so that might correspond to over here that might correspond to some location. So this location is very high and these locations are very interesting and only in these locations you take them out and then only those you process in full resolution. So you might have extracted let's say four patches so now you have four of these patches and each of them individually you run through a second neural network which is called another CNN which is called F the feature network. So the feature network will take a patch and output a vector of features. So it will feed those in and output the vector of features and then what you do is you simply your final output which they call G, let me colorize this so G which is G is now the final output let's not call it G let's call it O. Output is you sum over all the patches you have extracted down here so the patch number P over all your patches and you sum these features F of patch P right and P might be at location IJ let's put IJ here so IJ in the extracted patches and you weigh each feature by how much attention it got at that location. So it looks more complicated than it is what you do is you simply determine these features by using this neural network only at the position where this neural network says are interesting then you get the features from the interesting positions and you basically just weigh them by how much attention they got in the attention distribution and that will be your final output of the network and it makes intuitive sense like one network decides what is interesting the other network decides what are we going to do with the interesting things in this image. And the cool thing about this is you can basically decide how many of these patches here how many you want to extract you can decide at what resolution you want to process this image and all of this are parameters that you set by how much time you have for computation and how much memory you have for your computation so that's pretty cool pretty module we can scale up we can scale down and the another cool thing is the theoretical guarantees that they give so basically here they prove that the way they do it especially by extracting the patch especially if they have an unbiased sorry especially have if they have sampling without replacement is that if they weigh the things correctly and if they do the things correctly they show that this is actually an unbiased estimator of the true neural network if you were to evaluate on the full image basically on each patch in full resolution so only taking the ones where the attention focuses is an unbiased estimator and not only is it an unbiased estimator it is in fact the estimator with the smallest variance and that's what they prove here so the minimum variance estimator and this is this is pretty pretty interesting pretty cool and works pretty well they also show how to derive the gradient update when you train with this attention sampling so now you train your neural you train your machine learning system not on the whole image but only on a subset of the image patches but it still behaves in expectation as if you were to train on the entire image so pretty neat so here they show how this compares to full CNN in this case we have the full CNN where the picture is simply down sampled and then classified and this is what's called megapixel amnest so in megapixel amnest you have a large image and you put three digits in there there are the same for example five five five from the amnest data set you put two random digits others like two three and you put also a bunch of noise noise patches somewhere so the task is to recognize which is the dominant digit here in this case it would be five right five five where was the other one five here so if you give this to a regular CNN you see it does about this well this is the training loss here training loss and this is the test loss and it takes this much time right time per epoch here and this much time to evaluate sorry if you now use this attention sampling and as I said you can actually modulate how many patches you want to take so as you go down you take more patches we would expect it to take more time this is exactly what happens you see for example down here in the test error if you take five patches per image it takes very little time but the error I mean the error is still better than the if you use the CNN simply because you can now pay attention to details much more as you use more patches your test error drops the also your training loss they drop so using more patches will be actually give you a better and better and better performing model but you sacrifice a little bit of time but still not never as as slow as with the full with that with the CNN so even though it's a down sampled CNN right so that is very interesting and very cool that not only do they beat the the baseline in terms of error but also a lot in terms of speed if you look at what the model does as it learns here you see for a given image this is always the same image from the data set at the beginning they have actually marked where the relevant the three relevant digits are in the picture with the red circle so if you look at how over the training of this model how this distribution evolves is pretty interesting yellow basically means high attention so at the beginning you have high attention everywhere in the image right and then as you go on and on and on you see for example here it pays attention to all the locations where basically where there is something in the image right this could be one of these three digits but it could also be one of the digits that it's trying to that is trying to distract the model like the false digits or the noise patches and as you go more and more and more it really learns to only pay attention to the relevant digits and then classify those at full resolution so this really shows the this this kind of attention distribution learns something very meaningful they do more experiments on two data sets namely this is a histopathology data set right here where the goal is I think to recognize this epithelial cells this type of cell and you can see that this here is the baseline and this here is the new method and the baseline basically what it does is it does similar thing namely it processes the image in patches but it processes every single patch maybe in succession but it still processes every single patch where the attention sampling only processes the patches that the attention sampling distribution suggests and this other data set here is a street sign data set that you saw at the beginning right here and the the again I think this is the baseline and this is the attention sample so both learn to pay attention to the street signs but again the attention sampling much more efficient so here you see the baseline performance the attention sampling performance is similar in terms of test error but if you look at how much time the baseline uses per sample and how much memory and then compare this to the attention sampling you see that they save at least an order of magnitude in time and memory and the same thing goes for the street sign data set you see test error here and then test error is similar for the attention sampling but again time memory much much lower so the attention sampling is faster and is more memory efficient than the baseline and that makes it makes it easy to process these megapixel images even on here they say process megapixel images in a single CPU or GPU and that really I like this because it kind of brings their research back to let's say regular people or maybe universities that don't have as much money as large companies and so all in all very cool paper very neat experiments to have a lot in the appendix check it out where they show their attention distribution in these images their theoretical analysis is pretty easy to follow if you want to check that out and with that thanks for listening and bye bye
 Hi there, today we're looking at processing megapixel images with deep
 attention sampling models by Angelos Kateropoulos and François Fleuret.
 This is another paper that I saw the talk of at ICML and it's a pretty cool idea,
 it's pretty simple and apparently it works very well. So consider the
 following image here of a street situation and ask yourself if a
 self-driving car sees this, what are the kind of things it needs to be aware of?
 So of course one of the things it needs to be aware of is like the road, the cars
 and so on but also what's encircled in red here, the street sign and the street
 sign especially is important because there's a number on it and you want to
 see what the number is otherwise you won't be able to adjust your speed. So if
 this is now a really large image, so if the camera is really good and the
 dimensions of this image are really large, then current machine learning
 methods have a problem because current machine learning methods kind of go up
 to maybe something like 200 by 200 pixels or the current image net models,
 some down sample and so on. So if this is much larger than this, what current
 machine learning models would do is they would simply down sample, like compress
 the size, just compress it a bit and so on. And by that, as you see here on the
 right, if the original patch in the image you could cut it
 out and enlarge it, it would look like this. If you compress the whole image, the
 same patch would now look like this, blurred. So in the bottom half you'd be
 able to recognize the number, in the top half you wouldn't. So a standard CNN might
 be able to recognize the road and the car still at the lower resolution but
 not the speed sign. What we want is a method that can selectively pay
 attention to parts of the image that it finds interesting and then look at those
 parts in full detail while basically deciding to discard other parts
 completely such as the sky here. So this paper is one that does this and does so
 in a very efficient manner. So the basic premise is very simple. All right, I'm
 going to show you on this on the same image. So what you do is first you
 actually compress the image. So this image will become a smaller image, right?
 So here maybe this is 1000 by 2000, you compress it down to maybe 100 by 200.
 Still the same image but compressed. Here's the road, here's a bunch of
 trees. I'm very good at drawing trees. And here's this street sign and here is a
 car and here is another car. All right, so and there is a sky up here. So now
 what you do is on this smaller version you classify every location. I guess
 you could classify, you could subsample but you want to classify every single
 location on it on how interesting is it. And what they do is they take this and
 just put it through what they call an attention network which is just this it
 just a neural network. In their case it's a CNN that for each location here for
 each blue location outputs a function a of a and let's call it a x y at
 coordinates x and y of this image x. Okay, this is stupid notation. That's a of x
 so the image is x at coordinates i, j. Right, so all of these blue things here
 are i's and j's. Different i's and j's. And then what does this gives you now if
 you normalize correctly, so if you normalize over all the a's and i, j, a, i, j. If you
 normalize this gives you a distribution over this image. So if we look at it in
 like 1D this gives you like a distribution not a continuous one in
 this case a discrete one. How interesting is each patch and at the end if you have
 this distribution, so let's finish here, what you want to do is you want to say
 which are the most interesting locations. So this one's pretty high and these are
 very high so that might correspond to over here that might correspond to some
 location. So this location is very high and these locations are very interesting
 and only in these locations you take them out and then only those you process
 in full resolution. So you might have extracted let's say four patches so now
 you have four of these patches and each of them individually you run through a
 second neural network which is called another CNN which is called F the
 feature network. So the feature network will take a patch and output a vector of
 features. So it will feed those in and output the vector of features and
 then what you do is you simply your final output which they call G, let me
 colorize this so G which is G is now the final output let's not call it G let's
 call it O. Output is you sum over all the patches you have extracted down here so
 the patch number P over all your patches and you sum these features F of patch P
 right and P might be at location IJ let's put IJ here so IJ in the extracted
 patches and you weigh each feature by how much attention it got at that
 location. So it looks more complicated than it is what you do is you
 simply determine these features by using this neural network only at the position
 where this neural network says are interesting then you get the features
 from the interesting positions and you basically just weigh them by how much
 attention they got in the attention distribution and that will be your final
 output of the network and it makes intuitive sense like one network decides
 what is interesting the other network decides what are we going to do with the
 interesting things in this image. And the cool thing about this is you
 can basically decide how many of these patches here how many you want to
 extract you can decide at what resolution you want to process this
 image and all of this are parameters that you set by how much time you have
 for computation and how much memory you have for your computation so that's
 pretty cool pretty module we can scale up we can scale down and the another cool
 thing is the theoretical guarantees that they give so basically here they prove
 that the way they do it especially by extracting the patch especially if they
 have an unbiased sorry especially have if they have sampling without replacement
 is that if they weigh the things correctly and if they do the things
 correctly they show that this is actually an unbiased estimator of the
 true neural network if you were to evaluate on the full image basically on
 each patch in full resolution so only taking the ones where the attention
 focuses is an unbiased estimator and not only is it an unbiased estimator it is
 in fact the estimator with the smallest variance and that's what they prove
 here so the minimum variance estimator and this is this is pretty pretty
 interesting pretty cool and works pretty well they also show how to derive the
 gradient update when you train with this attention sampling so now you train your
 neural you train your machine learning system not on the whole image but only
 on a subset of the image patches but it still behaves in expectation as if you
 were to train on the entire image so pretty neat so here they show how this
 compares to full CNN in this case we have the full CNN where the picture is
 simply down sampled and then classified and this is what's called megapixel
 amnest so in megapixel amnest you have a large image and you put three digits in
 there there are the same for example five five five from the amnest data set
 you put two random digits others like two three and you put also a bunch of
 noise noise patches somewhere so the task is to recognize which is the
 dominant digit here in this case it would be five right five five where was
 the other one five here so if you give this to a regular CNN you see it does
 about this well this is the training loss here training loss and this is the
 test loss and it takes this much time right time per epoch here and this much
 time to evaluate sorry if you now use this attention sampling and as I said
 you can actually modulate how many patches you want to take so as you go
 down you take more patches we would expect it to take more time this is
 exactly what happens you see for example down here in the test error if you take
 five patches per image it takes very little time but the error I mean the
 error is still better than the if you use the CNN simply because you can now
 pay attention to details much more as you use more patches your test error
 drops the also your training loss they drop so using more patches will be
 actually give you a better and better and better performing model but you
 sacrifice a little bit of time but still not never as as slow as with the full
 with that with the CNN so even though it's a down sampled CNN right so that
 is very interesting and very cool that not only do they beat the the baseline
 in terms of error but also a lot in terms of speed if you look at what the
 model does as it learns here you see for a given image this is always the same
 image from the data set at the beginning they have actually marked where the
 relevant the three relevant digits are in the picture with the red circle so if
 you look at how over the training of this model how this distribution evolves
 is pretty interesting yellow basically means high attention so at the beginning
 you have high attention everywhere in the image right and then as you go on and
 on and on you see for example here it pays attention to all the locations
 where basically where there is something in the image right this could be one of
 these three digits but it could also be one of the digits that it's trying to
 that is trying to distract the model like the false digits or the noise
 patches and as you go more and more and more it really learns to only pay
 attention to the relevant digits and then classify those at full resolution
 so this really shows the this this kind of attention distribution learns
 something very meaningful they do more experiments on two data sets namely this
 is a histopathology data set right here where the goal is I think to recognize
 this epithelial cells this type of cell and you can see that this here is the
 baseline and this here is the new method and the baseline basically what it does
 is it does similar thing namely it processes the image in patches but it
 processes every single patch maybe in succession but it still processes every
 single patch where the attention sampling only processes the patches that
 the attention sampling distribution suggests and this other data set here
 is a street sign data set that you saw at the beginning right here and the
 the again I think this is the baseline and this is the attention sample so both
 learn to pay attention to the street signs but again the attention sampling
 much more efficient so here you see the baseline performance the attention
 sampling performance is similar in terms of test error but if you look at how
 much time the baseline uses per sample and how much memory and then compare
 this to the attention sampling you see that they save at least an order of
 magnitude in time and memory and the same thing goes for the street sign
 data set you see test error here and then test error is similar for the
 attention sampling but again time memory much much lower so the attention
 sampling is faster and is more memory efficient than the baseline and that
 makes it makes it easy to process these megapixel images even on here they say
 process megapixel images in a single CPU or GPU and that really I like this
 because it kind of brings their research back to let's say regular people or
 maybe universities that don't have as much money as large companies and so all
 in all very cool paper very neat experiments to have a lot in the
 appendix check it out where they show their attention distribution in these
 images their theoretical analysis is pretty easy to follow if you want to
 check that out and with that thanks for listening and bye bye