Deep image reconstruction from human brain activity (Paper Explained)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper fmri mind reading thoughts visual cortex vc v1 v4 vgg reconstruction iterative deep dream microscope activity imagine visualize introspection human telepathy
Can you peek into people's brains? Reading human thoughts is a long-standing dream of the AI field. This paper reads fMRI signals from a person and then reconstructs what that person's eyes currently see. This is achieved by translating the fMRI signal to features of a Deep Neural Network and then iteratively optimizing the input of the network to match those features. The results are impressive.

OUTLINE:
0:00 - Overview
1:35 - Pipeline
4:00 - Training
5:20 - Image Reconstruction
7:00 - Deep Generator Network
8:15 - Results

Paper: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1006633
My Video on OpenAI Microscope (what I called Atlas): https://youtu.be/Ok44otx90D4

Abstract:
The mental contents of perception and imagery are thought to be encoded in hierarchical representations in the brain, but previous attempts to visualize perceptual contents have failed to capitalize on multiple levels of the hierarchy, leaving it challenging to reconstruct internal imagery. Recent work showed that visual cortical activity measured by functional magnetic resonance imaging (fMRI) can be decoded (translated) into the hierarchical features of a pre-trained deep neural network (DNN) for the same input image, providing a way to make use of the information from hierarchical visual features. Here, we present a novel image reconstruction method, in which the pixel values of an image are optimized to make its DNN features similar to those decoded from human brain activity at multiple layers. We found that our method was able to reliably produce reconstructions that resembled the viewed natural images. A natural image prior introduced by a deep generator neural network effectively rendered semantically meaningful details to the reconstructions. Human judgment of the reconstructions supported the effectiveness of combining multiple DNN layers to enhance the visual quality of generated images. While our model was solely trained with natural images, it successfully generalized to artificial shapes, indicating that our model was not simply matching to exemplars. The same analysis applied to mental imagery demonstrated rudimentary reconstructions of the subjective content. Our results suggest that our method can effectively combine hierarchical neural representations to reconstruct perceptual and subjective images, providing a new window into the internal contents of the brain.

Authors: Guohua Shen, Tomoyasu Horikawa, Kei Majima, Yukiyasu Kamitani

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there! Today we're looking at deep image reconstruction from human brain activity by Gwa-wa Shen, Tomoyasu Horikawa, Kai Majima and Yuki Yazu Kamitani. This is like the reading thoughts. So I was excited when I saw this paper. I saw this on reddit and it is a bit older. It is from the beginning of last year. So I'm sure there have been developments in this area. But basically what this paper does is it will have a human look at a picture as you can see for example right up here. It will measure the MRI activity. Then it will use a what they call a feature decoder in order to map that MRI activity to features of a deep neural network. Then they will reconstruct the image that is closest to those features in the neural network. By reconstruction they get basically an image of what the human sees. So we're going to explore this pipeline right here. But it's pretty cool and if it works it basically means that we can read someone's thoughts. But of course there's going to be issues and problems. So first of all this is all visual. They measure the activity in the visual cortex right here. Let's break it down to the individual parts. The individual parts here, the fMRI, that is a machine. We cannot control. So you measure the fMRI activity and that basically measures which of these cells in your brain use oxygen. It's functional fMRI, it's not structural. So it measures which ones are active and that's how you would see which parts of the brains are active. I think the resolution on these things has gotten very very good. So you can make out very fine grained activation patterns in the neurons. And they measure the visual cortex which is the part responsible for basically visual stimuli. So for seeing things. Now they need this feature decoder but because ultimately what they want to do is they want to have these features correspond to features in a neural network. This DNN here is like a VGG, I think a VGG-16 or a VGG-19. It's called a VGG-16 network. This is a couple of years ago these architectures were very popular for ImageNet and they're fairly basic. So what that means it's not like a super duper inception net where you have like layers within layers and so on. They're pretty straightforward convolutional neural networks with nonlinearities and pooling here. You can see there is a bunch of layers, then there's pooling, there is a bunch of layers, there's pooling and so on. So what you want to look at are these layers of the deep neural network. The individual layers and you want to basically put an image right here into the neural network and then observe its features in the neural network. Then you want to put the same image through the human and then you observe the MRI features. You know that this is the same image so you can basically learn a feature decoder. This is going to be another sort of machine learned maybe in neural network. I haven't actually read this could be just like a linear regression or a neural network. Just a regression that maps the fMRI to the features. So this is what you have to learn. Basically they took a bunch of humans, stuck them in an MRI machine, got out their their fMRI data for the same image. So for a given image X they got the human fMRI data and they got the VGG. They got the features when they put the X, the image through the neural network and now they learn a function that minimizes the error. So there is an error. So basically they run this on a test set of images and then you end up with a function that can basically map fMRI features to neural network features. Now the second step, now you can give the human, let's say this works, you can give the human an arbitrary image and the human will basically interpret it through its visual cortex. You measure the activity in the visual cortex and then you can predict the neural network features if that image were given to the neural network. But now you don't give the image to the neural network. Instead of what you do, you do something like deep dream does. That means you start from a noisy image at the beginning and you try to find the image. So you start from this noisy image right here and through iterative gradient descent you refine this image. That's this arrow right here. You try to find the image that as closely as possible in the internal representation matches the features that you predict from the fMRI signal. Because these are the features that the neural network should see. If your feature decoder is good, these are the features that the neural network should output for that image. So you're basically trying to find this image right here but you're not looking at it. You only look at the features that should be in the neural network. So after a bunch of steps of refining that image, you hope you basically end up with an image that corresponds to these features. Then you can look at it. It usually looks something like this. We're used to these kind of things from neural networks. If I invite you to look at something like the OpenAI Atlas, if you want to understand how this is done. But basically we can get the image that most faithfully corresponds to these features. This doesn't always work super well because there are actually, since the neural network is sort of a dimensional reduction technique, there are many images that correspond to the same features and they often end up like really weird. So what they do is they have this deep generator network as a prior. Basically this is the generator from a GAN, from a Generative Adversarial Network. So this network right here is really good at just producing naturally looking images. And now because we have that, our task is not going to be to start from this image right here. Our task will be to do the exact same thing but with the input to the deep generator network. So basically what we're trying to do is we're trying to find the input vector to the deep generator network such that these features right here correspond to the features that we predict from the fMRI activity. And because the deep generator network is trained to produce natural looking images, this will always give us sort of a natural looking image no matter what our input vector is. So thereby we basically constrain the optimization procedure to only output natural looking images. So let's see how well this works. So these are the reconstructions. I have to say they're training set, they're training and testing set. So up here, this procedure where we learned right here, this would be done on a training set of images. And then they expose the humans to testing set of images. So this reconstruction right here would happen on images that the feature decoder wasn't trained on. But the humans are looking at it. So the humans would be looking at the picture here on the left. And then this would be to the right you'll see as more and more iterations of this process of reconstructing happens, the image gets basically clearer and clearer. And you can see on the right you get pretty good looking images for the ones on the left. Now, these researchers they tend to they tout the success like to say, oh, look, look at this. But you know, honestly, like this is a leopard. This is a dog sheep. This is an owl. This is a dog. So, so, you know, this is a fish. And this is a like a shell, like a muscle. So I'm and you can go through these. And basically, you know, this is a sled. And then this is a truck. So they go basically, and they say, wow, the accuracy is really good. So they do a pixel correlation accuracy, which is, you know, you just try to pixel correlate things, but they have human judgment, the accuracy via human judgment is over 95%. That's crazy. But how do they do it? So basically, they tell you, okay, see, this is the image at the beginning, no, sorry, they give you this image right here as a human radar. And then they give you two other images, they say, okay, here are two images, let's say these two, which one did it come from? And if the human can can determine it correctly counts as a hit. So the baseline probability here is 50%. So basically, right, is so this right here, is it rather the owl? Or is it rather the VCR? And I mean, in that respect, it's pretty impressive what you can read from a brain, but in no way, like zero way, this is reading your thoughts. This isn't like, it seems to basically just reconstruct a example from the ImageNet training set. And the ImageNet Explorer is down right now. So I, I try to look at this. But it seems to me it's just kind of reconstructing something it knows that sort of bit resembles the image on the left. Yeah, but it is not it is not reconstructing that image. Not at all. Like look, like a bit, but vaguely, vaguely. But they do some more investigation into this. Okay, so well, first of all, here you can see what happens without this deep generator network. So when you have unconstrained search, then it is even worse, right? You get like big pixel meshes right here. So you need this this kind of prior over natural images. But the prior, I think the prior here comes through a bit much because I the prior might be in part responsible for why the images just show something else than you see. They go into an investigation of if and they discover if you use more layers to reconstruct, the reconstruction gets better. So here, according to human judgment, if you just reconstruct from the first layer, you don't get very good reconstruction. But if you incorporate the signal across many layers of the neural network, so you're basically trying to match to predict many layers of signal different layers, then the reconstruction gets really good. And, you know, we know this from things like style transfer, you can modify how close you are to the original or to the target by basically seeing which layers and how many you reconstruct to which accuracy. So this makes kind of sense. So if you if you only take the first layer to match the features, then you get basically this blob here. But if you get layers one through seven, you get pretty pretty okay ish thing that looks like this thing. I guess these are without the deep generator network so far. But no, that is interesting. And I think the novel thing here is one of the novel thing is that they actually use multiple layers of the neural network to reconstruct. The interesting thing is, I think this is this is pretty cool. They can now do this with these shapes. So these shapes aren't natural images, and they have not been seen in training. But still, as you can see, when the human sees, for example, the plus shape, it will get you pretty clear plus shape. And it happens for a lot of these things right here. So these are actually, I would say, fairly okay ish reconstructions of what the human sees. Here neuron, that is fairly neat. And for the alphabetical letters and shapes, you see again, the even the pixel correlation now is pretty high, but the human judgment again high. And here the human judgment kind of makes sense, right? If you ask, is it like this shape or this shape? Then it makes more sense to evaluate it like this. So the shapes, I am fairly impressed that they can reconstruct these shapes. And what they're now trying to do is they're trying to infer imagined images. So basically, they're telling a human, please imagine an image, and they show it to the human. It's not really imagining, it's basically recalling, they show you this image. And then you whatever close your eyes, they take the image away, and you just try to imagine that. And you can see through the reconstruction process, that works out, you know, sort of ish, right? You can see that the cross here, it kind of comes through. And this the plus kind of it sort of comes through. And so these are the high accuracy, these are actually the samples where where it worked. And there are also samples where it didn't work, like here, where you see it doesn't really come through. So there, either there is, you know, really a difference between imagining something and seeing something, or this method just isn't very good, per se, and you actually need or humans aren't really good at imagining, like there's lots of explanations. And here is the same thing if you imagine these images. Now they report that if humans just recall or imagine these images, then the reconstruction doesn't work at all. So that might be to the fact that, you know, you cannot in your recollection, you basically just remember the important things about something, you don't remember the exact pixel values, and therefore, your visual cortex doesn't respond in the same way. But I mean, it's interesting, even per se to think about it. But I have my doubts about you know, this entire system. So I don't want to make too many conclusions here about these things. Suffice to say that sometimes it actually can read your thoughts. Because if you just think so this is the stuff up here, if you just think of a shape, it can sort of kind of a bit make out the shape that you're thinking about. Alright, this was it for this paper. I'm basically mainly wanted to show you what I found. And I'm I have to say I'm pretty impressed with this, even though like, this is a laptop. This is not a VCR. This is a VCR. It's, it's more of a nearest neighbor thing, really, than I reconstruction, I think. But that's my opinion, right? Yes. So if you like this, give it a like, subscribe if you're still here. And I look forward to next time. Bye bye.
 Hi there! Today we're looking at deep image reconstruction from human brain activity by
 Gwa-wa Shen, Tomoyasu Horikawa, Kai Majima and Yuki Yazu Kamitani. This is like the reading thoughts.
 So I was excited when I saw this paper. I saw this on reddit and it is a bit older. It is from the
 beginning of last year. So I'm sure there have been developments in this area. But basically what this
 paper does is it will have a human look at a picture as you can see for example right up here.
 It will measure the MRI activity. Then it will use a what they call a feature decoder in order to
 map that MRI activity to features of a deep neural network. Then they will reconstruct the image that
 is closest to those features in the neural network. By reconstruction they get basically
 an image of what the human sees. So we're going to explore this pipeline right here. But it's pretty
 cool and if it works it basically means that we can read someone's thoughts. But of course there's
 going to be issues and problems. So first of all this is all visual. They measure the activity in
 the visual cortex right here. Let's break it down to the individual parts. The individual parts here,
 the fMRI, that is a machine. We cannot control. So you measure the fMRI activity and that basically
 measures which of these cells in your brain use oxygen. It's functional fMRI, it's not structural.
 So it measures which ones are active and that's how you would see which parts of the brains are
 active. I think the resolution on these things has gotten very very good. So you can make out very
 fine grained activation patterns in the neurons. And they measure the visual cortex which is the
 part responsible for basically visual stimuli. So for seeing things. Now they need this feature
 decoder but because ultimately what they want to do is they want to have these features correspond
 to features in a neural network. This DNN here is like a VGG, I think a VGG-16 or a VGG-19. It's
 called a VGG-16 network. This is a couple of years ago these architectures were very popular for
 ImageNet and they're fairly basic. So what that means it's not like a super duper inception net
 where you have like layers within layers and so on. They're pretty straightforward convolutional
 neural networks with nonlinearities and pooling here. You can see there is a bunch of layers,
 then there's pooling, there is a bunch of layers, there's pooling and so on. So what you want to
 look at are these layers of the deep neural network. The individual layers and you want to
 basically put an image right here into the neural network and then observe its features in the
 neural network. Then you want to put the same image through the human and then you observe the
 MRI features. You know that this is the same image so you can basically learn a feature decoder. This
 is going to be another sort of machine learned maybe in neural network. I haven't actually read
 this could be just like a linear regression or a neural network. Just a regression that maps the
 fMRI to the features. So this is what you have to learn. Basically they took a bunch of humans,
 stuck them in an MRI machine, got out their their fMRI data for the same image. So for a given image
 X they got the human fMRI data and they got the VGG. They got the features when they put the X,
 the image through the neural network and now they learn a function that minimizes the error. So there
 is an error. So basically they run this on a test set of images and then you end up with a function
 that can basically map fMRI features to neural network features. Now the second step, now you
 can give the human, let's say this works, you can give the human an arbitrary image and the
 human will basically interpret it through its visual cortex. You measure the activity in the
 visual cortex and then you can predict the neural network features if that image were given to the
 neural network. But now you don't give the image to the neural network. Instead of what you do,
 you do something like deep dream does. That means you start from a noisy image at the beginning and
 you try to find the image. So you start from this noisy image right here and through iterative
 gradient descent you refine this image. That's this arrow right here. You try to find the image
 that as closely as possible in the internal representation matches the features that you
 predict from the fMRI signal. Because these are the features that the neural network should see.
 If your feature decoder is good, these are the features that the neural network should output
 for that image. So you're basically trying to find this image right here but you're not looking at
 it. You only look at the features that should be in the neural network. So after a bunch of steps
 of refining that image, you hope you basically end up with an image that corresponds to these
 features. Then you can look at it. It usually looks something like this. We're used to these
 kind of things from neural networks. If I invite you to look at something like the OpenAI Atlas,
 if you want to understand how this is done. But basically we can get the image that most
 faithfully corresponds to these features. This doesn't always work super well because there
 are actually, since the neural network is sort of a dimensional reduction technique, there are many
 images that correspond to the same features and they often end up like really weird. So what they
 do is they have this deep generator network as a prior. Basically this is the generator from a GAN,
 from a Generative Adversarial Network. So this network right here is really good at just producing
 naturally looking images. And now because we have that, our task is not going to be to start from
 this image right here. Our task will be to do the exact same thing but with the input to the deep
 generator network. So basically what we're trying to do is we're trying to find the input vector to
 the deep generator network such that these features right here correspond to the features that we
 predict from the fMRI activity. And because the deep generator network is trained to produce
 natural looking images, this will always give us sort of a natural looking image no matter what
 our input vector is. So thereby we basically constrain the optimization procedure to only
 output natural looking images. So let's see how well this works. So these are the reconstructions.
 I have to say they're training set, they're training and testing set. So up here, this
 procedure where we learned right here, this would be done on a training set of images. And then they
 expose the humans to testing set of images. So this reconstruction right here would happen on
 images that the feature decoder wasn't trained on. But the humans are looking at it. So the humans
 would be looking at the picture here on the left. And then this would be to the right you'll see
 as more and more iterations of this process of reconstructing happens, the image gets basically
 clearer and clearer. And you can see on the right you get pretty good looking images for the ones on
 the left. Now, these researchers they tend to they tout the success like to say, oh, look, look at
 this. But you know, honestly, like this is a leopard. This is a dog sheep. This is an owl. This
 is a dog. So, so, you know, this is a fish. And this is a like a shell, like a muscle. So I'm and
 you can go through these. And basically, you know, this is a sled. And then this is a truck. So they
 go basically, and they say, wow, the accuracy is really good. So they do a pixel correlation
 accuracy, which is, you know, you just try to pixel correlate things, but they have human
 judgment, the accuracy via human judgment is over 95%. That's crazy. But how do they do it? So
 basically, they tell you, okay, see, this is the image at the beginning, no, sorry, they give you
 this image right here as a human radar. And then they give you two other images, they say, okay,
 here are two images, let's say these two, which one did it come from? And if the human can can
 determine it correctly counts as a hit. So the baseline probability here is 50%. So basically,
 right, is so this right here, is it rather the owl? Or is it rather the VCR? And I mean, in that
 respect, it's pretty impressive what you can read from a brain, but in no way, like zero way, this
 is reading your thoughts. This isn't like, it seems to basically just reconstruct a example from the
 ImageNet training set. And the ImageNet Explorer is down right now. So I, I try to look at this.
 But it seems to me it's just kind of reconstructing something it knows that sort of bit resembles the
 image on the left. Yeah, but it is not it is not reconstructing that image. Not at all. Like look,
 like a bit, but vaguely, vaguely. But they do some more investigation into this. Okay, so well,
 first of all, here you can see what happens without this deep generator network. So when you have
 unconstrained search, then it is even worse, right? You get like big pixel meshes right here. So you
 need this this kind of prior over natural images. But the prior, I think the prior here comes through
 a bit much because I the prior might be in part responsible for why the images just show something
 else than you see. They go into an investigation of if and they discover if you use more layers to
 reconstruct, the reconstruction gets better. So here, according to human judgment, if you just
 reconstruct from the first layer, you don't get very good reconstruction. But if you incorporate
 the signal across many layers of the neural network, so you're basically trying to match to
 predict many layers of signal different layers, then the reconstruction gets really good. And,
 you know, we know this from things like style transfer, you can modify how close you are to
 the original or to the target by basically seeing which layers and how many you reconstruct to which
 accuracy. So this makes kind of sense. So if you if you only take the first layer to match the
 features, then you get basically this blob here. But if you get layers one through seven, you get
 pretty pretty okay ish thing that looks like this thing. I guess these are without the deep generator
 network so far. But no, that is interesting. And I think the novel thing here is one of the novel
 thing is that they actually use multiple layers of the neural network to reconstruct. The interesting
 thing is, I think this is this is pretty cool. They can now do this with these shapes. So these
 shapes aren't natural images, and they have not been seen in training. But still, as you can see,
 when the human sees, for example, the plus shape, it will get you pretty clear plus shape. And it
 happens for a lot of these things right here. So these are actually, I would say, fairly okay ish
 reconstructions of what the human sees. Here neuron, that is fairly neat. And for the alphabetical
 letters and shapes, you see again, the even the pixel correlation now is pretty high, but the human
 judgment again high. And here the human judgment kind of makes sense, right? If you ask, is it like
 this shape or this shape? Then it makes more sense to evaluate it like this. So the shapes, I am
 fairly impressed that they can reconstruct these shapes. And what they're now trying to do is they're
 trying to infer imagined images. So basically, they're telling a human, please imagine an image,
 and they show it to the human. It's not really imagining, it's basically recalling, they show
 you this image. And then you whatever close your eyes, they take the image away, and you just try
 to imagine that. And you can see through the reconstruction process, that works out, you know,
 sort of ish, right? You can see that the cross here, it kind of comes through. And this the plus
 kind of it sort of comes through. And so these are the high accuracy, these are actually the samples
 where where it worked. And there are also samples where it didn't work, like here, where you see it
 doesn't really come through. So there, either there is, you know, really a difference between
 imagining something and seeing something, or this method just isn't very good, per se, and you
 actually need or humans aren't really good at imagining, like there's lots of explanations. And
 here is the same thing if you imagine these images. Now they report that if humans just recall or
 imagine these images, then the reconstruction doesn't work at all. So that might be to the fact
 that, you know, you cannot in your recollection, you basically just remember the important things
 about something, you don't remember the exact pixel values, and therefore, your visual cortex
 doesn't respond in the same way. But I mean, it's interesting, even per se to think about it. But I
 have my doubts about you know, this entire system. So I don't want to make too many conclusions here
 about these things. Suffice to say that sometimes it actually can read your thoughts. Because if you
 just think so this is the stuff up here, if you just think of a shape, it can sort of kind of a
 bit make out the shape that you're thinking about. Alright, this was it for this paper. I'm basically
 mainly wanted to show you what I found. And I'm I have to say I'm pretty impressed with this, even
 though like, this is a laptop. This is not a VCR. This is a VCR. It's, it's more of a nearest neighbor
 thing, really, than I reconstruction, I think. But that's my opinion, right? Yes. So if you like this,
 give it a like, subscribe if you're still here. And I look forward to next time. Bye bye.