Multimodal Neurons in Artificial Neural Networks (w/ OpenAI Microscope, Research Paper Explained)
deep learning machine learning explained neural networks ai artificial intelligence paper openai openai emotions openai dalle openai clip openai microscope openai clip microscope alec radford emotion neuron deep learning emotion chris olah chris olah openai neural network feature visualization multimodal neural network what does a neural network learn what do neural networks learn how do neural networks work what does openai do faceted visualization
#openai #clip #microscope

OpenAI does a huge investigation into the inner workings of their recent CLIP model via faceted feature visualization and finds amazing things: Some neurons in the last layer respond to distinct concepts across multiple modalities, meaning they fire for photographs, drawings, and signs depicting the same concept, even when the images are vastly distinct. Through manual examination, they identify and investigate neurons corresponding to persons, geographical regions, religions, emotions, and much more. In this video, I go through the publication and then I present my own findings from digging around in the OpenAI Microscope.

OUTLINE:
0:00 - Intro & Overview
3:35 - OpenAI Microscope
7:10 - Categories of found neurons
11:10 - Person Neurons
13:00 - Donald Trump Neuron
17:15 - Emotion Neurons
22:45 - Region Neurons
26:40 - Sparse Mixture of Emotions
28:05 - Emotion Atlas
29:45 - Adversarial Typographic Attacks
31:55 - Stroop Test
33:10 - My Findings in OpenAI Microscope
33:30 - Superman Neuron
33:50 - Resting B*tchface Neuron
34:10 - Trash Bag Neuron
35:25 - God Weightlifting Neuron
36:40 - Organ Neuron
38:35 - Film Spool Neuron
39:05 - Feather Neuron
39:20 - Spartan Neuron
40:25 - Letter E Neuron
40:35 - Cleanin Neuron
40:45 - Frown Neuron
40:55 - Lion Neuron
41:05 - Fashion Model Neuron
41:20 - Baseball Neuron
41:50 - Bride Neuron
42:00 - Navy Neuron
42:30 - Hemp Neuron
43:25 - Staircase Neuron
43:45 - Disney Neuron
44:15 - Hillary Clinton Neuron
44:50 - God Neuron
45:15 - Blurry Neuron
45:35 - Arrow Neuron
45:55 - Trophy Presentation Neuron
46:10 - Receding Hairline Neuron
46:30 - Traffic Neuron
46:40 - Raised Hand Neuron
46:50 - Google Maps Neuron
47:15 - Nervous Smile Neuron
47:30 - Elvis Neuron
47:55 - The Flash Neuron
48:05 - Beard Neuron
48:15 - Kilt Neuron
48:25 - Rainy Neuron
48:35 - Electricity Neuron
48:50 - Droplets Neuron
49:00 - Escape Neuron
49:25 - King Neuron
49:35 - Country Neuron
49:45 - Overweight Men Neuron
49:55 - Wedding
50:05 - Australia Neuron
50:15 - Yawn Neuron
50:30 - Bees & Simpsons Neuron
50:40 - Mussles Neuron
50:50 - Spice Neuron
51:00 - Conclusion

Paper: https://distill.pub/2021/multimodal-neurons/
My Findings: https://www.notion.so/CLIP-OpenAI-Microscope-Findings-27465eac373c451d8083428443e0837c
My Video on CLIP: https://youtu.be/T9XSU0pKX2E
My Video on Feature Visualizations & The OpenAI Microscope: https://youtu.be/Ok44otx90D4

Abstract:
In 2005, a letter published in Nature described human neurons responding to specific people, such as Jennifer Aniston or Halle Berry. The exciting thing wasn’t just that they selected for particular people, but that they did so regardless of whether they were shown photographs, drawings, or even images of the person’s name. The neurons were multimodal. As the lead author would put it: "You are looking at the far end of the transformation from metric, visual shapes to conceptual... information." We report the existence of similar multimodal neurons in artificial neural networks. This includes neurons selecting for prominent public figures or fictional characters, such as Lady Gaga or Spiderman. Like the biological multimodal neurons, these artificial neurons respond to the same subject in photographs, drawings, and images of their name.

Authors: Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, Chris Olah

Links:
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://discord.gg/4H8xxDF
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
Parler: https://parler.com/profile/YannicKilcher
LinkedIn: https://www.linkedin.com/in/yannic-kilcher-488534136/
BiliBili: https://space.bilibili.com/1824646584

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hi there and welcome back my dear fellow scholars. Today we're going to look at multimodal neurons in artificial neural networks by Gabriel Goh, Nick Camarada, Chelsea Voss, Shan Carter, Michael Petroff, Ludwig Schubert, Alec Radford and Chris Ola that has appeared in this Distillpub journal which I think is a pretty cool journal going beyond the classic PDF publishing. So this paper is an investigation into the new CLIP model by OpenAI and specifically the discovery of what they call multimodal neurons in this model. So this is an investigative work. They work with visualizations and I've made a video about both the CLIP model as well as the feature visualizations that has appeared previously. So safe to say what they are claiming as the high-level claim here is that in biology we sort of expect there to be neurons that respond not to individual patterns or to individual words but to concepts. So there could be a concept neuron of Halle Berry as you can see here and that neuron would respond to photographs of Halle Berry, to drawings and sketches of Halle Berry and also to text. So if we see the text, the rasterized text or we hear the word, that neuron, that same neuron would fire. Now so far in artificial neural networks we had not seen this kind of multimodal perception. So we have seen neurons responding in general to the same class of images because we train them as image classifiers but we have not seen that generalize to other modalities such as drawings or text. What they find in this CLIP model right here is that exactly what we expect in humans or in general in biological neural networks that happens. So they find for example a neuron that responds to Spider-Man. That is you know photos of Spider-Man in the real world or some person in a Spider-Man costume, drawings of Spider-Man and also text that says spider. So that would always the neuron would respond to all of these things, the same neuron and that is a sort of sign that these models have learned to connect to different modalities together. We've already discussed in the CLIP video that the model sort of learns to do OCR so it learns to recognize text because the CLIP model is fundamentally a model that connects images to text and my claim here is going to be that this addition of text, the model I think is very much a text model. So a lot of the connection it makes go via the textual level and a lot of the responses you're going to see here, the visualizations are going to deal with text rather than with images. So here you can see what this neuron responds to. If you thought it was the spider web here, no there's spider as a text, spider here, spider there, drawings of Spider-Man. So this neuron would respond to all of these things which is pretty pretty cool. So what they do, what they present here is an overview over the different neurons they find and as I understand it what they have done is they've gone through these neurons and they use their feature visualization technique with every single one of them. So I can show you what that looks like. Here is the open AI microscope and you can find that and this is the exact model they're looking at. So what you can do is you can simply click around in these neurons over here and then these are the visualizations right here. So now the visualizations are twofold. So on the left hand you have channel optimization, on the right hand you have neuron optimization. We've treated them in a previous video if you want to know how they come about but for now what you should know is that these are images that activate that particular neuron or that particular channel very much. So these images activate this particular thing in the neural network but not other things. So this is a way to see what these neurons respond to heavily. So here you can see on the left you often have kind of pattern structures, on the right you more have kind of in the center individual things. So maybe it's not really clear what this is. So what they also portray is data samples from the ImageNet data set that activate mostly that particular neuron. So you can pretty clearly see that this responds to popsicle ice cream. Now they also have a different data set down here. There is a Flickr Creative Commons and very much the same you see this is kind of ice and ice cream and at the bottom you have text that goes along with it. So here it's not really ice cream so this is a bit of a failure case but you always have to keep in mind that it could also be because of the lack in power in searching for text. So what they do down here is they have a search algorithm that finds pieces of text that that neuron responds to highly. So text that maximizes the dot product. So in the clip model you have an image part, you have a text part and you have a dot product at the end. So this is text that when you input it to the text part maximizes the dot product with that particular neuron. So it's not always going to be you know really good text but very often you can give you a hint in what the neuron thinks. Note that this isn't the same text as we're going to see later like the text that you saw in Spider-Man because the text you saw in Spider-Man that was rendered text. So they do a lot of investigation into rendered text because the clip model is quite good at responding to rendered text in the image side. Alright so they find they look at these neurons literally I think they just click here on the left boom and you look at them. So this seems to be like a hamburger pancake neuron and it is I did this for hours and I'll show you later what I found. This is absolutely fascinating what you'll find here by just clicking through and every now and then you find something like yeah alright but let's get back to the paper first. So the paper they find region neurons so neurons that respond to different regions of the world for example the USA. Now they not only do they have not only do they have this visualization technique for a for kind of the whole image they have faceted visualization so in this paper they introduce faceted visualization which they can so they can produce specifically faces that are US that respond to USA. They can produce specifically indoor things so this is all the same neuron these are images that are made such that they represent indoor scenes and there is an appendix if you want to know how that's done they can trim it to only produce nature pictures that this particular neuron responds to. So here you can get a much better insight into what into what the neuron looks at for example in if you create faces for the USA this is I don't know I call this one I call this one Benjamin Washington because it's a sort of a blend of Ben Franklin and George Washington but in general it's pretty cool so you can even yeah nature you can do pose for North America pose for the US I think that's kind of a GI a pose for Europe I don't know what that is but it doesn't always you know work out super well but they find person neurons so neurons that respond to individual people be that faces be that text so this is Donald Trump be that poses yeah Elvis is also pretty cool I've actually found I don't know if it I found the Elvis neuron myself or if I found a different one yeah so they also have emotion neurons which is also pretty cool where they so they find the neurons that respond to particular emotions so when they tell these neurons when they make a faceted reconstruction and tell please give me a face this is what comes out and that you know it's just shocking when you do something like a pose for shocked this I think we're only scratching the surface here honestly but you can see the claim here the claim is that the same neuron responds to this picture and to this picture this is supposed to be text you can only guide it you can't you know force it to this picture indoor to this picture so the same neuron will respond to all of these and they call that multimodal neuron because it represents a concept the concept of being shocked rather than in a particular fine-grained pattern which was always the kind of problem so far with these neural networks that the they were more looking at you know low level patterns than high level concepts it seems with clip with by combining modalities like images and text and by not forcing this constraint like in a classifier into 1000 predefined classes we can gain much more we can go up the hierarchy of features so they have art style they have holiday neurons religion neurons person trait neurons abstract concept neurons the star I found the star I yeah I remember time neurons counting neurons pairs of force they are not always so super good but it clearly goes into the good direction so here they highlight specific things first person neurons so they find neurons that respond for example to Jesus Christ so they would respond to all of these images here on the right you see their crosses Jesus Christ and so on depictions of Jesus drawings of Jesus and when you ask the model to generate you a image that reconstructs this neurons activation and you can force it or you guide it to make a face this turns out if you got it to make a pose this turns out a logo obviously they also have Hitler right here which is also pretty cool though I have if you click on these things you'll get actually to the microscope thing and this is the one for for Hitler and you know I'm I'm not entirely sure that this is the case like I can see you know the kind of mustache thing but if you look at what in the data set activates this one it's it is a bunch of swastikas but it is also just a bunch of kind of German political stuff but yeah I mean the concept the concept here even if it's not Hitler directly it's pretty pretty cool I yeah also found that domain endings rendered as images will activate the same neuron as the flag of that country and activate the same neuron as like the architecture of that country it is super duper interesting alright so they have these person neurons which is already cool and they have so they've found these they do a case study here for the Donald Trump neuron so the Donald Trump neuron recognizes Donald Trump and then they want to see what images in the data set activate this neuron by how much so they make the claim here that if you for example choose profile pictures of Donald Trump and you see here is the zero line and here is the standard deviations from zero activation so pictures of Donald Trump activate this neuron like 30 times more than it is activated over the whole data set which makes sense if that neuron responds to Donald Trump but it also responds to art images containing Donald Trump by the way these are classified by the authors here they've gone through the images and they've classified them into these categories text containing Donald Trump's name the model also strongly responds with the same neuron right that's the that's the crazy part so a picture with text in it that says Trump activates the same neuron as a profile picture of Trump activates the same neuron as a mugger hat and activates sometimes the same neuron as political images activates so the if you look at games and music and so on that is very that neuron is very deactivated so not only is it zero it's actually negative which the authors interpreted as sort of being being counter to that in the space of all concepts they do so the this paper is is full of this kind of content warnings it might be disturbing and so on which you know you can you can do but I also find I also find the rest of the paper is kind of a fairly large hedge against certain things and it gets political at times for example when they want to when they want to claim that so here on the other hand it most negatively activates to musicians like Nicki Minaj and Eminem video games like fortnight civil rights activists like Martin Luther King jr. and LGBT symbols like rainbow flags so the games and the fortnight here yes we can see that but if you click on this and they have four images of this you can see that it's activated at relatively low magnet like negative magnitudes which is correct then it is also almost equally activated over here at high magnitudes so like I see the point you're trying to make but I mean if if you are in the political sphere this is not you have to you have to not interpret this as meaning that these things are kind of aligned but you have to interpret it as these things will appear together often which you know one can one can definitely understand in this case so here they search for profile pictures of other people when including Donald Trump himself and they plot how much these profile pictures of other people activate the Trump neuron and you can see that for example well yeah Pence activates this neuron by quite a bit I think yeah the selection here is you know up to the authors of course but it's it's fairly interesting to see that Clinton Cruz and Obama activated more than Hitler and almost as much as Steve Jobs for some reason so I'm not I'm not entirely sure what you can make of this but it's definitely interesting to in on this side like to observe the multimodality of pictures just the fact that text drawings symbols of that campaign and profile pictures will all activate the same neuron that is fairly impressive they go on and they identify emotion neurons so again there's a content warning by the way also here so here they identify a neuron that responds to surprise or shock and you can see that all of these pictures on the right will activate that neuron so there are faces being shocked there are horses being shocked and there is rendered text saying like WTF OMG and so on again if you I think we've we've gone through this this is the the shocked one there they're also secondary neurons that help let's say help the primary emotion neurons so here you can see an overview over the different emotion neurons they have found and it is pretty stunning so here they ask them obviously to create a face when they constrain them not constrain they guide them towards making poses by the way the way you guide them is they train linear probe classifiers on separate data sets so they would train a classifier on a face data set to distinguish all faces from all non faces and then that use that classifier to sort of guide this reconstruction process that's how you can sort of choose to end up with a face or with a pose or with a piece of text so as you can see it's pretty pretty cool that even the text that comes out of this reconstruction process these aren't real images right these are kind of reconstructed to activate those neurons like for evil you can see that there's devil and Satan for shocked it's like OMG for crowd for happy it's it's happy if you look at the poses for happy for serious evil is particularly cool incarcerated rejected this is I think this is absolutely cool there is the NSFW there is erotic there are erotic neurons and if I click on this it will show now if you click on this absolutely nothing not safe for work will happen I promise I don't promise but you know I I've tried it it's fine I will not click on it because if this model things that's not safe for work the YouTube algorithm will think is not safe for work so but what I can tell you is that if you go on that neuron and you go click through it to go to the microscope and you look at what image net pictures respond to that neuron heavily you'll find out that image net isn't the really clean dog breed data set that you might have known all right they found other neurons corresponding to silly facial expressions like duck faces and and and tongue showing and so on which is is pretty neat and they find this neuron that corresponds to mental illness which the reconstruction is just amazing like this is just mind-baffling nature kind of always looks the same but mental illness let's say face this is it's crazy how this model connects things and it connects these things to books and writings of sad mental health anxiety and so on now do I think the model understands what a mental illness is no I don't think so I think much like in GPT-3 it is learned to statistically associate things so it has learned that there might be and I think that happens via the textual input so in clip for every image you have a piece of text and I think the connection between the topics happens on the textual level because the text descriptions are the same between images so there will be images of people you know cowering like this being sad and the textual description for it would be something like mental illness anxiety sadness and then for these pictures of these books as well there the descriptions would be I mean this is one is literally called overcoming anxiety so if the picture is of a book and the description says what is on the picture obviously that text will be connected so I think that's how it learns to connect things via the text and I think this thing is in large part a text model so here they do the same study for images that are associated with mental illness so depression sad pictures like anxiety pictures are pretty high depressing jokes if you look at music and sports that's negatively activated so on so you can see that I think via the text the model can sort of learn about how different different concepts different things different patterns are connected to one another they have region neurons which I find pretty cool so they discover neurons that when they show them a crop of this world map this this world map when they show them a crop of the world map the the neuron will respond the neural will flare up and so the neuron this red neuron here that reacts to these pieces of text and now it reacts to the pieces of text when they are rendered into images right then the neuron responds if you render the word American in an image and then you give it to the network that neuron will flare up the same neuron will flare up if you show it a crop of this region here of the map which is crazy like crazy again I think the connection happens in the textual domain but still crazy you can have it do face facets for these different regions yeah if you if you go over here so the neuron that responds to this blue area responds to the rendered words Mumbai Singh Pakistan Afghanistan Bangladesh and responds strongly or if you make reconstructions that activate that neuron you get these kinds of pictures which is fairly cool the same here for Europe so this is kind of European and yeah I that looks like home so check this out a bit for yourself but it's immensely cool they even find these secondary regional neurons that aren't exactly regional but they also respond to crops of this map and they highlight this entrepreneur neuron that you know it's a response to sort of the words entrepreneur entrepreneurial and it you know it kind of looks like his company logos a little bit I guess but it you know the the model that responds to the word entrepreneur lights up when you show it the west coast of the US kind of the the California region interestingly it also lights up when you show it the west coast of the of the lower of the southern African continent which is cool like that's definitely unexpected I don't know I I'm not informed enough to know whether or not there is significant entrepreneurial drive going on there could also be that it the model simply confuses the west coast of the two countries right like they look in a crop they look the same could be I'm not I'm not I don't know so maybe I'm wrong it's also interesting that only these regions light up right if for this particular neuron so I have my doubts whether that's just kind of a a lucky cherry pick I'm not saying it's cherry picked but you know kind of the I can stumble upon and you make something of it or not they have more case study of African kind of subdivisions and let's go down here here is where they discuss that they can also produce text for the text side of clip so not only do they render and this this text here is what you're going to see the maximal text align with an image or with a neuron sorry is what you're going to see at the bottom of the microscope pages so lastly they force a they kind of make a sparse code out of their main neurons that they find and they try to build more complex emotions from them for example jealous and they do they do claim here that that makes sort of a bit of sense like jealous is champion plus hug plus grumpy minus crying I'm not exactly sure if you know if that makes super much sense so bored is relaxing plus grumpy maybe yeah intimate is soft smile plus heart minus sick you can you can probably make something out of that though yeah powerful is lightning miracle plus evil plus yoga that's that's definitely definitely the case do check it out it is very interesting to look at some of those things even though I think it does not make you know terrible much sense but in often cases but stressed being success plus mental disorder plus pink objects maybe but it is more kind of it is not claimed that this is you know kind of an absolute thing it's more an investigation into these networks if you lay them out in sort of a 2d surface you can see that these emotion neurons they come pretty close to sort of an atlas of what people when we just use two factors we roughly reconstruct the canonical mood axis of in much used in much of psychology valence and arousal so you can divide these emotions into two things so there is valence which is good or bad so I think that's top bottom here so here's mad angry hostile and so on maybe not no top bottom is probably valence like how strong something is and then left right might be good and bad no also not here insecure inspired aroused awful sad well these are all bad no hostile is here appalled is here and horrified is here where are you happy in the middle maybe creative okay happy is here also it might not be exactly axis aligned right you can also divide it into seven factors which we nearly reconstruct a well-known categorization of these emotions into happy surprised bad disgusted fearful and angry except with disgusted switch for a new category related to affection that includes valued loving lonely and insignificant all right so this next piece is really funny what they do is so given clip you can build a classifier so if you have the clip model that connects images to text what you can do is you feed one image and then you give it a bunch of texts to choose from and whichever one it responds highest with that's kind of the class so if you provide the class labels as text you can build a zero short classifier now clip papers demonstrated that that works well so here they do this so they have this Apple right here and the label is correctly Apple but if they just slap a sticker on it that says iPod the clip model will switch to iPod and here yeah here is where I really think that this model it is a textual model it responds even to rendered text it responds very heavily so here it responds to this iPod library like this iPod looks like something I bought off Craigslist last week so you can see it works like almost every single time you just slap a label on it and that tells me that we are still like the text is might be too dominant in these models especially you know this models they will connect the text with render text in the image and that that's a very strong signal for what's in the image right this is only zero shot though if you switch this to do linear probe so if you actually train a linear probe on the representation of clip then these attacks don't work anymore so this is going back again to sort of the old-school deep learning approach where you actually train a classifier and once you train it picks up on on other features and then it doesn't work anymore all right yeah so they they evaluate this on a large scale they can't always slap a label so they just fill the image with render text and that usually gets the classifier confused fairly fairly well they also do this with this strupe test which you can do with humans which is fairly difficult if you do it at a high speed and they discover that the model basically pays no attention whatsoever to the color of the word it pays much more attention to what the word says which is strange right because you think if I have a neural network and you know it basically needs to to recognize the color here it needs to filter out the white pixels but then just average the pixels it gets the correct answer that's so easy right it simply averages whereas to recognize that this says green is much more difficult but the model was trained to connect text and images images which often have text in them so it has learned to do OCR basically in the Dolly video I claimed that Dolly has learned to do reverse OCR and people correctly pointed out that that is more aptly called writing but I love reverse OCR I'm gonna call writing from now on reverse OCR so again this is evidence for the claim that this is mostly a textual model and now I want to show you what I found so if you're not in the mood I have all this in a notion page which I'll link down below so I'll show you just some interesting stuff sometimes it's multimodal sometimes it's not right so we already were here we just clicked around but now I want to kind of show you the good stuff so this is a Superman neuron that I found so it responds as you can see to symbols of Superman in the image net data set Superman Superman drawing Superman comics Superman spelled out rendered and so on this is exactly kind of what what the the article was about right but now it's Superman not spider-man this I call the resting bee face neuron so it responds to people being slightly annoyed yeah as you can see here this is trash bags so this responds to trash bags pretty cool right so not any kind of bag right specifically trash bags even if they are not black so there are a couple in there they don't necessarily breath black there is even trash cans like dump containers right here that have no bag in sight yet still that neuron response this sorry about sorry about that yeah for some reason you might want to I don't know maybe have something in your pockets yeah so so fairly cool oh there's a tree is not always you know perfect but these are the data set examples that most excite that neuron so you can also see the text isn't always good though I think I think if the text here isn't super good it might more be an effect of this method to search text because text is of course not a continuous signal so it's fairly hard to search text that maximizes some activation otherwise we could build GANs for text very easily which we still can't this one here I've titled this strength and a law and weightlifting which I'm aware of this is not you know iconography of a law however so this is pretty cool as an image right now if you look at what in the data set what samples it responds to it's kind of all weightlifting it's all weights so this is weight weight and if you go down here to the other data set this is why I called it sort of a law because you have also rendered names like the the rendered Allah you have the Quran you have symbols of Islam and if you go to the text that it searches goes like hammer workout prophet prophet Zana in lumber iron gym the brutal workout of God so you know pretty cool neuron honestly and you know that it that responds with this I don't even I don't even know what what that is is that is that Hindu imagery or Buddhist imagery so cool these are organs this is an organ neuron I hope like you you can see that and it responds to the render text of control I don't know what to make of it also canal viral but also to drawings you can see here a drawing of a heart for some reason also chins so it's not always super duper clear what a neuron does in fact most of these neurons you will find if you go look at what image net sound and these I believe these are crops of image net samples not entire pictures so if you look at what by the way control and CTRL if you look at what examples most often it will be rendered text so that the image that no matter what neuron most neurons actually pay attention to rendered text rather than to images the ones I've selected are the ones that do not but if you just go and click on some random neuron we can actually try and it's certainly going to probably fail this one looks pretty cool looks pretty cool actually that responds to printers yep demonstration effect fails horribly how about this one yeah so you can see that you know maybe you don't exactly know what that is so you want to look at what so here you see that it primarily responds to the text miss I guess mss I miss you Mississippi and so on you know Mississippi having it twice in there that got a respond pretty pretty heavily and most of the time you'll find something like this that it responds very much to the rendered pieces of text in images these are film spools and so not only does it respond to film spools but also to things like director screening popcorn the kind of movie theater labeling showing Hollywood cinemas there's also entertainment so you know the multimodality again this this is a this is a phenomenon because we introduced the text and it can connect it on the text level this is feather patterns and leaf patterns so even when it's in coffee you see the feather and leaf patterns even when it's a drawing it can it will still respond this one is strange so this responds to things like Sparta and front and Troy but so that it responds to rendered front Trojan Spartans front and it also has a lot of people doing sort of squats as you can see so and and fighting so this is kind of an iron so this is a bit of kind of a warrior neurons you can see oh there's lots of ah of course it's because of these Spartan runs and all they're called like this right these kind of sporting events I see Roman frontside Roman Roman so it connects the workout with the Spartan workout kind of division and then it connects the Trojan and so on via again via the text because it makes no sense to connect like the vodka and the and the weightlifting maybe so yeah I hope I hope you're fairly convinced by now we're gonna know a bit faster now because the videos already too long but this one here is the letter E so it's e it responds again to rendered text of E this one here is cleaning so it responds to cleaning products and cleaning things this one here is frown so this is frowning frowning frowning grumpy face grumpy face lion lion responding to lions rendered text of lions team names called lions and so on fashion model fashion model a bit by the way the labels are mine I just looked at them and decided what they are but you can see like there's a lot of these kind of runway shots here baseball stadium so cool so these are kind of top views of baseball stadium but it responds a lot to things saying park PNC park AT&T park but also kind of home team park lights and and baseball dugouts and even players I've seen some players logos of teams baseball depictions of actual baseballs immense immensely cool here bride this is bride you can see this is bride this one what do you think this one is Navy so super cool that it can I kind of connect these ropes with the emblems the the kind of your tags so and it connects it to render text saying Navy right so these are the crops of images that it responds to Navy of fish like officers Navy gravestones yeah so cool this one okay this for this I also had to look at sort of the pictures here and the text going along with it this is hemp but it is also kind of goa patterns it is also for some reason turn or earn it is also Hendrix so this isn't even Jimi Hendrix right like this this is definitely connected to these goa shirts there is also there's pictures of Jimi Hendrix which I guess you can understand there is also turn again whereas there's Bob no this is Bob Marley sorry this Bob Marley yeah so so it connects these things staircase and here for some reason also responds to text rendered human and to staircases and here I have I don't know why but there's there's this thing which I'm not sure so it has human in it but it is also arranged like a staircase so maybe that's why it responds extra extra yeah the Disney neuron this is a Disney neuron how cool is this how cool is this so you can clearly see that that but then it you know Disney these are the samples that it responds to simply something saying Disney the Mickey Mouse ear the mini bow no immensely cool the castle right the Disney castle this is the Hillary Clinton neuron you can see this is Hillary and the images it responds to is Hillary Hill pill Polly Hill pills so this is maybe it's more like the LL why the IL why neuron but it it does pick out Hillary Clinton as well yeah so image net of course is older than at least one of Hillary's campaigns I'm not sure this is God so I found this one this is yeah God if you so the reconstruction process it's not very good at generating text maybe because so they have a lot of priors in that if you look at the reconstruction article you can probably and they do this in in this article they reconstruct text but it's still not super clear maybe it has to do with the architecture this here is blurry it's just the concept of blurry so you look at the images they're kind of often blurry and if you look at the text going along with it it's all like blurry blurry blurry blurry blurry blurry blurry blurry cool like it's not even what's on the image but you can clearly see like this comes from the other description this is hand-drawn arrows or arrows in general this looks like my videos now right like this recognizes arrows is specifically a you know kind of color re arrows this one what does it do this is presenting a trophy you see this one here in the middle this is kind of so these are all you know people presenting some kind of thing holding some kind of thing in their hand showing it like fishermen or diplomas this one I was amazed by this is a neuron responding to receding hairlines like it responds to receding hairlines how cool is that how cool is that this is traffic tent and so on so it responds to tents and traffics and crowds of people this one is raised arms but also pancakes so pancakes and raised hands for some reason there's a connection no but I mean these these models they still overload when they can this one how cool is that this is the Google Maps neuron these are reconstructions these are not samples these are reconstructions you can see it's clearly it has kind of the street labels and the pins on it so this is a Google Google Maps like neuron what so cool this one I call nervous smile you can maybe see that it's like yeah here's Elvis this is the Elvis neuron I know it sort of it also looks like Hendrix a bit but the things it connects it to is that's not Elvis that's not Elvis kiss okay maybe it's not exactly Elvis maybe it's more like a pop star neuron yeah maybe it's not Elvis only Elvis Billy Elliot this one is the flash right that's the flash and the cool thing is it responds to images saying flash what okay beards response to beards generally beards lots of beards kilts kilts and bagpipes response to guilt kilts and bagpipes rainy this is a neuron that responds to things that are rainy rainy days so you can see here out the window it's raining rainy windows so cool this is flash and electricity so you will see like symbols these symbols of these flashes but also kind of electric hair curling up droplets how cool does that look like that's just cool and the occasional image net reconstruction thing where there must be like half a dog face in there that is just trippy this one is this one is escape okay escape like look at that like to connect these things how long would you like without contrastive learning how well I guess if as long as you have images and labels but still king this is king so the depicted are crowns but response to renderings of King this is nation how cool is that nation response to country country country oh it's country not nation but still this one response to overweight men there's a neuron that responds to over phases of overweight men this one is wedding this one is Australia and the cool thing here is that it responds to rendered domain names of Australia like the top-level domain of Australia what mind-blown this is yawning or screaming well I think you know like here we have a same neuron for bees and the Simpsons bees and the Simpsons this is muscles and seafood and lastly spices spices and other powdery things you know don't ask too many questions hmm alright so that was it for me for today I have many more that are linked in a notion description somewhere go check it out please try out this I've not yet looked through all of them there are so many there are literally thousands of these units and this is just one of the models they have available go look and share you know on our discord you know the best ones you find alright that was it thanks for listening bye bye
 Hi there and welcome back my dear fellow scholars. Today we're going to look at
 multimodal neurons in artificial neural networks by Gabriel Goh, Nick Camarada,
 Chelsea Voss, Shan Carter, Michael Petroff, Ludwig Schubert, Alec Radford and
 Chris Ola that has appeared in this Distillpub journal which I think is a
 pretty cool journal going beyond the classic PDF publishing. So this paper is
 an investigation into the new CLIP model by OpenAI and specifically the
 discovery of what they call multimodal neurons in this model. So this is an
 investigative work. They work with visualizations and I've made a video
 about both the CLIP model as well as the feature visualizations that has appeared
 previously. So safe to say what they are claiming as the high-level claim here is
 that in biology we sort of expect there to be neurons that respond not to
 individual patterns or to individual words but to concepts. So there could be
 a concept neuron of Halle Berry as you can see here and that neuron would
 respond to photographs of Halle Berry, to drawings and sketches of Halle Berry and
 also to text. So if we see the text, the rasterized text or we hear the word, that
 neuron, that same neuron would fire. Now so far in artificial neural networks we
 had not seen this kind of multimodal perception. So we have seen neurons
 responding in general to the same class of images because we train them as image
 classifiers but we have not seen that generalize to other modalities such as
 drawings or text. What they find in this CLIP model right here is that exactly
 what we expect in humans or in general in biological neural networks that
 happens. So they find for example a neuron that responds to Spider-Man. That
 is you know photos of Spider-Man in the real world or some person in a Spider-Man
 costume, drawings of Spider-Man and also text that says spider. So that would
 always the neuron would respond to all of these things, the same neuron and that
 is a sort of sign that these models have learned to connect to different
 modalities together. We've already discussed in the CLIP video that the
 model sort of learns to do OCR so it learns to recognize text because the
 CLIP model is fundamentally a model that connects images to text and my claim
 here is going to be that this addition of text, the model I think is very much a
 text model. So a lot of the connection it makes go via the textual level and a lot
 of the responses you're going to see here, the visualizations are going to
 deal with text rather than with images. So here you can see what this neuron
 responds to. If you thought it was the spider web here, no there's spider as a
 text, spider here, spider there, drawings of Spider-Man. So this neuron would
 respond to all of these things which is pretty pretty cool. So what they do, what
 they present here is an overview over the different neurons they find and as I
 understand it what they have done is they've gone through these neurons and
 they use their feature visualization technique with every single one of them.
 So I can show you what that looks like. Here is the open AI microscope
 and you can find that and this is the exact model they're looking at. So what
 you can do is you can simply click around in these neurons over here and
 then these are the visualizations right here. So now the visualizations are
 twofold. So on the left hand you have channel optimization, on the right hand
 you have neuron optimization. We've treated them in a previous video if you
 want to know how they come about but for now what you should know is that these
 are images that activate that particular neuron or that particular channel very
 much. So these images activate this particular thing in the neural
 network but not other things. So this is a way to see what these neurons
 respond to heavily. So here you can see on the left you often have kind of
 pattern structures, on the right you more have kind of in the center
 individual things. So maybe it's not really clear what this is. So what they
 also portray is data samples from the ImageNet data set that activate mostly
 that particular neuron. So you can pretty clearly see that this responds to popsicle
 ice cream. Now they also have a different data set down here. There is a Flickr
 Creative Commons and very much the same you see this is kind of ice and ice
 cream and at the bottom you have text that goes along with it. So here it's not
 really ice cream so this is a bit of a failure case but you always have to keep
 in mind that it could also be because of the lack in power in searching for text.
 So what they do down here is they have a search algorithm that finds pieces of
 text that that neuron responds to highly. So text that maximizes the dot product.
 So in the clip model you have an image part, you have a text part and you have a
 dot product at the end. So this is text that when you input it to the text part
 maximizes the dot product with that particular neuron. So it's not always
 going to be you know really good text but very often you can give you a hint
 in what the neuron thinks. Note that this isn't the same text as we're going to
 see later like the text that you saw in Spider-Man because the text you saw in
 Spider-Man that was rendered text. So they do a lot of investigation into
 rendered text because the clip model is quite good at responding to rendered
 text in the image side. Alright so they find they look at these neurons
 literally I think they just click here on the left boom and you look at them. So
 this seems to be like a hamburger pancake neuron and it is I did this for
 hours and I'll show you later what I found. This is absolutely fascinating
 what you'll find here by just clicking through and every now and then you find
 something like yeah alright but let's get back to the paper first. So the paper
 they find region neurons so neurons that respond to different regions of the
 world for example the USA. Now they not only do they have not only do they have
 this visualization technique for a for kind of the whole image they have
 faceted visualization so in this paper they introduce faceted visualization
 which they can so they can produce specifically faces that are US that
 respond to USA. They can produce specifically indoor things so this is
 all the same neuron these are images that are made such that they represent
 indoor scenes and there is an appendix if you want to know how that's done they
 can trim it to only produce nature pictures that this particular neuron
 responds to. So here you can get a much better insight into what into what the
 neuron looks at for example in if you create faces for the USA this is I don't
 know I call this one I call this one Benjamin Washington because it's a sort
 of a blend of Ben Franklin and George Washington but in general it's pretty
 cool so you can even yeah nature you can do pose for North America pose for the
 US I think that's kind of a GI a pose for Europe I don't know what that is but
 it doesn't always you know work out super well but they find person neurons
 so neurons that respond to individual people be that faces be that text so
 this is Donald Trump be that poses yeah Elvis is also pretty cool I've actually
 found I don't know if it I found the Elvis neuron myself or if I found a
 different one yeah so they also have emotion neurons which is also pretty
 cool where they so they find the neurons that respond to particular emotions so
 when they tell these neurons when they make a faceted reconstruction and tell
 please give me a face this is what comes out and that you know it's just shocking
 when you do something like a pose for shocked this I think we're only
 scratching the surface here honestly but you can see the claim here the claim is
 that the same neuron responds to this picture and to this picture this is
 supposed to be text you can only guide it you can't you know force it to this
 picture indoor to this picture so the same neuron will respond to all of these
 and they call that multimodal neuron because it represents a concept the
 concept of being shocked rather than in a particular fine-grained pattern which
 was always the kind of problem so far with these neural networks that the they
 were more looking at you know low level patterns than high level concepts it
 seems with clip with by combining modalities like images and text and by
 not forcing this constraint like in a classifier into 1000 predefined classes
 we can gain much more we can go up the hierarchy of features so they have art
 style they have holiday neurons religion neurons person trait neurons abstract
 concept neurons the star I found the star I yeah I remember time neurons
 counting neurons pairs of force they are not always so super good but it clearly
 goes into the good direction so here they highlight specific things first
 person neurons so they find neurons that respond for example to Jesus Christ so
 they would respond to all of these images here on the right you see their
 crosses Jesus Christ and so on depictions of Jesus drawings of Jesus and
 when you ask the model to generate you a image that reconstructs this neurons
 activation and you can force it or you guide it to make a face this turns out
 if you got it to make a pose this turns out a logo obviously they also have
 Hitler right here which is also pretty cool though I have if you click on these
 things you'll get actually to the microscope thing and this is the one for
 for Hitler and you know I'm I'm not entirely sure that this is the case like
 I can see you know the kind of mustache thing but if you look at what in the
 data set activates this one it's it is a bunch of swastikas but it is also just a
 bunch of kind of German political stuff but yeah I mean the concept the concept
 here even if it's not Hitler directly it's pretty pretty cool I yeah also
 found that domain endings rendered as images will activate the same neuron as
 the flag of that country and activate the same neuron as like the architecture
 of that country it is super duper interesting alright so they have these
 person neurons which is already cool and they have so they've found these they do
 a case study here for the Donald Trump neuron so the Donald Trump neuron
 recognizes Donald Trump and then they want to see what images in the data set
 activate this neuron by how much so they make the claim here that if you for
 example choose profile pictures of Donald Trump and you see here is the
 zero line and here is the standard deviations from zero activation so
 pictures of Donald Trump activate this neuron like 30 times more than it is
 activated over the whole data set which makes sense if that neuron responds to
 Donald Trump but it also responds to art images containing Donald Trump by the
 way these are classified by the authors here they've gone through the images
 and they've classified them into these categories text containing Donald
 Trump's name the model also strongly responds with the same neuron right
 that's the that's the crazy part so a picture with text in it that says Trump
 activates the same neuron as a profile picture of Trump activates the same
 neuron as a mugger hat and activates sometimes the same neuron as political
 images activates so the if you look at games and music and so on that is very
 that neuron is very deactivated so not only is it zero it's actually negative
 which the authors interpreted as sort of being being counter to that in the space
 of all concepts they do so the this paper is is full of this kind of content
 warnings it might be disturbing and so on which you know you can you can do but
 I also find I also find the rest of the paper is kind of a fairly large hedge
 against certain things and it gets political at times for example when they
 want to when they want to claim that so here on the other hand it most
 negatively activates to musicians like Nicki Minaj and Eminem video games like
 fortnight civil rights activists like Martin Luther King jr. and LGBT symbols
 like rainbow flags so the games and the fortnight here yes we can see that but
 if you click on this and they have four images of this you can see that it's
 activated at relatively low magnet like negative magnitudes which is correct
 then it is also almost equally activated over here at high magnitudes so like I
 see the point you're trying to make but I mean if if you are in the political
 sphere this is not you have to you have to not interpret this as meaning that
 these things are kind of aligned but you have to interpret it as these things
 will appear together often which you know one can one can definitely
 understand in this case so here they search for profile pictures of other
 people when including Donald Trump himself and they plot how much these
 profile pictures of other people activate the Trump neuron and you can
 see that for example well yeah Pence activates this neuron by quite a bit I
 think yeah the selection here is you know up to the authors of course but
 it's it's fairly interesting to see that Clinton Cruz and Obama activated more
 than Hitler and almost as much as Steve Jobs for some reason so I'm not I'm not
 entirely sure what you can make of this but it's definitely interesting to in on
 this side like to observe the multimodality of pictures just the fact
 that text drawings symbols of that campaign and profile pictures will all
 activate the same neuron that is fairly impressive they go on and they identify
 emotion neurons so again there's a content warning by the way also here so
 here they identify a neuron that responds to surprise or shock and you
 can see that all of these pictures on the right will activate that neuron so
 there are faces being shocked there are horses being shocked and there is
 rendered text saying like WTF OMG and so on again if you I think we've we've gone
 through this this is the the shocked one there they're also secondary neurons
 that help let's say help the primary emotion neurons so here you can see an
 overview over the different emotion neurons they have found and it is pretty
 stunning so here they ask them obviously to create a face when they constrain them
 not constrain they guide them towards making poses by the way the way you guide
 them is they train linear probe classifiers on separate data sets so
 they would train a classifier on a face data set to distinguish all faces from
 all non faces and then that use that classifier to sort of guide this
 reconstruction process that's how you can sort of choose to end up with a face
 or with a pose or with a piece of text so as you can see it's pretty pretty
 cool that even the text that comes out of this reconstruction process these
 aren't real images right these are kind of reconstructed to activate those
 neurons like for evil you can see that there's devil and Satan for shocked it's
 like OMG for crowd for happy it's it's happy if you look at the poses for happy
 for serious evil is particularly cool incarcerated rejected this is I think
 this is absolutely cool there is the NSFW there is erotic there are erotic
 neurons and if I click on this it will show now if you click on this absolutely
 nothing not safe for work will happen I promise I don't promise but you know I
 I've tried it it's fine I will not click on it because if this model things
 that's not safe for work the YouTube algorithm will think is not safe for
 work so but what I can tell you is that if you go on that neuron and you go
 click through it to go to the microscope and you look at what image net pictures
 respond to that neuron heavily you'll find out that image net isn't the really
 clean dog breed data set that you might have known all right they found other
 neurons corresponding to silly facial expressions like duck faces and and and
 tongue showing and so on which is is pretty neat and they find this neuron
 that corresponds to mental illness which the reconstruction is just amazing like
 this is just mind-baffling nature kind of always looks the same but mental
 illness let's say face this is it's crazy how this model connects things and
 it connects these things to books and writings of sad mental health anxiety
 and so on now do I think the model understands what a mental illness is no
 I don't think so I think much like in GPT-3 it is learned to statistically
 associate things so it has learned that there might be and I think that happens
 via the textual input so in clip for every image you have a piece of text and
 I think the connection between the topics happens on the textual level because the
 text descriptions are the same between images so there will be images of people
 you know cowering like this being sad and the textual description for it would
 be something like mental illness anxiety sadness and then for these pictures of
 these books as well there the descriptions would be I mean this is
 one is literally called overcoming anxiety so if the picture is of a book
 and the description says what is on the picture obviously that text will be
 connected so I think that's how it learns to connect things via the text
 and I think this thing is in large part a text model so here they do the same
 study for images that are associated with mental illness so depression sad
 pictures like anxiety pictures are pretty high depressing jokes if you look
 at music and sports that's negatively activated so on so you can see that I
 think via the text the model can sort of learn about how different different
 concepts different things different patterns are connected to one another
 they have region neurons which I find pretty cool so they discover neurons
 that when they show them a crop of this world map this this world map when they
 show them a crop of the world map the the neuron will respond the neural
 will flare up and so the neuron this red neuron here that reacts to these pieces
 of text and now it reacts to the pieces of text when they are rendered into
 images right then the neuron responds if you render the word American in an
 image and then you give it to the network that neuron will flare up the
 same neuron will flare up if you show it a crop of this region here of the map
 which is crazy like crazy again I think the connection happens in the textual
 domain but still crazy you can have it do face facets for these different
 regions yeah if you if you go over here so the neuron that responds to this blue
 area responds to the rendered words Mumbai Singh Pakistan Afghanistan
 Bangladesh and responds strongly or if you make reconstructions that activate
 that neuron you get these kinds of pictures which is fairly cool the same
 here for Europe so this is kind of European and yeah I that looks like home
 so check this out a bit for yourself but it's immensely cool they even find these
 secondary regional neurons that aren't exactly regional but they also respond
 to crops of this map and they highlight this entrepreneur neuron that you know
 it's a response to sort of the words entrepreneur entrepreneurial and it you
 know it kind of looks like his company logos a little bit I guess but it you
 know the the model that responds to the word entrepreneur lights up when you
 show it the west coast of the US kind of the the California region interestingly
 it also lights up when you show it the west coast of the of the lower of the
 southern African continent which is cool like that's definitely unexpected I
 don't know I I'm not informed enough to know whether or not there is significant
 entrepreneurial drive going on there could also be that it the model simply
 confuses the west coast of the two countries right like they look in a crop
 they look the same could be I'm not I'm not I don't know so maybe I'm wrong it's
 also interesting that only these regions light up right if for this particular
 neuron so I have my doubts whether that's just kind of a a lucky cherry
 pick I'm not saying it's cherry picked but you know kind of the I can stumble
 upon and you make something of it or not they have more case study of African
 kind of subdivisions and let's go down here here is where they discuss that
 they can also produce text for the text side of clip so not only do they render
 and this this text here is what you're going to see the maximal text align with
 an image or with a neuron sorry is what you're going to see at the bottom of the
 microscope pages so lastly they force a they kind of make a sparse code out of
 their main neurons that they find and they try to build more complex emotions
 from them for example jealous and they do they do claim here that that makes
 sort of a bit of sense like jealous is champion plus hug plus grumpy minus
 crying I'm not exactly sure if you know if that makes super much sense so bored
 is relaxing plus grumpy maybe yeah intimate is soft smile plus heart minus
 sick you can you can probably make something out of that though yeah
 powerful is lightning miracle plus evil plus yoga that's that's definitely
 definitely the case do check it out it is very interesting to look at some of
 those things even though I think it does not make you know terrible much sense
 but in often cases but stressed being success plus mental disorder plus pink
 objects maybe but it is more kind of it is not claimed that this is you know
 kind of an absolute thing it's more an investigation into these networks if you
 lay them out in sort of a 2d surface you can see that these emotion neurons they
 come pretty close to sort of an atlas of what people when we just use two factors
 we roughly reconstruct the canonical mood axis of in much used in much of
 psychology valence and arousal so you can divide these emotions into two
 things so there is valence which is good or bad so I think that's top bottom
 here so here's mad angry hostile and so on maybe not no top bottom is probably
 valence like how strong something is and then left right might be good and bad no
 also not here insecure inspired aroused awful sad well these are all bad no
 hostile is here appalled is here and horrified is here where are you happy in
 the middle maybe creative okay happy is here also it might not be exactly axis
 aligned right you can also divide it into seven factors which we nearly
 reconstruct a well-known categorization of these emotions into happy surprised
 bad disgusted fearful and angry except with disgusted switch for a new category
 related to affection that includes valued loving lonely and insignificant
 all right so this next piece is really funny what they do is so given clip you
 can build a classifier so if you have the clip model that connects images to
 text what you can do is you feed one image and then you give it a bunch of
 texts to choose from and whichever one it responds highest with that's kind of
 the class so if you provide the class labels as text you can build a zero
 short classifier now clip papers demonstrated that that works well so
 here they do this so they have this Apple right here and the label is
 correctly Apple but if they just slap a sticker on it that says iPod the clip
 model will switch to iPod and here yeah here is where I really think that this
 model it is a textual model it responds even to rendered text it responds very
 heavily so here it responds to this iPod library like this iPod looks like
 something I bought off Craigslist last week so you can see it works like almost
 every single time you just slap a label on it and that tells me that we are
 still like the text is might be too dominant in these models especially you
 know this models they will connect the text with render text in the image and
 that that's a very strong signal for what's in the image right this is only
 zero shot though if you switch this to do linear probe so if you actually train
 a linear probe on the representation of clip then these attacks don't work
 anymore so this is going back again to sort of the old-school deep learning
 approach where you actually train a classifier and once you train it picks
 up on on other features and then it doesn't work anymore
 all right yeah so they they evaluate this on a large scale they can't always
 slap a label so they just fill the image with render text and that usually gets
 the classifier confused fairly fairly well they also do this with this strupe
 test which you can do with humans which is fairly difficult if you do it at a
 high speed and they discover that the model basically pays no attention
 whatsoever to the color of the word it pays much more attention to what the
 word says which is strange right because you think if I have a neural network
 and you know it basically needs to to recognize the color here it needs to
 filter out the white pixels but then just average the pixels it gets the
 correct answer that's so easy right it simply averages whereas to recognize
 that this says green is much more difficult but the model was trained to
 connect text and images images which often have text in them so it has
 learned to do OCR basically in the Dolly video I claimed that Dolly has learned
 to do reverse OCR and people correctly pointed out that that is more aptly
 called writing but I love reverse OCR I'm gonna call writing from now on reverse
 OCR so again this is evidence for the claim that this is mostly a textual
 model and now I want to show you what I found so if you're not in the mood I
 have all this in a notion page which I'll link down below so I'll show you
 just some interesting stuff sometimes it's multimodal sometimes it's not
 right so we already were here we just clicked around but now I want to kind
 of show you the good stuff so this is a Superman neuron that I found so it
 responds as you can see to symbols of Superman in the image net data set
 Superman Superman drawing Superman comics Superman spelled out rendered and
 so on this is exactly kind of what what the the article was about right but now
 it's Superman not spider-man this I call the resting bee face neuron so it
 responds to people being slightly annoyed yeah as you can see here this is trash
 bags so this responds to trash bags pretty cool right so not any kind of
 bag right specifically trash bags even if they are not black so there are a
 couple in there they don't necessarily breath black there is even trash cans
 like dump containers right here that have no bag in sight yet still that
 neuron response this sorry about sorry about that yeah for some reason you
 might want to I don't know maybe have something in your pockets yeah so so
 fairly cool oh there's a tree is not always you know perfect but these are
 the data set examples that most excite that neuron so you can also see the text
 isn't always good though I think I think if the text here isn't super good it
 might more be an effect of this method to search text because text is of course
 not a continuous signal so it's fairly hard to search text that maximizes some
 activation otherwise we could build GANs for text very easily which we still
 can't this one here I've titled this strength and a law and weightlifting
 which I'm aware of this is not you know iconography of a law however so this is
 pretty cool as an image right now if you look at what in the data set what
 samples it responds to it's kind of all weightlifting it's all weights so this
 is weight weight and if you go down here to the other data set this is why I
 called it sort of a law because you have also rendered names like the the
 rendered Allah you have the Quran you have symbols of Islam and if you go to
 the text that it searches goes like hammer workout prophet prophet Zana in
 lumber iron gym the brutal workout of God so you know pretty cool neuron
 honestly and you know that it that responds with this I don't even I don't
 even know what what that is is that is that Hindu imagery or Buddhist imagery
 so cool these are organs this is an organ neuron I hope like you you can see
 that and it responds to the render text of control I don't know what to make of
 it also canal viral but also to drawings you can see here a drawing of a heart
 for some reason also chins so it's not always super duper clear what a neuron
 does in fact most of these neurons you will find if you go look at what image
 net sound and these I believe these are crops of image net samples not entire
 pictures so if you look at what by the way control and CTRL if you look at what
 examples most often it will be rendered text so that the image that no matter
 what neuron most neurons actually pay attention to rendered text rather than
 to images the ones I've selected are the ones that do not but if you just go and
 click on some random neuron we can actually try and it's certainly going to
 probably fail this one looks pretty cool looks pretty cool actually that responds
 to printers yep demonstration effect fails horribly how about this one yeah
 so you can see that you know maybe you don't exactly know what that is so you
 want to look at what so here you see that it primarily responds to the text
 miss I guess mss I miss you Mississippi and so on you know Mississippi having
 it twice in there that got a respond pretty pretty heavily and most of the
 time you'll find something like this that it responds very much to the
 rendered pieces of text in images these are film spools and so not only does it
 respond to film spools but also to things like director screening popcorn
 the kind of movie theater labeling showing Hollywood cinemas there's also
 entertainment so you know the multimodality again this this is a this
 is a phenomenon because we introduced the text and it can connect it on the
 text level this is feather patterns and leaf patterns so even when it's in coffee
 you see the feather and leaf patterns even when it's a drawing it can it will
 still respond this one is strange so this responds to things like Sparta and
 front and Troy but so that it responds to rendered front Trojan Spartans front
 and it also has a lot of people doing sort of squats as you can see so and and
 fighting so this is kind of an iron so this is a bit of kind of a warrior
 neurons you can see oh there's lots of ah of course it's because of these
 Spartan runs and all they're called like this right these kind of sporting events
 I see Roman frontside Roman Roman so it connects the workout with the Spartan
 workout kind of division and then it connects the Trojan and so on via again
 via the text because it makes no sense to connect like the vodka and the and
 the weightlifting maybe so yeah I hope I hope you're fairly convinced by now
 we're gonna know a bit faster now because the videos already too long but
 this one here is the letter E so it's e it responds again to rendered text of E
 this one here is cleaning so it responds to cleaning products and cleaning things
 this one here is frown so this is frowning frowning frowning grumpy face
 grumpy face lion lion responding to lions rendered text of lions team names
 called lions and so on fashion model fashion model a bit by the way the
 labels are mine I just looked at them and decided what they are but you can
 see like there's a lot of these kind of runway shots here baseball stadium so
 cool so these are kind of top views of baseball stadium but it responds a lot
 to things saying park PNC park AT&T park but also kind of home team park lights
 and and baseball dugouts and even players I've seen some players logos of
 teams baseball depictions of actual baseballs immense immensely cool here
 bride this is bride you can see this is bride this one what do you think this
 one is Navy so super cool that it can I kind of connect these ropes with the
 emblems the the kind of your tags so and it connects it to render text saying
 Navy right so these are the crops of images that it responds to Navy of fish
 like officers Navy gravestones yeah so cool this one okay this for this I also
 had to look at sort of the pictures here and the text going along with it this is
 hemp but it is also kind of goa patterns it is also for some reason turn or earn
 it is also Hendrix so this isn't even Jimi Hendrix right like this this is
 definitely connected to these goa shirts there is also there's pictures of Jimi
 Hendrix which I guess you can understand there is also turn again
 whereas there's Bob no this is Bob Marley sorry this Bob Marley yeah so so
 it connects these things staircase and here for some reason also responds to
 text rendered human and to staircases and here I have I don't know why but
 there's there's this thing which I'm not sure so it has human in it but it is
 also arranged like a staircase so maybe that's why it responds extra extra yeah
 the Disney neuron this is a Disney neuron how cool is this how cool is this
 so you can clearly see that that but then it you know Disney these are the
 samples that it responds to simply something saying Disney the Mickey Mouse
 ear the mini bow no immensely cool the castle right the Disney castle this is
 the Hillary Clinton neuron you can see this is Hillary and the images it
 responds to is Hillary Hill pill Polly Hill pills so this is maybe it's more
 like the LL why the IL why neuron but it it does pick out Hillary Clinton as
 well yeah so image net of course is older than at least one of Hillary's
 campaigns I'm not sure this is God so I found this one this is yeah God if you
 so the reconstruction process it's not very good at generating text maybe
 because so they have a lot of priors in that if you look at the reconstruction
 article you can probably and they do this in in this article they reconstruct
 text but it's still not super clear maybe it has to do with the architecture
 this here is blurry it's just the concept of blurry so you look at the
 images they're kind of often blurry and if you look at the text going along with
 it it's all like blurry blurry blurry blurry blurry blurry blurry blurry cool
 like it's not even what's on the image but you can clearly see like this comes
 from the other description this is hand-drawn arrows or arrows in general
 this looks like my videos now right like this recognizes arrows is specifically
 a you know kind of color re arrows this one what does it do this is presenting a
 trophy you see this one here in the middle this is kind of so these are all
 you know people presenting some kind of thing holding some kind of thing in
 their hand showing it like fishermen or diplomas this one I was amazed by this
 is a neuron responding to receding hairlines like it responds to receding
 hairlines how cool is that how cool is that this is traffic tent and so on so
 it responds to tents and traffics and crowds of people this one is raised arms
 but also pancakes so pancakes and raised hands for some reason there's a
 connection no but I mean these these models they still overload when they can
 this one how cool is that this is the Google Maps neuron these are
 reconstructions these are not samples these are reconstructions you can see
 it's clearly it has kind of the street labels and the pins on it so this is a
 Google Google Maps like neuron what so cool this one I call nervous smile you
 can maybe see that it's like yeah here's Elvis this is the Elvis neuron I know it
 sort of it also looks like Hendrix a bit but the things it connects it to is
 that's not Elvis that's not Elvis kiss okay maybe it's not exactly Elvis maybe
 it's more like a pop star neuron yeah maybe it's not Elvis only Elvis Billy
 Elliot this one is the flash right that's the flash and the cool thing is
 it responds to images saying flash what okay beards response to beards
 generally beards lots of beards kilts kilts and bagpipes response to guilt
 kilts and bagpipes rainy this is a neuron that responds to things that are rainy
 rainy days so you can see here out the window it's raining rainy windows so
 cool this is flash and electricity so you will see like symbols these symbols
 of these flashes but also kind of electric hair curling up droplets how
 cool does that look like that's just cool and the occasional image net
 reconstruction thing where there must be like half a dog face in there that is
 just trippy this one is this one is escape okay escape like look at that like
 to connect these things how long would you like without contrastive learning
 how well I guess if as long as you have images and labels but still king this is
 king so the depicted are crowns but response to renderings of King this is
 nation how cool is that nation response to country country country oh it's
 country not nation but still this one response to overweight men there's a
 neuron that responds to over phases of overweight men this one is wedding this
 one is Australia and the cool thing here is that it responds to rendered domain
 names of Australia like the top-level domain of Australia what mind-blown this
 is yawning or screaming well I think you know like here we have a same neuron
 for bees and the Simpsons bees and the Simpsons this is muscles and seafood and
 lastly spices spices and other powdery things you know don't ask too many
 questions hmm alright so that was it for me for today I have many more that are
 linked in a notion description somewhere go check it out please try out this I've
 not yet looked through all of them there are so many there are literally
 thousands of these units and this is just one of the models they have
 available go look and share you know on our discord you know the best ones you
 find alright that was it thanks for listening bye bye