The hidden dangers of loading open-source AI models (ARBITRARY CODE EXPLOIT!)
deep learning machine learning arxiv explained neural networks ai artificial intelligence paper wandb huggingface hugging face is hugging face dangerous is ai dangerous ai exploit pickle exploit pytorch exploit is hugging face safe reduce python pickle python pickletools python pickle exploit pytorch pickle exploit ai model backdoor arbitrary code execution pickle code injection pytorch danger pytorch load danger is pytorch safe is pytorch dangerous
#huggingface #pickle #exploit 

Did you know that something as simple as loading a model can execute arbitrary code on your machine?

Try the model: https://huggingface.co/ykilcher/totally-harmless-model
Get the code: https://github.com/yk/patch-torch-save

Sponsor: Weights & Biases
Go here: https://wandb.me/yannic

OUTLINE:
0:00 - Introduction
1:10 - Sponsor: Weights & Biases
3:20 - How Hugging Face models are loaded
5:30 - From PyTorch to pickle
7:10 - Understanding how pickle saves data
13:00 - Executing arbitrary code
15:05 - The final code
17:25 - How can you protect yourself?


Links:
Homepage: https://ykilcher.com
Merch: https://ykilcher.com/merch
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
LinkedIn: https://www.linkedin.com/in/ykilcher

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Well, what do we have here? Totally harmless model. I kind of wonder what it is. Seems to be kind of a Distilbert recent version of Transformers, Flow 32. I like this model. The Hugging Face Hub makes it very easy to try machine learning models. So let's let's give that a go. Python shell. Import auto model, model equals from pre trained. And let's go. And what's happening? Oh, wow. It loaded the model, but it also opened a random website. I don't know what this website is, but it seems very interesting. So if you actually look at that model, then you'll see this is a normal model, it actually works. So this is a model to distill model with all the weights, you can forward pass data through it. So this would pass any test of being a machine learning model. But every time you load it, it also does something else in the background. And that's what we're going to talk about today, the dangers of loading untrusted models, how does this work and how you may protect yourself against this. Just a quick aside, look at this binary number over here, I want you to take the first four of each and just kind of go like small circle and big circle in relation to zeros or one. So like small, big, small, big, small, small, big, small, small, big, small. And that's the logo of weights and biases. Look at this. It's actually pretty, pretty cool. So small, big, small, big, if you look at actually what the number translates to in ASCII, it's W and B. I did not figure this out on my own. Scott pointed it out on Twitter, but he's been working at weights and biases for over a year before he even realized it's just attention to detail. So I just think this is this is very cool. You're in the middle of a sponsor spot, by the way, if you didn't notice the weights and biases is not just a product that I advertise, it's actually a product that I use personally on a daily basis. And so should you weights and biases is a total solution for ml ops from experimentation all the way to deployment and monitoring and it is for everyone academics are using it hobbyists are using it personal accounts are completely free and academic teams as well. But it's not just for individuals very, very large companies are using weights and biases. Now if you happen to be a company small or large, then there's great offerings from weights and biases for you. The weights and biases cloud gives you an all in one solution. But if you're worried about where your data is, you can also go with a self managed instance. And now there is an even better solution. There is a weights and biases dedicated cloud. So what they'll do is they'll pull up an isolated environment on a cloud provider and a region of your choice. And that's just yours. It's managed by the weights and biases team, but it's fully yours. And if like most businesses today, you're on some cloud already, then this is an absolutely great balance between security, privacy and flexibility. Head over to the link one to be.me slash Yannick. This lets them know that I sent you and promise you won't be disappointed again, thanks to weights and biases for sponsoring this video really awesome to have them on board. And now let's get into it. So how does loading a model from the hugging face hub legit hugging face hub model open a random website on your browser as you load the model for that we have to dive a little bit into how the mechanics of saving and loading models work. So the hugging face hub is super popular, obviously for sharing models, getting models out there. And recently, I've been trying out a bunch of models on the hub for a problem that I had. So I just went through here, I was like, okay, I'm looking for image segmentation, filtering down the models. And it occurred to me, wait, I'm just kind of downloading stuff and executing it. Is this safe? And it turns out no, no, it's not safe at all. And the gist is there is absolutely nothing that can be done about it. But with more awareness, I hope the situation is going to improve. Alright, so how do models even get to the hub? And how do you download what happens when you download them? See, if you create a model, if you make a model in hugging face, and you want to save it either locally or on the hub to share it out, you use this function save pre trained. Now save pre trained is a method on a model. And it takes just one mandatory argument, the directory, you want to save it to now, how could that possibly go wrong? Well, you can also see a little bit of the mechanics of how this works already from the function signature. So optionally, it asks you for a state dict, if you don't provide a state dict, it simply takes that state dict from the model that you want to say. So essentially, this saved pre trained function takes the state dict and then saves that. Now, how does it save it? It doesn't use JSON or NumPy or anything like this, because well, JSON is text and is not accurate. And NumPy is very limiting. In fact, since the framework wants to support any kind of models that you might possibly think of, it needs a general protocol of saving and restoring stuff. Now hugging face makes it pretty easy right here. It simply calls this thing called the save function. And the save function by default is just torch dot save. So hugging face takes the state dict and then simply delegates to pytorch to save that and load it again. Save pre trained calls torch dot save and from pre trained calls torch dot load. All right, we're halfway down the rabbit hole. Let's dig into torch dot save. What does it do? So here's the pytorch documentation torch dot saves saves an object to a disk file. Easy enough. You can see here, it takes an object to save no conditions on what that object is, it takes a file like object, something that comes out of a Python open call. And interestingly, it takes a pickle module. And again, you can already see a little bit of how this actually works internally in pytorch documentation of serialization semantics, it says they use Python's pickle file by default. So you can also save multiple tensors or objects like tuples lists and dicts. And yes, if we look at the internals of the save function, then we can see right here, here is that implementation, here is that pickle module. And as we scroll down, we clearly see the pickle module creates a pickler and that pickler simply dumps the object. So what you might say pickle is a standard module of the Python library, it saves stuff to disk and then it loads that stuff up again. Well, let me introduce you to that last level of the rabbit hole. How does pickle work? Now you might think pickle might be something like saving a file to adjacent or a CSV or something like this, something where you take the data and put it on a file. That seems pretty straightforward. However, pickle, as I said, is used to save and load arbitrary things in Python. And since arbitrary things can be well arbitrary, you need an arbitrarily powerful protocol to save and load things. So by necessity, that means this is touring complete code. But let me show you what I mean. So here I have a little Python file, it has a dict. So there's a name and a company entry. And then I simply dump that dict to a file using pickle. All right, executed. Now here's the code to load that very easy. Open the file, pickle dot load, I should get my dict back. And I do. But what is actually in that file, we can look at that file. Well, that's pretty strange. As you can see right here, there's a bunch of signs and then name young company meta. So there seems to be a semblance of the data we put in, there's stuff around it. Now, Python has an internal module that you can use to actually dissect pickle files. It's called pickle tools. So we use it to look at that file. And we see a little bit more what's going on. You don't have to understand all of this. But essentially, here you can see that we first create an empty dictionary, then we load all of the data into memory. So here is name, young company meta. And at the end, we call this set items function. And we can already estimate that what happens here is first an empty dictionary is made, and then it's filled up by that data. It seems to be very specific. And you probably can only do that with dicts and not with an arbitrary object. So let's dig in a little bit deeper. All right, let's get a little bit more complicated. Here I have a class, the class is essentially the same as before, it takes a name and a company and its initializer saves that to the local dict of the instance. And we'll try to save that class to pickle file. All right, done. And let's now inspect that file. What is a slightly more interesting. So again, we'll have this closed curly bracket from before, followed by the data that we gave it. But now we also have this prefix right here, the class name. Interestingly, there's nowhere really a definition of our class. And if we look at the pickle file using pickle tools, you can see the ending is very much the same, there is a build call instead of a set items call. But at the beginning, we also kind of have a main my class stuff in the code right here, indicating that it tries to somehow create or construct or load that class. But you see the general principle, first we'll try to kind of create the object itself. And then we try to fill it in with the data. Now over here, I have the code to load from that file. And watch what happens when I do that, there's an error, it says it can't find my class. So actually, Python doesn't really store the definitions of classes you write into the pickle file. However, at runtime, it tries to automatically get those classes from somewhere and slowly it dawns on you, hey, pickle isn't just saving data to a file and loading that data again, pickle is saving executable code. And when you on pickle something, it actually executes that executable code, whatever that is. And you can nicely demonstrate that. All right, we'll go a couple of steps back, we'll have the original class here again. So this is a class and it has an init method. But I've also defined this method right here called reduce reduces in fact, what pickle calls in Python, lots of things they will call these dunder methods on objects that hook into a protocol and reduce is the hook to hook into pickling. So if I want to modify the pickling behavior of any class, then I have to implement the reduce method. What does the reduce method return? Well, the Python documentation says that the reduce method takes no argument and shall return either a string or preferably a tuple. When a tuple is returned, it must be between two and six items long. The first item is a callable object that will be called to create the initial version of the object. So that means whatever you return from the reduce method, that's the code that will be executed whenever you load the file back up. So the code that you return here is stored as executable code in the file, which will then be executed. So I have my class right here, it has a bunch of data. However, the reduce method simply returns a list actually returns the constructor for a list needs to return a callable and the first argument to that constructor is the list 123. Now I'm going to make that object as before filling it with data. However, if I save that object, watch what happens. So I've done that and just for giggles, I've also simply dumped the list 123. So my object here should have like a young and meta in it. But if we look at the pickle files, built ins list, yeah, none of that. And pickle tools tells us yes, it's importing built ins, it gets the function list, it fills it up with 123. And it depends that to the list. Very good. Now the pickle file for the second thing where I actually just dumped the list is a tiny bit different as it just constructs an empty list from the beginning and then it pushes 123. But it's just a more efficient implementation of doing exactly the same thing. And when I load the two objects up again, and I'm also emitting their type right here, and I'm even checking if they're equal. Then yes, in fact, I just have twice that same list, even though the first one was a pickle of an object that had a name and the company attribute. So again, pickle stores objects by calling their reduce method, whatever that reduce method returns is then executed upon loading. And it's essentially up to the goodwill of people who make these objects or mostly to the default behavior of Python to give you the correct result. However, this is fully executable code and it can do whatever any Python program can do. So why don't we just write a function that opens a web browser and in our reduce function, we simply return that as a callable. Nothing easier than that. Now we actually save it and load it back up. What happens? browser opens, there you go. But you see, there is a little problem right here. As I told you before, we cannot simply do this and then load it up in some other file because we've defined a class right here. And most importantly, we've defined this open browser function that is not going to be available if we upload to the hugging face hub and then someone else downloads it, they're not going to have that open browser function. However, according to the pickle file, that's what's going to be called and it should be in the main module. So we'll need to get a bit more creative to make sure that whatever we want to do is going to be available on any computer that loads up our model. And secondly, you also see that the return type here is none. So we've substituted saving our data and we can now open a browser. However, the user is going to notice something is wrong because they're loading a file and is not actually giving them the thing they want. Now we can solve both of those things with some neat tools of Python called eval and exec Python as you might know is quite dynamic. In fact, it's so dynamic, you can just load up code at runtime and have Python parse the string of code and execute it two methods here are eval and exec. However, eval only works on expressions. So two plus two is an expression because there is a return value, it's four. However, if we try to eval something like import web browser, it's not going to work because that's not an expression import web browser is a statement, we need something that executes statements and that is exec. exec is another function that takes in an argument and simply executes that thing import web browser, good. And now web browser is available. However, exec is not exactly as eval. So if we exec two plus two, it does it but there's no return value. But with a little clever combination of the two, we can achieve anything that we want. So I've written a small library patch towards safe, very small library, you can install directly from GitHub, what you do is you provide a function that you want to execute before any model loads, in this case, opening a web browser, it can be arbitrary Python codes with import statements with whatever you want, you then call my module with that function, which will return a patched version of torch dot save. And now you can provide that patched version to hugging face in the safe pre train. Remember, it takes as an argument, the save function that's usually torch dot save. Now you simply provide that patched function. And that's that if anyone loads your model from local folder from the hub from wherever it is, it will act like a normal model, it will in fact be that model. However, as you load it, that side effect up here will happen. The whole library is just these 21 lines of code, it's actually very small. So here's what I do, I get the source code of that function you provide as a string, I strip away the top, so the def whatever, I just want the body of the function, I indent it by one because I want this to be executable Python code in sort of the top level. And I construct this thing called bad dict, and I replace your dictionary that you want to save that you would give to torch dot save with a bad dict version of it. And then I call torch dot save. So my function is simply a proxy for torch dot save that wraps whatever you want to save into this bad dict class, the bad dict itself has the reduce method implemented, it simply calls a val as a function, the argument to eval is a string with source code, the string with source code does two things. First, it uses exec to execute whatever the body of the function you provided was, and then it simply returns an empty dict, which it later fills with the items of your original dictionary. So line 10 really does most of the work right here. And as you can see, it's astonishingly simple and allows again for arbitrary execution of code. So whatever you could do in Python, any of these models could do as soon as you call from pre trained and you wouldn't even know anything, they could be running some crypto miner in the background, they could be running a key logger, anything that you can think of. So what can be done about it? Pretty sad outlook, if you ask me. Now, if you look into the documentation of the Python pickle module, it very prominently says the pickle module is not secure only on pickle data you trust this will execute arbitrary code during on pickling. So they're very clear what's happening right here. high torch itself in torch dot load, they say warning torch dot load uses the pickle module, which is known to be insecure, it is possible to construct malicious pickle data, which will execute arbitrary code during on pickling never load data that comes from an untrusted source only load data you trust. So both Python and pytorch are adamant about warning you of only loading trusted code. However, on hugging face, I was so far unable to find any of these warnings, not that they would matter much, I guess most people wouldn't read them anyway, but it's simply nowhere. Okay, quick addendum to this video for releasing it, I've actually contacted hugging face and made them aware of the problem and now there is a nice banner nice warning in the hugging face documentation, I feel at some point hugging face just going to be full of features they implemented because I did something stupid, but very appreciated. So there's now warning and I'm going to be working with them to make things more secure, at least to share the little bit I know all the while my model is being marked safe by their malware scanner, but their malware scanner is only just starting to ramp up and it actually looks kind of promising that some of these things can be mitigated. So I'm looking forward to that. If you want to try out totally harmless model feel absolutely free. It's available on the hugging face hub, you're also free to use this library here to create your own funny models that do funny things on loading up and in the spirit of responsible disclosure, I've actually contacted hugging face ahead of time here and warn them and ask them to maybe implement one of the suggestions again, there is very little that can be done other than awareness. So be aware, stay hydrated and I'll see you around. Bye bye.
 Well, what do we have here? Totally harmless model. I kind of wonder what it is. Seems
 to be kind of a Distilbert recent version of Transformers, Flow 32. I like this model.
 The Hugging Face Hub makes it very easy to try machine learning models. So let's let's
 give that a go. Python shell. Import auto model, model equals from pre trained. And
 let's go. And what's happening? Oh, wow. It loaded the model, but it also opened a random
 website. I don't know what this website is, but it seems very interesting. So if you actually
 look at that model, then you'll see this is a normal model, it actually works. So this
 is a model to distill model with all the weights, you can forward pass data through it. So this
 would pass any test of being a machine learning model. But every time you load it, it also
 does something else in the background. And that's what we're going to talk about today,
 the dangers of loading untrusted models, how does this work and how you may protect yourself
 against this. Just a quick aside, look at this binary number over here, I want you to
 take the first four of each and just kind of go like small circle and big circle in
 relation to zeros or one. So like small, big, small, big, small, small, big, small, small,
 big, small. And that's the logo of weights and biases. Look at this. It's actually pretty,
 pretty cool. So small, big, small, big, if you look at actually what the number translates
 to in ASCII, it's W and B. I did not figure this out on my own. Scott pointed it out on
 Twitter, but he's been working at weights and biases for over a year before he even
 realized it's just attention to detail. So I just think this is this is very cool. You're
 in the middle of a sponsor spot, by the way, if you didn't notice the weights and biases
 is not just a product that I advertise, it's actually a product that I use personally on
 a daily basis. And so should you weights and biases is a total solution for ml ops from
 experimentation all the way to deployment and monitoring and it is for everyone academics
 are using it hobbyists are using it personal accounts are completely free and academic
 teams as well. But it's not just for individuals very, very large companies are using weights
 and biases. Now if you happen to be a company small or large, then there's great offerings
 from weights and biases for you. The weights and biases cloud gives you an all in one solution.
 But if you're worried about where your data is, you can also go with a self managed instance.
 And now there is an even better solution. There is a weights and biases dedicated cloud.
 So what they'll do is they'll pull up an isolated environment on a cloud provider and a region
 of your choice. And that's just yours. It's managed by the weights and biases team, but
 it's fully yours. And if like most businesses today, you're on some cloud already, then
 this is an absolutely great balance between security, privacy and flexibility. Head over
 to the link one to be.me slash Yannick. This lets them know that I sent you and promise
 you won't be disappointed again, thanks to weights and biases for sponsoring this video
 really awesome to have them on board. And now let's get into it. So how does loading
 a model from the hugging face hub legit hugging face hub model open a random website on your
 browser as you load the model for that we have to dive a little bit into how the mechanics
 of saving and loading models work. So the hugging face hub is super popular, obviously
 for sharing models, getting models out there. And recently, I've been trying out a bunch
 of models on the hub for a problem that I had. So I just went through here, I was like,
 okay, I'm looking for image segmentation, filtering down the models. And it occurred
 to me, wait, I'm just kind of downloading stuff and executing it. Is this safe? And
 it turns out no, no, it's not safe at all. And the gist is there is absolutely nothing
 that can be done about it. But with more awareness, I hope the situation is going to improve.
 Alright, so how do models even get to the hub? And how do you download what happens
 when you download them? See, if you create a model, if you make a model in hugging face,
 and you want to save it either locally or on the hub to share it out, you use this function
 save pre trained. Now save pre trained is a method on a model. And it takes just one
 mandatory argument, the directory, you want to save it to now, how could that possibly
 go wrong? Well, you can also see a little bit of the mechanics of how this works already
 from the function signature. So optionally, it asks you for a state dict, if you don't
 provide a state dict, it simply takes that state dict from the model that you want to
 say. So essentially, this saved pre trained function takes the state dict and then saves
 that. Now, how does it save it? It doesn't use JSON or NumPy or anything like this, because
 well, JSON is text and is not accurate. And NumPy is very limiting. In fact, since the
 framework wants to support any kind of models that you might possibly think of, it needs
 a general protocol of saving and restoring stuff. Now hugging face makes it pretty easy
 right here. It simply calls this thing called the save function. And the save function by
 default is just torch dot save. So hugging face takes the state dict and then simply delegates
 to pytorch to save that and load it again. Save pre trained calls torch dot save and
 from pre trained calls torch dot load. All right, we're halfway down the rabbit hole.
 Let's dig into torch dot save. What does it do? So here's the pytorch documentation torch
 dot saves saves an object to a disk file. Easy enough. You can see here, it takes an
 object to save no conditions on what that object is, it takes a file like object, something
 that comes out of a Python open call. And interestingly, it takes a pickle module. And
 again, you can already see a little bit of how this actually works internally in pytorch
 documentation of serialization semantics, it says they use Python's pickle file by default.
 So you can also save multiple tensors or objects like tuples lists and dicts. And yes, if we
 look at the internals of the save function, then we can see right here, here is that implementation,
 here is that pickle module. And as we scroll down, we clearly see the pickle module creates
 a pickler and that pickler simply dumps the object. So what you might say pickle is a
 standard module of the Python library, it saves stuff to disk and then it loads that
 stuff up again. Well, let me introduce you to that last level of the rabbit hole. How
 does pickle work? Now you might think pickle might be something like saving a file to adjacent
 or a CSV or something like this, something where you take the data and put it on a file.
 That seems pretty straightforward. However, pickle, as I said, is used to save and load
 arbitrary things in Python. And since arbitrary things can be well arbitrary, you need an
 arbitrarily powerful protocol to save and load things. So by necessity, that means this
 is touring complete code. But let me show you what I mean. So here I have a little Python
 file, it has a dict. So there's a name and a company entry. And then I simply dump that
 dict to a file using pickle. All right, executed. Now here's the code to load that very easy.
 Open the file, pickle dot load, I should get my dict back. And I do. But what is actually
 in that file, we can look at that file. Well, that's pretty strange. As you can see right
 here, there's a bunch of signs and then name young company meta. So there seems to be a
 semblance of the data we put in, there's stuff around it. Now, Python has an internal module
 that you can use to actually dissect pickle files. It's called pickle tools. So we use
 it to look at that file. And we see a little bit more what's going on. You don't have to
 understand all of this. But essentially, here you can see that we first create an empty
 dictionary, then we load all of the data into memory. So here is name, young company meta.
 And at the end, we call this set items function. And we can already estimate that what happens
 here is first an empty dictionary is made, and then it's filled up by that data. It seems
 to be very specific. And you probably can only do that with dicts and not with an arbitrary
 object. So let's dig in a little bit deeper. All right, let's get a little bit more complicated.
 Here I have a class, the class is essentially the same as before, it takes a name and a
 company and its initializer saves that to the local dict of the instance. And we'll
 try to save that class to pickle file. All right, done. And let's now inspect that file.
 What is a slightly more interesting. So again, we'll have this closed curly bracket from
 before, followed by the data that we gave it. But now we also have this prefix right
 here, the class name. Interestingly, there's nowhere really a definition of our class.
 And if we look at the pickle file using pickle tools, you can see the ending is very much
 the same, there is a build call instead of a set items call. But at the beginning, we
 also kind of have a main my class stuff in the code right here, indicating that it tries
 to somehow create or construct or load that class. But you see the general principle,
 first we'll try to kind of create the object itself. And then we try to fill it in with
 the data. Now over here, I have the code to load from that file. And watch what happens
 when I do that, there's an error, it says it can't find my class. So actually, Python
 doesn't really store the definitions of classes you write into the pickle file. However, at
 runtime, it tries to automatically get those classes from somewhere and slowly it dawns
 on you, hey, pickle isn't just saving data to a file and loading that data again, pickle
 is saving executable code. And when you on pickle something, it actually executes that
 executable code, whatever that is. And you can nicely demonstrate that. All right, we'll
 go a couple of steps back, we'll have the original class here again. So this is a class
 and it has an init method. But I've also defined this method right here called reduce reduces
 in fact, what pickle calls in Python, lots of things they will call these dunder methods
 on objects that hook into a protocol and reduce is the hook to hook into pickling. So if I
 want to modify the pickling behavior of any class, then I have to implement the reduce
 method. What does the reduce method return? Well, the Python documentation says that the
 reduce method takes no argument and shall return either a string or preferably a tuple.
 When a tuple is returned, it must be between two and six items long. The first item is
 a callable object that will be called to create the initial version of the object. So that
 means whatever you return from the reduce method, that's the code that will be executed
 whenever you load the file back up. So the code that you return here is stored as executable
 code in the file, which will then be executed. So I have my class right here, it has a bunch
 of data. However, the reduce method simply returns a list actually returns the constructor
 for a list needs to return a callable and the first argument to that constructor is
 the list 123. Now I'm going to make that object as before filling it with data. However, if
 I save that object, watch what happens. So I've done that and just for giggles, I've
 also simply dumped the list 123. So my object here should have like a young and meta in it.
 But if we look at the pickle files, built ins list, yeah, none of that. And pickle tools
 tells us yes, it's importing built ins, it gets the function list, it fills it up with
 123. And it depends that to the list. Very good. Now the pickle file for the second thing
 where I actually just dumped the list is a tiny bit different as it just constructs an
 empty list from the beginning and then it pushes 123. But it's just a more efficient
 implementation of doing exactly the same thing. And when I load the two objects up again,
 and I'm also emitting their type right here, and I'm even checking if they're equal. Then
 yes, in fact, I just have twice that same list, even though the first one was a pickle
 of an object that had a name and the company attribute. So again, pickle stores objects
 by calling their reduce method, whatever that reduce method returns is then executed upon
 loading. And it's essentially up to the goodwill of people who make these objects or mostly
 to the default behavior of Python to give you the correct result. However, this is fully
 executable code and it can do whatever any Python program can do. So why don't we just
 write a function that opens a web browser and in our reduce function, we simply return
 that as a callable. Nothing easier than that. Now we actually save it and load it back up.
 What happens? browser opens, there you go. But you see, there is a little problem right
 here. As I told you before, we cannot simply do this and then load it up in some other
 file because we've defined a class right here. And most importantly, we've defined this open
 browser function that is not going to be available if we upload to the hugging face hub and then
 someone else downloads it, they're not going to have that open browser function. However,
 according to the pickle file, that's what's going to be called and it should be in the
 main module. So we'll need to get a bit more creative to make sure that whatever we want
 to do is going to be available on any computer that loads up our model. And secondly, you
 also see that the return type here is none. So we've substituted saving our data and we
 can now open a browser. However, the user is going to notice something is wrong because
 they're loading a file and is not actually giving them the thing they want. Now we can
 solve both of those things with some neat tools of Python called eval and exec Python
 as you might know is quite dynamic. In fact, it's so dynamic, you can just load up code
 at runtime and have Python parse the string of code and execute it two methods here are
 eval and exec. However, eval only works on expressions. So two plus two is an expression
 because there is a return value, it's four. However, if we try to eval something like
 import web browser, it's not going to work because that's not an expression import web
 browser is a statement, we need something that executes statements and that is exec.
 exec is another function that takes in an argument and simply executes that thing import
 web browser, good. And now web browser is available. However, exec is not exactly as
 eval. So if we exec two plus two, it does it but there's no return value. But with a
 little clever combination of the two, we can achieve anything that we want. So I've written
 a small library patch towards safe, very small library, you can install directly from GitHub,
 what you do is you provide a function that you want to execute before any model loads,
 in this case, opening a web browser, it can be arbitrary Python codes with import statements
 with whatever you want, you then call my module with that function, which will return a patched
 version of torch dot save. And now you can provide that patched version to hugging face
 in the safe pre train. Remember, it takes as an argument, the save function that's usually
 torch dot save. Now you simply provide that patched function. And that's that if anyone
 loads your model from local folder from the hub from wherever it is, it will act like
 a normal model, it will in fact be that model. However, as you load it, that side effect
 up here will happen. The whole library is just these 21 lines of code, it's actually
 very small. So here's what I do, I get the source code of that function you provide as
 a string, I strip away the top, so the def whatever, I just want the body of the function,
 I indent it by one because I want this to be executable Python code in sort of the top
 level. And I construct this thing called bad dict, and I replace your dictionary that you
 want to save that you would give to torch dot save with a bad dict version of it. And
 then I call torch dot save. So my function is simply a proxy for torch dot save that
 wraps whatever you want to save into this bad dict class, the bad dict itself has the
 reduce method implemented, it simply calls a val as a function, the argument to eval
 is a string with source code, the string with source code does two things. First, it uses
 exec to execute whatever the body of the function you provided was, and then it simply returns
 an empty dict, which it later fills with the items of your original dictionary. So line
 10 really does most of the work right here. And as you can see, it's astonishingly simple
 and allows again for arbitrary execution of code. So whatever you could do in Python,
 any of these models could do as soon as you call from pre trained and you wouldn't even
 know anything, they could be running some crypto miner in the background, they could
 be running a key logger, anything that you can think of. So what can be done about it?
 Pretty sad outlook, if you ask me. Now, if you look into the documentation of the Python
 pickle module, it very prominently says the pickle module is not secure only on pickle
 data you trust this will execute arbitrary code during on pickling. So they're very clear
 what's happening right here. high torch itself in torch dot load, they say warning torch
 dot load uses the pickle module, which is known to be insecure, it is possible to construct
 malicious pickle data, which will execute arbitrary code during on pickling never load
 data that comes from an untrusted source only load data you trust. So both Python and pytorch
 are adamant about warning you of only loading trusted code. However, on hugging face, I
 was so far unable to find any of these warnings, not that they would matter much, I guess most
 people wouldn't read them anyway, but it's simply nowhere. Okay, quick addendum to this
 video for releasing it, I've actually contacted hugging face and made them aware of the problem
 and now there is a nice banner nice warning in the hugging face documentation, I feel
 at some point hugging face just going to be full of features they implemented because
 I did something stupid, but very appreciated. So there's now warning and I'm going to be
 working with them to make things more secure, at least to share the little bit I know all
 the while my model is being marked safe by their malware scanner, but their malware scanner
 is only just starting to ramp up and it actually looks kind of promising that some of these
 things can be mitigated. So I'm looking forward to that. If you want to try out totally harmless
 model feel absolutely free. It's available on the hugging face hub, you're also free
 to use this library here to create your own funny models that do funny things on loading
 up and in the spirit of responsible disclosure, I've actually contacted hugging face ahead
 of time here and warn them and ask them to maybe implement one of the suggestions again,
 there is very little that can be done other than awareness. So be aware, stay hydrated
 and I'll see you around. Bye bye.