Memsad
Sprundel, Ilja van
This presentation will start off with a simple problem (how do you clear memory that holds sensitive content). It explores numerous possible solutions, and presents real live facts and figures. bugs in common applications will be shown.
 Mem said that's the title of the next talk. Why cleaning memory is hard? Ilya Fansprudl is a security researcher who loves to find out new things. And he found out that it's quite hard to get rid of sensitive content in the memory. So today he's going to give us an overview, presentation. Please give a warm round of applause to Ilya Fansprudl. OK, perfect. Yeah, so as the hair will just explain, my presentation is called MENSAD. Why Clearing Memory is Hard. Before I dive in that, once upon a time that was me. A lot more hair, a lot less fat. This is my 17th Congress... You have pot kettle here, buddy. This is my 17th Congress in a row. I've spoken here a number of times before. I haven't added 35 in there yet, but obviously that should mean there as well. I work for a company called IOActive. I'm the director of penetration testing. But that really just means that I lead teams of pentesters. No double entendre there. Obviously, we're always looking for good security people, so if you're interested, come talk to me afterwards. I like looking at low level stuff, kernels, drivers, type of reasons, that type of stuff. And I enjoy reading code. Okay, enough about me. What's the kind of audience that I think would enjoy this? Pretty all around security people, crypto people, if you like code review, if you like compiler stuff, and if you're just generally curious about technology, I think you might enjoy this. In terms of the knowledge required, the first half of these slides is relatively basic. And so if you have a basic technology understanding, you should be able to understand most of the first half. If you have some C background, that would be nice. And then as I move forward past the first half, things become a bit more advanced. But if you just only grasp the first half, I think that will still be useful. Right. So what does this talk about? Actually it's one very simple, easy to explain crypto implementation problem. And the reason I've dedicated an entire talk about it is because while the problem is easy, the solution is not. There's a lot of moving parts, a lot of subtlety, a lot of nuance. And I'll get into that in a little bit. Now I can hear some of you thinking, well, Ilya, WTF, 2018, why the hell are you talking about this? This is very, very, very well known. To them, I say, well, because this stuff is still everywhere. And I will show that in the slides. I will show this with data. I will show this with bugs. But the driver of this talk, the reason why I started making this presentation is that this year alone, I did engagements for three different customers on three entirely different software projects where they all had this exact same type of bug. And you tell them about the bug, and the customer goes back and says, OK, well, yeah, that's great. We understand. Now tell us how to fix this in a portable way. And that is not very easy. The other thing is that even though this problem is known conceptually, people kind of blasé about it, practically not many people understand how out pervasive this problem is, how realistic it is. It isn't just like, oh, well, the compiler might do this. No, the compiler will do this. And it does it everywhere. And these bugs do show up everywhere. Even though, because it doesn't, it's hard to tell from the code that it is there. If you look at the binary, if you look at the compiler in bits, you see that it's there. And then the third is, given that one of the teams of the Congress this year is Foundations and to talk about things that aren't necessarily new, but to sort of try and help bring the subject to the next generation, I think this fits in perfectly in the concept of Foundations. Right. So, before I dive in, there's a couple of people, we're actually a long list of people that sort of helped me out. As I said, the problem is well known. It's been well known for at least 20 or 30 years. And so many, many, many people have published papers and presentations about this. And I don't know all of them personally. And I wish I could include them all. But the people I've included in here are sort of the one or two away from me that have had some kind of impact in these slides. Some of you are sitting in this audience. And your help has been appreciated. Okay. So, let's actually start. Now let's say you're going to write some piece of code, right? And it's going to be doing something and it's going to be handling sensitive data. Be that keys or decrypted plain text or session tokens or passwords or password hashes or anything that could be considered sensitive, right? Now, if you're a smart security conscious person, right, the moment you are done with that sensitive data, you want to dispose of it, right? You want to purge it from memory, right? Now why do you want to do this? Well, because otherwise, if there's some kind of infolique that is discovered later on, then whatever secrets or lingering memory could be used in your infolique and all of a sudden, you know, your tokens or your keys are leaked out, right? And I mean, that may sound like a stretch, but, you know, things like heart bleed, you know, happen, right? So this is very practical. This can really happen. And for a move on, I want to say, if you think and make the step where you say, okay, I need to dispose of sensitive material once I'm done with it, that's really big, right? Your software that deals with, you know, sensitive material does not do this. So if you make the step thinking I need to purge this, you're ahead of the curve, right? So now, concretely, it would look something like this, right? You would write a little, this is a sample code I have that basically generates a little key and it's a function you give it and you give it a key pointer and this thing declares a local variable that search invites, goes and reads a bunch of random bits, puts it in K, copies K to key and then before it returns, because K is about to go out of scope and it contains sensitive key material, you go and say memset and then you clear the thing and then you return, right? Perfect. You run this, you compile it, you add a main and it does exactly what it's supposed to do. You just use the assembly and it's all perfect. Problem there is what you're doing is that's not release based code, right? When you sort of make code ready to be released, you sort of tell the compiler that it should enable the optimizer, right? You'll give it OS or O2. Those are the most common ones. You just see O3 when people want to live on the edge but usually O2 or OS. Now if you look at the assembly again, you get a whole different picture of what's going on and I want to illustrate this and there's a website called Compiler Explorer which is beautiful, integrates a whole bunch of compilers and it has on the left it shows you the C code and the right shows you the assembly and it's like color based and it's easy to make connections. So let's take our little example and on the left we see the generate key and on the right we see the compiler and sure enough if you follow the colors and left and right you can see what C code translates to which assembly, right? If they make it a little bit easier, that memset clearly gets translated to assembly, right? Now that is when you do minus O0 which is the default which is what you would do if you're developing code and you want to debug this stuff, right? Now once you're done developing and you're about to ship this thing, you do minus O1 for example and the optimizer kicks in and all of a sudden your assembly looks a whole lot different. You'll notice it's shorter. You'll notice that all of a sudden the color of your memset changed whereas in O0 it was this sort of red-ish and all of a sudden it became white and it's nowhere to be found in your assembly, right? It just does not show up, right? That's a problem, okay? Well, what happened, right? Let's, yeah, I stole that. So what happened is a thing called dead store optimization or dead store elimination. And so basically that memset at the end that what you're doing is you are writing into a buffer that is never, ever going to get used again. And an optimizer compiler looks at that and says, hey, you know, I can just take that memset out and I just saved you a couple of cycles and you have a smaller binary, huge win. And because it doesn't really change what the program does, it's fully compliant with all the relevant language standards. Right, so that's essentially, that is in a nutshell our problem, right? And so one of the things I want to do is I want to look at all common compilers and see for which of these I can get it to effectively, practically optimize out a memset like this. And I had to, with some of them it was easy, some of it was harder. I had to fill around with it for some straight up memset works for others. I had to like kind of twiddle and make a for loop or you know, kind of jump around a bit. But essentially that these are lists of 10 compilers I tested. I try to get my hands on the IBM compiler, but I don't have $20,000, so I couldn't do that. But these are the ones that I did test. And then so the first five or the first four you will know, you know, the GCC and Clang and Intel compiler and the Microsoft compiler. And they're all also on the compiler explorer, so it was easy to test those. And it was very easy to get those to optimize out memsets. And then I moved on, downloaded a bunch of others. You know, the Sun Studio compiler and the, and Barcadero's SQL Splash Builder and the ARM compiler and a bunch of others. And out of these 10, I was able to get eight to optimize it out, right? 80% of the most common compilers do this in any, in a practical sense. So it isn't just like a theoretical thing. This really happens. A funny note, I tried really hard to get the PGI compiler to do it. In fact, it has a switch called that's for elimination. And of course I played with it and I tried it and I spend over an hour trying to get it to optimize out my memset. God damn thing wouldn't move. I don't know what it, I don't know what it's doing there. I couldn't get it to do anything. But basically most compilers, if you ask them to do optimization, will gladly optimize out a lot of memsets. Right, so the next question is how common is it to actually see projects use optimization, right? And this sort of stems from a conversation I had earlier this year with a couple of colleagues where a bunch of people said, well, you know, I don't see 02 or 01 or OS all that often. I don't think optimization is all that common. And so I started looking around and I said, okay, well, where can I get some data? And the first thing I saw was, okay, well, I can go to opsewers.apple.com and that lists about 200 projects and I'll just go through all their make files and look for 02 or OS and so on. And there's about 100 out of there. And then I realized that they actually don't use make files. They have a really bizarre build system and that build system, by default, uses OS. So even though it says 100 out of 200, it's probably closer to 200 out of 200. And then I had a whole list of programs I wanted to test in like FreeBSD and you've put into it a bunch of Linux distros. But that's pretty boring and I ran out of time. So I kind of stopped there. But these numbers should be good enough. In addition, if you look at the common IDEs, in particular Visual Studio and Xcode, when you tell them to build a project in release mode, Visual Studio by default does 02. Xcode by default does OS. So the fact that these tools by default give you optimization should make you confident enough in knowing that, yes, in fact, optimization is incredibly common in release builds. It isn't everywhere, but it is almost everywhere. Right. So now that we know the problem and now that we know it isn't just theoretical and that we know it's practical and that, in fact, it does occur very, very often and with most compilers, in fact, basically it's a real problem. How do we fix this? And this is sort of where things get difficult. There are many sort of solutions. Nothing is portable. Right. It's sort of the, OK, well, this solution works if you use this compiler and this solution works if you use this Lipsy and this solution works if you use this OS and this solution works with this version of the language spec and this solution works if you have this particular executable file format. Right. So and before I dive into any of those, let's first talk about the elf in the room. Don't just roll your own. I've seen people do this where they go like, oh, well, you know, I'll just, I'll fight with the compiler. I know what I'm doing and they'll just, you know, they'll kind of leader where Jenkins this and they'll totally screw it up and they'll come up with, you know, some really stupid ID. One of the ones I heard was like, oh, well, I'll just, you know, I'll just do IO with the buffer and then it's cool. Yeah, you could do that, but then you're doing IO, right? For no reason. So don't just roll your own. You're going to come up with a solution that's probably stupid. You're going to look really stupid. And it'll be one of these things where, okay, you're sort of, your bad solution might work for this particular version of the compiler, but if you don't understand the concepts behind it, then chances are the next version of the compiler that is somewhat slightly smarter will sort of just bypass whatever you implement it. So don't roll your own or if you want to roll your own, at least listen to the advice of the next 10 and then base your solution on at least some of the advice that I'll be giving out in the next couple of slides. So with that, let's move on to actual solutions. The first one is a Lipsy function called explicit B0. This is not part of any standard, as far as I can tell, at the present time. But this was sort of concocted in May 2014 by the OpenBSD guys. If you note the date, it's pretty close to what hardly happened. It's a few months later. I think that may have some relation. Anyway, this function basically does a B0, but explicitly guarantees that it does not get optimized out. Now what that means is that what it does or it doesn't, it's no longer your problem, it's the Lipsy's problem because they've made the guarantee, so now it's on them. So that's really nice. OpenBSD did first and then NetBSD said, that's a great idea and we're going to steal it. But we're going to rename it though. So they changed the name to explicit memset, but it's essentially the same thing. And then about two years ago, FreeBSD came up sort of the same thing. And then almost two years ago, the GLipsy guys came up with this too. And then DietLipsy supports this too. OSX, however, does not support it. So if you're limited to those platforms, explicit B0 is a perfect solution. Similarly, if you are developing for the Windows world, there is an API called Secure Zero Memory which basically is Microsoft saying, we guarantee that this thing doesn't get optimized out. So if you want to securely clear sensitive material, just use this API. MSN says it will ensure that your data will be overwritten promptly. And this is one of the cases where Microsoft was ahead of the curve by like 15 years. They've had this thing since the early 2000s. It was in XP and it was in Windows 2003. Both operating systems are no longer supported, but the API is. OK, so now there's another function called memset underscore s. And it guarantees it doesn't get optimized out. And it's guaranteed by spec. It's guaranteed by the language spec. It is standardized. It is in C11. It is wonderful. It is great. Except it's not great. Because even though it's in the standard and it's there, it's in what's called the optional NXK. And if you read the spec, and it's like pages and pages of boring crap. But if you end up reading the K2, it says optional extension. What does optional extension mean? It means you can be entirely C11 compliant and not offer memset s. So it's kind of this as Reverend Lafayette would say, yes, but if, no, it abut. So if it's there, great. If it's not, it has the potential of being this great portable solution and then it isn't. Right. And then, of course, the sort of obvious choice, but somehow a lot of people seem to miss this is if you end up doing something with sensitive material, chances are you're using a crypto library. And if you're using crypto library, chances are the crypto library offers you an API to do secure memory cleaning. And so I listed the common ones. If you're using OpenSL, there's OpenSL cleanse and OpenSL guarantees they don't get optimized out. If you use Canutee less, there's Canutee less memset. Same thing, Canutee less guarantees it does not get optimized out. If you're using LeapSodium, which is one of the newer ones, they have sodium memzero. Same thing, they guarantee it doesn't get optimized out. And I'll get down to a minute. So this is basically up until here, I've sort of given you a list of, okay, well, here are specific API functions you can call. If you're using this library or using this OS, use this. The next sort of solutions are sort of the, okay, well, what if you can't rely on the APIs? Maybe we can get something out of the compiler. The first solution is, and this isn't portable, but most compilers have this or something like it where you can go to the compiler and say, hey, don't use the built-in memset, just use the one from libc. And what that means is you tell the compiler that it shouldn't assume that it knows what memset does. And if you do that, then sure enough, memset won't get optimized out. GCC, this is originally GCC specific. And then the Intel compiler supports it too. And then Clang supports it too, but up until, and this is true, up until Clang 3.7, which is maybe two years old, it's not that old. Clang basically supported FNobilt in memset. And then what they did is they kind of dropped it on the floor and it got optimized out anyway. So it's kind of annoying. It kind of ruins the whole use this because it works, except if you're using an older version of Clang. And also, it's not overly portable, but it's a solution that works. Other things might still get optimized out. If you have some kind of for loop that clears memory, that might still get optimized out. But at least if you use memset and you're using no built-in memset, then you have a pretty strong guarantee that it shouldn't get optimized out. So another sort of solution is just don't use optimization. That works. You're guaranteed not to get optimized if you don't use the optimizer. Obviously that isn't perfect. For one, Fortify source doesn't work if you don't use optimization. So if you want to use Fortify source, you have to use optimization. The other one, of course, is that you've got to change your build environment. I guess it's not portable, but then again, most compiles will have some way to tell it to not optimize anything. But obviously the reason you don't want to use this particular solution is because you don't get the optimizer. So your product will probably be slower. Sort of a spin-off of this is some compiles, in particular the Microsoft one. And then GCC also kind of supports it, is where you can localize optimizations based on scopes and functions. And so you can say, oh, for this function, do o0. It's not a commonly used feature. It seems like it might have some side effects and it doesn't support all switches that the compiler generally does. I've seen this recommended by a few people. I played around with it. It seems to work. But it doesn't seem to be a sort of commonly adopt way of doing things. The other thing, of course, is again, this is, I mean, these are pragmas. This is very, very compiler-specific stuff. Another solution is using what's called weak symbols. Anybody familiar with weak symbols? Oh, yeah, a little bit. I'll try to sort of very briefly get into this. So the L file format basically is a format that specifies how to have an executable that can run on OSes that support this file format. So, Ls, if it's compiled, for example, for Linux, it'll be in this particular format. And one of the, obviously, as part of format, you can store symbols for things like functions and variables and so on. And generally, a symbol is what's called a strong symbol. You can mark one as weak. And what weak means is that a symbol may change in runtime. And what that means is if you declare a function or a symbol of a function as weak, that means that compile time, the compiler would have a very hard time to reason about what that thing does because of the sheer fact that you've declared it as weak. And in fact, this particular solution is what it means to use this in their implementation of explicit B0. And what I really like about this is that this is the commit message for the OpenBZ guys. And they're very pragmatic about it. They say, well, you know, we think our solution is pretty clever, but it's not foolproof. There are still ways to defeat this. And they list, you know, a bunch of ways to do this. In particular, you know, well, you know, the compiler could emit runtime code that checks what this thing is in runtime right before it's called. And then you could still optimize it out if the thing matches or doesn't match. But then they go on and say, well, in the foreseeable future, we don't think that's going to happen. But it's possible that at some point down the road this might happen. And so I like the way of reasoning about this, where solution is pretty clever, but it's not foolproof. It may at some point in the future break. But at the present time, it's a fairly good solution, I think. Right. So another solution is to use memory barriers. How many people know what a memory barrier is or what it does? Okay, about the same number of hands. Okay. Let me try to very briefly sort of explain what a memory barrier is. And bear with me here because it's, and I'm going to oversimplify it because it's not a particularly simple concept if you've never heard of it. But okay, let's say you have a piece of code, two global variables, A and B, and you assign a value. You say A equals something, and B equals something, right? And there's no relation between A and B. Right? What that means is both the compiler and the hardware, because they have no relation. Both of them are allowed to reorder it. So B can be assigned first, and then A can be assigned later. Because there's no correlation, that's perfectly valid. Now let's say you have a second thread somewhere, and your second thread says, okay, you know, while not B spin, and then once B is set, you use A. Right? This is sort of where you're basically waiting on something to be set, and the idea is that you wrote your code to that B set after A is set. And that seems logical, and that would work, except if the compiler and hardware don't know, there's a relation between your loop on one end and the assignment on the other, and either the hardware or the compiler reorders it and sets B before it sets A. Really, really nasty things happen. This has been the source of numerous security bugs, very, very subtle stuff. The K, SAN, and T, SAN kind of stuff. We've seen Linux kernel last couple of years. A bunch of that is related to these kind of bugs. And so the way that you fix this is you introduce what's called a memory barrier, and that is basically when you write your code and you say A equals something, and then before you say B equals something, you basically, and it will say memory barrier, and then you say B equals something. And what that means is it gives a signal to the hardware and to the compiler, and it says, whatever happens before this and after this, you are not allowed to reorder this. There is a correlation there that I know that you're not aware of, so don't reorder it. And I hope I explained it well. This usually takes a lot longer to explain, but I hope I got the message across well enough to sort of give you an idea of what a memory barrier is. And now the cool thing about memory barriers is that it's a way for a programmer to tell the hardware or the compiler, I know something about this memory, you don't. Stay away, don't touch. And because of, I mean, it works for reordering, but it also works really well to not get something optimized out, right? The idea is you could basically just do your memset, and then on the thing you memset it, you basically do a memory barrier, and that tells the compiler not to optimize it out. And I know the concept sounds complicated, but it's pretty clever, and I've oversimplified this because it's a relatively complicated subject. But this works really well, and this is used by Dytlipsy, and it's used by Glypsy, and Nginx recently had a fix where they have their own explicit memzero, and it also uses a memory barrier. So this is a tried and tested concept, and it works. So those are kind of the solutions that are known and that work, and have been tried and tested by various fairly well-known pieces of software. If none of those things, if somehow you're in an environment somewhere, and none of this is available to you, or it's not portable enough, and you're looking for a solution that works everywhere, the best you can do is fall back on constructs that are known in the C language, and this is basically the use of the volatile keyword. I call this a fallback. People often go, well, just use volatile, and then that solves the problem. It turns out optimizers can be very clever and very tricky, and even when you use volatile, there are cases that can be made where if the optimizer is clever enough, your data may still get optimized out. So the volatile solutions are sort of best effort fallback solutions, and there's sort of two variants of this. One is a volatile pointer, right? That's the fallback solution in the Lipsodium, and the other one is a volatile memset function pointer, which is what OpenSL uses. And here's what that looks like. This is the Lipsodium fallback, and this looks like, you know, volatile is sort of the, you know, you tell the compiler, hey, you know, I know something about this, you don't. And this is where it gets very language-loyalry. If you look at the spec, it says something along the lines of the access object, something, something, and they're describing the actual memory volatile, not just the pointer L value. And so if the compiler looks at this code, and it can trace, and it can prove that wherever PNT came from, and if that isn't actually volatile, then this volatile doesn't really mean all that much, and it can still optimize it out. That sounds very theoretical. I don't know if that actually happens, but a number of people smarter than me or that know more about the sort of, this negrity little C language things have told me that, yes, in fact, you would be allowed to do that if you're a very smart, optimized, and compiler. The fact that it's a fallback solution for Sogium and a few others leads me to believe that it probably doesn't, but it could. Right. And so this is the solution that OpenSL uses, which also uses volatile. It doesn't do a pointer right, but instead it creates a volatile function pointer that points to memset. And it sort of gives you, you get the same concept more or less as with the weak symbols. The idea being is that your volatile function pointer can change at any time without the compiler knowing about it. And that seems like a pretty good solution, except when you, one way of, in theory, getting around this is if the compiler emits runtime code that right before the function pointer gets called, it looks and goes, well, like it captures it and then goes, is it memset or is it something else? If it's something else, then we call it. If it's memset, we just return and then you optimize out the runtime and save a few cycles. In theory, the compiler is allowed to do that and emit code like that. And I don't know if that actually happens anywhere, but it's a possibility. So think of these last two solutions as fallback. They may not work in theory. In reality, they probably do. Right. So this is sort of the first half of my presentation and I'm perfectly on time, which is great. So now there isn't one portable solution and this is why, this is why clear memory is hard. The problem is well understood, but if you're looking for like an all-around solution that works everywhere regardless of compilers and the operating system and so on, it's very hard to have a good solution. And this is what, where customers have come back to me and said, give me a portable solution. I need something better than this or that. And so the best solution I have for this is sort of, you know, apply all of the above as best as possible. And my initial idea was I'll just write a little function that does this and put it on GitHub and people can use it. But if you look at lip sodium and then, yeah, you see, I mean, I'm not going to click on it now, but that's the link. If you download the slides later, you'll see it points to GitHub and it shows you the actual limitation. Lip sodium's MEM-0 is really well written and it's beautiful and it sort of has this fairly elegant, you know, if, you know, this, this and this or if deaf this, then do this solution. If, you know, this particular setup, then do this solution and it does, has this for six or seven of the cases I've covered. It's really nice. It's really elegant. If you're looking for inspiration, point people to lip sodium. I think it's a good portable-ish way of solving this problem. Right. Okay. So that now we've discovered, we've talked about the problem. We've talked some solutions. Okay. Well, I want detection. When does this really happen? I want to see this, right? And I want compilers to tell me this. Like why doesn't GCC tell me it's not writing something out? Like if it has security consequences, it should tell me. I want, why are they not doing this? I don't understand. So I set out and I modified GCC. I looked at a deathceral elimination and I came up with this patch. And instead of the, this is, this is, there are three SSA deathceral elimination paths and when it calls, delete dead call, I sort of take that out and say, if, you know, it's a built-in memset before you call, delete call, emit this warning, tell me to file and tell me to line number. And then do still optimize it out. And what this means is every time a memset gets optimized out, GCC now tells me. And this sort of, this is very interesting because I not only get, get, you know, detection for my own code, this is a great way to get, to get really cheap, fast zero day. And in fact, that's what I did. I downloaded a whole bunch of very well-known open source projects and I ran them through a modified version of GCC and I came up with a list of things. Oh, awesome. Thank you. So I know of this particular problem in, in like practically affecting open SSL, MIT KERP, Heimdall KERP, Matrix SSL, PHP, DHCP, bind, squid cache, and the list goes on. I have R sync as well and there's more. So we know this problem is very widespread, right? If the stuff we all rely on, the stuff that is built on has these problems, that means your code probably has this as well. And of course, I'm just giving names out here. But let's give you guys some zero day. That's MIT KERP. That mem sets optimized out. That's PHP. That mem sets optimized out. This I think is Matrix SSL. That decrypted plain text gets mem set it. That gets optimized out. That's the lingers around memory. This is open SSL. That crypto extended data, that doesn't get optimized out. This is NGINX. That password, that mem zero gets optimized out. This is a bind and DHCP. That mem set of private key data gets optimized out. This is quid that it goes to LDAP and gets creds. And basically it tries to clear the creds and then that gets optimized out. Yeah, well, I had to play around with Power of Wind a little bit. So and then same thing, this is a key that gets optimized out. And then this is R sync. And these are stored credentials in a file and that gets random memory and those mem sets get optimized out. So that's nine bugs right there. And all it took was five lines of code change in GCC. And GCC just gave me all these bugs. The other thing about seeing exactly what. Thank you. The other thing that was really nice about getting the data back from GCC isn't just that it gave me bugs. It also showed me things that optimized out that I thought it wouldn't. Obviously what I was expecting is a variable that's about to go out of scope and that you get mem set that would get optimized out, obviously. But what I also noticed was that, you know, when you there's a common code pattern when you just what you just malloc something or you declare something on a stack. And the first thing you do is mem set to clear the whole thing and then you move on. It turns out that in a number of cases that also gets optimized out. The idea is that if the compiler it only gets optimized out if the compiler can prove that every element in the struct or that the whole field gets filled in. I'm not sure if that's entirely true. And we were talking about this earlier. But what about, you know, things like structure padding or maybe enums or, you know, something like that or, you know, unions, I mean, that I'm not quite sure how that works. I suspect I haven't dug into it because this is I ran it I found this yet. I mean, I wrote this patch yesterday, right? These bugs are like they're fresh. So I don't know exactly how much potential there is here, but it smells like there's room for bugs here. So I was surprised to see this and some more research is needed. So if anybody wants to feel free. And then the other thing I sort of noticed is that obviously the common case when I was looking for in terms of bugs was something that was sensitive material. And then obviously I saw a whole bunch of things where non-sensitive material was being mem set and then freed. And then that mem set would get optimized out as well. And that struck me as odd at the beginning, but then obviously there's a common sort of coding pattern that I've seen where, you know, anytime somebody does a malloc and then before using doesn't mem set null or when they free right before they do a mem set zero, not because they want to clear sensitive material, but because they always they want to have a guarantee they always start from a clean slate. And so they end up building code that ends up working for something that always has a clean slate, right? And if that if that mem set gets optimized out, then those guarantees no longer hold. And so code that works around the sort of, well, we always have a clean slate when we you know, when we get a fresh piece of memory, that is no longer true. And so I think that coding pattern doesn't jive well with compiler optimization. Again, this is the sort of realization I made yesterday. I don't have all the facts on this yet, but it seemed interesting. And I think there's some room for research here. The other thing I noticed is that sort of close to the mem sets that were optimized out, I noticed like other bugs, and this is kind of like, you know, it made me think is like, well, bad code attracts other bad code. So one of them was null derapsing the other was the use after free, where, you know, instead of doing mem set free, the code was doing free mem sets, right? And that obviously is a use after free. So this is the case of the null derp where basically a malloc happens and then the mem set null happens and then there's a check to see if the variable that was allocated is null, right? Obviously that mem set, if the value is null, that mem set would cause a null derp. Except the mem set gets optimized out so it never generates the null derp. It kind of catches 22 there. But that construct is clearly broken. So now, I mostly spoke about mem set, but really there's a thousand variations that clear memory and sort of comes down to the same thing, right? Obviously you can do a for loop or you can use other APIs or you can like roll your own and do something very exotic. And then there's C++ and which there's a gazillion ways of doing it. You can have like weird classes with like a constructor and like, you know, inheritance, multiple objects, like virtual. It gets really crazy once C++ comes in the mix. And so basically it all kind of looks different, but it all does the same thing. It has the same root cause and the same problem. And the thing is when you look at this from just a code perspective is that sometimes the optimizer is smart enough to see it. Sometimes the optimizer is not, but it could in the future be smart enough to see it. So it's one of these things where if you're looking at a piece of code and you're doing some kind of security assessment and you see this and like, should you report to bug, should you not? I think you should because even if the compiler doesn't optimize it out today, it may very well optimize it out tomorrow. Right. Okay. So now oldest talk has been about C and when you write it in C, well, what if you're not writing C code? What if you're using other languages, non-native languages, you know, go Rust, Objective C, C sharp, Java, flavor of the month. And really I wanted to spend more time on this, but then my slides would have gotten so long I couldn't. I only have one slide on non-C, but I spent a little bit of time on this in C sharp. There's something called secure string, which is supposed to hold a string in a safe way. And the problem I have with secure string isn't the implementation. It's how do I get something securely into secure string and how do I get something securely out of it? And then in terms of Java, there's a Java crypto guide, which basically says it recommends not using strings to hold sensitive material, but to use a biderane instead. And there was some reason behind it. I don't remember. But the idea is basically this. Most managed languages don't really offer any decent way to clear memory or to hold sensitive material in memory without it leaking. And it will leak and it'll kind of happen behind your back because you wouldn't know it leaked, especially when you're dealing with garbage collection where something can get reallocated without you ever knowing. And all of a sudden, before you know it, there's like five different copies of your key sprayed all over memory. These languages, as far as I can tell, they don't have the infrastructure in place to deal with sensitive material. It seems to be entirely missing in a lot of places. In other places, it's kind of like a shoe horned on or bolted on with some varying degrees of success. I remember seeing there was some in place for go, but it had been in revision three or four because there was always something wrong with it. And so, again, I wish I had more time than I could elaborate on this. From what I saw is that it's a pretty sad state of affairs in Nazi that most people haven't tried and those who have have not tried hard enough. So now that I've run through all of the sort of sort of memset problems and how to clear up memory, I want to sort of talk about some related issues. First of all, what I said initially is that when people make this step and go, oh, well, I should clear this memory because it's sensitive, I said that's huge because it really is. Most code doesn't even try. There's an unbelievable amount of code that just keeps keys in memory and sensitive material and it goes out of scope and it never gets cleared and it just ends up lingering on the stack or the heap. I mean, often it would get overwritten fast, but sometimes it could linger on and sit there for a very, very long time. Problem with this is that it's hard to find in any kind of automated fashion because you're looking for the absence of something. So that means the only way you can really find these kinds of bugs is to manually look at it and go, oh, this is sensitive material. No effort was made to clear this. A second related issue, and this is really cute, actually. So when you call memset the way it's done there, that's wrong. The length and the byte you want to memset with are transposed. The zero should be the second argument and the length should be the third argument. So what that really does is a no op. It basically says memset and then use len as the pattern and then the pattern is used as the length. In this case, the pattern is zero. So a memset of zero, which becomes a no op. And what's really cool is that GCC actually had, you can tell GCC to have made a warning for this. And one of the things I wanted to do is I wanted to run through all the same code I had tested before and enabled the warning and then sort of show it on a list of bugs, but I kind of ran out of time. But I strongly suspect if you use this warning and you go download a whole bunch of like well known open source code and you run it through, you'll end up with a very similar list of bugs. So another list of sort of related bugs, and related in the sense that no longer clearing secrets but related in the sense that optimization was involved. That is to say, if the optimizer wasn't turned on, a security bug wouldn't have occurred, right, or would have been less severe, right. As there are sort of three cases of bugs I sort of ran into that I think is somewhat relevant that I sort of want to talk about. So the first one is what's called pointer overflow. It turns out, let's say you have this code, pdr and then there's a lend and the lend is untrusted, right. The idea is that you want to validate that pdr plus lend isn't beyond the end of your buffer. Before you do that, you also want to make sure that pdr plus lend doesn't overflow, right. And so you would do code like pdr plus lend is smaller than pdr. If that's the case, then the pointer overflow and you bail out, right. Problem is, according to the C standard, pointer overflow can't happen and so that is undefined behavior. And the optimizer sees that and goes oh, I'm defined behavior optimized out, gone. So you bound check just got optimized out. That is a relatively common bug to see. If you don't know it's undefined behavior, if you don't know the compiler can optimize it out, you would just read over it. But once you know, like you start reading code and you'll see it everywhere. Also the way to fix this is basically to cast your pointer to an integer type that's big enough to hold a pointer and all of a sudden the optimizer can no longer optimize it out and it'll be in your code. So that's the first one. The second one is a lot more subtle. This has to do with switch case optimization. So when you have a switch in C and you translate it to assembly, you could just do like a one one sort of translation. But if you have, if you use the optimizer, one of two things will happen. Either it'll generate a binary tree that's generally observed in the Microsoft compiler. If you look at GCC and Clang, what they do is they'll create a jump table. And what that means is they'll look at the value to compare and then if they compare a certain number, they get the value again and they use that value as an offset in a jump table. And that usually isn't the problem and that's an abstraction and most people don't care about it in most situations except if you're dealing with a shared memory trust boundary because all of a sudden there is a subtle double fetch that was emitted by the compiler behind your back that doesn't show up in the actual C code. And these are situations like a hypervisor trust boundary. These are very, very strong trust boundaries. And that thing is actually a link if you look and once I publish the slides, if you click on it, it links to a blog post that shows just that. It's a virtual box guest host pre-resolation because of a switch case jump table optimization. And so the bug there basically is that when you fetch the first time you do the compare, it's fine. And then two instructions later you fetch again to do the jump table. Between the first and second fetch, the guy on the other end of the shared memory could change it. And all of a sudden you can jump outside of your jump table and basically cause an arbitrary jump which obviously is bad. Yeah, I'm not going to cover this. Okay. I got to wrap it up. I got three more slides and then we can get to questions. This is actually very important. So now after everything I've covered, we're good. We know what the problem is. We know what the solutions are. We know there's real world problems. But we have a good grasp of it now, right? Okay. It turns out I kind of lied to you. This is not the whole problem. If it was the whole problem, that'd be great. We know how to fix that more or less. It turns out that compile optimization is really, really clever and it does many, many clever things and they're all very subtle and a lot of them are architecture specific. And here's a scenario of things that can occur, right? You'll have the optimizer will do things like, oh, well, you're handling this string of a certain kind. You know what we'll do? I'll just shove it into a bunch of registers so it'll be faster. And then, you know, it passes something and then all of a sudden the optimizer goes, oh, you know what? You don't have enough registers. It's okay. What I'll do is I'll take whatever's in the registers and I'll dump it on the stack and then we'll go from there. All of a sudden what happened is you leak key material in registers and then you leak them on the stack. And this stuff happens throughout. And secrets leak out. It just happens. And it is because of optimization. Even if you try really, really hard by trying to do the really hard to do the right thing and the optimizer, it's just, they just screw you, right? And this problem is echoed in the blog post by Colin Percival who used to be the security guy for 3BSD, now has, I think, a cloud storage company, very, very smart security guy. I would recommend reading that blog post. This problem is also echoed in the Linux map page, Exocasive B0. And the map page basically says, well, yes, this is a fundamental problem. We still recommend using this as B0 and our hope, our thought process is that towards the future, we will have a way to get the compilers to not do this and we can move on. But at the present time, there's no good fix for this. And so this is the first statement I have that I want to make is that at the present time, optimizing compilers and cryptography are mutually exclusive. You can have one or the other. You cannot have both. It does not work at the present time. Okay. Before I get to my conclusion, I want to rant a bit about optimization as if I haven't already. But basically, I mean, I get that optimization is great and it gives you all these things and things get faster. But I have a real problem with optimization because, look, if you're a developer and you write code and you're pretty smart, you can reason about your code because you wrote it and you know what it does. But if you then compile it and it's gone through an optimization pass, you can no longer reason about it because you don't know what the optimizer did. And that is, I think, a fundamental problem. And sort of what I want people to sort of think about is whenever they do dash O is the, and I'm not saying you shouldn't use the optimizer because it has many pros, but don't be like blasé about it. Before you type dash O, really think what it means to do that because it will introduce all sorts of things that you weren't sure about or that suddenly changes the meaning of something. And what I really want for the compiler people and maybe the language people to implement this so the compiler people can do it, is I want strong accountability and control of the optimizer. That is, if I compile something, I want to be able to compile, I want to be able to go to the compiler and say, hey, before you do anything, I want you to give me a detailed list of everything you're about to do in terms of optimization so that I can take that list and look at my code and then with that list in my code, I can now reason again about what the binary is going to do. Without having that kind of accountability, you can't reason about your binary. And the other thing is control, right? I want to have fine grain control over optimization. What I mean by that is, you know, like the localized stuff that I mentioned before that the Microsoft compiler, for example, has, where I want to be able to go and say, this particular scope don't optimize or this particular scope don't do this particular optimization. I would like to see something like that. I'll just skip this. So okay, here's my conclusion. Right. We know what the problem is, the original problem. And then I have some solutions and then in retrospect, they're just partial solutions but they're still kind of solutions. Okay, and then I also have a call to action of things I think should happen and hopefully will happen at some point in the future. The problem, as I've illustrated, I think is rampant. And so basically, I would like people to use that GC patch that I've shown or create a better one and go find some bugs or better yet, go fix some bugs. In terms of compilers, as I just mentioned, what I want is optimization, accountability, and control. It's kind of the wild, wild West in terms of optimization where the compilers just go and do all these things. And I mean, we have some flags but there's not enough control and there's not enough accountability. There's not enough transparency. You just don't know what's going on. And some of the problems have like a dumb functionality where it's like a needle in a haystack. You want something that's easy to work with, easy to read or easy to parse and that tells you exactly all the optimization steps you're doing. And ideally, I'd like the language people to get involved and the standardize on this because if they do, we can now demand this of all the compilers. And then lastly, sort of the, you know, coming back to my Nazi, sort of the, what about the Nazi Ruby Python Perl, Go, Rust, and so on. It smells bad. It looks bad, especially when there are runtimes involved. This is probably worthy of a presentation of its own or multiple presentations. But I wish I had done more there. That is essentially it. I hope you enjoyed it.
 Mem said that's the title of the next talk.  Why cleaning memory is hard?  Ilya Fansprudl is a security researcher who loves to find out new things.  And he found out that it's quite hard to get rid of sensitive content in the memory.  So today he's going to give us an overview, presentation.  Please give a warm round of applause to Ilya Fansprudl.  OK, perfect.  Yeah, so as the hair will just explain, my presentation is called MENSAD.  Why Clearing Memory is Hard.  Before I dive in that, once upon a time that was me.  A lot more hair, a lot less fat.  This is my 17th Congress...  You have pot kettle here, buddy.  This is my 17th Congress in a row.  I've spoken here a number of times before.  I haven't added 35 in there yet, but obviously that should mean there as well.  I work for a company called IOActive.  I'm the director of penetration testing.  But that really just means that I lead teams of pentesters.  No double entendre there.  Obviously, we're always looking for good security people, so if you're interested, come talk  to me afterwards.  I like looking at low level stuff, kernels, drivers, type of reasons, that type of stuff.  And I enjoy reading code.  Okay, enough about me.  What's the kind of audience that I think would enjoy this?  Pretty all around security people, crypto people, if you like code review, if you like  compiler stuff, and if you're just generally curious about technology, I think you might  enjoy this.  In terms of the knowledge required, the first half of these slides is relatively basic.  And so if you have a basic technology understanding, you should be able to understand most of the  first half.  If you have some C background, that would be nice.  And then as I move forward past the first half, things become a bit more advanced.  But if you just only grasp the first half, I think that will still be useful.  Right.  So what does this talk about?  Actually it's one very simple, easy to explain crypto implementation problem.  And the reason I've dedicated an entire talk about it is because while the problem is easy,  the solution is not.  There's a lot of moving parts, a lot of subtlety, a lot of nuance.  And I'll get into that in a little bit.  Now I can hear some of you thinking, well, Ilya, WTF, 2018, why the hell are you talking  about this?  This is very, very, very well known.  To them, I say, well, because this stuff is still everywhere.  And I will show that in the slides.  I will show this with data.  I will show this with bugs.  But the driver of this talk, the reason why I started making this presentation is that  this year alone, I did engagements for three different customers on three entirely different  software projects where they all had this exact same type of bug.  And you tell them about the bug, and the customer goes back and says, OK, well, yeah, that's  great.  We understand.  Now tell us how to fix this in a portable way.  And that is not very easy.  The other thing is that even though this problem is known conceptually, people kind of blasé  about it, practically not many people understand how out pervasive this problem is, how realistic  it is.  It isn't just like, oh, well, the compiler might do this.  No, the compiler will do this.  And it does it everywhere.  And these bugs do show up everywhere.  Even though, because it doesn't, it's hard to tell from the code that it is there.  If you look at the binary, if you look at the compiler in bits, you see that it's there.  And then the third is, given that one of the teams of the Congress this year is Foundations  and to talk about things that aren't necessarily new, but to sort of try and help bring the  subject to the next generation, I think this fits in perfectly in the concept of Foundations.  Right.  So, before I dive in, there's a couple of people, we're actually a long list of people  that sort of helped me out.  As I said, the problem is well known.  It's been well known for at least 20 or 30 years.  And so many, many, many people have published papers and presentations about this.  And I don't know all of them personally.  And I wish I could include them all.  But the people I've included in here are sort of the one or two away from me that have  had some kind of impact in these slides.  Some of you are sitting in this audience.  And your help has been appreciated.  Okay.  So, let's actually start.  Now let's say you're going to write some piece of code, right?  And it's going to be doing something and it's going to be handling sensitive data.  Be that keys or decrypted plain text or session tokens or passwords or password hashes or anything  that could be considered sensitive, right?  Now, if you're a smart security conscious person, right, the moment you are done with  that sensitive data, you want to dispose of it, right?  You want to purge it from memory, right?  Now why do you want to do this?  Well, because otherwise, if there's some kind of infolique that is discovered later on,  then whatever secrets or lingering memory could be used in your infolique and all of a sudden,  you know, your tokens or your keys are leaked out, right?  And I mean, that may sound like a stretch, but, you know, things like heart bleed, you  know, happen, right?  So this is very practical.  This can really happen.  And for a move on, I want to say, if you think and make the step where you say, okay, I need  to dispose of sensitive material once I'm done with it, that's really big, right?  Your software that deals with, you know, sensitive material does not do this.  So if you make the step thinking I need to purge this, you're ahead of the curve, right?  So now, concretely, it would look something like this, right?  You would write a little, this is a sample code I have that basically generates a little  key and it's a function you give it and you give it a key pointer and this thing declares  a local variable that search invites, goes and reads a bunch of random bits, puts it  in K, copies K to key and then before it returns, because K is about to go out of scope and  it contains sensitive key material, you go and say memset and then you clear the thing  and then you return, right?  Perfect.  You run this, you compile it, you add a main and it does exactly what it's supposed to  do.  You just use the assembly and it's all perfect.  Problem there is what you're doing is that's not release based code, right?  When you sort of make code ready to be released, you sort of tell the compiler that it should  enable the optimizer, right?  You'll give it OS or O2.  Those are the most common ones.  You just see O3 when people want to live on the edge but usually O2 or OS.  Now if you look at the assembly again, you get a whole different picture of what's going  on and I want to illustrate this and there's a website called Compiler Explorer which is  beautiful, integrates a whole bunch of compilers and it has on the left it shows you the C  code and the right shows you the assembly and it's like color based and it's easy to  make connections.  So let's take our little example and on the left we see the generate key and on the right  we see the compiler and sure enough if you follow the colors and left and right you can  see what C code translates to which assembly, right?  If they make it a little bit easier, that memset clearly gets translated to assembly,  right?  Now that is when you do minus O0 which is the default which is what you would do if you're  developing code and you want to debug this stuff, right?  Now once you're done developing and you're about to ship this thing, you do minus O1  for example and the optimizer kicks in and all of a sudden your assembly looks a whole  lot different.  You'll notice it's shorter.  You'll notice that all of a sudden the color of your memset changed whereas in O0 it was  this sort of red-ish and all of a sudden it became white and it's nowhere to be found  in your assembly, right?  It just does not show up, right?  That's a problem, okay?  Well, what happened, right?  Let's, yeah, I stole that.  So what happened is a thing called dead store optimization or dead store elimination.  And so basically that memset at the end that what you're doing is you are writing into  a buffer that is never, ever going to get used again.  And an optimizer compiler looks at that and says, hey, you know, I can just take that  memset out and I just saved you a couple of cycles and you have a smaller binary, huge  win.  And because it doesn't really change what the program does, it's fully compliant with  all the relevant language standards.  Right, so that's essentially, that is in a nutshell our problem, right?  And so one of the things I want to do is I want to look at all common compilers and see  for which of these I can get it to effectively, practically optimize out a memset like this.  And I had to, with some of them it was easy, some of it was harder.  I had to fill around with it for some straight up memset works for others.  I had to like kind of twiddle and make a for loop or you know, kind of jump around a bit.  But essentially that these are lists of 10 compilers I tested.  I try to get my hands on the IBM compiler, but I don't have $20,000, so I couldn't do  that.  But these are the ones that I did test.  And then so the first five or the first four you will know, you know, the GCC and Clang  and Intel compiler and the Microsoft compiler.  And they're all also on the compiler explorer, so it was easy to test those.  And it was very easy to get those to optimize out memsets.  And then I moved on, downloaded a bunch of others.  You know, the Sun Studio compiler and the, and Barcadero's SQL Splash Builder and the  ARM compiler and a bunch of others.  And out of these 10, I was able to get eight to optimize it out, right?  80% of the most common compilers do this in any, in a practical sense.  So it isn't just like a theoretical thing.  This really happens.  A funny note, I tried really hard to get the PGI compiler to do it.  In fact, it has a switch called that's for elimination.  And of course I played with it and I tried it and I spend over an hour trying to get  it to optimize out my memset.  God damn thing wouldn't move.  I don't know what it, I don't know what it's doing there.  I couldn't get it to do anything.  But basically most compilers, if you ask them to do optimization, will gladly optimize out  a lot of memsets.  Right, so the next question is how common is it to actually see projects use optimization,  right?  And this sort of stems from a conversation I had earlier this year with a couple of  colleagues where a bunch of people said, well, you know, I don't see 02 or 01 or OS all that  often.  I don't think optimization is all that common.  And so I started looking around and I said, okay, well, where can I get some data?  And the first thing I saw was, okay, well, I can go to opsewers.apple.com and that lists  about 200 projects and I'll just go through all their make files and look for 02 or OS  and so on.  And there's about 100 out of there.  And then I realized that they actually don't use make files.  They have a really bizarre build system and that build system, by default, uses OS.  So even though it says 100 out of 200, it's probably closer to 200 out of 200.  And then I had a whole list of programs I wanted to test in like FreeBSD and you've  put into it a bunch of Linux distros.  But that's pretty boring and I ran out of time.  So I kind of stopped there.  But these numbers should be good enough.  In addition, if you look at the common IDEs, in particular Visual Studio and Xcode, when  you tell them to build a project in release mode, Visual Studio by default does 02.  Xcode by default does OS.  So the fact that these tools by default give you optimization should make you confident  enough in knowing that, yes, in fact, optimization is incredibly common in release builds.  It isn't everywhere, but it is almost everywhere.  Right.  So now that we know the problem and now that we know it isn't just theoretical and that  we know it's practical and that, in fact, it does occur very, very often and with most  compilers, in fact, basically it's a real problem.  How do we fix this?  And this is sort of where things get difficult.  There are many sort of solutions.  Nothing is portable.  Right.  It's sort of the, OK, well, this solution works if you use this compiler and this solution  works if you use this Lipsy and this solution works if you use this OS and this solution  works with this version of the language spec and this solution works if you have this particular  executable file format.  Right.  So and before I dive into any of those, let's first talk about the elf in the room.  Don't just roll your own.  I've seen people do this where they go like, oh, well, you know, I'll just, I'll fight  with the compiler.  I know what I'm doing and they'll just, you know, they'll kind of leader where Jenkins  this and they'll totally screw it up and they'll come up with, you know, some really stupid  ID.  One of the ones I heard was like, oh, well, I'll just, you know, I'll just do IO with  the buffer and then it's cool.  Yeah, you could do that, but then you're doing IO, right?  For no reason.  So don't just roll your own.  You're going to come up with a solution that's probably stupid.  You're going to look really stupid.  And it'll be one of these things where, okay, you're sort of, your bad solution might work  for this particular version of the compiler, but if you don't understand the concepts behind  it, then chances are the next version of the compiler that is somewhat slightly smarter  will sort of just bypass whatever you implement it.  So don't roll your own or if you want to roll your own, at least listen to the advice  of the next 10 and then base your solution on at least some of the advice that I'll be  giving out in the next couple of slides.  So with that, let's move on to actual solutions.  The first one is a Lipsy function called explicit B0.  This is not part of any standard, as far as I can tell, at the present time.  But this was sort of concocted in May 2014 by the OpenBSD guys.  If you note the date, it's pretty close to what hardly happened.  It's a few months later.  I think that may have some relation.  Anyway, this function basically does a B0, but explicitly guarantees that it does not  get optimized out.  Now what that means is that what it does or it doesn't, it's no longer your problem,  it's the Lipsy's problem because they've made the guarantee, so now it's on them.  So that's really nice.  OpenBSD did first and then NetBSD said, that's a great idea and we're going to steal it.  But we're going to rename it though.  So they changed the name to explicit memset, but it's essentially the same thing.  And then about two years ago, FreeBSD came up sort of the same thing.  And then almost two years ago, the GLipsy guys came up with this too.  And then DietLipsy supports this too.  OSX, however, does not support it.  So if you're limited to those platforms, explicit B0 is a perfect solution.  Similarly, if you are developing for the Windows world, there is an API called Secure Zero  Memory which basically is Microsoft saying, we guarantee that this thing doesn't get optimized  out.  So if you want to securely clear sensitive material, just use this API.  MSN says it will ensure that your data will be overwritten promptly.  And this is one of the cases where Microsoft was ahead of the curve by like 15 years.  They've had this thing since the early 2000s.  It was in XP and it was in Windows 2003.  Both operating systems are no longer supported, but the API is.  OK, so now there's another function called memset underscore s.  And it guarantees it doesn't get optimized out.  And it's guaranteed by spec.  It's guaranteed by the language spec.  It is standardized.  It is in C11.  It is wonderful.  It is great.  Except it's not great.  Because even though it's in the standard and it's there, it's in what's called the  optional NXK.  And if you read the spec, and it's like pages and pages of boring crap.  But if you end up reading the K2, it says optional extension.  What does optional extension mean?  It means you can be entirely C11 compliant and not offer memset s.  So it's kind of this as Reverend Lafayette would say, yes, but if, no, it abut.  So if it's there, great.  If it's not, it has the potential of being this great portable solution and then it isn't.  Right.  And then, of course, the sort of obvious choice, but somehow a lot of people seem to miss this  is if you end up doing something with sensitive material, chances are you're using a crypto  library.  And if you're using crypto library, chances are the crypto library offers you an API to  do secure memory cleaning.  And so I listed the common ones.  If you're using OpenSL, there's OpenSL cleanse and OpenSL guarantees they don't get optimized  out.  If you use Canutee less, there's Canutee less memset.  Same thing, Canutee less guarantees it does not get optimized out.  If you're using LeapSodium, which is one of the newer ones, they have sodium memzero.  Same thing, they guarantee it doesn't get optimized out.  And I'll get down to a minute.  So this is basically up until here, I've sort of given you a list of, okay, well, here are  specific API functions you can call.  If you're using this library or using this OS, use this.  The next sort of solutions are sort of the, okay, well, what if you can't rely on the  APIs?  Maybe we can get something out of the compiler.  The first solution is, and this isn't portable, but most compilers have this or something  like it where you can go to the compiler and say, hey, don't use the built-in memset, just  use the one from libc.  And what that means is you tell the compiler that it shouldn't assume that it knows what  memset does.  And if you do that, then sure enough, memset won't get optimized out.  GCC, this is originally GCC specific.  And then the Intel compiler supports it too.  And then Clang supports it too, but up until, and this is true, up until Clang 3.7, which  is maybe two years old, it's not that old.  Clang basically supported FNobilt in memset.  And then what they did is they kind of dropped it on the floor and it got optimized out anyway.  So it's kind of annoying.  It kind of ruins the whole use this because it works, except if you're using an older  version of Clang.  And also, it's not overly portable, but it's a solution that works.  Other things might still get optimized out.  If you have some kind of for loop that clears memory, that might still get optimized out.  But at least if you use memset and you're using no built-in memset, then you have a  pretty strong guarantee that it shouldn't get optimized out.  So another sort of solution is just don't use optimization.  That works.  You're guaranteed not to get optimized if you don't use the optimizer.  Obviously that isn't perfect.  For one, Fortify source doesn't work if you don't use optimization.  So if you want to use Fortify source, you have to use optimization.  The other one, of course, is that you've got to change your build environment.  I guess it's not portable, but then again, most compiles will have some way to tell it  to not optimize anything.  But obviously the reason you don't want to use this particular solution is because you  don't get the optimizer.  So your product will probably be slower.  Sort of a spin-off of this is some compiles, in particular the Microsoft one.  And then GCC also kind of supports it, is where you can localize optimizations based  on scopes and functions.  And so you can say, oh, for this function, do o0.  It's not a commonly used feature.  It seems like it might have some side effects and it doesn't support all switches that the  compiler generally does.  I've seen this recommended by a few people.  I played around with it.  It seems to work.  But it doesn't seem to be a sort of commonly adopt way of doing things.  The other thing, of course, is again, this is, I mean, these are pragmas.  This is very, very compiler-specific stuff.  Another solution is using what's called weak symbols.  Anybody familiar with weak symbols?  Oh, yeah, a little bit.  I'll try to sort of very briefly get into this.  So the L file format basically is a format that specifies how to have an executable that  can run on OSes that support this file format.  So, Ls, if it's compiled, for example, for Linux, it'll be in this particular format.  And one of the, obviously, as part of format, you can store symbols for things like functions  and variables and so on.  And generally, a symbol is what's called a strong symbol.  You can mark one as weak.  And what weak means is that a symbol may change in runtime.  And what that means is if you declare a function or a symbol of a function as weak, that means  that compile time, the compiler would have a very hard time to reason about what that  thing does because of the sheer fact that you've declared it as weak.  And in fact, this particular solution is what it means to use this in their implementation  of explicit B0.  And what I really like about this is that this is the commit message for the OpenBZ  guys.  And they're very pragmatic about it.  They say, well, you know, we think our solution is pretty clever, but it's not foolproof.  There are still ways to defeat this.  And they list, you know, a bunch of ways to do this.  In particular, you know, well, you know, the compiler could emit runtime code that checks  what this thing is in runtime right before it's called.  And then you could still optimize it out if the thing matches or doesn't match.  But then they go on and say, well, in the foreseeable future, we don't think that's  going to happen.  But it's possible that at some point down the road this might happen.  And so I like the way of reasoning about this, where solution is pretty clever, but it's  not foolproof.  It may at some point in the future break.  But at the present time, it's a fairly good solution, I think.  Right.  So another solution is to use memory barriers.  How many people know what a memory barrier is or what it does?  Okay, about the same number of hands.  Okay.  Let me try to very briefly sort of explain what a memory barrier is.  And bear with me here because it's, and I'm going to oversimplify it because it's not  a particularly simple concept if you've never heard of it.  But okay, let's say you have a piece of code, two global variables, A and B, and you assign  a value.  You say A equals something, and B equals something, right?  And there's no relation between A and B. Right?  What that means is both the compiler and the hardware, because they have no relation.  Both of them are allowed to reorder it.  So B can be assigned first, and then A can be assigned later.  Because there's no correlation, that's perfectly valid.  Now let's say you have a second thread somewhere, and your second thread says, okay, you know,  while not B spin, and then once B is set, you use A. Right?  This is sort of where you're basically waiting on something to be set, and the idea is that  you wrote your code to that B set after A is set.  And that seems logical, and that would work, except if the compiler and hardware don't  know, there's a relation between your loop on one end and the assignment on the other,  and either the hardware or the compiler reorders it and sets B before it sets A. Really, really  nasty things happen.  This has been the source of numerous security bugs, very, very subtle stuff.  The K, SAN, and T, SAN kind of stuff.  We've seen Linux kernel last couple of years.  A bunch of that is related to these kind of bugs.  And so the way that you fix this is you introduce what's called a memory barrier, and that is  basically when you write your code and you say A equals something, and then before you  say B equals something, you basically, and it will say memory barrier, and then you say  B equals something.  And what that means is it gives a signal to the hardware and to the compiler, and it says,  whatever happens before this and after this, you are not allowed to reorder this.  There is a correlation there that I know that you're not aware of, so don't reorder it.  And I hope I explained it well.  This usually takes a lot longer to explain, but I hope I got the message across well enough  to sort of give you an idea of what a memory barrier is.  And now the cool thing about memory barriers is that it's a way for a programmer to tell  the hardware or the compiler, I know something about this memory, you don't.  Stay away, don't touch.  And because of, I mean, it works for reordering, but it also works really well to not get something  optimized out, right?  The idea is you could basically just do your memset, and then on the thing you memset it,  you basically do a memory barrier, and that tells the compiler not to optimize it out.  And I know the concept sounds complicated, but it's pretty clever, and I've oversimplified  this because it's a relatively complicated subject.  But this works really well, and this is used by Dytlipsy, and it's used by Glypsy, and  Nginx recently had a fix where they have their own explicit memzero, and it also uses a memory  barrier.  So this is a tried and tested concept, and it works.  So those are kind of the solutions that are known and that work, and have been tried and  tested by various fairly well-known pieces of software.  If none of those things, if somehow you're in an environment somewhere, and none of this  is available to you, or it's not portable enough, and you're looking for a solution  that works everywhere, the best you can do is fall back on constructs that are known  in the C language, and this is basically the use of the volatile keyword.  I call this a fallback.  People often go, well, just use volatile, and then that solves the problem.  It turns out optimizers can be very clever and very tricky, and even when you use volatile,  there are cases that can be made where if the optimizer is clever enough, your data may  still get optimized out.  So the volatile solutions are sort of best effort fallback solutions, and there's sort  of two variants of this.  One is a volatile pointer, right?  That's the fallback solution in the Lipsodium, and the other one is a volatile memset function  pointer, which is what OpenSL uses.  And here's what that looks like.  This is the Lipsodium fallback, and this looks like, you know, volatile is sort of the,  you know, you tell the compiler, hey, you know, I know something about this, you don't.  And this is where it gets very language-loyalry.  If you look at the spec, it says something along the lines of the access object, something,  something, and they're describing the actual memory volatile, not just the pointer L value.  And so if the compiler looks at this code, and it can trace, and it can prove that wherever  PNT came from, and if that isn't actually volatile, then this volatile doesn't really  mean all that much, and it can still optimize it out.  That sounds very theoretical.  I don't know if that actually happens, but a number of people smarter than me or that  know more about the sort of, this negrity little C language things have told me that,  yes, in fact, you would be allowed to do that if you're a very smart, optimized, and  compiler.  The fact that it's a fallback solution for Sogium and a few others leads me to believe  that it probably doesn't, but it could.  Right.  And so this is the solution that OpenSL uses, which also uses volatile.  It doesn't do a pointer right, but instead it creates a volatile function pointer that  points to memset.  And it sort of gives you, you get the same concept more or less as with the weak symbols.  The idea being is that your volatile function pointer can change at any time without the  compiler knowing about it.  And that seems like a pretty good solution, except when you, one way of, in theory, getting  around this is if the compiler emits runtime code that right before the function pointer  gets called, it looks and goes, well, like it captures it and then goes, is it memset  or is it something else?  If it's something else, then we call it.  If it's memset, we just return and then you optimize out the runtime and save a few cycles.  In theory, the compiler is allowed to do that and emit code like that.  And I don't know if that actually happens anywhere, but it's a possibility.  So think of these last two solutions as fallback.  They may not work in theory.  In reality, they probably do.  Right.  So this is sort of the first half of my presentation and I'm perfectly on time, which is great.  So now there isn't one portable solution and this is why, this is why clear memory is hard.  The problem is well understood, but if you're looking for like an all-around solution that  works everywhere regardless of compilers and the operating system and so on, it's very  hard to have a good solution.  And this is what, where customers have come back to me and said, give me a portable solution.  I need something better than this or that.  And so the best solution I have for this is sort of, you know, apply all of the above  as best as possible.  And my initial idea was I'll just write a little function that does this and put it  on GitHub and people can use it.  But if you look at lip sodium and then, yeah, you see, I mean, I'm not going to click on  it now, but that's the link.  If you download the slides later, you'll see it points to GitHub and it shows you the actual  limitation.  Lip sodium's MEM-0 is really well written and it's beautiful and it sort of has this  fairly elegant, you know, if, you know, this, this and this or if deaf this, then do this  solution.  If, you know, this particular setup, then do this solution and it does, has this for  six or seven of the cases I've covered.  It's really nice.  It's really elegant.  If you're looking for inspiration, point people to lip sodium.  I think it's a good portable-ish way of solving this problem.  Right.  Okay.  So that now we've discovered, we've talked about the problem.  We've talked some solutions.  Okay.  Well, I want detection.  When does this really happen?  I want to see this, right?  And I want compilers to tell me this.  Like why doesn't GCC tell me it's not writing something out?  Like if it has security consequences, it should tell me.  I want, why are they not doing this?  I don't understand.  So I set out and I modified GCC.  I looked at a deathceral elimination and I came up with this patch.  And instead of the, this is, this is, there are three SSA deathceral elimination paths  and when it calls, delete dead call, I sort of take that out and say, if, you know, it's  a built-in memset before you call, delete call, emit this warning, tell me to file and  tell me to line number.  And then do still optimize it out.  And what this means is every time a memset gets optimized out, GCC now tells me.  And this sort of, this is very interesting because I not only get, get, you know, detection  for my own code, this is a great way to get, to get really cheap, fast zero day.  And in fact, that's what I did.  I downloaded a whole bunch of very well-known open source projects and I ran them through  a modified version of GCC and I came up with a list of things.  Oh, awesome.  Thank you.  So I know of this particular problem in, in like practically affecting open SSL, MIT  KERP, Heimdall KERP, Matrix SSL, PHP, DHCP, bind, squid cache, and the list goes on.  I have R sync as well and there's more.  So we know this problem is very widespread, right?  If the stuff we all rely on, the stuff that is built on has these problems, that means  your code probably has this as well.  And of course, I'm just giving names out here.  But let's give you guys some zero day.  That's MIT KERP.  That mem sets optimized out.  That's PHP.  That mem sets optimized out.  This I think is Matrix SSL.  That decrypted plain text gets mem set it.  That gets optimized out.  That's the lingers around memory.  This is open SSL.  That crypto extended data, that doesn't get optimized out.  This is NGINX.  That password, that mem zero gets optimized out.  This is a bind and DHCP.  That mem set of private key data gets optimized out.  This is quid that it goes to LDAP and gets creds.  And basically it tries to clear the creds and then that gets optimized out.  Yeah, well, I had to play around with Power of Wind a little bit.  So and then same thing, this is a key that gets optimized out.  And then this is R sync.  And these are stored credentials in a file and that gets random memory and those mem  sets get optimized out.  So that's nine bugs right there.  And all it took was five lines of code change in GCC.  And GCC just gave me all these bugs.  The other thing about seeing exactly what.  Thank you.  The other thing that was really nice about getting the data back from GCC isn't just  that it gave me bugs.  It also showed me things that optimized out that I thought it wouldn't.  Obviously what I was expecting is a variable that's about to go out of scope and that you  get mem set that would get optimized out, obviously.  But what I also noticed was that, you know, when you there's a common code pattern when  you just what you just malloc something or you declare something on a stack.  And the first thing you do is mem set to clear the whole thing and then you move on.  It turns out that in a number of cases that also gets optimized out.  The idea is that if the compiler it only gets optimized out if the compiler can prove that  every element in the struct or that the whole field gets filled in.  I'm not sure if that's entirely true.  And we were talking about this earlier.  But what about, you know, things like structure padding or maybe enums or, you know, something  like that or, you know, unions, I mean, that I'm not quite sure how that works.  I suspect I haven't dug into it because this is I ran it I found this yet.  I mean, I wrote this patch yesterday, right?  These bugs are like they're fresh.  So I don't know exactly how much potential there is here, but it smells like there's  room for bugs here.  So I was surprised to see this and some more research is needed.  So if anybody wants to feel free.  And then the other thing I sort of noticed is that obviously the common case when I was  looking for in terms of bugs was something that was sensitive material.  And then obviously I saw a whole bunch of things where non-sensitive material was being  mem set and then freed.  And then that mem set would get optimized out as well.  And that struck me as odd at the beginning, but then obviously there's a common sort of  coding pattern that I've seen where, you know, anytime somebody does a malloc and then before  using doesn't mem set null or when they free right before they do a mem set zero, not because  they want to clear sensitive material, but because they always they want to have a guarantee  they always start from a clean slate.  And so they end up building code that ends up working for something that always has a  clean slate, right?  And if that if that mem set gets optimized out, then those guarantees no longer hold.  And so code that works around the sort of, well, we always have a clean slate when we  you know, when we get a fresh piece of memory, that is no longer true.  And so I think that coding pattern doesn't jive well with compiler optimization.  Again, this is the sort of realization I made yesterday.  I don't have all the facts on this yet, but it seemed interesting.  And I think there's some room for research here.  The other thing I noticed is that sort of close to the mem sets that were optimized out, I  noticed like other bugs, and this is kind of like, you know, it made me think is like,  well, bad code attracts other bad code.  So one of them was null derapsing the other was the use after free, where, you know, instead  of doing mem set free, the code was doing free mem sets, right?  And that obviously is a use after free.  So this is the case of the null derp where basically a malloc happens and then the mem  set null happens and then there's a check to see if the variable that was allocated  is null, right?  Obviously that mem set, if the value is null, that mem set would cause a null derp.  Except the mem set gets optimized out so it never generates the null derp.  It kind of catches 22 there.  But that construct is clearly broken.  So now, I mostly spoke about mem set, but really there's a thousand variations that  clear memory and sort of comes down to the same thing, right?  Obviously you can do a for loop or you can use other APIs or you can like roll your own  and do something very exotic.  And then there's C++ and which there's a gazillion ways of doing it.  You can have like weird classes with like a constructor and like, you know, inheritance,  multiple objects, like virtual.  It gets really crazy once C++ comes in the mix.  And so basically it all kind of looks different, but it all does the same thing.  It has the same root cause and the same problem.  And the thing is when you look at this from just a code perspective is that sometimes  the optimizer is smart enough to see it.  Sometimes the optimizer is not, but it could in the future be smart enough to see it.  So it's one of these things where if you're looking at a piece of code and you're doing  some kind of security assessment and you see this and like, should you report to bug, should  you not?  I think you should because even if the compiler doesn't optimize it out today, it may very  well optimize it out tomorrow.  Right.  Okay.  So now oldest talk has been about C and when you write it in C, well, what if you're not  writing C code?  What if you're using other languages, non-native languages, you know, go Rust, Objective C,  C sharp, Java, flavor of the month.  And really I wanted to spend more time on this, but then my slides would have gotten  so long I couldn't.  I only have one slide on non-C, but I spent a little bit of time on this in C sharp.  There's something called secure string, which is supposed to hold a string in a safe way.  And the problem I have with secure string isn't the implementation.  It's how do I get something securely into secure string and how do I get something securely  out of it?  And then in terms of Java, there's a Java crypto guide, which basically says it recommends  not using strings to hold sensitive material, but to use a biderane instead.  And there was some reason behind it.  I don't remember.  But the idea is basically this.  Most managed languages don't really offer any decent way to clear memory or to hold  sensitive material in memory without it leaking.  And it will leak and it'll kind of happen behind your back because you wouldn't know  it leaked, especially when you're dealing with garbage collection where something can  get reallocated without you ever knowing.  And all of a sudden, before you know it, there's like five different copies of your key sprayed  all over memory.  These languages, as far as I can tell, they don't have the infrastructure in place to  deal with sensitive material.  It seems to be entirely missing in a lot of places.  In other places, it's kind of like a shoe horned on or bolted on with some varying degrees  of success.  I remember seeing there was some in place for go, but it had been in revision three or  four because there was always something wrong with it.  And so, again, I wish I had more time than I could elaborate on this.  From what I saw is that it's a pretty sad state of affairs in Nazi that most people  haven't tried and those who have have not tried hard enough.  So now that I've run through all of the sort of sort of memset problems and how to clear  up memory, I want to sort of talk about some related issues.  First of all, what I said initially is that when people make this step and go, oh, well,  I should clear this memory because it's sensitive, I said that's huge because it really is.  Most code doesn't even try.  There's an unbelievable amount of code that just keeps keys in memory and sensitive material  and it goes out of scope and it never gets cleared and it just ends up lingering on the  stack or the heap.  I mean, often it would get overwritten fast, but sometimes it could linger on and sit there  for a very, very long time.  Problem with this is that it's hard to find in any kind of automated fashion because you're  looking for the absence of something.  So that means the only way you can really find these kinds of bugs is to manually look  at it and go, oh, this is sensitive material.  No effort was made to clear this.  A second related issue, and this is really cute, actually.  So when you call memset the way it's done there, that's wrong.  The length and the byte you want to memset with are transposed.  The zero should be the second argument and the length should be the third argument.  So what that really does is a no op.  It basically says memset and then use len as the pattern and then the pattern is used  as the length.  In this case, the pattern is zero.  So a memset of zero, which becomes a no op.  And what's really cool is that GCC actually had, you can tell GCC to have made a warning  for this.  And one of the things I wanted to do is I wanted to run through all the same code I had tested  before and enabled the warning and then sort of show it on a list of bugs, but I kind of  ran out of time.  But I strongly suspect if you use this warning and you go download a whole bunch of like well  known open source code and you run it through, you'll end up with a very similar list of  bugs.  So another list of sort of related bugs, and related in the sense that no longer clearing  secrets but related in the sense that optimization was involved.  That is to say, if the optimizer wasn't turned on, a security bug wouldn't have occurred,  right, or would have been less severe, right.  As there are sort of three cases of bugs I sort of ran into that I think is somewhat  relevant that I sort of want to talk about.  So the first one is what's called pointer overflow.  It turns out, let's say you have this code, pdr and then there's a lend and the lend is  untrusted, right.  The idea is that you want to validate that pdr plus lend isn't beyond the end of your  buffer.  Before you do that, you also want to make sure that pdr plus lend doesn't overflow, right.  And so you would do code like pdr plus lend is smaller than pdr.  If that's the case, then the pointer overflow and you bail out, right.  Problem is, according to the C standard, pointer overflow can't happen and so that is undefined  behavior.  And the optimizer sees that and goes oh, I'm defined behavior optimized out, gone.  So you bound check just got optimized out.  That is a relatively common bug to see.  If you don't know it's undefined behavior, if you don't know the compiler can optimize  it out, you would just read over it.  But once you know, like you start reading code and you'll see it everywhere.  Also the way to fix this is basically to cast your pointer to an integer type that's  big enough to hold a pointer and all of a sudden the optimizer can no longer optimize  it out and it'll be in your code.  So that's the first one.  The second one is a lot more subtle.  This has to do with switch case optimization.  So when you have a switch in C and you translate it to assembly, you could just do like a one  one sort of translation.  But if you have, if you use the optimizer, one of two things will happen.  Either it'll generate a binary tree that's generally observed in the Microsoft compiler.  If you look at GCC and Clang, what they do is they'll create a jump table.  And what that means is they'll look at the value to compare and then if they compare  a certain number, they get the value again and they use that value as an offset in a  jump table.  And that usually isn't the problem and that's an abstraction and most people don't care  about it in most situations except if you're dealing with a shared memory trust boundary  because all of a sudden there is a subtle double fetch that was emitted by the compiler  behind your back that doesn't show up in the actual C code.  And these are situations like a hypervisor trust boundary.  These are very, very strong trust boundaries.  And that thing is actually a link if you look and once I publish the slides, if you click  on it, it links to a blog post that shows just that.  It's a virtual box guest host pre-resolation because of a switch case jump table optimization.  And so the bug there basically is that when you fetch the first time you do the compare,  it's fine.  And then two instructions later you fetch again to do the jump table.  Between the first and second fetch, the guy on the other end of the shared memory could  change it.  And all of a sudden you can jump outside of your jump table and basically cause an arbitrary  jump which obviously is bad.  Yeah, I'm not going to cover this.  Okay.  I got to wrap it up.  I got three more slides and then we can get to questions.  This is actually very important.  So now after everything I've covered, we're good.  We know what the problem is.  We know what the solutions are.  We know there's real world problems.  But we have a good grasp of it now, right?  Okay.  It turns out I kind of lied to you.  This is not the whole problem.  If it was the whole problem, that'd be great.  We know how to fix that more or less.  It turns out that compile optimization is really, really clever and it does many, many  clever things and they're all very subtle and a lot of them are architecture specific.  And here's a scenario of things that can occur, right?  You'll have the optimizer will do things like, oh, well, you're handling this string of a  certain kind.  You know what we'll do?  I'll just shove it into a bunch of registers so it'll be faster.  And then, you know, it passes something and then all of a sudden the optimizer goes, oh,  you know what?  You don't have enough registers.  It's okay.  What I'll do is I'll take whatever's in the registers and I'll dump it on the stack and  then we'll go from there.  All of a sudden what happened is you leak key material in registers and then you leak  them on the stack.  And this stuff happens throughout.  And secrets leak out.  It just happens.  And it is because of optimization.  Even if you try really, really hard by trying to do the really hard to do the right thing  and the optimizer, it's just, they just screw you, right?  And this problem is echoed in the blog post by Colin Percival who used to be the security  guy for 3BSD, now has, I think, a cloud storage company, very, very smart security guy.  I would recommend reading that blog post.  This problem is also echoed in the Linux map page, Exocasive B0.  And the map page basically says, well, yes, this is a fundamental problem.  We still recommend using this as B0 and our hope, our thought process is that towards  the future, we will have a way to get the compilers to not do this and we can move on.  But at the present time, there's no good fix for this.  And so this is the first statement I have that I want to make is that at the present  time, optimizing compilers and cryptography are mutually exclusive.  You can have one or the other.  You cannot have both.  It does not work at the present time.  Okay.  Before I get to my conclusion, I want to rant a bit about optimization as if I haven't already.  But basically, I mean, I get that optimization is great and it gives you all these things  and things get faster.  But I have a real problem with optimization because, look, if you're a developer and you  write code and you're pretty smart, you can reason about your code because you wrote it  and you know what it does.  But if you then compile it and it's gone through an optimization pass, you can no longer reason  about it because you don't know what the optimizer did.  And that is, I think, a fundamental problem.  And sort of what I want people to sort of think about is whenever they do dash O is  the, and I'm not saying you shouldn't use the optimizer because it has many pros, but  don't be like blasé about it.  Before you type dash O, really think what it means to do that because it will introduce  all sorts of things that you weren't sure about or that suddenly changes the meaning  of something.  And what I really want for the compiler people and maybe the language people to implement  this so the compiler people can do it, is I want strong accountability and control of  the optimizer.  That is, if I compile something, I want to be able to compile, I want to be able to  go to the compiler and say, hey, before you do anything, I want you to give me a detailed  list of everything you're about to do in terms of optimization so that I can take that list  and look at my code and then with that list in my code, I can now reason again about what  the binary is going to do.  Without having that kind of accountability, you can't reason about your binary.  And the other thing is control, right?  I want to have fine grain control over optimization.  What I mean by that is, you know, like the localized stuff that I mentioned before that  the Microsoft compiler, for example, has, where I want to be able to go and say, this particular  scope don't optimize or this particular scope don't do this particular optimization.  I would like to see something like that.  I'll just skip this.  So okay, here's my conclusion.  Right.  We know what the problem is, the original problem.  And then I have some solutions and then in retrospect, they're just partial solutions  but they're still kind of solutions.  Okay, and then I also have a call to action of things I think should happen and hopefully  will happen at some point in the future.  The problem, as I've illustrated, I think is rampant.  And so basically, I would like people to use that GC patch that I've shown or create a  better one and go find some bugs or better yet, go fix some bugs.  In terms of compilers, as I just mentioned, what I want is optimization, accountability,  and control.  It's kind of the wild, wild West in terms of optimization where the compilers just go  and do all these things.  And I mean, we have some flags but there's not enough control and there's not enough  accountability.  There's not enough transparency.  You just don't know what's going on.  And some of the problems have like a dumb functionality where it's like a needle in  a haystack.  You want something that's easy to work with, easy to read or easy to parse and that tells  you exactly all the optimization steps you're doing.  And ideally, I'd like the language people to get involved and the standardize on this  because if they do, we can now demand this of all the compilers.  And then lastly, sort of the, you know, coming back to my Nazi, sort of the, what about the  Nazi Ruby Python Perl, Go, Rust, and so on.  It smells bad.  It looks bad, especially when there are runtimes involved.  This is probably worthy of a presentation of its own or multiple presentations.  But I wish I had done more there.  That is essentially it.  I hope you enjoyed it.
35c3-9788-memsad_0_59.png 35c3-9788-memsad_60_707.png 35c3-9788-memsad_708_779.png 35c3-9788-memsad_780_851.png 35c3-9788-memsad_852_1019.png 35c3-9788-memsad_1020_1343.png 35c3-9788-memsad_1344_1523.png 35c3-9788-memsad_1524_2483.png 35c3-9788-memsad_2484_3695.png 35c3-9788-memsad_3696_3779.png 35c3-9788-memsad_3780_4175.png 35c3-9788-memsad_4176_6335.png 35c3-9788-memsad_6336_6551.png 35c3-9788-memsad_6552_9851.png 35c3-9788-memsad_9852_10199.png 35c3-9788-memsad_10200_10259.png 35c3-9788-memsad_10260_10319.png 35c3-9788-memsad_10320_10403.png 35c3-9788-memsad_10404_13319.png 35c3-9788-memsad_13320_14711.png 35c3-9788-memsad_14712_14783.png 35c3-9788-memsad_14784_17903.png 35c3-9788-memsad_17904_17975.png 35c3-9788-memsad_17976_20915.png 35c3-9788-memsad_20916_23759.png 35c3-9788-memsad_23760_28187.png 35c3-9788-memsad_28188_28379.png 35c3-9788-memsad_28380_28991.png 35c3-9788-memsad_28992_29411.png 35c3-9788-memsad_29412_37547.png 35c3-9788-memsad_37548_39179.png 35c3-9788-memsad_39180_40259.png 35c3-9788-memsad_40260_40439.png 35c3-9788-memsad_40440_40535.png 35c3-9788-memsad_40536_41507.png 35c3-9788-memsad_41508_41579.png 35c3-9788-memsad_41580_41831.png 35c3-9788-memsad_41832_41927.png 35c3-9788-memsad_41928_44471.png 35c3-9788-memsad_44472_61439.png 35c3-9788-memsad_61440_68759.png 35c3-9788-memsad_68760_68819.png 35c3-9788-memsad_68820_68903.png 35c3-9788-memsad_68904_70847.png 35c3-9788-memsad_70848_72659.png 35c3-9788-memsad_72660_79091.png 35c3-9788-memsad_79092_79139.png 35c3-9788-memsad_79140_79307.png 35c3-9788-memsad_79308_80015.png 35c3-9788-memsad_80016_80507.png 35c3-9788-memsad_80508_80783.png 35c3-9788-memsad_80784_83579.png 35c3-9788-memsad_83580_83675.png 35c3-9788-memsad_83676_88235.png 35c3-9788-memsad_88236_88295.png 35c3-9788-memsad_88296_88427.png 35c3-9788-memsad_88428_92015.png 35c3-9788-memsad_92016_92183.png 35c3-9788-memsad_92184_92291.png 35c3-9788-memsad_92292_92460.png
Mem said that's the title of the next talk. Why cleaning memory is hard? Ilya Fansprudl is a security researcher who loves to find out new things. And he found out that it's quite hard to get rid of sensitive content in the memory. So today he's going to give us an overview, presentation. Please give a warm round of applause to Ilya Fansprudl. OK, perfect. Yeah, so as the hair will just explain, my presentation is called MENSAD. Why Clearing Memory is Hard. Before I dive in that, once upon a time that was me. A lot more hair, a lot less fat. This is my 17th Congress... You have pot kettle here, buddy. This is my 17th Congress in a row. I've spoken here a number of times before. I haven't added 35 in there yet, but obviously that should mean there as well. I work for a company called IOActive. I'm the director of penetration testing. But that really just means that I lead teams of pentesters. No double entendre there. Obviously, we're always looking for good security people, so if you're interested, come talk to me afterwards. I like looking at low level stuff, kernels, drivers, type of reasons, that type of stuff. And I enjoy reading code. Okay, enough about me. What's the kind of audience that I think would enjoy this? Pretty all around security people, crypto people, if you like code review, if you like compiler stuff, and if you're just generally curious about technology, I think you might enjoy this. In terms of the knowledge required, the first half of these slides is relatively basic. And so if you have a basic technology understanding, you should be able to understand most of the first half. If you have some C background, that would be nice. And then as I move forward past the first half, things become a bit more advanced. But if you just only grasp the first half, I think that will still be useful. Right. So what does this talk about? Actually it's one very simple, easy to explain crypto implementation problem. And the reason I've dedicated an entire talk about it is because while the problem is easy, the solution is not. There's a lot of moving parts, a lot of subtlety, a lot of nuance. And I'll get into that in a little bit. Now I can hear some of you thinking, well, Ilya, WTF, 2018, why the hell are you talking about this? This is very, very, very well known. To them, I say, well, because this stuff is still everywhere. And I will show that in the slides. I will show this with data. I will show this with bugs. But the driver of this talk, the reason why I started making this presentation is that this year alone, I did engagements for three different customers on three entirely different software projects where they all had this exact same type of bug. And you tell them about the bug, and the customer goes back and says, OK, well, yeah, that's great. We understand. Now tell us how to fix this in a portable way. And that is not very easy. The other thing is that even though this problem is known conceptually, people kind of blasé about it, practically not many people understand how out pervasive this problem is, how realistic it is. It isn't just like, oh, well, the compiler might do this. No, the compiler will do this. And it does it everywhere. And these bugs do show up everywhere. Even though, because it doesn't, it's hard to tell from the code that it is there. If you look at the binary, if you look at the compiler in bits, you see that it's there. And then the third is, given that one of the teams of the Congress this year is Foundations and to talk about things that aren't necessarily new, but to sort of try and help bring the subject to the next generation, I think this fits in perfectly in the concept of Foundations. Right. So, before I dive in, there's a couple of people, we're actually a long list of people that sort of helped me out. As I said, the problem is well known. It's been well known for at least 20 or 30 years. And so many, many, many people have published papers and presentations about this. And I don't know all of them personally. And I wish I could include them all. But the people I've included in here are sort of the one or two away from me that have had some kind of impact in these slides. Some of you are sitting in this audience. And your help has been appreciated. Okay. So, let's actually start. Now let's say you're going to write some piece of code, right? And it's going to be doing something and it's going to be handling sensitive data. Be that keys or decrypted plain text or session tokens or passwords or password hashes or anything that could be considered sensitive, right? Now, if you're a smart security conscious person, right, the moment you are done with that sensitive data, you want to dispose of it, right? You want to purge it from memory, right? Now why do you want to do this? Well, because otherwise, if there's some kind of infolique that is discovered later on, then whatever secrets or lingering memory could be used in your infolique and all of a sudden, you know, your tokens or your keys are leaked out, right? And I mean, that may sound like a stretch, but, you know, things like heart bleed, you know, happen, right? So this is very practical. This can really happen. And for a move on, I want to say, if you think and make the step where you say, okay, I need to dispose of sensitive material once I'm done with it, that's really big, right? Your software that deals with, you know, sensitive material does not do this. So if you make the step thinking I need to purge this, you're ahead of the curve, right? So now, concretely, it would look something like this, right? You would write a little, this is a sample code I have that basically generates a little key and it's a function you give it and you give it a key pointer and this thing declares a local variable that search invites, goes and reads a bunch of random bits, puts it in K, copies K to key and then before it returns, because K is about to go out of scope and it contains sensitive key material, you go and say memset and then you clear the thing and then you return, right? Perfect. You run this, you compile it, you add a main and it does exactly what it's supposed to do. You just use the assembly and it's all perfect. Problem there is what you're doing is that's not release based code, right? When you sort of make code ready to be released, you sort of tell the compiler that it should enable the optimizer, right? You'll give it OS or O2. Those are the most common ones. You just see O3 when people want to live on the edge but usually O2 or OS. Now if you look at the assembly again, you get a whole different picture of what's going on and I want to illustrate this and there's a website called Compiler Explorer which is beautiful, integrates a whole bunch of compilers and it has on the left it shows you the C code and the right shows you the assembly and it's like color based and it's easy to make connections. So let's take our little example and on the left we see the generate key and on the right we see the compiler and sure enough if you follow the colors and left and right you can see what C code translates to which assembly, right? If they make it a little bit easier, that memset clearly gets translated to assembly, right? Now that is when you do minus O0 which is the default which is what you would do if you're developing code and you want to debug this stuff, right? Now once you're done developing and you're about to ship this thing, you do minus O1 for example and the optimizer kicks in and all of a sudden your assembly looks a whole lot different. You'll notice it's shorter. You'll notice that all of a sudden the color of your memset changed whereas in O0 it was this sort of red-ish and all of a sudden it became white and it's nowhere to be found in your assembly, right? It just does not show up, right? That's a problem, okay? Well, what happened, right? Let's, yeah, I stole that. So what happened is a thing called dead store optimization or dead store elimination. And so basically that memset at the end that what you're doing is you are writing into a buffer that is never, ever going to get used again. And an optimizer compiler looks at that and says, hey, you know, I can just take that memset out and I just saved you a couple of cycles and you have a smaller binary, huge win. And because it doesn't really change what the program does, it's fully compliant with all the relevant language standards. Right, so that's essentially, that is in a nutshell our problem, right? And so one of the things I want to do is I want to look at all common compilers and see for which of these I can get it to effectively, practically optimize out a memset like this. And I had to, with some of them it was easy, some of it was harder. I had to fill around with it for some straight up memset works for others. I had to like kind of twiddle and make a for loop or you know, kind of jump around a bit. But essentially that these are lists of 10 compilers I tested. I try to get my hands on the IBM compiler, but I don't have $20,000, so I couldn't do that. But these are the ones that I did test. And then so the first five or the first four you will know, you know, the GCC and Clang and Intel compiler and the Microsoft compiler. And they're all also on the compiler explorer, so it was easy to test those. And it was very easy to get those to optimize out memsets. And then I moved on, downloaded a bunch of others. You know, the Sun Studio compiler and the, and Barcadero's SQL Splash Builder and the ARM compiler and a bunch of others. And out of these 10, I was able to get eight to optimize it out, right? 80% of the most common compilers do this in any, in a practical sense. So it isn't just like a theoretical thing. This really happens. A funny note, I tried really hard to get the PGI compiler to do it. In fact, it has a switch called that's for elimination. And of course I played with it and I tried it and I spend over an hour trying to get it to optimize out my memset. God damn thing wouldn't move. I don't know what it, I don't know what it's doing there. I couldn't get it to do anything. But basically most compilers, if you ask them to do optimization, will gladly optimize out a lot of memsets. Right, so the next question is how common is it to actually see projects use optimization, right? And this sort of stems from a conversation I had earlier this year with a couple of colleagues where a bunch of people said, well, you know, I don't see 02 or 01 or OS all that often. I don't think optimization is all that common. And so I started looking around and I said, okay, well, where can I get some data? And the first thing I saw was, okay, well, I can go to opsewers.apple.com and that lists about 200 projects and I'll just go through all their make files and look for 02 or OS and so on. And there's about 100 out of there. And then I realized that they actually don't use make files. They have a really bizarre build system and that build system, by default, uses OS. So even though it says 100 out of 200, it's probably closer to 200 out of 200. And then I had a whole list of programs I wanted to test in like FreeBSD and you've put into it a bunch of Linux distros. But that's pretty boring and I ran out of time. So I kind of stopped there. But these numbers should be good enough. In addition, if you look at the common IDEs, in particular Visual Studio and Xcode, when you tell them to build a project in release mode, Visual Studio by default does 02. Xcode by default does OS. So the fact that these tools by default give you optimization should make you confident enough in knowing that, yes, in fact, optimization is incredibly common in release builds. It isn't everywhere, but it is almost everywhere. Right. So now that we know the problem and now that we know it isn't just theoretical and that we know it's practical and that, in fact, it does occur very, very often and with most compilers, in fact, basically it's a real problem. How do we fix this? And this is sort of where things get difficult. There are many sort of solutions. Nothing is portable. Right. It's sort of the, OK, well, this solution works if you use this compiler and this solution works if you use this Lipsy and this solution works if you use this OS and this solution works with this version of the language spec and this solution works if you have this particular executable file format. Right. So and before I dive into any of those, let's first talk about the elf in the room. Don't just roll your own. I've seen people do this where they go like, oh, well, you know, I'll just, I'll fight with the compiler. I know what I'm doing and they'll just, you know, they'll kind of leader where Jenkins this and they'll totally screw it up and they'll come up with, you know, some really stupid ID. One of the ones I heard was like, oh, well, I'll just, you know, I'll just do IO with the buffer and then it's cool. Yeah, you could do that, but then you're doing IO, right? For no reason. So don't just roll your own. You're going to come up with a solution that's probably stupid. You're going to look really stupid. And it'll be one of these things where, okay, you're sort of, your bad solution might work for this particular version of the compiler, but if you don't understand the concepts behind it, then chances are the next version of the compiler that is somewhat slightly smarter will sort of just bypass whatever you implement it. So don't roll your own or if you want to roll your own, at least listen to the advice of the next 10 and then base your solution on at least some of the advice that I'll be giving out in the next couple of slides. So with that, let's move on to actual solutions. The first one is a Lipsy function called explicit B0. This is not part of any standard, as far as I can tell, at the present time. But this was sort of concocted in May 2014 by the OpenBSD guys. If you note the date, it's pretty close to what hardly happened. It's a few months later. I think that may have some relation. Anyway, this function basically does a B0, but explicitly guarantees that it does not get optimized out. Now what that means is that what it does or it doesn't, it's no longer your problem, it's the Lipsy's problem because they've made the guarantee, so now it's on them. So that's really nice. OpenBSD did first and then NetBSD said, that's a great idea and we're going to steal it. But we're going to rename it though. So they changed the name to explicit memset, but it's essentially the same thing. And then about two years ago, FreeBSD came up sort of the same thing. And then almost two years ago, the GLipsy guys came up with this too. And then DietLipsy supports this too. OSX, however, does not support it. So if you're limited to those platforms, explicit B0 is a perfect solution. Similarly, if you are developing for the Windows world, there is an API called Secure Zero Memory which basically is Microsoft saying, we guarantee that this thing doesn't get optimized out. So if you want to securely clear sensitive material, just use this API. MSN says it will ensure that your data will be overwritten promptly. And this is one of the cases where Microsoft was ahead of the curve by like 15 years. They've had this thing since the early 2000s. It was in XP and it was in Windows 2003. Both operating systems are no longer supported, but the API is. OK, so now there's another function called memset underscore s. And it guarantees it doesn't get optimized out. And it's guaranteed by spec. It's guaranteed by the language spec. It is standardized. It is in C11. It is wonderful. It is great. Except it's not great. Because even though it's in the standard and it's there, it's in what's called the optional NXK. And if you read the spec, and it's like pages and pages of boring crap. But if you end up reading the K2, it says optional extension. What does optional extension mean? It means you can be entirely C11 compliant and not offer memset s. So it's kind of this as Reverend Lafayette would say, yes, but if, no, it abut. So if it's there, great. If it's not, it has the potential of being this great portable solution and then it isn't. Right. And then, of course, the sort of obvious choice, but somehow a lot of people seem to miss this is if you end up doing something with sensitive material, chances are you're using a crypto library. And if you're using crypto library, chances are the crypto library offers you an API to do secure memory cleaning. And so I listed the common ones. If you're using OpenSL, there's OpenSL cleanse and OpenSL guarantees they don't get optimized out. If you use Canutee less, there's Canutee less memset. Same thing, Canutee less guarantees it does not get optimized out. If you're using LeapSodium, which is one of the newer ones, they have sodium memzero. Same thing, they guarantee it doesn't get optimized out. And I'll get down to a minute. So this is basically up until here, I've sort of given you a list of, okay, well, here are specific API functions you can call. If you're using this library or using this OS, use this. The next sort of solutions are sort of the, okay, well, what if you can't rely on the APIs? Maybe we can get something out of the compiler. The first solution is, and this isn't portable, but most compilers have this or something like it where you can go to the compiler and say, hey, don't use the built-in memset, just use the one from libc. And what that means is you tell the compiler that it shouldn't assume that it knows what memset does. And if you do that, then sure enough, memset won't get optimized out. GCC, this is originally GCC specific. And then the Intel compiler supports it too. And then Clang supports it too, but up until, and this is true, up until Clang 3.7, which is maybe two years old, it's not that old. Clang basically supported FNobilt in memset. And then what they did is they kind of dropped it on the floor and it got optimized out anyway. So it's kind of annoying. It kind of ruins the whole use this because it works, except if you're using an older version of Clang. And also, it's not overly portable, but it's a solution that works. Other things might still get optimized out. If you have some kind of for loop that clears memory, that might still get optimized out. But at least if you use memset and you're using no built-in memset, then you have a pretty strong guarantee that it shouldn't get optimized out. So another sort of solution is just don't use optimization. That works. You're guaranteed not to get optimized if you don't use the optimizer. Obviously that isn't perfect. For one, Fortify source doesn't work if you don't use optimization. So if you want to use Fortify source, you have to use optimization. The other one, of course, is that you've got to change your build environment. I guess it's not portable, but then again, most compiles will have some way to tell it to not optimize anything. But obviously the reason you don't want to use this particular solution is because you don't get the optimizer. So your product will probably be slower. Sort of a spin-off of this is some compiles, in particular the Microsoft one. And then GCC also kind of supports it, is where you can localize optimizations based on scopes and functions. And so you can say, oh, for this function, do o0. It's not a commonly used feature. It seems like it might have some side effects and it doesn't support all switches that the compiler generally does. I've seen this recommended by a few people. I played around with it. It seems to work. But it doesn't seem to be a sort of commonly adopt way of doing things. The other thing, of course, is again, this is, I mean, these are pragmas. This is very, very compiler-specific stuff. Another solution is using what's called weak symbols. Anybody familiar with weak symbols? Oh, yeah, a little bit. I'll try to sort of very briefly get into this. So the L file format basically is a format that specifies how to have an executable that can run on OSes that support this file format. So, Ls, if it's compiled, for example, for Linux, it'll be in this particular format. And one of the, obviously, as part of format, you can store symbols for things like functions and variables and so on. And generally, a symbol is what's called a strong symbol. You can mark one as weak. And what weak means is that a symbol may change in runtime. And what that means is if you declare a function or a symbol of a function as weak, that means that compile time, the compiler would have a very hard time to reason about what that thing does because of the sheer fact that you've declared it as weak. And in fact, this particular solution is what it means to use this in their implementation of explicit B0. And what I really like about this is that this is the commit message for the OpenBZ guys. And they're very pragmatic about it. They say, well, you know, we think our solution is pretty clever, but it's not foolproof. There are still ways to defeat this. And they list, you know, a bunch of ways to do this. In particular, you know, well, you know, the compiler could emit runtime code that checks what this thing is in runtime right before it's called. And then you could still optimize it out if the thing matches or doesn't match. But then they go on and say, well, in the foreseeable future, we don't think that's going to happen. But it's possible that at some point down the road this might happen. And so I like the way of reasoning about this, where solution is pretty clever, but it's not foolproof. It may at some point in the future break. But at the present time, it's a fairly good solution, I think. Right. So another solution is to use memory barriers. How many people know what a memory barrier is or what it does? Okay, about the same number of hands. Okay. Let me try to very briefly sort of explain what a memory barrier is. And bear with me here because it's, and I'm going to oversimplify it because it's not a particularly simple concept if you've never heard of it. But okay, let's say you have a piece of code, two global variables, A and B, and you assign a value. You say A equals something, and B equals something, right? And there's no relation between A and B. Right? What that means is both the compiler and the hardware, because they have no relation. Both of them are allowed to reorder it. So B can be assigned first, and then A can be assigned later. Because there's no correlation, that's perfectly valid. Now let's say you have a second thread somewhere, and your second thread says, okay, you know, while not B spin, and then once B is set, you use A. Right? This is sort of where you're basically waiting on something to be set, and the idea is that you wrote your code to that B set after A is set. And that seems logical, and that would work, except if the compiler and hardware don't know, there's a relation between your loop on one end and the assignment on the other, and either the hardware or the compiler reorders it and sets B before it sets A. Really, really nasty things happen. This has been the source of numerous security bugs, very, very subtle stuff. The K, SAN, and T, SAN kind of stuff. We've seen Linux kernel last couple of years. A bunch of that is related to these kind of bugs. And so the way that you fix this is you introduce what's called a memory barrier, and that is basically when you write your code and you say A equals something, and then before you say B equals something, you basically, and it will say memory barrier, and then you say B equals something. And what that means is it gives a signal to the hardware and to the compiler, and it says, whatever happens before this and after this, you are not allowed to reorder this. There is a correlation there that I know that you're not aware of, so don't reorder it. And I hope I explained it well. This usually takes a lot longer to explain, but I hope I got the message across well enough to sort of give you an idea of what a memory barrier is. And now the cool thing about memory barriers is that it's a way for a programmer to tell the hardware or the compiler, I know something about this memory, you don't. Stay away, don't touch. And because of, I mean, it works for reordering, but it also works really well to not get something optimized out, right? The idea is you could basically just do your memset, and then on the thing you memset it, you basically do a memory barrier, and that tells the compiler not to optimize it out. And I know the concept sounds complicated, but it's pretty clever, and I've oversimplified this because it's a relatively complicated subject. But this works really well, and this is used by Dytlipsy, and it's used by Glypsy, and Nginx recently had a fix where they have their own explicit memzero, and it also uses a memory barrier. So this is a tried and tested concept, and it works. So those are kind of the solutions that are known and that work, and have been tried and tested by various fairly well-known pieces of software. If none of those things, if somehow you're in an environment somewhere, and none of this is available to you, or it's not portable enough, and you're looking for a solution that works everywhere, the best you can do is fall back on constructs that are known in the C language, and this is basically the use of the volatile keyword. I call this a fallback. People often go, well, just use volatile, and then that solves the problem. It turns out optimizers can be very clever and very tricky, and even when you use volatile, there are cases that can be made where if the optimizer is clever enough, your data may still get optimized out. So the volatile solutions are sort of best effort fallback solutions, and there's sort of two variants of this. One is a volatile pointer, right? That's the fallback solution in the Lipsodium, and the other one is a volatile memset function pointer, which is what OpenSL uses. And here's what that looks like. This is the Lipsodium fallback, and this looks like, you know, volatile is sort of the, you know, you tell the compiler, hey, you know, I know something about this, you don't. And this is where it gets very language-loyalry. If you look at the spec, it says something along the lines of the access object, something, something, and they're describing the actual memory volatile, not just the pointer L value. And so if the compiler looks at this code, and it can trace, and it can prove that wherever PNT came from, and if that isn't actually volatile, then this volatile doesn't really mean all that much, and it can still optimize it out. That sounds very theoretical. I don't know if that actually happens, but a number of people smarter than me or that know more about the sort of, this negrity little C language things have told me that, yes, in fact, you would be allowed to do that if you're a very smart, optimized, and compiler. The fact that it's a fallback solution for Sogium and a few others leads me to believe that it probably doesn't, but it could. Right. And so this is the solution that OpenSL uses, which also uses volatile. It doesn't do a pointer right, but instead it creates a volatile function pointer that points to memset. And it sort of gives you, you get the same concept more or less as with the weak symbols. The idea being is that your volatile function pointer can change at any time without the compiler knowing about it. And that seems like a pretty good solution, except when you, one way of, in theory, getting around this is if the compiler emits runtime code that right before the function pointer gets called, it looks and goes, well, like it captures it and then goes, is it memset or is it something else? If it's something else, then we call it. If it's memset, we just return and then you optimize out the runtime and save a few cycles. In theory, the compiler is allowed to do that and emit code like that. And I don't know if that actually happens anywhere, but it's a possibility. So think of these last two solutions as fallback. They may not work in theory. In reality, they probably do. Right. So this is sort of the first half of my presentation and I'm perfectly on time, which is great. So now there isn't one portable solution and this is why, this is why clear memory is hard. The problem is well understood, but if you're looking for like an all-around solution that works everywhere regardless of compilers and the operating system and so on, it's very hard to have a good solution. And this is what, where customers have come back to me and said, give me a portable solution. I need something better than this or that. And so the best solution I have for this is sort of, you know, apply all of the above as best as possible. And my initial idea was I'll just write a little function that does this and put it on GitHub and people can use it. But if you look at lip sodium and then, yeah, you see, I mean, I'm not going to click on it now, but that's the link. If you download the slides later, you'll see it points to GitHub and it shows you the actual limitation. Lip sodium's MEM-0 is really well written and it's beautiful and it sort of has this fairly elegant, you know, if, you know, this, this and this or if deaf this, then do this solution. If, you know, this particular setup, then do this solution and it does, has this for six or seven of the cases I've covered. It's really nice. It's really elegant. If you're looking for inspiration, point people to lip sodium. I think it's a good portable-ish way of solving this problem. Right. Okay. So that now we've discovered, we've talked about the problem. We've talked some solutions. Okay. Well, I want detection. When does this really happen? I want to see this, right? And I want compilers to tell me this. Like why doesn't GCC tell me it's not writing something out? Like if it has security consequences, it should tell me. I want, why are they not doing this? I don't understand. So I set out and I modified GCC. I looked at a deathceral elimination and I came up with this patch. And instead of the, this is, this is, there are three SSA deathceral elimination paths and when it calls, delete dead call, I sort of take that out and say, if, you know, it's a built-in memset before you call, delete call, emit this warning, tell me to file and tell me to line number. And then do still optimize it out. And what this means is every time a memset gets optimized out, GCC now tells me. And this sort of, this is very interesting because I not only get, get, you know, detection for my own code, this is a great way to get, to get really cheap, fast zero day. And in fact, that's what I did. I downloaded a whole bunch of very well-known open source projects and I ran them through a modified version of GCC and I came up with a list of things. Oh, awesome. Thank you. So I know of this particular problem in, in like practically affecting open SSL, MIT KERP, Heimdall KERP, Matrix SSL, PHP, DHCP, bind, squid cache, and the list goes on. I have R sync as well and there's more. So we know this problem is very widespread, right? If the stuff we all rely on, the stuff that is built on has these problems, that means your code probably has this as well. And of course, I'm just giving names out here. But let's give you guys some zero day. That's MIT KERP. That mem sets optimized out. That's PHP. That mem sets optimized out. This I think is Matrix SSL. That decrypted plain text gets mem set it. That gets optimized out. That's the lingers around memory. This is open SSL. That crypto extended data, that doesn't get optimized out. This is NGINX. That password, that mem zero gets optimized out. This is a bind and DHCP. That mem set of private key data gets optimized out. This is quid that it goes to LDAP and gets creds. And basically it tries to clear the creds and then that gets optimized out. Yeah, well, I had to play around with Power of Wind a little bit. So and then same thing, this is a key that gets optimized out. And then this is R sync. And these are stored credentials in a file and that gets random memory and those mem sets get optimized out. So that's nine bugs right there. And all it took was five lines of code change in GCC. And GCC just gave me all these bugs. The other thing about seeing exactly what. Thank you. The other thing that was really nice about getting the data back from GCC isn't just that it gave me bugs. It also showed me things that optimized out that I thought it wouldn't. Obviously what I was expecting is a variable that's about to go out of scope and that you get mem set that would get optimized out, obviously. But what I also noticed was that, you know, when you there's a common code pattern when you just what you just malloc something or you declare something on a stack. And the first thing you do is mem set to clear the whole thing and then you move on. It turns out that in a number of cases that also gets optimized out. The idea is that if the compiler it only gets optimized out if the compiler can prove that every element in the struct or that the whole field gets filled in. I'm not sure if that's entirely true. And we were talking about this earlier. But what about, you know, things like structure padding or maybe enums or, you know, something like that or, you know, unions, I mean, that I'm not quite sure how that works. I suspect I haven't dug into it because this is I ran it I found this yet. I mean, I wrote this patch yesterday, right? These bugs are like they're fresh. So I don't know exactly how much potential there is here, but it smells like there's room for bugs here. So I was surprised to see this and some more research is needed. So if anybody wants to feel free. And then the other thing I sort of noticed is that obviously the common case when I was looking for in terms of bugs was something that was sensitive material. And then obviously I saw a whole bunch of things where non-sensitive material was being mem set and then freed. And then that mem set would get optimized out as well. And that struck me as odd at the beginning, but then obviously there's a common sort of coding pattern that I've seen where, you know, anytime somebody does a malloc and then before using doesn't mem set null or when they free right before they do a mem set zero, not because they want to clear sensitive material, but because they always they want to have a guarantee they always start from a clean slate. And so they end up building code that ends up working for something that always has a clean slate, right? And if that if that mem set gets optimized out, then those guarantees no longer hold. And so code that works around the sort of, well, we always have a clean slate when we you know, when we get a fresh piece of memory, that is no longer true. And so I think that coding pattern doesn't jive well with compiler optimization. Again, this is the sort of realization I made yesterday. I don't have all the facts on this yet, but it seemed interesting. And I think there's some room for research here. The other thing I noticed is that sort of close to the mem sets that were optimized out, I noticed like other bugs, and this is kind of like, you know, it made me think is like, well, bad code attracts other bad code. So one of them was null derapsing the other was the use after free, where, you know, instead of doing mem set free, the code was doing free mem sets, right? And that obviously is a use after free. So this is the case of the null derp where basically a malloc happens and then the mem set null happens and then there's a check to see if the variable that was allocated is null, right? Obviously that mem set, if the value is null, that mem set would cause a null derp. Except the mem set gets optimized out so it never generates the null derp. It kind of catches 22 there. But that construct is clearly broken. So now, I mostly spoke about mem set, but really there's a thousand variations that clear memory and sort of comes down to the same thing, right? Obviously you can do a for loop or you can use other APIs or you can like roll your own and do something very exotic. And then there's C++ and which there's a gazillion ways of doing it. You can have like weird classes with like a constructor and like, you know, inheritance, multiple objects, like virtual. It gets really crazy once C++ comes in the mix. And so basically it all kind of looks different, but it all does the same thing. It has the same root cause and the same problem. And the thing is when you look at this from just a code perspective is that sometimes the optimizer is smart enough to see it. Sometimes the optimizer is not, but it could in the future be smart enough to see it. So it's one of these things where if you're looking at a piece of code and you're doing some kind of security assessment and you see this and like, should you report to bug, should you not? I think you should because even if the compiler doesn't optimize it out today, it may very well optimize it out tomorrow. Right. Okay. So now oldest talk has been about C and when you write it in C, well, what if you're not writing C code? What if you're using other languages, non-native languages, you know, go Rust, Objective C, C sharp, Java, flavor of the month. And really I wanted to spend more time on this, but then my slides would have gotten so long I couldn't. I only have one slide on non-C, but I spent a little bit of time on this in C sharp. There's something called secure string, which is supposed to hold a string in a safe way. And the problem I have with secure string isn't the implementation. It's how do I get something securely into secure string and how do I get something securely out of it? And then in terms of Java, there's a Java crypto guide, which basically says it recommends not using strings to hold sensitive material, but to use a biderane instead. And there was some reason behind it. I don't remember. But the idea is basically this. Most managed languages don't really offer any decent way to clear memory or to hold sensitive material in memory without it leaking. And it will leak and it'll kind of happen behind your back because you wouldn't know it leaked, especially when you're dealing with garbage collection where something can get reallocated without you ever knowing. And all of a sudden, before you know it, there's like five different copies of your key sprayed all over memory. These languages, as far as I can tell, they don't have the infrastructure in place to deal with sensitive material. It seems to be entirely missing in a lot of places. In other places, it's kind of like a shoe horned on or bolted on with some varying degrees of success. I remember seeing there was some in place for go, but it had been in revision three or four because there was always something wrong with it. And so, again, I wish I had more time than I could elaborate on this. From what I saw is that it's a pretty sad state of affairs in Nazi that most people haven't tried and those who have have not tried hard enough. So now that I've run through all of the sort of sort of memset problems and how to clear up memory, I want to sort of talk about some related issues. First of all, what I said initially is that when people make this step and go, oh, well, I should clear this memory because it's sensitive, I said that's huge because it really is. Most code doesn't even try. There's an unbelievable amount of code that just keeps keys in memory and sensitive material and it goes out of scope and it never gets cleared and it just ends up lingering on the stack or the heap. I mean, often it would get overwritten fast, but sometimes it could linger on and sit there for a very, very long time. Problem with this is that it's hard to find in any kind of automated fashion because you're looking for the absence of something. So that means the only way you can really find these kinds of bugs is to manually look at it and go, oh, this is sensitive material. No effort was made to clear this. A second related issue, and this is really cute, actually. So when you call memset the way it's done there, that's wrong. The length and the byte you want to memset with are transposed. The zero should be the second argument and the length should be the third argument. So what that really does is a no op. It basically says memset and then use len as the pattern and then the pattern is used as the length. In this case, the pattern is zero. So a memset of zero, which becomes a no op. And what's really cool is that GCC actually had, you can tell GCC to have made a warning for this. And one of the things I wanted to do is I wanted to run through all the same code I had tested before and enabled the warning and then sort of show it on a list of bugs, but I kind of ran out of time. But I strongly suspect if you use this warning and you go download a whole bunch of like well known open source code and you run it through, you'll end up with a very similar list of bugs. So another list of sort of related bugs, and related in the sense that no longer clearing secrets but related in the sense that optimization was involved. That is to say, if the optimizer wasn't turned on, a security bug wouldn't have occurred, right, or would have been less severe, right. As there are sort of three cases of bugs I sort of ran into that I think is somewhat relevant that I sort of want to talk about. So the first one is what's called pointer overflow. It turns out, let's say you have this code, pdr and then there's a lend and the lend is untrusted, right. The idea is that you want to validate that pdr plus lend isn't beyond the end of your buffer. Before you do that, you also want to make sure that pdr plus lend doesn't overflow, right. And so you would do code like pdr plus lend is smaller than pdr. If that's the case, then the pointer overflow and you bail out, right. Problem is, according to the C standard, pointer overflow can't happen and so that is undefined behavior. And the optimizer sees that and goes oh, I'm defined behavior optimized out, gone. So you bound check just got optimized out. That is a relatively common bug to see. If you don't know it's undefined behavior, if you don't know the compiler can optimize it out, you would just read over it. But once you know, like you start reading code and you'll see it everywhere. Also the way to fix this is basically to cast your pointer to an integer type that's big enough to hold a pointer and all of a sudden the optimizer can no longer optimize it out and it'll be in your code. So that's the first one. The second one is a lot more subtle. This has to do with switch case optimization. So when you have a switch in C and you translate it to assembly, you could just do like a one one sort of translation. But if you have, if you use the optimizer, one of two things will happen. Either it'll generate a binary tree that's generally observed in the Microsoft compiler. If you look at GCC and Clang, what they do is they'll create a jump table. And what that means is they'll look at the value to compare and then if they compare a certain number, they get the value again and they use that value as an offset in a jump table. And that usually isn't the problem and that's an abstraction and most people don't care about it in most situations except if you're dealing with a shared memory trust boundary because all of a sudden there is a subtle double fetch that was emitted by the compiler behind your back that doesn't show up in the actual C code. And these are situations like a hypervisor trust boundary. These are very, very strong trust boundaries. And that thing is actually a link if you look and once I publish the slides, if you click on it, it links to a blog post that shows just that. It's a virtual box guest host pre-resolation because of a switch case jump table optimization. And so the bug there basically is that when you fetch the first time you do the compare, it's fine. And then two instructions later you fetch again to do the jump table. Between the first and second fetch, the guy on the other end of the shared memory could change it. And all of a sudden you can jump outside of your jump table and basically cause an arbitrary jump which obviously is bad. Yeah, I'm not going to cover this. Okay. I got to wrap it up. I got three more slides and then we can get to questions. This is actually very important. So now after everything I've covered, we're good. We know what the problem is. We know what the solutions are. We know there's real world problems. But we have a good grasp of it now, right? Okay. It turns out I kind of lied to you. This is not the whole problem. If it was the whole problem, that'd be great. We know how to fix that more or less. It turns out that compile optimization is really, really clever and it does many, many clever things and they're all very subtle and a lot of them are architecture specific. And here's a scenario of things that can occur, right? You'll have the optimizer will do things like, oh, well, you're handling this string of a certain kind. You know what we'll do? I'll just shove it into a bunch of registers so it'll be faster. And then, you know, it passes something and then all of a sudden the optimizer goes, oh, you know what? You don't have enough registers. It's okay. What I'll do is I'll take whatever's in the registers and I'll dump it on the stack and then we'll go from there. All of a sudden what happened is you leak key material in registers and then you leak them on the stack. And this stuff happens throughout. And secrets leak out. It just happens. And it is because of optimization. Even if you try really, really hard by trying to do the really hard to do the right thing and the optimizer, it's just, they just screw you, right? And this problem is echoed in the blog post by Colin Percival who used to be the security guy for 3BSD, now has, I think, a cloud storage company, very, very smart security guy. I would recommend reading that blog post. This problem is also echoed in the Linux map page, Exocasive B0. And the map page basically says, well, yes, this is a fundamental problem. We still recommend using this as B0 and our hope, our thought process is that towards the future, we will have a way to get the compilers to not do this and we can move on. But at the present time, there's no good fix for this. And so this is the first statement I have that I want to make is that at the present time, optimizing compilers and cryptography are mutually exclusive. You can have one or the other. You cannot have both. It does not work at the present time. Okay. Before I get to my conclusion, I want to rant a bit about optimization as if I haven't already. But basically, I mean, I get that optimization is great and it gives you all these things and things get faster. But I have a real problem with optimization because, look, if you're a developer and you write code and you're pretty smart, you can reason about your code because you wrote it and you know what it does. But if you then compile it and it's gone through an optimization pass, you can no longer reason about it because you don't know what the optimizer did. And that is, I think, a fundamental problem. And sort of what I want people to sort of think about is whenever they do dash O is the, and I'm not saying you shouldn't use the optimizer because it has many pros, but don't be like blasé about it. Before you type dash O, really think what it means to do that because it will introduce all sorts of things that you weren't sure about or that suddenly changes the meaning of something. And what I really want for the compiler people and maybe the language people to implement this so the compiler people can do it, is I want strong accountability and control of the optimizer. That is, if I compile something, I want to be able to compile, I want to be able to go to the compiler and say, hey, before you do anything, I want you to give me a detailed list of everything you're about to do in terms of optimization so that I can take that list and look at my code and then with that list in my code, I can now reason again about what the binary is going to do. Without having that kind of accountability, you can't reason about your binary. And the other thing is control, right? I want to have fine grain control over optimization. What I mean by that is, you know, like the localized stuff that I mentioned before that the Microsoft compiler, for example, has, where I want to be able to go and say, this particular scope don't optimize or this particular scope don't do this particular optimization. I would like to see something like that. I'll just skip this. So okay, here's my conclusion. Right. We know what the problem is, the original problem. And then I have some solutions and then in retrospect, they're just partial solutions but they're still kind of solutions. Okay, and then I also have a call to action of things I think should happen and hopefully will happen at some point in the future. The problem, as I've illustrated, I think is rampant. And so basically, I would like people to use that GC patch that I've shown or create a better one and go find some bugs or better yet, go fix some bugs. In terms of compilers, as I just mentioned, what I want is optimization, accountability, and control. It's kind of the wild, wild West in terms of optimization where the compilers just go and do all these things. And I mean, we have some flags but there's not enough control and there's not enough accountability. There's not enough transparency. You just don't know what's going on. And some of the problems have like a dumb functionality where it's like a needle in a haystack. You want something that's easy to work with, easy to read or easy to parse and that tells you exactly all the optimization steps you're doing. And ideally, I'd like the language people to get involved and the standardize on this because if they do, we can now demand this of all the compilers. And then lastly, sort of the, you know, coming back to my Nazi, sort of the, what about the Nazi Ruby Python Perl, Go, Rust, and so on. It smells bad. It looks bad, especially when there are runtimes involved. This is probably worthy of a presentation of its own or multiple presentations. But I wish I had done more there. That is essentially it. I hope you enjoyed it.
get rid of sensitive content in the memory. So memset and then you clear the thing and then you a conversation I had earlier this year with a data will be overwritten promptly. And this is volatile, and then that solves the problem. It string and how do I get something securely out of