PACKET HACKING VILLAGE - Defense in Depth The Path to SGX at Akamai
Erb, Samuel
In this presentation you will learn how Akamai has spent the past 4 years working toward preventing the next TLS heartbleed incident. Nothing hypothetical -only deployed defense-in-depth systems will be discussed. This talk will include how we deployed Intel SGX at scale in our network.
 It is absolutely my pleasure to introduce to you from the Council of Nine. Sam Herb. Hello. So, this is going to be a bit of a different talk. It's definitely more blue team than most of what's normally presented at Defcon. So, apologize for that in advance, but it's something that I was, I really wanted to give a presentation on. I think it's important to share with the wider community what we're doing at Akamai. So, with that in mind, this is Defense in Depth, the path to Shaxx at Akamai. So, who am I? I am a software engineer at Akamai. At Defcon, I have won two black badges in a badge challenge with the Council of Nine. Yeah, if you want to reach me, probably the best way right now is to find me on Twitter. So, disclaimer, generic disclaimer. Don't poke and prod at Akamai, please. And everything presented here is the work of many individuals, some of whom are actually in the room today. Which is exciting. This wasn't just me. So, get started. Defense in Depth. I'm talking about Defense in Depth, and what I'm really talking about is the classical computing term, Defense in Depth, where we're adding additional security controls or layers to mitigate or lower the impact of vulnerability in our product. What I'm talking about in my definition, though, what I want to do is I want to avoid the next sales heart bleed at Akamai and build systems that prevent this or otherwise mitigate the impact of that. So, I'm going to ask the question, what is heart bleed? So, open SSL users were affected by heart bleed. We have a blog post on how Akamai was impacted by this. In general, this doesn't turn out very well on the slide there, but this is a wire shark output that shows a bad pack payload length. And what really heart bleed was, was you would request a certain length and you'd say, hey, like, give me five bytes, and it would return five bytes back to you. And it was supposed to be the same five bytes that you sent to it. That's the RFC 6520, which actually defines this, where the receiver must send the corresponding heart bleed response back to the requester. The implementers neglected to verify that the payload length matches the length in the field itself and would send you back arbitrary memory from the server itself. So, this allowed you to read memory from the server and then subsequently potentially get access to the TLS private keys, which we're seeing on that server, or other information for that matter. Just go louder. So, what is TLS? So, TLS is HTTPS. I would, if you haven't had a chance to, I would really recommend just going to read in the RFC. It's super approachable as far as RFCs go. This is straight out of the TLS 1.2 RFC. And what I'm really going to be talking about today more than anything else is the server key exchange message. So, the way TLS works in general is you have a, like, a Hello exchange, and then you have a, in blue there, and then in yellow, you have this Key Exchange period. And then in red is where you're actually sending your bulk crypto data. And I'm really going to be zeroing in on the Key Exchange, and in particular the server Key Exchange. And what that does is proves that the server owns a copy of the certificate by signing certain parameters that get used with the certificate's private key. So, the server itself has to have, somehow have access to the private key, which traditionally has meant that it's just sitting in memory on the server, which is why Harpley was so dangerous. TLS 1.3 makes some slight modifications to this, but that's also something that's worth going and actually reading the RFC. So, with that in mind, I'm getting all that out of the way, I'm going to be walking us through a TLS security journey. It's not the most exciting name, but our first stop is we're going to make secrets harder to find in memory. And to accomplish this, we're actually going to use OpenSSL itself. And in particular, we're going to use one of OpenSSL, one of Occamized Contribution OpenSSL, which is the OpenSSL secure heap. And this is just standard Linux memory protection features to get the keys into a more protected area of memory and to prevent them from appearing in core dumps, etc. And this also applies to other key material, which can be used to derive the private keys as well. I believe the function name that you're going to need to look for is OpenSSL secure malloc. And it's also externally visible or externally exposed so that your application can actually make use of this as well. So, yeah. So, our second stop is we're going to make the, we're going to move the secrets entirely out of process. The reason we're doing this is that when you could dump the memory of the process, you could potentially dump the secrets. So, if they're not in the process memory, you can't access the secrets. And to do that, we're actually going to use the OpenSSL engine interface. I posted some example code here. This is a OpenSSL RSA engine. And really the important thing here is that RSA underscore method. And what that will do is it will pass the RSA or what needs to be signed by the server into a function. And once it's within that function, it's really up to you to make the signature happen and return it back to OpenSSL. And so this gives you a hook into OpenSSL to accomplish this. And so using this hook, we're able to go outside of the process, you know, using a, either shared memory or some other inter-process channel to get to the TLS private key that we need. So what this actually looks like in terms of the protocol, I'm defining a web server here. And that special RSA sign function that I defined previously. I'm going to use that to send client random server random and key parameters off, as well as a certificate identifier off to some external key process. And this external key process is actually going to make the signature and then return it back to the process itself. So if we had had this in place, when Hartbleed rolled around, we wouldn't have been as vulnerable at least. The attacker still would have got access to private memory, which could have contained sensitive information, but at least we wouldn't have had to potentially rotate this key. So what we did here was nothing, or what we did here was nothing particularly special. So we decided to take this to the next step and we moved the secrets not only out of process, but also out of the machine. This means that if you had got a, if you have some sort of bad exploit, like a remote code execution, you have some confidence that this key wasn't leaked. And so what that looks like is we just put the internet between our web server and the external key server. The protocol is identical. Only probably want to add some additional security to this channel. And so for some background here, Akamai has, I think we say over 200,000 servers at this point, distributed internationally. We have a lot of these web servers. So that also means that also implies that we're going to need a lot of these external key servers. And obviously we know what slow handshakes. This only impacts the initial TLS handshake. Subsequent ones will use session keys, which allow for faster session resumption. And also means you don't have to have access to the server private key as well. But this, this puts this problem squarely into the engineering space and not into security space. This is a strictly engineering problem. And it's also a performance security trade off here, at least on this initial TLS handshake. Yeah, it really creates a many, many writing problem, which is something that Akamai is luckily good at solving. We solve that every day with our end users. So in this case, we're turning that problem on its head and we're having web servers connect to these external key servers. So this involves a lot of planning. So this, one of my colleagues put this together, I believe, was Richard Wiley. This is a map, a heat map that shows connectivity between different parts of the world. It was really fascinating to learn about how poorly connected parts of the world are while doing this. So in general, you don't want to be crossing between not just national, but in some cases, continent boundaries. And that create, that will lead to slow connectivity, slow handshakes, and slow performance. And this is just a visualization of this. We actually use the raw data to figure out where to best put these key servers. And this was actually in our, included in our state of the internet report in 2017. So in practice, so this is actually a visual, an internal visualization of our network. It's a little bit hard to see from further back, but the blue dots on there represent locations. We have web servers and the red dots represent locations that we actually put key servers, I should say, because each one represents more than one, or each dot represents more than one machine. And then the yellow and green lines just represent different traffic levels. And this is actually a real-time dashboard that I pulled this off of. So we're actually visualizing our web servers connecting to our key processors in real-time at scale. And Europe is honestly the most exciting one. It has the most lines as closely bunched together, simply due to geographical constraints. Yeah, I've always found the visualization extremely cool. So that's great that we've moved our secrets now to another machine. Let's get a bit of a paranoid, and we're going to put them into an HSM. For those of you who are not familiar with an HSM, this would be something like a UB key, for example, where it's a hardware device and hardware security module, I believe is what it stands for. It's a hardware device that holds your private key, and you're able to set it a request to use that private key, and it will give you back a signed response of some form. And so to accomplish this, I'm going to use Intel SGX. And I'm going to explain a lot more about what Intel SGX actually is in subsequent slides. So this is a little bit paranoid, honestly. This is kind of taking us to the next level as an experiment, but we were actually able to make this work and deploy it, which is very exciting. So we put an SGX key store into each of these external key servers. So what is SGX? So this is an Intel technology. We have it in our certain chips that we deploy in our network. It's not available in all Intel chips yet. I think they're working on that still. And most importantly, it's a new set of CPU instructions for handling enclaves. So enclaves can be thought of as a shared object library, where in the untrusted library, you make a function call and it drops you into this trusted state where you can't really do a whole lot, to be honest. You can do memory operations, but you can't ever leave the chip, and anything that leaves the chip itself is either going to be encrypted or you're not going to have access to. And so what that means is that you can only drop into the state and come back out of the state, which on a first pass doesn't really seem very useful, but you can securely get data into this, which means it can function as an HSM if you code it correctly. So it can be best thought of as a reverse sandbox. Oncleaves can read and write to external memory of a process, not to any external memory, just of a single process, and applications at any level cannot read or write memory within that enclave. So it's kind of when you actually call the enclave, it really will then reach out back to you, fetch the information it needs, do something with it, and then return that data back to you. And one of the great things about this, about SGX, is it has roughly the same performance as the CPU itself, it's running on the same hardware. The hardware just simply drops into this SGX state and comes back out of it. That's the best way to think about this. I'm not the first one to use that term. It was originally termed going down a 2016 Black Hat talk. So oncleaves, these are the shared object libraries that I had in a previous slide. Oncleaves must be signed, and the root of trust is actually traced back to your BIOS. So if you control your BIOS, you control who can run enclaves on your system. It really roots the chains of trust to a very low level. Oncleaves have to be statically compiled. All of the code is signed. So that kind of implies that SG statically compiled. It requires BIOS and kernel support just to make sure everything works correctly. And remote and localization are possible. And this is one of the cooler features of SGX, and is what allows you to securely get your secrets into these oncleaves and trust that they're going to not only the right machine, but the right oncleave itself. So there are a bunch of new instructions, and most of them aren't really that exciting. They're for memory operations and launching and stopping oncleaves. The important two are going to be eReport and eGekki. So eReport allows you to tell another oncleave on the system that you're on the same system as it, and you generate a report, and you sign it in a special way, and then another oncleave can take that data and verify it, even if it passed through an untrusted boundary. eGekki, which is a little bit more interesting, allows you to actually generate a unique key within a given oncleave. And this allows for sealing, which I'll talk about a little bit more later. But the important thing here is that you can generate a unique key on that machine, and then subsequently decrypt data based on that key. I should say that that is actually a symmetric key. So sgxcelein, so that eGekki parameter allows you to generate a key that's shared between all oncleaves that are on the same system and signed by the same key. And this key is persistent, so as long as you call this function the same way, you'll always get the same key back from it, which is awesome, between OS installs, you don't have to maintain any state on that machine. You can wipe it completely, and the key will still come back the same way the next time. They also have counters in there to handle if their vulnerability is found in the process, there are counters in there that will increment, auto-increment, that will allow you to create certain actions based on the key. That's a little bit too much detail, honestly. So we can actually use this sgxcelein system. And per machine, what we ended up doing was we created a public-private key pair within an enclave, and then we had the private key encrypted using the ceiling key. And then once it's encrypted using the ceiling key, we can then export this public-key sealed private key, and that becomes a unique public-private key pair tied to that machine that will survive OS installs. And so we actually were able to use this to do key provisioning at scale. This is across a large number of machines where we had each one of these sgxcelein keystores issue this sealed private key, public key combination. We would then, we have been a massive advantage because we control our machine deployment and our pipeline to get machines deployed. And so we were able to export that key. We then were able to encrypt a key to that public key and then send that and a bunch of other data back down to the enclave. And assuming that this whole process works, only that enclave will actually be able to decrypt the secrets that are encrypted to that public key. I should say this key pair is an RSA key pair, which allows you to encrypt things to the public key such that they can only be decrypted using the private key. Yeah, so at a high level, that's kind of how that all works. So there are a bunch of complexity with this sgxcelein key that I didn't get into. You will spend all of your time worrying about how to rotate every single key you create. Initial trust establishment is a little complicated, which I described previously, especially if you don't control your machine build process or pipeline. Secure handling of multiple enclaves, keys, and most importantly, replay attack prevention are all things that I didn't talk about, but you should worry about if you're actually deploying this. You will spend most of your time trying to figure out how this should work and handle key rotation, handle all these really complex cases, and not a whole lot of time writing code, which to some extent is a good thing. So, yeah, so I should add all of this was deployed within Akamai. If you connect to Akamai, there's a chance that your TLS handshake is actually at some point actually traversing a sgxonclave, which I think is pretty cool. So yeah, thank you. applause
 It is absolutely my pleasure to introduce to you from the Council of Nine.  Sam Herb.  Hello.  So, this is going to be a bit of a different talk.  It's definitely more blue team than most of what's normally presented at Defcon.  So, apologize for that in advance, but it's something that I was, I really wanted to give a presentation on.  I think it's important to share with the wider community what we're doing at Akamai.  So, with that in mind, this is Defense in Depth, the path to Shaxx at Akamai.  So, who am I?  I am a software engineer at Akamai.  At Defcon, I have won two black badges in a badge challenge with the Council of Nine.  Yeah, if you want to reach me, probably the best way right now is to find me on Twitter.  So, disclaimer, generic disclaimer.  Don't poke and prod at Akamai, please.  And everything presented here is the work of many individuals, some of whom are actually in the room today.  Which is exciting.  This wasn't just me.  So, get started. Defense in Depth.  I'm talking about Defense in Depth, and what I'm really talking about is the classical computing term,  Defense in Depth, where we're adding additional security controls or layers to mitigate or lower the impact of vulnerability in our product.  What I'm talking about in my definition, though, what I want to do is I want to avoid the next sales heart bleed at Akamai  and build systems that prevent this or otherwise mitigate the impact of that.  So, I'm going to ask the question, what is heart bleed?  So, open SSL users were affected by heart bleed.  We have a blog post on how Akamai was impacted by this.  In general, this doesn't turn out very well on the slide there, but this is a wire shark output that shows a bad pack payload length.  And what really heart bleed was, was you would request a certain length and you'd say, hey, like, give me five bytes,  and it would return five bytes back to you.  And it was supposed to be the same five bytes that you sent to it.  That's the RFC 6520, which actually defines this, where the receiver must send the corresponding heart bleed response back to the requester.  The implementers neglected to verify that the payload length matches the length in the field itself and would send you back arbitrary memory from the server itself.  So, this allowed you to read memory from the server and then subsequently potentially get access to the TLS private keys,  which we're seeing on that server, or other information for that matter.  Just go louder.  So, what is TLS?  So, TLS is HTTPS.  I would, if you haven't had a chance to, I would really recommend just going to read in the RFC.  It's super approachable as far as RFCs go.  This is straight out of the TLS 1.2 RFC.  And what I'm really going to be talking about today more than anything else is the server key exchange message.  So, the way TLS works in general is you have a, like, a Hello exchange, and then you have a, in blue there, and then in yellow,  you have this Key Exchange period.  And then in red is where you're actually sending your bulk crypto data.  And I'm really going to be zeroing in on the Key Exchange, and in particular the server Key Exchange.  And what that does is proves that the server owns a copy of the certificate by signing certain parameters that get used with the certificate's private key.  So, the server itself has to have, somehow have access to the private key, which traditionally has meant that it's just sitting in memory on the server,  which is why Harpley was so dangerous.  TLS 1.3 makes some slight modifications to this, but that's also something that's worth going and actually reading the RFC.  So, with that in mind, I'm getting all that out of the way, I'm going to be walking us through a TLS security journey.  It's not the most exciting name, but our first stop is we're going to make secrets harder to find in memory.  And to accomplish this, we're actually going to use OpenSSL itself.  And in particular, we're going to use one of OpenSSL, one of Occamized Contribution OpenSSL, which is the OpenSSL secure heap.  And this is just standard Linux memory protection features to get the keys into a more protected area of memory and to prevent them from appearing in core dumps, etc.  And this also applies to other key material, which can be used to derive the private keys as well.  I believe the function name that you're going to need to look for is OpenSSL secure malloc.  And it's also externally visible or externally exposed so that your application can actually make use of this as well.  So, yeah.  So, our second stop is we're going to make the, we're going to move the secrets entirely out of process.  The reason we're doing this is that when you could dump the memory of the process, you could potentially dump the secrets.  So, if they're not in the process memory, you can't access the secrets.  And to do that, we're actually going to use the OpenSSL engine interface.  I posted some example code here.  This is a OpenSSL RSA engine.  And really the important thing here is that RSA underscore method.  And what that will do is it will pass the RSA or what needs to be signed by the server into a function.  And once it's within that function, it's really up to you to make the signature happen and return it back to OpenSSL.  And so this gives you a hook into OpenSSL to accomplish this.  And so using this hook, we're able to go outside of the process, you know, using a, either shared memory or some other inter-process channel to get to the TLS private key that we need.  So what this actually looks like in terms of the protocol, I'm defining a web server here.  And that special RSA sign function that I defined previously.  I'm going to use that to send client random server random and key parameters off, as well as a certificate identifier off to some external key process.  And this external key process is actually going to make the signature and then return it back to the process itself.  So if we had had this in place, when Hartbleed rolled around, we wouldn't have been as vulnerable at least.  The attacker still would have got access to private memory, which could have contained sensitive information, but at least we wouldn't have had to potentially rotate this key.  So what we did here was nothing, or what we did here was nothing particularly special.  So we decided to take this to the next step and we moved the secrets not only out of process, but also out of the machine.  This means that if you had got a, if you have some sort of bad exploit, like a remote code execution, you have some confidence that this key wasn't leaked.  And so what that looks like is we just put the internet between our web server and the external key server.  The protocol is identical. Only probably want to add some additional security to this channel.  And so for some background here, Akamai has, I think we say over 200,000 servers at this point, distributed internationally.  We have a lot of these web servers. So that also means that also implies that we're going to need a lot of these external key servers.  And obviously we know what slow handshakes. This only impacts the initial TLS handshake.  Subsequent ones will use session keys, which allow for faster session resumption.  And also means you don't have to have access to the server private key as well.  But this, this puts this problem squarely into the engineering space and not into security space.  This is a strictly engineering problem. And it's also a performance security trade off here, at least on this initial TLS handshake.  Yeah, it really creates a many, many writing problem, which is something that Akamai is luckily good at solving.  We solve that every day with our end users. So in this case, we're turning that problem on its head and we're having web servers connect to these external key servers.  So this involves a lot of planning. So this, one of my colleagues put this together, I believe, was Richard Wiley.  This is a map, a heat map that shows connectivity between different parts of the world.  It was really fascinating to learn about how poorly connected parts of the world are while doing this.  So in general, you don't want to be crossing between not just national, but in some cases, continent boundaries.  And that create, that will lead to slow connectivity, slow handshakes, and slow performance.  And this is just a visualization of this. We actually use the raw data to figure out where to best put these key servers.  And this was actually in our, included in our state of the internet report in 2017.  So in practice, so this is actually a visual, an internal visualization of our network.  It's a little bit hard to see from further back, but the blue dots on there represent locations.  We have web servers and the red dots represent locations that we actually put key servers, I should say, because each one represents more than one,  or each dot represents more than one machine. And then the yellow and green lines just represent different traffic levels.  And this is actually a real-time dashboard that I pulled this off of. So we're actually visualizing our web servers connecting to our key processors in real-time at scale.  And Europe is honestly the most exciting one. It has the most lines as closely bunched together, simply due to geographical constraints.  Yeah, I've always found the visualization extremely cool.  So that's great that we've moved our secrets now to another machine. Let's get a bit of a paranoid, and we're going to put them into an HSM.  For those of you who are not familiar with an HSM, this would be something like a UB key, for example, where it's a hardware device and hardware security module,  I believe is what it stands for. It's a hardware device that holds your private key, and you're able to set it a request to use that private key,  and it will give you back a signed response of some form. And so to accomplish this, I'm going to use Intel SGX.  And I'm going to explain a lot more about what Intel SGX actually is in subsequent slides.  So this is a little bit paranoid, honestly. This is kind of taking us to the next level as an experiment,  but we were actually able to make this work and deploy it, which is very exciting.  So we put an SGX key store into each of these external key servers.  So what is SGX? So this is an Intel technology. We have it in our certain chips that we deploy in our network.  It's not available in all Intel chips yet. I think they're working on that still.  And most importantly, it's a new set of CPU instructions for handling enclaves.  So enclaves can be thought of as a shared object library, where in the untrusted library,  you make a function call and it drops you into this trusted state where you can't really do a whole lot, to be honest.  You can do memory operations, but you can't ever leave the chip, and anything that leaves the chip itself is either going to be encrypted  or you're not going to have access to. And so what that means is that you can only drop into the state  and come back out of the state, which on a first pass doesn't really seem very useful,  but you can securely get data into this, which means it can function as an HSM if you code it correctly.  So it can be best thought of as a reverse sandbox.  Oncleaves can read and write to external memory of a process, not to any external memory, just of a single process, and applications at any level cannot  read or write memory within that enclave. So it's kind of when you actually call the enclave, it really will then reach out back to you,  fetch the information it needs, do something with it, and then return that data back to you.  And one of the great things about this, about SGX, is it has roughly the same performance as the CPU itself, it's running on the same hardware.  The hardware just simply drops into this SGX state and comes back out of it.  That's the best way to think about this. I'm not the first one to use that term.  It was originally termed going down a 2016 Black Hat talk.  So oncleaves, these are the shared object libraries that I had in a previous slide.  Oncleaves must be signed, and the root of trust is actually traced back to your BIOS.  So if you control your BIOS, you control who can run enclaves on your system.  It really roots the chains of trust to a very low level.  Oncleaves have to be statically compiled. All of the code is signed.  So that kind of implies that SG statically compiled.  It requires BIOS and kernel support just to make sure everything works correctly.  And remote and localization are possible.  And this is one of the cooler features of SGX, and is what allows you to securely get your secrets into these oncleaves  and trust that they're going to not only the right machine, but the right oncleave itself.  So there are a bunch of new instructions, and most of them aren't really that exciting. They're for memory operations and launching and stopping oncleaves.  The important two are going to be eReport and eGekki.  So eReport allows you to tell another oncleave on the system that you're on the same system as it,  and you generate a report, and you sign it in a special way, and then another oncleave can take that data and verify it,  even if it passed through an untrusted boundary.  eGekki, which is a little bit more interesting, allows you to actually generate a unique key within a given oncleave.  And this allows for sealing, which I'll talk about a little bit more later.  But the important thing here is that you can generate a unique key on that machine,  and then subsequently decrypt data based on that key.  I should say that that is actually a symmetric key.  So sgxcelein, so that eGekki parameter allows you to generate a key that's shared between all oncleaves that are on the same system and signed by the same key.  And this key is persistent, so as long as you call this function the same way, you'll always get the same key back from it,  which is awesome, between OS installs, you don't have to maintain any state on that machine.  You can wipe it completely, and the key will still come back the same way the next time.  They also have counters in there to handle if their vulnerability is found in the process,  there are counters in there that will increment, auto-increment, that will allow you to create certain actions based on the key.  That's a little bit too much detail, honestly.  So we can actually use this sgxcelein system.  And per machine, what we ended up doing was we created a public-private key pair within an enclave,  and then we had the private key encrypted using the ceiling key.  And then once it's encrypted using the ceiling key, we can then export this public-key sealed private key,  and that becomes a unique public-private key pair tied to that machine that will survive OS installs.  And so we actually were able to use this to do key provisioning at scale.  This is across a large number of machines where we had each one of these sgxcelein keystores issue this sealed private key, public key combination.  We would then, we have been a massive advantage because we control our machine deployment and our pipeline to get machines deployed.  And so we were able to export that key.  We then were able to encrypt a key to that public key and then send that and a bunch of other data back down to the enclave.  And assuming that this whole process works, only that enclave will actually be able to decrypt the secrets that are encrypted to that public key.  I should say this key pair is an RSA key pair, which allows you to encrypt things to the public key such that they can only be decrypted using the private key.  Yeah, so at a high level, that's kind of how that all works.  So there are a bunch of complexity with this sgxcelein key that I didn't get into.  You will spend all of your time worrying about how to rotate every single key you create.  Initial trust establishment is a little complicated, which I described previously,  especially if you don't control your machine build process or pipeline.  Secure handling of multiple enclaves, keys, and most importantly, replay attack prevention are all things that I didn't talk about,  but you should worry about if you're actually deploying this.  You will spend most of your time trying to figure out how this should work and handle key rotation,  handle all these really complex cases, and not a whole lot of time writing code, which to some extent is a good thing.  So, yeah, so I should add all of this was deployed within Akamai.  If you connect to Akamai, there's a chance that your TLS handshake is actually at some point actually traversing a sgxonclave,  which I think is pretty cool. So yeah, thank you.  applause
DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_0_143.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_144_191.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_192_239.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_240_923.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_924_1331.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_1332_1679.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_1680_2195.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_2196_2519.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_2520_3107.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_3108_4451.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_4452_6239.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_6240_6599.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_6600_7619.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_7620_8039.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_8040_9347.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_9348_10415.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_10416_10907.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_10908_12899.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_12900_14051.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_14052_15263.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_15264_16151.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_16152_16679.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_16680_18203.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_18204_19391.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_19392_20759.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_20760_22031.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_22032_22247.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_22248_24179.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_24180_25715.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_25716_26495.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_26496_26855.png DEF_CON_26_PACKET_HACKING_VILLAGE_-_Sam_Erb_-_Defense_in_Depth_The_Path_to_SGX_at_Akamai_gU6Zehq-eXM_26856_27377.png
It is absolutely my pleasure to introduce to you from the Council of Nine. Sam Herb. Hello. So, this is going to be a bit of a different talk. It's definitely more blue team than most of what's normally presented at Defcon. So, apologize for that in advance, but it's something that I was, I really wanted to give a presentation on. I think it's important to share with the wider community what we're doing at Akamai. So, with that in mind, this is Defense in Depth, the path to Shaxx at Akamai. So, who am I? I am a software engineer at Akamai. At Defcon, I have won two black badges in a badge challenge with the Council of Nine. Yeah, if you want to reach me, probably the best way right now is to find me on Twitter. So, disclaimer, generic disclaimer. Don't poke and prod at Akamai, please. And everything presented here is the work of many individuals, some of whom are actually in the room today. Which is exciting. This wasn't just me. So, get started. Defense in Depth. I'm talking about Defense in Depth, and what I'm really talking about is the classical computing term, Defense in Depth, where we're adding additional security controls or layers to mitigate or lower the impact of vulnerability in our product. What I'm talking about in my definition, though, what I want to do is I want to avoid the next sales heart bleed at Akamai and build systems that prevent this or otherwise mitigate the impact of that. So, I'm going to ask the question, what is heart bleed? So, open SSL users were affected by heart bleed. We have a blog post on how Akamai was impacted by this. In general, this doesn't turn out very well on the slide there, but this is a wire shark output that shows a bad pack payload length. And what really heart bleed was, was you would request a certain length and you'd say, hey, like, give me five bytes, and it would return five bytes back to you. And it was supposed to be the same five bytes that you sent to it. That's the RFC 6520, which actually defines this, where the receiver must send the corresponding heart bleed response back to the requester. The implementers neglected to verify that the payload length matches the length in the field itself and would send you back arbitrary memory from the server itself. So, this allowed you to read memory from the server and then subsequently potentially get access to the TLS private keys, which we're seeing on that server, or other information for that matter. Just go louder. So, what is TLS? So, TLS is HTTPS. I would, if you haven't had a chance to, I would really recommend just going to read in the RFC. It's super approachable as far as RFCs go. This is straight out of the TLS 1.2 RFC. And what I'm really going to be talking about today more than anything else is the server key exchange message. So, the way TLS works in general is you have a, like, a Hello exchange, and then you have a, in blue there, and then in yellow, you have this Key Exchange period. And then in red is where you're actually sending your bulk crypto data. And I'm really going to be zeroing in on the Key Exchange, and in particular the server Key Exchange. And what that does is proves that the server owns a copy of the certificate by signing certain parameters that get used with the certificate's private key. So, the server itself has to have, somehow have access to the private key, which traditionally has meant that it's just sitting in memory on the server, which is why Harpley was so dangerous. TLS 1.3 makes some slight modifications to this, but that's also something that's worth going and actually reading the RFC. So, with that in mind, I'm getting all that out of the way, I'm going to be walking us through a TLS security journey. It's not the most exciting name, but our first stop is we're going to make secrets harder to find in memory. And to accomplish this, we're actually going to use OpenSSL itself. And in particular, we're going to use one of OpenSSL, one of Occamized Contribution OpenSSL, which is the OpenSSL secure heap. And this is just standard Linux memory protection features to get the keys into a more protected area of memory and to prevent them from appearing in core dumps, etc. And this also applies to other key material, which can be used to derive the private keys as well. I believe the function name that you're going to need to look for is OpenSSL secure malloc. And it's also externally visible or externally exposed so that your application can actually make use of this as well. So, yeah. So, our second stop is we're going to make the, we're going to move the secrets entirely out of process. The reason we're doing this is that when you could dump the memory of the process, you could potentially dump the secrets. So, if they're not in the process memory, you can't access the secrets. And to do that, we're actually going to use the OpenSSL engine interface. I posted some example code here. This is a OpenSSL RSA engine. And really the important thing here is that RSA underscore method. And what that will do is it will pass the RSA or what needs to be signed by the server into a function. And once it's within that function, it's really up to you to make the signature happen and return it back to OpenSSL. And so this gives you a hook into OpenSSL to accomplish this. And so using this hook, we're able to go outside of the process, you know, using a, either shared memory or some other inter-process channel to get to the TLS private key that we need. So what this actually looks like in terms of the protocol, I'm defining a web server here. And that special RSA sign function that I defined previously. I'm going to use that to send client random server random and key parameters off, as well as a certificate identifier off to some external key process. And this external key process is actually going to make the signature and then return it back to the process itself. So if we had had this in place, when Hartbleed rolled around, we wouldn't have been as vulnerable at least. The attacker still would have got access to private memory, which could have contained sensitive information, but at least we wouldn't have had to potentially rotate this key. So what we did here was nothing, or what we did here was nothing particularly special. So we decided to take this to the next step and we moved the secrets not only out of process, but also out of the machine. This means that if you had got a, if you have some sort of bad exploit, like a remote code execution, you have some confidence that this key wasn't leaked. And so what that looks like is we just put the internet between our web server and the external key server. The protocol is identical. Only probably want to add some additional security to this channel. And so for some background here, Akamai has, I think we say over 200,000 servers at this point, distributed internationally. We have a lot of these web servers. So that also means that also implies that we're going to need a lot of these external key servers. And obviously we know what slow handshakes. This only impacts the initial TLS handshake. Subsequent ones will use session keys, which allow for faster session resumption. And also means you don't have to have access to the server private key as well. But this, this puts this problem squarely into the engineering space and not into security space. This is a strictly engineering problem. And it's also a performance security trade off here, at least on this initial TLS handshake. Yeah, it really creates a many, many writing problem, which is something that Akamai is luckily good at solving. We solve that every day with our end users. So in this case, we're turning that problem on its head and we're having web servers connect to these external key servers. So this involves a lot of planning. So this, one of my colleagues put this together, I believe, was Richard Wiley. This is a map, a heat map that shows connectivity between different parts of the world. It was really fascinating to learn about how poorly connected parts of the world are while doing this. So in general, you don't want to be crossing between not just national, but in some cases, continent boundaries. And that create, that will lead to slow connectivity, slow handshakes, and slow performance. And this is just a visualization of this. We actually use the raw data to figure out where to best put these key servers. And this was actually in our, included in our state of the internet report in 2017. So in practice, so this is actually a visual, an internal visualization of our network. It's a little bit hard to see from further back, but the blue dots on there represent locations. We have web servers and the red dots represent locations that we actually put key servers, I should say, because each one represents more than one, or each dot represents more than one machine. And then the yellow and green lines just represent different traffic levels. And this is actually a real-time dashboard that I pulled this off of. So we're actually visualizing our web servers connecting to our key processors in real-time at scale. And Europe is honestly the most exciting one. It has the most lines as closely bunched together, simply due to geographical constraints. Yeah, I've always found the visualization extremely cool. So that's great that we've moved our secrets now to another machine. Let's get a bit of a paranoid, and we're going to put them into an HSM. For those of you who are not familiar with an HSM, this would be something like a UB key, for example, where it's a hardware device and hardware security module, I believe is what it stands for. It's a hardware device that holds your private key, and you're able to set it a request to use that private key, and it will give you back a signed response of some form. And so to accomplish this, I'm going to use Intel SGX. And I'm going to explain a lot more about what Intel SGX actually is in subsequent slides. So this is a little bit paranoid, honestly. This is kind of taking us to the next level as an experiment, but we were actually able to make this work and deploy it, which is very exciting. So we put an SGX key store into each of these external key servers. So what is SGX? So this is an Intel technology. We have it in our certain chips that we deploy in our network. It's not available in all Intel chips yet. I think they're working on that still. And most importantly, it's a new set of CPU instructions for handling enclaves. So enclaves can be thought of as a shared object library, where in the untrusted library, you make a function call and it drops you into this trusted state where you can't really do a whole lot, to be honest. You can do memory operations, but you can't ever leave the chip, and anything that leaves the chip itself is either going to be encrypted or you're not going to have access to. And so what that means is that you can only drop into the state and come back out of the state, which on a first pass doesn't really seem very useful, but you can securely get data into this, which means it can function as an HSM if you code it correctly. So it can be best thought of as a reverse sandbox. Oncleaves can read and write to external memory of a process, not to any external memory, just of a single process, and applications at any level cannot read or write memory within that enclave. So it's kind of when you actually call the enclave, it really will then reach out back to you, fetch the information it needs, do something with it, and then return that data back to you. And one of the great things about this, about SGX, is it has roughly the same performance as the CPU itself, it's running on the same hardware. The hardware just simply drops into this SGX state and comes back out of it. That's the best way to think about this. I'm not the first one to use that term. It was originally termed going down a 2016 Black Hat talk. So oncleaves, these are the shared object libraries that I had in a previous slide. Oncleaves must be signed, and the root of trust is actually traced back to your BIOS. So if you control your BIOS, you control who can run enclaves on your system. It really roots the chains of trust to a very low level. Oncleaves have to be statically compiled. All of the code is signed. So that kind of implies that SG statically compiled. It requires BIOS and kernel support just to make sure everything works correctly. And remote and localization are possible. And this is one of the cooler features of SGX, and is what allows you to securely get your secrets into these oncleaves and trust that they're going to not only the right machine, but the right oncleave itself. So there are a bunch of new instructions, and most of them aren't really that exciting. They're for memory operations and launching and stopping oncleaves. The important two are going to be eReport and eGekki. So eReport allows you to tell another oncleave on the system that you're on the same system as it, and you generate a report, and you sign it in a special way, and then another oncleave can take that data and verify it, even if it passed through an untrusted boundary. eGekki, which is a little bit more interesting, allows you to actually generate a unique key within a given oncleave. And this allows for sealing, which I'll talk about a little bit more later. But the important thing here is that you can generate a unique key on that machine, and then subsequently decrypt data based on that key. I should say that that is actually a symmetric key. So sgxcelein, so that eGekki parameter allows you to generate a key that's shared between all oncleaves that are on the same system and signed by the same key. And this key is persistent, so as long as you call this function the same way, you'll always get the same key back from it, which is awesome, between OS installs, you don't have to maintain any state on that machine. You can wipe it completely, and the key will still come back the same way the next time. They also have counters in there to handle if their vulnerability is found in the process, there are counters in there that will increment, auto-increment, that will allow you to create certain actions based on the key. That's a little bit too much detail, honestly. So we can actually use this sgxcelein system. And per machine, what we ended up doing was we created a public-private key pair within an enclave, and then we had the private key encrypted using the ceiling key. And then once it's encrypted using the ceiling key, we can then export this public-key sealed private key, and that becomes a unique public-private key pair tied to that machine that will survive OS installs. And so we actually were able to use this to do key provisioning at scale. This is across a large number of machines where we had each one of these sgxcelein keystores issue this sealed private key, public key combination. We would then, we have been a massive advantage because we control our machine deployment and our pipeline to get machines deployed. And so we were able to export that key. We then were able to encrypt a key to that public key and then send that and a bunch of other data back down to the enclave. And assuming that this whole process works, only that enclave will actually be able to decrypt the secrets that are encrypted to that public key. I should say this key pair is an RSA key pair, which allows you to encrypt things to the public key such that they can only be decrypted using the private key. Yeah, so at a high level, that's kind of how that all works. So there are a bunch of complexity with this sgxcelein key that I didn't get into. You will spend all of your time worrying about how to rotate every single key you create. Initial trust establishment is a little complicated, which I described previously, especially if you don't control your machine build process or pipeline. Secure handling of multiple enclaves, keys, and most importantly, replay attack prevention are all things that I didn't talk about, but you should worry about if you're actually deploying this. You will spend most of your time trying to figure out how this should work and handle key rotation, handle all these really complex cases, and not a whole lot of time writing code, which to some extent is a good thing. So, yeah, so I should add all of this was deployed within Akamai. If you connect to Akamai, there's a chance that your TLS handshake is actually at some point actually traversing a sgxonclave, which I think is pretty cool. So yeah, thank you. applause
blog post on how Akamai was impacted by this. In message. So, the way TLS works in general is you take this to the next step and we moved the Intel SGX actually is in subsequent slides. So chips that we deploy in our network. It's not this to do key provisioning at scale. This is works, only that enclave will actually be able to