Linux Containers: Future or Fantasy?
Grattafiori, Aaron
Linux Containers: Future or Fantasy? Aaron Grattafiori Principal Security Consultant, iSEC Partners/NCC Group Containers, a pinnacle of fast and secure deployment or a panacea of false security? In recent years Linux containers have developed from an insecure and loose collection of Linux kernel namespaces to a production-ready OS virtualization stack. In this talk, the audience will first learn the basics of how containers function, understanding namespaces, capabilities and cgroups in order to see how Linux containers and the supporting kernel features can offer an effective application and system sandboxing solution yet to be widely deployed or adopted. Understanding LXC or Docker use, weaknesses and security for PaaS and application sandboxing is only the beginning. Leveraging container technologies is rapidly becoming popular within the modern PaaS and devops world but little has been publicly discussed in terms of actual security risks or guarantees. Understanding prior container vulnerabilities or escapes, and current risks or pitfalls in major public platforms will be explored in this talk. I'll cover methods to harden containers against future attacks and common mistakes to avoid when using systems such as LXC and Docker. This will also include an analysis and discussion of techniques such as Linux kernel hardening, reduced capabilities, Mandatory Access Controls (MAC), the User kernel namespace and seccomp-bpf (syscall filtering); all of which help actually contain containers. The talk will end on some methods for creating minimal, highly-secure containers and end on where containers are going and why they might show up where you least expect them. Aaron Grattafiori (@dyn___) is a Principal Security Consultant and Research Lead with iSEC Partners/NCC Group. A jack-of-all-security, Aaron leads projects dealing with complex system analysis, mobile and web application security to network, protocol, and design reviews to red teams and other hybrid testing. With over nine years of security experience, Aaron utilizes a wide array of technology skills, historical research and security knowledge to consistently discover critical vulnerabilities. Aaron has spoke on a wide range of topics at security conferences such as Blackhat, DEF CON Kids, Toorcon:Seattle+SanDiego, ToorCamp, Source Seattle, EELive! and SecureWorld in addition to being a guest speaker at Stanford University. Prior to working at iSEC Partners, Aaron worked as a Security Consultant for Security Innovation and is a retired long time member of the Neg9 CTF team. This will be Aaron's 12th DEF CON, w00t! Twitter: @dyn___
 Welcome everybody. Everybody will come in and get us sat down. Erin is here to talk to us about Linux containers. Thanks. Yeah, so I'm Erin. I've been doing security a while. ISEC, if you don't know who NCC group is, ISEC, Montesano came together. We were brought by NCC group from the UK. But same people, same place. I've given some talks before on some different things. So this is my first DEF CON talk, but it is my 12th DEF CON. And this is my 10 second promotional motivational slide to not wait as long as I did and not be a slacker. And if you're, you know, you've got an idea and you know, we've got lots of inspirational talks and cool talks here, you know, keep that going and do research and submit to the CFP and give talks. And don't wait as long as I did. So I want to start with this story. So Bob, this is Bob here. He's got some web app or he's got something that's on the internet. It's on Linux. And somehow there's a bug. He gets popped. There's some kind of RCE. He gets arbitrary execution, something like that. And he wants to add some security, but he's not really sure. So somebody says, oh, you should use CH root. That's legit. You know, open SSH uses it. It must be good, right? But as we all know, it's broken if you have root. You essentially just CH root yourself. And then you can go up to the root. And yeah. So that's well, well known. So somebody else comes along and says, oh, you should use SE Linux, right? That's easy. The NSA made it. It must be secure. It's not in. It's a, you know, if you know anything about SE Linux, it's a super complicated type based system. It's really designed for multi-level security. It's really designed for kind of government things when you've got lots of different things to classify. It's got really good support on Red Hat, but other than that, it's not so great. Some of those reasons, right, complexity, it's really complicated to actually create it. It has the Linux Torvodes problem, which most of the other Linux security modules have where, you know, he doesn't really believe in any of them. The Set-in-4-0 problem where, you know, if you walk into a box that has SE Linux, the odds of it being disabled are high. And, you know, at the end of the day, the kernel's got a kernel, right? So if there's a kernel vulnerability and SE Linux is being enforced by the kernel, it just gets turned off. If you ever look at any of Spender's exploits, he loves to just do that first and then dig at them as he does. But speaking of Spender, so somebody else comes along to this, to Bob and says, no, you should use GR Security. That's the best. And yeah, it really is in a lot of ways. But that's not really solving some of the problem, right? If he has WordPress and he's got some arbitrary, you know, command injection, GR Security is going to do nothing for that, right? You're doing a lot of memory corruption protection. You're doing a lot of other things. But unless you do all the kind of turn everything to 11 with RBAC and everything, it's not going to solve that. So then somebody else says, no, you should use, just use VMs. You know, that's how to be ultra secure. But at the end of the day, you know, VMs aren't perfect either and you've got a lot of other problems with configuration management and everything to do there. So the second story is Glenn. He knows a lot about security. He's his crypto cat. And so he talks to this potential source and he's really, really paranoid. He wants to run Linux because, you know, he doesn't trust OS X. He doesn't take that seriously and, you know, malware is on Windows or whatever. And you know, all the cool kids run Linux and it's got to be super secure, right? But not really. I mean, he's one Tor browser bug exploited away from having his calculator popped or whatever may happen. You know, live purple track record is an awesome. Last time I updated my live CD, that didn't go so well. So the third story is Margaret, right? Margaret works at an IOT company. She is tired of getting these, everything running as root and everything getting popped and not having to be able to see any kind of security on this little platform. Then I actually inserted this slide after I saw Charlie and Chris's talk where, you know, maybe Margaret also works on the GBU Connect team and doesn't want to run D bus as root and have just come, you know, no ability to control things. But really, you know, a lot of that's done just because money obviously takes time to add security. There's no way to virtualize something that's on ARM or MIPS or this little embedded thing that's running Linux. A lot of these other things are too heavy duty solutions. There's got to be something we can do. And really, these stories have in common attack services, right? You know, that's the worst attack service possible there at the bottom. But really, you know, attack service is really important. And sandboxes and containers, it would be great to see them being the norm, not the exception. If you look at Chrome and Adobe Reader X, seat belt, you know, any of the kind of modern sandboxing environments, that really should be everywhere. There's a battle that we're fighting with native code and you're going to lose, right? And so until we can win and everything's rewritten in some crazy rust or whatever it may be, we have to cut our losses and we can do that with containers, you can do that with sandboxing, you can do that with other isolation. So for Linux, you know, open source loves to reinvent the wheel. But how I'd like to kind of set this up is, you know, being in Vegas, one of my favorite movies, so there's this movie, Moonbox Aintz, where, you know, they come out of this house and they're staring at six guys with guns drawn. And, you know, it's a firefight to paraphrase. And there's this kind of bumbling investigator that says, you know, what if it wasn't six guys with six guns, but one guy with six guns? And, you know, he's kind of an idiot, so William DeFoe is William DeFoe. And he says, you know, shut up, you don't know what you're talking about. But really Greenlee was right. It was one guy with six guns. So, you know, the person is proposed, what if it wasn't a whole bunch of kernels, but one kernel with multiple user lands? And that's really the core idea of containers, right? So this also is not a new idea. This has been done a long time before Docker and everything came along. So before we keep going, I just want to say that this is, if you are really scared about security or incredibly paranoid and you have something, you know, ridiculous system, you do not want to rely purely on any kind of containerization solution. You don't want to rely purely on any virtualization solution. You know, you need guys with guns and fences and, you know, air gaps, actual air gaps, not things connected by wires. And really, you know, I'm not up here to say that containers can do everything. But what I am here to say is that you don't want to depend on any one single method. And containers are getting a lot easier, they're getting a lot more powerful. And if you can do anything, you might as well just add things in layers, as we've seen. So to get to how this actually works, so the first major area is namespaces. So within the kernel, just like a lot of any other cool stuff on Unix, it really started in plan 9 where they had the idea of namespaces in the first place. But the Linux kernel is split up into these kind of five namespaces, network, mount, user, UTS, which is host name and pit and pit. And the way you get into one of these namespaces is when you clone, when you process clones and spawn a child, it's kind of the new fancy fork, the kind of phrase that you get set up into is you enter a new kernel execution context. And the way you do that is by when you run, when you call clone, you add these special flags at the end, depending on the namespaces that you want to enter. And so the mount namespace was added first a long time ago. It basically lets your process have a per process view of what the file system is. IPC, we can keep going through these. Your processes can have their own host name, they can have their own view of the other processes, and they can be nested and there's some cool tricks you can do. If you look at how this actually works, if you create a container and you run PS in it, you pretty much just see the processes that are in your container. Which leads me to the awesome snake oil solution. You can't hack what you can't see. So the new, there's also a network namespace that isolates your IP, your firewall, your routing table, all that kind of thing. And the user namespace, which is the newest one, which was added, that is really important for securing your containers and essentially lets you be root inside of a container, but that's still treated as a low rights user in the context of the kernel outside of the container. Which obviously is also a high risk area of the container. You're really controlling a lot of how the UID system works within the kernel in very sensitive areas. And so there have been a few vulnerabilities of using the user namespace to break things outside the context of containers. The way that it looks is, you know, if you attach to our food container, if you, your root you can see inside, you run sleep and then outside of it you're actually UID 100,000. Essentially just shifts all your UIDs up by some offset. So the other major area of containers is capabilities. The idea of capabilities is you take the user root, right, who can do everything and you blow that up into a whole bunch of little pieces. You know, whether you need to bind to something lower than 1024, whether you need to be able to run ptrace, whether you need to be able to, you know, change your network settings. And capabilities are great, right? That lets us get rid of this just kind of god mode and split it up into a whole bunch of little pieces that really can do, you know, only what you want that process to be able to do with root. The bad thing is that they started lumping them together a bit in weird ways. And so you end up with kind of a capabilities model that somewhat works but is confusing, hard and can be messed up. So some of the, you know, kind of dangerous capabilities or capabilities that you might commonly encounter are things like being able to receive raw packets or bind to something lower than 1024 or change resource controls or send, kill. So everyone always asks, you know, what capabilities should be dropped? You know, should I drop this or that? And really you want to drop all of them. You want to set up everything for your container or your namespace ahead of time and then when you pivot it, you want to throw away all your capabilities and just live in your little cocoon and you can't do anything at all. And then you get questions on forums or even just people as implementing this, you're thinking like, what if I just leave this one capability enabled? What happens? And the answer is it depends. So if we go and look at an example of this, so we look at ping, ping is set UID root. Everybody from forever ago remembers that, you know, like CD record was set UID root and you could pop shells on Linux and do privsk by doing, you know, any kind of set UID root binary with a vulnerability, you get instant root. So it's kind of ridiculous that ping, you know, on these days still has root and set UID root. Obviously the attack surface is ping and has been pounded on a bunch. If you copy ping to somewhere else, you'll lose your set UID root bit. And if you try to run that ping, you'll get operation not permitted because you need to be able to create raw sockets for ping or ICMP. So the way to fix that is you can set a capability of cap net raw on that new elf. If we, you know, look at that, we can see that we have cap net raw and then when you try to run it, even as a low rights user, now I can ping, but that's the only thing I can do. That's the only capability that I have. So obviously there's a lot of dangerous capabilities that you don't want to give things. Being able to road a kernel module, that could be bad. Being able to override, you know, just criss-ary access controls or turn off the manager access control system, you know, things like that, that's definitely things you don't want to have. The other bad thing is there's a capability cap sys admin. And obviously that sounds pretty important. It's basically root. There's a whole lot of things that cap sys admin can do. There's also a great post by Radspender that goes into all the detail and all the different capabilities and how they can be used to get root or do bad things. And that's the link to it. And so the other major aspect of containers is control groups. The basic idea of those is they're hierarchical and inheritable system for controlling resources across a set of processes. And that can be devices or CPU usage or physical CPU, memory amounts, IO rates of a certain device, network. And it's really you limit on serial rates if you want to think about it that way. And it's mainly used to kind of fill the gaps of some namespaces. So right now there is no dev namespace or there's no namespace. So you kind of have to get around that a little bit by using C groups in a way that someone intended to do that, but not really. They're typically controlled through a proc file system, you know, pseudo-it. So it's just a directory and you put create directories and this corresponds to PIDs and things will read that directory. The good thing is obviously that it's a file system based and everything on Unix is a file that used to be at least following that model. But at the end of the day you can do tricky over mount attacks and other things that is a side effect of having it be a file. So it can be controlled through CGA Jack, CGA manager, but really most of the container platforms will abstract this away. So when you put all that together, namespaces isolate the elements of the kernel, the capabilities, help enforce those namespaces and limit the capabilities within that container and then the C groups limit the access. So really those three elements along with some other, you know, secrets, you know, magic sauce will create containers and it is better than C H group, right? There's a lot of background to those. You get special mount options, you know, you can do things like an overlay file system where, you know, part of the directory is shown to the container and part of it is shown to the host and there's kind of some go in between. You can do crazy stuff with SSHFS now. So where are Linux capability or, you know, where are containers being used now on servers? So a lot of platform as a service systems, you know, EC2, you can do Docker, Google App Engine uses containers, Heroku has been doing it for way before anybody did and they, you know, have Rackspace, everybody probably more than I don't have on that list. It's also being used in clients. So, you know, Chrome OS is a huge user of these technologies. They've done a lot of work hard, hard work on making, you know, awesome sandboxing in Chrome and Chrome OS. It's somewhat used in Android. It's somewhat used on some Linux distributions. There's also some cool sandboxing tools if you're into those that use these without kind of going to full blown containers. So the kind of major one that started on the kind of path of Docker and the current trend was LXC. So LXC is kind of vanilla and I won't go into the details of this but essentially it's controlled through a template, you know, you set various options, you know, what devices you want to have, some security different, some security things like Set Comp and App Armor or, you know, what capabilities you want to keep or drop. There's also been some recent advancement and I say recent within like the last year or two of containers. So, unpriloged containers, I mentioned that the user namespace, so that really creates the ability for a non-privileged user to create a new container that they can then be root in but is all treated as a non-root user when it comes down to it. Obviously there's some weird things that can happen with that situation so there's more work and auditing needs to be done there but that'll be, you know, a futuristic thing where, you know, a low rights user that doesn't have root and never will get you root will be able to create containers and do things. But you have this problem, right? We talked about attack surface. So, SysCalls, right, are one of the main entry points to the kernel. There was a lot back in 2.2. Now there's a whole lot more. What are all those and does your app really need all those, you know, 300 of some odd SysCalls? Probably not. So then this thing came along called Set Comp and the guys at Google Chrome team are doing a ton of awesome work to really push the boundaries of what you can do here in a very good way. This has been tried before but the way they're doing it is much better than some of the methods that have been tried in the past. But it's important to point out that Set Comp by itself is not a sandbox. It's purely made to just limit the attack surface to things that your app really doesn't need to have. So you can also, not only can you filter on certain SysCalls like this process, it's just crunching numbers. So it just needs to call, you know, return and open and that's it. But you can actually control the arguments to those SysCalls and say, you know, it only needs to open this file or it needs to, it can only do these flags on this file. And so that's pretty powerful, right? You do encounter a performance hit. So when you've got a lot of filters, there will be some impact. I think that we're still kind of figuring out what that is and how to optimize it. But it definitely is something to consider if you're going to be rolling it out. And right now you will need LXC or use MiniJail directly or use Mbox or something like that. You can't, if you're going to use Docker with Set Comp, you have to use the slash code and tribb or wait till 1.8, which I think is when they're going to actually launch Set Comp support. But LXC has it right now and you can use it. It's typically done through the PR control. You basically can set one of two flags, the strict mode, which I think is only four or five SysCalls, which should be used for just like a, you know, once you get your process into a spot where it's just going to be crunching numbers or doing something very basic, you can put that on there. Or you can filter and by filter, I mean a Berkeley facet filter, which everybody remembers from TCP dump. This is how you would use Set Comp in an actual program directly. And Berkeley facet filter is pretty cool. You can actually use TCP dump to dump out some of the language that I don't really know how it works underneath, but it's cool that you can see it. So where is Set Comp being used right now? Lots of places. You can turn it on in lots of places at least. It's not necessarily on by default. It is definitely in Chrome and they've been doing a lot of good work. It's also in Tor, but it's not on by default. So containers, you know, who's doing these, right? So Docker, obviously, CoreOS, they have Rocket. There's lots of interesting other companies that are either doing these in the background or they're offering them as a service or they're going to be offering this as a service. You know, big shout out actually to theSansProm.io guys. They have a crazy, like, customized system and they do GRSEC for everything. And they're super serious about security and everything else. And also the RancherOS is an interesting idea where everything in that kind of minimal system runs in a container. It's also funny. I don't know if this has been fixed or not, but when you would Google that, it would actually bring up Wave with RancherOS. Google would fix it. So yeah, they might have some SEO problems. But we're really here to talk about the big two, right? So Docker. When I usually mention Docker, this is the response. I'm from Silicon Valley. So, you know, this is usually the response I get from some devs. It's really packaging and kind of development focused. But one thing that is really good about that, I think, is that it's also very app focused. So the kind of the philosophy of Docker is that it's not just a container with, you know, some things in it. It's ideally one process in that container and that's it. And that, you know, as a security person, I really like that. Because then not only is it that one process in that container, but it's that one process with just the libraries that it needs and just the files it needs and nothing else. And so you really are cutting off a lot of attack service there. And the big deal about Docker and why it's gotten so popular is that it really makes containers easy, right? LXC was kind of doing it, you know, in kind of the harder mode. Whereas Docker is, you know, you just do Docker pull and magic happens. But Docker isn't just containers, right? So there's all these other things that they've done. It used to be based off of LXC. Now they have something called libcontainers written in go. They also have libchan and libswarm and all these other things. They have a REST API. It's by default, it uses a Unix socket, but you can turn on the REST API. It also, one of the kind of downsides of it that they're working on fixing, I think, is related to the fact that, you know, there's this root daemon that runs and everything, all your containers and everything from that goes under that daemon. And runs, that runs as root. You have to be root to interact with it, which then makes people run as root on the container host. And, you know, that's never a good thing. So, you know, you can pull down your images. You can map things from the host into your container. There's also this idea of Docker hub, which is kind of like GitHub for containers. You know, you can do commits and all this stuff. And there's all the, you know, orchestration, communication and management. I don't think the word orchestration was really mentioned much at all in the last, you know, hundred years, except for maybe last year. Anyway, so CoreOS is the other big one. It's really started out as a minimal OS for hosting containers. They're launching Rocket in this app container spec. They're trying to kind of distance themselves from LXC and Docker. They're kind of main ideas that they're trying to be more secure and lean and be what Docker should be or used to be. And, but, you know, why would anybody use any of these systems? And it's mostly just to make it easy, right? Or to package a lot of the functionality with it. And one way that you might kind of relate to it is, you know, you can run for EBSD and you can secure that, but you kind of need to know what you're doing. Or you can run OS 10 and it's easy and it's kind of BSD still. But you kind of have to take some things, you know, for granted about control. And so, you know, LXC, it's kind of vanilla, it lets you do whatever you want and you do get some bleeding edge there. But you kind of have to know what you're doing in order to do everything correctly, especially based on the defaults. Whereas Docker, you know, they do a lot of stuff for you. They drop most of the capabilities that you don't need, but they still leave some around and you need to kind of explicitly disable those if your app doesn't need them. And we'll get into that. So, you know, LXC, hard mode, but you got flexibility. Docker, easy, but you know, you have some costs or some risks. CoreOS, it's new. It's very, very new. We'll get to that. So, but we're here to talk about attacks, right? That's all just stuff. So if you think about this, there's a lot of different ways that we can go about thinking about what are the attacks in a container or what should I be worried about? So there's containers to other containers. There's things in the container against itself essentially, trying to elevate your privileges within a container if you're not just a single process. Container to the container host, things supporting infrastructure, whether that's orchestration and all that other stuff or whether that's other things on the network. And you can go from there. So kind of starting at the top, the kernel obviously is doing all this virtualization stuff. So in the kernel, you've got all these old drivers. You've got all this old code. You've got weird file system support. Pretty much nobody compiles a kernel from scratch anymore that doesn't have all this bloaty stuff that you probably don't even need. Which means you're also inheriting all those vulnerabilities of some random can bus, you know, special can bus driver thing in Linux that you don't use on your server. The other way that this kind of typically goes bad is not dropping capabilities. So CapNet Admin is kind of a bad offender. There's been, I don't know how many different CVEs were of documentation and vulnerabilities where you can go from CapNet Admin to something else that you shouldn't be doing that isn't related to networking. The way that happened in a lot of cases is being able to force CapNet Admin to load an arbitrary kernel module that the youth then control. That was, I think, one of the worst cases. But, you know, speaking of dropping capabilities, Docker was really stung by this. So they kept around for a long time the CapDac read search capability. And if you read the man page and maybe this is what they did, you know, it says something like, you know, you can read some files related to, you know, permissions. You can also invoke open handle at, which is a syscall that doesn't really have a ton of documentation. They probably thought, oh, whatever. But it turns out that if you actually use that syscall and you do some other tricks based on iNodes and other things, you can essentially brute force the iNode of EtsyShadow in the host and read it from inside the container. And obviously there's other attacks you can do related to that. But, you know, that was something that really gave the capability model a good example of why you need to drop the ones that you don't really need because they're complicated. And, you know, if you don't drop those capabilities, you have to rely on some other thing to, you know, enforce the rules on your container. And, you know, that's typically done through a manager access control system like SC Linux or AppArmor. But if you don't use that, the capabilities are your only line of defense. Because if you're root in a container, you're still root on a host. Unless you're using the user namespace, which again doesn't have a ton of support yet or adoption. You also have the problem of limiting access. So we mentioned how the kernel, there's not a namespace for PROC or there's not a namespace for dev. And so what happens is you have things in the kernel or things in the container that are exposed from the host or the bridge of the host and the container. So things like PROC-K core where you can expose the kernel memory into the container. You have things like, you know, sysrec, which everyone forgets about, but stung, you know, LXC for a number of years where they, you know, were leaving that open. You also have things like being able to make new devices or even just see D message, right? That's not a ton of attack surface there, but, you know, you're leaking things like maybe firewall rules or other stuff. And then unattended access to the network, right? So a lot of these container systems use a bridge system by default. If you remember networking 101, a bridge is a switch. So most of the default systems by, I think, almost all the containers will create a bridge and let your containers talk to each other. And that isn't something that most people realize when they deploy five containers on a host. They don't think that all those containers will be able to communicate, but they can unless you've written firewall rules or switched to some other networking type. You also, you know, see people not limiting any resources. Obviously, if you fork bomb a container with my favorite bash function there, you're going to be fork bombing the host. And then kind of some other things relating to, you know, Docker has been stung by this a couple times of, you know, this idea of taking your pack, taking your application, packaging it all up with the libraries that you need, putting a lock on it and shipping it off everywhere. You know, that is, doesn't really happen. What happens when there's a vulnerability in one of those libraries or that application and you need to update it, what do you do? You know, you can't just, you're not supposed to run app get upgrade on your containers. It's kind of a no-no in the container world because they're supposed to be ephemeral and immutable. And so what do you do? And so you kind of need to define, to have a system to adopt that, have a patching system. And then the lack of mandatory access control. So most of the container systems also have some form of app armor support or SE linux support that works quite well. Obviously it's not perfect. There's been some bugs even very recently that were published there. But at the end of the day, you know, the kernel is still enforcing things. So this isn't a perfect solution either. And really it should just be one facet of many different things. And we mentioned that the networking defaults and how that can be bad. For LXC specifically, the defaults are quite bad for dropping capabilities. But that kind of relates to the way that the model of LXC is, you have a minimal container, but it's not supposed to really just be a single process. It's supposed to be, you know, a few different things. Maybe an SSH daemon or system D or something like that. Whereas Docker is the idea of one process. But speaking of Docker, so Docker does a pretty good job of dropping capabilities. But if you read the between the lines, they mention that they drop all the capabilities except for those required. But if they don't know what you're going to be running, how do they know the ones that are still required? And then, you know, you have to be root to use it. So that ends up being kind of bad. I've been on a number of engagements where there's been a, you know, people have been added to the Docker group. But, you know, if you're in the Docker group, you can essentially root the host trivially. There's also things like the REST API by default doesn't have authentication. So if you've turned that on and it's bound to all interfaces, then you have some problem where, you know, some outside attacker can get to it or, you know, some server side, your cross forgery or some other thing where you can get to that system. And it could even be exposed to containers through that bridge system if you don't have firewall. And then it also has the kind of GitHub all the way down problem, which I think is not just Docker has the problem where, you know, you've got something like it's including these from these five vendors and they just accept commit requests from these five other random people on GitHub. And all that code is just being chained down to some kid in a basement. And he doesn't know that the code he's committing is going into data centers and, you know, a crazy company. And all it would take would be one commit. There's been backdoor AMIs published before, so we'll see what happens with GitHub and all that. And also, you know, I mentioned it doesn't drop all the capabilities, you know, by default, it still keeps cap net raw, it still keeps cap f owner. You can still make devices in your container. You're going to be limited by app armor, but, you know, most of the time your app probably isn't going to need those, so you should drop them. When you bind a port to Docker by default, it binds it to all the interfaces. And so if you have multiple interfaces that you're exposing to your container, it's going to be on all of those. There's some other things related to base images are also really huge. So a lot of people in their Docker file, which is, you know, the main way of spinning up your Docker image, they'll, you know, from Ubuntu or something like that, but that pulls in like 250 megabytes of packages that, you know, you don't need. And, you know, that's kind of a problem where you think you're just running a single process in that container, but really there's a whole bunch of other stuff there that could be used to try to attack or break out of that container. I mentioned giving the, you know, giving users access to the Docker user or the Docker group on a host, you know, is essentially giving them root. It doesn't have support right now for SACC Comp or username spaces, which are two, you know, really critical security benefits, but it will vary soon. I think 1.8 is what they've mentioned about that. And the other thing I've mentioned that I've seen on GitHub is people for trying to do introspection with their container, if they have something in the container, be able to see, you know, what's going on with itself. They'll expose the Docker socket inside the container. Don't do that. That's giving your container root on your host. And speaking of the user namespace, I have to kind of poke at them a little bit more. They mentioned on Hacker News that, you know, I'm the maintainer of Docker. You know, we will soon have support for user namespaces. When we feel comfortable, the Docker can run UID untrusted zero. We will say so clearly. That was a year ago. No, probably more than a year ago now. And they still don't have it. To give them a low credit, though, I think that one of the major reasons why they don't have the user namespace is I was actually a limitation of go. And not necessarily their fault, but they're on it now. And so hopefully it will be soon. Also, every container talk has to have an image of some broken container. So I can check that box. So Rocket is the other one that kind of came on the stage. It's still, you know, their whole selling point was that there's no root daemon. It's just a binary that you can run and it does the stuff, but there's no thing that's running on your host. It still requires root to run it. So I don't know if they've really solved the problem. It also doesn't drop almost any of the dangerous capabilities. It also doesn't support the user namespace. Let's see. What else? Oh, yeah. It doesn't support set com. It doesn't support app armor. Kind of supports SE Linux, but there's no documentation. For a long time they had a bug where you couldn't actually run things inside a container without running them as root inside the container, which just seems like crazy. They don't have a lot of limits they should on proc or proxys. So, you know, if you read the documentation, they have this very impressive design with this multi-stage system. And it sounds really good, but unfortunately I think this is the current implementation. And, you know, I hope it improves because I like to see competition, but I think right now this is where they're at. So then very recently there was, I think at DockerCon 15 they announced the open container project. I was really, really bummed that they didn't go with this logo because it's awesome and a great movie. So they actually changed their name to the open container initiative. And they really have this specification for containers and kind of to standardize the system and not have all this different kind of competing things. They launched something called Run C, which is a very, very minimal container. Essentially is a kind of competition for Rocket. It's very minimal. You know, you just define your container with a Gison file or something like that. And then it's backed by Lib container, which is reported by Docker and some other people, but they still are not working on Robocop. So, you know, all that stuff sounds kind of bad and messed up and, you know, what can we do? So there's a lot of recommendations I have. I actually have a ton of stuff in a white paper that I'm going to be publishing really soon. So I kind of designed my slides to be somewhat of high level of each area. But the white paper goes into a ton of detail and all this stuff. But really the big message is defense in depth. And I know everybody says that and, you know, I can toot that horn a bunch. But really, you know, if you're trying to defend against things, you want to build your defenses in layers. And that's really the only way you're going to do it. So if we're going to do that, we want to start with a kernel. Because it's got a kernel. So you want to do GR security. Like just everyone should be running it. It's ridiculous, especially on a server where there's no special hardware. You know, sometimes on a laptop, GR security can be a little tricky with drivers or whatever it may be. But on a server, when you're just running a few things or you're doing containers, just do it. And so, you know, obviously hardening your kernel in other ways. Dropping all the capabilities that you don't want. Trying to design for the smallest possible set. And also assuming the worst. So you're giving these capabilities out, but you want to be careful about what they are and try to fully understand what each capability really can do. You also want to use AppArmor or GR security RBAC or SMAC or some other thing. If you can, most of the container systems will have AppArmor by default enabled, which is awesome. It also can be nested. So that's kind of crazy to think about. But you can create an AppArmor profile for a container. And then in that container, if you want to then have something be able to apply a secondary but limited AppArmor profile that's a restricted subset of that first one, you can do that. That's kind of neat. For Docker specifically, you know, you don't want to allow users to run Docker if you don't want to give them root on that host. You don't want to run containers that are privileges or containers that are root. As we've seen, either using an IP table trick or using the CapNet bind service, you do not need to be root to bind to lower the 1024. That's, you know, if you, I don't know how many times you've been on a project. I'm sure you guys know, but you've been on a project or you've looked at some security of something. And the only reason why they're using as root is because they need to bind to port 80 or 443. And that's just stupid and that doesn't exist anymore and nobody should be doing it. Using small base images is also cool. There's a really good talk about building a crazy, tiny, go Docker image. And yeah, that's going to be documented. Using setcomp if you can. So, you know, right now, with LXC, you can use a whitelist of syscalls that you want to allow your app to be able to call. That can really cut off a ton of attack surface in ways that, you know, the kernel is typically, you know, accessed. You want to do all your normal system hardening, right? That's a long list of things, but there's a lot of stuff to do there and that's, you know, what should be done. You want to avoid things that are listing on all interfaces or just keep that in mind, you know, that cross container networking. The other thing that's cool about containers is if you have these different interfaces, it's really easy to tell what's going in and out of my processes in that container. So, it can be great for TraceFlow. So, the core picture is, you know, you've got this hardened application, then under that you've got your user name space, you've dropped capabilities, then you have mount protections, right? Let's say that container is read only except for maybe like the log files or something like that. And then you've got a minimal container distro and then you do sitcom filtering and then you've got a GR security and then you can even do, you know, a hypervisor or some other kind of virtual machine to gain some like actual hardware separation and then you isolate that whole thing by some trust that you give it and then do that differently for other things. So, you know, your containers that are getting hit by your API that are on the front end of your system, those should be treated with different trust than the sum container back end that has your database, right? You don't want to have those on the same system if you can avoid it, even in separate hardware VMs, that's not ideal. So, where do we go from here, right? So, more namespaces hopefully will be coming. That'll help clear that up a little bit. Some of the edge cases where, you know, PROC right now is kind of a shit show. Minimal hypervisors, right? So, there was a thing recently where Intel came out with this thing called clear containers, which essentially like a 20 megabyte hypervisor that will give you some hardware virtualization aspects that you can apply to containers. That's a really cool hybrid model that I think is going to be need to explore. More minimal container features will probably be coming out soon or more refined going forward. And also, you know, these, this, because this is Linux on Linux and it can be, most of these features can be implemented anywhere Linux can run. You can actually do this kind of stuff on embedded devices. You can do these kinds of isolation mechanisms or run containers on a phone or on whatever, whereas, you know, virtualization or other things is less capable there. There's also, you know, we've talked a lot about servers, but really there's no reason why desktop Linux can't benefit from all these isolation mechanisms or using containers. So, hopefully, you know, there's a lot more efforts there. You know, I personally run everything in a container that speaks anything externally or that parses anything from the outside. So, there's all of this micro services idea, which is somewhat new. It's kind of cool. You end up something with something like this. I think one of the ways that's neat about it is that actually isn't a micro services picture. That's the Chrome sandbox. And so if you think about it, the way that micro services, the idea of them, you know, splitting up everything into 100 different REST APIs, that sounds like a pain in the ass, but really it lets you do a lot of least privileges and least access. And, you know, you can do a lot of crazy cool security where everyone always draws this, you know, all these boxes on a white board of this is what our system looks like. But at the end of the day, there's a whole bunch of other pieces that get connected there and they don't have the security they should. It's not just, you know, my auth service talks to the database. It also could talk to some other thing because nobody writes firewall rules for all that stuff. So, you know, the micro services model is, you know, this kind of splitting everything up into pieces, running it all in little containers, having, you know, message queues and all kinds of things. Unfortunately, you know, it starts off with something kind of like this and ends up something like this when it gets implemented. So really, you know, it's not about perfect security. Containers, you know, you can't just drop that and think you have security, but it's improving the work, right? The number of times that I've been on a pen test or some red team or something, I've been stopped by popping something and then having it be inside a sandbox or a container is close to zero. And that's just silly these days because it's not that hard to do containers. It can secure a lot of existing systems that are Linux, you know, and the micro services architecture is, you know, it's coming, it's going to be adding, that's going to be a lot more popular, and I actually like it from the security aspects of it. So coming soon, so my white paper is coming out, it covers everything that I've talked about in a whole lot more depth. It covers, you know, all kinds of past attacks and looking at new areas and everything. And hopefully it'll be released soon. If you want to make sure that you get it, I'm going to be writing everybody that emails me in a text file. You can follow me on Twitter and I'll talk about it at some point. But yeah, any questions or comments? I'm going to answer that. Three. And I'll be around after, you know, this is the last talk. And thanks a lot for hanging out. It's kind of late, but I appreciate it.
 Welcome everybody. Everybody will come in and get us sat down. Erin is here to talk to us about Linux  containers. Thanks. Yeah, so I'm Erin. I've been doing security a while. ISEC, if you don't  know who NCC group is, ISEC, Montesano came together. We were brought by NCC group from the UK.  But same people, same place. I've given some talks before on some different things. So this is my  first DEF CON talk, but it is my 12th DEF CON. And this is my 10 second promotional motivational  slide to not wait as long as I did and not be a slacker. And if you're, you know, you've got an  idea and you know, we've got lots of inspirational talks and cool talks here, you know, keep that  going and do research and submit to the CFP and give talks. And don't wait as long as I did. So I  want to start with this story. So Bob, this is Bob here. He's got some web app or he's got something  that's on the internet. It's on Linux. And somehow there's a bug. He gets popped. There's some kind  of RCE. He gets arbitrary execution, something like that. And he wants to add some security, but  he's not really sure. So somebody says, oh, you should use CH root. That's legit. You know, open  SSH uses it. It must be good, right? But as we all know, it's broken if you have root. You  essentially just CH root yourself. And then you can go up to the root. And yeah. So that's  well, well known. So somebody else comes along and says, oh, you should use SE Linux, right?  That's easy. The NSA made it. It must be secure. It's not in. It's a, you know, if you know anything  about SE Linux, it's a super complicated type based system. It's really designed for multi-level  security. It's really designed for kind of government things when you've got lots of different  things to classify. It's got really good support on Red Hat, but other than that, it's not so great.  Some of those reasons, right, complexity, it's really complicated to actually create it. It has the  Linux Torvodes problem, which most of the other Linux security modules have where, you know, he  doesn't really believe in any of them. The Set-in-4-0 problem where, you know, if you walk into a  box that has SE Linux, the odds of it being disabled are high. And, you know, at the end of the  day, the kernel's got a kernel, right? So if there's a kernel vulnerability and SE Linux is being  enforced by the kernel, it just gets turned off. If you ever look at any of Spender's exploits, he  loves to just do that first and then dig at them as he does. But speaking of Spender, so somebody  else comes along to this, to Bob and says, no, you should use GR Security. That's the best. And  yeah, it really is in a lot of ways. But that's not really solving some of the problem, right? If he  has WordPress and he's got some arbitrary, you know, command injection, GR Security is going to do  nothing for that, right? You're doing a lot of memory corruption protection. You're doing a lot of  other things. But unless you do all the kind of turn everything to 11 with RBAC and everything, it's  not going to solve that. So then somebody else says, no, you should use, just use VMs. You know,  that's how to be ultra secure. But at the end of the day, you know, VMs aren't perfect either and  you've got a lot of other problems with configuration management and everything to do there. So the  second story is Glenn. He knows a lot about security. He's his crypto cat. And so he talks to  this potential source and he's really, really paranoid. He wants to run Linux because, you know, he  doesn't trust OS X. He doesn't take that seriously and, you know, malware is on Windows or whatever. And  you know, all the cool kids run Linux and it's got to be super secure, right? But not really. I mean, he's  one Tor browser bug exploited away from having his calculator popped or whatever may happen. You  know, live purple track record is an awesome. Last time I updated my live CD, that didn't go so  well. So the third story is Margaret, right? Margaret works at an IOT company. She is tired of  getting these, everything running as root and everything getting popped and not having to be able to  see any kind of security on this little platform. Then I actually inserted this slide after I saw  Charlie and Chris's talk where, you know, maybe Margaret also works on the GBU Connect team and  doesn't want to run D bus as root and have just come, you know, no ability to control things. But  really, you know, a lot of that's done just because money obviously takes time to add security.  There's no way to virtualize something that's on ARM or MIPS or this little embedded thing that's  running Linux. A lot of these other things are too heavy duty solutions. There's got to be something we  can do. And really, these stories have in common attack services, right? You know, that's the worst  attack service possible there at the bottom. But really, you know, attack service is really  important. And sandboxes and containers, it would be great to see them being the norm, not the exception.  If you look at Chrome and Adobe Reader X, seat belt, you know, any of the kind of modern sandboxing  environments, that really should be everywhere. There's a battle that we're fighting with native code  and you're going to lose, right? And so until we can win and everything's rewritten in some crazy  rust or whatever it may be, we have to cut our losses and we can do that with containers, you can do  that with sandboxing, you can do that with other isolation. So for Linux, you know, open source  loves to reinvent the wheel. But how I'd like to kind of set this up is, you know, being in Vegas,  one of my favorite movies, so there's this movie, Moonbox Aintz, where, you know, they come out of  this house and they're staring at six guys with guns drawn. And, you know, it's a firefight to  paraphrase. And there's this kind of bumbling investigator that says, you know, what if it wasn't  six guys with six guns, but one guy with six guns? And, you know, he's kind of an idiot, so  William DeFoe is William DeFoe. And he says, you know, shut up, you don't know what you're talking  about. But really Greenlee was right. It was one guy with six guns. So, you know, the person is  proposed, what if it wasn't a whole bunch of kernels, but one kernel with multiple user lands? And  that's really the core idea of containers, right? So this also is not a new idea. This has been done  a long time before Docker and everything came along. So before we keep going, I just want to say  that this is, if you are really scared about security or incredibly paranoid and you have  something, you know, ridiculous system, you do not want to rely purely on any kind of  containerization solution. You don't want to rely purely on any virtualization solution. You  know, you need guys with guns and fences and, you know, air gaps, actual air gaps, not things  connected by wires. And really, you know, I'm not up here to say that containers can do  everything. But what I am here to say is that you don't want to depend on any one single method.  And containers are getting a lot easier, they're getting a lot more powerful. And if you can do  anything, you might as well just add things in layers, as we've seen. So to get to how this  actually works, so the first major area is namespaces. So within the kernel, just like a lot  of any other cool stuff on Unix, it really started in plan 9 where they had the idea of  namespaces in the first place. But the Linux kernel is split up into these kind of five  namespaces, network, mount, user, UTS, which is host name and pit and pit. And the way you get  into one of these namespaces is when you clone, when you process clones and spawn a child, it's  kind of the new fancy fork, the kind of phrase that you get set up into is you enter a new  kernel execution context. And the way you do that is by when you run, when you call clone, you  add these special flags at the end, depending on the namespaces that you want to enter. And so the  mount namespace was added first a long time ago. It basically lets your process have a per  process view of what the file system is. IPC, we can keep going through these. Your  processes can have their own host name, they can have their own view of the other processes, and  they can be nested and there's some cool tricks you can do. If you look at how this actually  works, if you create a container and you run PS in it, you pretty much just see the processes that  are in your container. Which leads me to the awesome snake oil solution. You can't hack what  you can't see. So the new, there's also a network namespace that isolates your IP, your  firewall, your routing table, all that kind of thing. And the user namespace, which is the  newest one, which was added, that is really important for securing your containers and  essentially lets you be root inside of a container, but that's still treated as a low  rights user in the context of the kernel outside of the container. Which obviously is also a high  risk area of the container. You're really controlling a lot of how the UID system works within  the kernel in very sensitive areas. And so there have been a few vulnerabilities of using the user  namespace to break things outside the context of containers. The way that it looks is, you know,  if you attach to our food container, if you, your root you can see inside, you run sleep and  then outside of it you're actually UID 100,000. Essentially just shifts all your UIDs up by  some offset. So the other major area of containers is capabilities. The idea of  capabilities is you take the user root, right, who can do everything and you blow that up into a  whole bunch of little pieces. You know, whether you need to bind to something lower than 1024,  whether you need to be able to run ptrace, whether you need to be able to, you know, change your  network settings. And capabilities are great, right? That lets us get rid of this just kind of god  mode and split it up into a whole bunch of little pieces that really can do, you know, only what  you want that process to be able to do with root. The bad thing is that they started lumping them  together a bit in weird ways. And so you end up with kind of a capabilities model that  somewhat works but is confusing, hard and can be messed up. So some of the, you know, kind of  dangerous capabilities or capabilities that you might commonly encounter are things like being  able to receive raw packets or bind to something lower than 1024 or change resource controls or  send, kill. So everyone always asks, you know, what capabilities should be dropped? You know,  should I drop this or that? And really you want to drop all of them. You want to set up  everything for your container or your namespace ahead of time and then when you pivot it, you  want to throw away all your capabilities and just live in your little cocoon and you can't do  anything at all. And then you get questions on forums or even just people as implementing this,  you're thinking like, what if I just leave this one capability enabled? What happens? And the  answer is it depends. So if we go and look at an example of this, so we look at ping, ping is set  UID root. Everybody from forever ago remembers that, you know, like CD record was set UID root and  you could pop shells on Linux and do privsk by doing, you know, any kind of set UID root binary  with a vulnerability, you get instant root. So it's kind of ridiculous that ping, you know, on  these days still has root and set UID root. Obviously the attack surface is ping and has been  pounded on a bunch. If you copy ping to somewhere else, you'll lose your set UID root bit. And if  you try to run that ping, you'll get operation not permitted because you need to be able to create  raw sockets for ping or ICMP. So the way to fix that is you can set a capability of cap net  raw on that new elf. If we, you know, look at that, we can see that we have cap net raw and then  when you try to run it, even as a low rights user, now I can ping, but that's the only thing I can do.  That's the only capability that I have. So obviously there's a lot of dangerous capabilities that you  don't want to give things. Being able to road a kernel module, that could be bad. Being able to  override, you know, just criss-ary access controls or turn off the manager access control system,  you know, things like that, that's definitely things you don't want to have. The other bad thing is  there's a capability cap sys admin. And obviously that sounds pretty important. It's basically root.  There's a whole lot of things that cap sys admin can do. There's also a great post by Radspender  that goes into all the detail and all the different capabilities and how they can be used to get root  or do bad things. And that's the link to it. And so the other major aspect of containers is  control groups. The basic idea of those is they're hierarchical and inheritable system for  controlling resources across a set of processes. And that can be devices or CPU usage or physical  CPU, memory amounts, IO rates of a certain device, network. And it's really you limit on  serial rates if you want to think about it that way. And it's mainly used to kind of fill the gaps  of some namespaces. So right now there is no dev namespace or there's no namespace. So you kind of  have to get around that a little bit by using C groups in a way that someone intended to do that,  but not really. They're typically controlled through a proc file system, you know, pseudo-it.  So it's just a directory and you put create directories and this corresponds to PIDs and things  will read that directory. The good thing is obviously that it's a file system based and everything  on Unix is a file that used to be at least following that model. But at the end of the day you can do  tricky over mount attacks and other things that is a side effect of having it be a file. So it can be  controlled through CGA Jack, CGA manager, but really most of the container platforms will  abstract this away. So when you put all that together, namespaces isolate the elements of the  kernel, the capabilities, help enforce those namespaces and limit the capabilities within that  container and then the C groups limit the access. So really those three elements along with some  other, you know, secrets, you know, magic sauce will create containers and it is better than C H  group, right? There's a lot of background to those. You get special mount options, you know, you can do  things like an overlay file system where, you know, part of the directory is shown to the container  and part of it is shown to the host and there's kind of some go in between. You can do crazy stuff  with SSHFS now. So where are Linux capability or, you know, where are containers being used now on  servers? So a lot of platform as a service systems, you know, EC2, you can do Docker, Google App  Engine uses containers, Heroku has been doing it for way before anybody did and they, you know,  have Rackspace, everybody probably more than I don't have on that list. It's also being used in  clients. So, you know, Chrome OS is a huge user of these technologies. They've done a lot of work  hard, hard work on making, you know, awesome sandboxing in Chrome and Chrome OS. It's somewhat used  in Android. It's somewhat used on some Linux distributions. There's also some cool sandboxing  tools if you're into those that use these without kind of going to full blown containers. So the  kind of major one that started on the kind of path of Docker and the current trend was LXC. So LXC is  kind of vanilla and I won't go into the details of this but essentially it's controlled through a  template, you know, you set various options, you know, what devices you want to have, some security  different, some security things like Set Comp and App Armor or, you know, what capabilities you want to  keep or drop. There's also been some recent advancement and I say recent within like the last year or  two of containers. So, unpriloged containers, I mentioned that the user namespace, so that  really creates the ability for a non-privileged user to create a new container that they can then be  root in but is all treated as a non-root user when it comes down to it. Obviously there's some weird  things that can happen with that situation so there's more work and auditing needs to be done there  but that'll be, you know, a futuristic thing where, you know, a low rights user that doesn't have root  and never will get you root will be able to create containers and do things. But you have this  problem, right? We talked about attack surface. So, SysCalls, right, are one of the main entry points  to the kernel. There was a lot back in 2.2. Now there's a whole lot more. What are all those and does  your app really need all those, you know, 300 of some odd SysCalls? Probably not. So then this  thing came along called Set Comp and the guys at Google Chrome team are doing a ton of awesome work to  really push the boundaries of what you can do here in a very good way. This has been tried  before but the way they're doing it is much better than some of the methods that have been tried in the  past. But it's important to point out that Set Comp by itself is not a sandbox. It's purely made to  just limit the attack surface to things that your app really doesn't need to have. So you can also,  not only can you filter on certain SysCalls like this process, it's just crunching numbers. So it  just needs to call, you know, return and open and that's it. But you can actually control the  arguments to those SysCalls and say, you know, it only needs to open this file or it needs to, it can  only do these flags on this file. And so that's pretty powerful, right? You do encounter a  performance hit. So when you've got a lot of filters, there will be some impact. I think that we're  still kind of figuring out what that is and how to optimize it. But it definitely is something to  consider if you're going to be rolling it out. And right now you will need LXC or use MiniJail  directly or use Mbox or something like that. You can't, if you're going to use Docker with Set Comp,  you have to use the slash code and tribb or wait till 1.8, which I think is when they're going to  actually launch Set Comp support. But LXC has it right now and you can use it. It's typically done  through the PR control. You basically can set one of two flags, the strict mode, which I think is  only four or five SysCalls, which should be used for just like a, you know, once you get your process  into a spot where it's just going to be crunching numbers or doing something very basic, you can put  that on there. Or you can filter and by filter, I mean a Berkeley facet filter, which everybody  remembers from TCP dump. This is how you would use Set Comp in an actual program directly. And  Berkeley facet filter is pretty cool. You can actually use TCP dump to dump out some of the language  that I don't really know how it works underneath, but it's cool that you can see it. So where is Set Comp  being used right now? Lots of places. You can turn it on in lots of places at least. It's not  necessarily on by default. It is definitely in Chrome and they've been doing a lot of good work. It's  also in Tor, but it's not on by default. So containers, you know, who's doing these, right? So  Docker, obviously, CoreOS, they have Rocket. There's lots of interesting other companies that are either  doing these in the background or they're offering them as a service or they're going to be offering  this as a service. You know, big shout out actually to theSansProm.io guys. They have a crazy, like,  customized system and they do GRSEC for everything. And they're super serious about security and  everything else. And also the RancherOS is an interesting idea where everything in that kind of  minimal system runs in a container. It's also funny. I don't know if this has been fixed or not, but  when you would Google that, it would actually bring up Wave with RancherOS. Google would fix it. So  yeah, they might have some SEO problems. But we're really here to talk about the big two, right? So  Docker. When I usually mention Docker, this is the response. I'm from Silicon Valley. So, you know, this  is usually the response I get from some devs. It's really packaging and kind of development focused. But  one thing that is really good about that, I think, is that it's also very app focused. So the kind of  the philosophy of Docker is that it's not just a container with, you know, some things in it. It's  ideally one process in that container and that's it. And that, you know, as a security person, I really  like that. Because then not only is it that one process in that container, but it's that one process  with just the libraries that it needs and just the files it needs and nothing else. And so you really  are cutting off a lot of attack service there. And the big deal about Docker and why it's gotten so  popular is that it really makes containers easy, right? LXC was kind of doing it, you know, in kind of the  harder mode. Whereas Docker is, you know, you just do Docker pull and magic happens. But Docker isn't  just containers, right? So there's all these other things that they've done. It used to be based off of  LXC. Now they have something called libcontainers written in go. They also have libchan and libswarm and  all these other things. They have a REST API. It's by default, it uses a Unix socket, but you can turn on  the REST API. It also, one of the kind of downsides of it that they're working on fixing, I think, is  related to the fact that, you know, there's this root daemon that runs and everything, all your  containers and everything from that goes under that daemon. And runs, that runs as root. You have to be  root to interact with it, which then makes people run as root on the container host. And, you know, that's  never a good thing. So, you know, you can pull down your images. You can map things from the host into  your container. There's also this idea of Docker hub, which is kind of like GitHub for containers. You  know, you can do commits and all this stuff. And there's all the, you know, orchestration, communication  and management. I don't think the word orchestration was really mentioned much at all in the last, you know,  hundred years, except for maybe last year. Anyway, so CoreOS is the other big one. It's really started  out as a minimal OS for hosting containers. They're launching Rocket in this app container spec. They're  trying to kind of distance themselves from LXC and Docker. They're kind of main ideas that they're trying  to be more secure and lean and be what Docker should be or used to be. And, but, you know, why would anybody  use any of these systems? And it's mostly just to make it easy, right? Or to package a lot of the  functionality with it. And one way that you might kind of relate to it is, you know, you can run for  EBSD and you can secure that, but you kind of need to know what you're doing. Or you can run OS 10 and  it's easy and it's kind of BSD still. But you kind of have to take some things, you know, for granted about  control. And so, you know, LXC, it's kind of vanilla, it lets you do whatever you want and you do get some  bleeding edge there. But you kind of have to know what you're doing in order to do everything correctly,  especially based on the defaults. Whereas Docker, you know, they do a lot of stuff for you. They drop most of  the capabilities that you don't need, but they still leave some around and you need to kind of explicitly  disable those if your app doesn't need them. And we'll get into that. So, you know, LXC, hard mode, but you  got flexibility. Docker, easy, but you know, you have some costs or some risks. CoreOS, it's new. It's very, very  new. We'll get to that. So, but we're here to talk about attacks, right? That's all just stuff. So if you  think about this, there's a lot of different ways that we can go about thinking about what are the attacks in a  container or what should I be worried about? So there's containers to other containers. There's things in the  container against itself essentially, trying to elevate your privileges within a container if you're not just a  single process. Container to the container host, things supporting infrastructure, whether that's  orchestration and all that other stuff or whether that's other things on the network. And you can go from  there. So kind of starting at the top, the kernel obviously is doing all this virtualization stuff. So in the  kernel, you've got all these old drivers. You've got all this old code. You've got weird file system support.  Pretty much nobody compiles a kernel from scratch anymore that doesn't have all this bloaty stuff that you  probably don't even need. Which means you're also inheriting all those vulnerabilities of some random can  bus, you know, special can bus driver thing in Linux that you don't use on your server. The other way that  this kind of typically goes bad is not dropping capabilities. So CapNet Admin is kind of a bad offender.  There's been, I don't know how many different CVEs were of documentation and vulnerabilities where you can go  from CapNet Admin to something else that you shouldn't be doing that isn't related to networking. The way that  happened in a lot of cases is being able to force CapNet Admin to load an arbitrary kernel module that the  youth then control. That was, I think, one of the worst cases. But, you know, speaking of dropping capabilities,  Docker was really stung by this. So they kept around for a long time the CapDac read search capability. And  if you read the man page and maybe this is what they did, you know, it says something like, you know, you can  read some files related to, you know, permissions. You can also invoke open handle at, which is a syscall that  doesn't really have a ton of documentation. They probably thought, oh, whatever. But it turns out that if you  actually use that syscall and you do some other tricks based on iNodes and other things, you can  essentially brute force the iNode of EtsyShadow in the host and read it from inside the container. And  obviously there's other attacks you can do related to that. But, you know, that was something that really  gave the capability model a good example of why you need to drop the ones that you don't really need because  they're complicated. And, you know, if you don't drop those capabilities, you have to rely on some other  thing to, you know, enforce the rules on your container. And, you know, that's typically done through a  manager access control system like SC Linux or AppArmor. But if you don't use that, the capabilities are your  only line of defense. Because if you're root in a container, you're still root on a host. Unless you're  using the user namespace, which again doesn't have a ton of support yet or adoption. You also have the  problem of limiting access. So we mentioned how the kernel, there's not a namespace for PROC or there's  not a namespace for dev. And so what happens is you have things in the kernel or things in the  container that are exposed from the host or the bridge of the host and the container. So things like  PROC-K core where you can expose the kernel memory into the container. You have things like, you know,  sysrec, which everyone forgets about, but stung, you know, LXC for a number of years where they, you  know, were leaving that open. You also have things like being able to make new devices or even just  see D message, right? That's not a ton of attack surface there, but, you know, you're leaking things like  maybe firewall rules or other stuff. And then unattended access to the network, right? So a lot of  these container systems use a bridge system by default. If you remember networking 101, a bridge is  a switch. So most of the default systems by, I think, almost all the containers will create a  bridge and let your containers talk to each other. And that isn't something that most people realize  when they deploy five containers on a host. They don't think that all those containers will be able  to communicate, but they can unless you've written firewall rules or switched to some other networking  type. You also, you know, see people not limiting any resources. Obviously, if you fork bomb a  container with my favorite bash function there, you're going to be fork bombing the host. And then  kind of some other things relating to, you know, Docker has been stung by this a couple times of, you  know, this idea of taking your pack, taking your application, packaging it all up with the libraries  that you need, putting a lock on it and shipping it off everywhere. You know, that is, doesn't really  happen. What happens when there's a vulnerability in one of those libraries or that application and you  need to update it, what do you do? You know, you can't just, you're not supposed to run app get  upgrade on your containers. It's kind of a no-no in the container world because they're supposed to be  ephemeral and immutable. And so what do you do? And so you kind of need to define, to have a system  to adopt that, have a patching system. And then the lack of mandatory access control. So most of the  container systems also have some form of app armor support or SE linux support that works quite well.  Obviously it's not perfect. There's been some bugs even very recently that were published there. But at  the end of the day, you know, the kernel is still enforcing things. So this isn't a perfect solution  either. And really it should just be one facet of many different things. And we mentioned that the  networking defaults and how that can be bad. For LXC specifically, the defaults are quite bad for  dropping capabilities. But that kind of relates to the way that the model of LXC is, you have a  minimal container, but it's not supposed to really just be a single process. It's supposed to be, you  know, a few different things. Maybe an SSH daemon or system D or something like that. Whereas  Docker is the idea of one process. But speaking of Docker, so Docker does a pretty good job of  dropping capabilities. But if you read the between the lines, they mention that they drop all the  capabilities except for those required. But if they don't know what you're going to be running, how do  they know the ones that are still required? And then, you know, you have to be root to use it. So  that ends up being kind of bad. I've been on a number of engagements where there's been a, you know,  people have been added to the Docker group. But, you know, if you're in the Docker group, you can  essentially root the host trivially. There's also things like the REST API by default doesn't have  authentication. So if you've turned that on and it's bound to all interfaces, then you have some  problem where, you know, some outside attacker can get to it or, you know, some server side, your  cross forgery or some other thing where you can get to that system. And it could even be exposed to  containers through that bridge system if you don't have firewall. And then it also has the kind of  GitHub all the way down problem, which I think is not just Docker has the problem where, you know, you've  got something like it's including these from these five vendors and they just accept commit requests from  these five other random people on GitHub. And all that code is just being chained down to some kid in a  basement. And he doesn't know that the code he's committing is going into data centers and, you know, a  crazy company. And all it would take would be one commit. There's been backdoor AMIs published before,  so we'll see what happens with GitHub and all that. And also, you know, I mentioned it doesn't drop all  the capabilities, you know, by default, it still keeps cap net raw, it still keeps cap f owner. You can  still make devices in your container. You're going to be limited by app armor, but, you know, most of the  time your app probably isn't going to need those, so you should drop them. When you bind a port to  Docker by default, it binds it to all the interfaces. And so if you have multiple interfaces that you're  exposing to your container, it's going to be on all of those. There's some other things related to base  images are also really huge. So a lot of people in their Docker file, which is, you know, the main way of  spinning up your Docker image, they'll, you know, from Ubuntu or something like that, but that pulls in  like 250 megabytes of packages that, you know, you don't need. And, you know, that's kind of a problem where  you think you're just running a single process in that container, but really there's a whole bunch of other  stuff there that could be used to try to attack or break out of that container. I mentioned giving the,  you know, giving users access to the Docker user or the Docker group on a host, you know, is essentially  giving them root. It doesn't have support right now for SACC Comp or username spaces, which are two, you  know, really critical security benefits, but it will vary soon. I think 1.8 is what they've mentioned about  that. And the other thing I've mentioned that I've seen on GitHub is people for trying to do introspection with  their container, if they have something in the container, be able to see, you know, what's going on with  itself. They'll expose the Docker socket inside the container. Don't do that. That's giving your container root on  your host. And speaking of the user namespace, I have to kind of poke at them a little bit more. They  mentioned on Hacker News that, you know, I'm the maintainer of Docker. You know, we will soon have support for  user namespaces. When we feel comfortable, the Docker can run UID untrusted zero. We will say so clearly.  That was a year ago. No, probably more than a year ago now. And they still don't have it. To give them a  low credit, though, I think that one of the major reasons why they don't have the user namespace is I was  actually a limitation of go. And not necessarily their fault, but they're on it now. And so hopefully  it will be soon. Also, every container talk has to have an image of some broken container. So I can  check that box. So Rocket is the other one that kind of came on the stage. It's still, you know, their whole  selling point was that there's no root daemon. It's just a binary that you can run and it does the stuff, but  there's no thing that's running on your host. It still requires root to run it. So I don't know if they've  really solved the problem. It also doesn't drop almost any of the dangerous capabilities. It also doesn't  support the user namespace. Let's see. What else? Oh, yeah. It doesn't support set com. It doesn't support  app armor. Kind of supports SE Linux, but there's no documentation. For a long time they had a bug where  you couldn't actually run things inside a container without running them as root inside the container,  which just seems like crazy. They don't have a lot of limits they should on proc or proxys. So, you know, if  you read the documentation, they have this very impressive design with this multi-stage system. And it  sounds really good, but unfortunately I think this is the current implementation. And, you know, I hope it  improves because I like to see competition, but I think right now this is where they're at. So then very  recently there was, I think at DockerCon 15 they announced the open container project. I was really, really  bummed that they didn't go with this logo because it's awesome and a great movie. So they actually  changed their name to the open container initiative. And they really have this specification for  containers and kind of to standardize the system and not have all this different kind of competing  things. They launched something called Run C, which is a very, very minimal container. Essentially is a kind of  competition for Rocket. It's very minimal. You know, you just define your container with a Gison file or  something like that. And then it's backed by Lib container, which is reported by Docker and some other  people, but they still are not working on Robocop. So, you know, all that stuff sounds kind of bad and  messed up and, you know, what can we do? So there's a lot of recommendations I have. I actually have a ton of  stuff in a white paper that I'm going to be publishing really soon. So I kind of designed my slides to be  somewhat of high level of each area. But the white paper goes into a ton of detail and all this stuff. But  really the big message is defense in depth. And I know everybody says that and, you know, I can toot that horn a  bunch. But really, you know, if you're trying to defend against things, you want to build your defenses in  layers. And that's really the only way you're going to do it. So if we're going to do that, we want to start  with a kernel. Because it's got a kernel. So you want to do GR security. Like just everyone should be running it.  It's ridiculous, especially on a server where there's no special hardware. You know, sometimes on a laptop,  GR security can be a little tricky with drivers or whatever it may be. But on a server, when you're just  running a few things or you're doing containers, just do it. And so, you know, obviously hardening your kernel in  other ways. Dropping all the capabilities that you don't want. Trying to design for the smallest possible set.  And also assuming the worst. So you're giving these capabilities out, but you want to be careful about what they  are and try to fully understand what each capability really can do. You also want to use AppArmor or  GR security RBAC or SMAC or some other thing. If you can, most of the container systems will have AppArmor by  default enabled, which is awesome. It also can be nested. So that's kind of crazy to think about. But you can  create an AppArmor profile for a container. And then in that container, if you want to then have something be able to  apply a secondary but limited AppArmor profile that's a restricted subset of that first one, you can do that.  That's kind of neat. For Docker specifically, you know, you don't want to allow users to run Docker if you don't  want to give them root on that host. You don't want to run containers that are privileges or containers that are  root. As we've seen, either using an IP table trick or using the CapNet bind service, you do not need to be root to  bind to lower the 1024. That's, you know, if you, I don't know how many times you've been on a project. I'm sure you  guys know, but you've been on a project or you've looked at some security of something. And the only reason why they're  using as root is because they need to bind to port 80 or 443. And that's just stupid and that doesn't exist anymore  and nobody should be doing it. Using small base images is also cool. There's a really good talk about building a  crazy, tiny, go Docker image. And yeah, that's going to be documented. Using setcomp if you can. So, you know, right now,  with LXC, you can use a whitelist of syscalls that you want to allow your app to be able to call. That can really  cut off a ton of attack surface in ways that, you know, the kernel is typically, you know, accessed. You want to do all  your normal system hardening, right? That's a long list of things, but there's a lot of stuff to do there and that's,  you know, what should be done. You want to avoid things that are listing on all interfaces or just keep that in mind,  you know, that cross container networking. The other thing that's cool about containers is if you have these different  interfaces, it's really easy to tell what's going in and out of my processes in that container. So, it can be great for  TraceFlow. So, the core picture is, you know, you've got this hardened application, then under that you've got your  user name space, you've dropped capabilities, then you have mount protections, right? Let's say that container is read  only except for maybe like the log files or something like that. And then you've got a minimal container distro and then  you do sitcom filtering and then you've got a GR security and then you can even do, you know, a hypervisor or some other  kind of virtual machine to gain some like actual hardware separation and then you isolate that whole thing by some trust  that you give it and then do that differently for other things. So, you know, your containers that are getting hit by  your API that are on the front end of your system, those should be treated with different trust than the sum container  back end that has your database, right? You don't want to have those on the same system if you can avoid it, even in  separate hardware VMs, that's not ideal. So, where do we go from here, right? So, more namespaces hopefully will be  coming. That'll help clear that up a little bit. Some of the edge cases where, you know, PROC right now is kind of a  shit show. Minimal hypervisors, right? So, there was a thing recently where Intel came out with this thing called clear containers,  which essentially like a 20 megabyte hypervisor that will give you some hardware virtualization aspects that you can  apply to containers. That's a really cool hybrid model that I think is going to be need to explore. More minimal container  features will probably be coming out soon or more refined going forward. And also, you know, these, this, because this is  Linux on Linux and it can be, most of these features can be implemented anywhere Linux can run. You can actually do this  kind of stuff on embedded devices. You can do these kinds of isolation mechanisms or run containers on a phone or on  whatever, whereas, you know, virtualization or other things is less capable there. There's also, you know, we've talked a lot  about servers, but really there's no reason why desktop Linux can't benefit from all these isolation mechanisms or using  containers. So, hopefully, you know, there's a lot more efforts there. You know, I personally run everything in a  container that speaks anything externally or that parses anything from the outside. So, there's all of this micro  services idea, which is somewhat new. It's kind of cool. You end up something with something like this. I think one of the  ways that's neat about it is that actually isn't a micro services picture. That's the Chrome sandbox. And so if you  think about it, the way that micro services, the idea of them, you know, splitting up everything into 100 different  REST APIs, that sounds like a pain in the ass, but really it lets you do a lot of least privileges and least access. And,  you know, you can do a lot of crazy cool security where everyone always draws this, you know, all these boxes on a white  board of this is what our system looks like. But at the end of the day, there's a whole bunch of other pieces that get  connected there and they don't have the security they should. It's not just, you know, my auth service talks to the  database. It also could talk to some other thing because nobody writes firewall rules for all that stuff. So, you  know, the micro services model is, you know, this kind of splitting everything up into pieces, running it all in little  containers, having, you know, message queues and all kinds of things. Unfortunately, you know, it starts off with  something kind of like this and ends up something like this when it gets implemented. So really, you know, it's not  about perfect security. Containers, you know, you can't just drop that and think you have security, but it's  improving the work, right? The number of times that I've been on a pen test or some red team or something, I've been  stopped by popping something and then having it be inside a sandbox or a container is close to zero. And that's  just silly these days because it's not that hard to do containers. It can secure a lot of existing systems that  are Linux, you know, and the micro services architecture is, you know, it's coming, it's going to be adding,  that's going to be a lot more popular, and I actually like it from the security aspects of it. So coming soon, so my  white paper is coming out, it covers everything that I've talked about in a whole lot more depth. It covers, you  know, all kinds of past attacks and looking at new areas and everything. And hopefully it'll be released soon. If you  want to make sure that you get it, I'm going to be writing everybody that emails me in a text file. You can  follow me on Twitter and I'll talk about it at some point. But yeah, any questions or comments? I'm going to  answer that. Three. And I'll be around after, you know, this is the last talk. And thanks a lot for  hanging out. It's kind of late, but I appreciate it.
DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_0_467.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_468_887.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_888_1535.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_1536_5015.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_5016_5483.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_5484_6311.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_6312_7787.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_7788_8783.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_8784_9155.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_9156_9311.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_9312_9695.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_9696_11099.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_11100_12527.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_12528_13127.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_13128_13715.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_13716_15191.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_15192_15455.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_15456_15803.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_15804_16931.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_16932_20195.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_20196_22535.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_22536_23567.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_23568_25355.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_25356_26723.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_26724_30047.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_30048_30395.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_30396_50363.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_50364_51371.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_51372_51563.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_51564_52031.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_52032_52775.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_52776_53219.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_53220_54455.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_54456_61139.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_61140_64895.png DEF_CON_23_-_Aaron_Grattafiori_-_Linux_Containers__Future_or_Fantasy__-_101_Track_iN6QbszB1R8_64896_67008.png
Welcome everybody. Everybody will come in and get us sat down. Erin is here to talk to us about Linux containers. Thanks. Yeah, so I'm Erin. I've been doing security a while. ISEC, if you don't know who NCC group is, ISEC, Montesano came together. We were brought by NCC group from the UK. But same people, same place. I've given some talks before on some different things. So this is my first DEF CON talk, but it is my 12th DEF CON. And this is my 10 second promotional motivational slide to not wait as long as I did and not be a slacker. And if you're, you know, you've got an idea and you know, we've got lots of inspirational talks and cool talks here, you know, keep that going and do research and submit to the CFP and give talks. And don't wait as long as I did. So I want to start with this story. So Bob, this is Bob here. He's got some web app or he's got something that's on the internet. It's on Linux. And somehow there's a bug. He gets popped. There's some kind of RCE. He gets arbitrary execution, something like that. And he wants to add some security, but he's not really sure. So somebody says, oh, you should use CH root. That's legit. You know, open SSH uses it. It must be good, right? But as we all know, it's broken if you have root. You essentially just CH root yourself. And then you can go up to the root. And yeah. So that's well, well known. So somebody else comes along and says, oh, you should use SE Linux, right? That's easy. The NSA made it. It must be secure. It's not in. It's a, you know, if you know anything about SE Linux, it's a super complicated type based system. It's really designed for multi-level security. It's really designed for kind of government things when you've got lots of different things to classify. It's got really good support on Red Hat, but other than that, it's not so great. Some of those reasons, right, complexity, it's really complicated to actually create it. It has the Linux Torvodes problem, which most of the other Linux security modules have where, you know, he doesn't really believe in any of them. The Set-in-4-0 problem where, you know, if you walk into a box that has SE Linux, the odds of it being disabled are high. And, you know, at the end of the day, the kernel's got a kernel, right? So if there's a kernel vulnerability and SE Linux is being enforced by the kernel, it just gets turned off. If you ever look at any of Spender's exploits, he loves to just do that first and then dig at them as he does. But speaking of Spender, so somebody else comes along to this, to Bob and says, no, you should use GR Security. That's the best. And yeah, it really is in a lot of ways. But that's not really solving some of the problem, right? If he has WordPress and he's got some arbitrary, you know, command injection, GR Security is going to do nothing for that, right? You're doing a lot of memory corruption protection. You're doing a lot of other things. But unless you do all the kind of turn everything to 11 with RBAC and everything, it's not going to solve that. So then somebody else says, no, you should use, just use VMs. You know, that's how to be ultra secure. But at the end of the day, you know, VMs aren't perfect either and you've got a lot of other problems with configuration management and everything to do there. So the second story is Glenn. He knows a lot about security. He's his crypto cat. And so he talks to this potential source and he's really, really paranoid. He wants to run Linux because, you know, he doesn't trust OS X. He doesn't take that seriously and, you know, malware is on Windows or whatever. And you know, all the cool kids run Linux and it's got to be super secure, right? But not really. I mean, he's one Tor browser bug exploited away from having his calculator popped or whatever may happen. You know, live purple track record is an awesome. Last time I updated my live CD, that didn't go so well. So the third story is Margaret, right? Margaret works at an IOT company. She is tired of getting these, everything running as root and everything getting popped and not having to be able to see any kind of security on this little platform. Then I actually inserted this slide after I saw Charlie and Chris's talk where, you know, maybe Margaret also works on the GBU Connect team and doesn't want to run D bus as root and have just come, you know, no ability to control things. But really, you know, a lot of that's done just because money obviously takes time to add security. There's no way to virtualize something that's on ARM or MIPS or this little embedded thing that's running Linux. A lot of these other things are too heavy duty solutions. There's got to be something we can do. And really, these stories have in common attack services, right? You know, that's the worst attack service possible there at the bottom. But really, you know, attack service is really important. And sandboxes and containers, it would be great to see them being the norm, not the exception. If you look at Chrome and Adobe Reader X, seat belt, you know, any of the kind of modern sandboxing environments, that really should be everywhere. There's a battle that we're fighting with native code and you're going to lose, right? And so until we can win and everything's rewritten in some crazy rust or whatever it may be, we have to cut our losses and we can do that with containers, you can do that with sandboxing, you can do that with other isolation. So for Linux, you know, open source loves to reinvent the wheel. But how I'd like to kind of set this up is, you know, being in Vegas, one of my favorite movies, so there's this movie, Moonbox Aintz, where, you know, they come out of this house and they're staring at six guys with guns drawn. And, you know, it's a firefight to paraphrase. And there's this kind of bumbling investigator that says, you know, what if it wasn't six guys with six guns, but one guy with six guns? And, you know, he's kind of an idiot, so William DeFoe is William DeFoe. And he says, you know, shut up, you don't know what you're talking about. But really Greenlee was right. It was one guy with six guns. So, you know, the person is proposed, what if it wasn't a whole bunch of kernels, but one kernel with multiple user lands? And that's really the core idea of containers, right? So this also is not a new idea. This has been done a long time before Docker and everything came along. So before we keep going, I just want to say that this is, if you are really scared about security or incredibly paranoid and you have something, you know, ridiculous system, you do not want to rely purely on any kind of containerization solution. You don't want to rely purely on any virtualization solution. You know, you need guys with guns and fences and, you know, air gaps, actual air gaps, not things connected by wires. And really, you know, I'm not up here to say that containers can do everything. But what I am here to say is that you don't want to depend on any one single method. And containers are getting a lot easier, they're getting a lot more powerful. And if you can do anything, you might as well just add things in layers, as we've seen. So to get to how this actually works, so the first major area is namespaces. So within the kernel, just like a lot of any other cool stuff on Unix, it really started in plan 9 where they had the idea of namespaces in the first place. But the Linux kernel is split up into these kind of five namespaces, network, mount, user, UTS, which is host name and pit and pit. And the way you get into one of these namespaces is when you clone, when you process clones and spawn a child, it's kind of the new fancy fork, the kind of phrase that you get set up into is you enter a new kernel execution context. And the way you do that is by when you run, when you call clone, you add these special flags at the end, depending on the namespaces that you want to enter. And so the mount namespace was added first a long time ago. It basically lets your process have a per process view of what the file system is. IPC, we can keep going through these. Your processes can have their own host name, they can have their own view of the other processes, and they can be nested and there's some cool tricks you can do. If you look at how this actually works, if you create a container and you run PS in it, you pretty much just see the processes that are in your container. Which leads me to the awesome snake oil solution. You can't hack what you can't see. So the new, there's also a network namespace that isolates your IP, your firewall, your routing table, all that kind of thing. And the user namespace, which is the newest one, which was added, that is really important for securing your containers and essentially lets you be root inside of a container, but that's still treated as a low rights user in the context of the kernel outside of the container. Which obviously is also a high risk area of the container. You're really controlling a lot of how the UID system works within the kernel in very sensitive areas. And so there have been a few vulnerabilities of using the user namespace to break things outside the context of containers. The way that it looks is, you know, if you attach to our food container, if you, your root you can see inside, you run sleep and then outside of it you're actually UID 100,000. Essentially just shifts all your UIDs up by some offset. So the other major area of containers is capabilities. The idea of capabilities is you take the user root, right, who can do everything and you blow that up into a whole bunch of little pieces. You know, whether you need to bind to something lower than 1024, whether you need to be able to run ptrace, whether you need to be able to, you know, change your network settings. And capabilities are great, right? That lets us get rid of this just kind of god mode and split it up into a whole bunch of little pieces that really can do, you know, only what you want that process to be able to do with root. The bad thing is that they started lumping them together a bit in weird ways. And so you end up with kind of a capabilities model that somewhat works but is confusing, hard and can be messed up. So some of the, you know, kind of dangerous capabilities or capabilities that you might commonly encounter are things like being able to receive raw packets or bind to something lower than 1024 or change resource controls or send, kill. So everyone always asks, you know, what capabilities should be dropped? You know, should I drop this or that? And really you want to drop all of them. You want to set up everything for your container or your namespace ahead of time and then when you pivot it, you want to throw away all your capabilities and just live in your little cocoon and you can't do anything at all. And then you get questions on forums or even just people as implementing this, you're thinking like, what if I just leave this one capability enabled? What happens? And the answer is it depends. So if we go and look at an example of this, so we look at ping, ping is set UID root. Everybody from forever ago remembers that, you know, like CD record was set UID root and you could pop shells on Linux and do privsk by doing, you know, any kind of set UID root binary with a vulnerability, you get instant root. So it's kind of ridiculous that ping, you know, on these days still has root and set UID root. Obviously the attack surface is ping and has been pounded on a bunch. If you copy ping to somewhere else, you'll lose your set UID root bit. And if you try to run that ping, you'll get operation not permitted because you need to be able to create raw sockets for ping or ICMP. So the way to fix that is you can set a capability of cap net raw on that new elf. If we, you know, look at that, we can see that we have cap net raw and then when you try to run it, even as a low rights user, now I can ping, but that's the only thing I can do. That's the only capability that I have. So obviously there's a lot of dangerous capabilities that you don't want to give things. Being able to road a kernel module, that could be bad. Being able to override, you know, just criss-ary access controls or turn off the manager access control system, you know, things like that, that's definitely things you don't want to have. The other bad thing is there's a capability cap sys admin. And obviously that sounds pretty important. It's basically root. There's a whole lot of things that cap sys admin can do. There's also a great post by Radspender that goes into all the detail and all the different capabilities and how they can be used to get root or do bad things. And that's the link to it. And so the other major aspect of containers is control groups. The basic idea of those is they're hierarchical and inheritable system for controlling resources across a set of processes. And that can be devices or CPU usage or physical CPU, memory amounts, IO rates of a certain device, network. And it's really you limit on serial rates if you want to think about it that way. And it's mainly used to kind of fill the gaps of some namespaces. So right now there is no dev namespace or there's no namespace. So you kind of have to get around that a little bit by using C groups in a way that someone intended to do that, but not really. They're typically controlled through a proc file system, you know, pseudo-it. So it's just a directory and you put create directories and this corresponds to PIDs and things will read that directory. The good thing is obviously that it's a file system based and everything on Unix is a file that used to be at least following that model. But at the end of the day you can do tricky over mount attacks and other things that is a side effect of having it be a file. So it can be controlled through CGA Jack, CGA manager, but really most of the container platforms will abstract this away. So when you put all that together, namespaces isolate the elements of the kernel, the capabilities, help enforce those namespaces and limit the capabilities within that container and then the C groups limit the access. So really those three elements along with some other, you know, secrets, you know, magic sauce will create containers and it is better than C H group, right? There's a lot of background to those. You get special mount options, you know, you can do things like an overlay file system where, you know, part of the directory is shown to the container and part of it is shown to the host and there's kind of some go in between. You can do crazy stuff with SSHFS now. So where are Linux capability or, you know, where are containers being used now on servers? So a lot of platform as a service systems, you know, EC2, you can do Docker, Google App Engine uses containers, Heroku has been doing it for way before anybody did and they, you know, have Rackspace, everybody probably more than I don't have on that list. It's also being used in clients. So, you know, Chrome OS is a huge user of these technologies. They've done a lot of work hard, hard work on making, you know, awesome sandboxing in Chrome and Chrome OS. It's somewhat used in Android. It's somewhat used on some Linux distributions. There's also some cool sandboxing tools if you're into those that use these without kind of going to full blown containers. So the kind of major one that started on the kind of path of Docker and the current trend was LXC. So LXC is kind of vanilla and I won't go into the details of this but essentially it's controlled through a template, you know, you set various options, you know, what devices you want to have, some security different, some security things like Set Comp and App Armor or, you know, what capabilities you want to keep or drop. There's also been some recent advancement and I say recent within like the last year or two of containers. So, unpriloged containers, I mentioned that the user namespace, so that really creates the ability for a non-privileged user to create a new container that they can then be root in but is all treated as a non-root user when it comes down to it. Obviously there's some weird things that can happen with that situation so there's more work and auditing needs to be done there but that'll be, you know, a futuristic thing where, you know, a low rights user that doesn't have root and never will get you root will be able to create containers and do things. But you have this problem, right? We talked about attack surface. So, SysCalls, right, are one of the main entry points to the kernel. There was a lot back in 2.2. Now there's a whole lot more. What are all those and does your app really need all those, you know, 300 of some odd SysCalls? Probably not. So then this thing came along called Set Comp and the guys at Google Chrome team are doing a ton of awesome work to really push the boundaries of what you can do here in a very good way. This has been tried before but the way they're doing it is much better than some of the methods that have been tried in the past. But it's important to point out that Set Comp by itself is not a sandbox. It's purely made to just limit the attack surface to things that your app really doesn't need to have. So you can also, not only can you filter on certain SysCalls like this process, it's just crunching numbers. So it just needs to call, you know, return and open and that's it. But you can actually control the arguments to those SysCalls and say, you know, it only needs to open this file or it needs to, it can only do these flags on this file. And so that's pretty powerful, right? You do encounter a performance hit. So when you've got a lot of filters, there will be some impact. I think that we're still kind of figuring out what that is and how to optimize it. But it definitely is something to consider if you're going to be rolling it out. And right now you will need LXC or use MiniJail directly or use Mbox or something like that. You can't, if you're going to use Docker with Set Comp, you have to use the slash code and tribb or wait till 1.8, which I think is when they're going to actually launch Set Comp support. But LXC has it right now and you can use it. It's typically done through the PR control. You basically can set one of two flags, the strict mode, which I think is only four or five SysCalls, which should be used for just like a, you know, once you get your process into a spot where it's just going to be crunching numbers or doing something very basic, you can put that on there. Or you can filter and by filter, I mean a Berkeley facet filter, which everybody remembers from TCP dump. This is how you would use Set Comp in an actual program directly. And Berkeley facet filter is pretty cool. You can actually use TCP dump to dump out some of the language that I don't really know how it works underneath, but it's cool that you can see it. So where is Set Comp being used right now? Lots of places. You can turn it on in lots of places at least. It's not necessarily on by default. It is definitely in Chrome and they've been doing a lot of good work. It's also in Tor, but it's not on by default. So containers, you know, who's doing these, right? So Docker, obviously, CoreOS, they have Rocket. There's lots of interesting other companies that are either doing these in the background or they're offering them as a service or they're going to be offering this as a service. You know, big shout out actually to theSansProm.io guys. They have a crazy, like, customized system and they do GRSEC for everything. And they're super serious about security and everything else. And also the RancherOS is an interesting idea where everything in that kind of minimal system runs in a container. It's also funny. I don't know if this has been fixed or not, but when you would Google that, it would actually bring up Wave with RancherOS. Google would fix it. So yeah, they might have some SEO problems. But we're really here to talk about the big two, right? So Docker. When I usually mention Docker, this is the response. I'm from Silicon Valley. So, you know, this is usually the response I get from some devs. It's really packaging and kind of development focused. But one thing that is really good about that, I think, is that it's also very app focused. So the kind of the philosophy of Docker is that it's not just a container with, you know, some things in it. It's ideally one process in that container and that's it. And that, you know, as a security person, I really like that. Because then not only is it that one process in that container, but it's that one process with just the libraries that it needs and just the files it needs and nothing else. And so you really are cutting off a lot of attack service there. And the big deal about Docker and why it's gotten so popular is that it really makes containers easy, right? LXC was kind of doing it, you know, in kind of the harder mode. Whereas Docker is, you know, you just do Docker pull and magic happens. But Docker isn't just containers, right? So there's all these other things that they've done. It used to be based off of LXC. Now they have something called libcontainers written in go. They also have libchan and libswarm and all these other things. They have a REST API. It's by default, it uses a Unix socket, but you can turn on the REST API. It also, one of the kind of downsides of it that they're working on fixing, I think, is related to the fact that, you know, there's this root daemon that runs and everything, all your containers and everything from that goes under that daemon. And runs, that runs as root. You have to be root to interact with it, which then makes people run as root on the container host. And, you know, that's never a good thing. So, you know, you can pull down your images. You can map things from the host into your container. There's also this idea of Docker hub, which is kind of like GitHub for containers. You know, you can do commits and all this stuff. And there's all the, you know, orchestration, communication and management. I don't think the word orchestration was really mentioned much at all in the last, you know, hundred years, except for maybe last year. Anyway, so CoreOS is the other big one. It's really started out as a minimal OS for hosting containers. They're launching Rocket in this app container spec. They're trying to kind of distance themselves from LXC and Docker. They're kind of main ideas that they're trying to be more secure and lean and be what Docker should be or used to be. And, but, you know, why would anybody use any of these systems? And it's mostly just to make it easy, right? Or to package a lot of the functionality with it. And one way that you might kind of relate to it is, you know, you can run for EBSD and you can secure that, but you kind of need to know what you're doing. Or you can run OS 10 and it's easy and it's kind of BSD still. But you kind of have to take some things, you know, for granted about control. And so, you know, LXC, it's kind of vanilla, it lets you do whatever you want and you do get some bleeding edge there. But you kind of have to know what you're doing in order to do everything correctly, especially based on the defaults. Whereas Docker, you know, they do a lot of stuff for you. They drop most of the capabilities that you don't need, but they still leave some around and you need to kind of explicitly disable those if your app doesn't need them. And we'll get into that. So, you know, LXC, hard mode, but you got flexibility. Docker, easy, but you know, you have some costs or some risks. CoreOS, it's new. It's very, very new. We'll get to that. So, but we're here to talk about attacks, right? That's all just stuff. So if you think about this, there's a lot of different ways that we can go about thinking about what are the attacks in a container or what should I be worried about? So there's containers to other containers. There's things in the container against itself essentially, trying to elevate your privileges within a container if you're not just a single process. Container to the container host, things supporting infrastructure, whether that's orchestration and all that other stuff or whether that's other things on the network. And you can go from there. So kind of starting at the top, the kernel obviously is doing all this virtualization stuff. So in the kernel, you've got all these old drivers. You've got all this old code. You've got weird file system support. Pretty much nobody compiles a kernel from scratch anymore that doesn't have all this bloaty stuff that you probably don't even need. Which means you're also inheriting all those vulnerabilities of some random can bus, you know, special can bus driver thing in Linux that you don't use on your server. The other way that this kind of typically goes bad is not dropping capabilities. So CapNet Admin is kind of a bad offender. There's been, I don't know how many different CVEs were of documentation and vulnerabilities where you can go from CapNet Admin to something else that you shouldn't be doing that isn't related to networking. The way that happened in a lot of cases is being able to force CapNet Admin to load an arbitrary kernel module that the youth then control. That was, I think, one of the worst cases. But, you know, speaking of dropping capabilities, Docker was really stung by this. So they kept around for a long time the CapDac read search capability. And if you read the man page and maybe this is what they did, you know, it says something like, you know, you can read some files related to, you know, permissions. You can also invoke open handle at, which is a syscall that doesn't really have a ton of documentation. They probably thought, oh, whatever. But it turns out that if you actually use that syscall and you do some other tricks based on iNodes and other things, you can essentially brute force the iNode of EtsyShadow in the host and read it from inside the container. And obviously there's other attacks you can do related to that. But, you know, that was something that really gave the capability model a good example of why you need to drop the ones that you don't really need because they're complicated. And, you know, if you don't drop those capabilities, you have to rely on some other thing to, you know, enforce the rules on your container. And, you know, that's typically done through a manager access control system like SC Linux or AppArmor. But if you don't use that, the capabilities are your only line of defense. Because if you're root in a container, you're still root on a host. Unless you're using the user namespace, which again doesn't have a ton of support yet or adoption. You also have the problem of limiting access. So we mentioned how the kernel, there's not a namespace for PROC or there's not a namespace for dev. And so what happens is you have things in the kernel or things in the container that are exposed from the host or the bridge of the host and the container. So things like PROC-K core where you can expose the kernel memory into the container. You have things like, you know, sysrec, which everyone forgets about, but stung, you know, LXC for a number of years where they, you know, were leaving that open. You also have things like being able to make new devices or even just see D message, right? That's not a ton of attack surface there, but, you know, you're leaking things like maybe firewall rules or other stuff. And then unattended access to the network, right? So a lot of these container systems use a bridge system by default. If you remember networking 101, a bridge is a switch. So most of the default systems by, I think, almost all the containers will create a bridge and let your containers talk to each other. And that isn't something that most people realize when they deploy five containers on a host. They don't think that all those containers will be able to communicate, but they can unless you've written firewall rules or switched to some other networking type. You also, you know, see people not limiting any resources. Obviously, if you fork bomb a container with my favorite bash function there, you're going to be fork bombing the host. And then kind of some other things relating to, you know, Docker has been stung by this a couple times of, you know, this idea of taking your pack, taking your application, packaging it all up with the libraries that you need, putting a lock on it and shipping it off everywhere. You know, that is, doesn't really happen. What happens when there's a vulnerability in one of those libraries or that application and you need to update it, what do you do? You know, you can't just, you're not supposed to run app get upgrade on your containers. It's kind of a no-no in the container world because they're supposed to be ephemeral and immutable. And so what do you do? And so you kind of need to define, to have a system to adopt that, have a patching system. And then the lack of mandatory access control. So most of the container systems also have some form of app armor support or SE linux support that works quite well. Obviously it's not perfect. There's been some bugs even very recently that were published there. But at the end of the day, you know, the kernel is still enforcing things. So this isn't a perfect solution either. And really it should just be one facet of many different things. And we mentioned that the networking defaults and how that can be bad. For LXC specifically, the defaults are quite bad for dropping capabilities. But that kind of relates to the way that the model of LXC is, you have a minimal container, but it's not supposed to really just be a single process. It's supposed to be, you know, a few different things. Maybe an SSH daemon or system D or something like that. Whereas Docker is the idea of one process. But speaking of Docker, so Docker does a pretty good job of dropping capabilities. But if you read the between the lines, they mention that they drop all the capabilities except for those required. But if they don't know what you're going to be running, how do they know the ones that are still required? And then, you know, you have to be root to use it. So that ends up being kind of bad. I've been on a number of engagements where there's been a, you know, people have been added to the Docker group. But, you know, if you're in the Docker group, you can essentially root the host trivially. There's also things like the REST API by default doesn't have authentication. So if you've turned that on and it's bound to all interfaces, then you have some problem where, you know, some outside attacker can get to it or, you know, some server side, your cross forgery or some other thing where you can get to that system. And it could even be exposed to containers through that bridge system if you don't have firewall. And then it also has the kind of GitHub all the way down problem, which I think is not just Docker has the problem where, you know, you've got something like it's including these from these five vendors and they just accept commit requests from these five other random people on GitHub. And all that code is just being chained down to some kid in a basement. And he doesn't know that the code he's committing is going into data centers and, you know, a crazy company. And all it would take would be one commit. There's been backdoor AMIs published before, so we'll see what happens with GitHub and all that. And also, you know, I mentioned it doesn't drop all the capabilities, you know, by default, it still keeps cap net raw, it still keeps cap f owner. You can still make devices in your container. You're going to be limited by app armor, but, you know, most of the time your app probably isn't going to need those, so you should drop them. When you bind a port to Docker by default, it binds it to all the interfaces. And so if you have multiple interfaces that you're exposing to your container, it's going to be on all of those. There's some other things related to base images are also really huge. So a lot of people in their Docker file, which is, you know, the main way of spinning up your Docker image, they'll, you know, from Ubuntu or something like that, but that pulls in like 250 megabytes of packages that, you know, you don't need. And, you know, that's kind of a problem where you think you're just running a single process in that container, but really there's a whole bunch of other stuff there that could be used to try to attack or break out of that container. I mentioned giving the, you know, giving users access to the Docker user or the Docker group on a host, you know, is essentially giving them root. It doesn't have support right now for SACC Comp or username spaces, which are two, you know, really critical security benefits, but it will vary soon. I think 1.8 is what they've mentioned about that. And the other thing I've mentioned that I've seen on GitHub is people for trying to do introspection with their container, if they have something in the container, be able to see, you know, what's going on with itself. They'll expose the Docker socket inside the container. Don't do that. That's giving your container root on your host. And speaking of the user namespace, I have to kind of poke at them a little bit more. They mentioned on Hacker News that, you know, I'm the maintainer of Docker. You know, we will soon have support for user namespaces. When we feel comfortable, the Docker can run UID untrusted zero. We will say so clearly. That was a year ago. No, probably more than a year ago now. And they still don't have it. To give them a low credit, though, I think that one of the major reasons why they don't have the user namespace is I was actually a limitation of go. And not necessarily their fault, but they're on it now. And so hopefully it will be soon. Also, every container talk has to have an image of some broken container. So I can check that box. So Rocket is the other one that kind of came on the stage. It's still, you know, their whole selling point was that there's no root daemon. It's just a binary that you can run and it does the stuff, but there's no thing that's running on your host. It still requires root to run it. So I don't know if they've really solved the problem. It also doesn't drop almost any of the dangerous capabilities. It also doesn't support the user namespace. Let's see. What else? Oh, yeah. It doesn't support set com. It doesn't support app armor. Kind of supports SE Linux, but there's no documentation. For a long time they had a bug where you couldn't actually run things inside a container without running them as root inside the container, which just seems like crazy. They don't have a lot of limits they should on proc or proxys. So, you know, if you read the documentation, they have this very impressive design with this multi-stage system. And it sounds really good, but unfortunately I think this is the current implementation. And, you know, I hope it improves because I like to see competition, but I think right now this is where they're at. So then very recently there was, I think at DockerCon 15 they announced the open container project. I was really, really bummed that they didn't go with this logo because it's awesome and a great movie. So they actually changed their name to the open container initiative. And they really have this specification for containers and kind of to standardize the system and not have all this different kind of competing things. They launched something called Run C, which is a very, very minimal container. Essentially is a kind of competition for Rocket. It's very minimal. You know, you just define your container with a Gison file or something like that. And then it's backed by Lib container, which is reported by Docker and some other people, but they still are not working on Robocop. So, you know, all that stuff sounds kind of bad and messed up and, you know, what can we do? So there's a lot of recommendations I have. I actually have a ton of stuff in a white paper that I'm going to be publishing really soon. So I kind of designed my slides to be somewhat of high level of each area. But the white paper goes into a ton of detail and all this stuff. But really the big message is defense in depth. And I know everybody says that and, you know, I can toot that horn a bunch. But really, you know, if you're trying to defend against things, you want to build your defenses in layers. And that's really the only way you're going to do it. So if we're going to do that, we want to start with a kernel. Because it's got a kernel. So you want to do GR security. Like just everyone should be running it. It's ridiculous, especially on a server where there's no special hardware. You know, sometimes on a laptop, GR security can be a little tricky with drivers or whatever it may be. But on a server, when you're just running a few things or you're doing containers, just do it. And so, you know, obviously hardening your kernel in other ways. Dropping all the capabilities that you don't want. Trying to design for the smallest possible set. And also assuming the worst. So you're giving these capabilities out, but you want to be careful about what they are and try to fully understand what each capability really can do. You also want to use AppArmor or GR security RBAC or SMAC or some other thing. If you can, most of the container systems will have AppArmor by default enabled, which is awesome. It also can be nested. So that's kind of crazy to think about. But you can create an AppArmor profile for a container. And then in that container, if you want to then have something be able to apply a secondary but limited AppArmor profile that's a restricted subset of that first one, you can do that. That's kind of neat. For Docker specifically, you know, you don't want to allow users to run Docker if you don't want to give them root on that host. You don't want to run containers that are privileges or containers that are root. As we've seen, either using an IP table trick or using the CapNet bind service, you do not need to be root to bind to lower the 1024. That's, you know, if you, I don't know how many times you've been on a project. I'm sure you guys know, but you've been on a project or you've looked at some security of something. And the only reason why they're using as root is because they need to bind to port 80 or 443. And that's just stupid and that doesn't exist anymore and nobody should be doing it. Using small base images is also cool. There's a really good talk about building a crazy, tiny, go Docker image. And yeah, that's going to be documented. Using setcomp if you can. So, you know, right now, with LXC, you can use a whitelist of syscalls that you want to allow your app to be able to call. That can really cut off a ton of attack surface in ways that, you know, the kernel is typically, you know, accessed. You want to do all your normal system hardening, right? That's a long list of things, but there's a lot of stuff to do there and that's, you know, what should be done. You want to avoid things that are listing on all interfaces or just keep that in mind, you know, that cross container networking. The other thing that's cool about containers is if you have these different interfaces, it's really easy to tell what's going in and out of my processes in that container. So, it can be great for TraceFlow. So, the core picture is, you know, you've got this hardened application, then under that you've got your user name space, you've dropped capabilities, then you have mount protections, right? Let's say that container is read only except for maybe like the log files or something like that. And then you've got a minimal container distro and then you do sitcom filtering and then you've got a GR security and then you can even do, you know, a hypervisor or some other kind of virtual machine to gain some like actual hardware separation and then you isolate that whole thing by some trust that you give it and then do that differently for other things. So, you know, your containers that are getting hit by your API that are on the front end of your system, those should be treated with different trust than the sum container back end that has your database, right? You don't want to have those on the same system if you can avoid it, even in separate hardware VMs, that's not ideal. So, where do we go from here, right? So, more namespaces hopefully will be coming. That'll help clear that up a little bit. Some of the edge cases where, you know, PROC right now is kind of a shit show. Minimal hypervisors, right? So, there was a thing recently where Intel came out with this thing called clear containers, which essentially like a 20 megabyte hypervisor that will give you some hardware virtualization aspects that you can apply to containers. That's a really cool hybrid model that I think is going to be need to explore. More minimal container features will probably be coming out soon or more refined going forward. And also, you know, these, this, because this is Linux on Linux and it can be, most of these features can be implemented anywhere Linux can run. You can actually do this kind of stuff on embedded devices. You can do these kinds of isolation mechanisms or run containers on a phone or on whatever, whereas, you know, virtualization or other things is less capable there. There's also, you know, we've talked a lot about servers, but really there's no reason why desktop Linux can't benefit from all these isolation mechanisms or using containers. So, hopefully, you know, there's a lot more efforts there. You know, I personally run everything in a container that speaks anything externally or that parses anything from the outside. So, there's all of this micro services idea, which is somewhat new. It's kind of cool. You end up something with something like this. I think one of the ways that's neat about it is that actually isn't a micro services picture. That's the Chrome sandbox. And so if you think about it, the way that micro services, the idea of them, you know, splitting up everything into 100 different REST APIs, that sounds like a pain in the ass, but really it lets you do a lot of least privileges and least access. And, you know, you can do a lot of crazy cool security where everyone always draws this, you know, all these boxes on a white board of this is what our system looks like. But at the end of the day, there's a whole bunch of other pieces that get connected there and they don't have the security they should. It's not just, you know, my auth service talks to the database. It also could talk to some other thing because nobody writes firewall rules for all that stuff. So, you know, the micro services model is, you know, this kind of splitting everything up into pieces, running it all in little containers, having, you know, message queues and all kinds of things. Unfortunately, you know, it starts off with something kind of like this and ends up something like this when it gets implemented. So really, you know, it's not about perfect security. Containers, you know, you can't just drop that and think you have security, but it's improving the work, right? The number of times that I've been on a pen test or some red team or something, I've been stopped by popping something and then having it be inside a sandbox or a container is close to zero. And that's just silly these days because it's not that hard to do containers. It can secure a lot of existing systems that are Linux, you know, and the micro services architecture is, you know, it's coming, it's going to be adding, that's going to be a lot more popular, and I actually like it from the security aspects of it. So coming soon, so my white paper is coming out, it covers everything that I've talked about in a whole lot more depth. It covers, you know, all kinds of past attacks and looking at new areas and everything. And hopefully it'll be released soon. If you want to make sure that you get it, I'm going to be writing everybody that emails me in a text file. You can follow me on Twitter and I'll talk about it at some point. But yeah, any questions or comments? I'm going to answer that. Three. And I'll be around after, you know, this is the last talk. And thanks a lot for hanging out. It's kind of late, but I appreciate it.
my first DEF CON talk, but it is my 12th DEF CON. know, keep that going and do research and submit Security is going to do nothing for that, right? of security on this little platform. Then I Linux. A lot of these other things are too heavy can do. And really, these stories have in common it would be great to see them being the norm, not modern sandboxing environments, that really with sandboxing, you can do that with other idea. This has been done a long time before virtualization solution. You know, you need guys major area is namespaces. So within the kernel, So the new, there's also a network namespace that controlling a lot of how the UID system works there have been a few vulnerabilities of using containers is capabilities. The idea of capabilities that you might commonly encounter the day you can do tricky over mount attacks and container platforms will abstract this away. So capabilities, help enforce those namespaces and things like an overlay file system where, you and the current trend was LXC. So LXC is kind of also been some recent advancement and I say will be able to create containers and do things. only needs to open this file or it needs to, it really started out as a minimal OS for hosting distance themselves from LXC and Docker. They're order to do everything correctly, especially but you know, you have some costs or some risks. also invoke open handle at, which is a syscall you know, LXC for a number of years where they, right? So a lot of these container systems use a in one of those libraries or that application and container, it's going to be on all of those. two, you know, really critical security benefits, that one of the major reasons why they don't have impressive design with this multi-stage system. some other people, but they still are not working system hardening, right? That's a long list of want to avoid things that are listing on all be, most of these features can be implemented You end up something with something like this. I least privileges and least access. And, you know, like this when it gets implemented. So really, been on a pen test or some red team or something, can secure a lot of existing systems that are popular, and I actually like it from the security You can follow me on Twitter and I'll talk about