Automated Firewall Testing
Provost, Kristof
Automatically testing the pf firewall, using network stack virtualisation. For fun and profit, or at least for the sake of fewer bugs. We're all convinced that automated tests are a good idea. For some applications (e.g. grep, awk, cc, ...) this is very straightforward. Others are a lot harder to test, for example firewalls. Typically testing firewalls takes two to three hosts. One to send traffic, the firewall test host and one to receive traffic. This makes automated test orchestration complex and brittle. This in turn means that tests either don't get written, are difficult to write and/or suffer random failures unrelated to issues in the firewall itself. Virtualisation has made this all somewhat easier, but it's still fiddly and difficult to make robust. It's also slow. FreeBSD 12 will ship with network stack virtualisation (known as VIMAGE or vnet). This is an important feature for many applications, one of which is automated network stack and firewall testing. As of FreeBSD 12 PF fully support VIMAGE, allowing users to configure a firewall for each jail. This talk will introduce VIMAGE and show how it can be used to easily write firewall tests. If there's time a few interesting bugs and their test cases will also be discussed.
 So thank you all for coming. This is the automated firewall testing talk. Before we get started, let's talk about something nobody cares about, who I am. So I'm Christoph, KP at FreeBSD.org. I sort of maintain PF in FreeBSD. So if it doesn't work in FreeBSD, it's all the fault of the openBSD people. But if it does work, it's thanks to my glorious efforts. Or possibly the other way around. Professionally, I do a lot of embellished Linux projects. I will happily do FreeBSD or other BSD projects. I got suckered into being on the EuroBSD-Cone Foundation board, which means, well, that is the reason I've been choosing everyone to submit to the call for papers for EuroBSD-Cone. I will repeat that. Submit your papers and tutorials to EuroBSD-Cone. You have until the 26th. No, seriously, do this. Otherwise, it's going to be about 20 slots of me repeating this talk. I also always feel the need to point out that I'm not for sale, but I can be rented at reasonable rates. That's where you're failing. You rented at unreasonable. Well, I didn't say reasonable to whom. Anyway, what is PF? PF is a packet filter. We stole it from OpenBSD. Imported it from OpenBSD as a word. Last import was a while ago. What we have is an older version. It's not exactly what's in OpenBSD at the moment. We also have a couple of shiny things in FreeBSD that are not in OpenBSD, one of which is VNAT and all more about VNAT later. Another one is that our PF sort of runs multi-core, which means that it's faster. Note the ER. I didn't say fast. I said faster. They're different concepts. I keep recommending that people use IPFW, but I saw the survey results from the FreeBSD users and they don't listen to me. Thank you, OpenBSD. It turns out our users really like your firewall. On to relevant things for this talk. Why automated testing? I think the obvious answer is because we're all lazy and testing is boring, repetitive, mind-bombing work. Somebody should invent a machine that's good at boring and repetitive and mind-bombing work. Oh, wait. Slightly more seriously, why do we want automated tests to make sure that things work? It turns out that our users like it when they install something and it works. They don't get excited when it explodes in new and unexpected ways. Yeah, users are strange, but what are you going to do? It's also really nice to have a convenient test case. Does this thing actually work? I'm going to try to roll out something on my network. Let's test to make sure that it works. Also, about preventing regressions, it works today. It would be nice if it still works tomorrow. It turns out that with complicated software, it's really, really easy to accidentally make a mistake that breaks it. There are a lot of ways things can go wrong and you're not always going to notice immediately when it does go wrong. Actually, I have some examples here. That might be related to the end of my bio about not talking to me about IPv6 fragmentation. Fragment handling is broken several times. One of the things I liberated from OpenBSD was to teach PF to deal with IPv6 fragments. We gather up all of the fragments, we reassemble, we filter all the reassembled packets, and when we send out, we have to re-fragment because IPv6 and bottom to you discovery and all of those things. I'm not going to go into that. The main point is that it's been broken several times. The first time it broke on December 12th of 2016 and we noticed October 25th of 2017, or rather we fixed it October 25th of 2017. There's a pretty long time for it to be broken. The problem in the first case was that we checked the packet size before the P-fill hook, so before we went into PF. PF didn't have the chance to re-fragment before it got transmitted. The stack looked at it and went, yes, but this packet is way too big. Let me send an ICMP packet too big error back to a host who has no idea what you're talking about because he sent much more packets. The second one is a better example. Well, they say better. It would have been better if we didn't break it, but it's software we're going to be breaking things. It was broken as a result of a security fix. If that number doesn't mean anything to you, basically the problem was that the way we handled packet fragments could be exploited to trigger worst case performance, which would make things very, very slow and your machine would not be doing anything else other than sitting there trying to reassemble packets that were designed to be very difficult to reassemble. That got fixed, except it accidentally broke V6 reassembly. Oops. But, you know, Bjorn, nobody cares about IPv6, right? He's in the back. That's why I'm brave enough to say this. It broke. The tests noticed this immediately. Only it was the F test, so everybody just went, it's Kristof's fault. We don't need to look at this. I looked at it, managed to find a fix for it, and it only took two weeks. Now, two weeks is longer than we would like it to be broken, but given the choice between being broken for two weeks and being broken for nine months, I think it's a fairly obvious choice. For additional fun, that bug was one of these very strange bugs. When I started detracing things, it went away. I've actually got the code here. I'll give you a few seconds to try to work out what the problem was. I have Jeopardy music, and I can't figure out how to make it play. So just imagine the Jeopardy jingle, and I'll give you a few more seconds to. Thank you very much, Paul. So we've all spotted the bug, right? So I don't need to be embarrassed that it took me an embarrassingly long time to figure it out. I can show you the patch. Does that help? Well, going back to the code, basically, we hash the fragments into buckets now to speed up the lookup. So we allocate hash key, which means to be the size of two IPv6 addresses and another 32 bits from fragment ID, except the size of I6 adder is 128 bits, so mumble number of bytes, times four, because you win 32t. And then we hash over the entire size of the array. So we grab some random stack garbage, include that in the hash, and then stick it in the buckets. And if your random stack garbage is different on every entry into the function, you're going to hash your fragments into different buckets, and you're never, ever going to reassemble them. Unless, of course, you detrace Frag6 input, and then your stack garbage is, you know, whatever's left behind by detrace, so it's predictable. So it all hashes back into the same buckets, and it works. I will not point to guilty people, mostly because I don't actually remember whose fault it was. I just remember being annoyed at figuring this one out. So, you know, we fixed it, and that was awesome. I hope I've convinced you of, you know, tests are a good idea. What do we want out of these tests? I want them to be easy to write for two reasons. The first and most important one being, you know, I'm lazy, and I want things to be easy. But another one is that I would like more people to write tests. I'll get back to this, but when I get bug reports from users about PF issues, the vast majority of the time I spend on trying to fix them is not looking at the code, debugging it. It's not writing a patch. It's, what the hell are you doing? What's your setup? And, you know, given your setup, what goes on to make it break? So if you want me to fix a bug for you, well, not the best way. The second best way is to give me the test case. Because I have to write the test anyway. The very best way to get me to fix a bug for you, of course, is to pay me. So anyway, we want them to be easy to write. We want them to be easy for everyone to run them, which means that hopefully people will run those tests before they commit something and will notice that they've, that they broke something before they commit it. That has a number of advantages, you know. They work smarter because they didn't break, they didn't commit something that broke stuff. We don't break things for our users and we have some extremely brave users who run, you know, large chunks of the internet or at least large chunks of video across the internet who like using latest and greatest. It would be nice if we didn't break things for them. But it also means, you know, from a strictly selfish point of view that you're going to notice that you broke something rather than, you know, committing something and then me getting shouted at for the PF test breaking again. You know, I might be very selfish. We also want them to be fast to run. That sort of fits in with, you know, wanting everyone to run them. Because if it only takes two minutes to run, they'll run them. If it takes four hours to run, they won't. We also want to integrate with the automated testing framework, which is an automated testing framework in QA, the automated test for PF or built around that. They are wonderful tools that we came up with ourselves and totally didn't steal from that PSD. Because if they integrate with that, they will turn up in our Jenkins instance. This is a very old screenshot. We've got more tests now, but this one 7,300 tests. And in this case, well, two failures. That's two more than we'd like, but at least most of the testing works. These get run often. Lee-Wen, you know, happened to know off the top of your head how often they get run. That's every couple of hours at the worst, right? So every couple of hours, so ideally when you commit something and it breaks, there will be evidence of this a couple of hours later. With any luck, you'll actually look at this and notice that, oh, wait, I committed something and everything is broken now. Lee-Wen very kindly writes weekly reports about the state of our tests and sends them to 3DSD testing at and a couple of other mailing lists. So there's someone actually following up on this. So how do we build these tests? The obvious way to do this is we get a bunch of hardware and get a server, we get a switch, we get another server, we send packets from server A to server B and maybe we get some replies and we look at them. And that's great, unless we want to forward, okay, server, switch, server, switch, server, and we send packets through that. Unless, of course, we want to do pf sync or cart or more complicated setup, so even more hardware. That's good advantages. You can do things like performance testing when you have real hardware, but it's got a couple of downsides as well, like, you know, this is a firewall test. One of the tests we'd like to do is block all traffic. Kind of hard to remotely configure a system if you block all traffic. You can go and, you know, configure it over serial lines, but then you need more hardware and more machines to talk to and all of those things. We want to run the same 3BSE version on all of these systems. Okay, we'll net boot them, but yeah, firewall. Pretty annoying that, you know, you net boot, you mount your root file system over NFS and then you block all traffic and your root file system goes away. There are ways around this, but it, you know, it becomes complicated. Then what happens if the machine panics? You need to have control code that will manage or will understand that, hey, this machine panics. I should collect the information out of it and I should give it a hard reset. Oh, by the way, we need some more hardware to do this. You can imagine complicated setups. Where does all of this hardware live? I have too much old hardware at home, but not enough hardware to build all of this, so it would have to live somewhere else. But if we need five machines and serial interfaces and power switches, we can't really expect all developers to have this setup. So how do other people write tests for this when they can test their tests? There are advantages to this setup. Alexander did a great talk about it. One of the advantages is you can do meaningful performance testing against it, but I'm sure he'll agree that it's a complicated setup that needs some care and feeling. So, you know, that's a downside. The obvious way around all of these is, you know, virtual hardware. We've got Beehive. Beehive is great. We can run any number of machines, well, you know, low number of machines, but enough to build our setups in Beehive. And that's actually the approach taken by a student in Google Summer of Code 2017 to try to build this. We've still got some issues, you know. We walk all traffic and oops, we've got a serial port or we can pretend to have a serial port and we can make that work. Great. But our automated tests on the IDOP3BAS.org run in Beehive. So we have to run Beehive inside Beehive. And I don't actually know if that's possible, but if it is, it's bound to be slow. And the darker, darker is good enough for everyone in the cloud. Beehive, Beehive should be good enough for you. Yeah. Another issue with that, and that's where the what the Google Summer of Code student ran into is it is really, really annoying to build the images during your build run. It takes forever and you need a lot of storage and all of those things. You've still got issues coping with panics. It's still possible, but it's still annoying. It's also really slow because you're waiting for emulated, well, virtualized machines to boot. Best case 20, 30 seconds depending on what hardware you've got. It's an improvement, but it's still annoying. I keep telling you what solutions won't work, what will work. Well, VNet or network stack virtualization hands up everyone who knows and loves VNet. Okay, for those of you who don't know, it's network stack virtualization. Here we go. I've explained that. It's associated with a jail and it basically turns a jail into a pretend virtual machine. They all run the same kernel, but every jail can have its own network interfaces. It can have its own IP address, you can set whatever IP address you like. You can also run your firewall inside the jail. For all intents and purposes, you have your own machine. Of course, it's not 100% the same as running on actual hardware, but it's 99% the same code. PF supports this. Actually, to be entirely fair, all of our firewalls support this now. I'm mostly focused on PF because I spent some time arguing with PF about, you know, maybe you should try not panicking when we run you in a jail. After a while that sank in and now it usually doesn't panic when you run it in a jail. The usual disclaimer there, right? I do development. I look at bugs that users report. So I spend all of my time looking at bugs. And once I've fixed one of them, it goes away, but I go on and look at the next bug. So there tends to be a perception of everything is broken and everything is horrible. Turns out that as a user, and I actually use PF in the net jails myself on one of my machines, and it works just fine. I don't actually think that machine's ever panicked over one of those things. So it works a lot better than I get the impression. But bugs will happen. Please report them. As of 12.0, I will try to help you if you run PF in a jail and it panics for you. If you do it in 11, I will just point and laugh. Unless you threaten to pay me in which case I will quietly laugh and help you. Anyway, I've been talking about this forever now. How do we actually run a VNet jail with its own IP stock? I bet it's really, really hard. That's it. We start the jail. We give it a name, Alcatraz in this case, because I think that's hilarious. We tell it that we want it to be a VNet jail and we want to persist it. Persist just tells the jail that the jail should keep existing even if there are no processes in it. Usually, that's a silly thing to want because jails isolate processes. If you don't have a process, why would you bother to isolate it? But in this case, the jail has an IP stack and you could forward packets. So that's it. We're done here. All right. Well, to be entirely fair, that's not all that there's to it. We also need some network interfaces. It's all easier to test network things when you can communicate with the outside world. So to do that, we've got something called ePair. ePair is actually really simple. It's two network interfaces with a cable between the mixup. The cable doesn't exist and the network interfaces are virtual, but it's two network interfaces with a cable. So you get an ePair zero, which has an A and a B side and A and B are linked with each other. And if you stick a packet into A, you can add to B and the other way around. The usual way you would use them is you would give an IP address to the A interface. You would stick the D interface in the jail. So same thing. We start the jail with VNAT and oh, by the way, you have an interface called ePair zero B. And then we can actually execute a command inside the jail. And I still haven't fixed my slides. This mix misses Alcatraz. So execute inside Alcatraz. If can fake ePair zero B, there's an IP address, and then you can ping it. And that's that's all there's to it. And we're already testing our IP stack or testing that we can send by CMP Echo requests and get replies. It's not terribly exciting, but it actually runs a whole lot of code. And we we had, you know, any number of panics around this, because it turns out that setting up and shutting down my IP stack or a network stack is something that we only ever did when we booted them, when we rebooted. And if you do 20 of those, rather than one, and you do several of them one after the other, or you do them at the same time, you run into bugs that you never ever run into in other situations. But that's basically all we need to do. So, so let's take a look at writing a test for this. That's not all that there's to the test, but you know, the entire test case doesn't fit all my slides. So what we do, we use the automated testing framework, which again, you know, we totally enhance ourselves and in no way stole from the SD includes some utilities, declare a test case, we have a v4 test case cleanup just says that there is a cleanup function and you should call it after you're done with the tests. Test head just describes the test. So, in the description, basic password test for IPv4, we require a user. For some reason, people think that security is important. And maybe we shouldn't allow people to spin up new IP sacks and network configuration unless they're actually, you know, root. The only thing this does is make sure that when you run the test as more brews, this test will just skip because it can't possibly succeed. The interesting bits is test bolly. And that's, that's all there's to it. Although admittedly, I've stripped out a bunch of tests, but this is a functional test. This is useful. So we have general initialization codes. The initialization code doesn't do a whole lot of exciting work. It just goes and checks that VNet feature is available in your kernel because if it's not, there's no point in running this test. It also checks that PF is available because, you know, testing PF when the PF module is not loaded again, not terribly useful. We create an EPIR. We assign an IP address to it, just like we discussed earlier. Create a jail, help a script to create a jail, create a jail arbitrage, give us an interface, set an IP address in the jail, and then just, you know, basic sanity check. So ATF check will just go and verify that the next command will return exit state to zero. We ignore the standard about it because we don't care what it says as long as it returns with zero. One thing only, and the time out of one second to that IP address. Hopefully, you know, the jail will apply and everything will be well. Next up, we enable PF. PF by default passes everything, so this should still work. So, yeah, check exit state to zero, and then we're going to walk everything. So we set rules in the jail, walk input. This is a simple helper function. It just makes life easier when you can write this rather than write it to a text file or echo it and pipe it into the jail. It's much shorter to write this. Remember, I'm lazy. So we check that we get an error output. We still don't care about standard outs, and that this thing will fail. Something about this, as we'll see later when I show some output, is this entire test run takes 1.2 seconds. The entire second is taken up by this because we spend the second waiting for a reply that will never, ever come. I mean, it should unless we totally broke it something. Everything else is 200 milliseconds to top start the jail. So, you know, maybe we should do some boot time improvements so we can get things to boot faster in beehive, but I don't think we're ever going to get to the point where we can boot an entire machine in 200 milliseconds. We'll actually boot it, do some work in it, and shut it down in 200 milliseconds. But we can do that with VNetJails, which makes it a lot easier to run all of these tests. I say a lot. We have some tests for PFRite now, not that many, but enough that they catch bugs, and they only tend to take about a minute. So, cleanup, pretty easy. There's a utility function. The other utility functions, whenever they create an interface or a jail, helpfully keep track of this in a temporary file. So, the cleanup function just goes and looks, what's in the temporary file? We've created the EPAR0A, delete EPAR0A. We've created EPAR1A, delete EPAR1A. We've created the Alcatraz jail, okay, remove the Alcatraz jail, just so that we clean up everything so that the next test isn't going to conflict with whatever we've left behind. And then this is just ATF infrastructure. If you're really, really interested in how all of the ATF things work, there's an article in the FreeBSD Journal about this. So, you can take a look at that. The FreeBSD Journal is now free, so you can just get onto the FreeBSD Foundation website and go take a look at it. What business output? Well, there we go. We do, you know, sudo because we need to be root to run this. Obviously, QA test, password V4. Password V4, the test passes. Okay, it works. 1.2 seconds. We even have some results. We can go and look at, you know, what sort of output did this test generate? Which is useful. Although, to be entirely fair, usually when the tests fail, I will start off by adding SIT-X to the test body so that I can see what commands it's all executing. And we'll probably end up setting a sleep 3600 somewhere so that it doesn't clean up the jail so I can go look at, you know, we have the test set up and now I can send some things and I can TCP dump and I can, I might even detrace or look at some other statistics or some other counters to, you know, what is actually going on here. But they run, and these things run automatically. These things run several times a day. So, the next time somebody breaks PF so that it, you know, rather than dropping packets, just lets them all through, which, you know, how would you notice unless you test? And it is kind of important for your firewall to occasionally drop packets and maybe even the right packets. Anyway, next example is a much, much more complicated example because it's a test for PF sync. For those of you who don't know, PF has this wonderful feature where you can synchronize states between two machines. So, the typical use case for this is when you have a gateway machine, you just have a second one. And if one of them dies or you take it down for an upgrade, the other one will take over. But you wanted to remember states because if your TCP session is running, you would like it to keep running. So, PF sync will send state from one machine to the other so that it can't notionally take over. So, very basic tests for this. We create a couple of interfaces because, you know, we'll have two machines, we'll have two jails, jail one, jail two. We'll have interfaces to all of those jails and we'll have a link between them so that they can exchange states. Hence the e-pair sync, e-pair one, e-pair two, jail one, jail two. I got tired of the naming my jails after this jail's joke here, as you can tell. Give them the two interfaces, configure interfaces in jail one, enable PF sync, synchronize over, sync a device, send some up, max update if I remember correctly, that just tells PF sync that the moment you have a single update ready, send it out because usually it tries to gather up a whole bunch of updates before it starts to send packets off to the other side, send it up, do the same thing with the second jail. So, there's a lot going on there but it's basically all related to how do I actually configure PF sync to do something. Enable PF in both of the jails, set IP addresses and then we send the ping. This ping will travel through the first jail and after some time that state will get synchronized to the other jail so we can go ask jail two, you know, what states do you have and do you have an ICMP state for this particular IP address? Should have one because we send an ICMP echo through it. We go look for it, make sure that it matches what we expected to do and if we don't find it, the test fails. So, very primitive, well, it's very basic test. It's set up PF sync, make sure that the one state we deliberately create in it ends up on the other side. It's primitive but it tests a lot of the functionality of PF sync. It's basically, you know, the basic functionality of PF sync has to work for this test to work. I did some work on PF sync recently and, you know, this test failed very often. It also demonstrates that, you know, it is actually that hard to test something that needs two machines. You could test things with even more machines. I have tested for things like fragment handling as well. I don't have examples in the slides but there's one where, you know, we send a bunch of fragments at it and we test what happens. So, these are all fairly simple tests because we send, you know, correctly format packets. There are some tests that do things like use KB, the PyFold tool to build arbitrary packets to send malformed packets which is another fun thing to do to your network stack. Yeah, I'm trying to remember, I think for some of the recent security issues, I also wrote something to send specifically malformed packets to trigger something that would otherwise have been a panic but now is no longer a panic because Alexander fixed the bug for us. Was it you or Sasha? Sasha, please. Okay. So, I've set the credit to the wrong side then. Anyway, for those of you interested in the math tests, the PF tests live under user source, assuming that's where you install your source. Test sys netpvlpf to mirror where our PF code lives, our PF code lives user source sys netpvlpf. When they get installed, they're in user test sys netpvlpf. If you want to run them, sudo package install qr and then go to, well, actually there we go, install qr, installs kb because that tests a few more things. Kld loads pf sync which will implicitly load pf. Go to user tests sys netpvlpf and then probably sudo qr test unless you've done all of this as root already. There's a lot more information in the journal. Well, there's information on how to write ATF tests. You should all be writing tests. Yes, even you open BSD people, you should be writing tests for free BSD. Just copy that. Well, there is actually a lot to be said for we should take a look at some of your tests. Are you threatening your payers to do it? No. I'm threatening you with my internal gratitude. So there's an article in the free BSD journal, the March April edition of this year, how do you write some test things? Hopefully one of our documentation people will actually go and add some of this to the handbook as well. Benedict's working on this. Benedict's not here but I'm going to apply social pressure anyway. Important things, well, if I still haven't convinced you now what we need is some sort of deeply profound quote by someone famous. I didn't have the budget for that. So you get me telling you, write tests. Tests are good. I don't know about how profound that is but it is actually true. Tests are wonderful. What's in that for you? You can prototype some of your setups because sometimes you try to do things and they don't actually work. You can prototype a lot of networking setups with just jails, which means that you can do it in a virtual machine all playing over whatever ocean you like to fly over. It also means that if you write a test for this and you give us the test case, we'll include it and we'll run it, which means that your use case is not going to break or when it breaks, we're going to know. And we're going to know pretty much immediately after we commit the breaking change rather than three years later when you finally get around to upgrading and then notice that you're careful use case no longer works. And we broke it three years ago. Developers have, well, I might be projecting here, it might just be me, but in general developers have a very short attention span. So if you notice that they broke something a couple of days after they did it, you're a lot more likely to get them to care than when you notice three years later. What else is in it for you? Right. Yes. As I mentioned before, write a test case because it makes it really, it makes it much easier for me to fix your bug. I'm lazy. Make it easy for me. Recruiting bugs is hard. I need to understand your entire test setup. And I often get, you know, well, I get two types of bug reports usually. I get the one that says it's broken. Please fix it with no information in it as to, you know, what is broken? What is your test setup? How do you break it? How often does it break? In what way does it break? No information whatsoever. The other type and that is somewhat more rare is here's 200,000 lines of output about my environment. IP addresses, firewalls, net start output, state output. And I need three hours to run through all of that and work out what your setup actually is. If you write a minimal test case, it will take me 30 seconds to work that out. Also, if you want to get a patch into our code, if you've written a use case for it, sorry, if you've written a test case for it, it means that I will understand what your use case is for that patch. It also tells me that, you know, you've actually tested this patch. And I'm going to have to write a test for this anyway. I'm trying very hard lately to, every time I make a change to actually have a corresponding test. And it'll always succeed, but often enough that we're getting more and more tests for PF. So I'm going to have to write that test anyway. Unless you do it for me, of course. If all of that seems a lot like, you know, work, you can also just pay me and then I will happily do whatever for you. I would like to encourage you all to write tests. You wouldn't be the first to do this. Olivier has already written some tests for IPsec. Olivier was tired of IPsec being broken with certain algorithms. So he wrote a test and now IPsec is not broken. And if it does break, Lee Wen here will shout at them. Well, actually, that's a lie. Lee Wen will politely ask them to fix it, which is very nearly just as good. Also, somebody's working on a Google Summer of Code project right now, and I'm totally blanking on their name, to write more tests. So the objective of that project is to write firewall tests in such a way that we can use all three of our firewalls. Because right now, PF is the only firewall with tests. So, you know, if you need to choose a firewall, choose IDFW because it's great and it doesn't need tests. Or possibly choose PF and send me tests or send me test cases rather. I think I've covered pretty much everything. So you have questions. Do you have heckles? Do you have bags of money to throw at me? Yes, bags of money. Also good. Right. So the question is about user run testing for existing rule sets. So I think there's two aspects to this. One of the tests I did talk about is also a product of the 2017 Google Summer of Code project. So you wrote some tests to test the PFCable parser. There's quite a lot of code in user space that parses rules. And we have had some regressions in there where we accidentally broke previously accepted syntax. So there are some tests there that take some rules, parse them, and have the PFCable spit them back out. And then we compare to see, you know, did it actually read what we expect? So that's one part of it. But that's straight syntax checking. That's loc functionality checking. If you're interested in, you know, we have a rule set and we want to know, does it actually do what we expect it to do? You could write very similar tests to these. You would want to start some jails, probably one or well, probably between two and three or between one and two, depending on how you do it. You want a jail that pretends to be your firewall. You want to put some interfaces in it. And you could rename your interfaces so you can literally reuse your rule sets, or you can just define aliases and then fill them out differently. And then you can inject traffic and watch at the other end of the interface is to see what traffic goes through and what traffic does not go through. The test cases I write, I typically try to restrict very much to testing one aspect of the functionality at a time, both because that makes for shorter and easier tests and makes for shorter and easier debugging sessions. But you could certainly imagine writing a test like this with, I don't know how many interfaces your firewall has, but let's say five different interfaces with a bunch of different networks and some V-lines attached to it. You should be able to replicate that basically entirely as a V-net J-0 and then some traffic at the end see what happens. It should be entirely possible to do. Any other questions? Yes. So you might agree that our multitask code could use the tests. And I thought a lot about using this framework. It was part of the problem is that I really want to write C code for these tests because they are the most important for talking on this. Do you have any plan on knowing any working vent or like the equivalent for the J exactly? So like perfected JL commands that you use? So the question is, would you like some tests for this for multicast? And because multicast testing involves to a large extent socket options that need to be set, Mark would like some C codes to do this in. Yes and no. You can write ADF tests in C, but I don't know if that's what you actually want to do because I think you'll want to write some shell code to build a test setup because your multicast tests might involve multiple hosts depending on which side of things that you're testing. Probably thinking about this live, you might want to see it as two different sets of tests. You have the tests for the Cisco layer interface, so between user space and the kernel. And those you probably just want to write with ADFC and then you can set socket options and you can test all sorts of things. Where maybe you want to run your test program inside the VNet Jail. That's an interesting question, we should talk about that. The second part of what you might want to do is the network side of multicast. When we send IGMP join and leave messages and for that you probably want to do the shell thing where you set up a VNet Jail and you trigger it to do things either with some test programs, also even from Python because you can do some sets of all from Python just as well as you can from C. Or you know whatever other scripting language takes your fancy, I understand that not everybody is a huge Python fan. So you could do things like that. Combining them gets slightly tricky, but I'm sure that it's possible. To get on a different course there is a lot of our tools really want to be split up into a binary and a library. So if we would tear up the Jail commands and the JXX commands in a very thin binary wrapper around the library, that would make that problem a lot easier as well. But I'm going to assign that work to someone else. Any other questions? So the question is can I take multiple of these Jails and plug them together to simulate a network? Any answer is yes. You can simulate arbitrarily complex networks with arbitrary number of routers in between. So what you would do is you would create Jail1, Jail2, Jail3, you would create a bunch of ePair interfaces and then you would use an ePair interface to create a link from JailA to JailB and from JailA to JailC and from JailB to JailC and from JailC to A and B. Any topology you like, you can create them in there, you can set routing options or routes inside that Jail because perhaps one example or one advantage of this that I didn't really go into that much is, well, is this. It is trivial to execute commands inside the Jail. You just go JXX, JailName, do the commands. So you can go JXX, Alcatraz, routes, add default, whatever. You can set whatever route you like in those Jails, you can set whatever IP address you like. They don't need network connectivity because you're not interacting with them over the network. You're just giving them orders from the app side which makes it really, really easy to build setups like that. So the answer, you know, to make a very long answer short, yes, absolutely you can do that. Anyone else? Any heckling? We've gone through the questions, no heckling? When are you updating PF? Thank you, thank you. If you want us to write tests for you, why don't you update PF and then we'll really want to write tests for you. So the OpenBSD people's head on is why don't you update PF? The reply to that is I would like to but my users will kill me when I drop performance by 80%. Actually, 80% is slightly exaggerated. So I did some work on PFCync performance because our PFCync used to contain all the single mutics which is roughly the situation that OpenBSD is in right now. I saw no more than, well, no more. I saw 1.4 million packets a second through that without PFCync. So our PFCync is closer to 3.83.9 million packets a second. There's still some work to be done. There's a very vague plan for, you know, if ever I have the time and motivation to see what we can do about importing OpenBSD's PFCync and helping them or us working, collaborating I think is a better way to describe it to change the walking so that it will scale OpenBSD's working on this and I'm sure they'll get there but I'm also sure that it's a hard problem and it's not something they're going to do in the next two weeks. So without heckle return, the trouble... I thought you were telling me just this morning we were going to commit the lockless system calls tomorrow. I mean, Theo's hiking for two weeks. What could go wrong? You won't get in trouble for at least two weeks. But we don't have tests, so we... So it is about lunchtime, so you have a choice now. I can rant all about one more example of a test or I can let you all go and define... oh sorry, one more question. I was just going to say as a networker, please fill the love of God, nobody gets the right idea to run these against production. So the amazing disclaimer here and that is a legitimate point is don't run these tests on your production machine. Maybe run them before you take it into production but don't run them on your production machine. Certainly don't run them when your sysadmin can get at you because your sysadmin will get at you and you will have a bad day. I try to take care to use network, something that's that will not conflict with anything existing. But don't do that on a production system, do it on a development system. So that's what happens after a bunch of people are running production systems, you all know my system is still valid. So why not run tests like this once a week, once a month from 2 to 3 am, whatever works for your particular situation. Because it is most fun to page the guy running your switches between 2 and 3 am. That's when they like to get woken up in shit all the time. Just stop doing online network because I still need to cross. I'll punch you, I'm good at it. You're on online network. You know what I'm saying, you were careful to avoid IP conflicts. You know the janials there, they are separate IP spaces. But for my tests I tend to use the host to inject packets from. So the host will have a route to some of these subnets. So if they conflict, if I happen to use a subnet that you're using in your network, you shouldn't be because they're test subnets. But if you're using them then yes it will break your routing. The typical use case for these tests is very much you're developing and you're doing this during development. Can I give you a hand? Sure. Look up the NSA's AS, they have a big blob of addresses, just choose those, you'll never even play. It's been around for years. How use are they in 1918, everybody uses those. To use the NSA's subnets because they have a big subnet range and it's not public on the internet. But I make it a policy to not pick fights with people who have an answer for you and what are me. So Bjorn? There are official young asses next to a network that you use there. Yes. How easy is it to convert legacy tests that people have like shellcups and everything into the framework? I've, so the question is how easy is it to convert legacy tests that people have into this framework? And the answer is that I don't know. Because I've never tried. I wouldn't think that it's terribly hard. I think that depending on how those tests are built, you get to throw away a lot of your code and just start the jail, set some configuration in its sense of traffic. I honestly don't have a better answer for that. I don't have a lot of feedback that you might be able to find in all kinds of cases. This is a source tool somewhere from the old days. Some of the memory types that you leverage mean that they've never been used in the past. They've never been, you know, 80F wasn't around that time. Suddenly a pointer and I will add up to my ever growing to-do list. I'll give you five more answers. Yes. One comment to the addresses and networks. In the OpenBSD tests, I can pass the network settings via the environment to the test. So everyone can set up their own network environment and the reaction of other people using my test was give me your IP addresses, otherwise it's too complicated. Yeah, so yeah, that's a good point. I've chosen not to do that to keep it just easy. I like that. Well, let's wander into the next example here. I like that it is extremely easy to read these tests because everything you need to know is here. So when we do things like set firewall rules, you can just read the firewall rule and see everything that's involved in it. If you, you know, what IP address are we setting, you can just see what IP address we're setting. You can certainly make a case that that should be abstracted and it should be a variable and it's really easy to do because it's just a shell script. So we can have some common variables that define IP addresses and some of them are going to use and use those. But it's an extra layer of abstraction and it makes it harder to follow what's actually going on. So a lot of these tests take me 10, 15 minutes to write and then you know half an hour to get the robust and cleaned up. But they are remarkably easy to write and I think in part that is because there is what I feel is the right level of abstraction. Yeah, at least I don't want to advocate for a policy of oh, it's only test code, none of the rules apply. But it is only test code and there is a limit to the amount of abstraction that you need. Okay, that's it. Thank you very much for coming. Thank you very much for the limit of the Hackling Eye goals and enjoy lunch.
 So thank you all for coming. This is the automated firewall testing talk. Before we get started,  let's talk about something nobody cares about, who I am. So I'm Christoph, KP at FreeBSD.org.  I sort of maintain PF in FreeBSD. So if it doesn't work in FreeBSD, it's all the fault  of the openBSD people. But if it does work, it's thanks to my glorious efforts. Or possibly  the other way around. Professionally, I do a lot of embellished Linux projects. I will happily do  FreeBSD or other BSD projects. I got suckered into being on the EuroBSD-Cone Foundation board,  which means, well, that is the reason I've been choosing everyone to submit to the call for papers  for EuroBSD-Cone. I will repeat that. Submit your papers and tutorials to EuroBSD-Cone.  You have until the 26th. No, seriously, do this. Otherwise, it's going to be about 20 slots of  me repeating this talk. I also always feel the need to point out that I'm not for sale,  but I can be rented at reasonable rates. That's where you're failing. You rented at unreasonable.  Well, I didn't say reasonable to whom. Anyway, what is PF? PF is a packet filter. We stole it  from OpenBSD. Imported it from OpenBSD as a word. Last import was a while ago. What we have is an  older version. It's not exactly what's in OpenBSD at the moment. We also have a couple of shiny  things in FreeBSD that are not in OpenBSD, one of which is VNAT and all more about VNAT later.  Another one is that our PF sort of runs multi-core, which means that it's faster.  Note the ER. I didn't say fast. I said faster. They're different concepts. I keep recommending  that people use IPFW, but I saw the survey results from the FreeBSD users and they don't listen to me.  Thank you, OpenBSD. It turns out our users really like your firewall.  On to relevant things for this talk. Why automated testing? I think the obvious answer  is because we're all lazy and testing is boring, repetitive, mind-bombing work. Somebody  should invent a machine that's good at boring and repetitive and mind-bombing work.  Oh, wait. Slightly more seriously, why do we want automated tests to make sure that things work?  It turns out that our users like it when they install something and it works.  They don't get excited when it explodes in new and unexpected ways.  Yeah, users are strange, but what are you going to do? It's also really nice to have a convenient  test case. Does this thing actually work? I'm going to try to roll out something on my network.  Let's test to make sure that it works. Also, about preventing regressions, it works today.  It would be nice if it still works tomorrow. It turns out that with complicated software,  it's really, really easy to accidentally make a mistake that breaks it. There are a lot of ways  things can go wrong and you're not always going to notice immediately when it does go wrong.  Actually, I have some examples here. That might be related to the end of my bio about not talking  to me about IPv6 fragmentation. Fragment handling is broken several times. One of the things  I liberated from OpenBSD was to teach PF to deal with IPv6 fragments. We gather up all of the  fragments, we reassemble, we filter all the reassembled packets, and when we send out,  we have to re-fragment because IPv6 and bottom to you discovery and all of those things.  I'm not going to go into that. The main point is that it's been broken several times. The first  time it broke on December 12th of 2016 and we noticed October 25th of 2017, or rather we fixed it  October 25th of 2017. There's a pretty long time for it to be broken. The problem in the first case  was that we checked the packet size before the P-fill hook, so before we went into PF. PF didn't  have the chance to re-fragment before it got transmitted. The stack looked at it and went,  yes, but this packet is way too big. Let me send an ICMP packet too big error back to a host who  has no idea what you're talking about because he sent much more packets. The second one is a  better example. Well, they say better. It would have been better if we didn't break it, but  it's software we're going to be breaking things. It was broken as a result of a security fix. If  that number doesn't mean anything to you, basically the problem was that the way we handled packet  fragments could be exploited to trigger worst case performance, which would make things very,  very slow and your machine would not be doing anything else other than sitting there trying  to reassemble packets that were designed to be very difficult to reassemble. That got fixed,  except it accidentally broke V6 reassembly. Oops. But, you know, Bjorn, nobody cares about IPv6,  right? He's in the back. That's why I'm brave enough to say this. It broke. The tests noticed  this immediately. Only it was the F test, so everybody just went, it's Kristof's fault. We  don't need to look at this. I looked at it, managed to find a fix for it, and it only took two weeks.  Now, two weeks is longer than we would like it to be broken, but given the choice between being  broken for two weeks and being broken for nine months, I think it's a fairly obvious choice.  For additional fun, that bug was one of these very strange bugs. When I started  detracing things, it went away. I've actually got the code here. I'll give you a few seconds  to try to work out what the problem was. I have Jeopardy music, and I can't figure out  how to make it play. So just imagine the Jeopardy jingle, and I'll give you a few more seconds to.  Thank you very much, Paul.  So we've all spotted the bug, right? So I don't need to be embarrassed that it took me  an embarrassingly long time to figure it out. I can show you the patch. Does that help?  Well, going back to the code, basically, we hash the fragments into buckets now to speed up the  lookup. So we allocate hash key, which means to be the size of two IPv6 addresses and another 32  bits from fragment ID, except the size of I6 adder is 128 bits, so mumble number of bytes,  times four, because you win 32t. And then we hash over the entire size of the array. So we grab some  random stack garbage, include that in the hash, and then stick it in the buckets. And if your  random stack garbage is different on every entry into the function, you're going to hash your  fragments into different buckets, and you're never, ever going to reassemble them. Unless,  of course, you detrace Frag6 input, and then your stack garbage is, you know, whatever's left behind  by detrace, so it's predictable. So it all hashes back into the same buckets, and it works.  I will not point to guilty people, mostly because I don't actually remember whose fault it was.  I just remember being annoyed at figuring this one out. So, you know, we fixed it, and that was awesome.  I hope I've convinced you of, you know, tests are a good idea. What do we want out of these tests?  I want them to be easy to write for two reasons. The first and most important one being, you know,  I'm lazy, and I want things to be easy. But another one is that I would like more people to write tests.  I'll get back to this, but when I get bug reports from users about PF issues, the vast majority  of the time I spend on trying to fix them is not looking at the code, debugging it. It's not  writing a patch. It's, what the hell are you doing? What's your setup? And, you know, given your setup,  what goes on to make it break? So if you want me to fix a bug for you, well, not the best way.  The second best way is to give me the test case. Because I have to write the test anyway. The very  best way to get me to fix a bug for you, of course, is to pay me. So anyway, we want them to be easy  to write. We want them to be easy for everyone to run them, which means that hopefully people will  run those tests before they commit something and will notice that they've, that they broke something  before they commit it. That has a number of advantages, you know. They work smarter because  they didn't break, they didn't commit something that broke stuff. We don't break things for our  users and we have some extremely brave users who run, you know, large chunks of the internet  or at least large chunks of video across the internet who like using latest and greatest.  It would be nice if we didn't break things for them. But it also means, you know, from a strictly  selfish point of view that you're going to notice that you broke something rather than, you know,  committing something and then me getting shouted at for the PF test breaking again.  You know, I might be very selfish. We also want them to be fast to run. That sort of fits in with,  you know, wanting everyone to run them. Because if it only takes two minutes to run,  they'll run them. If it takes four hours to run, they won't. We also want to integrate with the  automated testing framework, which is an automated testing framework in QA, the automated test for  PF or built around that. They are wonderful tools that we came up with ourselves and totally didn't  steal from that PSD. Because if they integrate with that, they will turn up in our Jenkins instance.  This is a very old screenshot. We've got more tests now, but this one 7,300 tests.  And in this case, well, two failures. That's two more than we'd like, but at least most of the  testing works. These get run often. Lee-Wen, you know, happened to know off the top of your head  how often they get run. That's every couple of hours at the worst, right? So every couple of hours,  so ideally when you commit something and it breaks, there will be evidence of this a couple  of hours later. With any luck, you'll actually look at this and notice that, oh, wait, I committed  something and everything is broken now. Lee-Wen very kindly writes weekly reports about the  state of our tests and sends them to 3DSD testing at and a couple of other mailing lists. So there's  someone actually following up on this. So how do we build these tests? The obvious way to do this  is we get a bunch of hardware and get a server, we get a switch, we get another server, we send  packets from server A to server B and maybe we get some replies and we look at them. And that's  great, unless we want to forward, okay, server, switch, server, switch, server, and we send packets  through that. Unless, of course, we want to do pf sync or cart or more complicated setup, so even  more hardware. That's good advantages. You can do things like performance testing when you have  real hardware, but it's got a couple of downsides as well, like, you know, this is a firewall test.  One of the tests we'd like to do is block all traffic. Kind of hard to remotely configure a  system if you block all traffic. You can go and, you know, configure it over serial lines, but then  you need more hardware and more machines to talk to and all of those things. We want to run the  same 3BSE version on all of these systems. Okay, we'll net boot them, but yeah, firewall. Pretty  annoying that, you know, you net boot, you mount your root file system over NFS and then you block  all traffic and your root file system goes away. There are ways around this, but it, you know,  it becomes complicated. Then what happens if the machine panics? You need to have control code that  will manage or will understand that, hey, this machine panics. I should collect the information  out of it and I should give it a hard reset. Oh, by the way, we need some more hardware to do this.  You can imagine complicated setups. Where does all of this hardware live? I have too much old  hardware at home, but not enough hardware to build all of this, so it would have to live somewhere  else. But if we need five machines and serial interfaces and power switches, we can't really  expect all developers to have this setup. So how do other people write tests for this when they  can test their tests? There are advantages to this setup. Alexander did a great talk about it.  One of the advantages is you can do meaningful performance testing against it,  but I'm sure he'll agree that it's a complicated setup that needs some care and feeling.  So, you know, that's a downside. The obvious way around all of these is, you know, virtual hardware.  We've got Beehive. Beehive is great. We can run any number of machines, well, you know, low number  of machines, but enough to build our setups in Beehive. And that's actually the approach taken  by a student in Google Summer of Code 2017 to try to build this. We've still got some issues, you  know. We walk all traffic and oops, we've got a serial port or we can pretend to have a serial  port and we can make that work. Great. But our automated tests on the IDOP3BAS.org run in Beehive.  So we have to run Beehive inside Beehive. And I don't actually know if that's possible, but if it  is, it's bound to be slow. And the darker, darker is good enough for everyone in the cloud. Beehive,  Beehive should be good enough for you. Yeah. Another issue with that, and that's where the  what the Google Summer of Code student ran into is it is really, really annoying to build the  images during your build run. It takes forever and you need a lot of storage and all of those things.  You've still got issues coping with panics. It's still possible, but it's still annoying.  It's also really slow because you're waiting for emulated, well,  virtualized machines to boot. Best case 20, 30 seconds depending on what hardware you've got.  It's an improvement, but it's still annoying. I keep telling you what solutions won't work,  what will work. Well, VNet or network stack virtualization hands up everyone who knows  and loves VNet. Okay, for those of you who don't know, it's network stack virtualization. Here  we go. I've explained that. It's associated with a jail and it basically turns a jail into  a pretend virtual machine. They all run the same kernel, but every jail can have its own network  interfaces. It can have its own IP address, you can set whatever IP address you like.  You can also run your firewall inside the jail. For all intents and purposes, you have your own  machine. Of course, it's not 100% the same as running on actual hardware, but it's 99% the same  code. PF supports this. Actually, to be entirely fair, all of our firewalls support this now.  I'm mostly focused on PF because I spent some time arguing with PF about, you know, maybe you  should try not panicking when we run you in a jail. After a while that sank in and now it usually  doesn't panic when you run it in a jail. The usual disclaimer there, right? I do development. I  look at bugs that users report. So I spend all of my time looking at bugs. And once I've fixed one  of them, it goes away, but I go on and look at the next bug. So there tends to be a perception of  everything is broken and everything is horrible. Turns out that as a user, and I actually use  PF in the net jails myself on one of my machines, and it works just fine.  I don't actually think that machine's ever panicked over one of those things. So it works  a lot better than I get the impression. But bugs will happen. Please report them. As of 12.0,  I will try to help you if you run PF in a jail and it panics for you. If you do it in 11, I will  just point and laugh. Unless you threaten to pay me in which case I will quietly laugh and help you.  Anyway, I've been talking about this forever now. How do we actually run a VNet jail with  its own IP stock? I bet it's really, really hard. That's it. We start the jail. We give it a name,  Alcatraz in this case, because I think that's hilarious. We tell it that we want it to be a  VNet jail and we want to persist it. Persist just tells the jail that the jail should keep existing  even if there are no processes in it. Usually, that's a silly thing to want because jails isolate  processes. If you don't have a process, why would you bother to isolate it? But in this case,  the jail has an IP stack and you could forward packets. So that's it. We're done here.  All right. Well, to be entirely fair, that's not all that there's to it. We also need some network  interfaces. It's all easier to test network things when you can communicate with the outside world.  So to do that, we've got something called ePair. ePair is actually really simple. It's two network  interfaces with a cable between the mixup. The cable doesn't exist and the network interfaces are  virtual, but it's two network interfaces with a cable. So you get an ePair zero, which has an A  and a B side and A and B are linked with each other. And if you stick a packet into A, you can  add to B and the other way around. The usual way you would use them is you would give an IP address  to the A interface. You would stick the D interface in the jail. So same thing. We start the jail with  VNAT and oh, by the way, you have an interface called ePair zero B. And then we can actually  execute a command inside the jail. And I still haven't fixed my slides. This mix misses Alcatraz.  So execute inside Alcatraz. If can fake ePair zero B, there's an IP address, and then you can  ping it. And that's that's all there's to it. And we're already testing our IP stack or testing  that we can send by CMP Echo requests and get replies. It's not terribly exciting, but it  actually runs a whole lot of code. And we we had, you know, any number of panics around this,  because it turns out that setting up and shutting down my IP stack or a network stack is something  that we only ever did when we booted them, when we rebooted. And if you do 20 of those,  rather than one, and you do several of them one after the other, or you do them at the same time,  you run into bugs that you never ever run into in other situations. But that's basically all we  need to do. So, so let's take a look at writing a test for this. That's not all that there's to  the test, but you know, the entire test case doesn't fit all my slides. So what we do, we use the  automated testing framework, which again, you know, we totally enhance ourselves and in no way stole  from the SD includes some utilities, declare a test case, we have a v4 test case cleanup just says  that there is a cleanup function and you should call it after you're done with the tests.  Test head just describes the test. So, in the description, basic password test for IPv4,  we require a user. For some reason, people think that security is important. And maybe we shouldn't  allow people to spin up new IP sacks and network configuration unless they're actually, you know,  root. The only thing this does is make sure that when you run the test as more brews,  this test will just skip because it can't possibly succeed. The interesting bits is test bolly.  And that's, that's all there's to it. Although admittedly, I've stripped out a bunch of tests,  but this is a functional test. This is useful. So we have general initialization codes. The  initialization code doesn't do a whole lot of exciting work. It just goes and checks that  VNet feature is available in your kernel because if it's not, there's no point in running this test.  It also checks that PF is available because, you know, testing PF when the PF module is not  loaded again, not terribly useful. We create an EPIR. We assign an IP address to it, just like  we discussed earlier. Create a jail, help a script to create a jail, create a jail arbitrage, give us  an interface, set an IP address in the jail, and then just, you know, basic sanity check.  So ATF check will just go and verify that the next command will return exit state to zero.  We ignore the standard about it because we don't care what it says as long as it returns with zero.  One thing only, and the time out of one second to that IP address. Hopefully, you know, the jail  will apply and everything will be well. Next up, we enable PF. PF by default passes everything,  so this should still work. So, yeah, check exit state to zero, and then we're going to walk everything.  So we set rules in the jail, walk input. This is a simple helper function. It just makes life  easier when you can write this rather than write it to a text file or echo it and pipe it into the jail.  It's much shorter to write this. Remember, I'm lazy. So we check that we get an error output.  We still don't care about standard outs, and that this thing will fail. Something about this,  as we'll see later when I show some output, is this entire test run takes 1.2 seconds. The  entire second is taken up by this because we spend the second waiting for a reply that will never,  ever come. I mean, it should unless we totally broke it something. Everything else is 200 milliseconds  to top start the jail. So, you know, maybe we should do some boot time improvements so we can  get things to boot faster in beehive, but I don't think we're ever going to get to the point where  we can boot an entire machine in 200 milliseconds. We'll actually boot it, do some work in it,  and shut it down in 200 milliseconds. But we can do that with VNetJails, which makes it a lot easier  to run all of these tests. I say a lot. We have some tests for PFRite now, not that many, but enough  that they catch bugs, and they only tend to take about a minute. So, cleanup, pretty easy. There's  a utility function. The other utility functions, whenever they create an interface or a jail,  helpfully keep track of this in a temporary file. So, the cleanup function just goes and looks,  what's in the temporary file? We've created the EPAR0A, delete EPAR0A. We've created EPAR1A,  delete EPAR1A. We've created the Alcatraz jail, okay, remove the Alcatraz jail, just so that we  clean up everything so that the next test isn't going to conflict with whatever we've left behind.  And then this is just ATF infrastructure. If you're really, really interested in how all of the ATF  things work, there's an article in the FreeBSD Journal about this. So, you can take a look at  that. The FreeBSD Journal is now free, so you can just get onto the FreeBSD Foundation website  and go take a look at it. What business output? Well, there we go. We do, you know, sudo because  we need to be root to run this. Obviously, QA test, password V4. Password V4, the test passes.  Okay, it works. 1.2 seconds. We even have some results. We can go and look at, you know,  what sort of output did this test generate? Which is useful. Although, to be entirely fair,  usually when the tests fail, I will start off by adding SIT-X to the test body so that I can see  what commands it's all executing. And we'll probably end up setting a sleep 3600 somewhere  so that it doesn't clean up the jail so I can go look at, you know, we have the test set up and  now I can send some things and I can TCP dump and I can, I might even detrace or look at some other  statistics or some other counters to, you know, what is actually going on here. But they run,  and these things run automatically. These things run several times a day. So, the next time somebody  breaks PF so that it, you know, rather than dropping packets, just lets them all through,  which, you know, how would you notice unless you test? And it is kind of important for your firewall  to occasionally drop packets and maybe even the right packets. Anyway, next example is a much,  much more complicated example because it's a test for PF sync. For those of you who don't know,  PF has this wonderful feature where you can synchronize states between two machines. So,  the typical use case for this is when you have a gateway machine, you just have a second one.  And if one of them dies or you take it down for an upgrade, the other one will take over.  But you wanted to remember states because if your TCP session is running, you would like it to keep  running. So, PF sync will send state from one machine to the other so that it can't notionally take  over. So, very basic tests for this. We create a couple of interfaces because, you know, we'll  have two machines, we'll have two jails, jail one, jail two. We'll have interfaces to all of those  jails and we'll have a link between them so that they can exchange states. Hence the e-pair sync,  e-pair one, e-pair two, jail one, jail two. I got tired of the naming my jails after this jail's  joke here, as you can tell. Give them the two interfaces, configure interfaces in jail one,  enable PF sync, synchronize over, sync a device, send some up, max update if I remember correctly,  that just tells PF sync that the moment you have a single update ready, send it out because usually  it tries to gather up a whole bunch of updates before it starts to send packets off to the other  side, send it up, do the same thing with the second jail. So, there's a lot going on there but it's  basically all related to how do I actually configure PF sync to do something. Enable PF in both of the  jails, set IP addresses and then we send the ping. This ping will travel through the first jail  and after some time that state will get synchronized to the other jail so we can go ask jail two,  you know, what states do you have and do you have an ICMP state for this particular IP address?  Should have one because we send an ICMP echo through it. We go look for it, make sure that it  matches what we expected to do and if we don't find it, the test fails. So, very primitive, well,  it's very basic test. It's set up PF sync, make sure that the one state we deliberately create  in it ends up on the other side. It's primitive but it tests a lot of the functionality of PF  sync. It's basically, you know, the basic functionality of PF sync has to work for this test  to work. I did some work on PF sync recently and, you know, this test failed very often. It also  demonstrates that, you know, it is actually that hard to test something that needs two machines.  You could test things with even more machines. I have tested for things like fragment handling as  well. I don't have examples in the slides but there's one where, you know, we send a bunch of  fragments at it and we test what happens. So, these are all fairly simple tests because we send,  you know, correctly format packets. There are some tests that do things like use KB, the PyFold  tool to build arbitrary packets to send malformed packets which is another fun thing to do to your  network stack. Yeah, I'm trying to remember, I think for some of the recent security issues,  I also wrote something to send specifically malformed packets to trigger something that would  otherwise have been a panic but now is no longer a panic because Alexander fixed the bug for us.  Was it you or Sasha? Sasha, please. Okay. So, I've set the credit to the wrong side then.  Anyway, for those of you interested in the math tests, the PF tests live under user source,  assuming that's where you install your source. Test sys netpvlpf to mirror where our PF code lives,  our PF code lives user source sys netpvlpf. When they get installed, they're in user test sys netpvlpf.  If you want to run them, sudo package install qr and then go to, well, actually there we go,  install qr, installs kb because that tests a few more things. Kld loads pf sync which will implicitly  load pf. Go to user tests sys netpvlpf and then probably sudo qr test unless you've done all of  this as root already. There's a lot more information in the journal. Well, there's information on how  to write ATF tests. You should all be writing tests. Yes, even you open BSD people, you should be  writing tests for free BSD. Just copy that. Well, there is actually a lot to be said for we should  take a look at some of your tests. Are you threatening your payers to do it? No. I'm threatening you  with my internal gratitude. So there's an article in the free BSD journal, the March April edition  of this year, how do you write some test things? Hopefully one of our documentation people will  actually go and add some of this to the handbook as well. Benedict's working on this. Benedict's  not here but I'm going to apply social pressure anyway. Important things, well, if I still haven't  convinced you now what we need is some sort of deeply profound quote by someone famous.  I didn't have the budget for that. So you get me telling you, write tests. Tests are good. I don't  know about how profound that is but it is actually true. Tests are wonderful. What's in that for you?  You can prototype some of your setups because sometimes you try to do things and they don't  actually work. You can prototype a lot of networking setups with just jails, which means that you can  do it in a virtual machine all playing over whatever ocean you like to fly over. It also means  that if you write a test for this and you give us the test case, we'll include it and we'll run it,  which means that your use case is not going to break or when it breaks, we're going to know.  And we're going to know pretty much immediately after we commit the breaking change rather than  three years later when you finally get around to upgrading and then notice that you're careful  use case no longer works. And we broke it three years ago. Developers have, well, I might be  projecting here, it might just be me, but in general developers have a very short attention  span. So if you notice that they broke something a couple of days after they did it, you're a lot  more likely to get them to care than when you notice three years later. What else is in it for  you? Right. Yes. As I mentioned before, write a test case because it makes it really, it makes it  much easier for me to fix your bug. I'm lazy. Make it easy for me. Recruiting bugs is hard.  I need to understand your entire test setup. And I often get, you know, well, I get two types of  bug reports usually. I get the one that says it's broken. Please fix it with no information in it  as to, you know, what is broken? What is your test setup? How do you break it? How often does it  break? In what way does it break? No information whatsoever. The other type and that is somewhat  more rare is here's 200,000 lines of output about my environment. IP addresses, firewalls,  net start output, state output. And I need three hours to run through all of that and work out  what your setup actually is. If you write a minimal test case, it will take me 30 seconds to work  that out. Also, if you want to get a patch into our code, if you've written a use case for it,  sorry, if you've written a test case for it, it means that I will understand what your use case  is for that patch. It also tells me that, you know, you've actually tested this patch. And I'm  going to have to write a test for this anyway. I'm trying very hard lately to, every time I make a  change to actually have a corresponding test. And it'll always succeed, but often enough that  we're getting more and more tests for PF. So I'm going to have to write that test anyway. Unless  you do it for me, of course. If all of that seems a lot like, you know, work, you can also just pay  me and then I will happily do whatever for you. I would like to encourage you all to write tests.  You wouldn't be the first to do this. Olivier has already written some tests for IPsec. Olivier  was tired of IPsec being broken with certain algorithms. So he wrote a test and now IPsec  is not broken. And if it does break, Lee Wen here will shout at them. Well, actually, that's a lie.  Lee Wen will politely ask them to fix it, which is very nearly just as good. Also, somebody's working  on a Google Summer of Code project right now, and I'm totally blanking on their name, to write  more tests. So the objective of that project is to write firewall tests in such a way that we can  use all three of our firewalls. Because right now, PF is the only firewall with tests. So, you know,  if you need to choose a firewall, choose IDFW because it's great and it doesn't need tests. Or  possibly choose PF and send me tests or send me test cases rather. I think I've covered pretty  much everything. So you have questions. Do you have heckles? Do you have bags of money to throw at me?  Yes, bags of money. Also good.  Right. So the question is about user run testing for existing rule sets. So I think there's two  aspects to this. One of the tests I did talk about is also a product of the 2017 Google Summer  of Code project. So you wrote some tests to test the PFCable parser. There's quite a lot of code  in user space that parses rules. And we have had some regressions in there where we accidentally  broke previously accepted syntax. So there are some tests there that take some rules,  parse them, and have the PFCable spit them back out. And then we compare to see, you know,  did it actually read what we expect? So that's one part of it. But that's straight syntax checking.  That's loc functionality checking. If you're interested in, you know, we have a rule set  and we want to know, does it actually do what we expect it to do? You could write very similar  tests to these. You would want to start some jails, probably one or well, probably between two and three  or between one and two, depending on how you do it. You want a jail that pretends to be your firewall.  You want to put some interfaces in it. And you could rename your interfaces so you can literally  reuse your rule sets, or you can just define aliases and then fill them out differently.  And then you can inject traffic and watch at the other end of the interface is to see what  traffic goes through and what traffic does not go through. The test cases I write, I typically try  to restrict very much to testing one aspect of the functionality at a time, both because that makes  for shorter and easier tests and makes for shorter and easier debugging sessions. But you could  certainly imagine writing a test like this with, I don't know how many interfaces your firewall has,  but let's say five different interfaces with a bunch of different networks and some V-lines attached to it.  You should be able to replicate that basically entirely as a V-net J-0 and then some traffic  at the end see what happens. It should be entirely possible to do. Any other questions? Yes.  So you might agree that our multitask code could use the tests. And I thought a lot about using  this framework. It was part of the problem is that I really want to write C code for these tests  because they are the most important for talking on this. Do you have any plan on  knowing any working vent or like the equivalent for the J exactly? So like perfected JL commands that you use?  So the question is, would you like some tests for this for multicast? And because multicast  testing involves to a large extent socket options that need to be set, Mark would like some C codes  to do this in. Yes and no. You can write ADF tests in C, but I don't know if that's what you actually  want to do because I think you'll want to write some shell code to build a test setup  because your multicast tests might involve multiple hosts depending on which side of things that you're testing.  Probably thinking about this live, you might want to see it as two different sets of tests.  You have the tests for the Cisco layer interface, so between user space and the kernel.  And those you probably just want to write with ADFC and then you can set socket options and you can test all sorts of things.  Where maybe you want to run your test program inside the VNet Jail. That's an interesting question, we should talk about that.  The second part of what you might want to do is the network side of multicast. When we send  IGMP join and leave messages and for that you probably want to do the shell thing where you set up a VNet Jail  and you trigger it to do things either with some test programs, also even from Python because you can do some  sets of all from Python just as well as you can from C.  Or you know whatever other scripting language takes your fancy, I understand that not everybody is a huge Python fan.  So you could do things like that. Combining them gets slightly tricky, but I'm sure that it's possible.  To get on a different course there is a lot of our tools really want to be split up into a binary and a library.  So if we would tear up the Jail commands and the JXX commands in a very thin binary wrapper around the library,  that would make that problem a lot easier as well. But I'm going to assign that work to someone else.  Any other questions?  So the question is can I take multiple of these Jails and plug them together to simulate a network?  Any answer is yes. You can simulate arbitrarily complex networks with arbitrary number of  routers in between. So what you would do is you would create Jail1, Jail2, Jail3,  you would create a bunch of ePair interfaces and then you would use an ePair interface to create a link  from JailA to JailB and from JailA to JailC and from JailB to JailC and from JailC to A and B.  Any topology you like, you can create them in there, you can set routing options  or routes inside that Jail because perhaps one example or one advantage of this that I didn't  really go into that much is, well, is this. It is trivial to execute commands inside the Jail.  You just go JXX, JailName, do the commands. So you can go JXX, Alcatraz, routes, add default,  whatever. You can set whatever route you like in those Jails, you can set whatever IP address you  like. They don't need network connectivity because you're not interacting with them over the network.  You're just giving them orders from the app side which makes it really, really easy to build setups  like that. So the answer, you know, to make a very long answer short, yes, absolutely you can do that.  Anyone else?  Any heckling? We've gone through the questions, no heckling?  When are you updating PF?  Thank you, thank you.  If you want us to write tests for you, why don't you update PF and then we'll really want to write  tests for you. So the OpenBSD people's head on is why don't you update PF? The reply to that is  I would like to but my users will kill me when I drop performance by 80%.  Actually, 80% is slightly exaggerated. So I did some work on PFCync performance  because our PFCync used to contain all the single mutics which is roughly the situation that OpenBSD  is in right now. I saw no more than, well, no more. I saw 1.4 million packets a second through that  without PFCync. So our PFCync is closer to 3.83.9 million packets a second.  There's still some work to be done. There's a very vague plan for, you know, if ever I have the time  and motivation to see what we can do about importing OpenBSD's PFCync and helping them  or us working, collaborating I think is a better way to describe it to change the walking  so that it will scale OpenBSD's working on this and I'm sure they'll get there but I'm also sure  that it's a hard problem and it's not something they're going to do in the next two weeks.  So without heckle return, the trouble... I thought you were telling me just this morning  we were going to commit the lockless system calls tomorrow. I mean, Theo's hiking for two weeks.  What could go wrong? You won't get in trouble for at least two weeks.  But we don't have tests, so we...  So it is about lunchtime, so you have a choice now. I can rant all about one more example of a test  or I can let you all go and define... oh sorry, one more question.  I was just going to say as a networker, please fill the love of God, nobody gets the right idea to run these against production.  So the amazing disclaimer here and that is a legitimate point is don't run these tests on your production machine.  Maybe run them before you take it into production but don't run them on your production machine.  Certainly don't run them when your sysadmin can get at you because your sysadmin will get at you and you will have a bad day.  I try to take care to use network, something that's that will not conflict with anything existing.  But don't do that on a production system, do it on a development system.  So that's what happens after a bunch of people are running production systems, you all know my system is still valid.  So why not run tests like this once a week, once a month from 2 to 3 am, whatever works for your particular situation.  Because it is most fun to page the guy running your switches between 2 and 3 am.  That's when they like to get woken up in shit all the time.  Just stop doing online network because I still need to cross.  I'll punch you, I'm good at it.  You're on online network.  You know what I'm saying, you were careful to avoid IP conflicts.  You know the janials there, they are separate IP spaces.  But for my tests I tend to use the host to inject packets from.  So the host will have a route to some of these subnets.  So if they conflict, if I happen to use a subnet that you're using in your network, you shouldn't be because they're test subnets.  But if you're using them then yes it will break your routing.  The typical use case for these tests is very much you're developing and you're doing this during development.  Can I give you a hand?  Sure.  Look up the NSA's AS, they have a big blob of addresses, just choose those, you'll never even play.  It's been around for years.  How use are they in 1918, everybody uses those.  To use the NSA's subnets because they have a big subnet range and it's not public on the internet.  But I make it a policy to not pick fights with people who have an answer for you and what are me.  So Bjorn?  There are official young asses next to a network that you use there.  Yes.  How easy is it to convert legacy tests that people have like shellcups and everything into the framework?  I've, so the question is how easy is it to convert legacy tests that people have into this framework?  And the answer is that I don't know.  Because I've never tried. I wouldn't think that it's terribly hard.  I think that depending on how those tests are built, you get to throw away a lot of your code and just start the jail, set some configuration in its sense of traffic.  I honestly don't have a better answer for that.  I don't have a lot of feedback that you might be able to find in all kinds of cases.  This is a source tool somewhere from the old days.  Some of the memory types that you leverage mean that they've never been used in the past.  They've never been, you know, 80F wasn't around that time.  Suddenly a pointer and I will add up to my ever growing to-do list.  I'll give you five more answers.  Yes.  One comment to the addresses and networks.  In the OpenBSD tests, I can pass the network settings via the environment to the test.  So everyone can set up their own network environment and the reaction of other people using my test was give me your IP addresses, otherwise it's too complicated.  Yeah, so yeah, that's a good point.  I've chosen not to do that to keep it just easy.  I like that.  Well, let's wander into the next example here.  I like that it is extremely easy to read these tests because everything you need to know is here.  So when we do things like set firewall rules, you can just read the firewall rule and see everything that's involved in it.  If you, you know, what IP address are we setting, you can just see what IP address we're setting.  You can certainly make a case that that should be abstracted and it should be a variable and it's really easy to do because it's just a shell script.  So we can have some common variables that define IP addresses and some of them are going to use and use those.  But it's an extra layer of abstraction and it makes it harder to follow what's actually going on.  So a lot of these tests take me 10, 15 minutes to write and then you know half an hour to get the robust and cleaned up.  But they are remarkably easy to write and I think in part that is because there is what I feel is the right level of abstraction.  Yeah, at least I don't want to advocate for a policy of oh, it's only test code, none of the rules apply.  But it is only test code and there is a limit to the amount of abstraction that you need.  Okay, that's it. Thank you very much for coming.  Thank you very much for the limit of the Hackling Eye goals and enjoy lunch.
Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_0_107.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_108_203.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_204_311.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_312_563.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_564_2423.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_2424_2591.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_2592_2639.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_2640_2975.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_2976_3023.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_3024_4067.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_4068_5987.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_5988_10583.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_10584_10739.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_10740_11783.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_11784_11939.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_11940_13895.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_13896_14015.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_14016_17951.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_17952_19727.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_19728_19787.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_19788_20483.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_20484_20555.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_20556_20675.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_20676_20927.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_20928_20975.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_20976_21359.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_21360_21407.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_21408_21455.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_21456_21503.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_21504_21563.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_21564_21707.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_21708_21887.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_21888_22067.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_22068_22451.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_22452_22499.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_22500_22547.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_22548_22595.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_22596_22643.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_22644_22703.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_22704_22823.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_22824_22883.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_22884_23207.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_23208_23255.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_23256_23495.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_23496_23963.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_23964_25823.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_25824_29291.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_29292_29531.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_29532_30803.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_30804_33407.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_33408_34907.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_34908_37775.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_37776_39239.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_39240_39287.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_39288_39947.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_39948_40535.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_40536_42647.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_42648_44375.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_44376_45359.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_45360_48587.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_48588_49223.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_49224_49739.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_49740_49835.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_49836_49883.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_49884_51371.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_51372_51719.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_51720_56315.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_56316_58223.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_58224_68195.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_68196_68243.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_68244_68291.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_68292_68339.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_68340_69347.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_69348_69395.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_69396_69443.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_69444_69491.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_69492_72059.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_72060_79343.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_79344_81611.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_81612_81659.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_81660_81719.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_81720_81791.png Testing_firewalls_Why_vimage_is_awesome_by_Kristof_Provost_gTyt7KLz1mw_81792_81809.png
So thank you all for coming. This is the automated firewall testing talk. Before we get started, let's talk about something nobody cares about, who I am. So I'm Christoph, KP at FreeBSD.org. I sort of maintain PF in FreeBSD. So if it doesn't work in FreeBSD, it's all the fault of the openBSD people. But if it does work, it's thanks to my glorious efforts. Or possibly the other way around. Professionally, I do a lot of embellished Linux projects. I will happily do FreeBSD or other BSD projects. I got suckered into being on the EuroBSD-Cone Foundation board, which means, well, that is the reason I've been choosing everyone to submit to the call for papers for EuroBSD-Cone. I will repeat that. Submit your papers and tutorials to EuroBSD-Cone. You have until the 26th. No, seriously, do this. Otherwise, it's going to be about 20 slots of me repeating this talk. I also always feel the need to point out that I'm not for sale, but I can be rented at reasonable rates. That's where you're failing. You rented at unreasonable. Well, I didn't say reasonable to whom. Anyway, what is PF? PF is a packet filter. We stole it from OpenBSD. Imported it from OpenBSD as a word. Last import was a while ago. What we have is an older version. It's not exactly what's in OpenBSD at the moment. We also have a couple of shiny things in FreeBSD that are not in OpenBSD, one of which is VNAT and all more about VNAT later. Another one is that our PF sort of runs multi-core, which means that it's faster. Note the ER. I didn't say fast. I said faster. They're different concepts. I keep recommending that people use IPFW, but I saw the survey results from the FreeBSD users and they don't listen to me. Thank you, OpenBSD. It turns out our users really like your firewall. On to relevant things for this talk. Why automated testing? I think the obvious answer is because we're all lazy and testing is boring, repetitive, mind-bombing work. Somebody should invent a machine that's good at boring and repetitive and mind-bombing work. Oh, wait. Slightly more seriously, why do we want automated tests to make sure that things work? It turns out that our users like it when they install something and it works. They don't get excited when it explodes in new and unexpected ways. Yeah, users are strange, but what are you going to do? It's also really nice to have a convenient test case. Does this thing actually work? I'm going to try to roll out something on my network. Let's test to make sure that it works. Also, about preventing regressions, it works today. It would be nice if it still works tomorrow. It turns out that with complicated software, it's really, really easy to accidentally make a mistake that breaks it. There are a lot of ways things can go wrong and you're not always going to notice immediately when it does go wrong. Actually, I have some examples here. That might be related to the end of my bio about not talking to me about IPv6 fragmentation. Fragment handling is broken several times. One of the things I liberated from OpenBSD was to teach PF to deal with IPv6 fragments. We gather up all of the fragments, we reassemble, we filter all the reassembled packets, and when we send out, we have to re-fragment because IPv6 and bottom to you discovery and all of those things. I'm not going to go into that. The main point is that it's been broken several times. The first time it broke on December 12th of 2016 and we noticed October 25th of 2017, or rather we fixed it October 25th of 2017. There's a pretty long time for it to be broken. The problem in the first case was that we checked the packet size before the P-fill hook, so before we went into PF. PF didn't have the chance to re-fragment before it got transmitted. The stack looked at it and went, yes, but this packet is way too big. Let me send an ICMP packet too big error back to a host who has no idea what you're talking about because he sent much more packets. The second one is a better example. Well, they say better. It would have been better if we didn't break it, but it's software we're going to be breaking things. It was broken as a result of a security fix. If that number doesn't mean anything to you, basically the problem was that the way we handled packet fragments could be exploited to trigger worst case performance, which would make things very, very slow and your machine would not be doing anything else other than sitting there trying to reassemble packets that were designed to be very difficult to reassemble. That got fixed, except it accidentally broke V6 reassembly. Oops. But, you know, Bjorn, nobody cares about IPv6, right? He's in the back. That's why I'm brave enough to say this. It broke. The tests noticed this immediately. Only it was the F test, so everybody just went, it's Kristof's fault. We don't need to look at this. I looked at it, managed to find a fix for it, and it only took two weeks. Now, two weeks is longer than we would like it to be broken, but given the choice between being broken for two weeks and being broken for nine months, I think it's a fairly obvious choice. For additional fun, that bug was one of these very strange bugs. When I started detracing things, it went away. I've actually got the code here. I'll give you a few seconds to try to work out what the problem was. I have Jeopardy music, and I can't figure out how to make it play. So just imagine the Jeopardy jingle, and I'll give you a few more seconds to. Thank you very much, Paul. So we've all spotted the bug, right? So I don't need to be embarrassed that it took me an embarrassingly long time to figure it out. I can show you the patch. Does that help? Well, going back to the code, basically, we hash the fragments into buckets now to speed up the lookup. So we allocate hash key, which means to be the size of two IPv6 addresses and another 32 bits from fragment ID, except the size of I6 adder is 128 bits, so mumble number of bytes, times four, because you win 32t. And then we hash over the entire size of the array. So we grab some random stack garbage, include that in the hash, and then stick it in the buckets. And if your random stack garbage is different on every entry into the function, you're going to hash your fragments into different buckets, and you're never, ever going to reassemble them. Unless, of course, you detrace Frag6 input, and then your stack garbage is, you know, whatever's left behind by detrace, so it's predictable. So it all hashes back into the same buckets, and it works. I will not point to guilty people, mostly because I don't actually remember whose fault it was. I just remember being annoyed at figuring this one out. So, you know, we fixed it, and that was awesome. I hope I've convinced you of, you know, tests are a good idea. What do we want out of these tests? I want them to be easy to write for two reasons. The first and most important one being, you know, I'm lazy, and I want things to be easy. But another one is that I would like more people to write tests. I'll get back to this, but when I get bug reports from users about PF issues, the vast majority of the time I spend on trying to fix them is not looking at the code, debugging it. It's not writing a patch. It's, what the hell are you doing? What's your setup? And, you know, given your setup, what goes on to make it break? So if you want me to fix a bug for you, well, not the best way. The second best way is to give me the test case. Because I have to write the test anyway. The very best way to get me to fix a bug for you, of course, is to pay me. So anyway, we want them to be easy to write. We want them to be easy for everyone to run them, which means that hopefully people will run those tests before they commit something and will notice that they've, that they broke something before they commit it. That has a number of advantages, you know. They work smarter because they didn't break, they didn't commit something that broke stuff. We don't break things for our users and we have some extremely brave users who run, you know, large chunks of the internet or at least large chunks of video across the internet who like using latest and greatest. It would be nice if we didn't break things for them. But it also means, you know, from a strictly selfish point of view that you're going to notice that you broke something rather than, you know, committing something and then me getting shouted at for the PF test breaking again. You know, I might be very selfish. We also want them to be fast to run. That sort of fits in with, you know, wanting everyone to run them. Because if it only takes two minutes to run, they'll run them. If it takes four hours to run, they won't. We also want to integrate with the automated testing framework, which is an automated testing framework in QA, the automated test for PF or built around that. They are wonderful tools that we came up with ourselves and totally didn't steal from that PSD. Because if they integrate with that, they will turn up in our Jenkins instance. This is a very old screenshot. We've got more tests now, but this one 7,300 tests. And in this case, well, two failures. That's two more than we'd like, but at least most of the testing works. These get run often. Lee-Wen, you know, happened to know off the top of your head how often they get run. That's every couple of hours at the worst, right? So every couple of hours, so ideally when you commit something and it breaks, there will be evidence of this a couple of hours later. With any luck, you'll actually look at this and notice that, oh, wait, I committed something and everything is broken now. Lee-Wen very kindly writes weekly reports about the state of our tests and sends them to 3DSD testing at and a couple of other mailing lists. So there's someone actually following up on this. So how do we build these tests? The obvious way to do this is we get a bunch of hardware and get a server, we get a switch, we get another server, we send packets from server A to server B and maybe we get some replies and we look at them. And that's great, unless we want to forward, okay, server, switch, server, switch, server, and we send packets through that. Unless, of course, we want to do pf sync or cart or more complicated setup, so even more hardware. That's good advantages. You can do things like performance testing when you have real hardware, but it's got a couple of downsides as well, like, you know, this is a firewall test. One of the tests we'd like to do is block all traffic. Kind of hard to remotely configure a system if you block all traffic. You can go and, you know, configure it over serial lines, but then you need more hardware and more machines to talk to and all of those things. We want to run the same 3BSE version on all of these systems. Okay, we'll net boot them, but yeah, firewall. Pretty annoying that, you know, you net boot, you mount your root file system over NFS and then you block all traffic and your root file system goes away. There are ways around this, but it, you know, it becomes complicated. Then what happens if the machine panics? You need to have control code that will manage or will understand that, hey, this machine panics. I should collect the information out of it and I should give it a hard reset. Oh, by the way, we need some more hardware to do this. You can imagine complicated setups. Where does all of this hardware live? I have too much old hardware at home, but not enough hardware to build all of this, so it would have to live somewhere else. But if we need five machines and serial interfaces and power switches, we can't really expect all developers to have this setup. So how do other people write tests for this when they can test their tests? There are advantages to this setup. Alexander did a great talk about it. One of the advantages is you can do meaningful performance testing against it, but I'm sure he'll agree that it's a complicated setup that needs some care and feeling. So, you know, that's a downside. The obvious way around all of these is, you know, virtual hardware. We've got Beehive. Beehive is great. We can run any number of machines, well, you know, low number of machines, but enough to build our setups in Beehive. And that's actually the approach taken by a student in Google Summer of Code 2017 to try to build this. We've still got some issues, you know. We walk all traffic and oops, we've got a serial port or we can pretend to have a serial port and we can make that work. Great. But our automated tests on the IDOP3BAS.org run in Beehive. So we have to run Beehive inside Beehive. And I don't actually know if that's possible, but if it is, it's bound to be slow. And the darker, darker is good enough for everyone in the cloud. Beehive, Beehive should be good enough for you. Yeah. Another issue with that, and that's where the what the Google Summer of Code student ran into is it is really, really annoying to build the images during your build run. It takes forever and you need a lot of storage and all of those things. You've still got issues coping with panics. It's still possible, but it's still annoying. It's also really slow because you're waiting for emulated, well, virtualized machines to boot. Best case 20, 30 seconds depending on what hardware you've got. It's an improvement, but it's still annoying. I keep telling you what solutions won't work, what will work. Well, VNet or network stack virtualization hands up everyone who knows and loves VNet. Okay, for those of you who don't know, it's network stack virtualization. Here we go. I've explained that. It's associated with a jail and it basically turns a jail into a pretend virtual machine. They all run the same kernel, but every jail can have its own network interfaces. It can have its own IP address, you can set whatever IP address you like. You can also run your firewall inside the jail. For all intents and purposes, you have your own machine. Of course, it's not 100% the same as running on actual hardware, but it's 99% the same code. PF supports this. Actually, to be entirely fair, all of our firewalls support this now. I'm mostly focused on PF because I spent some time arguing with PF about, you know, maybe you should try not panicking when we run you in a jail. After a while that sank in and now it usually doesn't panic when you run it in a jail. The usual disclaimer there, right? I do development. I look at bugs that users report. So I spend all of my time looking at bugs. And once I've fixed one of them, it goes away, but I go on and look at the next bug. So there tends to be a perception of everything is broken and everything is horrible. Turns out that as a user, and I actually use PF in the net jails myself on one of my machines, and it works just fine. I don't actually think that machine's ever panicked over one of those things. So it works a lot better than I get the impression. But bugs will happen. Please report them. As of 12.0, I will try to help you if you run PF in a jail and it panics for you. If you do it in 11, I will just point and laugh. Unless you threaten to pay me in which case I will quietly laugh and help you. Anyway, I've been talking about this forever now. How do we actually run a VNet jail with its own IP stock? I bet it's really, really hard. That's it. We start the jail. We give it a name, Alcatraz in this case, because I think that's hilarious. We tell it that we want it to be a VNet jail and we want to persist it. Persist just tells the jail that the jail should keep existing even if there are no processes in it. Usually, that's a silly thing to want because jails isolate processes. If you don't have a process, why would you bother to isolate it? But in this case, the jail has an IP stack and you could forward packets. So that's it. We're done here. All right. Well, to be entirely fair, that's not all that there's to it. We also need some network interfaces. It's all easier to test network things when you can communicate with the outside world. So to do that, we've got something called ePair. ePair is actually really simple. It's two network interfaces with a cable between the mixup. The cable doesn't exist and the network interfaces are virtual, but it's two network interfaces with a cable. So you get an ePair zero, which has an A and a B side and A and B are linked with each other. And if you stick a packet into A, you can add to B and the other way around. The usual way you would use them is you would give an IP address to the A interface. You would stick the D interface in the jail. So same thing. We start the jail with VNAT and oh, by the way, you have an interface called ePair zero B. And then we can actually execute a command inside the jail. And I still haven't fixed my slides. This mix misses Alcatraz. So execute inside Alcatraz. If can fake ePair zero B, there's an IP address, and then you can ping it. And that's that's all there's to it. And we're already testing our IP stack or testing that we can send by CMP Echo requests and get replies. It's not terribly exciting, but it actually runs a whole lot of code. And we we had, you know, any number of panics around this, because it turns out that setting up and shutting down my IP stack or a network stack is something that we only ever did when we booted them, when we rebooted. And if you do 20 of those, rather than one, and you do several of them one after the other, or you do them at the same time, you run into bugs that you never ever run into in other situations. But that's basically all we need to do. So, so let's take a look at writing a test for this. That's not all that there's to the test, but you know, the entire test case doesn't fit all my slides. So what we do, we use the automated testing framework, which again, you know, we totally enhance ourselves and in no way stole from the SD includes some utilities, declare a test case, we have a v4 test case cleanup just says that there is a cleanup function and you should call it after you're done with the tests. Test head just describes the test. So, in the description, basic password test for IPv4, we require a user. For some reason, people think that security is important. And maybe we shouldn't allow people to spin up new IP sacks and network configuration unless they're actually, you know, root. The only thing this does is make sure that when you run the test as more brews, this test will just skip because it can't possibly succeed. The interesting bits is test bolly. And that's, that's all there's to it. Although admittedly, I've stripped out a bunch of tests, but this is a functional test. This is useful. So we have general initialization codes. The initialization code doesn't do a whole lot of exciting work. It just goes and checks that VNet feature is available in your kernel because if it's not, there's no point in running this test. It also checks that PF is available because, you know, testing PF when the PF module is not loaded again, not terribly useful. We create an EPIR. We assign an IP address to it, just like we discussed earlier. Create a jail, help a script to create a jail, create a jail arbitrage, give us an interface, set an IP address in the jail, and then just, you know, basic sanity check. So ATF check will just go and verify that the next command will return exit state to zero. We ignore the standard about it because we don't care what it says as long as it returns with zero. One thing only, and the time out of one second to that IP address. Hopefully, you know, the jail will apply and everything will be well. Next up, we enable PF. PF by default passes everything, so this should still work. So, yeah, check exit state to zero, and then we're going to walk everything. So we set rules in the jail, walk input. This is a simple helper function. It just makes life easier when you can write this rather than write it to a text file or echo it and pipe it into the jail. It's much shorter to write this. Remember, I'm lazy. So we check that we get an error output. We still don't care about standard outs, and that this thing will fail. Something about this, as we'll see later when I show some output, is this entire test run takes 1.2 seconds. The entire second is taken up by this because we spend the second waiting for a reply that will never, ever come. I mean, it should unless we totally broke it something. Everything else is 200 milliseconds to top start the jail. So, you know, maybe we should do some boot time improvements so we can get things to boot faster in beehive, but I don't think we're ever going to get to the point where we can boot an entire machine in 200 milliseconds. We'll actually boot it, do some work in it, and shut it down in 200 milliseconds. But we can do that with VNetJails, which makes it a lot easier to run all of these tests. I say a lot. We have some tests for PFRite now, not that many, but enough that they catch bugs, and they only tend to take about a minute. So, cleanup, pretty easy. There's a utility function. The other utility functions, whenever they create an interface or a jail, helpfully keep track of this in a temporary file. So, the cleanup function just goes and looks, what's in the temporary file? We've created the EPAR0A, delete EPAR0A. We've created EPAR1A, delete EPAR1A. We've created the Alcatraz jail, okay, remove the Alcatraz jail, just so that we clean up everything so that the next test isn't going to conflict with whatever we've left behind. And then this is just ATF infrastructure. If you're really, really interested in how all of the ATF things work, there's an article in the FreeBSD Journal about this. So, you can take a look at that. The FreeBSD Journal is now free, so you can just get onto the FreeBSD Foundation website and go take a look at it. What business output? Well, there we go. We do, you know, sudo because we need to be root to run this. Obviously, QA test, password V4. Password V4, the test passes. Okay, it works. 1.2 seconds. We even have some results. We can go and look at, you know, what sort of output did this test generate? Which is useful. Although, to be entirely fair, usually when the tests fail, I will start off by adding SIT-X to the test body so that I can see what commands it's all executing. And we'll probably end up setting a sleep 3600 somewhere so that it doesn't clean up the jail so I can go look at, you know, we have the test set up and now I can send some things and I can TCP dump and I can, I might even detrace or look at some other statistics or some other counters to, you know, what is actually going on here. But they run, and these things run automatically. These things run several times a day. So, the next time somebody breaks PF so that it, you know, rather than dropping packets, just lets them all through, which, you know, how would you notice unless you test? And it is kind of important for your firewall to occasionally drop packets and maybe even the right packets. Anyway, next example is a much, much more complicated example because it's a test for PF sync. For those of you who don't know, PF has this wonderful feature where you can synchronize states between two machines. So, the typical use case for this is when you have a gateway machine, you just have a second one. And if one of them dies or you take it down for an upgrade, the other one will take over. But you wanted to remember states because if your TCP session is running, you would like it to keep running. So, PF sync will send state from one machine to the other so that it can't notionally take over. So, very basic tests for this. We create a couple of interfaces because, you know, we'll have two machines, we'll have two jails, jail one, jail two. We'll have interfaces to all of those jails and we'll have a link between them so that they can exchange states. Hence the e-pair sync, e-pair one, e-pair two, jail one, jail two. I got tired of the naming my jails after this jail's joke here, as you can tell. Give them the two interfaces, configure interfaces in jail one, enable PF sync, synchronize over, sync a device, send some up, max update if I remember correctly, that just tells PF sync that the moment you have a single update ready, send it out because usually it tries to gather up a whole bunch of updates before it starts to send packets off to the other side, send it up, do the same thing with the second jail. So, there's a lot going on there but it's basically all related to how do I actually configure PF sync to do something. Enable PF in both of the jails, set IP addresses and then we send the ping. This ping will travel through the first jail and after some time that state will get synchronized to the other jail so we can go ask jail two, you know, what states do you have and do you have an ICMP state for this particular IP address? Should have one because we send an ICMP echo through it. We go look for it, make sure that it matches what we expected to do and if we don't find it, the test fails. So, very primitive, well, it's very basic test. It's set up PF sync, make sure that the one state we deliberately create in it ends up on the other side. It's primitive but it tests a lot of the functionality of PF sync. It's basically, you know, the basic functionality of PF sync has to work for this test to work. I did some work on PF sync recently and, you know, this test failed very often. It also demonstrates that, you know, it is actually that hard to test something that needs two machines. You could test things with even more machines. I have tested for things like fragment handling as well. I don't have examples in the slides but there's one where, you know, we send a bunch of fragments at it and we test what happens. So, these are all fairly simple tests because we send, you know, correctly format packets. There are some tests that do things like use KB, the PyFold tool to build arbitrary packets to send malformed packets which is another fun thing to do to your network stack. Yeah, I'm trying to remember, I think for some of the recent security issues, I also wrote something to send specifically malformed packets to trigger something that would otherwise have been a panic but now is no longer a panic because Alexander fixed the bug for us. Was it you or Sasha? Sasha, please. Okay. So, I've set the credit to the wrong side then. Anyway, for those of you interested in the math tests, the PF tests live under user source, assuming that's where you install your source. Test sys netpvlpf to mirror where our PF code lives, our PF code lives user source sys netpvlpf. When they get installed, they're in user test sys netpvlpf. If you want to run them, sudo package install qr and then go to, well, actually there we go, install qr, installs kb because that tests a few more things. Kld loads pf sync which will implicitly load pf. Go to user tests sys netpvlpf and then probably sudo qr test unless you've done all of this as root already. There's a lot more information in the journal. Well, there's information on how to write ATF tests. You should all be writing tests. Yes, even you open BSD people, you should be writing tests for free BSD. Just copy that. Well, there is actually a lot to be said for we should take a look at some of your tests. Are you threatening your payers to do it? No. I'm threatening you with my internal gratitude. So there's an article in the free BSD journal, the March April edition of this year, how do you write some test things? Hopefully one of our documentation people will actually go and add some of this to the handbook as well. Benedict's working on this. Benedict's not here but I'm going to apply social pressure anyway. Important things, well, if I still haven't convinced you now what we need is some sort of deeply profound quote by someone famous. I didn't have the budget for that. So you get me telling you, write tests. Tests are good. I don't know about how profound that is but it is actually true. Tests are wonderful. What's in that for you? You can prototype some of your setups because sometimes you try to do things and they don't actually work. You can prototype a lot of networking setups with just jails, which means that you can do it in a virtual machine all playing over whatever ocean you like to fly over. It also means that if you write a test for this and you give us the test case, we'll include it and we'll run it, which means that your use case is not going to break or when it breaks, we're going to know. And we're going to know pretty much immediately after we commit the breaking change rather than three years later when you finally get around to upgrading and then notice that you're careful use case no longer works. And we broke it three years ago. Developers have, well, I might be projecting here, it might just be me, but in general developers have a very short attention span. So if you notice that they broke something a couple of days after they did it, you're a lot more likely to get them to care than when you notice three years later. What else is in it for you? Right. Yes. As I mentioned before, write a test case because it makes it really, it makes it much easier for me to fix your bug. I'm lazy. Make it easy for me. Recruiting bugs is hard. I need to understand your entire test setup. And I often get, you know, well, I get two types of bug reports usually. I get the one that says it's broken. Please fix it with no information in it as to, you know, what is broken? What is your test setup? How do you break it? How often does it break? In what way does it break? No information whatsoever. The other type and that is somewhat more rare is here's 200,000 lines of output about my environment. IP addresses, firewalls, net start output, state output. And I need three hours to run through all of that and work out what your setup actually is. If you write a minimal test case, it will take me 30 seconds to work that out. Also, if you want to get a patch into our code, if you've written a use case for it, sorry, if you've written a test case for it, it means that I will understand what your use case is for that patch. It also tells me that, you know, you've actually tested this patch. And I'm going to have to write a test for this anyway. I'm trying very hard lately to, every time I make a change to actually have a corresponding test. And it'll always succeed, but often enough that we're getting more and more tests for PF. So I'm going to have to write that test anyway. Unless you do it for me, of course. If all of that seems a lot like, you know, work, you can also just pay me and then I will happily do whatever for you. I would like to encourage you all to write tests. You wouldn't be the first to do this. Olivier has already written some tests for IPsec. Olivier was tired of IPsec being broken with certain algorithms. So he wrote a test and now IPsec is not broken. And if it does break, Lee Wen here will shout at them. Well, actually, that's a lie. Lee Wen will politely ask them to fix it, which is very nearly just as good. Also, somebody's working on a Google Summer of Code project right now, and I'm totally blanking on their name, to write more tests. So the objective of that project is to write firewall tests in such a way that we can use all three of our firewalls. Because right now, PF is the only firewall with tests. So, you know, if you need to choose a firewall, choose IDFW because it's great and it doesn't need tests. Or possibly choose PF and send me tests or send me test cases rather. I think I've covered pretty much everything. So you have questions. Do you have heckles? Do you have bags of money to throw at me? Yes, bags of money. Also good. Right. So the question is about user run testing for existing rule sets. So I think there's two aspects to this. One of the tests I did talk about is also a product of the 2017 Google Summer of Code project. So you wrote some tests to test the PFCable parser. There's quite a lot of code in user space that parses rules. And we have had some regressions in there where we accidentally broke previously accepted syntax. So there are some tests there that take some rules, parse them, and have the PFCable spit them back out. And then we compare to see, you know, did it actually read what we expect? So that's one part of it. But that's straight syntax checking. That's loc functionality checking. If you're interested in, you know, we have a rule set and we want to know, does it actually do what we expect it to do? You could write very similar tests to these. You would want to start some jails, probably one or well, probably between two and three or between one and two, depending on how you do it. You want a jail that pretends to be your firewall. You want to put some interfaces in it. And you could rename your interfaces so you can literally reuse your rule sets, or you can just define aliases and then fill them out differently. And then you can inject traffic and watch at the other end of the interface is to see what traffic goes through and what traffic does not go through. The test cases I write, I typically try to restrict very much to testing one aspect of the functionality at a time, both because that makes for shorter and easier tests and makes for shorter and easier debugging sessions. But you could certainly imagine writing a test like this with, I don't know how many interfaces your firewall has, but let's say five different interfaces with a bunch of different networks and some V-lines attached to it. You should be able to replicate that basically entirely as a V-net J-0 and then some traffic at the end see what happens. It should be entirely possible to do. Any other questions? Yes. So you might agree that our multitask code could use the tests. And I thought a lot about using this framework. It was part of the problem is that I really want to write C code for these tests because they are the most important for talking on this. Do you have any plan on knowing any working vent or like the equivalent for the J exactly? So like perfected JL commands that you use? So the question is, would you like some tests for this for multicast? And because multicast testing involves to a large extent socket options that need to be set, Mark would like some C codes to do this in. Yes and no. You can write ADF tests in C, but I don't know if that's what you actually want to do because I think you'll want to write some shell code to build a test setup because your multicast tests might involve multiple hosts depending on which side of things that you're testing. Probably thinking about this live, you might want to see it as two different sets of tests. You have the tests for the Cisco layer interface, so between user space and the kernel. And those you probably just want to write with ADFC and then you can set socket options and you can test all sorts of things. Where maybe you want to run your test program inside the VNet Jail. That's an interesting question, we should talk about that. The second part of what you might want to do is the network side of multicast. When we send IGMP join and leave messages and for that you probably want to do the shell thing where you set up a VNet Jail and you trigger it to do things either with some test programs, also even from Python because you can do some sets of all from Python just as well as you can from C. Or you know whatever other scripting language takes your fancy, I understand that not everybody is a huge Python fan. So you could do things like that. Combining them gets slightly tricky, but I'm sure that it's possible. To get on a different course there is a lot of our tools really want to be split up into a binary and a library. So if we would tear up the Jail commands and the JXX commands in a very thin binary wrapper around the library, that would make that problem a lot easier as well. But I'm going to assign that work to someone else. Any other questions? So the question is can I take multiple of these Jails and plug them together to simulate a network? Any answer is yes. You can simulate arbitrarily complex networks with arbitrary number of routers in between. So what you would do is you would create Jail1, Jail2, Jail3, you would create a bunch of ePair interfaces and then you would use an ePair interface to create a link from JailA to JailB and from JailA to JailC and from JailB to JailC and from JailC to A and B. Any topology you like, you can create them in there, you can set routing options or routes inside that Jail because perhaps one example or one advantage of this that I didn't really go into that much is, well, is this. It is trivial to execute commands inside the Jail. You just go JXX, JailName, do the commands. So you can go JXX, Alcatraz, routes, add default, whatever. You can set whatever route you like in those Jails, you can set whatever IP address you like. They don't need network connectivity because you're not interacting with them over the network. You're just giving them orders from the app side which makes it really, really easy to build setups like that. So the answer, you know, to make a very long answer short, yes, absolutely you can do that. Anyone else? Any heckling? We've gone through the questions, no heckling? When are you updating PF? Thank you, thank you. If you want us to write tests for you, why don't you update PF and then we'll really want to write tests for you. So the OpenBSD people's head on is why don't you update PF? The reply to that is I would like to but my users will kill me when I drop performance by 80%. Actually, 80% is slightly exaggerated. So I did some work on PFCync performance because our PFCync used to contain all the single mutics which is roughly the situation that OpenBSD is in right now. I saw no more than, well, no more. I saw 1.4 million packets a second through that without PFCync. So our PFCync is closer to 3.83.9 million packets a second. There's still some work to be done. There's a very vague plan for, you know, if ever I have the time and motivation to see what we can do about importing OpenBSD's PFCync and helping them or us working, collaborating I think is a better way to describe it to change the walking so that it will scale OpenBSD's working on this and I'm sure they'll get there but I'm also sure that it's a hard problem and it's not something they're going to do in the next two weeks. So without heckle return, the trouble... I thought you were telling me just this morning we were going to commit the lockless system calls tomorrow. I mean, Theo's hiking for two weeks. What could go wrong? You won't get in trouble for at least two weeks. But we don't have tests, so we... So it is about lunchtime, so you have a choice now. I can rant all about one more example of a test or I can let you all go and define... oh sorry, one more question. I was just going to say as a networker, please fill the love of God, nobody gets the right idea to run these against production. So the amazing disclaimer here and that is a legitimate point is don't run these tests on your production machine. Maybe run them before you take it into production but don't run them on your production machine. Certainly don't run them when your sysadmin can get at you because your sysadmin will get at you and you will have a bad day. I try to take care to use network, something that's that will not conflict with anything existing. But don't do that on a production system, do it on a development system. So that's what happens after a bunch of people are running production systems, you all know my system is still valid. So why not run tests like this once a week, once a month from 2 to 3 am, whatever works for your particular situation. Because it is most fun to page the guy running your switches between 2 and 3 am. That's when they like to get woken up in shit all the time. Just stop doing online network because I still need to cross. I'll punch you, I'm good at it. You're on online network. You know what I'm saying, you were careful to avoid IP conflicts. You know the janials there, they are separate IP spaces. But for my tests I tend to use the host to inject packets from. So the host will have a route to some of these subnets. So if they conflict, if I happen to use a subnet that you're using in your network, you shouldn't be because they're test subnets. But if you're using them then yes it will break your routing. The typical use case for these tests is very much you're developing and you're doing this during development. Can I give you a hand? Sure. Look up the NSA's AS, they have a big blob of addresses, just choose those, you'll never even play. It's been around for years. How use are they in 1918, everybody uses those. To use the NSA's subnets because they have a big subnet range and it's not public on the internet. But I make it a policy to not pick fights with people who have an answer for you and what are me. So Bjorn? There are official young asses next to a network that you use there. Yes. How easy is it to convert legacy tests that people have like shellcups and everything into the framework? I've, so the question is how easy is it to convert legacy tests that people have into this framework? And the answer is that I don't know. Because I've never tried. I wouldn't think that it's terribly hard. I think that depending on how those tests are built, you get to throw away a lot of your code and just start the jail, set some configuration in its sense of traffic. I honestly don't have a better answer for that. I don't have a lot of feedback that you might be able to find in all kinds of cases. This is a source tool somewhere from the old days. Some of the memory types that you leverage mean that they've never been used in the past. They've never been, you know, 80F wasn't around that time. Suddenly a pointer and I will add up to my ever growing to-do list. I'll give you five more answers. Yes. One comment to the addresses and networks. In the OpenBSD tests, I can pass the network settings via the environment to the test. So everyone can set up their own network environment and the reaction of other people using my test was give me your IP addresses, otherwise it's too complicated. Yeah, so yeah, that's a good point. I've chosen not to do that to keep it just easy. I like that. Well, let's wander into the next example here. I like that it is extremely easy to read these tests because everything you need to know is here. So when we do things like set firewall rules, you can just read the firewall rule and see everything that's involved in it. If you, you know, what IP address are we setting, you can just see what IP address we're setting. You can certainly make a case that that should be abstracted and it should be a variable and it's really easy to do because it's just a shell script. So we can have some common variables that define IP addresses and some of them are going to use and use those. But it's an extra layer of abstraction and it makes it harder to follow what's actually going on. So a lot of these tests take me 10, 15 minutes to write and then you know half an hour to get the robust and cleaned up. But they are remarkably easy to write and I think in part that is because there is what I feel is the right level of abstraction. Yeah, at least I don't want to advocate for a policy of oh, it's only test code, none of the rules apply. But it is only test code and there is a limit to the amount of abstraction that you need. Okay, that's it. Thank you very much for coming. Thank you very much for the limit of the Hackling Eye goals and enjoy lunch.
automated firewall testing talk. Before we get in FreeBSD that are not in OpenBSD, one of which seriously, why do we want automated tests to make tests are a good idea. What do we want out of takes two minutes to run, they'll run them. If it framework, which is an automated testing framework in QA, the automated test for PF or than we'd like, but at least most of the testing firewall test. One of the tests we'd like to do configure a system if you block all traffic. You still possible, but it's still annoying. It's also really slow because you're waiting for easier to test network things when you can tests, but this is a functional test. This is VNetJails, which makes it a lot easier to run all know, PF has this wonderful feature where you can first jail and after some time that state will in the slides but there's one where, you know, we packets to send malformed packets which is another fun thing to do to your network stack. kb because that tests a few more things. Kld means that if you write a test for this and you them to fix it, which is very nearly just as not go through. The test cases I write, I a test like this with, I don't know how many using in your network, you shouldn't be because abstraction and it makes it harder to follow