Plumbing the Internet, BSD-style
Johnson, Thomas
If you have ever delved into the world of BGP routing on the Internet, you know that it can be quite an undertaking. Fortunately, we can utilize BSD and a couple of open source tools to build a solution that will provide high availability and easy management, at a fraction of the cost of commercial solutions. This talk is an introduction to the basics of building a BGP presence on the Internet, and orchestrating it with Ansible. If you have ever delved into the world of routing on the Internet, you probably know that it can be daunting. You learned that BGP is the protocol used to build the global routing table, but likely discover that every answer leads to two more questions. Bureaucracy and expensive, proprietary solutions are everywhere. BSD can't stop the red tape, but it does offer a fantastic solution for your technical dilemma. This talk is a whirlwind tour of building an Internet presence in three parts (and an opportunity to learn from my mistakes). Act one is a brief introduction to the BGP protocol, the process to obtain numbering resources, and connectivity. The basics of rigging a FreeBSD host for routing are covered, and the routing daemon BIRD. Next up is discussion of network design for high-availability, and the FreeBSD/BIRD configuration to make it happen. Setting up one router may be fun, but we're going to want automation. Enter Ansible. After a quick introduction, we'll use it to deploy a live demo of our configuration Finally, now that we have a fault-tolerant solution; so let's introduce some faults, and see what happens! We'll induce some failures, and watch how our network responds. Time-permitting, I will wrap up with some discussion of resources for checking the status of your installation. I'll also discuss some more complex tasks, such as multi-site failover, and taking a router out of service.
 Alright, good afternoon. Thanks for coming. This is Plumbing the Internet and we're going to be talking about BGP automation and BSD obviously because that's why we're all here. So first of all, I'm curious, who knows what in the room? Who's worked with BGP before? How about BGP on BSD? How about automation? Hopefully everyone in the room. How about with Ansible? Awesome. So we can skip through some of the boring parts then I hope. And I'm going to be cruising along here at a high rate of speed so if I lose you, by all means, flight me now. So before we get going here, my name is Thomas Johnson. I don't do many slideshows so it's a little rough around the edges. And how I got to be here, I got started with BSD in 2010. In 2013, we started planning for a high availability disaster recovery solution and decided that that was going to necessitate us figuring out BGP. So I bought a book and started reading and early 2014, we turned up our primary site for BGP. We turned up DR site a couple months later and then 2015, we got started with Ansible. And then early 2018, we had our first production test of our DR site because this was the parking lot of our primary data center the week before the Super Bowl. And we decided that we didn't really want to have our production there. But it wasn't all bad because when you get your poop in a group and actually use your DR facility, then you get away with stuff like this. So I'm going to start here by talking a little bit about Ansible. And since part of the presentation here is to demonstrate what you can do with automating your BGP with Ansible, it seemed like a really great and probably really foolish thing to build the demonstration lab as we go here. So we'll be turning that loose in a couple minutes. But real quick, couple of things about Ansible that are worth noting. It's written in Python. It's identigent operations so much like Puppet or the other automation systems. You're telling Ansible how you want things to be not necessarily what you want them to be doing. It's a very thin layer so your client nodes don't really need much in the way of prerequisites other than stuff that you're probably already running in your network anyways. As far as requirements, like I said, there's not a whole lot you need to install Ansible on your master nodes. And then your client nodes are going to need SSH, Python, stuff that we already have installed. So you can be up and running without, you know, going to great lengths. I'm going to start to jump right into some of the configuration of Ansible here. And we're going to start with inventory, which is the component of Ansible where we tell Ansible everything that we wanted to know about our network and everything that we wanted to know about our network. And at the heart of that is the hosts file, which is where we define all of the hosts that we want Ansible to know about. And we can also organize those into groups to simplify configuration. We can run Ansible plays against a group of hosts. So on the left-hand side here, we've got a group called Beehive Guests. And we've got a bunch of guests that are listed underneath that. The right-hand side is a variation on a theme that's a group composed of groups. So we're aggregating a bunch of subgroups into one group, makes it really handy to share configuration amongst other things. Moving deeper into the inventory, we get into the host files and group files. And this is where we really kind of get into the heart of what inventory is all about. It's a free-form AMLA file. Ansible doesn't really put much in the way of restrictions on how you can format data that you're putting into this file. In fact, pretty much everything on the left-hand side, which is a host files file, is completely arbitrary. It's put together the way that I want to use it in my implementation. So this is a host files file. A couple of things that are worth noting here. I can even use my laser pointer. We're defining our host configuration path where we want Ansible to go and look for configuration files if it needs them. And then down below, we're starting to describe how we want Ansible to structure the network build for this particular host. And the nice thing about that is we're defining it all in one single file or a group of files not spread out over a whole mess of files. On the right-hand side, we've got an example of a group vars file, which there again, it's configuration applies to an entire group. And it brings up a BSD gotcha, which is that Ansible user bin Python as its default path for the Python interpreter. So unless you've somehow got that in your BSD implementation, you're going to need to update that. And at this point, we have enough configuration in Ansible that we can actually start doing things. The other piece to Ansible that's just as important as inventory is going to be playbooks. And playbooks are really the top level unit of how we do things in Ansible. Playbook's nothing more than a text file. Again, it's written largely in YAML or entirely in YAML. It's composed of one or more plays that's run against a host or a group of hosts. And within that play, we have one or more, zero or more tasks that are actually applied to that host or the host in the play. So we have another example here. Can everyone read the small text? Okay. If you can't read things in the back, let me know and I'll do what I can to make them bigger, especially when I start pulling up terminals. But this is an example play. We're running this play against the Beehive Guests group. And then we've got a couple of tasks that are listed here. We're basically just checking a couple variables that are defined and we're starting a path and doing nothing more than that at this point. The other interesting part of Ansible, similar to other systems, is there's a concept of roles, which is basically just a predefined structure for code reuse. So if you have a piece of a group of tasks that are self-contained and you want to be able to use all over the place, roles are the way to do it. They have a predefined directory structure, which is what we see on the right-hand side here. And we've got directories for tasks, for default variables, handlers. We can even include other roles from a single role. So there's a lot of flexibility there. The other thing I'll point out here before we go too much further is VM Beehive, which is a tool that I found on GitHub when I was looking for something I could use to avoid having to figure out all of the Beehive provisioning by hand. And it's basically just a light CLI interface front end to Beehive. It lets you do all of your network configuration automatically. And the question is, as Baptiste pointed out earlier, is why doesn't FreeBSD have something like this in base already? So at this point, a little bit about the demo that we're going to be, that I'm going to be building up here. It's really essentially starting with nothing more than a Beehive host. And we've got an Ansible guest running on that host. And from that, we've got, on the host, we have a pre built stem cell of a Zvol. And all of our guests are going to be cloned from that. And then bootstrapped is necessary to build our network. And so once we get this going, about 17 minutes later from those first two hosts we end up with, should end up with a fully routed network. And if you guys saw the GitHub links at the beginning of the presentation and pulled the diagram that I forgot to mention that you should pull up, then you already have a copy of this in front of you. So real quick here, a high level view of what the bootstrap process looks like for these routers. The first thing that we're doing, you can see we're doing a ZFS clone to clone the stem cell into the host name that we want. And we import that volume so that we can do our bootstrap operations on it. We call a couple of very minimal roles that are going to build up the basic configuration that we need for this host to run, which really amounts to building out the config files for the routing daemon and templating RC.conf. And really there's nothing more than that and we're going to end up with a working router. At that point we export the Zvol and boot the guest. And this is my reminder to go and actually start the playbook. So we'll flip over here and can everybody see this? Well enough anyways. Good. So what we're looking at here is we're looking at the ants playbook command 001 build demo is just a real lightweight playbook that calls a role that does the bootstrapping of the entire network. And we're skipping a tag here for some slow tasks that we don't really need to run because they take even longer. And we'll set it loose. And the first thing that it's doing here is something I didn't mention. It's going through and it's doing some initial checks of the beehive host to make sure that the bridge is in such that we're going to need for all of the guests are in place and ready to go. So while that's running, I'm going to flip back and start talking about BGP and we'll start by talking about what the internet is. The internet, as I'm sure many of us are aware, is just a handful of autonomous systems that need to communicate with each other. And the way that everybody communicates with each other for better or for worse is using BGP. Now you guys can read Wikipedia just as well as I can regurgitate it, probably better. So I'm just going to point out a couple of things on BGP that are worth noting here. It's a distance vector protocol. It measures the shortest AS path between two autonomous systems, which basically means we're doing path determination based on the number of organizations that our traffic is going to be passing through. There's two varieties of BGP. There's an external and an internal variety. The external variety is used to pass routing information between organizations. The internal variety is used to pass that information amongst internal BGP speakers within an organization. The one got you here is that generally you still need an interior protocol to discover your own internal network. Generally BGP doesn't handle that for you. So in this case we'll be using a less PF, but any sort of interior protocol works. So we're going to do BGP. What do we need? Well, the first thing we need is money, but fortunately we don't need a whole heck of a lot of it to get started, especially if you're a smaller organization. We're going to need numbers. We're going to need an AS number. We need IP addresses. We're going to need routers and we're going to need connectivity. And for those last two points, we're going to want more than one. If you're going to go to the time and trouble of deploying BGP, there's no point in deploying it on just a single router, single connection. You're wasting your time. Internet registries are the place to go to get number resources. If you're in the United States or Canada, you're going to be talking to Aaron. That's where my experience is. Your mileage may vary if you're from outside of that region, but these are the folks that you need to go talk to. The one thing that I will mention is that if you start down this process of getting numbering resources and you have questions speaking the context of Aaron, Aaron is your friend. They're not a gatekeeper. They're not the bad guy. Their mission is to help you get resources to do things on the Internet. And so they're generally willing to go well out of their way to help you out. Autonomous system numbers simply is a number that identifies your organization. There's two byte and four byte varieties. Two byte are the legacy variety. I don't even know if they're issuing two byte AS numbers anymore. Who are you with? Okay. They will by default issue you a four byte number. I might see specifically ask for a two byte number. I have a two byte number because we were in early enough. Thanks for the clarification. IPv6, you get addresses from your registrar. Your allocation is going to depend on your organization size. And the one thing that I'll point out here, well, two things I'll point out here. Minimum announcement size that's going to be accepted generally is going to be a slash 48. And as you start to work out your addressing scheme, I really recommend not allocating anything less than a slash 64. Even if you deploy a smaller sum that allocates a slash 64. IPv4, it's the legacy protocol on the internet. Everyone in this room knows about it. It's exhausted. Minimum announcement size is a slash 24. I think there's enough said there. As far as obtaining resources, you can. And I'm going to get to that. But I like that picture. So blocks can be had. The internet registries have waiting lists. There's a transfer market that you can take your chances on if you're so inclined. But there's another trick. At least for Aaron, there's the 410 rule, which refers to the section in their numbering policy. It's a slash 10 that's set aside for IPv6 transition. And the bottom line here, the big takeaway here is that if you already have an IPv6 allocation and you want to dual stack your application, you have justification to request a v4 block. I may add for if you are with Rive NCC, become a local internet registry of the year. It's something like 2,000 and 1,000 euros. Okay. That's good to know. Does Rive still have a pool of blocks? I assume. Okay. Good to know. Okay. Okay. That's good to know. Connectivity real quick. There's generally three methods of obtaining connectivity. You can either buy transit, which means you're paying somebody else to provide you a path to the larger internet. Internet exchange points are a really great way to exchange traffic with geologically local peers. There are also great source of information. And then there's peering relationships where you're exchanging routes with another single organization. Couple of connectivity gotchas real quick. If you're starting to deploy BGP and bring up transit and things like that, you should expect to be providing a letter of agency, excuse me, basically acknowledging that you are authorized to announce the blocks that you say that you want to announce. Expect that you're going to pay cross-connect fees if you're in a data center, co-location. And the last point there, you may want to at least make some effort to make sure that your fancy redundant fiber isn't running through the same conduit because that can cause problems too. So switching back to FreeBSD here, talk about what we need to turn our FreeBSD into a router. And in a nutshell, you're looking at it. Pretty much everything we need at a high level is done through RC.conf. What we're looking at here, we're defining our network interfaces. We're enabling routing. And we're enabling our routing daemon. And there's really not too much more than that. In reality, things like VLAN, carp, lag bundles are going to make this more complicated, but we all know that's going to happen anyway. Real quick on firewalling. Firewalling, generally, I try to separate the filtering and the routing functionality roles on different machines, but there are cases where it's useful to do filtering on your firewall. The one thing to be aware of is keeping state on connections. When if you move to a, if you're moving to a VGP model where you've got multiple routers and uplinks, it's entirely likely and probable that traffic's going to come in one direction and go out another direction. And if we're keeping state for that, you're going to have all sorts of hard to die nose problems and you'll lose hair and just don't do it. So moving on to our routing daemon. The routing daemon I'm using here is Bird. It's open source. It's pretty widely, widely used. A couple of highlights on it. There's two, two versions that are current. We're going to be talking about version 1.6. That supports most of the common protocols that you're going to run into, stool stacked. The configuration structure is, is simple and easy to get your head around. And it has a utility called Bird C, which is very useful for controlling your routing instance and then monitoring it as well. So as we start to look back at our, at the test network here, we're going to take a look at some of these different relationships that, that we're, we're creating with Bird. And so we're looking at a, a portion of the test network here. And we're going to be looking at it largely in the context of this AS-123 router 1A. So, so our first relationship here is our relationship to another AS. And that's going to be where we're using EVGP. Within the AS, we have multiple BGP speakers. And this is where we're going to be using IBGP to, to share and communicate that external routing data to all of the different border routers on the, on the network. Internal to the network we're using OSP-APFAs I mentioned earlier to discover all of the routes that are available internally. And then additionally, we're using a couple of static protocols largely to, as a means of injecting routes into BGP. So rather than, especially from the point of view of an end-user rather than, than blindly redistributing things into BGP, we want to be very sure we know what we're, we're injecting into the, the global internet. So with that in mind, we'll take a look at how these relationships actually manifest themselves on a running bird. And so at this point, I've been moving too quick and the network isn't totally up yet. But hopefully it is up enough here that we can at least take a look at a couple of, couple of things that are going on in bird. And these are a couple of the commands that I use most frequently just to get an idea of what's going on. So show protocols is going to give you a quick bird's eye view of all the different protocol instances that are running on your, on your bird and what the status of them are. If you want to dig into a particular protocol instance and see more details about it, we can do that. And this one fortunately has come up so we can talk about it a little bit. So this is our, this is our EBGP instance from this router out to its, its upstream neighbor. And we can see that the protocol is up. The interesting thing, the usually the first thing that I go to look at when I'm pulling protocol is to see how many, how many routes are being imported from this, from this pier, how many we're exporting, which is, which should correspond to just the static routes that we're injecting manually. And then there's a preferred counter here. This is how many routes that we're actually preferring this, this path for. Birdsea is also an interactive shell, if I can spell it correctly, which is actually probably even more useful because it has inline help. So we can do whatever we want from here, like tell bird to reload its configuration if we've made some configuration changes to it. But generally we don't have to do that because we're running everything through Ansible right now and Ansible is going to handle that for us. So moving along here, we didn't look at the bird configuration, which is why I put the slide in here so I won't forget about it. So this is the actual configuration that's driving what's going on this router. So looking at the left-hand side here, a lot of what we see at the very top is boilerplate definition. I'm not going to spend a whole lot of time on it here. The more interesting bits, or as we get down into OSPF, we can see that we've defined interfaces for each of the OSPF, each of the interfaces we want to run OSPF on. Some of them are running normal OSPF, some of them are just stub interfaces, in particular our uplink interfaces we don't want to run OSPF on. You can see a couple of static instances defined here. For static BGP, portable BGP, this is where we're defining the routes we want to inject into BGP. And the distinction between the two is more applicable in a multi-site configuration. I'm going to get into that in a little bit. And then at the bottom here, we see our definitions for our EBGP and IBGP sessions. And they look pretty bare here. We're really just defining what our neighbor is. And the reason that they look so bare is because we're templating them. So these configurations are templated by an included file, which I forgot to point out at the top of the file, where we plug all of our common configuration that's going to be the same for all of our EBGP instances on this router. We just plug it into a single template and then we can reuse it and reuse it and reuse it. And there's not a whole lot more that we're defining here. We're defining our AS number. We are defining a little bit of a hack here to allow our own AS number to appear in the AS path. This is not generally allowed by BGP to prevent loops, but it is useful if you're running multi-site configuration. You don't have a point-to-point link. If we want to route between the two sites, we have to be able to allow the AS number. So looking at the bottom of this file here, I'm going to cut off here, but the IBGP template is largely the same as EBGP with just a couple of changes. So now that we've looked at the BIRD configuration, we'll look at the actual template that that configuration file is generated from. And we're doing that with Ansible. Ansible uses the Gingya templating language for all of its templating plays. It uses it for the playbooks and configuration as well. And it's very flexible. It draws information in from inventory, which is why we're finding it all there. And it's basically what allows me to generate 15 routers in this demonstration from what amounts to about one set of configurations. Oops, jumped ahead. So we will take a look at these templates. And so first we'll look back at the inventory a little bit here. And this is the inventory, again, for router 1A. We've got, again, some boilerplate at the top that's less interesting. But as we get down here into the network configuration, again, this is all arbitrary. You can structure it in whatever way makes sense to you. But we're defining the addressing for each of our interfaces. We're defining what interface name it should be associated with. In the case of some interfaces, we're defining carp addresses if we want to do high availability there. We're defining what the OSPF relationship is. And then as we move down into the bird section of this dictionary, we're starting to see all of our routing-related configurations, like what our AS number is. We've got a section here that's defining what prefixes we're going to be putting into the static and portable protocols. And if you look here, we've got additional variables that are being pulled in using the double braces, which basically just means that I'm defining what these lists are elsewhere in the configuration, actually, in groupfars files, because they're common across AS 123. And then I'm pulling them in here. At the bottom of the file, we see the definitions for our BGP sessions. And again, because everything's very templated, there's really not a whole lot that we need to define here. Looking over here on the right side, we're actually looking at the bird.cont file that's before it's templated by Ansible. So again, here we see double braces, which indicate that we're doing a variable substitution, bringing something in from inventory and plugging it into the file. The OSPF section here, we're starting to see some logic being used. We're starting to see a for loop where it's looping through all of the interfaces that are defined in inventory, looking to see if it should be configured for OSPF. Again, we're doing looping to define the static protocols, basically taking everything that's in a list of, a list in inventory and plugging it into the static protocol. And then at the bottom here, we have another couple of sections, again, to define our BGP instances. Find my slideshow and check to see if indeed our playbook finished and it did. And so at this point, our network should be fully routable, fully built and routing. And now that that's done, let's break it. So what I'm going to do is run a, create a couple of failures within the network, and we can watch and see how BGP responds to those. I'm also going to demonstrate a couple of useful management functions that are pretty easily implemented using Ansible. So the first thing that we're going to do is talk about link failures because they happen. So looking again here, a lot of this is going to be in the context of these bottom two clouds here, which are both AS123. They're just two different sites. Think of it as a primary site, DR site, backup site, what have you. So in our first case, we're going to look at what happens to traffic flowing between the two sites if we happen to just fail a link. So under normal operations, traffic going from site A over to site B is going to take the shortest path through AS100, AS300, and then back into the other site. Once this link goes down, obviously we want BGP to figure it out and just do the right thing and use its other available uplink through AS200, 100, 300, and life is good. So if we flip over to my other consoles that are running, we've got a couple of things going on here. We've got a connection into router 1A, and we're just watching the debug output coming from that daemon. We have a trace route running from the firewall that's at site A to the firewall that's at site B. So nothing particularly fancy there. And this is a case where we can use Ansible. We can use the Ansible command to run a one-off command against anything that's in the inventory. So in this case, we're going to tell the AS100 router that we want it to shut down its VTNET 2 link. So it's going to connect and do that in success, and we start to see packet loss on our trace route. And then within a couple seconds here, BGP is going to pick up on the fact that it can't talk to its neighbor anymore, readjust its routing table. And if we restart the trace route, we see that we're still communicating, we're just communicating over a slightly different path. The next thing we'll talk about here is site failover. And this is useful if you're running production services from one site, and you want to pick those services up and start running them from your backup site without having to monkey with DNS and other, resort to other tricks like that. So the process to do this is actually really simple. And if we look at the Ansible configuration here, it's really just driven by the, it's just a reuse of the roles and playbooks that we've already defined for bootstrapping the routers. If we look over an inventory, we're looking at a couple of group file files for AS123 on the left-hand side here. And the top pane is just for the site A routers. And we're defining that our location is A. Down below for the entire AS, we're just finding that the active location is also A. So really the process of doing the failover is just running a playbook and telling Ansible, passing in a variable that says, you should reformat everything so that site B becomes active. And then reload the Ansible, or reload the bird configs and will be active at the new site in no time flat, or something approximately that. In reality, the failover is actually pretty quick on the order of a couple of seconds. So looking back at our windows here, starting from what is actually our Ansible host, I'm going to start a trace route into www.as123, which is not going to work for me. But that will. So we've got a trace route running, and so what the network looks like here. No, I knew I had another slide here. So looking again at our network, we have our Ansible host way up here, which is currently routing into our portable address space, which didn't make it on the diagram, but it's just another slash 24 and another slash 48 that we're announcing for the next few days. So we're going to start a trace route here, and then we're going to do a trace route, which didn't make it on the diagram, but it's just another slash 24 and another slash 48 that we're announcing from whatever site is active. So it's routing into site A to those portable addresses. So what we want to do is pick those addresses up, and rather than bring them in here, we want to bring them in over here to site B. And then we have a pre-cooked playbook that here again, all we need to do is fire it up. And so looking at this playbook invocation, if we open up the playbook, there's really nothing going on there other than... Oh, I was going to talk about that, and then I didn't. This is the extent of the playbook that's going to handle this failover. And the most interesting bits are really these last three tasks where we... We're on our last four tasks here, where we template our bird configuration files by calling the bird configuration rule. We actually go and update the Ansible inventory once we've made the change. So the change is persistent within this system, and then we reload the bird configuration. And then we make noise warning you that you've done something that you should probably be aware of. So looking at the playbook invocation, we've got our playbook name, and we're specifying a couple of additional variables on the command line. We're telling it that its new active location should be B, and that once it's done doing what it wants to do, that it should reconfigure the bird instances and activate the changes. So as that runs here, there is one sanity check in the playbook where it pauses, and has you confirm that yes, indeed, you're sure that you want to move your services to the other site, and yes, for sure. So it's going to go and configure its bird files, updates the inventory, reloads the bird configuration. We see the debug start to update the paving, and then if we look back at this trace route, rather than going into site A, we now see that it's starting to go into site B on its way into that portable service. So next thing we'll talk about in terms of management is what happens if we want to deprioritize a router. If we want to bring a router down for maintenance, or we're having trouble with our upstream provider, we just want to keep as much traffic off of that router as possible to reduce the disruption. Deprioritizing it is the way to go. So coming back to our trusty network diagram, we already have a link failure, and currently our traffic between sites is coming in through AS300. But what happens if we want to bring this router down, perform maintenance on it well, we're failed over to our DR site, which may be getting a little bit contrived, but it could happen. So the way that we do that again is by leveraging the Ansible roles that we already have defined, and here again just passing them a couple of different variables so that they manipulate the bird configuration files a little bit differently. And I have an example of that right here. What we're looking at here is the bird BGP include file that's templated out by Ansible, and a couple of parts that are most interesting here is based on a variable definition, we're changing the preference of the BGP protocols for this router down from the default 100 down to 99 so that any routes that come in over this protocol instance are going to be skipped in the presence of a better route. Down below here, we've got an export filter for routes that are being sent out of this BGP instance, and we are going to do BGP path propending, which basically amounts to inserting our own AS a number of additional times into the AS path before we send it to our upstream router, basically makes the path look artificially long and is going to cause the rest of the internet to go, well, I've got a better path over there, I'm not going to send any traffic to this router. So here again, it's just a case of running another playbook where once again we have a couple of variables defined, cute names aside, we're telling Ansible that we indeed want to, that we indeed want to deprioritize the router, we're telling it which router that we want to deprioritize, which in this case is 2B, and we're telling it again that we want it to configure, the bird changes automatically, and the configure bird variable is just another sanity check that I like to use for cases where I may want to stage configurations but not necessarily activate them right away, but in this case we do. So if we look back at our trace route that's running between sites, we can see that here again it's running through AS 200, 100, 300, and back into our other site. Once we run this playbook, this route is going to become a heck of a lot less attractive. So what BGP is actually going to do is it's going to say, well, this is a really long route, but I've got this shorter route by going up and around through AS 50 and back into our other uplink. So with that, we'll see if it does what it's supposed to. Here again, it's templating bird configurations, and there's no changes being made to the main bird configuration cell happening in the included files. We reconfigure bird, we start to see changes immediately on our other router, and our route changes, and if we reset that trace route we see that we have, we still have connectivity just over a much longer path. Oh, forgot all of my other things that I already talked about. So the very last thing that we're going to talk about is what happens if we cause some more mischief because failing a link wasn't fun enough, what happens if we blow up a router? So we have our link down, we have our deprioritized router, but what's going to happen if we take out a router somewhere in AS 50? And the answer here is that even though this router is deprioritized, it's still a route into our network that we'll use if we don't have any other option. So looking again, we still have our trace route running, still have our log output. If we flip over to our beehive host, we can tell beehive to just power off the host, and we immediately start seeing packet loss on our path, and once again, once BGP figures out that there's a problem, starts to reconfigure itself, then the network comes back to life. And we can see that once again, it is taking that, what is actually a shorter path through 200, 100, and 300. And life goes on, your pager goes off, you say everything's good and you go back to bed. And with that, that is what I have. So I'm interested for any questions that you may have. What's the latency look like for using Ansible to cut over change configurations to switch which one's the active? For us, with Puppet, Puppet's a bit too slow to do switching between A and B. For things like our car IPs, if you want to do the cutover, you have a bunch of shell scripts that we can help promote these different changes very rapidly. Right, between sites or between routers? Well, it's between web servers and site-to-site. Ansible, I don't know that I would use it for failing over servers or clusters, things like that, because it is slow. Carp is a lot faster. So actually, for cases, I didn't really talk about it in the slide show anywhere here. But between our two border routers, we've got our routers, we've got a firewall or a cluster firewall sitting behind it. I'm actually using Carp on the border routers so that firewall can just set a default gateway. It doesn't actually have to run VGP. And so then if a router fails within a second, we're back up and running and life is good. Anybody else? Are you using Ansible on your routers? Yes. Did you also try to move this to actually how do I routers or just KBSD devices? We're in production. We're running free BSD physical hardware. Ansible does have a lot. They've been putting a lot of effort into building out modules for controlling various network hardware. So at this point, there's support for Cisco, Juniper. This tool in the part of it is pretty unusable. So I tried to use it a couple of times and it was a bit of a mess. Back to what? I'm not familiar with that one. But I'm not using Ansible to control any vendor hardware. I haven't had a use case. My free BSD boxes work great for BGP and my switching doesn't change enough for me to have a good reason to integrate Ansible with it. Yeah, a lot of libraries know how to speak Cisco, Router, whatever. So it creates a lot of fun to go and push the cloud whatever means of having all this. Yeah, I've tried a lot to look at the affections that work a lot on our vendors, but I think that is a bit of a factor in the results. Yeah, if I can pull it up here real quick. Ansible has a huge library of modules that are provided by core. If you look in the fault from yesterday at 10 from Zabino, he was using the SRO to talk to Cisco all the time. It's gotten a lot better just in the last few months. It has. Yeah, the answer team has increased their networking developers and network module developers. It has increased internal to the previous one. Yeah, I think the acquisition by Red Hat has really put some pressure under them to support a lot of enterprise products, which is a good thing. So VMware support, AWS support has gotten huge. So there's just a ton of flexibility that you don't have to reinvent. And additionally, there's Ansible Galaxy, which is a repository of pre-built roles and other reusable code that other people have submitted that's useful as well. If you can read it. If you can read it. Exactly. I think I'm using exactly one module from Galaxy. Did you have a question? Ansible has a lot of built-in protection against them. There have been a number of vulnerabilities that have been figured out surrounding that and escaping, quoting, things like that. And Ansible is actually pretty good about warning you when you're doing something you shouldn't be such as trusting client input that you shouldn't be. Yeah, it was an incidental client problem actually connected to some host and just by injecting some commands it was able to compromise the controller that was executing the commands. I think that's an area of this case. Yeah. It's only good to just wait. Yeah, that's always a possibility and a concern. Definitely. Yeah. If you have options for role-based access. Right. Right. Anybody else? Yes, it should be. Protect your controller, I think is the bottom line. Well, if nobody else has anything, I'd like to thank you for coming.
 Alright, good afternoon. Thanks for coming. This is Plumbing the Internet and we're going  to be talking about BGP automation and BSD obviously because that's why we're all here.  So first of all, I'm curious, who knows what in the room? Who's worked with BGP before?  How about BGP on BSD? How about automation? Hopefully everyone in the room. How about with  Ansible? Awesome. So we can skip through some of the boring parts then I hope. And I'm going  to be cruising along here at a high rate of speed so if I lose you, by all means, flight  me now. So before we get going here, my name is Thomas Johnson. I don't do many slideshows  so it's a little rough around the edges. And how I got to be here, I got started with BSD  in 2010. In 2013, we started planning for a high availability disaster recovery solution  and decided that that was going to necessitate us figuring out BGP. So I bought a book and  started reading and early 2014, we turned up our primary site for BGP. We turned up DR  site a couple months later and then 2015, we got started with Ansible. And then early  2018, we had our first production test of our DR site because this was the parking lot  of our primary data center the week before the Super Bowl. And we decided that we didn't  really want to have our production there. But it wasn't all bad because when you get  your poop in a group and actually use your DR facility, then you get away with stuff  like this. So I'm going to start here by talking a little bit about Ansible. And since part  of the presentation here is to demonstrate what you can do with automating your BGP with  Ansible, it seemed like a really great and probably really foolish thing to build the  demonstration lab as we go here. So we'll be turning that loose in a couple minutes.  But real quick, couple of things about Ansible that are worth noting. It's written in Python.  It's identigent operations so much like Puppet or the other automation systems. You're telling  Ansible how you want things to be not necessarily what you want them to be doing. It's a very  thin layer so your client nodes don't really need much in the way of prerequisites other  than stuff that you're probably already running in your network anyways. As far as requirements,  like I said, there's not a whole lot you need to install Ansible on your master nodes. And  then your client nodes are going to need SSH, Python, stuff that we already have installed.  So you can be up and running without, you know, going to great lengths.  I'm going to start to jump right into some of the configuration of Ansible here. And  we're going to start with inventory, which is the component of Ansible where we tell  Ansible everything that we wanted to know about our network and everything that we wanted  to know about our network. And at the heart of that is the hosts file, which is where  we define all of the hosts that we want Ansible to know about. And we can also organize those  into groups to simplify configuration. We can run Ansible plays against a group of hosts.  So on the left-hand side here, we've got a group called Beehive Guests. And we've got  a bunch of guests that are listed underneath that. The right-hand side is a variation on  a theme that's a group composed of groups. So we're aggregating a bunch of subgroups  into one group, makes it really handy to share configuration amongst other things. Moving  deeper into the inventory, we get into the host files and group files. And this is where  we really kind of get into the heart of what inventory is all about. It's a free-form  AMLA file. Ansible doesn't really put much in the way of restrictions on how you can  format data that you're putting into this file. In fact, pretty much everything on the  left-hand side, which is a host files file, is completely arbitrary. It's put together  the way that I want to use it in my implementation. So this is a host files file. A couple of  things that are worth noting here. I can even use my laser pointer. We're defining our host  configuration path where we want Ansible to go and look for configuration files if it  needs them. And then down below, we're starting to describe how we want Ansible to structure  the network build for this particular host. And the nice thing about that is we're defining  it all in one single file or a group of files not spread out over a whole mess of files.  On the right-hand side, we've got an example of a group vars file, which there again,  it's configuration applies to an entire group. And it brings up a BSD gotcha, which is that  Ansible user bin Python as its default path for the Python interpreter. So unless you've  somehow got that in your BSD implementation, you're going to need to update that. And at  this point, we have enough configuration in Ansible that we can actually start doing things.  The other piece to Ansible that's just as important as inventory is going to be playbooks.  And playbooks are really the top level unit of how we do things in Ansible. Playbook's  nothing more than a text file. Again, it's written largely in YAML or entirely in YAML.  It's composed of one or more plays that's run against a host or a group of hosts. And  within that play, we have one or more, zero or more tasks that are actually applied to  that host or the host in the play. So we have another example here. Can everyone read the  small text? Okay. If you can't read things in the back, let me know and I'll do what  I can to make them bigger, especially when I start pulling up terminals. But this is  an example play. We're running this play against the Beehive Guests group. And then we've got  a couple of tasks that are listed here. We're basically just checking a couple variables  that are defined and we're starting a path and doing nothing more than that at this point.  The other interesting part of Ansible, similar to other systems, is there's a concept of  roles, which is basically just a predefined structure for code reuse. So if you have a  piece of a group of tasks that are self-contained and you want to be able to use all over the  place, roles are the way to do it. They have a predefined directory structure, which is  what we see on the right-hand side here. And we've got directories for tasks, for default  variables, handlers. We can even include other roles from a single role. So there's a lot  of flexibility there. The other thing I'll point out here before we go too much further  is VM Beehive, which is a tool that I found on GitHub when I was looking for something  I could use to avoid having to figure out all of the Beehive provisioning by hand. And  it's basically just a light CLI interface front end to Beehive. It lets you do all of  your network configuration automatically. And the question is, as Baptiste pointed out  earlier, is why doesn't FreeBSD have something like this in base already? So at this point,  a little bit about the demo that we're going to be, that I'm going to be building up here.  It's really essentially starting with nothing more than a Beehive host. And we've got an  Ansible guest running on that host. And from that, we've got, on the host, we have a pre  built stem cell of a Zvol. And all of our guests are going to be cloned from that. And  then bootstrapped is necessary to build our network. And so once we get this going, about  17 minutes later from those first two hosts we end up with, should end up with a fully  routed network. And if you guys saw the GitHub links at the beginning of the presentation  and pulled the diagram that I forgot to mention that you should pull up, then you already  have a copy of this in front of you. So real quick here, a high level view of what  the bootstrap process looks like for these routers. The first thing that we're doing,  you can see we're doing a ZFS clone to clone the stem cell into the host name that we want.  And we import that volume so that we can do our bootstrap operations on it. We call a  couple of very minimal roles that are going to build up the basic configuration that we  need for this host to run, which really amounts to building out the config files for the routing  daemon and templating RC.conf. And really there's nothing more than that and we're  going to end up with a working router. At that point we export the Zvol and boot the  guest. And this is my reminder to go and actually start the playbook. So we'll flip over here  and can everybody see this? Well enough anyways. Good. So what we're looking at here is we're  looking at the ants playbook command 001 build demo is just a real lightweight playbook  that calls a role that does the bootstrapping of the entire network. And we're skipping  a tag here for some slow tasks that we don't really need to run because they take even  longer. And we'll set it loose. And the first thing that it's doing here is something I  didn't mention. It's going through and it's doing some initial checks of the beehive host  to make sure that the bridge is in such that we're going to need for all of the guests  are in place and ready to go. So while that's running, I'm going to flip back and start  talking about BGP and we'll start by talking about what the internet is. The internet,  as I'm sure many of us are aware, is just a handful of autonomous systems that need  to communicate with each other. And the way that everybody communicates with each other  for better or for worse is using BGP. Now you guys can read Wikipedia just as well as  I can regurgitate it, probably better. So I'm just going to point out a couple of things  on BGP that are worth noting here. It's a distance vector protocol. It measures the  shortest AS path between two autonomous systems, which basically means we're doing path determination  based on the number of organizations that our traffic is going to be passing through.  There's two varieties of BGP. There's an external and an internal variety. The external variety  is used to pass routing information between organizations. The internal variety is used  to pass that information amongst internal BGP speakers within an organization. The one  got you here is that generally you still need an interior protocol to discover your own  internal network. Generally BGP doesn't handle that for you. So in this case we'll be using  a less PF, but any sort of interior protocol works. So we're going to do BGP. What do  we need? Well, the first thing we need is money, but fortunately we don't need a whole  heck of a lot of it to get started, especially if you're a smaller organization. We're going  to need numbers. We're going to need an AS number. We need IP addresses. We're going  to need routers and we're going to need connectivity. And for those last two points, we're going  to want more than one. If you're going to go to the time and trouble of deploying BGP,  there's no point in deploying it on just a single router, single connection. You're  wasting your time. Internet registries are the place to go to get number resources. If  you're in the United States or Canada, you're going to be talking to Aaron. That's where  my experience is. Your mileage may vary if you're from outside of that region, but these  are the folks that you need to go talk to. The one thing that I will mention is that  if you start down this process of getting numbering resources and you have questions  speaking the context of Aaron, Aaron is your friend. They're not a gatekeeper. They're  not the bad guy. Their mission is to help you get resources to do things on the Internet.  And so they're generally willing to go well out of their way to help you out.  Autonomous system numbers simply is a number that identifies your organization. There's  two byte and four byte varieties. Two byte are the legacy variety. I don't even know  if they're issuing two byte AS numbers anymore. Who are you with?  Okay.  They will by default issue you a four byte number. I might see specifically ask for a  two byte number.  I have a two byte number because we were in early enough. Thanks for the clarification.  IPv6, you get addresses from your registrar. Your allocation is going to depend on your  organization size. And the one thing that I'll point out here, well, two things I'll  point out here. Minimum announcement size that's going to be accepted generally is going  to be a slash 48. And as you start to work out your addressing scheme, I really recommend  not allocating anything less than a slash 64. Even if you deploy a smaller sum that  allocates a slash 64.  IPv4, it's the legacy protocol on the internet. Everyone in this room knows about it. It's  exhausted. Minimum announcement size is a slash 24. I think there's enough said there.  As far as obtaining resources, you can. And I'm going to get to that. But I like that  picture. So blocks can be had. The internet registries have waiting lists. There's a transfer  market that you can take your chances on if you're so inclined. But there's another trick.  At least for Aaron, there's the 410 rule, which refers to the section in their numbering  policy. It's a slash 10 that's set aside for IPv6 transition. And the bottom line here,  the big takeaway here is that if you already have an IPv6 allocation and you want to dual  stack your application, you have justification to request a v4 block.  I may add for if you are with Rive NCC, become a local internet registry of the year. It's  something like 2,000 and 1,000 euros.  Okay.  That's good to know. Does Rive still have a pool of blocks? I assume. Okay. Good to  know.  Okay.  Okay. That's good to know.  Connectivity real quick. There's generally three methods of obtaining connectivity. You  can either buy transit, which means you're paying somebody else to provide you a path  to the larger internet. Internet exchange points are a really great way to exchange traffic  with geologically local peers. There are also great source of information. And then there's  peering relationships where you're exchanging routes with another single organization.  Couple of connectivity gotchas real quick. If you're starting to deploy BGP and bring  up transit and things like that, you should expect to be providing a letter of agency,  excuse me, basically acknowledging that you are authorized to announce the blocks that  you say that you want to announce. Expect that you're going to pay cross-connect fees  if you're in a data center, co-location. And the last point there, you may want to at least  make some effort to make sure that your fancy redundant fiber isn't running through the  same conduit because that can cause problems too.  So switching back to FreeBSD here, talk about what we need to turn our FreeBSD into a router.  And in a nutshell, you're looking at it. Pretty much everything we need at a high level is  done through RC.conf. What we're looking at here, we're defining our network interfaces.  We're enabling routing. And we're enabling our routing daemon. And there's really not  too much more than that. In reality, things like VLAN, carp, lag bundles are going to  make this more complicated, but we all know that's going to happen anyway.  Real quick on firewalling. Firewalling, generally, I try to separate the filtering and the routing  functionality roles on different machines, but there are cases where it's useful to do  filtering on your firewall. The one thing to be aware of is keeping state on connections.  When if you move to a, if you're moving to a VGP model where you've got multiple routers  and uplinks, it's entirely likely and probable that traffic's going to come in one direction  and go out another direction. And if we're keeping state for that, you're going to have  all sorts of hard to die nose problems and you'll lose hair and just don't do it.  So moving on to our routing daemon. The routing daemon I'm using here is Bird. It's open source.  It's pretty widely, widely used. A couple of highlights on it. There's two, two versions  that are current. We're going to be talking about version 1.6. That supports most of the  common protocols that you're going to run into, stool stacked. The configuration structure  is, is simple and easy to get your head around. And it has a utility called Bird C, which  is very useful for controlling your routing instance and then monitoring it as well. So  as we start to look back at our, at the test network here, we're going to take a look at  some of these different relationships that, that we're, we're creating with Bird. And  so we're looking at a, a portion of the test network here. And we're going to be looking  at it largely in the context of this AS-123 router 1A. So, so our first relationship here  is our relationship to another AS. And that's going to be where we're using EVGP.  Within the AS, we have multiple BGP speakers. And this is where we're going to be using  IBGP to, to share and communicate that external routing data to all of the different border  routers on the, on the network. Internal to the network we're using OSP-APFAs I mentioned  earlier to discover all of the routes that are available internally.  And then additionally, we're using a couple of static protocols largely to, as a means  of injecting routes into BGP. So rather than, especially from the point of view of an end-user  rather than, than blindly redistributing things into BGP, we want to be very sure we know  what we're, we're injecting into the, the global internet.  So with that in mind, we'll take a look at how these relationships actually manifest  themselves on a running bird. And so at this point, I've been moving too quick and the  network isn't totally up yet. But hopefully it is up enough here that we can at least  take a look at a couple of, couple of things that are going on in bird. And these are a  couple of the commands that I use most frequently just to get an idea of what's going on.  So show protocols is going to give you a quick bird's eye view of all the different protocol  instances that are running on your, on your bird and what the status of them are.  If you want to dig into a particular protocol instance and see more details about it, we  can do that. And this one fortunately has come up so we can talk about it a little bit.  So this is our, this is our EBGP instance from this router out to its, its upstream  neighbor. And we can see that the protocol is up. The interesting thing, the usually  the first thing that I go to look at when I'm pulling protocol is to see how many, how  many routes are being imported from this, from this pier, how many we're exporting,  which is, which should correspond to just the static routes that we're injecting manually.  And then there's a preferred counter here. This is how many routes that we're actually  preferring this, this path for. Birdsea is also an interactive shell, if I can spell  it correctly, which is actually probably even more useful because it has inline help. So  we can do whatever we want from here, like tell bird to reload its configuration if we've  made some configuration changes to it. But generally we don't have to do that because  we're running everything through Ansible right now and Ansible is going to handle that for  us. So moving along here, we didn't look at the bird configuration, which is why I put  the slide in here so I won't forget about it. So this is the actual configuration that's  driving what's going on this router. So looking at the left-hand side here, a lot of what  we see at the very top is boilerplate definition. I'm not going to spend a whole lot of time  on it here. The more interesting bits, or as we get down into OSPF, we can see that  we've defined interfaces for each of the OSPF, each of the interfaces we want to run OSPF  on. Some of them are running normal OSPF, some of them are just stub interfaces, in  particular our uplink interfaces we don't want to run OSPF on. You can see a couple  of static instances defined here. For static BGP, portable BGP, this is where we're defining  the routes we want to inject into BGP. And the distinction between the two is more applicable  in a multi-site configuration. I'm going to get into that in a little bit. And then at  the bottom here, we see our definitions for our EBGP and IBGP sessions. And they look  pretty bare here. We're really just defining what our neighbor is. And the reason that  they look so bare is because we're templating them. So these configurations are templated  by an included file, which I forgot to point out at the top of the file, where we plug  all of our common configuration that's going to be the same for all of our EBGP instances  on this router. We just plug it into a single template and then we can reuse it and reuse  it and reuse it. And there's not a whole lot more that we're defining here. We're defining  our AS number. We are defining a little bit of a hack here to allow our own AS number to  appear in the AS path. This is not generally allowed by BGP to prevent loops, but it is  useful if you're running multi-site configuration. You don't have a point-to-point link. If we  want to route between the two sites, we have to be able to allow the AS number. So looking  at the bottom of this file here, I'm going to cut off here, but the IBGP template is  largely the same as EBGP with just a couple of changes. So now that we've looked at the  BIRD configuration, we'll look at the actual template that that configuration file is generated  from. And we're doing that with Ansible. Ansible uses the Gingya templating language for all  of its templating plays. It uses it for the playbooks and configuration as well. And it's  very flexible. It draws information in from inventory, which is why we're finding it all  there. And it's basically what allows me to generate 15 routers in this demonstration  from what amounts to about one set of configurations. Oops, jumped ahead. So we will take a look  at these templates. And so first we'll look back at the inventory a little bit here. And  this is the inventory, again, for router 1A. We've got, again, some boilerplate at the  top that's less interesting. But as we get down here into the network configuration,  again, this is all arbitrary. You can structure it in whatever way makes sense to you. But  we're defining the addressing for each of our interfaces. We're defining what interface  name it should be associated with. In the case of some interfaces, we're defining carp  addresses if we want to do high availability there. We're defining what the OSPF relationship  is. And then as we move down into the bird section of this dictionary, we're starting  to see all of our routing-related configurations, like what our AS number is. We've got a section  here that's defining what prefixes we're going to be putting into the static and portable  protocols. And if you look here, we've got additional variables that are being pulled  in using the double braces, which basically just means that I'm defining what these lists  are elsewhere in the configuration, actually, in groupfars files, because they're common  across AS 123. And then I'm pulling them in here. At the bottom of the file, we see the  definitions for our BGP sessions. And again, because everything's very templated, there's  really not a whole lot that we need to define here. Looking over here on the right side,  we're actually looking at the bird.cont file that's before it's templated by Ansible.  So again, here we see double braces, which indicate that we're doing a variable substitution,  bringing something in from inventory and plugging it into the file. The OSPF section here, we're  starting to see some logic being used. We're starting to see a for loop where it's looping  through all of the interfaces that are defined in inventory, looking to see if it should  be configured for OSPF. Again, we're doing looping to define the static protocols, basically  taking everything that's in a list of, a list in inventory and plugging it into the static  protocol. And then at the bottom here, we have another couple of sections, again, to  define our BGP instances. Find my slideshow and check to see if indeed our playbook finished  and it did. And so at this point, our network should be fully routable, fully built and  routing. And now that that's done, let's break it. So what I'm going to do is run a, create  a couple of failures within the network, and we can watch and see how BGP responds to those.  I'm also going to demonstrate a couple of useful management functions that are pretty  easily implemented using Ansible. So the first thing that we're going to do is talk about  link failures because they happen. So looking again here, a lot of this is going to be in  the context of these bottom two clouds here, which are both AS123. They're just two different  sites. Think of it as a primary site, DR site, backup site, what have you. So in our first  case, we're going to look at what happens to traffic flowing between the two sites if  we happen to just fail a link. So under normal operations, traffic going from site A over  to site B is going to take the shortest path through AS100, AS300, and then back into the  other site. Once this link goes down, obviously we want BGP to figure it out and just do the  right thing and use its other available uplink through AS200, 100, 300, and life is good.  So if we flip over to my other consoles that are running, we've got a couple of things going  on here. We've got a connection into router 1A, and we're just watching the debug output  coming from that daemon. We have a trace route running from the firewall that's at site A  to the firewall that's at site B. So nothing particularly fancy there. And this is a case  where we can use Ansible. We can use the Ansible command to run a one-off command against  anything that's in the inventory. So in this case, we're going to tell the AS100 router  that we want it to shut down its VTNET 2 link. So it's going to connect and do that in success,  and we start to see packet loss on our trace route. And then within a couple seconds here,  BGP is going to pick up on the fact that it can't talk to its neighbor anymore, readjust  its routing table. And if we restart the trace route, we see that we're still communicating,  we're just communicating over a slightly different path.  The next thing we'll talk about here is site failover. And this is useful if you're running  production services from one site, and you want to pick those services up and start running  them from your backup site without having to monkey with DNS and other, resort to other  tricks like that. So the process to do this is actually really simple. And if we look  at the Ansible configuration here, it's really just driven by the, it's just a reuse of the  roles and playbooks that we've already defined for bootstrapping the routers. If we look  over an inventory, we're looking at a couple of group file files for AS123 on the left-hand  side here. And the top pane is just for the site A routers. And we're defining that our  location is A. Down below for the entire AS, we're just finding that the active location  is also A. So really the process of doing the failover is just running a playbook and  telling Ansible, passing in a variable that says, you should reformat everything so that  site B becomes active. And then reload the Ansible, or reload the bird configs and will  be active at the new site in no time flat, or something approximately that. In reality,  the failover is actually pretty quick on the order of a couple of seconds. So looking back  at our windows here, starting from what is actually our Ansible host, I'm going to start  a trace route into www.as123, which is not going to work for me. But that will. So we've  got a trace route running, and so what the network looks like here. No, I knew I had  another slide here. So looking again at our network, we have our Ansible host way up here,  which is currently routing into our portable address space, which didn't make it on the  diagram, but it's just another slash 24 and another slash 48 that we're announcing for  the next few days. So we're going to start a trace route here, and then we're going to  do a trace route, which didn't make it on the diagram, but it's just another slash 24  and another slash 48 that we're announcing from whatever site is active. So it's routing  into site A to those portable addresses. So what we want to do is pick those addresses  up, and rather than bring them in here, we want to bring them in over here to site B.  And then we have a pre-cooked playbook that here again, all we need to do is fire it up.  And so looking at this playbook invocation, if we open up the playbook, there's really  nothing going on there other than... Oh, I was going to talk about that, and then I  didn't. This is the extent of the playbook that's going to handle this failover. And  the most interesting bits are really these last three tasks where we... We're on our  last four tasks here, where we template our bird configuration files by calling the bird  configuration rule. We actually go and update the Ansible inventory once we've made the  change. So the change is persistent within this system, and then we reload the bird  configuration. And then we make noise warning you that you've done something that you should  probably be aware of. So looking at the playbook invocation, we've got our playbook name,  and we're specifying a couple of additional variables on the command line. We're telling  it that its new active location should be B, and that once it's done doing what it wants  to do, that it should reconfigure the bird instances and activate the changes. So as  that runs here, there is one sanity check in the playbook where it pauses, and has you  confirm that yes, indeed, you're sure that you want to move your services to the other  site, and yes, for sure. So it's going to go and configure its bird files, updates the  inventory, reloads the bird configuration. We see the debug start to update the paving,  and then if we look back at this trace route, rather than going into site A, we now see  that it's starting to go into site B on its way into that portable service.  So next thing we'll talk about in terms of management is what happens if we want to  deprioritize a router. If we want to bring a router down for maintenance, or we're having  trouble with our upstream provider, we just want to keep as much traffic off of that router  as possible to reduce the disruption. Deprioritizing it is the way to go. So coming back to our  trusty network diagram, we already have a link failure, and currently our traffic between  sites is coming in through AS300. But what happens if we want to bring this router down,  perform maintenance on it well, we're failed over to our DR site, which may be getting  a little bit contrived, but it could happen. So the way that we do that again is by leveraging  the Ansible roles that we already have defined, and here again just passing them a couple  of different variables so that they manipulate the bird configuration files a little bit differently.  And I have an example of that right here. What we're looking at here is the bird BGP  include file that's templated out by Ansible, and a couple of parts that are most interesting  here is based on a variable definition, we're changing the preference of the BGP protocols  for this router down from the default 100 down to 99 so that any routes that come in  over this protocol instance are going to be skipped in the presence of a better route.  Down below here, we've got an export filter for routes that are being sent out of this  BGP instance, and we are going to do BGP path propending, which basically amounts to  inserting our own AS a number of additional times into the AS path before we send it to  our upstream router, basically makes the path look artificially long and is going to cause  the rest of the internet to go, well, I've got a better path over there, I'm not going  to send any traffic to this router. So here again, it's just a case of running another  playbook where once again we have a couple of variables defined, cute names aside, we're  telling Ansible that we indeed want to, that we indeed want to deprioritize the router,  we're telling it which router that we want to deprioritize, which in this case is 2B,  and we're telling it again that we want it to configure, the bird changes automatically,  and the configure bird variable is just another sanity check that I like to use for cases  where I may want to stage configurations but not necessarily activate them right away,  but in this case we do. So if we look back at our trace route that's running between  sites, we can see that here again it's running through AS 200, 100, 300, and back into our  other site. Once we run this playbook, this route is going to become a heck of a lot less  attractive. So what BGP is actually going to do is it's going to say, well, this is a  really long route, but I've got this shorter route by going up and around through AS 50  and back into our other uplink. So with that, we'll see if it does what it's supposed to.  Here again, it's templating bird configurations, and there's no changes being made to the main  bird configuration cell happening in the included files. We reconfigure bird, we start to see  changes immediately on our other router, and our route changes, and if we reset that trace  route we see that we have, we still have connectivity just over a much longer path.  Oh, forgot all of my other things that I already talked about. So the very last thing that  we're going to talk about is what happens if we cause some more mischief because failing  a link wasn't fun enough, what happens if we blow up a router? So we have our link down,  we have our deprioritized router, but what's going to happen if we take out a router somewhere  in AS 50? And the answer here is that even though this router is deprioritized, it's  still a route into our network that we'll use if we don't have any other option. So  looking again, we still have our trace route running, still have our log output. If we flip  over to our beehive host, we can tell beehive to just power off the host, and we immediately  start seeing packet loss on our path, and once again, once BGP figures out that there's  a problem, starts to reconfigure itself, then the network comes back to life. And we can  see that once again, it is taking that, what is actually a shorter path through 200, 100,  and 300. And life goes on, your pager goes off, you say everything's good and you go  back to bed. And with that, that is what I have. So I'm interested for any questions  that you may have.  What's the latency look like for using Ansible to cut over change configurations to switch  which one's the active? For us, with Puppet, Puppet's a bit too slow to do switching between  A and B. For things like our car IPs, if you want to do the cutover, you have a bunch of  shell scripts that we can help promote these different changes very rapidly.  Right, between sites or between routers?  Well, it's between web servers and site-to-site.  Ansible, I don't know that I would use it for failing over servers or clusters, things  like that, because it is slow. Carp is a lot faster. So actually, for cases, I didn't really  talk about it in the slide show anywhere here. But between our two border routers, we've  got our routers, we've got a firewall or a cluster firewall sitting behind it. I'm  actually using Carp on the border routers so that firewall can just set a default gateway.  It doesn't actually have to run VGP. And so then if a router fails within a second, we're  back up and running and life is good.  Anybody else?  Are you using Ansible on your routers?  Yes.  Did you also try to move this to actually how do I routers or just KBSD devices?  We're in production. We're running free BSD physical hardware. Ansible does have a lot.  They've been putting a lot of effort into building out modules for controlling various  network hardware. So at this point, there's support for Cisco, Juniper.  This tool in the part of it is pretty unusable. So I tried to use it a couple of times and  it was a bit of a mess.  Back to what?  I'm not familiar with that one. But I'm not using Ansible to control any vendor hardware.  I haven't had a use case. My free BSD boxes work great for BGP and my switching doesn't  change enough for me to have a good reason to integrate Ansible with it.  Yeah, a lot of libraries know how to speak Cisco, Router, whatever. So it creates a lot  of fun to go and push the cloud whatever means of having all this.  Yeah, I've tried a lot to look at the affections that work a lot on our vendors, but I think  that is a bit of a factor in the results.  Yeah, if I can pull it up here real quick.  Ansible has a huge library of modules that are provided by core.  If you look in the fault from yesterday at 10 from Zabino, he was using the SRO to talk  to Cisco all the time.  It's gotten a lot better just in the last few months.  It has.  Yeah, the answer team has increased their networking developers and network module developers.  It has increased internal to the previous one.  Yeah, I think the acquisition by Red Hat has really put some pressure under them to support  a lot of enterprise products, which is a good thing.  So VMware support, AWS support has gotten huge.  So there's just a ton of flexibility that you don't have to reinvent.  And additionally, there's Ansible Galaxy, which is a repository of pre-built roles and  other reusable code that other people have submitted that's useful as well.  If you can read it.  If you can read it.  Exactly.  I think I'm using exactly one module from Galaxy.  Did you have a question?  Ansible has a lot of built-in protection against them.  There have been a number of vulnerabilities that have been figured out surrounding that  and escaping, quoting, things like that.  And Ansible is actually pretty good about warning you when you're doing something you shouldn't be  such as trusting client input that you shouldn't be.  Yeah, it was an incidental client problem actually connected to some host and just by  injecting some commands it was able to compromise the controller that was executing the commands.  I think that's an area of this case.  Yeah.  It's only good to just wait.  Yeah, that's always a possibility and a concern.  Definitely.  Yeah.  If you have options for role-based access.  Right.  Right.  Anybody else?  Yes, it should be.  Protect your controller, I think is the bottom line.  Well, if nobody else has anything, I'd like to thank you for coming.
Thomas_Johnson__Plumbing_the_Internet__BSD_style_0_107.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_108_251.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_252_515.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_516_2447.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_2448_2903.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_2904_3035.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_3036_13391.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_13392_13991.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_13992_15287.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_15288_16691.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_16692_19619.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_19620_20711.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_20712_32003.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_32004_34811.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_34812_34883.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_34884_35003.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_35004_38507.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_38508_38567.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_38568_39575.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_39576_39719.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_39720_43823.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_43824_43871.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_43872_43931.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_43932_44003.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_44004_44255.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_44256_44315.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_44316_44927.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_44928_46403.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_46404_46451.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_46452_48395.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_48396_48455.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_48456_49055.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_49056_49103.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_49104_50747.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_50748_51731.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_51732_51779.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_51780_51827.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_51828_52703.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_52704_52775.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_52776_53219.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_53220_53291.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_53292_54323.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_54324_54959.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_54960_56087.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_56088_56135.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_56136_56183.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_56184_56267.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_56268_56783.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_56784_57503.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_57504_57827.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_57828_57875.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_57876_59747.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_59748_61235.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_61236_61283.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_61284_61811.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_61812_62867.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_62868_62915.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_62916_63155.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_63156_63407.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_63408_64091.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_64092_64139.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_64140_65339.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_65340_65411.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_65412_66839.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_66840_69983.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_69984_70331.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_70332_70631.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_70632_72647.png Thomas_Johnson__Plumbing_the_Internet__BSD_style_72648_75576.png
Alright, good afternoon. Thanks for coming. This is Plumbing the Internet and we're going to be talking about BGP automation and BSD obviously because that's why we're all here. So first of all, I'm curious, who knows what in the room? Who's worked with BGP before? How about BGP on BSD? How about automation? Hopefully everyone in the room. How about with Ansible? Awesome. So we can skip through some of the boring parts then I hope. And I'm going to be cruising along here at a high rate of speed so if I lose you, by all means, flight me now. So before we get going here, my name is Thomas Johnson. I don't do many slideshows so it's a little rough around the edges. And how I got to be here, I got started with BSD in 2010. In 2013, we started planning for a high availability disaster recovery solution and decided that that was going to necessitate us figuring out BGP. So I bought a book and started reading and early 2014, we turned up our primary site for BGP. We turned up DR site a couple months later and then 2015, we got started with Ansible. And then early 2018, we had our first production test of our DR site because this was the parking lot of our primary data center the week before the Super Bowl. And we decided that we didn't really want to have our production there. But it wasn't all bad because when you get your poop in a group and actually use your DR facility, then you get away with stuff like this. So I'm going to start here by talking a little bit about Ansible. And since part of the presentation here is to demonstrate what you can do with automating your BGP with Ansible, it seemed like a really great and probably really foolish thing to build the demonstration lab as we go here. So we'll be turning that loose in a couple minutes. But real quick, couple of things about Ansible that are worth noting. It's written in Python. It's identigent operations so much like Puppet or the other automation systems. You're telling Ansible how you want things to be not necessarily what you want them to be doing. It's a very thin layer so your client nodes don't really need much in the way of prerequisites other than stuff that you're probably already running in your network anyways. As far as requirements, like I said, there's not a whole lot you need to install Ansible on your master nodes. And then your client nodes are going to need SSH, Python, stuff that we already have installed. So you can be up and running without, you know, going to great lengths. I'm going to start to jump right into some of the configuration of Ansible here. And we're going to start with inventory, which is the component of Ansible where we tell Ansible everything that we wanted to know about our network and everything that we wanted to know about our network. And at the heart of that is the hosts file, which is where we define all of the hosts that we want Ansible to know about. And we can also organize those into groups to simplify configuration. We can run Ansible plays against a group of hosts. So on the left-hand side here, we've got a group called Beehive Guests. And we've got a bunch of guests that are listed underneath that. The right-hand side is a variation on a theme that's a group composed of groups. So we're aggregating a bunch of subgroups into one group, makes it really handy to share configuration amongst other things. Moving deeper into the inventory, we get into the host files and group files. And this is where we really kind of get into the heart of what inventory is all about. It's a free-form AMLA file. Ansible doesn't really put much in the way of restrictions on how you can format data that you're putting into this file. In fact, pretty much everything on the left-hand side, which is a host files file, is completely arbitrary. It's put together the way that I want to use it in my implementation. So this is a host files file. A couple of things that are worth noting here. I can even use my laser pointer. We're defining our host configuration path where we want Ansible to go and look for configuration files if it needs them. And then down below, we're starting to describe how we want Ansible to structure the network build for this particular host. And the nice thing about that is we're defining it all in one single file or a group of files not spread out over a whole mess of files. On the right-hand side, we've got an example of a group vars file, which there again, it's configuration applies to an entire group. And it brings up a BSD gotcha, which is that Ansible user bin Python as its default path for the Python interpreter. So unless you've somehow got that in your BSD implementation, you're going to need to update that. And at this point, we have enough configuration in Ansible that we can actually start doing things. The other piece to Ansible that's just as important as inventory is going to be playbooks. And playbooks are really the top level unit of how we do things in Ansible. Playbook's nothing more than a text file. Again, it's written largely in YAML or entirely in YAML. It's composed of one or more plays that's run against a host or a group of hosts. And within that play, we have one or more, zero or more tasks that are actually applied to that host or the host in the play. So we have another example here. Can everyone read the small text? Okay. If you can't read things in the back, let me know and I'll do what I can to make them bigger, especially when I start pulling up terminals. But this is an example play. We're running this play against the Beehive Guests group. And then we've got a couple of tasks that are listed here. We're basically just checking a couple variables that are defined and we're starting a path and doing nothing more than that at this point. The other interesting part of Ansible, similar to other systems, is there's a concept of roles, which is basically just a predefined structure for code reuse. So if you have a piece of a group of tasks that are self-contained and you want to be able to use all over the place, roles are the way to do it. They have a predefined directory structure, which is what we see on the right-hand side here. And we've got directories for tasks, for default variables, handlers. We can even include other roles from a single role. So there's a lot of flexibility there. The other thing I'll point out here before we go too much further is VM Beehive, which is a tool that I found on GitHub when I was looking for something I could use to avoid having to figure out all of the Beehive provisioning by hand. And it's basically just a light CLI interface front end to Beehive. It lets you do all of your network configuration automatically. And the question is, as Baptiste pointed out earlier, is why doesn't FreeBSD have something like this in base already? So at this point, a little bit about the demo that we're going to be, that I'm going to be building up here. It's really essentially starting with nothing more than a Beehive host. And we've got an Ansible guest running on that host. And from that, we've got, on the host, we have a pre built stem cell of a Zvol. And all of our guests are going to be cloned from that. And then bootstrapped is necessary to build our network. And so once we get this going, about 17 minutes later from those first two hosts we end up with, should end up with a fully routed network. And if you guys saw the GitHub links at the beginning of the presentation and pulled the diagram that I forgot to mention that you should pull up, then you already have a copy of this in front of you. So real quick here, a high level view of what the bootstrap process looks like for these routers. The first thing that we're doing, you can see we're doing a ZFS clone to clone the stem cell into the host name that we want. And we import that volume so that we can do our bootstrap operations on it. We call a couple of very minimal roles that are going to build up the basic configuration that we need for this host to run, which really amounts to building out the config files for the routing daemon and templating RC.conf. And really there's nothing more than that and we're going to end up with a working router. At that point we export the Zvol and boot the guest. And this is my reminder to go and actually start the playbook. So we'll flip over here and can everybody see this? Well enough anyways. Good. So what we're looking at here is we're looking at the ants playbook command 001 build demo is just a real lightweight playbook that calls a role that does the bootstrapping of the entire network. And we're skipping a tag here for some slow tasks that we don't really need to run because they take even longer. And we'll set it loose. And the first thing that it's doing here is something I didn't mention. It's going through and it's doing some initial checks of the beehive host to make sure that the bridge is in such that we're going to need for all of the guests are in place and ready to go. So while that's running, I'm going to flip back and start talking about BGP and we'll start by talking about what the internet is. The internet, as I'm sure many of us are aware, is just a handful of autonomous systems that need to communicate with each other. And the way that everybody communicates with each other for better or for worse is using BGP. Now you guys can read Wikipedia just as well as I can regurgitate it, probably better. So I'm just going to point out a couple of things on BGP that are worth noting here. It's a distance vector protocol. It measures the shortest AS path between two autonomous systems, which basically means we're doing path determination based on the number of organizations that our traffic is going to be passing through. There's two varieties of BGP. There's an external and an internal variety. The external variety is used to pass routing information between organizations. The internal variety is used to pass that information amongst internal BGP speakers within an organization. The one got you here is that generally you still need an interior protocol to discover your own internal network. Generally BGP doesn't handle that for you. So in this case we'll be using a less PF, but any sort of interior protocol works. So we're going to do BGP. What do we need? Well, the first thing we need is money, but fortunately we don't need a whole heck of a lot of it to get started, especially if you're a smaller organization. We're going to need numbers. We're going to need an AS number. We need IP addresses. We're going to need routers and we're going to need connectivity. And for those last two points, we're going to want more than one. If you're going to go to the time and trouble of deploying BGP, there's no point in deploying it on just a single router, single connection. You're wasting your time. Internet registries are the place to go to get number resources. If you're in the United States or Canada, you're going to be talking to Aaron. That's where my experience is. Your mileage may vary if you're from outside of that region, but these are the folks that you need to go talk to. The one thing that I will mention is that if you start down this process of getting numbering resources and you have questions speaking the context of Aaron, Aaron is your friend. They're not a gatekeeper. They're not the bad guy. Their mission is to help you get resources to do things on the Internet. And so they're generally willing to go well out of their way to help you out. Autonomous system numbers simply is a number that identifies your organization. There's two byte and four byte varieties. Two byte are the legacy variety. I don't even know if they're issuing two byte AS numbers anymore. Who are you with? Okay. They will by default issue you a four byte number. I might see specifically ask for a two byte number. I have a two byte number because we were in early enough. Thanks for the clarification. IPv6, you get addresses from your registrar. Your allocation is going to depend on your organization size. And the one thing that I'll point out here, well, two things I'll point out here. Minimum announcement size that's going to be accepted generally is going to be a slash 48. And as you start to work out your addressing scheme, I really recommend not allocating anything less than a slash 64. Even if you deploy a smaller sum that allocates a slash 64. IPv4, it's the legacy protocol on the internet. Everyone in this room knows about it. It's exhausted. Minimum announcement size is a slash 24. I think there's enough said there. As far as obtaining resources, you can. And I'm going to get to that. But I like that picture. So blocks can be had. The internet registries have waiting lists. There's a transfer market that you can take your chances on if you're so inclined. But there's another trick. At least for Aaron, there's the 410 rule, which refers to the section in their numbering policy. It's a slash 10 that's set aside for IPv6 transition. And the bottom line here, the big takeaway here is that if you already have an IPv6 allocation and you want to dual stack your application, you have justification to request a v4 block. I may add for if you are with Rive NCC, become a local internet registry of the year. It's something like 2,000 and 1,000 euros. Okay. That's good to know. Does Rive still have a pool of blocks? I assume. Okay. Good to know. Okay. Okay. That's good to know. Connectivity real quick. There's generally three methods of obtaining connectivity. You can either buy transit, which means you're paying somebody else to provide you a path to the larger internet. Internet exchange points are a really great way to exchange traffic with geologically local peers. There are also great source of information. And then there's peering relationships where you're exchanging routes with another single organization. Couple of connectivity gotchas real quick. If you're starting to deploy BGP and bring up transit and things like that, you should expect to be providing a letter of agency, excuse me, basically acknowledging that you are authorized to announce the blocks that you say that you want to announce. Expect that you're going to pay cross-connect fees if you're in a data center, co-location. And the last point there, you may want to at least make some effort to make sure that your fancy redundant fiber isn't running through the same conduit because that can cause problems too. So switching back to FreeBSD here, talk about what we need to turn our FreeBSD into a router. And in a nutshell, you're looking at it. Pretty much everything we need at a high level is done through RC.conf. What we're looking at here, we're defining our network interfaces. We're enabling routing. And we're enabling our routing daemon. And there's really not too much more than that. In reality, things like VLAN, carp, lag bundles are going to make this more complicated, but we all know that's going to happen anyway. Real quick on firewalling. Firewalling, generally, I try to separate the filtering and the routing functionality roles on different machines, but there are cases where it's useful to do filtering on your firewall. The one thing to be aware of is keeping state on connections. When if you move to a, if you're moving to a VGP model where you've got multiple routers and uplinks, it's entirely likely and probable that traffic's going to come in one direction and go out another direction. And if we're keeping state for that, you're going to have all sorts of hard to die nose problems and you'll lose hair and just don't do it. So moving on to our routing daemon. The routing daemon I'm using here is Bird. It's open source. It's pretty widely, widely used. A couple of highlights on it. There's two, two versions that are current. We're going to be talking about version 1.6. That supports most of the common protocols that you're going to run into, stool stacked. The configuration structure is, is simple and easy to get your head around. And it has a utility called Bird C, which is very useful for controlling your routing instance and then monitoring it as well. So as we start to look back at our, at the test network here, we're going to take a look at some of these different relationships that, that we're, we're creating with Bird. And so we're looking at a, a portion of the test network here. And we're going to be looking at it largely in the context of this AS-123 router 1A. So, so our first relationship here is our relationship to another AS. And that's going to be where we're using EVGP. Within the AS, we have multiple BGP speakers. And this is where we're going to be using IBGP to, to share and communicate that external routing data to all of the different border routers on the, on the network. Internal to the network we're using OSP-APFAs I mentioned earlier to discover all of the routes that are available internally. And then additionally, we're using a couple of static protocols largely to, as a means of injecting routes into BGP. So rather than, especially from the point of view of an end-user rather than, than blindly redistributing things into BGP, we want to be very sure we know what we're, we're injecting into the, the global internet. So with that in mind, we'll take a look at how these relationships actually manifest themselves on a running bird. And so at this point, I've been moving too quick and the network isn't totally up yet. But hopefully it is up enough here that we can at least take a look at a couple of, couple of things that are going on in bird. And these are a couple of the commands that I use most frequently just to get an idea of what's going on. So show protocols is going to give you a quick bird's eye view of all the different protocol instances that are running on your, on your bird and what the status of them are. If you want to dig into a particular protocol instance and see more details about it, we can do that. And this one fortunately has come up so we can talk about it a little bit. So this is our, this is our EBGP instance from this router out to its, its upstream neighbor. And we can see that the protocol is up. The interesting thing, the usually the first thing that I go to look at when I'm pulling protocol is to see how many, how many routes are being imported from this, from this pier, how many we're exporting, which is, which should correspond to just the static routes that we're injecting manually. And then there's a preferred counter here. This is how many routes that we're actually preferring this, this path for. Birdsea is also an interactive shell, if I can spell it correctly, which is actually probably even more useful because it has inline help. So we can do whatever we want from here, like tell bird to reload its configuration if we've made some configuration changes to it. But generally we don't have to do that because we're running everything through Ansible right now and Ansible is going to handle that for us. So moving along here, we didn't look at the bird configuration, which is why I put the slide in here so I won't forget about it. So this is the actual configuration that's driving what's going on this router. So looking at the left-hand side here, a lot of what we see at the very top is boilerplate definition. I'm not going to spend a whole lot of time on it here. The more interesting bits, or as we get down into OSPF, we can see that we've defined interfaces for each of the OSPF, each of the interfaces we want to run OSPF on. Some of them are running normal OSPF, some of them are just stub interfaces, in particular our uplink interfaces we don't want to run OSPF on. You can see a couple of static instances defined here. For static BGP, portable BGP, this is where we're defining the routes we want to inject into BGP. And the distinction between the two is more applicable in a multi-site configuration. I'm going to get into that in a little bit. And then at the bottom here, we see our definitions for our EBGP and IBGP sessions. And they look pretty bare here. We're really just defining what our neighbor is. And the reason that they look so bare is because we're templating them. So these configurations are templated by an included file, which I forgot to point out at the top of the file, where we plug all of our common configuration that's going to be the same for all of our EBGP instances on this router. We just plug it into a single template and then we can reuse it and reuse it and reuse it. And there's not a whole lot more that we're defining here. We're defining our AS number. We are defining a little bit of a hack here to allow our own AS number to appear in the AS path. This is not generally allowed by BGP to prevent loops, but it is useful if you're running multi-site configuration. You don't have a point-to-point link. If we want to route between the two sites, we have to be able to allow the AS number. So looking at the bottom of this file here, I'm going to cut off here, but the IBGP template is largely the same as EBGP with just a couple of changes. So now that we've looked at the BIRD configuration, we'll look at the actual template that that configuration file is generated from. And we're doing that with Ansible. Ansible uses the Gingya templating language for all of its templating plays. It uses it for the playbooks and configuration as well. And it's very flexible. It draws information in from inventory, which is why we're finding it all there. And it's basically what allows me to generate 15 routers in this demonstration from what amounts to about one set of configurations. Oops, jumped ahead. So we will take a look at these templates. And so first we'll look back at the inventory a little bit here. And this is the inventory, again, for router 1A. We've got, again, some boilerplate at the top that's less interesting. But as we get down here into the network configuration, again, this is all arbitrary. You can structure it in whatever way makes sense to you. But we're defining the addressing for each of our interfaces. We're defining what interface name it should be associated with. In the case of some interfaces, we're defining carp addresses if we want to do high availability there. We're defining what the OSPF relationship is. And then as we move down into the bird section of this dictionary, we're starting to see all of our routing-related configurations, like what our AS number is. We've got a section here that's defining what prefixes we're going to be putting into the static and portable protocols. And if you look here, we've got additional variables that are being pulled in using the double braces, which basically just means that I'm defining what these lists are elsewhere in the configuration, actually, in groupfars files, because they're common across AS 123. And then I'm pulling them in here. At the bottom of the file, we see the definitions for our BGP sessions. And again, because everything's very templated, there's really not a whole lot that we need to define here. Looking over here on the right side, we're actually looking at the bird.cont file that's before it's templated by Ansible. So again, here we see double braces, which indicate that we're doing a variable substitution, bringing something in from inventory and plugging it into the file. The OSPF section here, we're starting to see some logic being used. We're starting to see a for loop where it's looping through all of the interfaces that are defined in inventory, looking to see if it should be configured for OSPF. Again, we're doing looping to define the static protocols, basically taking everything that's in a list of, a list in inventory and plugging it into the static protocol. And then at the bottom here, we have another couple of sections, again, to define our BGP instances. Find my slideshow and check to see if indeed our playbook finished and it did. And so at this point, our network should be fully routable, fully built and routing. And now that that's done, let's break it. So what I'm going to do is run a, create a couple of failures within the network, and we can watch and see how BGP responds to those. I'm also going to demonstrate a couple of useful management functions that are pretty easily implemented using Ansible. So the first thing that we're going to do is talk about link failures because they happen. So looking again here, a lot of this is going to be in the context of these bottom two clouds here, which are both AS123. They're just two different sites. Think of it as a primary site, DR site, backup site, what have you. So in our first case, we're going to look at what happens to traffic flowing between the two sites if we happen to just fail a link. So under normal operations, traffic going from site A over to site B is going to take the shortest path through AS100, AS300, and then back into the other site. Once this link goes down, obviously we want BGP to figure it out and just do the right thing and use its other available uplink through AS200, 100, 300, and life is good. So if we flip over to my other consoles that are running, we've got a couple of things going on here. We've got a connection into router 1A, and we're just watching the debug output coming from that daemon. We have a trace route running from the firewall that's at site A to the firewall that's at site B. So nothing particularly fancy there. And this is a case where we can use Ansible. We can use the Ansible command to run a one-off command against anything that's in the inventory. So in this case, we're going to tell the AS100 router that we want it to shut down its VTNET 2 link. So it's going to connect and do that in success, and we start to see packet loss on our trace route. And then within a couple seconds here, BGP is going to pick up on the fact that it can't talk to its neighbor anymore, readjust its routing table. And if we restart the trace route, we see that we're still communicating, we're just communicating over a slightly different path. The next thing we'll talk about here is site failover. And this is useful if you're running production services from one site, and you want to pick those services up and start running them from your backup site without having to monkey with DNS and other, resort to other tricks like that. So the process to do this is actually really simple. And if we look at the Ansible configuration here, it's really just driven by the, it's just a reuse of the roles and playbooks that we've already defined for bootstrapping the routers. If we look over an inventory, we're looking at a couple of group file files for AS123 on the left-hand side here. And the top pane is just for the site A routers. And we're defining that our location is A. Down below for the entire AS, we're just finding that the active location is also A. So really the process of doing the failover is just running a playbook and telling Ansible, passing in a variable that says, you should reformat everything so that site B becomes active. And then reload the Ansible, or reload the bird configs and will be active at the new site in no time flat, or something approximately that. In reality, the failover is actually pretty quick on the order of a couple of seconds. So looking back at our windows here, starting from what is actually our Ansible host, I'm going to start a trace route into www.as123, which is not going to work for me. But that will. So we've got a trace route running, and so what the network looks like here. No, I knew I had another slide here. So looking again at our network, we have our Ansible host way up here, which is currently routing into our portable address space, which didn't make it on the diagram, but it's just another slash 24 and another slash 48 that we're announcing for the next few days. So we're going to start a trace route here, and then we're going to do a trace route, which didn't make it on the diagram, but it's just another slash 24 and another slash 48 that we're announcing from whatever site is active. So it's routing into site A to those portable addresses. So what we want to do is pick those addresses up, and rather than bring them in here, we want to bring them in over here to site B. And then we have a pre-cooked playbook that here again, all we need to do is fire it up. And so looking at this playbook invocation, if we open up the playbook, there's really nothing going on there other than... Oh, I was going to talk about that, and then I didn't. This is the extent of the playbook that's going to handle this failover. And the most interesting bits are really these last three tasks where we... We're on our last four tasks here, where we template our bird configuration files by calling the bird configuration rule. We actually go and update the Ansible inventory once we've made the change. So the change is persistent within this system, and then we reload the bird configuration. And then we make noise warning you that you've done something that you should probably be aware of. So looking at the playbook invocation, we've got our playbook name, and we're specifying a couple of additional variables on the command line. We're telling it that its new active location should be B, and that once it's done doing what it wants to do, that it should reconfigure the bird instances and activate the changes. So as that runs here, there is one sanity check in the playbook where it pauses, and has you confirm that yes, indeed, you're sure that you want to move your services to the other site, and yes, for sure. So it's going to go and configure its bird files, updates the inventory, reloads the bird configuration. We see the debug start to update the paving, and then if we look back at this trace route, rather than going into site A, we now see that it's starting to go into site B on its way into that portable service. So next thing we'll talk about in terms of management is what happens if we want to deprioritize a router. If we want to bring a router down for maintenance, or we're having trouble with our upstream provider, we just want to keep as much traffic off of that router as possible to reduce the disruption. Deprioritizing it is the way to go. So coming back to our trusty network diagram, we already have a link failure, and currently our traffic between sites is coming in through AS300. But what happens if we want to bring this router down, perform maintenance on it well, we're failed over to our DR site, which may be getting a little bit contrived, but it could happen. So the way that we do that again is by leveraging the Ansible roles that we already have defined, and here again just passing them a couple of different variables so that they manipulate the bird configuration files a little bit differently. And I have an example of that right here. What we're looking at here is the bird BGP include file that's templated out by Ansible, and a couple of parts that are most interesting here is based on a variable definition, we're changing the preference of the BGP protocols for this router down from the default 100 down to 99 so that any routes that come in over this protocol instance are going to be skipped in the presence of a better route. Down below here, we've got an export filter for routes that are being sent out of this BGP instance, and we are going to do BGP path propending, which basically amounts to inserting our own AS a number of additional times into the AS path before we send it to our upstream router, basically makes the path look artificially long and is going to cause the rest of the internet to go, well, I've got a better path over there, I'm not going to send any traffic to this router. So here again, it's just a case of running another playbook where once again we have a couple of variables defined, cute names aside, we're telling Ansible that we indeed want to, that we indeed want to deprioritize the router, we're telling it which router that we want to deprioritize, which in this case is 2B, and we're telling it again that we want it to configure, the bird changes automatically, and the configure bird variable is just another sanity check that I like to use for cases where I may want to stage configurations but not necessarily activate them right away, but in this case we do. So if we look back at our trace route that's running between sites, we can see that here again it's running through AS 200, 100, 300, and back into our other site. Once we run this playbook, this route is going to become a heck of a lot less attractive. So what BGP is actually going to do is it's going to say, well, this is a really long route, but I've got this shorter route by going up and around through AS 50 and back into our other uplink. So with that, we'll see if it does what it's supposed to. Here again, it's templating bird configurations, and there's no changes being made to the main bird configuration cell happening in the included files. We reconfigure bird, we start to see changes immediately on our other router, and our route changes, and if we reset that trace route we see that we have, we still have connectivity just over a much longer path. Oh, forgot all of my other things that I already talked about. So the very last thing that we're going to talk about is what happens if we cause some more mischief because failing a link wasn't fun enough, what happens if we blow up a router? So we have our link down, we have our deprioritized router, but what's going to happen if we take out a router somewhere in AS 50? And the answer here is that even though this router is deprioritized, it's still a route into our network that we'll use if we don't have any other option. So looking again, we still have our trace route running, still have our log output. If we flip over to our beehive host, we can tell beehive to just power off the host, and we immediately start seeing packet loss on our path, and once again, once BGP figures out that there's a problem, starts to reconfigure itself, then the network comes back to life. And we can see that once again, it is taking that, what is actually a shorter path through 200, 100, and 300. And life goes on, your pager goes off, you say everything's good and you go back to bed. And with that, that is what I have. So I'm interested for any questions that you may have. What's the latency look like for using Ansible to cut over change configurations to switch which one's the active? For us, with Puppet, Puppet's a bit too slow to do switching between A and B. For things like our car IPs, if you want to do the cutover, you have a bunch of shell scripts that we can help promote these different changes very rapidly. Right, between sites or between routers? Well, it's between web servers and site-to-site. Ansible, I don't know that I would use it for failing over servers or clusters, things like that, because it is slow. Carp is a lot faster. So actually, for cases, I didn't really talk about it in the slide show anywhere here. But between our two border routers, we've got our routers, we've got a firewall or a cluster firewall sitting behind it. I'm actually using Carp on the border routers so that firewall can just set a default gateway. It doesn't actually have to run VGP. And so then if a router fails within a second, we're back up and running and life is good. Anybody else? Are you using Ansible on your routers? Yes. Did you also try to move this to actually how do I routers or just KBSD devices? We're in production. We're running free BSD physical hardware. Ansible does have a lot. They've been putting a lot of effort into building out modules for controlling various network hardware. So at this point, there's support for Cisco, Juniper. This tool in the part of it is pretty unusable. So I tried to use it a couple of times and it was a bit of a mess. Back to what? I'm not familiar with that one. But I'm not using Ansible to control any vendor hardware. I haven't had a use case. My free BSD boxes work great for BGP and my switching doesn't change enough for me to have a good reason to integrate Ansible with it. Yeah, a lot of libraries know how to speak Cisco, Router, whatever. So it creates a lot of fun to go and push the cloud whatever means of having all this. Yeah, I've tried a lot to look at the affections that work a lot on our vendors, but I think that is a bit of a factor in the results. Yeah, if I can pull it up here real quick. Ansible has a huge library of modules that are provided by core. If you look in the fault from yesterday at 10 from Zabino, he was using the SRO to talk to Cisco all the time. It's gotten a lot better just in the last few months. It has. Yeah, the answer team has increased their networking developers and network module developers. It has increased internal to the previous one. Yeah, I think the acquisition by Red Hat has really put some pressure under them to support a lot of enterprise products, which is a good thing. So VMware support, AWS support has gotten huge. So there's just a ton of flexibility that you don't have to reinvent. And additionally, there's Ansible Galaxy, which is a repository of pre-built roles and other reusable code that other people have submitted that's useful as well. If you can read it. If you can read it. Exactly. I think I'm using exactly one module from Galaxy. Did you have a question? Ansible has a lot of built-in protection against them. There have been a number of vulnerabilities that have been figured out surrounding that and escaping, quoting, things like that. And Ansible is actually pretty good about warning you when you're doing something you shouldn't be such as trusting client input that you shouldn't be. Yeah, it was an incidental client problem actually connected to some host and just by injecting some commands it was able to compromise the controller that was executing the commands. I think that's an area of this case. Yeah. It's only good to just wait. Yeah, that's always a possibility and a concern. Definitely. Yeah. If you have options for role-based access. Right. Right. Anybody else? Yes, it should be. Protect your controller, I think is the bottom line. Well, if nobody else has anything, I'd like to thank you for coming.
is Plumbing the Internet and we're going to be talking about BGP automation and BSD obviously for a high availability disaster recovery probably really foolish thing to build the quick, couple of things about Ansible that are and running without, you know, going to great some of the configuration of Ansible here. And put together the way that I want to use it in my and I'll do what I can to make them bigger, Beehive. It lets you do all of your network building up here. It's really essentially host. And from that, we've got, on the host, we external variety is used to pass routing an interior protocol to discover your own So we're going to do BGP. What do we need? Well, need to go talk to. The one thing that I will getting numbering resources and you have resources to do things on the Internet. And so point out here, well, two things I'll point out talk about what we need to turn our FreeBSD into on to our routing daemon. The routing daemon I'm using here is Bird. It's open source. It's pretty configuration structure is, is simple and easy to into the, the global internet. So with that in your bird and what the status of them are. If you multi-site configuration. You don't have a couple of changes. So now that we've looked at the network, and we can watch and see how BGP responds to those. I'm also going to demonstrate that it can't talk to its neighbor anymore, readjust its routing table. And if we restart the over an inventory, we're looking at a couple of really the process of doing the failover is just looking again at our network, we have our Ansible talk about that, and then I didn't. This is the what happens if we want to deprioritize a router. see if it does what it's supposed to. Here again, we see that we have, we still have connectivity about is what happens if we cause some more router, but what's going to happen if we take out you're doing something you shouldn't be such as