Tunnels as a Connectivity and Segregation Solution for Virtualized Networks
Muller, Assaf
Join me for an architectural, developer oriented overview of (GRE and VXLAN) tunnels in OpenStack Networking. In the virtualization environment virtual machines are hosted on hypervisors. These VMs then obtain network connectivity via software switches run in the same hypervisors. Data centers that provide infrastructure as a service have (hopefully) multiple customers (Or 'tenants'). As you can imagine we don't want tenants' VMs interacting with one another. VLANs are a natural approach to achieve tenant segregation. However, how do we maintain scalability with a growing number of hypervisors and VMs, when the administrator has to constantly configure the hardware switches manually? Is there another way? We all use VPNs to connect to our office resources remotely, or to connect two office sites into one seamless network. VPNs are essentially encrypted tunnels, but what are tunnels? Tunnels allow us to wrap packets inside more packets. In our context - VM traffic in exterior IP packets. That way, to the intermediate networking hardware, it looks like traffic between the hypervisors. Since the hypervisors should already be able to talk to each other, this makes VM connectivity a breeze! Let's explore how tunnels are used in the cloud as a means to achieve an overlay network. What is an overlay network? How does traffic flow between virtual machines on the same hypervisor, and on different hypervisors? What are the similarities between a layer 2 learning switch and tunnel logic in OpenStack? How does Open vSwitch fit in? Is there a cost to using tunnels? This talk will be useful to developers interested in learning about new networking concepts - Minimal background knowledge will be assumed.
 All right, so we're going to be talking about, it's a very networking oriented talk, the Vettapur, the Vettapur and networking oriented talk, we're going to be talking briefly about VLANs in a cloud solution kind of approach. We're specifically going to be talking about OpenStack and OpenStack Neutron, a bit about VLANs, just an introduction, how do they, how do we use VLANs to kind of hook up VMs and maybe even segregate to different networks and stuff like that and the shortcomings of VLANs and how do we use tunnels? So how do we use tunnels to achieve the, solve the same problem that VLANs currently solve and maybe even more? So I'll be, so I'm a staff builder, I'm an engineer, a software engineer at Red Hat, I'm working on networking oriented stuff in OpenStack and in Overt, which you've already seen multiple talks about and I'll be completely honest, I've probably never talked to more than 40 people at the same time and I don't know if anyone's ever told you this but you're, you're a lot of people, there's just a lot of you. So we're going to do just a quick warm up. This is actually for my mother. So, so we're going to do a one, two, three freedom sort of deal, if you would. So it's one, two, three. We can do better. We can do slightly better. So one, two, three. Thank you. That was very good. All right. So I'm kind of used to moving a lot but I was told not to do that so I'll try to reframe. So this is kind of a normal proof of concept or kind of a small scale networking deployment. So as you can see we have two different physical networks, the orange-ish stuff. That's for the management traffic, that's for like, you know, just telling a compute node that it should bring up a new VM, stuff like that. That's for management traffic and then we've got another physical network just for VM traffic, just for VMs talking to each other and talking to outside of the cloud. So you could create two different physical networks. Of course you could also just use two different VLANs. But the point of this slide is to just mention that we are only going to be talking about the VM data network. So this is just about how VMs talk to each other and to the internet. So just the teal network in this case. So how did we do that a long, long time ago, you know, yesterday? So we use VLANs. VLANs are a very kind of well-known concept. They come from the physical world. So it's kind of natural for networking people to just yell VLAN at kind of every problem. So VLANs have access ports and they have trunk ports. Right? So that's a switch configuration. You can enter the physical switch and you can configure each port to be either an access port or a trunk port. So a trunk port is untagged. It's basically telling, okay, this specific port is in VLAN 100 or the red VLAN and the traffic coming out of that access port will be completely untagged. So the VM or the host that's connected to that access port doesn't know that it's in a VLAN, but the switch does. So if the switch has two access ports and they're both in different VLANs, then the switch can do filtering. So it can either forward traffic from one port to the other or not. So that's access ports and trunk ports are kind of different. You, again, you enter the switch and you configure a specific port to be a trunk port and it actually carries tagged VLAN traffic. So the actual VLAN identification number, which is just a 10-bit number. So in this case, it would be like the color, but it would be like VLAN 100 or 200. That's actually carried with the message. So the message is carrying, what VLAN am I coming from? And that's configured on the trunk port. So a trunk port has a range of VLANs that are allowed on that trunk port. So like the dashed, the dashed cables, so they're carrying both VLAN 100 and VLAN 200. So if they get a message from VLAN 500, then, you know, that traffic will be just filtered and dropped. So that's kind of the point of a trunk port. So we could see that we have four compute nodes, right? And that bottom left one, that's kind of a blow-up or a zoom-in. So that's a physical switch, but that's actually a virtual switch, right? So we, in OpenStack, we use the VRInt, which was talked about in the last lecture. That's called the integration bridge. So that's a virtual switch like a Linux bridge or an OpenV switch for the open source plugins, right? Because Neutron, the whole OpenStack Neutron thing, it's pluggable. So you could have different implementations. You could have OpenStack, sorry, open source implementations for the API, and you can have proprietary closed source implementations for the same API, right? So for the open source implementations, you have virtual switch inside of the compute node. And so the bottom ports are access ports, right? So VMs are connected to access ports. And actually, the virtual switch is connected to the physical switch via a trunk port, right? So this works. This is fine. It's kind of awesome. But there's limitations, right? So just kind of hear me out. If this virtual machine, that's in the blue network, that's in the blue VLAN, right, VLAN 200. So if it sends a broadcast message, should that upper right compute node get that broadcast? I mean, we as human beings, we can tell that this specific compute node isn't hosting any red VMs, right? You don't have any red VMs in the red network on that compute node. So this broadcast that's originating from a computer on the blue network, right, that message shouldn't even reach that compute node, right? There's no point. So the way to accomplish that is would be to interacting with the physical network. And as a software person, I'm not really into that. I kind of like to work with virtual and software stuff. So configuring the virtual switch would be, the physical switch would be kind of a bummer. And I would have to connect to each physical switch, so I could have 100 of these in my cloud. And I would then have to enter each trunk port and manually configure each trunk port to carry or to trunk specific VLANs, right? And that would accomplish what we wanted to do. So this broadcast reaches the physical switch, the physical switch sees that this trunk port is only carrying the red VLAN, so it wouldn't forward the broadcast message. So mission accomplished, but obviously that's very manual, that's very tedious, it doesn't scale. It doesn't scale even for small deployments, actually. So what most system admins do, they just enter each switch. So for each trunk port, they just enter the entire range, right? Because that's the only thing that's feasible. So that means that basically every broadcast reaches everywhere, all of the compute nodes, even compute nodes that aren't hosting VMs in the specific network, right? Does that make sense? So that's kind of a problem. Also, there's kind of a philosophical issue of taking the VLANs and just, that's kind of a physical world concept, but our compute nodes are aware of the same VLANs, so we're kind of taking the VLANs and just bringing them into the virtual world, which is not ideal. So that's VLANs, right? So we can do better. So tunnels are basically a well-known and familiar concept. You know them as just VPNs, right? We all use VPNs, we connect from our home if we want to work from home and stuff like that. So tunnels are basically the, it's like a VPN, but it's unencrypted, right? So, or a better way of putting it would be that a VPN is an encrypted tunnel. So I don't care about encryption, I don't know anything about that. Also, who cares about security, right? So I wasn't kidding. So we're only going to be talking about tunnels, but only just the unencrypted stuff, right? So I could configure a tunnel and then just hack on encryption on top of that. So that's kind of uninteresting. And also, there's different types of tunnels. There are GRE tunnels and VX tunnels, and that doesn't matter. That's just the headers and stuff like that. We're just going to be talking about concepts. So there's two use cases, right? Just two that I could fit in this slide. So there's, let's say that I have a large corporation, you know, and I have a bunch of different sites. One is in Brussels and one is in Namur. So really, the kind of experience that I want to achieve is that computers hosted in different sites kind of feel like they're in the same site, right? Because, for example, say Brussels is the kind of the headquarters and it has a bunch of company resources, like company websites closed for employees only and a bunch of different company resources. So I'd like the people from other sites to be able to reach those, right? And I'm also using NUT. I'm using private IP addresses. So normally, without a tunnel, I wouldn't be able to ping from a Brussels computer to a Namur computer, right? Because I'm using NUT. It's private IP addresses can't reach those computers. Like in our own home network, we have private addresses. People theoretically can't ping or can't reach our own computers behind the NUT. So also, I want to be able, like just a personal example, I have a development laptop in my own office and I want to be able to SSH or reach hypervisors or computer resources that I'm using and that are in different sites. So we can use a tunnel. Now, mind you, a tunnel is a manually configured thing which we actually configure on the physical routers, right? So we basically, we SSH our talent into the left router and we just tell it, listen, if you want to reach the 172.16 network, then you take all of those, all of that traffic and you route it to a tunnel device. And the tunnel device is also a second configuration step, sorry, which I also manually configure, right? So I have this kind of tunnel interface and when I configure that tunnel interface, I tell it, what's the source IP? I want to this donation IP for the tunnel. And then I just, I have to configure the routing, just the static routing that I talked about earlier, right? So the left router knows that if it wants to reach this network, it has to leave through the tunnel device and the other way around, right? So that's kind of the manual configuration that you would do and you also have very similar, if you want to work from home, right? So it's kind of like, it's exactly like a VPN. So you have, you create the tunnel from your own computer to the router or the VPN server that you're connecting to and it's actually exactly the same thing. Right, so basically tunnels aren't magic, they're just a silly encapsulation trick, right? If you remember the OSI, 7 layer models, you've probably heard of this before, the DCP IP models, there's the different layers and different headers and encapsulation, right? So what are we doing here? We're just taking the data, whatever it is that you want to send. So this would be the IP packet representing or wrapping that data, right? So the source IP is some machine in Brussels and the destination IP is in some machine in the other site. So we take that packet and we wrap it in our tunnel header. That would be GRE in this case, right? So the only important header in the GRE header or the only important field, sorry, is what's the next protocol? So the next protocol in this case would also be IP, right? Because each header has data and the data is just a bunch of random bits. So the question is how does the computer know how to interpret that data? So you tell it, right? So each header has kind of what's the data inside of me, what's the next header? So the next header is also IP because in this case we're just taking IP packets and just wrapping them up in more IP packets. So the beauty here is that the outer IP packet, the source IP is the source of the tunnel, the destination IP is the destination of the tunnel, right? So the ISPs of the world, right, or the internet are just seeing a packet that's being routed from 1111 to 2222. So the only thing that we require from our physical network is that these two routers can actually ping each other. That's the only thing that we kind of require. And the ISPs or the routers on the internet, they don't know what's this Brussels network or what's that Namur network, right? They just don't know any of that. They just need to be able to route from one router to the next. So that's kind of what we do with tunnels and that's kind of the physical or the real world usage. So in OpenStack Neutron and in the cloud or in the data center, virtualization management, right, that whole context. So we kind of do it differently. We basically, these are compute nodes or hypervisors. They're hosting the VMs. So we just create a full batch of tunnels, right? We just hook up each compute node to every other compute node. We just create these tunnels. And the first thing that kind of should pop into your mind is that sounds expensive, right? Because, but you just kind of need to remember that these tunnels are, the overhead is basically just in the database of the compute node and we, that's, that's fine. So we just create this full mesh of tunnels between all of the compute nodes. And then it's kind of exactly like the previous example, right? So if a VM wants to talk to another VM, that's another compute node. So we just do the encapsulation trick and we just wrap up the packets exactly like in the previous example. And if two computers want to talk in the same hypervisor, then they just talk to each other directly via the same virtual switch, right? So both of the machines would be connected to the same virtual switch, BRint or the integration bridge, like an open V switch bridge or switch, and they would just talk to each other directly. Right. So what if we want to achieve segregation, right? Like in, in this previous example, you can see that I still have two networks. I have the red network and the blue network. So, all right. So about segregation or kind of splitting, splitting up networking. So two VMs on the same hypervisor, they're still using VLANs, which is kind of confusing. Okay. So we're, we're using tunneling for the actual, for forwarding traffic between the two different, between two different compute nodes. But the BRint on each compute node or the virtual switch is still using VLAN access ports to segregate between different networks, right? So VMs connected to the same virtual switch, they're still using VLAN access ports. So each network gets a different VLAN. So we're kind of mixing and matching. But more interesting is what, what's, what's going on with traffic on different compute nodes, right? We want to talk about the tunnels. So with VLANs, we use trunk ports, right? We tagged the frames with the VLAN identification number. So it's VLAN 100, VLAN 200. And that information isn't lost. It's being trunked or forwarded across the entire network. So the other end knows what VLAN was this message originated from. So I can either filter or not. So we're basically doing the exact same thing. All right? So we're using the GRE header or the VXLad header, basically the tunneling header. And we're using a field there which is called the tunnel ID. So the tunnel ID, it's basically the exact same thing, right? We take a network, we tag it with a specific number, and that number is placed inside the tunnel header. So it's, so that way we can color or we can kind of tag packets with the ID of that network, right? So that's how we do segregation across compute nodes. So we, we've kind of got to talk about how do we, what's the, what's the logic, right? How do we forward unicast traffic? How do we forward broadcast traffic? So this is kind of a short review. So physical, just kind of classical switches, could be physical switches, it could be virtual switches. They're called just layer two learning switches, right? So how do they work? They have this kind of table which is a binding or a map between the port number, right? So if I, if the ports are numbered 1 to 24, so each port is bound to a set of MAC addresses, right? So basically the switch is kind of sniffing traffic as it goes through it. It looks at the source MAC address, right? Where is this message coming from? And what, what port is it coming from? So if it, so if MAC A is the source MAC and it came from port one, then now I know that whenever someone wants to talk to MAC A, I can just forward it to port one, right? And that's where I will find MAC A. So that's kind of what physical, that's what kind of learning to layer two learning switches do, virtual or physical. So we're doing something that's very similar. So we have two virtual switches on each compute node. We have the integration bridge. So all the VMs are connected to that bridge and that bridge is connected to the tunneling bridge. So the tunneling bridge, we just control it and we can kind of create this sort of table, right? So each message as, each packet as it comes in from any other compute node into my own compute node, then I check what tunnel did it come in, come in on, right? Was it the first tunnel, the second tunnel? Because I'm connected, I have a tunnel to each other compute node. So I check what compute node this message came in on, right? So that's the peer, that's the IP part, the peer. And we just said that the messages are tagged, right? So I take the tunnel ID, which is actually the network ID, and I take those two, just numbers, and I bound them to the source MAC address. So it's basically the exact same concept as a learning switch. We just do that for different headers, right, for the tunnel and the tunnel ID and the source MAC address. So that way we know how to forward information through a specific tunnel, right? We need to know for a VM, basically what compute node should we send our messages to, right? So that's how we kind of do it. Let's try that again. Okay. Right. So we talked about unicast traffic, we talked about learning, right? We talked about how do we know which compute node is hosting the VM that I want to talk to. So we talked about learning, and now let's talk about broadcast traffic, because broadcast traffic is kind of different. Maybe we want to optimize some stuff there. So generally speaking, the kind of the first approach that you would think about would be for any broadcast traffic or multicast traffic as well, for that matter, that's leaving a VM on the red network, right? It would reach the tunneling bridge on that bottom right compute node. So we would just basically send it out through every tunnel, right? That's kind of the basic approach. And that works. That definitely works. But we want to be smarter, right? Because we can. And because we kind of, you know, as software engineers, we just, we like these optimizations. It's something that's nice and we kind of can reason about. So we can do two things. So we can just send less broadcasts. And we can, and about the broadcasts that are left, you know, that we didn't optimize away, we can kind of make them smarter. We can make them reach only the compute nodes that they should reach. So about minimizing broadcasts, turns out a lot of broadcasts are just about ARP requests, right? If you have, and this is in the physical and the virtual world as well, if you have two computers and they really want to talk to each other, right? So the first computer knows the IP address of the second computer, but it doesn't know its MAC address because it never talked to it. So it just asks. So it asks in a broadcast fashion. So it said in the broadcast, what is the MAC of IP address so and so? And that other machine answers it. So a lot of broadcast traffic is just because of ARP. So we go, okay. So there's a neutron controller. And this is without a dedicated SDN controller, right? We, in the previous session we talked about open daylight, SDN, that sort of stuff. So just in neutron, in OpenStack Neutron, we know each VM, what is its IP address, we know what is its MAC address, and we know where is it scheduled, right? So what compute node is hosting the VM? So that's based, so we're basically capable of filling ARP tables, right? We can, we know for each IP address, what is its MAC address, and we can send that information to all of the compute nodes. So all of the compute nodes actually have these full ARP tables, right, of all of the VMs. So that means that when this red VM is trying to ping or somewhere communicate with the other red VM, so the local compute node actually received a full ARP table from the neutron controller, and it knows, okay, so this VM, I know what's its IP address, I know what is its MAC address, and I'll just answer it locally, right? There's no reason to take the ARP broadcast and forward it to all of the tunnels. There's no reason to do that. So this compute node will actually answer the local ARP request. So that's basically minimizing broadcasts, right? Just a lot, just less broadcasts, that's good. The other thing that we can do about the remaining broadcasts, which there's still a lot of, we can just make them smarter. So generally speaking, again, going back to that centralized neutron controller, which knows all of the VMs, where are they, what are their IP addresses, what are their MAC addresses? So whenever this VM sends out a broadcast message, this compute node now knows, okay, I probably shouldn't forward it out that tunnel, right? Because someone, the controller, someone told me that there's no red VMs here. So that's the other thing that we can do, and there's kind of small asterisks. This barely works. It only works with the ML2 plug-in with a specific mechanism driver from specific versions, right? But that's in the small details, small font. Right. So we're going to do a very shallow deep dive into kind of the flow tables or the kind of the logic in the tunneling bridge, right? So I'm going to be talking about the open V-switch specifically, because that's what I know about. So I'm only going to be talking about something that I know about. So open V-switch, each bridge or each instance of an open virtual switch can operate in two modes, right? Two different modes. The first mode is just normal mode, which is just a learning switch, just a regular switch. Like we talked about earlier, just learning MAC addresses and all of that stuff. The other mode is called flow mode, which basically means that I can manually control the logic of that virtual switch. I can put in rules which will dictate how that switch works. And it's exclusive. It's either or. You can be either in normal mode or in flow mode. So about these flows, so you can create, you can put in new flows into a virtual switch in kind of two different approaches. You could do it locally via the command line, right? Via this open V-switch, open flow control, and you can just create new flows and manually put them in the virtual switch. You could, of course, also send SSH commands, so to do that locally. And you could also use an SDN protocol, like OpenFlow. So there's, right, so the OpenFlow protocol, basically you can have this kind of centralized node which can send these messages, OpenFlow messages to all of the virtual switches. And the virtual switches speak the same protocol, so they can configure themselves, right? They can add flows to their own tables. So in Neutron, right now what we do is that we have this software, the Neutron Open V-switch agent. So the agent is sitting locally on every compute node, and it can receive management traffic, right? Like bring up a new VM, bring up a new virtual network interface card. So whenever this piece of software receives management traffic, it can basically, what it does is that it's, it administrates local OpenFlow control messages. So it configures itself via management traffic that's being received from the Neutron controller. Okay. So what are these flows, right? We're basically taking rules or flows, and we're kind of manually configuring each virtual switch. So a flow is basically comprised of two parts. It's a match part, and it's an action part. So a match is basically, should this, right, so I have a bunch of different flows, a bunch of different rules. So should the first flow match the incoming packet, or should the second, or should the third? So I can match on a bunch of, of all, basically all the headers. The layer two headers, the layer three headers, right? So the ports, and the IP addresses, and the MAC addresses, and all that sort of stuff. So, okay, so basically a specific flow matched or caught a packet. So, all right, so it caught a packet, and now what should this flow actually do? So what is the action part? So the action part, it can change headers. So I can do, like, not. So I could change the source IP address. I can change the destination IP address. I can change ports. I could do all sorts of stuff. I can take the message and forward it to a specific port, to all ports, or to a group of ports. I can, obviously, do broadcasts. I can drop, right? I can filter messages, so I could basically implement a firewall. I can learn new flows on the fly, right? So I introduce the flow that matches packets, and according to those packets, it can actually generate new flows and write those new flows back into the table. And we'll talk about that in a minute. And we can resubmit to another table. So what's a table? So, as it turns out, a table is basically just a group of flows. And we can have a bunch of different tables that all belong to the same virtual switch. It's kind of just a convenient way to manage lots of flows. So all of the messages enter the first table, which is, of course, you know, table zero. We always count from zero because we're kind of idiots. So these messages can be resubmitted from table zero to just any other table. And for each table, right, a group of flows. So the flows are processed according to their priority, which was configured on each flow. And if, for example, a message came in on table zero, right, so the first flow doesn't match, the second flow doesn't match, and the final flow doesn't match either, then what happens? So that message is either dropped, or if there's an SDN controller configured, which with just like regular neutron, there isn't, so the message is dropped. That's kind of the default. Right, so kind of logically speaking, or what's the flow table, how do we use these different tables, or kind of how does the logic work? So traffic that's coming in from a VM on the local compute node, right, just a VM connected to the integration bridge, all VMs are connected to that bridge. So that enters the tunneling bridge, which is represented here. So it has a bunch of different tables. So table zero, as we said, that's the first table. So we basically say, oh, okay, this is traffic, traffic that's coming in from the integration bridge. So we'll just resubmit it to table one. Table one classifies unicast traffic and non-unicast, which is basically multicast and broadcast. So unicast traffic reaches table 20. Table 20 has a bunch of pre-learned flows. This is the unicast table. So this table should basically tell me, should I forward this message to table two, the first tunnel, the second tunnel, the third tunnel, right, what compute node should I be speaking to? So I either know the destination MAC address, so I just forward it to a specific tunnel, or I don't know, right? This is an unknown MAC address just because that MAC address hasn't spoken to me yet. So I forward it to table 21, and table 21 is multicast, broadcast, and unknown destination MAC addresses. So I basically forward it to all tunnel devices, right? Why the air quotes? Because we talked earlier about the broadcast optimization, right? So if that's enabled, then I don't need to send the message to all of the tunnels, just to some of the tunnels, right? So what about the other way around? If a message is coming in from a tunnel and not from a local VM, so it's coming in on table 20, it's traffic that's coming in from a tunnel and not the integration bridge, so it's going to table two. Table two is basically taking the tunnel headers, just throwing that away, adding the VLAN tag, which is kind of weird, and then forwarding that to table 10. Table 10 looks at the source MAC address and does the learning part, right, which we talked about previously. So it looks at the source MAC address, what tunnel that this came in on, both the tunnel ID and the actual tunnel itself, and it populates table 20, right, which was this one. That's the unicast table, right? And then finally, it forwards it to the integration bridge. So again, we drop the tunnel headers, all of the GRE or the VXLAN stuff, and we convert the tunnel ID to the VLAN ID, right? Because as we said, the actual, the local BR int or the virtual switch is still using VLANs to mark specific networks, right? So that's kind of the deal there. And just kind of the more information part. So we'll start at the end. The shameless plug. Basically, I have a blog which has all this entire thing just verbally written. So if you kind of didn't catch whatever, something that I said, so you can just read everything, examples, that sort of stuff. Scott Lowe, the big guy in networking. So he has an amazing blog about everything. A bunch of two different commands that you can use if you want to just kind of reverse engineer and kind of see how everything works. So you can use the show command to see all of the different bridges and what, like, the BR int, the BR tunnel, how are they connected, what are they connected to. You could use dump flows to see the flow tables, right? These are kind of these things, but just written in an actual table. And that's about it. So if questions part, any questions? Yeah. Is there a limit on the number of routes you can put on the flow? I imagine there's a limit. I'm not familiar with it. It's not a problem right now, anyway. Yeah. Okay. So the question is, we have these VLAN IDs, are they consistent between different compute nodes? So we are using tunnels between the different compute nodes. So the local VLANs are actually only locally significant. So because you only need to tag networks, right, you don't need to care that it's the same VLAN on the two different compute nodes. You just need to differentiate between two different networks. Any more questions? Yeah. So the number of VLANs should be two to the power of ten. But in this specific case, we're actually not, that limit comes from the tag, the size of the tag, but we're not actually using that with tunnels. We're only using, we're basically only using access ports. So the limit would be theoretically just what can open V-switch, you know, support. But that limit is talking about the number of different networks. So if you have, you know, two to the power of ten networks, then all the power to you. Yeah. Yeah. Yeah. So that's, and that could be handled via a tag inside of another tag and that sort of stuff. Any more questions? What's the performance hit on average? Who's speaking? Sorry. Yeah, hi. What's the performance hit of these tunnels in real life, so bandwidth and latency-wise? Yeah. So we have a bunch of guys running tests for a few months now. So historically, we were using the VLAN tag inside of the tunnel and we kind of forgot to remove the VLAN tag and that had a drastic, that was catastrophical. And we fixed that. And it's nearly the same as with VLANs. But tunneling doesn't have hardware offloading, which is a problem right now, right? Because the physical network interfaces cards of the physical machines, they can do hardware offloading for the VLAN stuff. That's actually done at the hardware level and the tunneling stuff isn't supported yet via hardware, but it will be in a couple of years. So as for actual numbers, I'm not the guy for that, but it's close. That's what I was told. Yeah. Right. So we're adding additional headers, right? The GRE or the VXLAN header. So there's MTU considerations. So basically, in any kind of real deployments, you would basically use jumbo frames or the 9000 MTU, which completely solves that issue, right? And we're actually, there's upstream patches in UTRON right now to kind of help with that stuff, the jumbo frames to even increase performance. Yeah. You told us about this, about this mic learning, vastly. So that's actually solved any problem. So this way, if you have an unknown Mac, you just send it over and you send a bunch of brochures. But like this, how you explain, it needs to be synchronized, it needs to be central working with R-table and stuff. So that's actually cheaper to synchronize the R-table between all type of writers from the app. It is. It is. Because the synchronization of the R-table is basically whenever you're bringing up VMs, right? And the other solution would be to just send the R-proquests, which is done on a much higher rate, right? That's dependent not on the number of VMs, that's dependent on the amount of traffic. So it's a very worthwhile optimization, yeah. Absolutely. All right, so my name is Asaf. I'll be at the Overhead stand, which is near the OpenStack stand. And I'll, you know, I'm a total geek for this stuff, so I'll be happy to talk about networking tunnels and whatever. Thank you. Thank you.
 All right, so we're going to be talking about, it's a very networking oriented talk, the  Vettapur, the Vettapur and networking oriented talk, we're going to be talking briefly about  VLANs in a cloud solution kind of approach. We're specifically going to be talking about  OpenStack and OpenStack Neutron, a bit about VLANs, just an introduction, how do they,  how do we use VLANs to kind of hook up VMs and maybe even segregate to different networks  and stuff like that and the shortcomings of VLANs and how do we use tunnels? So how do  we use tunnels to achieve the, solve the same problem that VLANs currently solve and maybe  even more? So I'll be, so I'm a staff builder, I'm an engineer, a software engineer at Red  Hat, I'm working on networking oriented stuff in OpenStack and in Overt, which you've already  seen multiple talks about and I'll be completely honest, I've probably never talked to more  than 40 people at the same time and I don't know if anyone's ever told you this but you're,  you're a lot of people, there's just a lot of you. So we're going to do just a quick  warm up. This is actually for my mother. So, so we're going to do a one, two, three freedom  sort of deal, if you would. So it's one, two, three. We can do better. We can do slightly  better. So one, two, three. Thank you. That was very good. All right. So I'm kind of used  to moving a lot but I was told not to do that so I'll try to reframe. So this is kind  of a normal proof of concept or kind of a small scale networking deployment. So as you can  see we have two different physical networks, the orange-ish stuff. That's for the management  traffic, that's for like, you know, just telling a compute node that it should bring up a new  VM, stuff like that. That's for management traffic and then we've got another physical  network just for VM traffic, just for VMs talking to each other and talking to outside  of the cloud. So you could create two different physical networks. Of course you could also  just use two different VLANs. But the point of this slide is to just mention that we are  only going to be talking about the VM data network. So this is just about how VMs talk  to each other and to the internet. So just the teal network in this case. So how did  we do that a long, long time ago, you know, yesterday? So we use VLANs. VLANs are a very  kind of well-known concept. They come from the physical world. So it's kind of natural  for networking people to just yell VLAN at kind of every problem. So VLANs have access  ports and they have trunk ports. Right? So that's a switch configuration. You can enter  the physical switch and you can configure each port to be either an access port or a  trunk port. So a trunk port is untagged. It's basically telling, okay, this specific port  is in VLAN 100 or the red VLAN and the traffic coming out of that access port will be completely  untagged. So the VM or the host that's connected to that access port doesn't know that it's  in a VLAN, but the switch does. So if the switch has two access ports and they're both in  different VLANs, then the switch can do filtering. So it can either forward traffic from one  port to the other or not. So that's access ports and trunk ports are kind of different.  You, again, you enter the switch and you configure a specific port to be a trunk port and it  actually carries tagged VLAN traffic. So the actual VLAN identification number, which is  just a 10-bit number. So in this case, it would be like the color, but it would be like VLAN 100  or 200. That's actually carried with the message. So the message is carrying, what VLAN am I  coming from? And that's configured on the trunk port. So a trunk port has a range of VLANs that  are allowed on that trunk port. So like the dashed, the dashed cables, so they're carrying  both VLAN 100 and VLAN 200. So if they get a message from VLAN 500, then, you know, that  traffic will be just filtered and dropped. So that's kind of the point of a trunk port.  So we could see that we have four compute nodes, right? And that bottom left one, that's kind of a  blow-up or a zoom-in. So that's a physical switch, but that's actually a virtual switch, right?  So we, in OpenStack, we use the VRInt, which was talked about in the last lecture. That's called  the integration bridge. So that's a virtual switch like a Linux bridge or an OpenV switch for the  open source plugins, right? Because Neutron, the whole OpenStack Neutron thing, it's  pluggable. So you could have different implementations. You could have OpenStack, sorry,  open source implementations for the API, and you can have proprietary closed source  implementations for the same API, right? So for the open source implementations, you have  virtual switch inside of the compute node. And so the bottom ports are access ports, right?  So VMs are connected to access ports. And actually, the virtual switch is connected to the  physical switch via a trunk port, right? So this works. This is fine. It's kind of awesome.  But there's limitations, right? So just kind of hear me out. If this virtual machine, that's in the blue  network, that's in the blue VLAN, right, VLAN 200. So if it sends a broadcast message, should that  upper right compute node get that broadcast? I mean, we as human beings, we can tell that this  specific compute node isn't hosting any red VMs, right? You don't have any red VMs in the red  network on that compute node. So this broadcast that's originating from a computer on the blue  network, right, that message shouldn't even reach that compute node, right? There's no point.  So the way to accomplish that is would be to interacting with the physical network. And as a  software person, I'm not really into that. I kind of like to work with virtual and software stuff.  So configuring the virtual switch would be, the physical switch would be kind of a bummer. And I  would have to connect to each physical switch, so I could have 100 of these in my cloud. And I would  then have to enter each trunk port and manually configure each trunk port to carry or to trunk  specific VLANs, right? And that would accomplish what we wanted to do. So this broadcast reaches the  physical switch, the physical switch sees that this trunk port is only carrying the red VLAN, so it  wouldn't forward the broadcast message. So mission accomplished, but obviously that's very manual,  that's very tedious, it doesn't scale. It doesn't scale even for small deployments, actually.  So what most system admins do, they just enter each switch. So for each trunk port, they just  enter the entire range, right? Because that's the only thing that's feasible. So that means that  basically every broadcast reaches everywhere, all of the compute nodes, even compute nodes that aren't  hosting VMs in the specific network, right? Does that make sense? So that's kind of a problem.  Also, there's kind of a philosophical issue of taking the VLANs and just, that's kind of a  physical world concept, but our compute nodes are aware of the same VLANs, so we're kind of  taking the VLANs and just bringing them into the virtual world, which is not ideal. So that's  VLANs, right? So we can do better. So tunnels are basically a well-known and familiar concept.  You know them as just VPNs, right? We all use VPNs, we connect from our home if we want to work from  home and stuff like that. So tunnels are basically the, it's like a VPN, but it's unencrypted, right?  So, or a better way of putting it would be that a VPN is an encrypted tunnel. So I don't care about  encryption, I don't know anything about that. Also, who cares about security, right? So I wasn't  kidding. So we're only going to be talking about tunnels, but only just the unencrypted stuff, right?  So I could configure a tunnel and then just hack on encryption on top of that. So that's kind of  uninteresting. And also, there's different types of tunnels. There are GRE tunnels and VX tunnels,  and that doesn't matter. That's just the headers and stuff like that. We're just going to be talking  about concepts. So there's two use cases, right? Just two that I could fit in this slide. So there's,  let's say that I have a large corporation, you know, and I have a bunch of different sites. One is in  Brussels and one is in Namur. So really, the kind of experience that I want to achieve is that  computers hosted in different sites kind of feel like they're in the same site, right? Because, for  example, say Brussels is the kind of the headquarters and it has a bunch of company resources, like  company websites closed for employees only and a bunch of different company resources. So I'd like the  people from other sites to be able to reach those, right? And I'm also using NUT. I'm using private IP  addresses. So normally, without a tunnel, I wouldn't be able to ping from a Brussels computer to a  Namur computer, right? Because I'm using NUT. It's private IP addresses can't reach those computers.  Like in our own home network, we have private addresses. People theoretically can't ping or can't  reach our own computers behind the NUT. So also, I want to be able, like just a personal example, I  have a development laptop in my own office and I want to be able to SSH or reach hypervisors or  computer resources that I'm using and that are in different sites. So we can use a tunnel. Now, mind  you, a tunnel is a manually configured thing which we actually configure on the physical routers, right?  So we basically, we SSH our talent into the left router and we just tell it, listen, if you want to  reach the 172.16 network, then you take all of those, all of that traffic and you route it to a  tunnel device. And the tunnel device is also a second configuration step, sorry, which I also manually  configure, right? So I have this kind of tunnel interface and when I configure that tunnel  interface, I tell it, what's the source IP? I want to this donation IP for the tunnel. And then I just, I  have to configure the routing, just the static routing that I talked about earlier, right? So the left  router knows that if it wants to reach this network, it has to leave through the tunnel device and the  other way around, right? So that's kind of the manual configuration that you would do and you also  have very similar, if you want to work from home, right? So it's kind of like, it's exactly like a  VPN. So you have, you create the tunnel from your own computer to the router or the VPN server that  you're connecting to and it's actually exactly the same thing. Right, so basically tunnels aren't  magic, they're just a silly encapsulation trick, right? If you remember the OSI, 7 layer models,  you've probably heard of this before, the DCP IP models, there's the different layers and different  headers and encapsulation, right? So what are we doing here? We're just taking the data, whatever it is that  you want to send. So this would be the IP packet representing or wrapping that data, right? So the source  IP is some machine in Brussels and the destination IP is in some machine in the other site. So we take that  packet and we wrap it in our tunnel header. That would be GRE in this case, right? So the only  important header in the GRE header or the only important field, sorry, is what's the next protocol? So the  next protocol in this case would also be IP, right? Because each header has data and the data is just a  bunch of random bits. So the question is how does the computer know how to interpret that data? So you tell  it, right? So each header has kind of what's the data inside of me, what's the next header? So the next  header is also IP because in this case we're just taking IP packets and just wrapping them up in more  IP packets. So the beauty here is that the outer IP packet, the source IP is the source of the tunnel, the  destination IP is the destination of the tunnel, right? So the ISPs of the world, right, or the internet are  just seeing a packet that's being routed from 1111 to 2222. So the only thing that we require from our  physical network is that these two routers can actually ping each other. That's the only thing that we kind of  require. And the ISPs or the routers on the internet, they don't know what's this Brussels network or what's  that Namur network, right? They just don't know any of that. They just need to be able to route from one  router to the next. So that's kind of what we do with tunnels and that's kind of the physical or the real  world usage. So in OpenStack Neutron and in the cloud or in the data center, virtualization management,  right, that whole context. So we kind of do it differently. We basically, these are compute nodes or  hypervisors. They're hosting the VMs. So we just create a full batch of tunnels, right? We just hook up each  compute node to every other compute node. We just create these tunnels. And the first thing that kind of  should pop into your mind is that sounds expensive, right? Because, but you just kind of need to remember that  these tunnels are, the overhead is basically just in the database of the compute node and we, that's, that's  fine. So we just create this full mesh of tunnels between all of the compute nodes. And then it's kind of  exactly like the previous example, right? So if a VM wants to talk to another VM, that's another compute node.  So we just do the encapsulation trick and we just wrap up the packets exactly like in the previous example.  And if two computers want to talk in the same hypervisor, then they just talk to each other directly via the  same virtual switch, right? So both of the machines would be connected to the same virtual switch, BRint or  the integration bridge, like an open V switch bridge or switch, and they would just talk to each other  directly. Right. So what if we want to achieve segregation, right? Like in, in this previous example, you  can see that I still have two networks. I have the red network and the blue network. So, all right. So about  segregation or kind of splitting, splitting up networking. So two VMs on the same hypervisor, they're still  using VLANs, which is kind of confusing. Okay. So we're, we're using tunneling for the actual, for forwarding  traffic between the two different, between two different compute nodes. But the BRint on each compute node or  the virtual switch is still using VLAN access ports to segregate between different networks, right? So VMs  connected to the same virtual switch, they're still using VLAN access ports. So each network gets a different  VLAN. So we're kind of mixing and matching. But more interesting is what, what's, what's going on with traffic  on different compute nodes, right? We want to talk about the tunnels. So with VLANs, we use trunk ports, right?  We tagged the frames with the VLAN identification number. So it's VLAN 100, VLAN 200. And that information  isn't lost. It's being trunked or forwarded across the entire network. So the other end knows what VLAN was this  message originated from. So I can either filter or not. So we're basically doing the exact same thing. All  right? So we're using the GRE header or the VXLad header, basically the tunneling header. And we're using a  field there which is called the tunnel ID. So the tunnel ID, it's basically the exact same thing, right?  We take a network, we tag it with a specific number, and that number is placed inside the tunnel header. So  it's, so that way we can color or we can kind of tag packets with the ID of that network, right? So that's how  we do segregation across compute nodes. So we, we've kind of got to talk about how do we, what's the, what's  the logic, right? How do we forward unicast traffic? How do we forward broadcast traffic? So this is kind of a  short review. So physical, just kind of classical switches, could be physical switches, it could be virtual  switches. They're called just layer two learning switches, right? So how do they work? They have this kind of  table which is a binding or a map between the port number, right? So if I, if the ports are numbered 1 to 24,  so each port is bound to a set of MAC addresses, right? So basically the switch is kind of sniffing traffic as it  goes through it. It looks at the source MAC address, right? Where is this message coming from? And what, what port  is it coming from? So if it, so if MAC A is the source MAC and it came from port one, then now I know that  whenever someone wants to talk to MAC A, I can just forward it to port one, right? And that's where I will find  MAC A. So that's kind of what physical, that's what kind of learning to layer two learning switches do,  virtual or physical. So we're doing something that's very similar. So we have two virtual switches on each  compute node. We have the integration bridge. So all the VMs are connected to that bridge and that bridge is  connected to the tunneling bridge. So the tunneling bridge, we just control it and we can kind of create this  sort of table, right? So each message as, each packet as it comes in from any other compute node into my  own compute node, then I check what tunnel did it come in, come in on, right? Was it the first tunnel, the  second tunnel? Because I'm connected, I have a tunnel to each other compute node. So I check what compute  node this message came in on, right? So that's the peer, that's the IP part, the peer. And we just said that  the messages are tagged, right? So I take the tunnel ID, which is actually the network ID, and I take those  two, just numbers, and I bound them to the source MAC address. So it's basically the exact same concept as a  learning switch. We just do that for different headers, right, for the tunnel and the tunnel ID and the  source MAC address. So that way we know how to forward information through a specific tunnel, right? We need  to know for a VM, basically what compute node should we send our messages to, right? So that's how we kind of do  it. Let's try that again. Okay. Right. So we talked about unicast traffic, we talked about learning, right?  We talked about how do we know which compute node is hosting the VM that I want to talk to. So we talked about  learning, and now let's talk about broadcast traffic, because broadcast traffic is kind of different. Maybe we  want to optimize some stuff there. So generally speaking, the kind of the first approach that you would think  about would be for any broadcast traffic or multicast traffic as well, for that matter, that's leaving a VM on  the red network, right? It would reach the tunneling bridge on that bottom right compute node. So we would just  basically send it out through every tunnel, right? That's kind of the basic approach. And that works. That  definitely works. But we want to be smarter, right? Because we can. And because we kind of, you know, as  software engineers, we just, we like these optimizations. It's something that's nice and we kind of can reason  about. So we can do two things. So we can just send less broadcasts. And we can, and about the broadcasts that are  left, you know, that we didn't optimize away, we can kind of make them smarter. We can make them reach only the  compute nodes that they should reach. So about minimizing broadcasts, turns out a lot of broadcasts are just about  ARP requests, right? If you have, and this is in the physical and the virtual world as well, if you have two computers and they really want to talk to each other, right?  So the first computer knows the IP address of the second computer, but it doesn't know its MAC address because it never talked to it.  So it just asks. So it asks in a broadcast fashion. So it said in the broadcast, what is the MAC of IP address so and so?  And that other machine answers it. So a lot of broadcast traffic is just because of ARP. So we go, okay. So there's a neutron controller.  And this is without a dedicated SDN controller, right? We, in the previous session we talked about open daylight, SDN, that sort of stuff.  So just in neutron, in OpenStack Neutron, we know each VM, what is its IP address, we know what is its MAC address, and we know where is it scheduled, right?  So what compute node is hosting the VM? So that's based, so we're basically capable of filling ARP tables, right?  We can, we know for each IP address, what is its MAC address, and we can send that information to all of the compute nodes.  So all of the compute nodes actually have these full ARP tables, right, of all of the VMs.  So that means that when this red VM is trying to ping or somewhere communicate with the other red VM,  so the local compute node actually received a full ARP table from the neutron controller, and it knows, okay, so this VM, I know what's its IP address,  I know what is its MAC address, and I'll just answer it locally, right?  There's no reason to take the ARP broadcast and forward it to all of the tunnels. There's no reason to do that.  So this compute node will actually answer the local ARP request.  So that's basically minimizing broadcasts, right? Just a lot, just less broadcasts, that's good.  The other thing that we can do about the remaining broadcasts, which there's still a lot of, we can just make them smarter.  So generally speaking, again, going back to that centralized neutron controller, which knows all of the VMs, where are they, what are their IP addresses,  what are their MAC addresses? So whenever this VM sends out a broadcast message, this compute node now knows, okay, I probably shouldn't forward it out that tunnel,  right? Because someone, the controller, someone told me that there's no red VMs here.  So that's the other thing that we can do, and there's kind of small asterisks. This barely works.  It only works with the ML2 plug-in with a specific mechanism driver from specific versions, right?  But that's in the small details, small font. Right.  So we're going to do a very shallow deep dive into kind of the flow tables or the kind of the logic in the tunneling bridge, right?  So I'm going to be talking about the open V-switch specifically, because that's what I know about.  So I'm only going to be talking about something that I know about.  So open V-switch, each bridge or each instance of an open virtual switch can operate in two modes, right?  Two different modes. The first mode is just normal mode, which is just a learning switch, just a regular switch.  Like we talked about earlier, just learning MAC addresses and all of that stuff.  The other mode is called flow mode, which basically means that I can manually control the logic of that virtual switch.  I can put in rules which will dictate how that switch works.  And it's exclusive. It's either or. You can be either in normal mode or in flow mode.  So about these flows, so you can create, you can put in new flows into a virtual switch in kind of two different approaches.  You could do it locally via the command line, right?  Via this open V-switch, open flow control, and you can just create new flows and manually put them in the virtual switch.  You could, of course, also send SSH commands, so to do that locally.  And you could also use an SDN protocol, like OpenFlow.  So there's, right, so the OpenFlow protocol, basically you can have this kind of centralized node which can send these messages,  OpenFlow messages to all of the virtual switches.  And the virtual switches speak the same protocol, so they can configure themselves, right?  They can add flows to their own tables.  So in Neutron, right now what we do is that we have this software, the Neutron Open V-switch agent.  So the agent is sitting locally on every compute node, and it can receive management traffic, right?  Like bring up a new VM, bring up a new virtual network interface card.  So whenever this piece of software receives management traffic, it can basically, what it does is that it's, it administrates local  OpenFlow control messages.  So it configures itself via management traffic that's being received from the Neutron controller.  Okay.  So what are these flows, right?  We're basically taking rules or flows, and we're kind of manually configuring each virtual switch.  So a flow is basically comprised of two parts.  It's a match part, and it's an action part.  So a match is basically, should this, right, so I have a bunch of different flows, a bunch of different rules.  So should the first flow match the incoming packet, or should the second, or should the third?  So I can match on a bunch of, of all, basically all the headers.  The layer two headers, the layer three headers, right?  So the ports, and the IP addresses, and the MAC addresses, and all that sort of stuff.  So, okay, so basically a specific flow matched or caught a packet.  So, all right, so it caught a packet, and now what should this flow actually do?  So what is the action part?  So the action part, it can change headers.  So I can do, like, not.  So I could change the source IP address.  I can change the destination IP address.  I can change ports.  I could do all sorts of stuff.  I can take the message and forward it to a specific port, to all ports, or to a group of ports.  I can, obviously, do broadcasts.  I can drop, right?  I can filter messages, so I could basically implement a firewall.  I can learn new flows on the fly, right?  So I introduce the flow that matches packets, and according to those packets, it can actually generate new flows  and write those new flows back into the table.  And we'll talk about that in a minute.  And we can resubmit to another table.  So what's a table?  So, as it turns out, a table is basically just a group of flows.  And we can have a bunch of different tables that all belong to the same virtual switch.  It's kind of just a convenient way to manage lots of flows.  So all of the messages enter the first table, which is, of course, you know, table zero.  We always count from zero because we're kind of idiots.  So these messages can be resubmitted from table zero to just any other table.  And for each table, right, a group of flows.  So the flows are processed according to their priority, which was configured on each flow.  And if, for example, a message came in on table zero, right, so the first flow doesn't match,  the second flow doesn't match, and the final flow doesn't match either, then what happens?  So that message is either dropped, or if there's an SDN controller configured,  which with just like regular neutron, there isn't, so the message is dropped.  That's kind of the default.  Right, so kind of logically speaking, or what's the flow table, how do we use these different tables,  or kind of how does the logic work?  So traffic that's coming in from a VM on the local compute node, right,  just a VM connected to the integration bridge, all VMs are connected to that bridge.  So that enters the tunneling bridge, which is represented here.  So it has a bunch of different tables.  So table zero, as we said, that's the first table.  So we basically say, oh, okay, this is traffic, traffic that's coming in from the integration bridge.  So we'll just resubmit it to table one.  Table one classifies unicast traffic and non-unicast, which is basically multicast and broadcast.  So unicast traffic reaches table 20.  Table 20 has a bunch of pre-learned flows.  This is the unicast table.  So this table should basically tell me, should I forward this message to table two,  the first tunnel, the second tunnel, the third tunnel, right, what compute node should I be speaking to?  So I either know the destination MAC address, so I just forward it to a specific tunnel,  or I don't know, right?  This is an unknown MAC address just because that MAC address hasn't spoken to me yet.  So I forward it to table 21, and table 21 is multicast, broadcast, and unknown destination MAC addresses.  So I basically forward it to all tunnel devices, right?  Why the air quotes?  Because we talked earlier about the broadcast optimization, right?  So if that's enabled, then I don't need to send the message to all of the tunnels, just to some of the tunnels, right?  So what about the other way around?  If a message is coming in from a tunnel and not from a local VM, so it's coming in on table 20,  it's traffic that's coming in from a tunnel and not the integration bridge, so it's going to table two.  Table two is basically taking the tunnel headers, just throwing that away,  adding the VLAN tag, which is kind of weird, and then forwarding that to table 10.  Table 10 looks at the source MAC address and does the learning part, right, which we talked about previously.  So it looks at the source MAC address, what tunnel that this came in on,  both the tunnel ID and the actual tunnel itself, and it populates table 20, right, which was this one.  That's the unicast table, right?  And then finally, it forwards it to the integration bridge.  So again, we drop the tunnel headers, all of the GRE or the VXLAN stuff,  and we convert the tunnel ID to the VLAN ID, right?  Because as we said, the actual, the local BR int or the virtual switch is still using VLANs to mark specific networks, right?  So that's kind of the deal there. And just kind of the more information part.  So we'll start at the end. The shameless plug.  Basically, I have a blog which has all this entire thing just verbally written.  So if you kind of didn't catch whatever, something that I said,  so you can just read everything, examples, that sort of stuff.  Scott Lowe, the big guy in networking.  So he has an amazing blog about everything.  A bunch of two different commands that you can use if you want to just kind of reverse engineer and kind of see how everything works.  So you can use the show command to see all of the different bridges and what, like, the BR int, the BR tunnel, how are they connected, what are they connected to.  You could use dump flows to see the flow tables, right?  These are kind of these things, but just written in an actual table.  And that's about it. So if questions part, any questions?  Yeah.  Is there a limit on the number of routes you can put on the flow?  I imagine there's a limit. I'm not familiar with it. It's not a problem right now, anyway.  Yeah.  Okay. So the question is, we have these VLAN IDs, are they consistent between different compute nodes?  So we are using tunnels between the different compute nodes.  So the local VLANs are actually only locally significant.  So because you only need to tag networks, right, you don't need to care that it's the same VLAN on the two different compute nodes.  You just need to differentiate between two different networks.  Any more questions? Yeah.  So the number of VLANs should be two to the power of ten.  But in this specific case, we're actually not, that limit comes from the tag, the size of the tag,  but we're not actually using that with tunnels.  We're only using, we're basically only using access ports.  So the limit would be theoretically just what can open V-switch, you know, support.  But that limit is talking about the number of different networks.  So if you have, you know, two to the power of ten networks, then all the power to you.  Yeah.  Yeah. Yeah.  So that's, and that could be handled via a tag inside of another tag and that sort of stuff.  Any more questions? What's the performance hit on average?  Who's speaking? Sorry.  Yeah, hi. What's the performance hit of these tunnels in real life, so bandwidth and latency-wise?  Yeah. So we have a bunch of guys running tests for a few months now.  So historically, we were using the VLAN tag inside of the tunnel and we kind of forgot to remove the VLAN tag  and that had a drastic, that was catastrophical.  And we fixed that. And it's nearly the same as with VLANs.  But tunneling doesn't have hardware offloading, which is a problem right now, right?  Because the physical network interfaces cards of the physical machines,  they can do hardware offloading for the VLAN stuff.  That's actually done at the hardware level and the tunneling stuff isn't supported yet via hardware,  but it will be in a couple of years.  So as for actual numbers, I'm not the guy for that, but it's close.  That's what I was told.  Yeah.  Right. So we're adding additional headers, right?  The GRE or the VXLAN header. So there's MTU considerations.  So basically, in any kind of real deployments, you would basically use jumbo frames or the 9000 MTU,  which completely solves that issue, right?  And we're actually, there's upstream patches in UTRON right now to kind of help with that stuff,  the jumbo frames to even increase performance.  Yeah.  You told us about this, about this mic learning, vastly.  So that's actually solved any problem.  So this way, if you have an unknown Mac, you just send it over and you send a bunch of brochures.  But like this, how you explain, it needs to be synchronized,  it needs to be central working with R-table and stuff.  So that's actually cheaper to synchronize the R-table between all type of writers from the app.  It is. It is. Because the synchronization of the R-table is basically whenever you're bringing up VMs, right?  And the other solution would be to just send the R-proquests, which is done on a much higher rate, right?  That's dependent not on the number of VMs, that's dependent on the amount of traffic.  So it's a very worthwhile optimization, yeah. Absolutely.  All right, so my name is Asaf. I'll be at the Overhead stand, which is near the OpenStack stand.  And I'll, you know, I'm a total geek for this stuff, so I'll be happy to talk about networking tunnels and whatever.  Thank you.  Thank you.
_FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_0_3215.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_3216_3527.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_3528_3827.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_3828_4175.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_4176_4667.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_4668_5123.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_5124_5363.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_5364_6563.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_6564_6647.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_6648_11843.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_11844_13163.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_13164_14435.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_14436_14507.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_14508_18683.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_18684_18935.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_18936_22283.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_22284_22511.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_22512_24095.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_24096_24623.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_24624_25199.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_25200_27455.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_27456_27719.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_27720_27803.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_27804_28019.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_28020_28367.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_28368_31931.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_31932_32303.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_32304_33275.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_33276_34259.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_34260_34883.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_34884_34991.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_34992_38375.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_38376_38711.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_38712_39515.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_39516_41627.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_41628_42227.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_42228_42587.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_42588_43223.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_43224_44819.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_44820_45059.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_45060_46523.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_46524_46811.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_46812_50951.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_50952_51107.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_51108_52031.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_52032_58367.png _FOSDEM_2014__Tunnels_as_a_Connectivity_and_Segregation_Solution_for_Virtualized_Networks_58368_59751.png
All right, so we're going to be talking about, it's a very networking oriented talk, the Vettapur, the Vettapur and networking oriented talk, we're going to be talking briefly about VLANs in a cloud solution kind of approach. We're specifically going to be talking about OpenStack and OpenStack Neutron, a bit about VLANs, just an introduction, how do they, how do we use VLANs to kind of hook up VMs and maybe even segregate to different networks and stuff like that and the shortcomings of VLANs and how do we use tunnels? So how do we use tunnels to achieve the, solve the same problem that VLANs currently solve and maybe even more? So I'll be, so I'm a staff builder, I'm an engineer, a software engineer at Red Hat, I'm working on networking oriented stuff in OpenStack and in Overt, which you've already seen multiple talks about and I'll be completely honest, I've probably never talked to more than 40 people at the same time and I don't know if anyone's ever told you this but you're, you're a lot of people, there's just a lot of you. So we're going to do just a quick warm up. This is actually for my mother. So, so we're going to do a one, two, three freedom sort of deal, if you would. So it's one, two, three. We can do better. We can do slightly better. So one, two, three. Thank you. That was very good. All right. So I'm kind of used to moving a lot but I was told not to do that so I'll try to reframe. So this is kind of a normal proof of concept or kind of a small scale networking deployment. So as you can see we have two different physical networks, the orange-ish stuff. That's for the management traffic, that's for like, you know, just telling a compute node that it should bring up a new VM, stuff like that. That's for management traffic and then we've got another physical network just for VM traffic, just for VMs talking to each other and talking to outside of the cloud. So you could create two different physical networks. Of course you could also just use two different VLANs. But the point of this slide is to just mention that we are only going to be talking about the VM data network. So this is just about how VMs talk to each other and to the internet. So just the teal network in this case. So how did we do that a long, long time ago, you know, yesterday? So we use VLANs. VLANs are a very kind of well-known concept. They come from the physical world. So it's kind of natural for networking people to just yell VLAN at kind of every problem. So VLANs have access ports and they have trunk ports. Right? So that's a switch configuration. You can enter the physical switch and you can configure each port to be either an access port or a trunk port. So a trunk port is untagged. It's basically telling, okay, this specific port is in VLAN 100 or the red VLAN and the traffic coming out of that access port will be completely untagged. So the VM or the host that's connected to that access port doesn't know that it's in a VLAN, but the switch does. So if the switch has two access ports and they're both in different VLANs, then the switch can do filtering. So it can either forward traffic from one port to the other or not. So that's access ports and trunk ports are kind of different. You, again, you enter the switch and you configure a specific port to be a trunk port and it actually carries tagged VLAN traffic. So the actual VLAN identification number, which is just a 10-bit number. So in this case, it would be like the color, but it would be like VLAN 100 or 200. That's actually carried with the message. So the message is carrying, what VLAN am I coming from? And that's configured on the trunk port. So a trunk port has a range of VLANs that are allowed on that trunk port. So like the dashed, the dashed cables, so they're carrying both VLAN 100 and VLAN 200. So if they get a message from VLAN 500, then, you know, that traffic will be just filtered and dropped. So that's kind of the point of a trunk port. So we could see that we have four compute nodes, right? And that bottom left one, that's kind of a blow-up or a zoom-in. So that's a physical switch, but that's actually a virtual switch, right? So we, in OpenStack, we use the VRInt, which was talked about in the last lecture. That's called the integration bridge. So that's a virtual switch like a Linux bridge or an OpenV switch for the open source plugins, right? Because Neutron, the whole OpenStack Neutron thing, it's pluggable. So you could have different implementations. You could have OpenStack, sorry, open source implementations for the API, and you can have proprietary closed source implementations for the same API, right? So for the open source implementations, you have virtual switch inside of the compute node. And so the bottom ports are access ports, right? So VMs are connected to access ports. And actually, the virtual switch is connected to the physical switch via a trunk port, right? So this works. This is fine. It's kind of awesome. But there's limitations, right? So just kind of hear me out. If this virtual machine, that's in the blue network, that's in the blue VLAN, right, VLAN 200. So if it sends a broadcast message, should that upper right compute node get that broadcast? I mean, we as human beings, we can tell that this specific compute node isn't hosting any red VMs, right? You don't have any red VMs in the red network on that compute node. So this broadcast that's originating from a computer on the blue network, right, that message shouldn't even reach that compute node, right? There's no point. So the way to accomplish that is would be to interacting with the physical network. And as a software person, I'm not really into that. I kind of like to work with virtual and software stuff. So configuring the virtual switch would be, the physical switch would be kind of a bummer. And I would have to connect to each physical switch, so I could have 100 of these in my cloud. And I would then have to enter each trunk port and manually configure each trunk port to carry or to trunk specific VLANs, right? And that would accomplish what we wanted to do. So this broadcast reaches the physical switch, the physical switch sees that this trunk port is only carrying the red VLAN, so it wouldn't forward the broadcast message. So mission accomplished, but obviously that's very manual, that's very tedious, it doesn't scale. It doesn't scale even for small deployments, actually. So what most system admins do, they just enter each switch. So for each trunk port, they just enter the entire range, right? Because that's the only thing that's feasible. So that means that basically every broadcast reaches everywhere, all of the compute nodes, even compute nodes that aren't hosting VMs in the specific network, right? Does that make sense? So that's kind of a problem. Also, there's kind of a philosophical issue of taking the VLANs and just, that's kind of a physical world concept, but our compute nodes are aware of the same VLANs, so we're kind of taking the VLANs and just bringing them into the virtual world, which is not ideal. So that's VLANs, right? So we can do better. So tunnels are basically a well-known and familiar concept. You know them as just VPNs, right? We all use VPNs, we connect from our home if we want to work from home and stuff like that. So tunnels are basically the, it's like a VPN, but it's unencrypted, right? So, or a better way of putting it would be that a VPN is an encrypted tunnel. So I don't care about encryption, I don't know anything about that. Also, who cares about security, right? So I wasn't kidding. So we're only going to be talking about tunnels, but only just the unencrypted stuff, right? So I could configure a tunnel and then just hack on encryption on top of that. So that's kind of uninteresting. And also, there's different types of tunnels. There are GRE tunnels and VX tunnels, and that doesn't matter. That's just the headers and stuff like that. We're just going to be talking about concepts. So there's two use cases, right? Just two that I could fit in this slide. So there's, let's say that I have a large corporation, you know, and I have a bunch of different sites. One is in Brussels and one is in Namur. So really, the kind of experience that I want to achieve is that computers hosted in different sites kind of feel like they're in the same site, right? Because, for example, say Brussels is the kind of the headquarters and it has a bunch of company resources, like company websites closed for employees only and a bunch of different company resources. So I'd like the people from other sites to be able to reach those, right? And I'm also using NUT. I'm using private IP addresses. So normally, without a tunnel, I wouldn't be able to ping from a Brussels computer to a Namur computer, right? Because I'm using NUT. It's private IP addresses can't reach those computers. Like in our own home network, we have private addresses. People theoretically can't ping or can't reach our own computers behind the NUT. So also, I want to be able, like just a personal example, I have a development laptop in my own office and I want to be able to SSH or reach hypervisors or computer resources that I'm using and that are in different sites. So we can use a tunnel. Now, mind you, a tunnel is a manually configured thing which we actually configure on the physical routers, right? So we basically, we SSH our talent into the left router and we just tell it, listen, if you want to reach the 172.16 network, then you take all of those, all of that traffic and you route it to a tunnel device. And the tunnel device is also a second configuration step, sorry, which I also manually configure, right? So I have this kind of tunnel interface and when I configure that tunnel interface, I tell it, what's the source IP? I want to this donation IP for the tunnel. And then I just, I have to configure the routing, just the static routing that I talked about earlier, right? So the left router knows that if it wants to reach this network, it has to leave through the tunnel device and the other way around, right? So that's kind of the manual configuration that you would do and you also have very similar, if you want to work from home, right? So it's kind of like, it's exactly like a VPN. So you have, you create the tunnel from your own computer to the router or the VPN server that you're connecting to and it's actually exactly the same thing. Right, so basically tunnels aren't magic, they're just a silly encapsulation trick, right? If you remember the OSI, 7 layer models, you've probably heard of this before, the DCP IP models, there's the different layers and different headers and encapsulation, right? So what are we doing here? We're just taking the data, whatever it is that you want to send. So this would be the IP packet representing or wrapping that data, right? So the source IP is some machine in Brussels and the destination IP is in some machine in the other site. So we take that packet and we wrap it in our tunnel header. That would be GRE in this case, right? So the only important header in the GRE header or the only important field, sorry, is what's the next protocol? So the next protocol in this case would also be IP, right? Because each header has data and the data is just a bunch of random bits. So the question is how does the computer know how to interpret that data? So you tell it, right? So each header has kind of what's the data inside of me, what's the next header? So the next header is also IP because in this case we're just taking IP packets and just wrapping them up in more IP packets. So the beauty here is that the outer IP packet, the source IP is the source of the tunnel, the destination IP is the destination of the tunnel, right? So the ISPs of the world, right, or the internet are just seeing a packet that's being routed from 1111 to 2222. So the only thing that we require from our physical network is that these two routers can actually ping each other. That's the only thing that we kind of require. And the ISPs or the routers on the internet, they don't know what's this Brussels network or what's that Namur network, right? They just don't know any of that. They just need to be able to route from one router to the next. So that's kind of what we do with tunnels and that's kind of the physical or the real world usage. So in OpenStack Neutron and in the cloud or in the data center, virtualization management, right, that whole context. So we kind of do it differently. We basically, these are compute nodes or hypervisors. They're hosting the VMs. So we just create a full batch of tunnels, right? We just hook up each compute node to every other compute node. We just create these tunnels. And the first thing that kind of should pop into your mind is that sounds expensive, right? Because, but you just kind of need to remember that these tunnels are, the overhead is basically just in the database of the compute node and we, that's, that's fine. So we just create this full mesh of tunnels between all of the compute nodes. And then it's kind of exactly like the previous example, right? So if a VM wants to talk to another VM, that's another compute node. So we just do the encapsulation trick and we just wrap up the packets exactly like in the previous example. And if two computers want to talk in the same hypervisor, then they just talk to each other directly via the same virtual switch, right? So both of the machines would be connected to the same virtual switch, BRint or the integration bridge, like an open V switch bridge or switch, and they would just talk to each other directly. Right. So what if we want to achieve segregation, right? Like in, in this previous example, you can see that I still have two networks. I have the red network and the blue network. So, all right. So about segregation or kind of splitting, splitting up networking. So two VMs on the same hypervisor, they're still using VLANs, which is kind of confusing. Okay. So we're, we're using tunneling for the actual, for forwarding traffic between the two different, between two different compute nodes. But the BRint on each compute node or the virtual switch is still using VLAN access ports to segregate between different networks, right? So VMs connected to the same virtual switch, they're still using VLAN access ports. So each network gets a different VLAN. So we're kind of mixing and matching. But more interesting is what, what's, what's going on with traffic on different compute nodes, right? We want to talk about the tunnels. So with VLANs, we use trunk ports, right? We tagged the frames with the VLAN identification number. So it's VLAN 100, VLAN 200. And that information isn't lost. It's being trunked or forwarded across the entire network. So the other end knows what VLAN was this message originated from. So I can either filter or not. So we're basically doing the exact same thing. All right? So we're using the GRE header or the VXLad header, basically the tunneling header. And we're using a field there which is called the tunnel ID. So the tunnel ID, it's basically the exact same thing, right? We take a network, we tag it with a specific number, and that number is placed inside the tunnel header. So it's, so that way we can color or we can kind of tag packets with the ID of that network, right? So that's how we do segregation across compute nodes. So we, we've kind of got to talk about how do we, what's the, what's the logic, right? How do we forward unicast traffic? How do we forward broadcast traffic? So this is kind of a short review. So physical, just kind of classical switches, could be physical switches, it could be virtual switches. They're called just layer two learning switches, right? So how do they work? They have this kind of table which is a binding or a map between the port number, right? So if I, if the ports are numbered 1 to 24, so each port is bound to a set of MAC addresses, right? So basically the switch is kind of sniffing traffic as it goes through it. It looks at the source MAC address, right? Where is this message coming from? And what, what port is it coming from? So if it, so if MAC A is the source MAC and it came from port one, then now I know that whenever someone wants to talk to MAC A, I can just forward it to port one, right? And that's where I will find MAC A. So that's kind of what physical, that's what kind of learning to layer two learning switches do, virtual or physical. So we're doing something that's very similar. So we have two virtual switches on each compute node. We have the integration bridge. So all the VMs are connected to that bridge and that bridge is connected to the tunneling bridge. So the tunneling bridge, we just control it and we can kind of create this sort of table, right? So each message as, each packet as it comes in from any other compute node into my own compute node, then I check what tunnel did it come in, come in on, right? Was it the first tunnel, the second tunnel? Because I'm connected, I have a tunnel to each other compute node. So I check what compute node this message came in on, right? So that's the peer, that's the IP part, the peer. And we just said that the messages are tagged, right? So I take the tunnel ID, which is actually the network ID, and I take those two, just numbers, and I bound them to the source MAC address. So it's basically the exact same concept as a learning switch. We just do that for different headers, right, for the tunnel and the tunnel ID and the source MAC address. So that way we know how to forward information through a specific tunnel, right? We need to know for a VM, basically what compute node should we send our messages to, right? So that's how we kind of do it. Let's try that again. Okay. Right. So we talked about unicast traffic, we talked about learning, right? We talked about how do we know which compute node is hosting the VM that I want to talk to. So we talked about learning, and now let's talk about broadcast traffic, because broadcast traffic is kind of different. Maybe we want to optimize some stuff there. So generally speaking, the kind of the first approach that you would think about would be for any broadcast traffic or multicast traffic as well, for that matter, that's leaving a VM on the red network, right? It would reach the tunneling bridge on that bottom right compute node. So we would just basically send it out through every tunnel, right? That's kind of the basic approach. And that works. That definitely works. But we want to be smarter, right? Because we can. And because we kind of, you know, as software engineers, we just, we like these optimizations. It's something that's nice and we kind of can reason about. So we can do two things. So we can just send less broadcasts. And we can, and about the broadcasts that are left, you know, that we didn't optimize away, we can kind of make them smarter. We can make them reach only the compute nodes that they should reach. So about minimizing broadcasts, turns out a lot of broadcasts are just about ARP requests, right? If you have, and this is in the physical and the virtual world as well, if you have two computers and they really want to talk to each other, right? So the first computer knows the IP address of the second computer, but it doesn't know its MAC address because it never talked to it. So it just asks. So it asks in a broadcast fashion. So it said in the broadcast, what is the MAC of IP address so and so? And that other machine answers it. So a lot of broadcast traffic is just because of ARP. So we go, okay. So there's a neutron controller. And this is without a dedicated SDN controller, right? We, in the previous session we talked about open daylight, SDN, that sort of stuff. So just in neutron, in OpenStack Neutron, we know each VM, what is its IP address, we know what is its MAC address, and we know where is it scheduled, right? So what compute node is hosting the VM? So that's based, so we're basically capable of filling ARP tables, right? We can, we know for each IP address, what is its MAC address, and we can send that information to all of the compute nodes. So all of the compute nodes actually have these full ARP tables, right, of all of the VMs. So that means that when this red VM is trying to ping or somewhere communicate with the other red VM, so the local compute node actually received a full ARP table from the neutron controller, and it knows, okay, so this VM, I know what's its IP address, I know what is its MAC address, and I'll just answer it locally, right? There's no reason to take the ARP broadcast and forward it to all of the tunnels. There's no reason to do that. So this compute node will actually answer the local ARP request. So that's basically minimizing broadcasts, right? Just a lot, just less broadcasts, that's good. The other thing that we can do about the remaining broadcasts, which there's still a lot of, we can just make them smarter. So generally speaking, again, going back to that centralized neutron controller, which knows all of the VMs, where are they, what are their IP addresses, what are their MAC addresses? So whenever this VM sends out a broadcast message, this compute node now knows, okay, I probably shouldn't forward it out that tunnel, right? Because someone, the controller, someone told me that there's no red VMs here. So that's the other thing that we can do, and there's kind of small asterisks. This barely works. It only works with the ML2 plug-in with a specific mechanism driver from specific versions, right? But that's in the small details, small font. Right. So we're going to do a very shallow deep dive into kind of the flow tables or the kind of the logic in the tunneling bridge, right? So I'm going to be talking about the open V-switch specifically, because that's what I know about. So I'm only going to be talking about something that I know about. So open V-switch, each bridge or each instance of an open virtual switch can operate in two modes, right? Two different modes. The first mode is just normal mode, which is just a learning switch, just a regular switch. Like we talked about earlier, just learning MAC addresses and all of that stuff. The other mode is called flow mode, which basically means that I can manually control the logic of that virtual switch. I can put in rules which will dictate how that switch works. And it's exclusive. It's either or. You can be either in normal mode or in flow mode. So about these flows, so you can create, you can put in new flows into a virtual switch in kind of two different approaches. You could do it locally via the command line, right? Via this open V-switch, open flow control, and you can just create new flows and manually put them in the virtual switch. You could, of course, also send SSH commands, so to do that locally. And you could also use an SDN protocol, like OpenFlow. So there's, right, so the OpenFlow protocol, basically you can have this kind of centralized node which can send these messages, OpenFlow messages to all of the virtual switches. And the virtual switches speak the same protocol, so they can configure themselves, right? They can add flows to their own tables. So in Neutron, right now what we do is that we have this software, the Neutron Open V-switch agent. So the agent is sitting locally on every compute node, and it can receive management traffic, right? Like bring up a new VM, bring up a new virtual network interface card. So whenever this piece of software receives management traffic, it can basically, what it does is that it's, it administrates local OpenFlow control messages. So it configures itself via management traffic that's being received from the Neutron controller. Okay. So what are these flows, right? We're basically taking rules or flows, and we're kind of manually configuring each virtual switch. So a flow is basically comprised of two parts. It's a match part, and it's an action part. So a match is basically, should this, right, so I have a bunch of different flows, a bunch of different rules. So should the first flow match the incoming packet, or should the second, or should the third? So I can match on a bunch of, of all, basically all the headers. The layer two headers, the layer three headers, right? So the ports, and the IP addresses, and the MAC addresses, and all that sort of stuff. So, okay, so basically a specific flow matched or caught a packet. So, all right, so it caught a packet, and now what should this flow actually do? So what is the action part? So the action part, it can change headers. So I can do, like, not. So I could change the source IP address. I can change the destination IP address. I can change ports. I could do all sorts of stuff. I can take the message and forward it to a specific port, to all ports, or to a group of ports. I can, obviously, do broadcasts. I can drop, right? I can filter messages, so I could basically implement a firewall. I can learn new flows on the fly, right? So I introduce the flow that matches packets, and according to those packets, it can actually generate new flows and write those new flows back into the table. And we'll talk about that in a minute. And we can resubmit to another table. So what's a table? So, as it turns out, a table is basically just a group of flows. And we can have a bunch of different tables that all belong to the same virtual switch. It's kind of just a convenient way to manage lots of flows. So all of the messages enter the first table, which is, of course, you know, table zero. We always count from zero because we're kind of idiots. So these messages can be resubmitted from table zero to just any other table. And for each table, right, a group of flows. So the flows are processed according to their priority, which was configured on each flow. And if, for example, a message came in on table zero, right, so the first flow doesn't match, the second flow doesn't match, and the final flow doesn't match either, then what happens? So that message is either dropped, or if there's an SDN controller configured, which with just like regular neutron, there isn't, so the message is dropped. That's kind of the default. Right, so kind of logically speaking, or what's the flow table, how do we use these different tables, or kind of how does the logic work? So traffic that's coming in from a VM on the local compute node, right, just a VM connected to the integration bridge, all VMs are connected to that bridge. So that enters the tunneling bridge, which is represented here. So it has a bunch of different tables. So table zero, as we said, that's the first table. So we basically say, oh, okay, this is traffic, traffic that's coming in from the integration bridge. So we'll just resubmit it to table one. Table one classifies unicast traffic and non-unicast, which is basically multicast and broadcast. So unicast traffic reaches table 20. Table 20 has a bunch of pre-learned flows. This is the unicast table. So this table should basically tell me, should I forward this message to table two, the first tunnel, the second tunnel, the third tunnel, right, what compute node should I be speaking to? So I either know the destination MAC address, so I just forward it to a specific tunnel, or I don't know, right? This is an unknown MAC address just because that MAC address hasn't spoken to me yet. So I forward it to table 21, and table 21 is multicast, broadcast, and unknown destination MAC addresses. So I basically forward it to all tunnel devices, right? Why the air quotes? Because we talked earlier about the broadcast optimization, right? So if that's enabled, then I don't need to send the message to all of the tunnels, just to some of the tunnels, right? So what about the other way around? If a message is coming in from a tunnel and not from a local VM, so it's coming in on table 20, it's traffic that's coming in from a tunnel and not the integration bridge, so it's going to table two. Table two is basically taking the tunnel headers, just throwing that away, adding the VLAN tag, which is kind of weird, and then forwarding that to table 10. Table 10 looks at the source MAC address and does the learning part, right, which we talked about previously. So it looks at the source MAC address, what tunnel that this came in on, both the tunnel ID and the actual tunnel itself, and it populates table 20, right, which was this one. That's the unicast table, right? And then finally, it forwards it to the integration bridge. So again, we drop the tunnel headers, all of the GRE or the VXLAN stuff, and we convert the tunnel ID to the VLAN ID, right? Because as we said, the actual, the local BR int or the virtual switch is still using VLANs to mark specific networks, right? So that's kind of the deal there. And just kind of the more information part. So we'll start at the end. The shameless plug. Basically, I have a blog which has all this entire thing just verbally written. So if you kind of didn't catch whatever, something that I said, so you can just read everything, examples, that sort of stuff. Scott Lowe, the big guy in networking. So he has an amazing blog about everything. A bunch of two different commands that you can use if you want to just kind of reverse engineer and kind of see how everything works. So you can use the show command to see all of the different bridges and what, like, the BR int, the BR tunnel, how are they connected, what are they connected to. You could use dump flows to see the flow tables, right? These are kind of these things, but just written in an actual table. And that's about it. So if questions part, any questions? Yeah. Is there a limit on the number of routes you can put on the flow? I imagine there's a limit. I'm not familiar with it. It's not a problem right now, anyway. Yeah. Okay. So the question is, we have these VLAN IDs, are they consistent between different compute nodes? So we are using tunnels between the different compute nodes. So the local VLANs are actually only locally significant. So because you only need to tag networks, right, you don't need to care that it's the same VLAN on the two different compute nodes. You just need to differentiate between two different networks. Any more questions? Yeah. So the number of VLANs should be two to the power of ten. But in this specific case, we're actually not, that limit comes from the tag, the size of the tag, but we're not actually using that with tunnels. We're only using, we're basically only using access ports. So the limit would be theoretically just what can open V-switch, you know, support. But that limit is talking about the number of different networks. So if you have, you know, two to the power of ten networks, then all the power to you. Yeah. Yeah. Yeah. So that's, and that could be handled via a tag inside of another tag and that sort of stuff. Any more questions? What's the performance hit on average? Who's speaking? Sorry. Yeah, hi. What's the performance hit of these tunnels in real life, so bandwidth and latency-wise? Yeah. So we have a bunch of guys running tests for a few months now. So historically, we were using the VLAN tag inside of the tunnel and we kind of forgot to remove the VLAN tag and that had a drastic, that was catastrophical. And we fixed that. And it's nearly the same as with VLANs. But tunneling doesn't have hardware offloading, which is a problem right now, right? Because the physical network interfaces cards of the physical machines, they can do hardware offloading for the VLAN stuff. That's actually done at the hardware level and the tunneling stuff isn't supported yet via hardware, but it will be in a couple of years. So as for actual numbers, I'm not the guy for that, but it's close. That's what I was told. Yeah. Right. So we're adding additional headers, right? The GRE or the VXLAN header. So there's MTU considerations. So basically, in any kind of real deployments, you would basically use jumbo frames or the 9000 MTU, which completely solves that issue, right? And we're actually, there's upstream patches in UTRON right now to kind of help with that stuff, the jumbo frames to even increase performance. Yeah. You told us about this, about this mic learning, vastly. So that's actually solved any problem. So this way, if you have an unknown Mac, you just send it over and you send a bunch of brochures. But like this, how you explain, it needs to be synchronized, it needs to be central working with R-table and stuff. So that's actually cheaper to synchronize the R-table between all type of writers from the app. It is. It is. Because the synchronization of the R-table is basically whenever you're bringing up VMs, right? And the other solution would be to just send the R-proquests, which is done on a much higher rate, right? That's dependent not on the number of VMs, that's dependent on the amount of traffic. So it's a very worthwhile optimization, yeah. Absolutely. All right, so my name is Asaf. I'll be at the Overhead stand, which is near the OpenStack stand. And I'll, you know, I'm a total geek for this stuff, so I'll be happy to talk about networking tunnels and whatever. Thank you. Thank you.
Vettapur, the Vettapur and networking oriented shortcomings of VLANs and how do we use tunnels? a compute node that it should bring up a new VM, for VM traffic, just for VMs talking to each yesterday? So we use VLANs. VLANs are a very kind virtual switch, right? So we, in OpenStack, we the way to accomplish that is would be to interacting with the physical network. And as a would have to connect to each physical switch, so manually configure each trunk port to carry or to know them as just VPNs, right? We all use VPNs, different sites. One is in Brussels and one is in want to achieve is that computers hosted in has a bunch of company resources, like company be able to SSH or reach hypervisors or computer our tunnel header. That would be GRE in this the data inside of me, what's the next header? So them up in more IP packets. So the beauty here is in the cloud or in the data center, context. So we kind of do it differently. We another VM, that's another compute node. So we same hypervisor, then they just talk to each to the same virtual switch, BRint or the or switch, and they would just talk to each other network. So, all right. So about segregation or forwarding traffic between the two different, the, what's the logic, right? How do we forward of learning to layer two learning switches do, So that means that when this red VM is trying to their IP addresses, what are their MAC addresses? dive into kind of the flow tables or the kind of software, the Neutron Open V-switch agent. So the don't know, right? This is an unknown MAC address there a limit on the number of routes you can put different compute nodes? So we are using tunnels via hardware, but it will be in a couple of happy to talk about networking tunnels and