The surprising complexity of checksums in TCP/IP
Brauer, Henning
The well-known IP and TCP/UDP (and less well known, ICMP) checksums seem pretty much straighforward. Digging into the network stack reveals a surprising complexity dealing with them and updating. The rise of hardware checksum offloading didn't exactly make things easier. It goes so far that the old "pseudo header checksum" hack where parts of the checksum are precaclucated on the template PCBs and updated on the way out made its way into some of the hardware offloading engines. The talk explains how IP and protocol (UDP/TCP and ICMP) checksums are handled in the OpenBSD network stack and pf, both traditionally and after redesigning. This includes a closer view on performance impact - while the IP checksum only covers the header, the protocol checksums cover the entire payload, which makes them comparably expensive to verify recalculate. While the actual math is dirt cheap, the data access is not, and for forwarded packets we would not access the payload otherwise. Several different output pathes like the regular IP output, the bridging case and various tunneling/encapsulation mechanisms make things harder. The redesigned checksumming mechanism pretty much centralizes the checksum handling instead of having it all over the place, making dealing with the checksums in the rest of the stack much easier. It also allows us to benefit a little more from the NICs' offloading capabilities and fixes a long-standing bug which prevented us from enabling protocol checksum offloading on the RX side on many chipsets.
 Welcome. I'm Henning. I'm saving you from the who am I part? No? Well, I drink beer. I hack an openBSD. You might have come across my name if you're using pf or openbgpd or ntpd or... What else did I do? Female. Pardon? Female, yes. Oh, whatever. It's been a long time. Before I actually start my talk, I do have a question. You know, we had this earthquake yesterday, and there's been this thing, BSD conference plus earthquake. I've been into that before, and so I have just one question. Is there a nuclear power plant near here? Okay, then I can go on. I want to talk about the checksum in TCP IP. We're mostly talking about two sets of checksums here. One is the IP checksum, and the other is the so-called protocol checksum that mostly is TCP and UDP. IPv6 does not have the IP header checksum one. I'll explain later why that doesn't make a difference at all. The actual checksumming algorithm is really, really, really simple. It's really just the lowest word of the once-compliment sum of all the words in the either IP header or almost entire packet. We'll come back to that later. The IP checksum just covers the IP header while the protocol checksum covers a part of the IP header, the actual protocol header, and the entire payload. So there's performance considerations in that. Me being mostly involved with firewalls and routers and everything, I care about forwarded packets, not locally generated ones all that much. The actual math for the checksum is dirt, dirt, dirt cheap. I spent many, many, many years profiling, and you cannot even find the checksumming function anywhere noticeably up in the call graphs. So the calculation can be considered completely free. We're just talking about the math here. That's basically talking about the CPU's integer units, right? The limiting factor that we run into is latency and bandwidth to memory. So we care about caches, RAM, and buses. This is mostly relevant for modern architectures. On the really old architectures, it doesn't really make a difference because their memory, relatively to the CPU, is fast. On modern systems, memory is comparably dog slow because the CPU's also got them fast. Check-summing data that already is in the cache is essentially free because the cache is fast. So in the forwarding case, when we are dealing with the IP pack and we are forwarding, the IP header is in cache. Of course, we need that information for forwarding, right? So IP header checksum free. IPv6 not having that checksum, which they often point out as a performance improvement, doesn't make a difference. You can't even measure it. So the few cycles we spent before to calculate the checksum in v6, we spent to wait for memory. Check-summing data that we have to fetch from memory, that is very, very expensive. So in the forwarding case, we usually don't touch the payload of a packet, right? So fetching that data just to check-sum it is expensive because, as I said, the memory is slow. Some platforms have optimized check-summing implementations. I find the list of architectures here kind of interesting. I3 to C, Spark64 and SH have a simpler implementations and alpha, HPPA, HPF64, the M68 case, the M88 case, Spark and the VAX have optimized C implementations that actually might contain a simpler code. I didn't check them all. The interesting bit is that the most modern architectures you see here do not have architecture-specific checksumming implementations because they don't need them. Are the optimized ones actually faster? The I386 assembler version is certainly written for an 8386, right? The assumptions being made about that CPU are almost certainly not true for the latest generation Xeon CPU. We've encountered, in other areas, several cases where the highly optimized assembler versions were actually way, way, way slower than the generic C version because the compilers do a better job these days. And as I said, the assembler versions were written for CPU architectures from 10 to 15 years ago. So there's a task here. Who was asking me about tasks? Where are you? There's the task. Take those benchmark and figure out whether they are actually any faster than the generic C version. Thank you. The IP checksum does cover the entire header. The IP header is rather simple. You have the IP version, which is always four, always. You have the header length, the type of service. You got it. The type of service. You have the total length of the IP packet. The other length is just the header length, and that is pretty much a constant these days. Pretty much. You have the IP ID and the IP offset for fragments. You have the TTL, which each and every router on the way has to decrement, and that is being part of the checksum. You have the protocol, that's just the number indicating whether it's TCP, UDP or something else. You have the actual checksum, and of course the source and destination addresses. So since every router has to decrement the TTL, it has to update the checksum, right? The checksum itself is part of the checksum. Because of the way the algorithm works, you have to set the checksum field to zero, run the algorithm over it, and the value that comes out of that, you put into that field. If you want to verify the checksum, you run the very same algorithm over the entire IP header again, with the checksum in place, and if the result is zero, you're golden. If it's not zero, the checksum doesn't match. So as I keep repeating, the IP header is in cache. That's almost guaranteed, because we're talking about the network just forwarding this, right? Recalculation is free, so don't worry. Offloading the IP checksum calculation to the other interface card doesn't make a difference anymore. I did this like 12 or 13 years ago, and there we saw a gain of something like 7 to 8%, today, not measurable. So the implementation in OpenBSD, the basic flow of a packet that we are forwarding, is IP input calling IP forward, calling IP output. IP input is quite a complicated beast. For most, it takes the decision whether the packet is local to be forwarded or not deliverable, and it does call pf test, which is the entry point to pf. If it's a packet to be delivered locally, it's just being handed off to the upper layers, like TCP and UDP. If it's to be forwarded, well, we're calling IP forward. All the books tell you that IP forward does the routing. That is actually a lie. The route lookup and everything else happens in IP input and is being passed on to IP forward. All that IP forward does is, well, decrement the Tdl, and I think that's on the next slide. No, it's not. Well, it deals with the ICMP error generation. If we figure out we cannot forward the packet, we have to generate an ICMP error in most cases, right? And the ICMP error quotes the original packet, or at least the tiny little bit of it. I'll get into the details on that later. So IP forward keeps a copy of a fraction of the packet that's forwarding, calls IP output, and if IP output fails, it does the IP error generation. It doesn't really do any routing. IP output is the most complicated of these functions. In the forwarding case, it's actually trivial. The mass of code in there deals with locally generated packets that do not have the IP header and TCP stuff filled in yet, or these not completely. IP output once again calls P of test for the outbound check. And right after P of test, IP output recalculates the IP checks unconditionally. Once to cover the decreased time to live field, of course, but there's also the possibility that PF changes, right, like in every data translation. Locally generated packet fitting IP output at that point do not have a full checksum yet. The checksum handling in PF. We used to update the IP checksum in PF for each and every change we made. Of course, with IP output calling the IP checksum algorithm right after PF, this was utterly and completely useless. Well, not completely, because there's the bridge. And the bridge, operating in layer two, of course, does not decrement the TTL in the IP header and doesn't make any changes to the IP packet whatsoever. So the bridge does not really need to update the IP checksum. But that doesn't lead to the bridge being special case all over the stack. There's second reason, but this is the worst one. So to solve that and to stop PF from having to fiddle with the checksum all the time, the solution, of course, is to make the bridge behave like a regular output path, right? And that's what I did in the end. Well, wait a second, there's checksum offloading. Pretty much each and every network interface card that has been made in this century can do it. Most that have been made in the last decade of the last century can do it as well. Our stack had been modified ages ago to use it. This probably came from previous, but I don't really know. As I said, that's been done ages ago. It basically works like this. We delay checksumming until we know the outbound interface. Once we know the outbound interface, we can check this interface flags for offloading capabilities. If it does have offloading capabilities, we just do the little magic to tell the chip, please update the checksum, which obviously is driver dependent, but there's a generic flag in the stack indicating this needs checksumming and move on. If we do not have the offloading capabilities in the output path, we used to calculate this in software right there. On the inbound side, the hardware, respectively, the driver just sets a flag in the MF packet header indicating the checksum is good or the checksum is bad. Our verification routines, well, we now have them before it was the Bill Joy disease of I know the internals, I fiddle directly with it all over the place and I don't need APIs. Now the routines, don't get me started. The verification routines look for those flag. If it does have the good flag, well, we're done, the checksum is fine, move on. If it does have the bad flag, well, we're done, just drop the packet. If it doesn't have any resort to software. Surprise, the bridge comes in the way again. The biggest problem I ran into with the bridge, and this is the second reason why I eventually was not able to remove all the special casing from the stack, but only half of it, is that the bridge has a gross hack for broadcast. It basically works like that. The packet comes in, it already has a pointer to the outbound interface, like the bridge decided on it's that outbound interface for broadcast that is a kind of random decision. So we look at the offloading capability flags of that kind of randomly chosen interface. If it happens to have offloading capabilities, we would not do this in software, right? So, well, as said, gross hack, the bridge just copies the packet over and over and over and sends it out to all interfaces. But this is after we've looked at those flags. So it would go out without a checksum on all interfaces that do not have offloading capabilities. Not all that good. If there's anybody who wants to do me a favor, remove that gross shortcut. It shouldn't be too hard, but I was in the middle of this mess and I didn't meet that little mess on top. Task. So the IP checksum status. All the IP checksum handling in PF, I was able to remove after I fixed the bridge. As expected, as already mentioned, I'm completely unable to measure any performance benefit on modern hardware because, as I said, it's basically free. But this considerably simplified the code and quite frankly the IP checksum handling in PF was just wrong. What happened there is 10, 12 years ago when we started hacking on PF, the PF group, including myself, considered, we considered us to be PF hackers. So we did everything in PF, but we didn't really look at the network stack around it. And well, that changed, fortunately. And we now consider, we now consider PF to be an integral part of the network stack and we don't necessarily do everything in PF just because we're PF hackers, but we look at the whole picture, which is what we should have done 10 years ago. So now let's go to the protocol checksums, which are much more problematic. As I mentioned, they cover only a part of the IP header. Why? I don't know. It doesn't really make sense to me, but that's what the standard says. They do cover the entire protocol header, of course, because the IP header checksum does not cover it and you want everything in the packet to be covered, right? And this is critical. They do cover the entire payload. So that makes it potentially very, very expensive to, well, relatively spoke, very expensive to recalculate the checksum. Of course, as I keep mentioning, the payload is almost certainly not in cash for a forward packet. The handling we had on PF for the protocol checksums, when we were doing that, when we changed ports or the IP header as a server thing, we obviously have to update the checksum as well. So we did exactly the same thing, update the checksum on the fly. And that's how that looks. And this makes me want to vomit. It's nested calls of PF checksum fuck up all over the place. Please note that the outmost call is different from all the inner inside calls. Like this is really easy to screw up. It's ugly as hell and it... I'll skip that. We do have to call PF checksum fuck up or sorry, fix up per word we touch in the packet for each and every single word. And PF checksum fix up needs the old word and the new word to calculate the delta and apply that. So it needs to see old word, new word and the checksum. All the BSD derived network stacks have so-called protocol control blocks to track connections and keep state in TCP and UDP. UDP is connectionless so it does not really have state, right? No wrong. In the state, of course, it has state. In OpenBSD and I think in the other BSDs as well, the PCBs are being looked up using hash tables. And in OpenBSD, that's a separate talk that I gave two years ago, the PCBs can be linked to the PF state. So if you're on your firewall and you already looked up the PF state, you just follow the pointer to the PCB instead of doing a new look up. What did that bring us? Something like 8, 9% or so? 7 to 8% or something. Well, that was not the big one. 7 to 8% was something that simple is nice. So whenever a socket is opened, there's a template PCB being created. I want to cover the bound listening socket here. As in you call it bind and listen because that's the path that is easiest to follow and to understand. Once you bind, you know the local IP address and the local part, right? So what did they do back then? Let's check some of the source IP address and the source part and install that partial. Check some in the template PCB. Now we're getting a connection in and we're calling accept. That leads to well in your socket obviously and we'll cover the copy the template PCB to a PCB for that new connection. We add the other sites IP and port information and update the checksum in the PCB for those fields we just added. This is called the pseudo header checksum. The reply packets like the packets coming in from the other side, we just have to verify right, but the reply packets that we send out, we obviously have to check some. So we take that partial checksum from the PCB into the packet, send this all the way down to the stack with the needs checksumming flag and somewhere way down there, we'll update the checksum. Actually update is not entirely true here, but I'll come back to that. We'll fix the checksum to cover the entire payload as well. So outbound in theory. Pretty late in the outbound path. We just look at the checksum flags in the protocol in the in the packet header. We compare that to the interface capabilities. So there's basically three cases. The packet says I need checksumming and the interface says I can do it. Well, fine. Just move on. It'll do it for us. There's the case where the packet says I need checksumming and the interface is not capable of doing this. Well, we need to do this in software. This used to be somewhere in the middle of the stack. I moved this as much down as possible. So we're basically operating in a model where we assume that we always have checksum offloading. And if not, we emulate the offload engine in software. If the packet says it doesn't need checksumming, we obviously don't need to do anything. It can go for beer. The partial checksum and pseudo header checksum from the PCB that gets copied into the actual packet. That's a hack that might have helped on HP 300. Today, it is actually counterproductive. It complicates things a lot. But unfortunately, some of the hardware vendors eternalized this hack in hardware. And unfortunately, this is mostly Intel and Broadcom. Should I say that real tech got this right? Real tech beats Intel and Broadcom on this. Intel and Broadcom require the pseudo header IP checksum to be there and to be correct. And they just update it for the payload. All the other offloading engines don't care what's there. Recalculate the checksum completely and fill it in. That leads us to the famous redirect to local host bug. So a packet coming from the local host, obviously, needs checksumming, right? So if you redirect to local host, say you redirect all packets coming from your LAN that go to FTP, you redirect them to local host, port 8021, to your FTP proxy, right? So your FTP proxy sends the reply back to the client. Now the reply to the client, obviously, must not come from 127.001, but from fdp.oppsd.org. So PF has to rewrite the source address. When PF rewrites the source address, it obviously needs to take care of the checksum. Now that packet being locally generated has the pseudo header checksum. PF has no way to see whether that packet has a full checksum or the pseudo header checksum or some other partial checksum. So PF updates those fields and updates the checksum for it. Wait a second. The pseudo header checksum doesn't cover the entire packet. We are modifying packets that are not being covered by the checksum, but we're updating the checksum for a packet, for a field that's not covered in it. Oops. So as long as we're just dealing with the IP addresses, that kind of works, but as soon as you touch something else, it blows up and you always touch something else. The software checksumming engine doesn't care, it just recalculates the checksum anyway or calculates. There's no recalculation because it's not there yet. The same hardware doesn't care, it just computes the checksum. But the hardware relying on pseudo header checksum being there is doomed. It updates an already broken checksum, and updating a broken checksum leads to a broken checksum. The result of that is that the protocol checksum offloading is disabled on that class of hardware for at least 10 years now. And that unfortunately at least means EM, BGE and BNX. That's what? 99% of the server market? So the offloading, almost everything made in this decade, know that I'm saying this decade and not the century, has offloading support. Not necessarily for IPv6, but let's stick to the relevant topics. Unfortunately, there are quite a few silicon bugs. Like the most spectacular we ran into lately, when you, I think it was an Intel 10GE1, when you enabled protocol checksum offloading, which only has business in touching TCP and UDP packets, right? Corrupted OSPF packets. It's not even supposed to touch them. Since it covers the payload and we have the different behavior of the offloading endges, this is considerably more complicated than the IP header checksum. But we can summarize this into three cases. We don't have offloading. We have the halfway offloading where the pseudo-hider checksum already is required. It's just being updated. And we have the full offloading. So we always want to work in the new world. We always want to work under the assumption that we have offloading. Of course, that considerably simplifies the handling all over the stack. When we hit an outbound path that does not have that capability, well, we emulate that in software. So for that, we now got a function, a proper API instead of fiddling all over the place called INPrototCheckSumOut. That does all the magic. It just looks at the flags, does the dance, and calls the software engine if needed. There's the INPdelateCheckSum for IPv4. And surprise, there was no IN6 DelateCheckSum, because I don't know. I don't know why it wasn't there. My theory is that we fix the source load that doesn't make a difference anyway. We call that prototCheckSumOut functions very, very late. We certainly know on which interface we are going to send the packet out by then. In the old world, the call was earlier. In most cases, we knew the right outgoing interface already, but not in all of them, unfortunately. So now that's much easier. That means the entire rest of the stack does not have to care about checksums anymore. All it needs to do is you modify the packet or you generate the packet, or whatever you do, change the packet, you set the flag, hey, this needs checksumming, done. And foremost, this means there doesn't have to be any knowledge about offloading anywhere that was all over the place before. This also means that we can remove pretty much all checksum handling from PF itself. If PF is modifying the packet, set the flag, I need checksumming. Well, there's ICMP. Did I miss a slide? No. There's ICMP. ICMP has a checksum as well. I remodeled that entire thing after the TCP and UDP handling. There actually is no hardware that has ICMP checksum offloading. It doesn't quite make sense because ICMP packets tend to be small. And it's, as I said, free in software. Should there ever be one, it would be trivial for us to implement that, but I don't expect that to happen. But the point is, ICMP being the same as TCP and UDP makes the code much easier and the handling all over the stack much easier. And the performance is the same anyway. Woo-hoo! Awesome. Now my timer is screwed. There we go. So, as I mentioned before, ICMP error message quotes the packet they refer to. Well, they have to. So, ICMP is a code that can be used in any way, but it's not a code that can be used in any way. It's a code that can be used in any way. So, I'll say it again, that's the code that can be used in any way. So, as I mentioned before, ICMP error message quotes the packet they refer to. Well, they have to. If you think about using the client, you send the TCP packet to the destination. For some reason, the destination is unreachable. Whether that's on the first packet or somewhere on an established connection, it doesn't really matter. The router dropping that packet because it's not deliverable, sends that ICMP error back. And you as a client have to have a way to match this ICMP error to your little PCB describing that TCP session, right? Because you want to peer the TCP session down. How's this done? The ICMP error quotes at least a part of the packet that it refers to. Now there's network address translation. Well, if we are translating the packets, we unfortunately also have to translate the quoted packet inside the ICMP packet. And also, surprise, there's no offloading support for that anywhere. The quoted packet, well, that has a checksum too. In most cases, it is truncated anyway. So, there's no way to verify the checksum of the quoted packet inside the ICMP packet. So, we don't have to care. If it's not truncated, nothing in the entire real world cares. There's one, I wouldn't call this an exception because it's not really the real world. There's a tool called SCAPE that we use to generate weird, weird, weird, weird IP packets, throw at our stack, not just IP, TCP and even payload, that we throw at our stack and try, on NPF and try to make it misbehave. And then SCAPE receives those packets as well. And SCAPE is the only thing in the world that I'm aware of that actually verifies the checksum inside the quoted packet inside the ICMP error message. This would be easily fixable because you could just recalculate the checksum in the non-truncated case. I have not implemented that, but could do it. It should be reasonably simple. And the cost is basically zero because it's tiny. We're talking topmost 68 bytes. So, consequences in PF. The old way was we update the checksum. What happens when you update a broken checksum? Well, you get a broken checksum. That's good. But now, we're not updating anymore. We're replacing the checksum. We are recalculating it. Now, we have a packet coming in with a broken checksum. We do some adjustments and we recalculate the checksum. Hey, it's not broken anymore. We just fixed it up. Oops, that's not good. We are hiding data corruption. So, before we do that, we unfortunately have to verify the existing checksum. On anything that's halfway sane, that's free because the offloading engines already tell us whether it was the checksum was correct or not. But if there's no offloading, that's potentially expensive because the payload, once again, is not in cash. Nonetheless, the new code is so much nicer and cleaner and there's so much more that we can clean up now. The redirect to local host bug is gone because we are doing the checksum much later anyway, so we don't have to care about the difference between pseudo-hider checksum and not. This eventually allows us to enable the offload engines on all these chips that require pseudo-hider checksums. Well, EM, BGE and BNX are pretty much 99% of the server market. Unfortunately, there is one case that suffers considerably. That is doing that when you do not have any offloading capabilities. Now, let's put this into perspective. If you're doing anything else but pure packet forwarding, the total cost of packet forwarding is basically nothing. If you run a proxy in userland, try to measure the cost of your packet handling in the kernel. It doesn't matter. It's barely measurable. If you're running IPsec, well, we don't even get to start on that. So the packet forwarding cost is tiny, so even if it becomes a tiny little bit more expensive, this would not really hurt us. If you're running your netting gateway on something without offloading and care about performance, sorry, you're doing something wrong. What we could do to fix that problem is calculate the checksum over the header before and after pf and apply the delta. Well, that means we cannot use any offloading engines because no offloading engine has support for verifying a part of the checksum and then apply a delta. So it's all software, which also means that we hurt the offloading case considerably. Given that pretty much everything has offloading, I rate it optimized for the offloading case than for the non-offloading case. And the old architectures that tend to not have any offloading capabilities don't suffer as much because, relatively speaking, the memory is fast there. In microbenchmarking, in that case, on a very, very modern AMD64 system like the latest generation Xeon, with, I think, we used EMs and disabled the offloading in the driver. This is a completely made up case that's not realistic at all. We are losing something between 5 and 10 percent depending on the traffic pattern. This depends a lot on the specific hardware implementation on the machine because this depends on caches, on the cache architectures. Is the layer too cache-shared between the CPU cores or not? It's stuff like that. It depends a lot on the buses. PCI is much worse than PCIe. And as said, that class of machines pretty certainly has offloading capabilities. So in my book, this is acceptable. If you have such a machine, let me find a nickel. Buy some of your Apple cards. So that's benchmark for the better use of the offloading engines we make now. On that mentioned machine, current generation Xeon, and pure forwarding, I cannot measure any difference. By the raw computing power that the current Xeon has, they are so fast, they have so much raw computing power. The caches are big and fast that it just doesn't make a difference anymore because our limiting factors are other factors. The limiting factors are latency, foremost latency. At Brandworth, we have plenty usually. It's latency to cache, to memory, to the buses, foremost on the buses. And we're not even talking about the latency outside the machine yet. So in microbenchmarking, even if there is an effect, it's completely hidden. This might be different if you're actually doing something else on the machine because you do save a few cycles. I don't think that really matters, but some people might care. On an older system, a Pentium M using an Intel GigabitNIC, we are seeing about a 7% increase. I keep coming back to the 7% when I do the checksum. That's the magic number there. But with that, with a much, much, much cleaner and much better structured code, we pave the road for future improvements. There's so much clean up possible afterwards. Last but not least, this is just less code, and this code is nicely contained in one file, in one spot, and not spread all over the stack. There's a nice and simple API to call instead of fiddling with the internals or the entire stack. We can actually remove that entire absolute header checksum dance in the PCVs. What's the point in pre-calculating that? There is no performance benefit at all. So just remove it and do the entire checksum late in the output path in those new IN and IN6 proto checksum out functions. And we only actually need to do that in the offloading case because the other two cases, the software engine does full recalculation anyway, and the same hardware does full recalculation anyway. We can unfortunately not distinguish the full offloading from the needs the pseudo checksum offloading. We could modify the drivers to signal that up, but there's no point since calculating that is so cheap that checking the flag is almost, well, checking the flag is still somewhat cheaper, but it just doesn't make a difference. The bridge special casing all over the stack needs to die. The biggest reason for the bridge special casing is gone with this. There's just that broadcast tag remaining. Yes. Is that how you get on the left towards the bar and the right to the bottom? No, that picture is actually from Warsaw. Oh, it looks familiar. So we briefly touched the surface here. This is actually complicated. I would have lost a lot of hair over it if she hadn't been faster. And we have to give acknowledgement to Christian Weisberger a lot that's Natty at OpenBSD, give him a lot of acknowledgement for this, because he was the first one to actually understand that redirect to local host bug. And I very, very remember him and us sitting there and him explaining this to us more than once. And eventually we understood what was going on as well. And just a couple of years later I started working on that. He also helped a lot with testing when I finally had something working. There was a lot of help and support from Mike Billow-Pouhauf. I think I got this right. That's Mike B. And Peter Hasler. They did a lot of the testing for me last time, please, because they have access to an Xeon. I don't. The IPv6 delay check something was written by Kenneth. That's a shithole. Finding the TCP header in IP is that simple. That's a fixed offset, basically. Finding the TCP header in IPv6. The function doesn't fit one screen. You have to loop over the chain of headers and hope that nothing goes wrong. So I'm working on this since late 2010. There's been a lot of hackathons that I used for this. There's been the Iceland one organized by the we've been in admin more than once organized by Theo, Bob and Theo on crew. We've been to Ljubljana in Slovenia. We've been in Budapest in Germany, Portugal. We just have been to New Zealand, which was awesome. And I really want to thank all the people that make this possible. The other occasion where I keep making progress on this are the other conferences. So there's EuroBSDCon commercial break. We are going to have EuroBSDCon in Malta in September. And Malta is a really nice island. So you do want to come. The call for papers is still open for another, I think nine days or so. Yes. So we want speakers. We already have plenty of submissions, but more is better. Bigger is better. We also want you as visitors, of course. So just tell your wife, girlfriend, whatever, let's go on vacation to Malta. Oh, by the way, there's a conference. I know the answer. Do you have a wife and how do you explain to her that you're sitting in a dirty garden? Isn't it bad enough that I have to explain your wife that you did last night? I'm thinking if she's in Malta, she doesn't care about women all day. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. Point. Are there any questions? Oh, I forgot another commercial break. I so need a jingle for this. I need a jingle stopped in the middle of the talk commercial break. So if you want to know more about the PF, there's the awesome book of PF written by Peter. You carry two copies, right? So I deeply suggest you get a copy. It's an awesome book. There's the almost brand new, brand new absolute open. It's the second edition by Michael Lucas. I've been the tech reviewer on both of these books. So opposed to many of the other authors who just write something, publish it, and then figure out that they got it all wrong. These are technically pretty as close to correct as it gets. Until we change it again. Until we change it, of course. Yes. I think you find Michael is going to sell it to you. You even get a second commercial for free. Enough. I know you benchmark. I know I've just made sure you do as well. You know, when I look at old stuff, can get hurt. The old stuff actually does not really get hurt because it doesn't the difference between the CPU speed and the memory access isn't all that big there. So the hit is not remotely as big as on new stuff where you forcefully disable the offloading. Have we looked at the kind of middle generation of hardware and so like what I'm saying is whatever happens to HP 300 in Spark, I kind of don't care. My concern is the Socracy and the Alex and the small embedded IPv6 that appeared a lot, people have to avoid it. With those generations of next that kind of suck. Everything you just mentioned has offloading that works. And it does work. Well, it's enabled because it's the cheap chips that implement the proper full offloading. It's not the server chips that eternalize the hack. So is it really worth doing hardware offloading? Never. For performance, not really anymore. So obviously you can't fully answer this question. Why are servers ramming this down my throat as the best thing since sliced bread? Because it's an item in the marketing checklist. Why is Microsoft claiming, okay, never mind. Because what servers will you buy that say support checks them offload or the ones that don't have a checkmark and not bought? Well, I can tell you how I order servers. I call our suppliers saying I need five more calls over. And when you pick nix to put in them, since it doesn't matter. But you don't. Exactly. You don't get to pick. It's on board anyway. Any more questions? Hey, that's a new one. Was that this year? No. My neighborhood is kind of crazy. It is. So my neighborhood declared autonomy from the surrounding city a couple of years ago. Of course, this is just, this is being a joke, right? But this kind of describes the feeling in the area. Everything works like a tiny, tiny, tiny village. Everybody knows everybody, but we're surrounded by the second biggest city in Germany and we're right in the middle of it. So once a year, we're having a big street fest. And for some reason that we don't understand the city of Hamburg things, we need to get a permit for this. Well, we don't think so. We're doing this for more than 25 years now. So each and every year the fest is a little bit bigger than the year before to give you an idea. It's more than 10,000 people throughout the day. And over the years, the, recently it got a little bit better. Like five years ago when it was first, the city of Hamburg sent us 4,500 policemen, seven water cannons, one tank. So what happens is they come in around 10 p.m. They get everybody wet, block the streets for two hours, then they piss off again and the party goes on. Works for me? No more questions. Of course the neighborhood is awesome. Okay. You can kick robocops out of your backyard. It's fucking amazing. We keep calling the riot control police guys robocops because that's how they look. And one year I look out of my rear window and there's two of them standing in my garden. Wait a second. So I open the door. I don't think you want to be in my garden. The house next door has been spotted. The house next door used to be an old abandoned building that nobody ever shit about. So it has been declared spotted each and every year. And at the evening everybody went home and well, case closed. But that year for some reason they thought they had to do something about it. Sent 200 policemen in, kicked everybody out. Then noticed, wait a second, we did not talk to the owner of this building. So they desperately tried to reach him, couldn't then realize shit. We don't have any right to be in this building at all. We have to go again and the people who spotted it came back. Two of the robocops suddenly standing in my backyard. So I'm stepping out. You don't want to be here. They keep saying, you need to secure the scene. Okay, whatever. Not in my backyard. But you can jump over the little fence style thingy. But yeah, there's a bush behind it. It's going to be too hurt. Not my problem. They did. Then I put up a sign stepping on the lawn as a private and for people wearing uniforms. Legally binding. Of course. Of course. Anybody has a technical question? Anybody has beer for me? No. Okay. Then please buy the books. Did I forget any commercials? Buy the books. Come to Europe ESD con. Come to Asia PSD con. Buy hosting from a company. Anything else? Buy open BSD CDs. Donate to the foundation. You have to come to the slide and say which one is the company. You need to change your t-shirt. So just list like Queensland is called. I don't know. So you have to tell. I'm not smart enough for that. Okay. Okay.
 Welcome. I'm Henning. I'm saving you from the who am I part?  No?  Well, I drink beer.  I hack an openBSD.  You might have come across my name if you're using pf or openbgpd or ntpd or...  What else did I do?  Female.  Pardon?  Female, yes.  Oh, whatever. It's been a long time.  Before I actually start my talk, I do have a question.  You know, we had this earthquake yesterday, and there's been this thing,  BSD conference plus earthquake. I've been into that before, and so I have just one question.  Is there a nuclear power plant near here?  Okay, then I can go on.  I want to talk about the checksum in TCP IP.  We're mostly talking about two sets of checksums here.  One is the IP checksum, and the other is the so-called protocol checksum that mostly is TCP and UDP.  IPv6 does not have the IP header checksum one.  I'll explain later why that doesn't make a difference at all.  The actual checksumming algorithm is really, really, really simple.  It's really just the lowest word of the once-compliment sum of all the words in the either IP header or almost entire packet.  We'll come back to that later.  The IP checksum just covers the IP header while the protocol checksum covers a part of the IP header,  the actual protocol header, and the entire payload.  So there's performance considerations in that.  Me being mostly involved with firewalls and routers and everything, I care about forwarded packets,  not locally generated ones all that much.  The actual math for the checksum is dirt, dirt, dirt cheap.  I spent many, many, many years profiling, and you cannot even find the checksumming function anywhere noticeably up in the call graphs.  So the calculation can be considered completely free.  We're just talking about the math here.  That's basically talking about the CPU's integer units, right?  The limiting factor that we run into is latency and bandwidth to memory.  So we care about caches, RAM, and buses.  This is mostly relevant for modern architectures.  On the really old architectures, it doesn't really make a difference because their memory, relatively to the CPU, is fast.  On modern systems, memory is comparably dog slow because the CPU's also got them fast.  Check-summing data that already is in the cache is essentially free because the cache is fast.  So in the forwarding case, when we are dealing with the IP pack and we are forwarding, the IP header is in cache.  Of course, we need that information for forwarding, right?  So IP header checksum free.  IPv6 not having that checksum, which they often point out as a performance improvement, doesn't make a difference.  You can't even measure it.  So the few cycles we spent before to calculate the checksum in v6, we spent to wait for memory.  Check-summing data that we have to fetch from memory, that is very, very expensive.  So in the forwarding case, we usually don't touch the payload of a packet, right?  So fetching that data just to check-sum it is expensive because, as I said, the memory is slow.  Some platforms have optimized check-summing implementations.  I find the list of architectures here kind of interesting.  I3 to C, Spark64 and SH have a simpler implementations and alpha, HPPA, HPF64, the M68 case, the M88 case, Spark and the VAX have optimized C implementations that actually might contain a simpler code.  I didn't check them all.  The interesting bit is that the most modern architectures you see here do not have architecture-specific checksumming implementations because they don't need them.  Are the optimized ones actually faster?  The I386 assembler version is certainly written for an 8386, right?  The assumptions being made about that CPU are almost certainly not true for the latest generation Xeon CPU.  We've encountered, in other areas, several cases where the highly optimized assembler versions were actually way, way, way slower than the generic C version because the compilers do a better job these days.  And as I said, the assembler versions were written for CPU architectures from 10 to 15 years ago.  So there's a task here.  Who was asking me about tasks?  Where are you?  There's the task.  Take those benchmark and figure out whether they are actually any faster than the generic C version.  Thank you.  The IP checksum does cover the entire header.  The IP header is rather simple.  You have the IP version, which is always four, always.  You have the header length, the type of service.  You got it.  The type of service.  You have the total length of the IP packet.  The other length is just the header length, and that is pretty much a constant these days.  Pretty much.  You have the IP ID and the IP offset for fragments.  You have the TTL, which each and every router on the way has to decrement, and that is being part of the checksum.  You have the protocol, that's just the number indicating whether it's TCP, UDP or something else.  You have the actual checksum, and of course the source and destination addresses.  So since every router has to decrement the TTL, it has to update the checksum, right?  The checksum itself is part of the checksum.  Because of the way the algorithm works, you have to set the checksum field to zero, run the algorithm over it,  and the value that comes out of that, you put into that field.  If you want to verify the checksum, you run the very same algorithm over the entire IP header again, with the checksum in place,  and if the result is zero, you're golden.  If it's not zero, the checksum doesn't match.  So as I keep repeating, the IP header is in cache.  That's almost guaranteed, because we're talking about the network just forwarding this, right?  Recalculation is free, so don't worry.  Offloading the IP checksum calculation to the other interface card doesn't make a difference anymore.  I did this like 12 or 13 years ago, and there we saw a gain of something like 7 to 8%, today, not measurable.  So the implementation in OpenBSD, the basic flow of a packet that we are forwarding, is IP input calling IP forward, calling IP output.  IP input is quite a complicated beast.  For most, it takes the decision whether the packet is local to be forwarded or not deliverable,  and it does call pf test, which is the entry point to pf.  If it's a packet to be delivered locally, it's just being handed off to the upper layers, like TCP and UDP.  If it's to be forwarded, well, we're calling IP forward.  All the books tell you that IP forward does the routing.  That is actually a lie.  The route lookup and everything else happens in IP input and is being passed on to IP forward.  All that IP forward does is, well, decrement the Tdl, and I think that's on the next slide.  No, it's not.  Well, it deals with the ICMP error generation.  If we figure out we cannot forward the packet, we have to generate an ICMP error in most cases, right?  And the ICMP error quotes the original packet, or at least the tiny little bit of it.  I'll get into the details on that later.  So IP forward keeps a copy of a fraction of the packet that's forwarding, calls IP output, and if IP output fails, it does the IP error generation.  It doesn't really do any routing.  IP output is the most complicated of these functions.  In the forwarding case, it's actually trivial.  The mass of code in there deals with locally generated packets that do not have the IP header and TCP stuff filled in yet, or these not completely.  IP output once again calls P of test for the outbound check.  And right after P of test, IP output recalculates the IP checks unconditionally.  Once to cover the decreased time to live field, of course, but there's also the possibility that PF changes, right, like in every data translation.  Locally generated packet fitting IP output at that point do not have a full checksum yet.  The checksum handling in PF.  We used to update the IP checksum in PF for each and every change we made.  Of course, with IP output calling the IP checksum algorithm right after PF, this was utterly and completely useless.  Well, not completely, because there's the bridge.  And the bridge, operating in layer two, of course, does not decrement the TTL in the IP header and doesn't make any changes to the IP packet whatsoever.  So the bridge does not really need to update the IP checksum.  But that doesn't lead to the bridge being special case all over the stack.  There's second reason, but this is the worst one.  So to solve that and to stop PF from having to fiddle with the checksum all the time, the solution, of course, is to make the bridge behave like a regular output path, right?  And that's what I did in the end.  Well, wait a second, there's checksum offloading.  Pretty much each and every network interface card that has been made in this century can do it.  Most that have been made in the last decade of the last century can do it as well.  Our stack had been modified ages ago to use it.  This probably came from previous, but I don't really know.  As I said, that's been done ages ago.  It basically works like this.  We delay checksumming until we know the outbound interface.  Once we know the outbound interface, we can check this interface flags for offloading capabilities.  If it does have offloading capabilities, we just do the little magic to tell the chip, please update the checksum, which obviously is driver dependent, but there's a generic flag in the stack indicating this needs checksumming and move on.  If we do not have the offloading capabilities in the output path, we used to calculate this in software right there.  On the inbound side, the hardware, respectively, the driver just sets a flag in the MF packet header indicating the checksum is good or the checksum is bad.  Our verification routines, well, we now have them before it was the Bill Joy disease of I know the internals, I fiddle directly with it all over the place and I don't need APIs.  Now the routines, don't get me started.  The verification routines look for those flag.  If it does have the good flag, well, we're done, the checksum is fine, move on.  If it does have the bad flag, well, we're done, just drop the packet.  If it doesn't have any resort to software.  Surprise, the bridge comes in the way again.  The biggest problem I ran into with the bridge, and this is the second reason why I eventually was not able to remove all the special casing from the stack, but only half of it, is that the bridge has a gross hack for broadcast.  It basically works like that.  The packet comes in, it already has a pointer to the outbound interface, like the bridge decided on it's that outbound interface for broadcast that is a kind of random decision.  So we look at the offloading capability flags of that kind of randomly chosen interface.  If it happens to have offloading capabilities, we would not do this in software, right?  So, well, as said, gross hack, the bridge just copies the packet over and over and over and sends it out to all interfaces.  But this is after we've looked at those flags.  So it would go out without a checksum on all interfaces that do not have offloading capabilities.  Not all that good.  If there's anybody who wants to do me a favor, remove that gross shortcut.  It shouldn't be too hard, but I was in the middle of this mess and I didn't meet that little mess on top.  Task.  So the IP checksum status.  All the IP checksum handling in PF, I was able to remove after I fixed the bridge.  As expected, as already mentioned, I'm completely unable to measure any performance benefit on modern hardware because, as I said, it's basically free.  But this considerably simplified the code and quite frankly the IP checksum handling in PF was just wrong.  What happened there is 10, 12 years ago when we started hacking on PF, the PF group, including myself, considered, we considered us to be PF hackers.  So we did everything in PF, but we didn't really look at the network stack around it.  And well, that changed, fortunately.  And we now consider, we now consider PF to be an integral part of the network stack and we don't necessarily do everything in PF just because we're PF hackers, but we look at the whole picture, which is what we should have done 10 years ago.  So now let's go to the protocol checksums, which are much more problematic.  As I mentioned, they cover only a part of the IP header. Why? I don't know. It doesn't really make sense to me, but that's what the standard says.  They do cover the entire protocol header, of course, because the IP header checksum does not cover it and you want everything in the packet to be covered, right?  And this is critical. They do cover the entire payload.  So that makes it potentially very, very expensive to, well, relatively spoke, very expensive to recalculate the checksum.  Of course, as I keep mentioning, the payload is almost certainly not in cash for a forward packet.  The handling we had on PF for the protocol checksums, when we were doing that, when we changed ports or the IP header as a server thing, we obviously have to update the checksum as well.  So we did exactly the same thing, update the checksum on the fly.  And that's how that looks. And this makes me want to vomit.  It's nested calls of PF checksum fuck up all over the place. Please note that the outmost call is different from all the inner inside calls.  Like this is really easy to screw up. It's ugly as hell and it...  I'll skip that.  We do have to call PF checksum fuck up or sorry, fix up per word we touch in the packet for each and every single word.  And PF checksum fix up needs the old word and the new word to calculate the delta and apply that.  So it needs to see old word, new word and the checksum.  All the BSD derived network stacks have so-called protocol control blocks to track connections and keep state in TCP and UDP.  UDP is connectionless so it does not really have state, right? No wrong.  In the state, of course, it has state.  In OpenBSD and I think in the other BSDs as well, the PCBs are being looked up using hash tables.  And in OpenBSD, that's a separate talk that I gave two years ago, the PCBs can be linked to the PF state.  So if you're on your firewall and you already looked up the PF state, you just follow the pointer to the PCB instead of doing a new look up.  What did that bring us? Something like 8, 9% or so?  7 to 8% or something.  Well, that was not the big one. 7 to 8% was something that simple is nice.  So whenever a socket is opened, there's a template PCB being created.  I want to cover the bound listening socket here.  As in you call it bind and listen because that's the path that is easiest to follow and to understand.  Once you bind, you know the local IP address and the local part, right?  So what did they do back then?  Let's check some of the source IP address and the source part and install that partial.  Check some in the template PCB.  Now we're getting a connection in and we're calling accept.  That leads to well in your socket obviously and we'll cover the copy the template PCB to a PCB for that new connection.  We add the other sites IP and port information and update the checksum in the PCB for those fields we just added.  This is called the pseudo header checksum.  The reply packets like the packets coming in from the other side, we just have to verify right, but the reply packets that we send out, we obviously have to check some.  So we take that partial checksum from the PCB into the packet, send this all the way down to the stack with the needs checksumming flag and somewhere way down there, we'll update the checksum.  Actually update is not entirely true here, but I'll come back to that.  We'll fix the checksum to cover the entire payload as well.  So outbound in theory.  Pretty late in the outbound path.  We just look at the checksum flags in the protocol in the in the packet header.  We compare that to the interface capabilities.  So there's basically three cases.  The packet says I need checksumming and the interface says I can do it.  Well, fine.  Just move on.  It'll do it for us.  There's the case where the packet says I need checksumming and the interface is not capable of doing this.  Well, we need to do this in software.  This used to be somewhere in the middle of the stack.  I moved this as much down as possible.  So we're basically operating in a model where we assume that we always have checksum offloading.  And if not, we emulate the offload engine in software.  If the packet says it doesn't need checksumming, we obviously don't need to do anything.  It can go for beer.  The partial checksum and pseudo header checksum from the PCB that gets copied into the actual packet.  That's a hack that might have helped on HP 300.  Today, it is actually counterproductive.  It complicates things a lot.  But unfortunately, some of the hardware vendors eternalized this hack in hardware.  And unfortunately, this is mostly Intel and Broadcom.  Should I say that real tech got this right?  Real tech beats Intel and Broadcom on this.  Intel and Broadcom require the pseudo header IP checksum to be there and to be correct.  And they just update it for the payload.  All the other offloading engines don't care what's there.  Recalculate the checksum completely and fill it in.  That leads us to the famous redirect to local host bug.  So a packet coming from the local host, obviously, needs checksumming, right?  So if you redirect to local host, say you redirect all packets coming from your LAN that go to FTP,  you redirect them to local host, port 8021, to your FTP proxy, right?  So your FTP proxy sends the reply back to the client.  Now the reply to the client, obviously, must not come from 127.001, but from fdp.oppsd.org.  So PF has to rewrite the source address.  When PF rewrites the source address, it obviously needs to take care of the checksum.  Now that packet being locally generated has the pseudo header checksum.  PF has no way to see whether that packet has a full checksum or the pseudo header checksum or some other partial checksum.  So PF updates those fields and updates the checksum for it.  Wait a second.  The pseudo header checksum doesn't cover the entire packet.  We are modifying packets that are not being covered by the checksum, but we're updating the checksum for a packet, for a field that's not covered in it.  Oops.  So as long as we're just dealing with the IP addresses, that kind of works, but as soon as you touch something else, it blows up and you always touch something else.  The software checksumming engine doesn't care, it just recalculates the checksum anyway or calculates.  There's no recalculation because it's not there yet.  The same hardware doesn't care, it just computes the checksum.  But the hardware relying on pseudo header checksum being there is doomed.  It updates an already broken checksum, and updating a broken checksum leads to a broken checksum.  The result of that is that the protocol checksum offloading is disabled on that class of hardware for at least 10 years now.  And that unfortunately at least means EM, BGE and BNX.  That's what? 99% of the server market?  So the offloading, almost everything made in this decade, know that I'm saying this decade and not the century, has offloading support.  Not necessarily for IPv6, but let's stick to the relevant topics.  Unfortunately, there are quite a few silicon bugs.  Like the most spectacular we ran into lately, when you, I think it was an Intel 10GE1,  when you enabled protocol checksum offloading, which only has business in touching TCP and UDP packets, right?  Corrupted OSPF packets.  It's not even supposed to touch them.  Since it covers the payload and we have the different behavior of the offloading endges, this is considerably more complicated than the IP header checksum.  But we can summarize this into three cases.  We don't have offloading.  We have the halfway offloading where the pseudo-hider checksum already is required.  It's just being updated.  And we have the full offloading.  So we always want to work in the new world.  We always want to work under the assumption that we have offloading.  Of course, that considerably simplifies the handling all over the stack.  When we hit an outbound path that does not have that capability, well, we emulate that in software.  So for that, we now got a function, a proper API instead of fiddling all over the place called INPrototCheckSumOut.  That does all the magic.  It just looks at the flags, does the dance, and calls the software engine if needed.  There's the INPdelateCheckSum for IPv4.  And surprise, there was no IN6 DelateCheckSum, because I don't know.  I don't know why it wasn't there.  My theory is that we fix the source load that doesn't make a difference anyway.  We call that prototCheckSumOut functions very, very late.  We certainly know on which interface we are going to send the packet out by then.  In the old world, the call was earlier.  In most cases, we knew the right outgoing interface already, but not in all of them, unfortunately.  So now that's much easier.  That means the entire rest of the stack does not have to care about checksums anymore.  All it needs to do is you modify the packet or you generate the packet,  or whatever you do, change the packet, you set the flag, hey, this needs checksumming, done.  And foremost, this means there doesn't have to be any knowledge about offloading anywhere that was all over the place before.  This also means that we can remove pretty much all checksum handling from PF itself.  If PF is modifying the packet, set the flag, I need checksumming.  Well, there's ICMP.  Did I miss a slide?  No.  There's ICMP.  ICMP has a checksum as well.  I remodeled that entire thing after the TCP and UDP handling.  There actually is no hardware that has ICMP checksum offloading.  It doesn't quite make sense because ICMP packets tend to be small.  And it's, as I said, free in software.  Should there ever be one, it would be trivial for us to implement that, but I don't expect that to happen.  But the point is, ICMP being the same as TCP and UDP makes the code much easier and the handling all over the stack much easier.  And the performance is the same anyway.  Woo-hoo!  Awesome.  Now my timer is screwed.  There we go.  So, as I mentioned before, ICMP error message quotes the packet they refer to.  Well, they have to.  So, ICMP is a code that can be used in any way,  but it's not a code that can be used in any way.  It's a code that can be used in any way.  So, I'll say it again, that's the code that can be used in any way.  So, as I mentioned before, ICMP error message quotes the packet they refer to.  Well, they have to.  If you think about using the client, you send the TCP packet to the destination.  For some reason, the destination is unreachable.  Whether that's on the first packet or somewhere on an established connection, it doesn't really matter.  The router dropping that packet because it's not deliverable, sends that ICMP error back.  And you as a client have to have a way to match this ICMP error to your little PCB  describing that TCP session, right?  Because you want to peer the TCP session down.  How's this done?  The ICMP error quotes at least a part of the packet that it refers to.  Now there's network address translation.  Well, if we are translating the packets, we unfortunately also have to translate the quoted packet inside the ICMP packet.  And also, surprise, there's no offloading support for that anywhere.  The quoted packet, well, that has a checksum too.  In most cases, it is truncated anyway.  So, there's no way to verify the checksum of the quoted packet inside the ICMP packet.  So, we don't have to care.  If it's not truncated, nothing in the entire real world cares.  There's one, I wouldn't call this an exception because it's not really the real world.  There's a tool called SCAPE that we use to generate weird, weird, weird, weird IP packets, throw at our stack,  not just IP, TCP and even payload, that we throw at our stack and try, on NPF and try to make it misbehave.  And then SCAPE receives those packets as well.  And SCAPE is the only thing in the world that I'm aware of that actually verifies the checksum inside the quoted packet inside the ICMP error message.  This would be easily fixable because you could just recalculate the checksum in the non-truncated case.  I have not implemented that, but could do it.  It should be reasonably simple.  And the cost is basically zero because it's tiny.  We're talking topmost 68 bytes.  So, consequences in PF.  The old way was we update the checksum.  What happens when you update a broken checksum?  Well, you get a broken checksum.  That's good.  But now, we're not updating anymore.  We're replacing the checksum.  We are recalculating it.  Now, we have a packet coming in with a broken checksum.  We do some adjustments and we recalculate the checksum.  Hey, it's not broken anymore.  We just fixed it up.  Oops, that's not good.  We are hiding data corruption.  So, before we do that, we unfortunately have to verify the existing checksum.  On anything that's halfway sane, that's free because the offloading engines already tell us whether it was the checksum was correct or not.  But if there's no offloading, that's potentially expensive because the payload, once again, is not in cash.  Nonetheless, the new code is so much nicer and cleaner and there's so much more that we can clean up now.  The redirect to local host bug is gone because we are doing the checksum much later anyway, so we don't have to care about the difference between pseudo-hider checksum and not.  This eventually allows us to enable the offload engines on all these chips that require pseudo-hider checksums.  Well, EM, BGE and BNX are pretty much 99% of the server market.  Unfortunately, there is one case that suffers considerably.  That is doing that when you do not have any offloading capabilities.  Now, let's put this into perspective.  If you're doing anything else but pure packet forwarding, the total cost of packet forwarding is basically nothing.  If you run a proxy in userland, try to measure the cost of your packet handling in the kernel.  It doesn't matter. It's barely measurable.  If you're running IPsec, well, we don't even get to start on that.  So the packet forwarding cost is tiny, so even if it becomes a tiny little bit more expensive, this would not really hurt us.  If you're running your netting gateway on something without offloading and care about performance, sorry, you're doing something wrong.  What we could do to fix that problem is calculate the checksum over the header before and after pf and apply the delta.  Well, that means we cannot use any offloading engines because no offloading engine has support for verifying a part of the checksum and then apply a delta.  So it's all software, which also means that we hurt the offloading case considerably.  Given that pretty much everything has offloading, I rate it optimized for the offloading case than for the non-offloading case.  And the old architectures that tend to not have any offloading capabilities don't suffer as much because, relatively speaking, the memory is fast there.  In microbenchmarking, in that case, on a very, very modern AMD64 system like the latest generation Xeon,  with, I think, we used EMs and disabled the offloading in the driver.  This is a completely made up case that's not realistic at all.  We are losing something between 5 and 10 percent depending on the traffic pattern.  This depends a lot on the specific hardware implementation on the machine because this depends on caches, on the cache architectures.  Is the layer too cache-shared between the CPU cores or not? It's stuff like that.  It depends a lot on the buses. PCI is much worse than PCIe.  And as said, that class of machines pretty certainly has offloading capabilities.  So in my book, this is acceptable.  If you have such a machine, let me find a nickel. Buy some of your Apple cards.  So that's benchmark for the better use of the offloading engines we make now.  On that mentioned machine, current generation Xeon, and pure forwarding, I cannot measure any difference.  By the raw computing power that the current Xeon has, they are so fast, they have so much raw computing power.  The caches are big and fast that it just doesn't make a difference anymore because our limiting factors are other factors.  The limiting factors are latency, foremost latency.  At Brandworth, we have plenty usually. It's latency to cache, to memory, to the buses, foremost on the buses.  And we're not even talking about the latency outside the machine yet.  So in microbenchmarking, even if there is an effect, it's completely hidden.  This might be different if you're actually doing something else on the machine because you do save a few cycles.  I don't think that really matters, but some people might care.  On an older system, a Pentium M using an Intel GigabitNIC, we are seeing about a 7% increase.  I keep coming back to the 7% when I do the checksum. That's the magic number there.  But with that, with a much, much, much cleaner and much better structured code, we pave the road for future improvements.  There's so much clean up possible afterwards.  Last but not least, this is just less code, and this code is nicely contained in one file, in one spot, and not spread all over the stack.  There's a nice and simple API to call instead of fiddling with the internals or the entire stack.  We can actually remove that entire absolute header checksum dance in the PCVs.  What's the point in pre-calculating that? There is no performance benefit at all.  So just remove it and do the entire checksum late in the output path in those new IN and IN6 proto checksum out functions.  And we only actually need to do that in the offloading case because the other two cases, the software engine does full recalculation anyway,  and the same hardware does full recalculation anyway.  We can unfortunately not distinguish the full offloading from the needs the pseudo checksum offloading.  We could modify the drivers to signal that up, but there's no point since calculating that is so cheap that checking the flag is almost,  well, checking the flag is still somewhat cheaper, but it just doesn't make a difference.  The bridge special casing all over the stack needs to die.  The biggest reason for the bridge special casing is gone with this.  There's just that broadcast tag remaining.  Yes.  Is that how you get on the left towards the bar and the right to the bottom?  No, that picture is actually from Warsaw.  Oh, it looks familiar.  So we briefly touched the surface here.  This is actually complicated.  I would have lost a lot of hair over it if she hadn't been faster.  And we have to give acknowledgement to Christian Weisberger a lot that's Natty at OpenBSD, give him a lot of acknowledgement for this,  because he was the first one to actually understand that redirect to local host bug.  And I very, very remember him and us sitting there and him explaining this to us more than once.  And eventually we understood what was going on as well.  And just a couple of years later I started working on that.  He also helped a lot with testing when I finally had something working.  There was a lot of help and support from Mike Billow-Pouhauf.  I think I got this right.  That's Mike B. And Peter Hasler.  They did a lot of the testing for me last time, please, because they have access to an Xeon.  I don't.  The IPv6 delay check something was written by Kenneth.  That's a shithole.  Finding the TCP header in IP is that simple.  That's a fixed offset, basically.  Finding the TCP header in IPv6.  The function doesn't fit one screen.  You have to loop over the chain of headers and hope that nothing goes wrong.  So I'm working on this since late 2010.  There's been a lot of hackathons that I used for this.  There's been the Iceland one organized by the we've been in admin more than once organized by Theo, Bob and Theo on crew.  We've been to Ljubljana in Slovenia.  We've been in Budapest in Germany, Portugal.  We just have been to New Zealand, which was awesome.  And I really want to thank all the people that make this possible.  The other occasion where I keep making progress on this are the other conferences.  So there's EuroBSDCon commercial break.  We are going to have EuroBSDCon in Malta in September.  And Malta is a really nice island.  So you do want to come.  The call for papers is still open for another, I think nine days or so.  Yes.  So we want speakers.  We already have plenty of submissions, but more is better.  Bigger is better.  We also want you as visitors, of course.  So just tell your wife, girlfriend, whatever, let's go on vacation to Malta.  Oh, by the way, there's a conference.  I know the answer.  Do you have a wife and how do you explain to her that you're sitting in a dirty garden?  Isn't it bad enough that I have to explain your wife that you did last night?  I'm thinking if she's in Malta, she doesn't care about women all day.  I don't know.  I don't know.  I don't know.  I don't know.  I don't know.  I don't know.  I don't know.  I don't know.  Point.  Are there any questions?  Oh, I forgot another commercial break.  I so need a jingle for this.  I need a jingle stopped in the middle of the talk commercial break.  So if you want to know more about the PF, there's the awesome book of PF written by Peter.  You carry two copies, right?  So I deeply suggest you get a copy.  It's an awesome book.  There's the almost brand new, brand new absolute open.  It's the second edition by Michael Lucas.  I've been the tech reviewer on both of these books.  So opposed to many of the other authors who just write something, publish it, and then figure out that they got it all wrong.  These are technically pretty as close to correct as it gets.  Until we change it again.  Until we change it, of course.  Yes.  I think you find Michael is going to sell it to you.  You even get a second commercial for free.  Enough.  I know you benchmark.  I know I've just made sure you do as well.  You know, when I look at old stuff, can get hurt.  The old stuff actually does not really get hurt because it doesn't the difference between the CPU speed and the memory access isn't all that big there.  So the hit is not remotely as big as on new stuff where you forcefully disable the offloading.  Have we looked at the kind of middle generation of hardware and so like what I'm saying is whatever happens to HP 300 in Spark, I kind of don't care.  My concern is the Socracy and the Alex and the small embedded IPv6 that appeared a lot, people have to avoid it.  With those generations of next that kind of suck.  Everything you just mentioned has offloading that works.  And it does work.  Well, it's enabled because it's the cheap chips that implement the proper full offloading.  It's not the server chips that eternalize the hack.  So is it really worth doing hardware offloading?  Never.  For performance, not really anymore.  So obviously you can't fully answer this question.  Why are servers ramming this down my throat as the best thing since sliced bread?  Because it's an item in the marketing checklist.  Why is Microsoft claiming, okay, never mind.  Because what servers will you buy that say support checks them offload or the ones that don't have a checkmark and not bought?  Well, I can tell you how I order servers.  I call our suppliers saying I need five more calls over.  And when you pick nix to put in them, since it doesn't matter.  But you don't.  Exactly. You don't get to pick.  It's on board anyway.  Any more questions?  Hey, that's a new one.  Was that this year?  No.  My neighborhood is kind of crazy.  It is.  So my neighborhood declared autonomy from the surrounding city a couple of years ago.  Of course, this is just, this is being a joke, right?  But this kind of describes the feeling in the area.  Everything works like a tiny, tiny, tiny village.  Everybody knows everybody, but we're surrounded by the second biggest city in Germany and we're right in the middle of it.  So once a year, we're having a big street fest.  And for some reason that we don't understand the city of Hamburg things, we need to get a permit for this.  Well, we don't think so.  We're doing this for more than 25 years now.  So each and every year the fest is a little bit bigger than the year before to give you an idea.  It's more than 10,000 people throughout the day.  And over the years, the, recently it got a little bit better.  Like five years ago when it was first, the city of Hamburg sent us 4,500 policemen, seven water cannons, one tank.  So what happens is they come in around 10 p.m.  They get everybody wet, block the streets for two hours, then they piss off again and the party goes on.  Works for me?  No more questions.  Of course the neighborhood is awesome.  Okay.  You can kick robocops out of your backyard.  It's fucking amazing.  We keep calling the riot control police guys robocops because that's how they look.  And one year I look out of my rear window and there's two of them standing in my garden.  Wait a second.  So I open the door.  I don't think you want to be in my garden.  The house next door has been spotted.  The house next door used to be an old abandoned building that nobody ever shit about.  So it has been declared spotted each and every year.  And at the evening everybody went home and well, case closed.  But that year for some reason they thought they had to do something about it.  Sent 200 policemen in, kicked everybody out.  Then noticed, wait a second, we did not talk to the owner of this building.  So they desperately tried to reach him, couldn't then realize shit.  We don't have any right to be in this building at all.  We have to go again and the people who spotted it came back.  Two of the robocops suddenly standing in my backyard.  So I'm stepping out.  You don't want to be here.  They keep saying, you need to secure the scene.  Okay, whatever.  Not in my backyard.  But you can jump over the little fence style thingy.  But yeah, there's a bush behind it.  It's going to be too hurt.  Not my problem.  They did.  Then I put up a sign stepping on the lawn as a private and for people wearing uniforms.  Legally binding.  Of course.  Of course.  Anybody has a technical question?  Anybody has beer for me?  No.  Okay.  Then please buy the books.  Did I forget any commercials?  Buy the books.  Come to Europe ESD con.  Come to Asia PSD con.  Buy hosting from a company.  Anything else?  Buy open BSD CDs.  Donate to the foundation.  You have to come to the slide and say which one is the company.  You need to change your t-shirt.  So just list like Queensland is called.  I don't know.  So you have to tell.  I'm not smart enough for that.  Okay.  Okay.
The_surprising_complexity_of_checksums_in_TCP_IP_2_0_1091.png The_surprising_complexity_of_checksums_in_TCP_IP_2_1092_1283.png The_surprising_complexity_of_checksums_in_TCP_IP_2_1284_2711.png The_surprising_complexity_of_checksums_in_TCP_IP_2_2712_4583.png The_surprising_complexity_of_checksums_in_TCP_IP_2_4584_6095.png The_surprising_complexity_of_checksums_in_TCP_IP_2_6096_6155.png The_surprising_complexity_of_checksums_in_TCP_IP_2_6156_6203.png The_surprising_complexity_of_checksums_in_TCP_IP_2_6204_6275.png The_surprising_complexity_of_checksums_in_TCP_IP_2_6276_6647.png The_surprising_complexity_of_checksums_in_TCP_IP_2_6648_6971.png The_surprising_complexity_of_checksums_in_TCP_IP_2_6972_7019.png The_surprising_complexity_of_checksums_in_TCP_IP_2_7020_7067.png The_surprising_complexity_of_checksums_in_TCP_IP_2_7068_8483.png The_surprising_complexity_of_checksums_in_TCP_IP_2_8484_10739.png The_surprising_complexity_of_checksums_in_TCP_IP_2_10740_11579.png The_surprising_complexity_of_checksums_in_TCP_IP_2_11580_12431.png The_surprising_complexity_of_checksums_in_TCP_IP_2_12432_12503.png The_surprising_complexity_of_checksums_in_TCP_IP_2_12504_12803.png The_surprising_complexity_of_checksums_in_TCP_IP_2_12804_13031.png The_surprising_complexity_of_checksums_in_TCP_IP_2_13032_13139.png The_surprising_complexity_of_checksums_in_TCP_IP_2_13140_13187.png The_surprising_complexity_of_checksums_in_TCP_IP_2_13188_15191.png The_surprising_complexity_of_checksums_in_TCP_IP_2_15192_16715.png The_surprising_complexity_of_checksums_in_TCP_IP_2_16716_18119.png The_surprising_complexity_of_checksums_in_TCP_IP_2_18120_19307.png The_surprising_complexity_of_checksums_in_TCP_IP_2_19308_21323.png The_surprising_complexity_of_checksums_in_TCP_IP_2_21324_22943.png The_surprising_complexity_of_checksums_in_TCP_IP_2_22944_24479.png The_surprising_complexity_of_checksums_in_TCP_IP_2_24480_24899.png The_surprising_complexity_of_checksums_in_TCP_IP_2_24900_25679.png The_surprising_complexity_of_checksums_in_TCP_IP_2_25680_28031.png The_surprising_complexity_of_checksums_in_TCP_IP_2_28032_29567.png The_surprising_complexity_of_checksums_in_TCP_IP_2_29568_31103.png The_surprising_complexity_of_checksums_in_TCP_IP_2_31104_32507.png The_surprising_complexity_of_checksums_in_TCP_IP_2_32508_34547.png The_surprising_complexity_of_checksums_in_TCP_IP_2_34548_35843.png The_surprising_complexity_of_checksums_in_TCP_IP_2_35844_37559.png The_surprising_complexity_of_checksums_in_TCP_IP_2_37560_38879.png The_surprising_complexity_of_checksums_in_TCP_IP_2_38880_40475.png The_surprising_complexity_of_checksums_in_TCP_IP_2_40476_40571.png The_surprising_complexity_of_checksums_in_TCP_IP_2_40572_41519.png The_surprising_complexity_of_checksums_in_TCP_IP_2_41520_41759.png The_surprising_complexity_of_checksums_in_TCP_IP_2_41760_41819.png The_surprising_complexity_of_checksums_in_TCP_IP_2_41820_41879.png The_surprising_complexity_of_checksums_in_TCP_IP_2_41880_41927.png The_surprising_complexity_of_checksums_in_TCP_IP_2_41928_41975.png The_surprising_complexity_of_checksums_in_TCP_IP_2_41976_42023.png The_surprising_complexity_of_checksums_in_TCP_IP_2_42024_42071.png The_surprising_complexity_of_checksums_in_TCP_IP_2_42072_42119.png The_surprising_complexity_of_checksums_in_TCP_IP_2_42120_42179.png The_surprising_complexity_of_checksums_in_TCP_IP_2_42180_45731.png The_surprising_complexity_of_checksums_in_TCP_IP_2_45732_47207.png The_surprising_complexity_of_checksums_in_TCP_IP_2_47208_48323.png The_surprising_complexity_of_checksums_in_TCP_IP_2_48324_50759.png The_surprising_complexity_of_checksums_in_TCP_IP_2_50760_52211.png The_surprising_complexity_of_checksums_in_TCP_IP_2_52212_54263.png The_surprising_complexity_of_checksums_in_TCP_IP_2_54264_56975.png The_surprising_complexity_of_checksums_in_TCP_IP_2_56976_57875.png The_surprising_complexity_of_checksums_in_TCP_IP_2_57876_58079.png The_surprising_complexity_of_checksums_in_TCP_IP_2_58080_59147.png The_surprising_complexity_of_checksums_in_TCP_IP_2_59148_60875.png The_surprising_complexity_of_checksums_in_TCP_IP_2_60876_66467.png The_surprising_complexity_of_checksums_in_TCP_IP_2_66468_68591.png The_surprising_complexity_of_checksums_in_TCP_IP_2_68592_68831.png The_surprising_complexity_of_checksums_in_TCP_IP_2_68832_72263.png The_surprising_complexity_of_checksums_in_TCP_IP_2_72264_72719.png The_surprising_complexity_of_checksums_in_TCP_IP_2_72720_73136.png
Welcome. I'm Henning. I'm saving you from the who am I part? No? Well, I drink beer. I hack an openBSD. You might have come across my name if you're using pf or openbgpd or ntpd or... What else did I do? Female. Pardon? Female, yes. Oh, whatever. It's been a long time. Before I actually start my talk, I do have a question. You know, we had this earthquake yesterday, and there's been this thing, BSD conference plus earthquake. I've been into that before, and so I have just one question. Is there a nuclear power plant near here? Okay, then I can go on. I want to talk about the checksum in TCP IP. We're mostly talking about two sets of checksums here. One is the IP checksum, and the other is the so-called protocol checksum that mostly is TCP and UDP. IPv6 does not have the IP header checksum one. I'll explain later why that doesn't make a difference at all. The actual checksumming algorithm is really, really, really simple. It's really just the lowest word of the once-compliment sum of all the words in the either IP header or almost entire packet. We'll come back to that later. The IP checksum just covers the IP header while the protocol checksum covers a part of the IP header, the actual protocol header, and the entire payload. So there's performance considerations in that. Me being mostly involved with firewalls and routers and everything, I care about forwarded packets, not locally generated ones all that much. The actual math for the checksum is dirt, dirt, dirt cheap. I spent many, many, many years profiling, and you cannot even find the checksumming function anywhere noticeably up in the call graphs. So the calculation can be considered completely free. We're just talking about the math here. That's basically talking about the CPU's integer units, right? The limiting factor that we run into is latency and bandwidth to memory. So we care about caches, RAM, and buses. This is mostly relevant for modern architectures. On the really old architectures, it doesn't really make a difference because their memory, relatively to the CPU, is fast. On modern systems, memory is comparably dog slow because the CPU's also got them fast. Check-summing data that already is in the cache is essentially free because the cache is fast. So in the forwarding case, when we are dealing with the IP pack and we are forwarding, the IP header is in cache. Of course, we need that information for forwarding, right? So IP header checksum free. IPv6 not having that checksum, which they often point out as a performance improvement, doesn't make a difference. You can't even measure it. So the few cycles we spent before to calculate the checksum in v6, we spent to wait for memory. Check-summing data that we have to fetch from memory, that is very, very expensive. So in the forwarding case, we usually don't touch the payload of a packet, right? So fetching that data just to check-sum it is expensive because, as I said, the memory is slow. Some platforms have optimized check-summing implementations. I find the list of architectures here kind of interesting. I3 to C, Spark64 and SH have a simpler implementations and alpha, HPPA, HPF64, the M68 case, the M88 case, Spark and the VAX have optimized C implementations that actually might contain a simpler code. I didn't check them all. The interesting bit is that the most modern architectures you see here do not have architecture-specific checksumming implementations because they don't need them. Are the optimized ones actually faster? The I386 assembler version is certainly written for an 8386, right? The assumptions being made about that CPU are almost certainly not true for the latest generation Xeon CPU. We've encountered, in other areas, several cases where the highly optimized assembler versions were actually way, way, way slower than the generic C version because the compilers do a better job these days. And as I said, the assembler versions were written for CPU architectures from 10 to 15 years ago. So there's a task here. Who was asking me about tasks? Where are you? There's the task. Take those benchmark and figure out whether they are actually any faster than the generic C version. Thank you. The IP checksum does cover the entire header. The IP header is rather simple. You have the IP version, which is always four, always. You have the header length, the type of service. You got it. The type of service. You have the total length of the IP packet. The other length is just the header length, and that is pretty much a constant these days. Pretty much. You have the IP ID and the IP offset for fragments. You have the TTL, which each and every router on the way has to decrement, and that is being part of the checksum. You have the protocol, that's just the number indicating whether it's TCP, UDP or something else. You have the actual checksum, and of course the source and destination addresses. So since every router has to decrement the TTL, it has to update the checksum, right? The checksum itself is part of the checksum. Because of the way the algorithm works, you have to set the checksum field to zero, run the algorithm over it, and the value that comes out of that, you put into that field. If you want to verify the checksum, you run the very same algorithm over the entire IP header again, with the checksum in place, and if the result is zero, you're golden. If it's not zero, the checksum doesn't match. So as I keep repeating, the IP header is in cache. That's almost guaranteed, because we're talking about the network just forwarding this, right? Recalculation is free, so don't worry. Offloading the IP checksum calculation to the other interface card doesn't make a difference anymore. I did this like 12 or 13 years ago, and there we saw a gain of something like 7 to 8%, today, not measurable. So the implementation in OpenBSD, the basic flow of a packet that we are forwarding, is IP input calling IP forward, calling IP output. IP input is quite a complicated beast. For most, it takes the decision whether the packet is local to be forwarded or not deliverable, and it does call pf test, which is the entry point to pf. If it's a packet to be delivered locally, it's just being handed off to the upper layers, like TCP and UDP. If it's to be forwarded, well, we're calling IP forward. All the books tell you that IP forward does the routing. That is actually a lie. The route lookup and everything else happens in IP input and is being passed on to IP forward. All that IP forward does is, well, decrement the Tdl, and I think that's on the next slide. No, it's not. Well, it deals with the ICMP error generation. If we figure out we cannot forward the packet, we have to generate an ICMP error in most cases, right? And the ICMP error quotes the original packet, or at least the tiny little bit of it. I'll get into the details on that later. So IP forward keeps a copy of a fraction of the packet that's forwarding, calls IP output, and if IP output fails, it does the IP error generation. It doesn't really do any routing. IP output is the most complicated of these functions. In the forwarding case, it's actually trivial. The mass of code in there deals with locally generated packets that do not have the IP header and TCP stuff filled in yet, or these not completely. IP output once again calls P of test for the outbound check. And right after P of test, IP output recalculates the IP checks unconditionally. Once to cover the decreased time to live field, of course, but there's also the possibility that PF changes, right, like in every data translation. Locally generated packet fitting IP output at that point do not have a full checksum yet. The checksum handling in PF. We used to update the IP checksum in PF for each and every change we made. Of course, with IP output calling the IP checksum algorithm right after PF, this was utterly and completely useless. Well, not completely, because there's the bridge. And the bridge, operating in layer two, of course, does not decrement the TTL in the IP header and doesn't make any changes to the IP packet whatsoever. So the bridge does not really need to update the IP checksum. But that doesn't lead to the bridge being special case all over the stack. There's second reason, but this is the worst one. So to solve that and to stop PF from having to fiddle with the checksum all the time, the solution, of course, is to make the bridge behave like a regular output path, right? And that's what I did in the end. Well, wait a second, there's checksum offloading. Pretty much each and every network interface card that has been made in this century can do it. Most that have been made in the last decade of the last century can do it as well. Our stack had been modified ages ago to use it. This probably came from previous, but I don't really know. As I said, that's been done ages ago. It basically works like this. We delay checksumming until we know the outbound interface. Once we know the outbound interface, we can check this interface flags for offloading capabilities. If it does have offloading capabilities, we just do the little magic to tell the chip, please update the checksum, which obviously is driver dependent, but there's a generic flag in the stack indicating this needs checksumming and move on. If we do not have the offloading capabilities in the output path, we used to calculate this in software right there. On the inbound side, the hardware, respectively, the driver just sets a flag in the MF packet header indicating the checksum is good or the checksum is bad. Our verification routines, well, we now have them before it was the Bill Joy disease of I know the internals, I fiddle directly with it all over the place and I don't need APIs. Now the routines, don't get me started. The verification routines look for those flag. If it does have the good flag, well, we're done, the checksum is fine, move on. If it does have the bad flag, well, we're done, just drop the packet. If it doesn't have any resort to software. Surprise, the bridge comes in the way again. The biggest problem I ran into with the bridge, and this is the second reason why I eventually was not able to remove all the special casing from the stack, but only half of it, is that the bridge has a gross hack for broadcast. It basically works like that. The packet comes in, it already has a pointer to the outbound interface, like the bridge decided on it's that outbound interface for broadcast that is a kind of random decision. So we look at the offloading capability flags of that kind of randomly chosen interface. If it happens to have offloading capabilities, we would not do this in software, right? So, well, as said, gross hack, the bridge just copies the packet over and over and over and sends it out to all interfaces. But this is after we've looked at those flags. So it would go out without a checksum on all interfaces that do not have offloading capabilities. Not all that good. If there's anybody who wants to do me a favor, remove that gross shortcut. It shouldn't be too hard, but I was in the middle of this mess and I didn't meet that little mess on top. Task. So the IP checksum status. All the IP checksum handling in PF, I was able to remove after I fixed the bridge. As expected, as already mentioned, I'm completely unable to measure any performance benefit on modern hardware because, as I said, it's basically free. But this considerably simplified the code and quite frankly the IP checksum handling in PF was just wrong. What happened there is 10, 12 years ago when we started hacking on PF, the PF group, including myself, considered, we considered us to be PF hackers. So we did everything in PF, but we didn't really look at the network stack around it. And well, that changed, fortunately. And we now consider, we now consider PF to be an integral part of the network stack and we don't necessarily do everything in PF just because we're PF hackers, but we look at the whole picture, which is what we should have done 10 years ago. So now let's go to the protocol checksums, which are much more problematic. As I mentioned, they cover only a part of the IP header. Why? I don't know. It doesn't really make sense to me, but that's what the standard says. They do cover the entire protocol header, of course, because the IP header checksum does not cover it and you want everything in the packet to be covered, right? And this is critical. They do cover the entire payload. So that makes it potentially very, very expensive to, well, relatively spoke, very expensive to recalculate the checksum. Of course, as I keep mentioning, the payload is almost certainly not in cash for a forward packet. The handling we had on PF for the protocol checksums, when we were doing that, when we changed ports or the IP header as a server thing, we obviously have to update the checksum as well. So we did exactly the same thing, update the checksum on the fly. And that's how that looks. And this makes me want to vomit. It's nested calls of PF checksum fuck up all over the place. Please note that the outmost call is different from all the inner inside calls. Like this is really easy to screw up. It's ugly as hell and it... I'll skip that. We do have to call PF checksum fuck up or sorry, fix up per word we touch in the packet for each and every single word. And PF checksum fix up needs the old word and the new word to calculate the delta and apply that. So it needs to see old word, new word and the checksum. All the BSD derived network stacks have so-called protocol control blocks to track connections and keep state in TCP and UDP. UDP is connectionless so it does not really have state, right? No wrong. In the state, of course, it has state. In OpenBSD and I think in the other BSDs as well, the PCBs are being looked up using hash tables. And in OpenBSD, that's a separate talk that I gave two years ago, the PCBs can be linked to the PF state. So if you're on your firewall and you already looked up the PF state, you just follow the pointer to the PCB instead of doing a new look up. What did that bring us? Something like 8, 9% or so? 7 to 8% or something. Well, that was not the big one. 7 to 8% was something that simple is nice. So whenever a socket is opened, there's a template PCB being created. I want to cover the bound listening socket here. As in you call it bind and listen because that's the path that is easiest to follow and to understand. Once you bind, you know the local IP address and the local part, right? So what did they do back then? Let's check some of the source IP address and the source part and install that partial. Check some in the template PCB. Now we're getting a connection in and we're calling accept. That leads to well in your socket obviously and we'll cover the copy the template PCB to a PCB for that new connection. We add the other sites IP and port information and update the checksum in the PCB for those fields we just added. This is called the pseudo header checksum. The reply packets like the packets coming in from the other side, we just have to verify right, but the reply packets that we send out, we obviously have to check some. So we take that partial checksum from the PCB into the packet, send this all the way down to the stack with the needs checksumming flag and somewhere way down there, we'll update the checksum. Actually update is not entirely true here, but I'll come back to that. We'll fix the checksum to cover the entire payload as well. So outbound in theory. Pretty late in the outbound path. We just look at the checksum flags in the protocol in the in the packet header. We compare that to the interface capabilities. So there's basically three cases. The packet says I need checksumming and the interface says I can do it. Well, fine. Just move on. It'll do it for us. There's the case where the packet says I need checksumming and the interface is not capable of doing this. Well, we need to do this in software. This used to be somewhere in the middle of the stack. I moved this as much down as possible. So we're basically operating in a model where we assume that we always have checksum offloading. And if not, we emulate the offload engine in software. If the packet says it doesn't need checksumming, we obviously don't need to do anything. It can go for beer. The partial checksum and pseudo header checksum from the PCB that gets copied into the actual packet. That's a hack that might have helped on HP 300. Today, it is actually counterproductive. It complicates things a lot. But unfortunately, some of the hardware vendors eternalized this hack in hardware. And unfortunately, this is mostly Intel and Broadcom. Should I say that real tech got this right? Real tech beats Intel and Broadcom on this. Intel and Broadcom require the pseudo header IP checksum to be there and to be correct. And they just update it for the payload. All the other offloading engines don't care what's there. Recalculate the checksum completely and fill it in. That leads us to the famous redirect to local host bug. So a packet coming from the local host, obviously, needs checksumming, right? So if you redirect to local host, say you redirect all packets coming from your LAN that go to FTP, you redirect them to local host, port 8021, to your FTP proxy, right? So your FTP proxy sends the reply back to the client. Now the reply to the client, obviously, must not come from 127.001, but from fdp.oppsd.org. So PF has to rewrite the source address. When PF rewrites the source address, it obviously needs to take care of the checksum. Now that packet being locally generated has the pseudo header checksum. PF has no way to see whether that packet has a full checksum or the pseudo header checksum or some other partial checksum. So PF updates those fields and updates the checksum for it. Wait a second. The pseudo header checksum doesn't cover the entire packet. We are modifying packets that are not being covered by the checksum, but we're updating the checksum for a packet, for a field that's not covered in it. Oops. So as long as we're just dealing with the IP addresses, that kind of works, but as soon as you touch something else, it blows up and you always touch something else. The software checksumming engine doesn't care, it just recalculates the checksum anyway or calculates. There's no recalculation because it's not there yet. The same hardware doesn't care, it just computes the checksum. But the hardware relying on pseudo header checksum being there is doomed. It updates an already broken checksum, and updating a broken checksum leads to a broken checksum. The result of that is that the protocol checksum offloading is disabled on that class of hardware for at least 10 years now. And that unfortunately at least means EM, BGE and BNX. That's what? 99% of the server market? So the offloading, almost everything made in this decade, know that I'm saying this decade and not the century, has offloading support. Not necessarily for IPv6, but let's stick to the relevant topics. Unfortunately, there are quite a few silicon bugs. Like the most spectacular we ran into lately, when you, I think it was an Intel 10GE1, when you enabled protocol checksum offloading, which only has business in touching TCP and UDP packets, right? Corrupted OSPF packets. It's not even supposed to touch them. Since it covers the payload and we have the different behavior of the offloading endges, this is considerably more complicated than the IP header checksum. But we can summarize this into three cases. We don't have offloading. We have the halfway offloading where the pseudo-hider checksum already is required. It's just being updated. And we have the full offloading. So we always want to work in the new world. We always want to work under the assumption that we have offloading. Of course, that considerably simplifies the handling all over the stack. When we hit an outbound path that does not have that capability, well, we emulate that in software. So for that, we now got a function, a proper API instead of fiddling all over the place called INPrototCheckSumOut. That does all the magic. It just looks at the flags, does the dance, and calls the software engine if needed. There's the INPdelateCheckSum for IPv4. And surprise, there was no IN6 DelateCheckSum, because I don't know. I don't know why it wasn't there. My theory is that we fix the source load that doesn't make a difference anyway. We call that prototCheckSumOut functions very, very late. We certainly know on which interface we are going to send the packet out by then. In the old world, the call was earlier. In most cases, we knew the right outgoing interface already, but not in all of them, unfortunately. So now that's much easier. That means the entire rest of the stack does not have to care about checksums anymore. All it needs to do is you modify the packet or you generate the packet, or whatever you do, change the packet, you set the flag, hey, this needs checksumming, done. And foremost, this means there doesn't have to be any knowledge about offloading anywhere that was all over the place before. This also means that we can remove pretty much all checksum handling from PF itself. If PF is modifying the packet, set the flag, I need checksumming. Well, there's ICMP. Did I miss a slide? No. There's ICMP. ICMP has a checksum as well. I remodeled that entire thing after the TCP and UDP handling. There actually is no hardware that has ICMP checksum offloading. It doesn't quite make sense because ICMP packets tend to be small. And it's, as I said, free in software. Should there ever be one, it would be trivial for us to implement that, but I don't expect that to happen. But the point is, ICMP being the same as TCP and UDP makes the code much easier and the handling all over the stack much easier. And the performance is the same anyway. Woo-hoo! Awesome. Now my timer is screwed. There we go. So, as I mentioned before, ICMP error message quotes the packet they refer to. Well, they have to. So, ICMP is a code that can be used in any way, but it's not a code that can be used in any way. It's a code that can be used in any way. So, I'll say it again, that's the code that can be used in any way. So, as I mentioned before, ICMP error message quotes the packet they refer to. Well, they have to. If you think about using the client, you send the TCP packet to the destination. For some reason, the destination is unreachable. Whether that's on the first packet or somewhere on an established connection, it doesn't really matter. The router dropping that packet because it's not deliverable, sends that ICMP error back. And you as a client have to have a way to match this ICMP error to your little PCB describing that TCP session, right? Because you want to peer the TCP session down. How's this done? The ICMP error quotes at least a part of the packet that it refers to. Now there's network address translation. Well, if we are translating the packets, we unfortunately also have to translate the quoted packet inside the ICMP packet. And also, surprise, there's no offloading support for that anywhere. The quoted packet, well, that has a checksum too. In most cases, it is truncated anyway. So, there's no way to verify the checksum of the quoted packet inside the ICMP packet. So, we don't have to care. If it's not truncated, nothing in the entire real world cares. There's one, I wouldn't call this an exception because it's not really the real world. There's a tool called SCAPE that we use to generate weird, weird, weird, weird IP packets, throw at our stack, not just IP, TCP and even payload, that we throw at our stack and try, on NPF and try to make it misbehave. And then SCAPE receives those packets as well. And SCAPE is the only thing in the world that I'm aware of that actually verifies the checksum inside the quoted packet inside the ICMP error message. This would be easily fixable because you could just recalculate the checksum in the non-truncated case. I have not implemented that, but could do it. It should be reasonably simple. And the cost is basically zero because it's tiny. We're talking topmost 68 bytes. So, consequences in PF. The old way was we update the checksum. What happens when you update a broken checksum? Well, you get a broken checksum. That's good. But now, we're not updating anymore. We're replacing the checksum. We are recalculating it. Now, we have a packet coming in with a broken checksum. We do some adjustments and we recalculate the checksum. Hey, it's not broken anymore. We just fixed it up. Oops, that's not good. We are hiding data corruption. So, before we do that, we unfortunately have to verify the existing checksum. On anything that's halfway sane, that's free because the offloading engines already tell us whether it was the checksum was correct or not. But if there's no offloading, that's potentially expensive because the payload, once again, is not in cash. Nonetheless, the new code is so much nicer and cleaner and there's so much more that we can clean up now. The redirect to local host bug is gone because we are doing the checksum much later anyway, so we don't have to care about the difference between pseudo-hider checksum and not. This eventually allows us to enable the offload engines on all these chips that require pseudo-hider checksums. Well, EM, BGE and BNX are pretty much 99% of the server market. Unfortunately, there is one case that suffers considerably. That is doing that when you do not have any offloading capabilities. Now, let's put this into perspective. If you're doing anything else but pure packet forwarding, the total cost of packet forwarding is basically nothing. If you run a proxy in userland, try to measure the cost of your packet handling in the kernel. It doesn't matter. It's barely measurable. If you're running IPsec, well, we don't even get to start on that. So the packet forwarding cost is tiny, so even if it becomes a tiny little bit more expensive, this would not really hurt us. If you're running your netting gateway on something without offloading and care about performance, sorry, you're doing something wrong. What we could do to fix that problem is calculate the checksum over the header before and after pf and apply the delta. Well, that means we cannot use any offloading engines because no offloading engine has support for verifying a part of the checksum and then apply a delta. So it's all software, which also means that we hurt the offloading case considerably. Given that pretty much everything has offloading, I rate it optimized for the offloading case than for the non-offloading case. And the old architectures that tend to not have any offloading capabilities don't suffer as much because, relatively speaking, the memory is fast there. In microbenchmarking, in that case, on a very, very modern AMD64 system like the latest generation Xeon, with, I think, we used EMs and disabled the offloading in the driver. This is a completely made up case that's not realistic at all. We are losing something between 5 and 10 percent depending on the traffic pattern. This depends a lot on the specific hardware implementation on the machine because this depends on caches, on the cache architectures. Is the layer too cache-shared between the CPU cores or not? It's stuff like that. It depends a lot on the buses. PCI is much worse than PCIe. And as said, that class of machines pretty certainly has offloading capabilities. So in my book, this is acceptable. If you have such a machine, let me find a nickel. Buy some of your Apple cards. So that's benchmark for the better use of the offloading engines we make now. On that mentioned machine, current generation Xeon, and pure forwarding, I cannot measure any difference. By the raw computing power that the current Xeon has, they are so fast, they have so much raw computing power. The caches are big and fast that it just doesn't make a difference anymore because our limiting factors are other factors. The limiting factors are latency, foremost latency. At Brandworth, we have plenty usually. It's latency to cache, to memory, to the buses, foremost on the buses. And we're not even talking about the latency outside the machine yet. So in microbenchmarking, even if there is an effect, it's completely hidden. This might be different if you're actually doing something else on the machine because you do save a few cycles. I don't think that really matters, but some people might care. On an older system, a Pentium M using an Intel GigabitNIC, we are seeing about a 7% increase. I keep coming back to the 7% when I do the checksum. That's the magic number there. But with that, with a much, much, much cleaner and much better structured code, we pave the road for future improvements. There's so much clean up possible afterwards. Last but not least, this is just less code, and this code is nicely contained in one file, in one spot, and not spread all over the stack. There's a nice and simple API to call instead of fiddling with the internals or the entire stack. We can actually remove that entire absolute header checksum dance in the PCVs. What's the point in pre-calculating that? There is no performance benefit at all. So just remove it and do the entire checksum late in the output path in those new IN and IN6 proto checksum out functions. And we only actually need to do that in the offloading case because the other two cases, the software engine does full recalculation anyway, and the same hardware does full recalculation anyway. We can unfortunately not distinguish the full offloading from the needs the pseudo checksum offloading. We could modify the drivers to signal that up, but there's no point since calculating that is so cheap that checking the flag is almost, well, checking the flag is still somewhat cheaper, but it just doesn't make a difference. The bridge special casing all over the stack needs to die. The biggest reason for the bridge special casing is gone with this. There's just that broadcast tag remaining. Yes. Is that how you get on the left towards the bar and the right to the bottom? No, that picture is actually from Warsaw. Oh, it looks familiar. So we briefly touched the surface here. This is actually complicated. I would have lost a lot of hair over it if she hadn't been faster. And we have to give acknowledgement to Christian Weisberger a lot that's Natty at OpenBSD, give him a lot of acknowledgement for this, because he was the first one to actually understand that redirect to local host bug. And I very, very remember him and us sitting there and him explaining this to us more than once. And eventually we understood what was going on as well. And just a couple of years later I started working on that. He also helped a lot with testing when I finally had something working. There was a lot of help and support from Mike Billow-Pouhauf. I think I got this right. That's Mike B. And Peter Hasler. They did a lot of the testing for me last time, please, because they have access to an Xeon. I don't. The IPv6 delay check something was written by Kenneth. That's a shithole. Finding the TCP header in IP is that simple. That's a fixed offset, basically. Finding the TCP header in IPv6. The function doesn't fit one screen. You have to loop over the chain of headers and hope that nothing goes wrong. So I'm working on this since late 2010. There's been a lot of hackathons that I used for this. There's been the Iceland one organized by the we've been in admin more than once organized by Theo, Bob and Theo on crew. We've been to Ljubljana in Slovenia. We've been in Budapest in Germany, Portugal. We just have been to New Zealand, which was awesome. And I really want to thank all the people that make this possible. The other occasion where I keep making progress on this are the other conferences. So there's EuroBSDCon commercial break. We are going to have EuroBSDCon in Malta in September. And Malta is a really nice island. So you do want to come. The call for papers is still open for another, I think nine days or so. Yes. So we want speakers. We already have plenty of submissions, but more is better. Bigger is better. We also want you as visitors, of course. So just tell your wife, girlfriend, whatever, let's go on vacation to Malta. Oh, by the way, there's a conference. I know the answer. Do you have a wife and how do you explain to her that you're sitting in a dirty garden? Isn't it bad enough that I have to explain your wife that you did last night? I'm thinking if she's in Malta, she doesn't care about women all day. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. Point. Are there any questions? Oh, I forgot another commercial break. I so need a jingle for this. I need a jingle stopped in the middle of the talk commercial break. So if you want to know more about the PF, there's the awesome book of PF written by Peter. You carry two copies, right? So I deeply suggest you get a copy. It's an awesome book. There's the almost brand new, brand new absolute open. It's the second edition by Michael Lucas. I've been the tech reviewer on both of these books. So opposed to many of the other authors who just write something, publish it, and then figure out that they got it all wrong. These are technically pretty as close to correct as it gets. Until we change it again. Until we change it, of course. Yes. I think you find Michael is going to sell it to you. You even get a second commercial for free. Enough. I know you benchmark. I know I've just made sure you do as well. You know, when I look at old stuff, can get hurt. The old stuff actually does not really get hurt because it doesn't the difference between the CPU speed and the memory access isn't all that big there. So the hit is not remotely as big as on new stuff where you forcefully disable the offloading. Have we looked at the kind of middle generation of hardware and so like what I'm saying is whatever happens to HP 300 in Spark, I kind of don't care. My concern is the Socracy and the Alex and the small embedded IPv6 that appeared a lot, people have to avoid it. With those generations of next that kind of suck. Everything you just mentioned has offloading that works. And it does work. Well, it's enabled because it's the cheap chips that implement the proper full offloading. It's not the server chips that eternalize the hack. So is it really worth doing hardware offloading? Never. For performance, not really anymore. So obviously you can't fully answer this question. Why are servers ramming this down my throat as the best thing since sliced bread? Because it's an item in the marketing checklist. Why is Microsoft claiming, okay, never mind. Because what servers will you buy that say support checks them offload or the ones that don't have a checkmark and not bought? Well, I can tell you how I order servers. I call our suppliers saying I need five more calls over. And when you pick nix to put in them, since it doesn't matter. But you don't. Exactly. You don't get to pick. It's on board anyway. Any more questions? Hey, that's a new one. Was that this year? No. My neighborhood is kind of crazy. It is. So my neighborhood declared autonomy from the surrounding city a couple of years ago. Of course, this is just, this is being a joke, right? But this kind of describes the feeling in the area. Everything works like a tiny, tiny, tiny village. Everybody knows everybody, but we're surrounded by the second biggest city in Germany and we're right in the middle of it. So once a year, we're having a big street fest. And for some reason that we don't understand the city of Hamburg things, we need to get a permit for this. Well, we don't think so. We're doing this for more than 25 years now. So each and every year the fest is a little bit bigger than the year before to give you an idea. It's more than 10,000 people throughout the day. And over the years, the, recently it got a little bit better. Like five years ago when it was first, the city of Hamburg sent us 4,500 policemen, seven water cannons, one tank. So what happens is they come in around 10 p.m. They get everybody wet, block the streets for two hours, then they piss off again and the party goes on. Works for me? No more questions. Of course the neighborhood is awesome. Okay. You can kick robocops out of your backyard. It's fucking amazing. We keep calling the riot control police guys robocops because that's how they look. And one year I look out of my rear window and there's two of them standing in my garden. Wait a second. So I open the door. I don't think you want to be in my garden. The house next door has been spotted. The house next door used to be an old abandoned building that nobody ever shit about. So it has been declared spotted each and every year. And at the evening everybody went home and well, case closed. But that year for some reason they thought they had to do something about it. Sent 200 policemen in, kicked everybody out. Then noticed, wait a second, we did not talk to the owner of this building. So they desperately tried to reach him, couldn't then realize shit. We don't have any right to be in this building at all. We have to go again and the people who spotted it came back. Two of the robocops suddenly standing in my backyard. So I'm stepping out. You don't want to be here. They keep saying, you need to secure the scene. Okay, whatever. Not in my backyard. But you can jump over the little fence style thingy. But yeah, there's a bush behind it. It's going to be too hurt. Not my problem. They did. Then I put up a sign stepping on the lawn as a private and for people wearing uniforms. Legally binding. Of course. Of course. Anybody has a technical question? Anybody has beer for me? No. Okay. Then please buy the books. Did I forget any commercials? Buy the books. Come to Europe ESD con. Come to Asia PSD con. Buy hosting from a company. Anything else? Buy open BSD CDs. Donate to the foundation. You have to come to the slide and say which one is the company. You need to change your t-shirt. So just list like Queensland is called. I don't know. So you have to tell. I'm not smart enough for that. Okay. Okay.
make a difference at all. The actual checksumming covers the IP header while the protocol checksum covers a part of the IP header, the actual actual math for the checksum is dirt, dirt, dirt case, when we are dealing with the IP pack and we having that checksum, which they often point out check them all. The interesting bit is that the and UDP. If it's to be forwarded, well, we're behave like a regular output path, right? And offloading capabilities, we would not do this in performance benefit on modern hardware because, integral part of the network stack and we don't mentioned, they cover only a part of the IP cover the entire payload. So that makes it protocol checksums, when we were doing that, when the checksum on the fly. And that's how that tables. And in OpenBSD, that's a separate talk cover the copy the template PCB to a PCB for that the PCB into the packet, send this all the way checksum and pseudo header checksum from the PCB things a lot. But unfortunately, some of the and updating a broken checksum leads to a broken instead of fiddling all over the place called That means the entire rest of the stack does not that has ICMP checksum offloading. It doesn't handling all over the stack much easier. And the the packets, we unfortunately also have to recalculate the checksum in the non-truncated unfortunately have to verify the existing checksum and not. This eventually allows us to tiny little bit more expensive, this would not non-offloading case. And the old architectures for the better use of the offloading engines we