Booting modular open source router
Hrusecky, Michal
We created an open source modular router. This by itself posed some challenges both on HW and SW side. Apart from it having to boot and discover all configurations, we also decided that it sounds like a good idea to use it as a simple AP without any local storage and boot it over the network. How it all comes together in the end? What open source tools are we using in the process? I'll try to describe our approach and how we solved the problems we were facing.
 So, hi everybody. Thank you for coming. And my name is Michal Hrushecki. I'm from a company called CZ-NIC and more specifically from the Therese project. And I'm going to speak here about a new router that we created. And we created a modular router. And we decided that we want to boot it and boot it over the network. And I'm going to talk about challenges that we were facing and how we went around to make it actually happen. So a few words about who we are. CZ-NIC is Czech Toplevel Domain Registry. Legally, we are some association of companies, but in fact we are working like a non-profit. And apart from managing Czech Toplevel Domain, we do quite some open source development. You might know us for doing not DNS alternative server or DNS resolver for doing BERT, BGP devoting daemon. And we also do some general good, mainly targeting Czech Republic, like education about internet and security and stuff like that. And one of the things that we do is also running Czech national CZ-NIC team. That's actually how this project started. We didn't just decide that we are going to do routers. The original idea was to figure out what are the type of attacks on the ordinary users at home. So basically we wanted to know who's attacking them, how often, and what are the attackers trying to do. So to do that, we created the first router because we needed some device to run it on. And the routers at that time were unable to basically do what we needed to do. And we gave it away for free and ran our security search on top of it. And when we started talking about that on various conferences, when we were presenting the results of the research, we found out that people like the open hardware and open software and having a root account on their device and being able to trust their devices and stuff like that. So we created a commercially available option, which was Tris Omnia. We ran some Indigo campaign. We got the money. We made the router. And it's now selling in retail and it's selling. And it's again open source out of the box with the root account and everything. And when we did that, we got some feedback from our users that they don't need SFP that we had there, that they don't need Ethernet ports because they have just Wi-Fi. Then they need more Ethernet ports and they don't need Wi-Fi. And they need more USB trees and stuff like that. So many conflicting requests. And that's how Mox came to being trying to make everybody happy. We created the modular router. And to show you how it works, we actually created some configuration web page where you get the CPU and then you decide what else you need. Where do you need some Ethernet ports? You just put some Ethernet ports. You need more. Let's put some more. Let's put some SFP in. And that's all nice. And you can do it even with the stuff that you buy in retail. And the generic idea is that if you need something special and if you have well-defined parameters that you want to achieve, you will just figure out the combination of components that will allow you to do so. And you buy it and you assemble it at home and you have a custom device as an end customer. Yeah, the idea is great. But there are a few challenges. Like what are actual components in there? From software side, we need to know what's in. And we also need to create some default configuration. So which port is actually going to your ISP? You might have SFP, but you might not have it. You can have up to 25 Ethernet ports. So which one would you pick as a one? And on the other hand, you might have just one Ethernet port. So is it one? Is it one? And what should we do when somebody buys a new module? He already had a router running and now he extended it and added more ports or took away some ports. How to handle that? And one of the things that we really wanted to do with this was also to allow people to basically extend their Wi-Fi coverage if they have some big house. So we wanted to create some setup where you can use the Mox as a simple AP and take configuration from your old Omnia and just get extended Wi-Fi coverage. That sounded like interesting and useful use case, but it poses another set of challenges. So how booting works in general? When the CPU starts, it runs Uboot that is stored in NOR and Uboot initializes some hardware. Uboot executes Distribute, which is a special command that is nowadays pretty much standardized across everybody. Well, everybody likes to use it. It's called Distribute because Distributions agrees that it's useful command. What it does is basically it tries several different ways how to boot your hardware. It tries EFI, it tries some configuration files, it tries Uboot Include file and if everything fails, it tries to boot over network using PXE. And then Uboot loads the kernel, Device3. Device3 is something that is quite common in those embedded devices. It basically describes the hardware that you have on the device because you have a kernel, you have a virus drivers, but then you need something to tell you how all the stuff is connected together and where it is and if it is there. And then Uboot continues with running the kernel and passes through some command line, interim disk, this Device3 and stuff like that. That's the generic embedded booting. We use this way, but we do some specific stuff because of our modularity. We have some SPI bus on which we can enumerate all the available modules. So in the Uboot, we use this SPI bus to figure out what modules are there and how they are connected to pass this information on. Then one thing that we did after discussions with some distributions was that we actually put our Device3 in normal memory that we have on the device itself as well. Why did we do that? Well, kernel has plenty of drivers and usually some CPUs are supported pretty fast, but so it's possible that you actually have support for some of the devices that we have in Mox even in your distribution kernel, but what you are missing and why you wouldn't be able to boot it actually is the Device3. So we took our Device3 and put it into NOR. So if your distribution has all the necessary drivers and doesn't have a Device3, it will use the Device3 that is available in NOR. What we did regarding the modularity, the Device3 describes how all the hardware is connected, but we would need quite some variations of Device3s because there is plenty of options how to actually connect the stuff together. If I take a look at the configurator, you see in just one way, but something like this is possible as well. So there is a lot of ways how to actually connect stuff together. Our kernel guy came up with the solution that basically we have a Device3 that contains all possible, it contains all the devices that could be connected and it connects them and then when UBoot actually runs the kernel, just before it runs the kernel, he modifies the Device3 based on the actual available devices and just disables the parts that are not there. And it actually works. And since we were deciding how to actually boot our system, usual way some years ago was that you created your own boot command and you specified where is your kernel, how is it named and where is your stuff on your device. But since distributions came up with this distribute command, we decided that we are going to use it and it's actually the default command that is run by Mox. And we are using this simple UBoot script that is actually booting our Mox and we have it saved in the root of partition which is one of the options that distribute tries. So if you have our system, then it will boot through normal distribute. If you have something else, which we, according to our documentation, we forbid that because if you run some other distribution, we cannot guarantee that it is within the specification and certifications and stuff like that. But if you accidentally decide to go against our documentation, then we have a distribute command. So it should boot any normal distribution and we have DTB in NOR, so it should work pretty nicely. So that's how we get to the kernel and to start some system. Then during the first boot of our distribution, which is open-lib reality-based, we do some first setup and we detect what modules are there and based on that, we decide what is actually the one and one, one of the problems that I spoke about in the beginning. We decided that if there is SFP, then it's one. If there are some switches, there are local network and the tricky part is the CPU port, the port that you have on the CPU module. It can be your connection to the Internet if you don't have SFP and if you have some other local network port. But also it can be a connection to your local network if you don't have any other port that you would use for that or if there is some other one port like SFP. So that's the first setup. In the following boots, we check on the boot whether user modified the configuration and if we edit some new switches, we put them into local network. Apart from that, we do almost no changes and we just remove switches that are no longer there. We don't touch the rest because it might either confuse the user or it might even put him at some security risk. If you imagine that user just buys a new SFP and plugs it in and if we would just change the one port, then maybe he didn't put the cable from ISP into SFP yet. So we would just bridge it with the local network. That's something that we don't want to do. Hopefully he still have some LAN ports and hopefully he can reconfigure which port is internet connection and which port is local network. We have a web user interface that allows you to do that. If the user is completely lost, he can always do a factory reset and go through the first configuration which automatically detects what should be where. That's for the easy part when we have just a router and want to decide what to do with it and want to decide what is on the local network and what is internet and what should we trust and what should we don't. Now the little bit more interesting part, although I think that the trick that our colleague did in Uboot is pretty interesting and pretty nice. He actually had a talk about that on FOSDEM so there is a recording of if you are interested in more details of that part. Why we decided to do network boot? We wanted to have some centrally managed APs so you would have your central router and you would configure something on it and then you will have some slave devices that you will just toss around and they will automatically do everything. It's in general central management of configuration. There are big projects that do that. It could be hard to set up and we just wanted to cheat a little bit and took the easy way out. The easiest way to do the central management is if you have well-defined version of OS that is running all those devices and if you are in control of what is installed there and if you can easily deploy configuration there. We decided that network boot will actually help us quite a lot because there is no local information stored on those devices that are booting over the network. Everything is always taken from the central router and it can just be nice and easy and always consistent. So how did we do it? As you remember when I was talking about this boot command I said that it tries various ways to get to the kernel and to figure out what is to boot actually. Well, if everything fails the last step is fixie boot and that's what we actually use. On Omnia we downloaded the specially prepared net boot root FS. A little bit about our distribution. We have packages as most of the distributions but we also have a few tar balls containing root file systems that are actually built out of those packages. One of them is root FS for the net boot. It's just ordinary root FS with preselected list of packages. No special configuration, nothing. What we do is we take out from net root FS kernel and DTB and so device tree and one special thing that is an interim disk. It's also again in one package but we have a special interim disk directly tailored for this net booting functionality. It contains script that does a few things and contains some basic drivers, basic kernel modules that we need. We create a fit image out of everything. A fit image is a nice thing that UBoot supports. You basically take your kernel, interim disk and device tree. You can take multiple of each and you put it into one file that UBoot can directly boot. It can contain checksums of all those parts. It can... All those parts could be signed and it can be... All those parts could be also compressed. So we take out the kernel and interim disk and device tree and create one file that contains everything to make it easy, to have just one file that we need to transfer over the network. We could do it with kernel that has embedded in the interim disk but the tricky part is that we want to use packages and standard root FS and we need the kernel compiled in the same way with same options as in the normal distributions. We don't want to maintain various kernel images. So that's why we are taking it from root FS and doing fit image. And then we prepare everything on Omnia for booting over Pixi. So we prepare the FTP server there and we let Moxboot... One thing that we need to take care of is to make sure that there is some security involved at some point because Mox will need access to some sensitive information like your keys for your Wi-Fi and stuff like that and potentially more. You don't want to display this information to everybody who is on your network. So what does this interim disk that we created, especially for the netboot, does? When it starts, it will generate SSH keys on Mox and then it will connect to Omnia and send over the public part of the key with its own serial number. And on Omnia, it will pop up in web user interface or you can list it on command line and you will see that there is a request from some device to be trusted and that you can decide whether you trust it or not. The serial number is there to help you decide if you have multiple Mox's, which one is it and if it is the new device that you just bought. When you approve it, as part of the approval process, Omnia will generate some random key for you as well for the Mox and Mox will download it. Now that it's trusted, it can use much more commands over from the Omnia and once it gets the IS key, it will save both the private SSH key and the IS key, we will see why we have the IS key later, to Uboot environment and it will also override the Uboot environment in such a way that on the next boot, Mox will no longer try Generic Disturb Boot but it will go directly for Pixi and will, well, not Generic Pixi, it will go directly to TFTP boot, download kernel image tailored specifically for this device and try to boot that. And once it sets up the Uboot, it will continue with normal booting, gets root FS, unpack it and starts. On following boots, when we start it, the Mox will contact the DHCP server and ask only for one file that has embedded its serial number into its name and that file is actually encrypted using the key that got exchanged during the handshake and Mox will download it, decrypt it because it has a key saved in Uboot environment and if it works, then it will boot this image. And again, use SSH to get root FS, unpack it into RAM, boot it and we have it running. It will also use SSH to get some initial Wi-Fi configuration, some SSL certificates that are used later on for the central management or remote management and sets up the remote access actually and it periodically also checks whether there is some reason to start over. For example, if the Omnia gets a download new root FS with some updates, with some security fixes or if the configuration of network boot changes or if it seems that the network is down and not working anymore, it will just reboot itself because that's the easiest way to recover and when it reboots, it will re-download all the configuration. For example, what can happen is that you decide to change your network IP addresses and you move to a whole different range and if you do that, then Mox wouldn't be able to communicate with the controlling router so it will decide to just panic, reboot itself and on the next boot from the HCP, it will get an IP address from the new range and new updated configuration and everything will work again. So we are using SSH a lot there. When we were designing it, we were thinking about what to use as a protocol but we decided to go with SSH for the initial setup because it's really easy to abuse it and it's also quite standard and well trusted and proven. So what we are doing is when the Mox is booting first time for the handshake, we pass on some private SSH key that is allowed to execute just one command on the Omnia and basically just send, I don't remember the exact limit but just a few bytes of the public key and that's everything that is allowed with that key that is sent during the generic pixie boot. To do that, we are using authorized keys. Just a question to the audience, who of you ever used authorized key for something else than just adding your key to have basically access to everything? Yeah, basically one of the features that quite some people don't know about is that authorized key is basically, it's not just a list of keys that have authorization to SSH but you can enforce some commands and you can associate commands with the keys and you can also add some additional options to every key that is logging in to your SSH server. Yeah, and that's what we are actually using there. I think it's heavily used by Gitulite to implement Git server. So the key that we are distributing to every Mox that tries to boot over the network, it's forced to use just one command and it has only the ability to store limited amount of data and nothing else. Once it's authorized, we get the public key that Mox created, that Mox send us and we add it to authorized keys and we allow it to use more commands but still really restricted environment. It cannot do any commands it wants. It can just run a few options of the script that we are forcing it to use. So it can just get its key, it can get some configuration, get root FS but it cannot access any random files on the router, it cannot get some network forwarding on and stuff like that. So that's a really good part that we are abusing. If you didn't know about it till now, I really suggest you to take a look at that. Just take a look at the month page for authorized keys and you will see that there is plenty of interesting features that you can abuse as well. So now I spoke about some configuration. We wrote some simple script that deploys the configuration when the router finally boots and this is the configuration that we have. That's the options that we are presenting to the advanced users. You can define some Wi-Fi networks that have some SSID and some key. If you don't define or if you use these magic keywords, it will pick the first network that you have configured in your Omnia and use SSID and key from there. You can override configuration per Wi-Fi card. This is actually vendor ID and product ID. You can override it also per MOX. This is actually the serial number of the device. So you can decide that this device will be broadcasting on specific channel. What we have there by default is we have actually two options in the device for Wi-Fi cards. One is SDI01, a little bit slower. We are using that one by default on 2.4. We have also the more powerful one that can do 5GHz AC and quite fast. So we are using that one on 5GHz. By default, we are using automatic configuration, which is not the automatic configuration if you set it up in Wi-Fi, but it's our. It basically picks a random channel. No. Finding a good channel is hard. The question for the recording, how our auto feature works and whether it tries to be clever. The answer is no. We decided to go with a dumb solution for now because getting the correct answer with channel to pick is hard because some APs might be down, some APs might be up. Actually, if we do this network configuration, if we will try to be clever and we will start, we will change something in the configuration of the moxies that are connected to the central router, so the Wi-Fi will be down, they will boot up, they will scan the environment and found the best channel to use and because all of them are booting up at the same time and using the same algorithm, they will pick the same channel, which is not the solution that you want. We decided to, by default, just do it randomly and hopefully it will work. If you find out what's the typical usage of the channels in which place is and which channel might be the most well suited one to use, then you can override it manually and decide by yourself. Okay. So, we also have some other options, how to customize the network booting because, all, we like to play with stuff and when we were designing the script, there were some really low hanging fruit that seemed really useful. So, you can provide custom root FS, you can create some overlay with few files that will get injected into root FS before booting. That might be handy if you don't want to have a net booted AP, but net booted network attached storage or something like that. We also, as I said, as I was speaking about the Wi-Fi setup, it's done basically by one simple script and that script can be also customized. You can replace it wholly or you can just add a few bits that will do something after we create the Wi-Fi configuration. So, some conclusion. Something to remember, SSH is really great and with authorized keys, it can be abused to do a lot of stuff and if you don't want to design your own protocol and do everything from scratch and if you are using just shell script, you can design your protocols over SSH and it's really simple, easy to do and yeah, it should be secure. Another interesting takeout from this presentation, hopefully, could be that U-boot is also quite great and flexible and you can do plenty of crazy stuff with that and it's scriptable and fit image if you haven't heard about it until now, it's also something nice and interesting to look at. So, thank you for your attention and if you have some questions, let's get to them. Okay? We have actual point, pointers to the actual documentation where I can read up of fit images and stuff. Point is to actual documentation about fit image, I don't know. I hope it should be on U-boot documentation somewhere and I don't have a network now. But yeah, if you take a look at the U-boot source repository, there is a directory with several text files and there is some documentation about fit image, what it actually is. Or other option is you can take a look at our netboot scripts because we are doing open source routers, so you can take a look at our scripts and see how we are using it. But basically, you create something that looks like a device tree, it has similar syntax and you put the addresses where everything should be, you put there if it is kernel, intram disk or DTB and then you call some make image with this file and it will fix everything, calculate checksums and create one binary file that you can use to boot. That's the simplest usage. I was wondering about this part with the serial number which doesn't seem very secure to me, so for instance, many of the middle attacker could intercept this request and just forward the serial number and get the traffic, did you think about some way? So suppose the attacker of course cannot directly interact with the new router that you put on the network but maybe has access to the network itself. So did you think about the... There's the weak spot. We solved it by our documentation. We explicitly stated in our documentation that you should disconnect other devices when you are doing the parent procedure. We were thinking about how to do it... Yeah, there are always two things, so you want to do it secure and user friendly. So we haven't found a way that would be user friendly but this setup is secure if you can trust your local network and basically make sure that during the pairing you will disconnect other devices or that if you make sure that nobody can intercept your communication during the handshake which can be done that basically you take the device that you are pairing, you will connect it directly to the device you are pairing it with and after pairing you will take it away and put it somewhere far away. But I mean you could also think about I don't know putting in some USB drive with a key to the device that you are pairing and taking it apart, taking it out after you complete the pairing, right? Yeah, in theory. Yeah, maybe we will implement something like that in future. What we already have is flashing of the rotors by themselves via the USB drive and yeah saving some stuff in your environment shouldn't be that hard. But initializing the pairing procedure would be a little bit harder than... Yeah, it would require some skills on the user side. This setup we were trying to make it as simple as possible for people to actually follow. Okay, some more questions? If not, then thank you for your attention. If you have some more questions later on, we have a boot in Mensa area and we have our devices there so you can come and put them together and apart again and play with our brakes. So yeah, thank you again. Bye.
 So, hi everybody. Thank you for coming. And my name is Michal Hrushecki. I'm from a company  called CZ-NIC and more specifically from the Therese project. And I'm going to speak here  about a new router that we created. And we created a modular router. And we decided that  we want to boot it and boot it over the network. And I'm going to talk about challenges that  we were facing and how we went around to make it actually happen.  So a few words about who we are. CZ-NIC is Czech Toplevel Domain Registry. Legally, we  are some association of companies, but in fact we are working like a non-profit. And  apart from managing Czech Toplevel Domain, we do quite some open source development.  You might know us for doing not DNS alternative server or DNS resolver for doing BERT, BGP  devoting daemon. And we also do some general good, mainly targeting Czech Republic, like  education about internet and security and stuff like that. And one of the things that  we do is also running Czech national CZ-NIC team. That's actually how this project started.  We didn't just decide that we are going to do routers. The original idea was to figure  out what are the type of attacks on the ordinary users at home. So basically we wanted to know  who's attacking them, how often, and what are the attackers trying to do. So to do that,  we created the first router because we needed some device to run it on. And the routers  at that time were unable to basically do what we needed to do. And we gave it away for free  and ran our security search on top of it. And when we started talking about that on  various conferences, when we were presenting the results of the research, we found out  that people like the open hardware and open software and having a root account on their  device and being able to trust their devices and stuff like that. So we created a commercially  available option, which was Tris Omnia. We ran some Indigo campaign. We got the money.  We made the router. And it's now selling in retail and it's selling. And it's again open  source out of the box with the root account and everything. And when we did that, we got  some feedback from our users that they don't need SFP that we had there, that they don't  need Ethernet ports because they have just Wi-Fi. Then they need more Ethernet ports  and they don't need Wi-Fi. And they need more USB trees and stuff like that. So many conflicting  requests. And that's how Mox came to being trying to make everybody happy. We created  the modular router. And to show you how it works, we actually created some configuration  web page where you get the CPU and then you decide what else you need. Where do you need  some Ethernet ports? You just put some Ethernet ports. You need more. Let's put some more.  Let's put some SFP in. And that's all nice. And you can do it even with the stuff that  you buy in retail. And the generic idea is that if you need something special and if  you have well-defined parameters that you want to achieve, you will just figure out  the combination of components that will allow you to do so. And you buy it and you assemble  it at home and you have a custom device as an end customer. Yeah, the idea is great.  But there are a few challenges. Like what are actual components in there? From software  side, we need to know what's in. And we also need to create some default configuration.  So which port is actually going to your ISP? You might have SFP, but you might not have  it. You can have up to 25 Ethernet ports. So which one would you pick as a one? And on  the other hand, you might have just one Ethernet port. So is it one? Is it one? And what should  we do when somebody buys a new module? He already had a router running and now he extended  it and added more ports or took away some ports. How to handle that? And one of the  things that we really wanted to do with this was also to allow people to basically extend  their Wi-Fi coverage if they have some big house. So we wanted to create some setup where  you can use the Mox as a simple AP and take configuration from your old Omnia and just  get extended Wi-Fi coverage. That sounded like interesting and useful use case, but  it poses another set of challenges. So how booting works in general? When the CPU starts,  it runs Uboot that is stored in NOR and Uboot initializes some hardware. Uboot executes  Distribute, which is a special command that is nowadays pretty much standardized across  everybody. Well, everybody likes to use it. It's called Distribute because Distributions  agrees that it's useful command. What it does is basically it tries several different ways  how to boot your hardware. It tries EFI, it tries some configuration files, it tries Uboot  Include file and if everything fails, it tries to boot over network using PXE. And then Uboot  loads the kernel, Device3. Device3 is something that is quite common in those embedded devices.  It basically describes the hardware that you have on the device because you have a kernel,  you have a virus drivers, but then you need something to tell you how all the stuff is  connected together and where it is and if it is there. And then Uboot continues with running  the kernel and passes through some command line, interim disk, this Device3 and stuff like that.  That's the generic embedded booting. We use this way, but we do some specific stuff because  of our modularity. We have some SPI bus on which we can enumerate all the available modules.  So in the Uboot, we use this SPI bus to figure out what modules are there and how they are  connected to pass this information on. Then one thing that we did after discussions with  some distributions was that we actually put our Device3 in normal memory that we have on  the device itself as well. Why did we do that? Well, kernel has plenty of drivers and usually  some CPUs are supported pretty fast, but so it's possible that you actually have support  for some of the devices that we have in Mox even in your distribution kernel, but what  you are missing and why you wouldn't be able to boot it actually is the Device3. So we  took our Device3 and put it into NOR. So if your distribution has all the necessary drivers  and doesn't have a Device3, it will use the Device3 that is available in NOR. What we  did regarding the modularity, the Device3 describes how all the hardware is connected,  but we would need quite some variations of Device3s because there is plenty of options  how to actually connect the stuff together. If I take a look at the configurator, you  see in just one way, but something like this is possible as well. So there is a lot of  ways how to actually connect stuff together. Our kernel guy came up with the solution that  basically we have a Device3 that contains all possible, it contains all the devices that  could be connected and it connects them and then when UBoot actually runs the kernel,  just before it runs the kernel, he modifies the Device3 based on the actual available  devices and just disables the parts that are not there. And it actually works. And since  we were deciding how to actually boot our system, usual way some years ago was that  you created your own boot command and you specified where is your kernel, how is it  named and where is your stuff on your device. But since distributions came up with this  distribute command, we decided that we are going to use it and it's actually the default  command that is run by Mox. And we are using this simple UBoot script that is actually  booting our Mox and we have it saved in the root of partition which is one of the options  that distribute tries. So if you have our system, then it will boot through normal distribute.  If you have something else, which we, according to our documentation, we forbid that because  if you run some other distribution, we cannot guarantee that it is within the specification  and certifications and stuff like that. But if you accidentally decide to go against our  documentation, then we have a distribute command. So it should boot any normal distribution  and we have DTB in NOR, so it should work pretty nicely. So that's how we get to the  kernel and to start some system. Then during the first boot of our distribution, which  is open-lib reality-based, we do some first setup and we detect what modules are there  and based on that, we decide what is actually the one and one, one of the problems that  I spoke about in the beginning. We decided that if there is SFP, then it's one. If there  are some switches, there are local network and the tricky part is the CPU port, the port  that you have on the CPU module. It can be your connection to the Internet if you don't  have SFP and if you have some other local network port. But also it can be a connection  to your local network if you don't have any other port that you would use for that or if  there is some other one port like SFP. So that's the first setup. In the following boots,  we check on the boot whether user modified the configuration and if we edit some new  switches, we put them into local network. Apart from that, we do almost no changes and  we just remove switches that are no longer there. We don't touch the rest because it  might either confuse the user or it might even put him at some security risk. If you  imagine that user just buys a new SFP and plugs it in and if we would just change the  one port, then maybe he didn't put the cable from ISP into SFP yet. So we would just bridge  it with the local network. That's something that we don't want to do. Hopefully he still  have some LAN ports and hopefully he can reconfigure which port is internet connection and which  port is local network. We have a web user interface that allows you to do that. If the  user is completely lost, he can always do a factory reset and go through the first configuration  which automatically detects what should be where. That's for the easy part when we have  just a router and want to decide what to do with it and want to decide what is on the  local network and what is internet and what should we trust and what should we don't.  Now the little bit more interesting part, although I think that the trick that our colleague  did in Uboot is pretty interesting and pretty nice. He actually had a talk about that on  FOSDEM so there is a recording of if you are interested in more details of that part.  Why we decided to do network boot? We wanted to have some centrally managed APs so you  would have your central router and you would configure something on it and then you will  have some slave devices that you will just toss around and they will automatically do  everything. It's in general central management of configuration. There are big projects that  do that. It could be hard to set up and we just wanted to cheat a little bit and took  the easy way out. The easiest way to do the central management is if you have well-defined  version of OS that is running all those devices and if you are in control of what is installed  there and if you can easily deploy configuration there. We decided that network boot will actually  help us quite a lot because there is no local information stored on those devices that are  booting over the network. Everything is always taken from the central router and it can just  be nice and easy and always consistent. So how did we do it? As you remember when I was  talking about this boot command I said that it tries various ways to get to the kernel  and to figure out what is to boot actually. Well, if everything fails the last step is  fixie boot and that's what we actually use. On Omnia we downloaded the specially prepared  net boot root FS. A little bit about our distribution. We have packages as most of the distributions  but we also have a few tar balls containing root file systems that are actually built  out of those packages. One of them is root FS for the net boot. It's just ordinary root  FS with preselected list of packages. No special configuration, nothing. What we do is we take  out from net root FS kernel and DTB and so device tree and one special thing that is  an interim disk. It's also again in one package but we have a special interim disk directly  tailored for this net booting functionality. It contains script that does a few things  and contains some basic drivers, basic kernel modules that we need. We create a fit image  out of everything. A fit image is a nice thing that UBoot supports. You basically take your  kernel, interim disk and device tree. You can take multiple of each and you put it into  one file that UBoot can directly boot. It can contain checksums of all those parts.  It can... All those parts could be signed and it can be... All those parts could be also  compressed. So we take out the kernel and interim disk and device tree and create one file that  contains everything to make it easy, to have just one file that we need to transfer over  the network. We could do it with kernel that has embedded  in the interim disk but the tricky part is that we want to use packages and standard  root FS and we need the kernel compiled in the same way with same options as in the normal  distributions. We don't want to maintain various kernel images. So that's why we are taking  it from root FS and doing fit image. And then we prepare everything on Omnia for  booting over Pixi. So we prepare the FTP server there and we let Moxboot...  One thing that we need to take care of is to make sure that there is some security involved  at some point because Mox will need access to some sensitive information like your keys  for your Wi-Fi and stuff like that and potentially more. You don't want to display this information  to everybody who is on your network. So what does this interim disk that we created, especially  for the netboot, does? When it starts, it will generate SSH keys on Mox and then it  will connect to Omnia and send over the public part of the key with its own serial number.  And on Omnia, it will pop up in web user interface or you can list it on command line and you  will see that there is a request from some device to be trusted and that you can decide  whether you trust it or not. The serial number is there to help you decide if you have multiple  Mox's, which one is it and if it is the new device that you just bought. When you approve  it, as part of the approval process, Omnia will generate some random key for you as well  for the Mox and Mox will download it. Now that it's trusted, it can use much more commands  over from the Omnia and once it gets the IS key, it will save both the private SSH key  and the IS key, we will see why we have the IS key later, to Uboot environment and it  will also override the Uboot environment in such a way that on the next boot, Mox will  no longer try Generic Disturb Boot but it will go directly for Pixi and will, well, not  Generic Pixi, it will go directly to TFTP boot, download kernel image tailored specifically  for this device and try to boot that. And once it sets up the Uboot, it will continue  with normal booting, gets root FS, unpack it and starts. On following boots, when we  start it, the Mox will contact the DHCP server and ask only for one file that has embedded  its serial number into its name and that file is actually encrypted using the key that got  exchanged during the handshake and Mox will download it, decrypt it because it has a key  saved in Uboot environment and if it works, then it will boot this image. And again, use  SSH to get root FS, unpack it into RAM, boot it and we have it running. It will also use  SSH to get some initial Wi-Fi configuration, some SSL certificates that are used later  on for the central management or remote management and sets up the remote access actually and  it periodically also checks whether there is some reason to start over. For example, if  the Omnia gets a download new root FS with some updates, with some security fixes or if  the configuration of network boot changes or if it seems that the network is down and  not working anymore, it will just reboot itself because that's the easiest way to recover  and when it reboots, it will re-download all the configuration. For example, what can happen  is that you decide to change your network IP addresses and you move to a whole different  range and if you do that, then Mox wouldn't be able to communicate with the controlling  router so it will decide to just panic, reboot itself and on the next boot from the HCP,  it will get an IP address from the new range and new updated configuration and everything  will work again. So we are using SSH a lot there. When we were designing it, we were  thinking about what to use as a protocol but we decided to go with SSH for the initial  setup because it's really easy to abuse it and it's also quite standard and well trusted  and proven. So what we are doing is when the Mox is booting first time for the handshake,  we pass on some private SSH key that is allowed to execute just one command on the Omnia and  basically just send, I don't remember the exact limit but just a few bytes of the public  key and that's everything that is allowed with that key that is sent during the generic  pixie boot. To do that, we are using authorized keys. Just a question to the audience, who  of you ever used authorized key for something else than just adding your key to have basically  access to everything? Yeah, basically one of the features that quite  some people don't know about is that authorized key is basically, it's not just a list of  keys that have authorization to SSH but you can enforce some commands and you can associate  commands with the keys and you can also add some additional options to every key that  is logging in to your SSH server. Yeah, and that's what we are actually using there. I  think it's heavily used by Gitulite to implement Git server. So the key that we are distributing  to every Mox that tries to boot over the network, it's forced to use just one command and it  has only the ability to store limited amount of data and nothing else. Once it's authorized,  we get the public key that Mox created, that Mox send us and we add it to authorized keys  and we allow it to use more commands but still really restricted environment. It cannot  do any commands it wants. It can just run a few options of the script that we are forcing  it to use. So it can just get its key, it can get some configuration, get root FS but  it cannot access any random files on the router, it cannot get some network forwarding on and  stuff like that. So that's a really good part that we are abusing. If you didn't know about  it till now, I really suggest you to take a look at that. Just take a look at the month  page for authorized keys and you will see that there is plenty of interesting features  that you can abuse as well. So now I spoke about some configuration. We wrote some simple  script that deploys the configuration when the router finally boots and this is the configuration  that we have. That's the options that we are presenting to the advanced users. You can  define some Wi-Fi networks that have some SSID and some key. If you don't define or  if you use these magic keywords, it will pick the first network that you have configured  in your Omnia and use SSID and key from there. You can override configuration per Wi-Fi card.  This is actually vendor ID and product ID. You can override it also per MOX. This is actually  the serial number of the device. So you can decide that this device will be broadcasting  on specific channel. What we have there by default is we have actually two options in  the device for Wi-Fi cards. One is SDI01, a little bit slower. We are using that one  by default on 2.4. We have also the more powerful one that can do 5GHz AC and quite fast. So  we are using that one on 5GHz. By default, we are using automatic configuration, which  is not the automatic configuration if you set it up in Wi-Fi, but it's our. It basically  picks a random channel. No. Finding a good channel is hard. The question for the recording,  how our auto feature works and whether it tries to be clever. The answer is no. We decided  to go with a dumb solution for now because getting the correct answer with channel to  pick is hard because some APs might be down, some APs might be up. Actually, if we do this  network configuration, if we will try to be clever and we will start, we will change something  in the configuration of the moxies that are connected to the central router, so the Wi-Fi  will be down, they will boot up, they will scan the environment and found the best channel  to use and because all of them are booting up at the same time and using the same algorithm,  they will pick the same channel, which is not the solution that you want. We decided to,  by default, just do it randomly and hopefully it will work. If you find out what's the typical  usage of the channels in which place is and which channel might be the most well suited  one to use, then you can override it manually and decide by yourself.  Okay. So, we also have some other options, how to customize the network booting because,  all, we like to play with stuff and when we were designing the script, there were some  really low hanging fruit that seemed really useful. So, you can provide custom root FS,  you can create some overlay with few files that will get injected into root FS before  booting. That might be handy if you don't want to have a net booted AP, but net booted  network attached storage or something like that. We also, as I said, as I was speaking  about the Wi-Fi setup, it's done basically by one simple script and that script can be  also customized. You can replace it wholly or you can just add a few bits that will do  something after we create the Wi-Fi configuration. So, some conclusion. Something to remember,  SSH is really great and with authorized keys, it can be abused to do a lot of stuff and  if you don't want to design your own protocol and do everything from scratch and if you  are using just shell script, you can design your protocols over SSH and it's really simple,  easy to do and yeah, it should be secure. Another interesting takeout from this presentation,  hopefully, could be that U-boot is also quite great and flexible and you can do plenty of  crazy stuff with that and it's scriptable and fit image if you haven't heard about it  until now, it's also something nice and interesting to look at. So, thank you for your attention  and if you have some questions, let's get to them. Okay?  We have actual point, pointers to the actual documentation where I can read up of fit  images and stuff. Point is to actual documentation about fit image, I don't know. I hope it should  be on U-boot documentation somewhere and I don't have a network now. But yeah, if you  take a look at the U-boot source repository, there is a directory with several text files  and there is some documentation about fit image, what it actually is. Or other option  is you can take a look at our netboot scripts because we are doing open source routers,  so you can take a look at our scripts and see how we are using it. But basically, you  create something that looks like a device tree, it has similar syntax and you put the  addresses where everything should be, you put there if it is kernel, intram disk or  DTB and then you call some make image with this file and it will fix everything, calculate  checksums and create one binary file that you can use to boot. That's the simplest usage.  I was wondering about this part with the serial number which doesn't seem very secure to me,  so for instance, many of the middle attacker could intercept this request and just forward  the serial number and get the traffic, did you think about some way? So suppose the attacker  of course cannot directly interact with the new router that you put on the network but  maybe has access to the network itself. So did you think about the...  There's the weak spot. We solved it by our documentation. We explicitly stated in our  documentation that you should disconnect other devices when you are doing the parent procedure.  We were thinking about how to do it... Yeah, there are always two things, so you want to  do it secure and user friendly. So we haven't found a way that would be user friendly but  this setup is secure if you can trust your local network and basically make sure that  during the pairing you will disconnect other devices or that if you make sure that nobody  can intercept your communication during the handshake which can be done that basically  you take the device that you are pairing, you will connect it directly to the device  you are pairing it with and after pairing you will take it away and put it somewhere  far away. But I mean you could also think about I don't know putting in some USB drive  with a key to the device that you are pairing and taking it apart, taking it out after you  complete the pairing, right? Yeah, in theory. Yeah, maybe we will implement something like  that in future. What we already have is flashing of the rotors by themselves via the USB drive  and yeah saving some stuff in your environment shouldn't be that hard. But initializing the  pairing procedure would be a little bit harder than... Yeah, it would require some skills  on the user side. This setup we were trying to make it as simple as possible for people  to actually follow. Okay, some more questions? If not, then thank you for your attention.  If you have some more questions later on, we have a boot in Mensa area and we have our  devices there so you can come and put them together and apart again and play with our  brakes. So yeah, thank you again. Bye.
2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_0_59.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_60_179.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_180_2651.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_2652_3407.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_3408_6167.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_6168_6647.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_6648_7187.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_7188_12527.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_12528_12623.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_12624_12815.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_12816_16427.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_16428_16619.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_16620_18155.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_18156_18287.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_18288_19151.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_19152_19631.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_19632_19883.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_19884_20291.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_20292_22847.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_22848_22943.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_22944_23711.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_23712_24455.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_24456_24527.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_24528_26075.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_26076_26663.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_26664_26759.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_26760_27119.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_27120_27947.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_27948_28055.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_28056_28151.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_28152_28391.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_28392_29243.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_29244_29519.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_29520_30887.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_30888_31391.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_31392_31727.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_31728_38543.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_38544_38939.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_38940_40031.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_40032_42527.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_42528_42899.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_42900_47255.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_47256_49259.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_49260_51071.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_51072_51215.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_51216_51275.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_51276_53963.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_53964_55151.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_55152_55967.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_55968_56879.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_56880_57227.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_57228_57431.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_57432_57707.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_57708_57983.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_57984_58919.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_58920_58967.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_58968_59039.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_59040_59159.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_59160_61163.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_61164_61751.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_61752_65495.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_65496_65783.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_65784_65831.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_65832_66083.png 2019_-_Booting_modular_open_source_router_j8YM-QTaQT4_66084_66295.png
So, hi everybody. Thank you for coming. And my name is Michal Hrushecki. I'm from a company called CZ-NIC and more specifically from the Therese project. And I'm going to speak here about a new router that we created. And we created a modular router. And we decided that we want to boot it and boot it over the network. And I'm going to talk about challenges that we were facing and how we went around to make it actually happen. So a few words about who we are. CZ-NIC is Czech Toplevel Domain Registry. Legally, we are some association of companies, but in fact we are working like a non-profit. And apart from managing Czech Toplevel Domain, we do quite some open source development. You might know us for doing not DNS alternative server or DNS resolver for doing BERT, BGP devoting daemon. And we also do some general good, mainly targeting Czech Republic, like education about internet and security and stuff like that. And one of the things that we do is also running Czech national CZ-NIC team. That's actually how this project started. We didn't just decide that we are going to do routers. The original idea was to figure out what are the type of attacks on the ordinary users at home. So basically we wanted to know who's attacking them, how often, and what are the attackers trying to do. So to do that, we created the first router because we needed some device to run it on. And the routers at that time were unable to basically do what we needed to do. And we gave it away for free and ran our security search on top of it. And when we started talking about that on various conferences, when we were presenting the results of the research, we found out that people like the open hardware and open software and having a root account on their device and being able to trust their devices and stuff like that. So we created a commercially available option, which was Tris Omnia. We ran some Indigo campaign. We got the money. We made the router. And it's now selling in retail and it's selling. And it's again open source out of the box with the root account and everything. And when we did that, we got some feedback from our users that they don't need SFP that we had there, that they don't need Ethernet ports because they have just Wi-Fi. Then they need more Ethernet ports and they don't need Wi-Fi. And they need more USB trees and stuff like that. So many conflicting requests. And that's how Mox came to being trying to make everybody happy. We created the modular router. And to show you how it works, we actually created some configuration web page where you get the CPU and then you decide what else you need. Where do you need some Ethernet ports? You just put some Ethernet ports. You need more. Let's put some more. Let's put some SFP in. And that's all nice. And you can do it even with the stuff that you buy in retail. And the generic idea is that if you need something special and if you have well-defined parameters that you want to achieve, you will just figure out the combination of components that will allow you to do so. And you buy it and you assemble it at home and you have a custom device as an end customer. Yeah, the idea is great. But there are a few challenges. Like what are actual components in there? From software side, we need to know what's in. And we also need to create some default configuration. So which port is actually going to your ISP? You might have SFP, but you might not have it. You can have up to 25 Ethernet ports. So which one would you pick as a one? And on the other hand, you might have just one Ethernet port. So is it one? Is it one? And what should we do when somebody buys a new module? He already had a router running and now he extended it and added more ports or took away some ports. How to handle that? And one of the things that we really wanted to do with this was also to allow people to basically extend their Wi-Fi coverage if they have some big house. So we wanted to create some setup where you can use the Mox as a simple AP and take configuration from your old Omnia and just get extended Wi-Fi coverage. That sounded like interesting and useful use case, but it poses another set of challenges. So how booting works in general? When the CPU starts, it runs Uboot that is stored in NOR and Uboot initializes some hardware. Uboot executes Distribute, which is a special command that is nowadays pretty much standardized across everybody. Well, everybody likes to use it. It's called Distribute because Distributions agrees that it's useful command. What it does is basically it tries several different ways how to boot your hardware. It tries EFI, it tries some configuration files, it tries Uboot Include file and if everything fails, it tries to boot over network using PXE. And then Uboot loads the kernel, Device3. Device3 is something that is quite common in those embedded devices. It basically describes the hardware that you have on the device because you have a kernel, you have a virus drivers, but then you need something to tell you how all the stuff is connected together and where it is and if it is there. And then Uboot continues with running the kernel and passes through some command line, interim disk, this Device3 and stuff like that. That's the generic embedded booting. We use this way, but we do some specific stuff because of our modularity. We have some SPI bus on which we can enumerate all the available modules. So in the Uboot, we use this SPI bus to figure out what modules are there and how they are connected to pass this information on. Then one thing that we did after discussions with some distributions was that we actually put our Device3 in normal memory that we have on the device itself as well. Why did we do that? Well, kernel has plenty of drivers and usually some CPUs are supported pretty fast, but so it's possible that you actually have support for some of the devices that we have in Mox even in your distribution kernel, but what you are missing and why you wouldn't be able to boot it actually is the Device3. So we took our Device3 and put it into NOR. So if your distribution has all the necessary drivers and doesn't have a Device3, it will use the Device3 that is available in NOR. What we did regarding the modularity, the Device3 describes how all the hardware is connected, but we would need quite some variations of Device3s because there is plenty of options how to actually connect the stuff together. If I take a look at the configurator, you see in just one way, but something like this is possible as well. So there is a lot of ways how to actually connect stuff together. Our kernel guy came up with the solution that basically we have a Device3 that contains all possible, it contains all the devices that could be connected and it connects them and then when UBoot actually runs the kernel, just before it runs the kernel, he modifies the Device3 based on the actual available devices and just disables the parts that are not there. And it actually works. And since we were deciding how to actually boot our system, usual way some years ago was that you created your own boot command and you specified where is your kernel, how is it named and where is your stuff on your device. But since distributions came up with this distribute command, we decided that we are going to use it and it's actually the default command that is run by Mox. And we are using this simple UBoot script that is actually booting our Mox and we have it saved in the root of partition which is one of the options that distribute tries. So if you have our system, then it will boot through normal distribute. If you have something else, which we, according to our documentation, we forbid that because if you run some other distribution, we cannot guarantee that it is within the specification and certifications and stuff like that. But if you accidentally decide to go against our documentation, then we have a distribute command. So it should boot any normal distribution and we have DTB in NOR, so it should work pretty nicely. So that's how we get to the kernel and to start some system. Then during the first boot of our distribution, which is open-lib reality-based, we do some first setup and we detect what modules are there and based on that, we decide what is actually the one and one, one of the problems that I spoke about in the beginning. We decided that if there is SFP, then it's one. If there are some switches, there are local network and the tricky part is the CPU port, the port that you have on the CPU module. It can be your connection to the Internet if you don't have SFP and if you have some other local network port. But also it can be a connection to your local network if you don't have any other port that you would use for that or if there is some other one port like SFP. So that's the first setup. In the following boots, we check on the boot whether user modified the configuration and if we edit some new switches, we put them into local network. Apart from that, we do almost no changes and we just remove switches that are no longer there. We don't touch the rest because it might either confuse the user or it might even put him at some security risk. If you imagine that user just buys a new SFP and plugs it in and if we would just change the one port, then maybe he didn't put the cable from ISP into SFP yet. So we would just bridge it with the local network. That's something that we don't want to do. Hopefully he still have some LAN ports and hopefully he can reconfigure which port is internet connection and which port is local network. We have a web user interface that allows you to do that. If the user is completely lost, he can always do a factory reset and go through the first configuration which automatically detects what should be where. That's for the easy part when we have just a router and want to decide what to do with it and want to decide what is on the local network and what is internet and what should we trust and what should we don't. Now the little bit more interesting part, although I think that the trick that our colleague did in Uboot is pretty interesting and pretty nice. He actually had a talk about that on FOSDEM so there is a recording of if you are interested in more details of that part. Why we decided to do network boot? We wanted to have some centrally managed APs so you would have your central router and you would configure something on it and then you will have some slave devices that you will just toss around and they will automatically do everything. It's in general central management of configuration. There are big projects that do that. It could be hard to set up and we just wanted to cheat a little bit and took the easy way out. The easiest way to do the central management is if you have well-defined version of OS that is running all those devices and if you are in control of what is installed there and if you can easily deploy configuration there. We decided that network boot will actually help us quite a lot because there is no local information stored on those devices that are booting over the network. Everything is always taken from the central router and it can just be nice and easy and always consistent. So how did we do it? As you remember when I was talking about this boot command I said that it tries various ways to get to the kernel and to figure out what is to boot actually. Well, if everything fails the last step is fixie boot and that's what we actually use. On Omnia we downloaded the specially prepared net boot root FS. A little bit about our distribution. We have packages as most of the distributions but we also have a few tar balls containing root file systems that are actually built out of those packages. One of them is root FS for the net boot. It's just ordinary root FS with preselected list of packages. No special configuration, nothing. What we do is we take out from net root FS kernel and DTB and so device tree and one special thing that is an interim disk. It's also again in one package but we have a special interim disk directly tailored for this net booting functionality. It contains script that does a few things and contains some basic drivers, basic kernel modules that we need. We create a fit image out of everything. A fit image is a nice thing that UBoot supports. You basically take your kernel, interim disk and device tree. You can take multiple of each and you put it into one file that UBoot can directly boot. It can contain checksums of all those parts. It can... All those parts could be signed and it can be... All those parts could be also compressed. So we take out the kernel and interim disk and device tree and create one file that contains everything to make it easy, to have just one file that we need to transfer over the network. We could do it with kernel that has embedded in the interim disk but the tricky part is that we want to use packages and standard root FS and we need the kernel compiled in the same way with same options as in the normal distributions. We don't want to maintain various kernel images. So that's why we are taking it from root FS and doing fit image. And then we prepare everything on Omnia for booting over Pixi. So we prepare the FTP server there and we let Moxboot... One thing that we need to take care of is to make sure that there is some security involved at some point because Mox will need access to some sensitive information like your keys for your Wi-Fi and stuff like that and potentially more. You don't want to display this information to everybody who is on your network. So what does this interim disk that we created, especially for the netboot, does? When it starts, it will generate SSH keys on Mox and then it will connect to Omnia and send over the public part of the key with its own serial number. And on Omnia, it will pop up in web user interface or you can list it on command line and you will see that there is a request from some device to be trusted and that you can decide whether you trust it or not. The serial number is there to help you decide if you have multiple Mox's, which one is it and if it is the new device that you just bought. When you approve it, as part of the approval process, Omnia will generate some random key for you as well for the Mox and Mox will download it. Now that it's trusted, it can use much more commands over from the Omnia and once it gets the IS key, it will save both the private SSH key and the IS key, we will see why we have the IS key later, to Uboot environment and it will also override the Uboot environment in such a way that on the next boot, Mox will no longer try Generic Disturb Boot but it will go directly for Pixi and will, well, not Generic Pixi, it will go directly to TFTP boot, download kernel image tailored specifically for this device and try to boot that. And once it sets up the Uboot, it will continue with normal booting, gets root FS, unpack it and starts. On following boots, when we start it, the Mox will contact the DHCP server and ask only for one file that has embedded its serial number into its name and that file is actually encrypted using the key that got exchanged during the handshake and Mox will download it, decrypt it because it has a key saved in Uboot environment and if it works, then it will boot this image. And again, use SSH to get root FS, unpack it into RAM, boot it and we have it running. It will also use SSH to get some initial Wi-Fi configuration, some SSL certificates that are used later on for the central management or remote management and sets up the remote access actually and it periodically also checks whether there is some reason to start over. For example, if the Omnia gets a download new root FS with some updates, with some security fixes or if the configuration of network boot changes or if it seems that the network is down and not working anymore, it will just reboot itself because that's the easiest way to recover and when it reboots, it will re-download all the configuration. For example, what can happen is that you decide to change your network IP addresses and you move to a whole different range and if you do that, then Mox wouldn't be able to communicate with the controlling router so it will decide to just panic, reboot itself and on the next boot from the HCP, it will get an IP address from the new range and new updated configuration and everything will work again. So we are using SSH a lot there. When we were designing it, we were thinking about what to use as a protocol but we decided to go with SSH for the initial setup because it's really easy to abuse it and it's also quite standard and well trusted and proven. So what we are doing is when the Mox is booting first time for the handshake, we pass on some private SSH key that is allowed to execute just one command on the Omnia and basically just send, I don't remember the exact limit but just a few bytes of the public key and that's everything that is allowed with that key that is sent during the generic pixie boot. To do that, we are using authorized keys. Just a question to the audience, who of you ever used authorized key for something else than just adding your key to have basically access to everything? Yeah, basically one of the features that quite some people don't know about is that authorized key is basically, it's not just a list of keys that have authorization to SSH but you can enforce some commands and you can associate commands with the keys and you can also add some additional options to every key that is logging in to your SSH server. Yeah, and that's what we are actually using there. I think it's heavily used by Gitulite to implement Git server. So the key that we are distributing to every Mox that tries to boot over the network, it's forced to use just one command and it has only the ability to store limited amount of data and nothing else. Once it's authorized, we get the public key that Mox created, that Mox send us and we add it to authorized keys and we allow it to use more commands but still really restricted environment. It cannot do any commands it wants. It can just run a few options of the script that we are forcing it to use. So it can just get its key, it can get some configuration, get root FS but it cannot access any random files on the router, it cannot get some network forwarding on and stuff like that. So that's a really good part that we are abusing. If you didn't know about it till now, I really suggest you to take a look at that. Just take a look at the month page for authorized keys and you will see that there is plenty of interesting features that you can abuse as well. So now I spoke about some configuration. We wrote some simple script that deploys the configuration when the router finally boots and this is the configuration that we have. That's the options that we are presenting to the advanced users. You can define some Wi-Fi networks that have some SSID and some key. If you don't define or if you use these magic keywords, it will pick the first network that you have configured in your Omnia and use SSID and key from there. You can override configuration per Wi-Fi card. This is actually vendor ID and product ID. You can override it also per MOX. This is actually the serial number of the device. So you can decide that this device will be broadcasting on specific channel. What we have there by default is we have actually two options in the device for Wi-Fi cards. One is SDI01, a little bit slower. We are using that one by default on 2.4. We have also the more powerful one that can do 5GHz AC and quite fast. So we are using that one on 5GHz. By default, we are using automatic configuration, which is not the automatic configuration if you set it up in Wi-Fi, but it's our. It basically picks a random channel. No. Finding a good channel is hard. The question for the recording, how our auto feature works and whether it tries to be clever. The answer is no. We decided to go with a dumb solution for now because getting the correct answer with channel to pick is hard because some APs might be down, some APs might be up. Actually, if we do this network configuration, if we will try to be clever and we will start, we will change something in the configuration of the moxies that are connected to the central router, so the Wi-Fi will be down, they will boot up, they will scan the environment and found the best channel to use and because all of them are booting up at the same time and using the same algorithm, they will pick the same channel, which is not the solution that you want. We decided to, by default, just do it randomly and hopefully it will work. If you find out what's the typical usage of the channels in which place is and which channel might be the most well suited one to use, then you can override it manually and decide by yourself. Okay. So, we also have some other options, how to customize the network booting because, all, we like to play with stuff and when we were designing the script, there were some really low hanging fruit that seemed really useful. So, you can provide custom root FS, you can create some overlay with few files that will get injected into root FS before booting. That might be handy if you don't want to have a net booted AP, but net booted network attached storage or something like that. We also, as I said, as I was speaking about the Wi-Fi setup, it's done basically by one simple script and that script can be also customized. You can replace it wholly or you can just add a few bits that will do something after we create the Wi-Fi configuration. So, some conclusion. Something to remember, SSH is really great and with authorized keys, it can be abused to do a lot of stuff and if you don't want to design your own protocol and do everything from scratch and if you are using just shell script, you can design your protocols over SSH and it's really simple, easy to do and yeah, it should be secure. Another interesting takeout from this presentation, hopefully, could be that U-boot is also quite great and flexible and you can do plenty of crazy stuff with that and it's scriptable and fit image if you haven't heard about it until now, it's also something nice and interesting to look at. So, thank you for your attention and if you have some questions, let's get to them. Okay? We have actual point, pointers to the actual documentation where I can read up of fit images and stuff. Point is to actual documentation about fit image, I don't know. I hope it should be on U-boot documentation somewhere and I don't have a network now. But yeah, if you take a look at the U-boot source repository, there is a directory with several text files and there is some documentation about fit image, what it actually is. Or other option is you can take a look at our netboot scripts because we are doing open source routers, so you can take a look at our scripts and see how we are using it. But basically, you create something that looks like a device tree, it has similar syntax and you put the addresses where everything should be, you put there if it is kernel, intram disk or DTB and then you call some make image with this file and it will fix everything, calculate checksums and create one binary file that you can use to boot. That's the simplest usage. I was wondering about this part with the serial number which doesn't seem very secure to me, so for instance, many of the middle attacker could intercept this request and just forward the serial number and get the traffic, did you think about some way? So suppose the attacker of course cannot directly interact with the new router that you put on the network but maybe has access to the network itself. So did you think about the... There's the weak spot. We solved it by our documentation. We explicitly stated in our documentation that you should disconnect other devices when you are doing the parent procedure. We were thinking about how to do it... Yeah, there are always two things, so you want to do it secure and user friendly. So we haven't found a way that would be user friendly but this setup is secure if you can trust your local network and basically make sure that during the pairing you will disconnect other devices or that if you make sure that nobody can intercept your communication during the handshake which can be done that basically you take the device that you are pairing, you will connect it directly to the device you are pairing it with and after pairing you will take it away and put it somewhere far away. But I mean you could also think about I don't know putting in some USB drive with a key to the device that you are pairing and taking it apart, taking it out after you complete the pairing, right? Yeah, in theory. Yeah, maybe we will implement something like that in future. What we already have is flashing of the rotors by themselves via the USB drive and yeah saving some stuff in your environment shouldn't be that hard. But initializing the pairing procedure would be a little bit harder than... Yeah, it would require some skills on the user side. This setup we were trying to make it as simple as possible for people to actually follow. Okay, some more questions? If not, then thank you for your attention. If you have some more questions later on, we have a boot in Mensa area and we have our devices there so you can come and put them together and apart again and play with our brakes. So yeah, thank you again. Bye.
created a modular router. And we decided that we want to boot it and boot it over the network. And facing and how we went around to make it actually in. And we also need to create some default setup where you can use the Mox as a simple AP command, we decided that we are going to use it local network. Apart from that, we do almost no netboot scripts because we are doing open source solved it by our documentation. We explicitly user side. This setup we were trying to make it