Bringing NixOS to my school
Schmitt, Marc
EPITA, a French school of engineers in computer science, is bringing NixOS to its students. Here is a presentation of the challenges of maintaining an infrastructure of more than 800 machines used by students. The presentation will start with a current state of our learning infrastructure running on Arch Linux, the challenges we face with it, the up- and downsides of migrating to NixOS and what has already been done.
 Okay, the next talk up is bringing next to us to my school by Mark Schmidt. And this presentation will start with a current state of the learning infrastructure running on Arch Linux and the challenges that we faced with it, ups and downs, signs of migrating to next to us and what has already been done. So I believe the school that this is about is school that marco's to EPI TA, I believe is pronounced EPI TA, a French school of engineers in computer science. And it's bringing next to us to its students. So it's going to be a presentation sort of showing the challenges of maintaining infrastructure of more than 800 machines used by students. So they started using next to us a year and a half ago when they were in California for a school trip. That's pretty cool. And they wanted a way to like manage config with like it. And I know a lot of people like just like I'm a mark tried Ansible, then then found next to us and I will say that next to us is much better than Ansible for this kind of thing. And he is also an organizer at prologin, the French national computer science contest. And a cool thing we share in common is that well, I don't play the same restaurant by a musician is he played by, he plays viola and he's done that for 14 years now. Okay, take it away. Right, so first I'd like to thank the organizers for their work. What they're doing is terrific. So as you said, we'll have peace. I am a student that's at my fourth year. I'm part of a laboratory, which is called the tree, which stands for central resource informatics, which in English means roughly IT resources center. So we're also a school departments in charge of managing the educational IT needs. What that means is anything from hosting Moodle to managing the computer rooms, which is what I will be talking about today. Right, so we'll skip over that. So Epita is still a school that has computer rooms much much like this one. This photo actually is not accurate because now we have screens that stand on top of the tables and that aren't inside and we are using Intel NUX as our computers that are attached behind the screen. It's very, very convenient. However, the tree does not manage everything in those computer rooms. Indeed, Epita is part of a group of school called UNIS that tries to mutualize needs, especially in IT. The department responsible for those is called the Bokel and they manage everything from phones, office 365 accounts, Wi-Fi and our computer rooms network. This means that we actually have some impose requirements when managing the computers in those. Our computers used to not have any disk, which meant that we had to boot the machines via network. However, our computer rooms only have a one gig uplink for each computer room. And since we have 13 computer rooms across six campuses in France, that would just kill our solar room uplink. That's also one gigabyte if we booted all the machines at the same time. So what we are doing is basically having our in-it RAM FS that downloads the root FS using a Aria 2, so BitSorrent. So we don't have to actually download everything from each machine from our server room. So as I said, we have 13 computer rooms that adds up to about 800 to 1000 machines, which probably makes it the biggest IT infrastructure running Arch Linux in France, maybe Europe. And to give students some persistence on those machines for the files when they're working on it, we're using Kerberos for authentication on Open AFS. For those who don't know what Open AFS is, it's basically a network file system that you can log in using Kerberos. Only you can see your files, not like in NFS. We also actually have several images that the computers can boot from. We do that because depending on what year you're in at Eputa, you're going to be working with different software and we don't want to include all the software you use that Eputa in one image. It would just be too big. Right now our images are about 2GB size, so all the software would be way too much. So currently our boot process goes as this. So the machines are configured to Pixi by default. They get their IP from the switches in the computer rooms, which forward the HTTP request to our servers. Then in this HTTP request, our servers tell the computers to download IPXE from our servers again using TFTP. Then IPXE queries a menu from a homemade service that actually manages IPXE menus, which allows us to select a default image for a room depending on what year, what promotion of student is going to be using it. And then once the image has been selected, IPXE will then download the kernel and in its RAMFS via HTTP from our S3 servers. And then in its RAMFS, we'll download a torrent file via HTTPS again. It finds what torrent file to download from the kernel command line. And then in this torrent, there is a rootFS-quashFS, which it downloads and then it mounts it and switch routes on it. If the computer has a disk, so our old computers didn't have a disk, but now we have one in our internal look, and we can actually do some boot cache. So we have a partition in which we store the torrent file and the contents of that torrent. And so the machine doesn't have to actually redownload the image if it hasn't changed and it had downloaded it before. Also once the machine is booted up, there's a service that seeds those images. So a booted up image is capable of seeding its own image and also any other images that it would have been downloaded before. Okay so how do we build an image? So using Arch Linux, as I said previously, we use Drakot for our In-N-Tar D because it's quite convenient, easily customizable, and it just works. We use Salt to manage our configuration and what packages are going to be installed on the machine or in the image. And then some custom tools including Arch Creator, which is basically a script to bootstrap an image. So the first step is what you would do in any classic Arch Linux installation, that's packstrap a root FS with some basic software. The second step we do is we install Salt. We actually have to pin this Salt version for a very simple reason. The current Salt version in Arch Linux repositories is broken, so we have to install our own. And then we have to patch system, well actually system CTL because the latest version of Salt doesn't support the latest version of system D that's shipped with Arch Linux. So we actually have to hack the output of system CTL-Version to remove that version and replace it with a version supported by Salt. Otherwise, Salt will just not start. Which is what we do just right here. Salt call and the important part is this, state of high state. What that does is Salt is going to create Saltmaster, which then looks in a Git repository on the master branch and applies a configuration on your current system, which in this case is a root FS, so we're inside a shoot right here. And then we undo our hack, we install some kernel modules, we generate our initRT using drag cuts, and then we create a squash FS with everything that we've done just here. Okay, so this setup actually has a few problems. First it's not reproducible for a very simple reason. Every time we start an image build, we don't know what packages are going to be installed because well, Arch Linux is a rolling release, so they just release packages whenever they want. So we would just install what's latest in the Arch Linux repositories. We could actually pin all those packages to a version, but there would just be too much maintenance for not much result. So we just hope that every time we rebuild an image, everything just goes fine. Salt is a pain because of what we've seen just before. It's also very hard to test the changes we do on these images. Right now we have two ways of testing changes. The first one is to push on a branch on our soltrapestory, build the image. We usually have an image called Arch Linux test for this purpose. So once the image is built, which takes in somewhere between 20 to 60 minutes, we can actually start a machine using this image, downloading the image depending on where you are in the school, how good is your connection can take between 2 minutes to 15 minutes. So you're in for about 30 to an hour and 30 minutes before you can actually test your changes. So it's very hard to iterate on any configuration changes that you are doing. The second way we have to test changes is actually to commit on our soltrapestory on master because of course, and then go on an already booted up machine and run solt called state.highstates, which is going to query the configuration it should have from its solt master and then apply it on the system. And then we can test it, which is pretty fine, but again pushing on master not great. And you can test anything if you make any changes to a system D unit that gets loaded on startup, you can test that, for example. CI takes forever as I said, and we also don't have any packages cache, which is probably why the CI takes forever, because we have no time to implement it. As far as I know, it's not very much trivial. So there's quite a running gag in our team that says, if we switch to NixOS, it's going to solve all our problems and everything is going to be fine. Everything is going to be fine just right after that. And what we, well, I actually decided to go and find out. So I mainly got inspired from the netboots configuration in Nix packages. And then what I did is basically hack on it. You can find everything here. The two things I needed to add on top of the netboots configuration was torrent downloading, which is basically putting aria2 in the initrd and then launching the command here. And add boot cache support, which is basically checking if a partition exists. I do right here. If it does, you mount it. If it fails, we just mount a tmpfs so the image can still seed while the torrent file. Right. You might wonder, how do I get the init file? In a standard NixOS installation, you would have a kernel command line argument called init that equals to something like slash nix slash tor slash hash, nixOS system slash init. But the thing is to add that in our current setup, I would have to have an API route on our service that manages images to upload, to update this path. And I just couldn't be bothered. So I did something that's a bit of a hack and frankly not quite satisfying. What I do when I'm creating my squashfs, so the closure info, that's basically everything that has to go in the image store contents is a list that contains config.system.build.topLevel. So I have that, I have to put the nix path registration also in my squashfs. And then when the system boots up, I have to run nix store loaddb and everything is in there. And then I echo config.system.build.topLevel slash init. That's going to evaluate to something like slash nix slash tor slash hash dash nixOS slash system dash whatever slash init which I put here. And I echo that in a stage2 init file. This stage2 init files end up in the squashfs. And then in my init.rd when I am done mounting everything, which means I have my nix store mounted and just right before I switch to stage2, I actually export this variable stage2init which is the variable that's usually set when the stage1 parses the kernel command line arguments. If there's init, it will set stage2init to whatever value is provided. And here I just set it to what would be in slash nix slash tor slash stage2init which comes from here. There's a better way of doing this actually. I would have to generate an IPXE script like that's what's done in the netbook module in Nix packages which is actually the right way of doing it. And then upload that IPXE script also to our S3 where things get downloaded from. And our service that manages images instead of providing a kernel and install.j to download it should just chain load on that script in our S3. I have to test it but it should work and it probably will be a better solution than this. So let's go over the list of the problems we had before and see if we solved some. Not reproducible. Well, Nix helps a lot with that since we can pin our packages which is the main problem we had before. However, my images are not yet fully reproducible. I noticed that when doing some tests to see if I could just upload an image on S3 even if it hadn't been updated. And some bits actually change between the two images I built at separate times. I haven't had time to investigate it but the only thing it adds up is that we don't have to have the manual step of going into our GitLab UI and start the job that uploads the images to the S3 which we want to have anyway so we just don't care about this. So we're pinning Nix packages using Flakes so we are only upgrading packages when we want. Salt is no longer paying because we are now only using it to actually run commands on several machines across a computer room and not using it to build a system anymore. So that's better. It's not hard to test changes anymore because you can just run a Nix build, config.system.build.vm and launch a VM which takes about a minute which is much better than the R we would take you before and you can iterate on hacking configuration pretty fast using that method. The VM is actually a bit broken right now because I won't accept my netboot configuration but I just have to override that output and disable the netboot. But detail. We now have a CI which only builds the top level configuration and not what I call the netboot top level which includes the squashFS which takes some time. So the CI is pretty fast because it's only building a Nix-VS configuration. Some packages does not take a very long time. It allows you to have some feedback when you push on the repository if your configuration is valid and builds. The squashFS part of the build which is what takes most of the time of the build is separated into another job that's launched manually as I said before and also uploads the images to our S3. And we also now have a binary cache that we set up in a few minutes so we can cache our packages and our configuration. Now let's go over what that provides to our students because what we've seen right now until then is what it helps us to do. Let's see what it allows our students to do. First they can reuse the configuration. For example our Kerberos and OpenFS configuration is exported as part of a Nix-VS module in our Flake which means they can import it in their configuration and use it from there. So they don't have to reconfigure all the servers and everything from their configuration. They can just use ours and if we update it they will be updated up to date with it. Students can also install the same packages we have at school which means they'll have the same environment as they would have at school which means it's also the same environment that's used in the test when we automatically test their code to grade it. And then they can, so on the commutators at school install new packages. Right now the only way they have is to download the source and compile it from there. Now because Nix is a multi-user they can actually install packages. So what's done, as I said, Netboot works. Kerberos and LDAP configuration and thus OpenFS support also works. I already have included some packages needed for development by the students. I think there's only Python right now. I have to add a bunch more but that's for later. And the CI for the images deployment works too. What's left to do is some extensive testing. I've never used them before, I have to see how they work, will they integrate with our workflow and that's some more work left to do. I have to write some introductory blog posts for the students that already worked one. Explaining from the very basics what's a package manager, what's its role, what does it do, how does it do it for a classic one and then how Nix does it. I have to do some team training because currently in our team we only have two people proficient with Nix and NixOS. But if you want to use that in production as a long-term solution, well, we have to train the whole team so they get on board and then know how it works. And then I have to write a small script to live update the images. So with Sol you could do a state.high state which you can't do anymore. So I have to find a way to update the images while they're running which is pretty simple, I guess. And then what's next for this project? It will be on a one-year trial so the student can test it, they can provide us with feedback and we'll see how it goes and maybe we'll do a full switch to NixOS. And then the last item here is just some blue sky insanity for me. Maybe I would be able to somehow share the Nix store between all the machines in a room. I don't know how I could do it, maybe using IPFS, I don't know how that works. So if one student installed a package and another one installed the same package, they could just download it from the other computer instead of downloading it from cache.nixos.org. All right, I guess that's all from me. If there are any questions, I'll be happy to answer them. If you let me just switch to the JITSEE room. They just rebooted. Yep. Live streaming is on. Okay, everyone, sorry about that. We had a little slight technical difficulty with the stream and the JITSEE room being cancelled but we ironed that out. So this now enters into the Q&A portion of this talk and we do have, I think, three questions to which are actually from the speaker who's going to be speaking right up next. So first one is, have you tried to bring NixOS to the routers? Is there a plan to do so? Sorry, to the router, you said? Yeah, to the routers. Okay, so we don't have actually any plans for that. The only plan right now is to use it for the computers used by the students. Okay, gotcha. And what kind of network setup does EP to have? Or runs? Well, what do you mean network? Like, okay, so, well, it's quite complicated. Basically, it's a mess. We have a computer room has a net. Our server room also has a subnet. The book I spoke of at the beginning of my talk actually routes them for us. So I don't exactly what goes on behind the scenes on their infrastructure because I'm not working. And then the only network we have to do is the one in our server room. So that's pretty basic. Even though we have some weird stuff going on for adding between the LAN and all that stuff. I'm not sure that is a question. I have my eye on the, oh yeah, okay. I have my next con QA channel if you want to. Okay. So, yes, there is isolation. Probably, but again, that's not what we do. That's the job of the book out to do that. Right. And someone just said you should not tell that to strangers. Okay. The next question we have is what and if there is CI, what is it? Is it being used? So we're using GitLab's CI because that's what was used up until then. We're not really planning to use Hercules or Hydra because, well, that just doesn't fit in our flow currently. I'm not quite sure there's a way for them to automatically upload the outputs of builds. Like not the path in the next store, but actually it's in it. So I don't want to, you know, when you next build you have a results sim link. I don't want to upload what's behind that result in link, but I want to upload only results slash, let's say, nixOS-test.nitrg and nixOS-test.squashfs and nixOS-test.colonel. I want to upload these three files fully to three with the same name. So I actually change my image. So we're not currently planning on any nix-focused CI system. Right. I've used Cache 6 for some things like that. I'm not sure if it's entirely possible to do what you're doing, though. It might be worth looking into maybe momentarily. Okay. So I don't think I see any more questions in the channel. Right. Yeah. I think that concludes the Q&A session. Thank you so much. Thanks to you.
 Okay, the next talk up is bringing next to us to my school by Mark Schmidt.  And this presentation will start with a current state of the learning infrastructure running  on Arch Linux and the challenges that we faced with it, ups and downs, signs of migrating  to next to us and what has already been done.  So I believe the school that this is about is school that marco's to EPI TA, I believe  is pronounced EPI TA, a French school of engineers in computer science.  And it's bringing next to us to its students.  So it's going to be a presentation sort of showing the challenges of maintaining infrastructure  of more than 800 machines used by students.  So they started using next to us a year and a half ago when they were in California for  a school trip.  That's pretty cool.  And they wanted a way to like manage config with like it.  And I know a lot of people like just like I'm a mark tried Ansible, then then found  next to us and I will say that next to us is much better than Ansible for this kind  of thing.  And he is also an organizer at prologin, the French national computer science contest.  And a cool thing we share in common is that well, I don't play the same restaurant by  a musician is he played by, he plays viola and he's done that for 14 years now.  Okay, take it away.  Right, so first I'd like to thank the organizers for their work.  What they're doing is terrific.  So as you said, we'll have peace.  I am a student that's at my fourth year.  I'm part of a laboratory, which is called the tree, which stands for central resource  informatics, which in English means roughly IT resources center.  So we're also a school departments in charge of managing the educational IT needs.  What that means is anything from hosting Moodle to managing the computer rooms, which is what  I will be talking about today.  Right, so we'll skip over that.  So Epita is still a school that has computer rooms much much like this one.  This photo actually is not accurate because now we have screens that stand on top of the  tables and that aren't inside and we are using Intel NUX as our computers that are attached  behind the screen.  It's very, very convenient.  However, the tree does not manage everything in those computer rooms.  Indeed, Epita is part of a group of school called UNIS that tries to mutualize needs,  especially in IT.  The department responsible for those is called the Bokel and they manage everything from  phones, office 365 accounts, Wi-Fi and our computer rooms network.  This means that we actually have some impose requirements when managing the computers in  those.  Our computers used to not have any disk, which meant that we had to boot the machines via  network.  However, our computer rooms only have a one gig uplink for each computer room.  And since we have 13 computer rooms across six campuses in France, that would just kill  our solar room uplink.  That's also one gigabyte if we booted all the machines at the same time.  So what we are doing is basically having our in-it RAM FS that downloads the root FS  using a Aria 2, so BitSorrent.  So we don't have to actually download everything from each machine from our server room.  So as I said, we have 13 computer rooms that adds up to about 800 to 1000 machines, which  probably makes it the biggest IT infrastructure running Arch Linux in France, maybe Europe.  And to give students some persistence on those machines for the files when they're working  on it, we're using Kerberos for authentication on Open AFS.  For those who don't know what Open AFS is, it's basically a network file system that  you can log in using Kerberos.  Only you can see your files, not like in NFS.  We also actually have several images that the computers can boot from.  We do that because depending on what year you're in at Eputa, you're going to be working  with different software and we don't want to include all the software you use that  Eputa in one image.  It would just be too big.  Right now our images are about 2GB size, so all the software would be way too much.  So currently our boot process goes as this.  So the machines are configured to Pixi by default.  They get their IP from the switches in the computer rooms, which forward the HTTP request  to our servers.  Then in this HTTP request, our servers tell the computers to download IPXE from our servers  again using TFTP.  Then IPXE queries a menu from a homemade service that actually manages IPXE menus,  which allows us to select a default image for a room depending on what year, what promotion  of student is going to be using it.  And then once the image has been selected, IPXE will then download the kernel and in  its RAMFS via HTTP from our S3 servers.  And then in its RAMFS, we'll download a torrent file via HTTPS again.  It finds what torrent file to download from the kernel command line.  And then in this torrent, there is a rootFS-quashFS, which it downloads and then it mounts it  and switch routes on it.  If the computer has a disk, so our old computers didn't have a disk, but now we have one in  our internal look, and we can actually do some boot cache.  So we have a partition in which we store the torrent file and the contents of that torrent.  And so the machine doesn't have to actually redownload the image if it hasn't changed  and it had downloaded it before.  Also once the machine is booted up, there's a service that seeds those images.  So a booted up image is capable of seeding its own image and also any other images that  it would have been downloaded before.  Okay so how do we build an image?  So using Arch Linux, as I said previously, we use Drakot for our In-N-Tar D because it's  quite convenient, easily customizable, and it just works.  We use Salt to manage our configuration and what packages are going to be installed on  the machine or in the image.  And then some custom tools including Arch Creator, which is basically a script to bootstrap  an image.  So the first step is what you would do in any classic Arch Linux installation, that's  packstrap a root FS with some basic software.  The second step we do is we install Salt.  We actually have to pin this Salt version for a very simple reason.  The current Salt version in Arch Linux repositories is broken, so we have to install our own.  And then we have to patch system, well actually system CTL because the latest version of Salt  doesn't support the latest version of system D that's shipped with Arch Linux.  So we actually have to hack the output of system CTL-Version to remove that version  and replace it with a version supported by Salt.  Otherwise, Salt will just not start.  Which is what we do just right here.  Salt call and the important part is this, state of high state.  What that does is Salt is going to create Saltmaster, which then looks in a Git repository  on the master branch and applies a configuration on your current system, which in this case  is a root FS, so we're inside a shoot right here.  And then we undo our hack, we install some kernel modules, we generate our initRT using  drag cuts, and then we create a squash FS with everything that we've done just here.  Okay, so this setup actually has a few problems.  First it's not reproducible for a very simple reason.  Every time we start an image build, we don't know what packages are going to be installed  because well, Arch Linux is a rolling release, so they just release packages whenever they  want.  So we would just install what's latest in the Arch Linux repositories.  We could actually pin all those packages to a version, but there would just be too much  maintenance for not much result.  So we just hope that every time we rebuild an image, everything just goes fine.  Salt is a pain because of what we've seen just before.  It's also very hard to test the changes we do on these images.  Right now we have two ways of testing changes.  The first one is to push on a branch on our soltrapestory, build the image.  We usually have an image called Arch Linux test for this purpose.  So once the image is built, which takes in somewhere between 20 to 60 minutes, we can  actually start a machine using this image, downloading the image depending on where you  are in the school, how good is your connection can take between 2 minutes to 15 minutes.  So you're in for about 30 to an hour and 30 minutes before you can actually test your  changes.  So it's very hard to iterate on any configuration changes that you are doing.  The second way we have to test changes is actually to commit on our soltrapestory on  master because of course, and then go on an already booted up machine and run solt called  state.highstates, which is going to query the configuration it should have from its  solt master and then apply it on the system.  And then we can test it, which is pretty fine, but again pushing on master not great.  And you can test anything if you make any changes to a system D unit that gets loaded  on startup, you can test that, for example.  CI takes forever as I said, and we also don't have any packages cache, which is probably  why the CI takes forever, because we have no time to implement it.  As far as I know, it's not very much trivial.  So there's quite a running gag in our team that says, if we switch to NixOS, it's going  to solve all our problems and everything is going to be fine.  Everything is going to be fine just right after that.  And what we, well, I actually decided to go and find out.  So I mainly got inspired from the netboots configuration in Nix packages.  And then what I did is basically hack on it.  You can find everything here.  The two things I needed to add on top of the netboots configuration was torrent downloading,  which is basically putting aria2 in the initrd and then launching the command here.  And add boot cache support, which is basically checking if a partition exists.  I do right here.  If it does, you mount it.  If it fails, we just mount a tmpfs so the image can still seed while the torrent file.  Right.  You might wonder, how do I get the init file?  In a standard NixOS installation, you would have a kernel command line argument called  init that equals to something like slash nix slash tor slash hash, nixOS system slash  init.  But the thing is to add that in our current setup, I would have to have an API route on  our service that manages images to upload, to update this path.  And I just couldn't be bothered.  So I did something that's a bit of a hack and frankly not quite satisfying.  What I do when I'm creating my squashfs, so the closure info, that's basically everything  that has to go in the image store contents is a list that contains config.system.build.topLevel.  So I have that, I have to put the nix path registration also in my squashfs.  And then when the system boots up, I have to run nix store loaddb and everything is  in there.  And then I echo config.system.build.topLevel slash init.  That's going to evaluate to something like slash nix slash tor slash hash dash nixOS  slash system dash whatever slash init which I put here.  And I echo that in a stage2 init file.  This stage2 init files end up in the squashfs.  And then in my init.rd when I am done mounting everything, which means I have my nix store  mounted and just right before I switch to stage2, I actually export this variable stage2init  which is the variable that's usually set when the stage1 parses the kernel command  line arguments.  If there's init, it will set stage2init to whatever value is provided.  And here I just set it to what would be in slash nix slash tor slash stage2init which  comes from here.  There's a better way of doing this actually.  I would have to generate an IPXE script like that's what's done in the netbook module in  Nix packages which is actually the right way of doing it.  And then upload that IPXE script also to our S3 where things get downloaded from.  And our service that manages images instead of providing a kernel and install.j to download  it should just chain load on that script in our S3.  I have to test it but it should work and it probably will be a better solution than this.  So let's go over the list of the problems we had before and see if we solved some.  Not reproducible.  Well, Nix helps a lot with that since we can pin our packages which is the main problem  we had before.  However, my images are not yet fully reproducible.  I noticed that when doing some tests to see if I could just upload an image on S3 even  if it hadn't been updated.  And some bits actually change between the two images I built at separate times.  I haven't had time to investigate it but the only thing it adds up is that we don't have  to have the manual step of going into our GitLab UI and start the job that uploads the  images to the S3 which we want to have anyway so we just don't care about this.  So we're pinning Nix packages using Flakes so we are only upgrading packages when we  want.  Salt is no longer paying because we are now only using it to actually run commands on  several machines across a computer room and not using it to build a system anymore.  So that's better.  It's not hard to test changes anymore because you can just run a Nix build, config.system.build.vm  and launch a VM which takes about a minute which is much better than the R we would take  you before and you can iterate on hacking configuration pretty fast using that method.  The VM is actually a bit broken right now because I won't accept my netboot configuration  but I just have to override that output and disable the netboot.  But detail.  We now have a CI which only builds the top level configuration and not what I call the  netboot top level which includes the squashFS which takes some time.  So the CI is pretty fast because it's only building a Nix-VS configuration.  Some packages does not take a very long time.  It allows you to have some feedback when you push on the repository if your configuration  is valid and builds.  The squashFS part of the build which is what takes most of the time of the build is separated  into another job that's launched manually as I said before and also uploads the images  to our S3.  And we also now have a binary cache that we set up in a few minutes so we can cache our  packages and our configuration.  Now let's go over what that provides to our students because what we've seen right now  until then is what it helps us to do.  Let's see what it allows our students to do.  First they can reuse the configuration.  For example our Kerberos and OpenFS configuration is exported as part of a Nix-VS module in  our Flake which means they can import it in their configuration and use it from there.  So they don't have to reconfigure all the servers and everything from their configuration.  They can just use ours and if we update it they will be updated up to date with it.  Students can also install the same packages we have at school which means they'll have  the same environment as they would have at school which means it's also the same environment  that's used in the test when we automatically test their code to grade it.  And then they can, so on the commutators at school install new packages.  Right now the only way they have is to download the source and compile it from there.  Now because Nix is a multi-user they can actually install packages.  So what's done, as I said, Netboot works.  Kerberos and LDAP configuration and thus OpenFS support also works.  I already have included some packages needed for development by the students.  I think there's only Python right now.  I have to add a bunch more but that's for later.  And the CI for the images deployment works too.  What's left to do is some extensive testing.  I've never used them before, I have to see how they work, will they integrate with our  workflow and that's some more work left to do.  I have to write some introductory blog posts for the students that already worked one.  Explaining from the very basics what's a package manager, what's its role, what does it do,  how does it do it for a classic one and then how Nix does it.  I have to do some team training because currently in our team we only have two people proficient  with Nix and NixOS.  But if you want to use that in production as a long-term solution, well, we have to train  the whole team so they get on board and then know how it works.  And then I have to write a small script to live update the images.  So with Sol you could do a state.high state which you can't do anymore.  So I have to find a way to update the images while they're running which is pretty simple,  I guess.  And then what's next for this project?  It will be on a one-year trial so the student can test it, they can provide us with feedback  and we'll see how it goes and maybe we'll do a full switch to NixOS.  And then the last item here is just some blue sky insanity for me.  Maybe I would be able to somehow share the Nix store between all the machines in a room.  I don't know how I could do it, maybe using IPFS, I don't know how that works.  So if one student installed a package and another one installed the same package, they  could just download it from the other computer instead of downloading it from cache.nixos.org.  All right, I guess that's all from me.  If there are any questions, I'll be happy to answer them.  If you let me just switch to the JITSEE room.  They just rebooted.  Yep.  Live streaming is on.  Okay, everyone, sorry about that.  We had a little slight technical difficulty with the stream and the JITSEE room being  cancelled but we ironed that out.  So this now enters into the Q&A portion of this talk and we do have, I think, three questions  to which are actually from the speaker who's going to be speaking right up next.  So first one is, have you tried to bring NixOS to the routers?  Is there a plan to do so?  Sorry, to the router, you said?  Yeah, to the routers.  Okay, so we don't have actually any plans for that.  The only plan right now is to use it for the computers used by the students.  Okay, gotcha.  And what kind of network setup does EP to have?  Or runs?  Well, what do you mean network?  Like, okay, so, well, it's quite complicated.  Basically, it's a mess.  We have a computer room has a net.  Our server room also has a subnet.  The book I spoke of at the beginning of my talk actually routes them for us.  So I don't exactly what goes on behind the scenes on their infrastructure because I'm  not working.  And then the only network we have to do is the one in our server room.  So that's pretty basic.  Even though we have some weird stuff going on for adding between the LAN and all that  stuff.  I'm not sure that is a question.  I have my eye on the, oh yeah, okay.  I have my next con QA channel if you want to.  Okay.  So, yes, there is isolation.  Probably, but again, that's not what we do.  That's the job of the book out to do that.  Right.  And someone just said you should not tell that to strangers.  Okay.  The next question we have is what and if there is CI, what is it?  Is it being used?  So we're using GitLab's CI because that's what was used up until then.  We're not really planning to use Hercules or Hydra because, well, that just doesn't  fit in our flow currently.  I'm not quite sure there's a way for them to automatically upload the outputs of builds.  Like not the path in the next store, but actually it's in it.  So I don't want to, you know, when you next build you have a results sim link.  I don't want to upload what's behind that result in link, but I want to upload only  results slash, let's say, nixOS-test.nitrg and nixOS-test.squashfs and nixOS-test.colonel.  I want to upload these three files fully to three with the same name.  So I actually change my image.  So we're not currently planning on any nix-focused CI system.  Right.  I've used Cache 6 for some things like that.  I'm not sure if it's entirely possible to do what you're doing, though.  It might be worth looking into maybe momentarily.  Okay.  So I don't think I see any more questions in the channel.  Right.  Yeah.  I think that concludes the Q&A session.  Thank you so much.  Thanks to you.
NixCon_2020_Vid_8_0_2015.png NixCon_2020_Vid_8_2016_2063.png NixCon_2020_Vid_8_2064_3047.png NixCon_2020_Vid_8_3048_3095.png NixCon_2020_Vid_8_3096_3191.png NixCon_2020_Vid_8_3192_4127.png NixCon_2020_Vid_8_4128_7667.png NixCon_2020_Vid_8_7668_9827.png NixCon_2020_Vid_8_9828_9959.png NixCon_2020_Vid_8_9960_10727.png NixCon_2020_Vid_8_10728_11387.png NixCon_2020_Vid_8_11388_12155.png NixCon_2020_Vid_8_12156_13967.png NixCon_2020_Vid_8_13968_17747.png NixCon_2020_Vid_8_17748_18815.png NixCon_2020_Vid_8_18816_18863.png NixCon_2020_Vid_8_18864_19199.png NixCon_2020_Vid_8_19200_19367.png NixCon_2020_Vid_8_19368_19607.png NixCon_2020_Vid_8_19608_24023.png NixCon_2020_Vid_8_24024_28307.png NixCon_2020_Vid_8_28308_30431.png NixCon_2020_Vid_8_30432_30983.png NixCon_2020_Vid_8_30984_32615.png NixCon_2020_Vid_8_32616_33791.png NixCon_2020_Vid_8_33792_33959.png NixCon_2020_Vid_8_33960_34007.png NixCon_2020_Vid_8_34008_34067.png NixCon_2020_Vid_8_34068_34235.png NixCon_2020_Vid_8_34236_34907.png NixCon_2020_Vid_8_34908_35435.png NixCon_2020_Vid_8_35436_37487.png NixCon_2020_Vid_8_37488_38567.png NixCon_2020_Vid_8_38568_38687.png NixCon_2020_Vid_8_38688_38819.png NixCon_2020_Vid_8_38820_38939.png NixCon_2020_Vid_8_38940_39563.png NixCon_2020_Vid_8_39564_39635.png NixCon_2020_Vid_8_39636_40235.png NixCon_2020_Vid_8_40236_40252.png
Okay, the next talk up is bringing next to us to my school by Mark Schmidt. And this presentation will start with a current state of the learning infrastructure running on Arch Linux and the challenges that we faced with it, ups and downs, signs of migrating to next to us and what has already been done. So I believe the school that this is about is school that marco's to EPI TA, I believe is pronounced EPI TA, a French school of engineers in computer science. And it's bringing next to us to its students. So it's going to be a presentation sort of showing the challenges of maintaining infrastructure of more than 800 machines used by students. So they started using next to us a year and a half ago when they were in California for a school trip. That's pretty cool. And they wanted a way to like manage config with like it. And I know a lot of people like just like I'm a mark tried Ansible, then then found next to us and I will say that next to us is much better than Ansible for this kind of thing. And he is also an organizer at prologin, the French national computer science contest. And a cool thing we share in common is that well, I don't play the same restaurant by a musician is he played by, he plays viola and he's done that for 14 years now. Okay, take it away. Right, so first I'd like to thank the organizers for their work. What they're doing is terrific. So as you said, we'll have peace. I am a student that's at my fourth year. I'm part of a laboratory, which is called the tree, which stands for central resource informatics, which in English means roughly IT resources center. So we're also a school departments in charge of managing the educational IT needs. What that means is anything from hosting Moodle to managing the computer rooms, which is what I will be talking about today. Right, so we'll skip over that. So Epita is still a school that has computer rooms much much like this one. This photo actually is not accurate because now we have screens that stand on top of the tables and that aren't inside and we are using Intel NUX as our computers that are attached behind the screen. It's very, very convenient. However, the tree does not manage everything in those computer rooms. Indeed, Epita is part of a group of school called UNIS that tries to mutualize needs, especially in IT. The department responsible for those is called the Bokel and they manage everything from phones, office 365 accounts, Wi-Fi and our computer rooms network. This means that we actually have some impose requirements when managing the computers in those. Our computers used to not have any disk, which meant that we had to boot the machines via network. However, our computer rooms only have a one gig uplink for each computer room. And since we have 13 computer rooms across six campuses in France, that would just kill our solar room uplink. That's also one gigabyte if we booted all the machines at the same time. So what we are doing is basically having our in-it RAM FS that downloads the root FS using a Aria 2, so BitSorrent. So we don't have to actually download everything from each machine from our server room. So as I said, we have 13 computer rooms that adds up to about 800 to 1000 machines, which probably makes it the biggest IT infrastructure running Arch Linux in France, maybe Europe. And to give students some persistence on those machines for the files when they're working on it, we're using Kerberos for authentication on Open AFS. For those who don't know what Open AFS is, it's basically a network file system that you can log in using Kerberos. Only you can see your files, not like in NFS. We also actually have several images that the computers can boot from. We do that because depending on what year you're in at Eputa, you're going to be working with different software and we don't want to include all the software you use that Eputa in one image. It would just be too big. Right now our images are about 2GB size, so all the software would be way too much. So currently our boot process goes as this. So the machines are configured to Pixi by default. They get their IP from the switches in the computer rooms, which forward the HTTP request to our servers. Then in this HTTP request, our servers tell the computers to download IPXE from our servers again using TFTP. Then IPXE queries a menu from a homemade service that actually manages IPXE menus, which allows us to select a default image for a room depending on what year, what promotion of student is going to be using it. And then once the image has been selected, IPXE will then download the kernel and in its RAMFS via HTTP from our S3 servers. And then in its RAMFS, we'll download a torrent file via HTTPS again. It finds what torrent file to download from the kernel command line. And then in this torrent, there is a rootFS-quashFS, which it downloads and then it mounts it and switch routes on it. If the computer has a disk, so our old computers didn't have a disk, but now we have one in our internal look, and we can actually do some boot cache. So we have a partition in which we store the torrent file and the contents of that torrent. And so the machine doesn't have to actually redownload the image if it hasn't changed and it had downloaded it before. Also once the machine is booted up, there's a service that seeds those images. So a booted up image is capable of seeding its own image and also any other images that it would have been downloaded before. Okay so how do we build an image? So using Arch Linux, as I said previously, we use Drakot for our In-N-Tar D because it's quite convenient, easily customizable, and it just works. We use Salt to manage our configuration and what packages are going to be installed on the machine or in the image. And then some custom tools including Arch Creator, which is basically a script to bootstrap an image. So the first step is what you would do in any classic Arch Linux installation, that's packstrap a root FS with some basic software. The second step we do is we install Salt. We actually have to pin this Salt version for a very simple reason. The current Salt version in Arch Linux repositories is broken, so we have to install our own. And then we have to patch system, well actually system CTL because the latest version of Salt doesn't support the latest version of system D that's shipped with Arch Linux. So we actually have to hack the output of system CTL-Version to remove that version and replace it with a version supported by Salt. Otherwise, Salt will just not start. Which is what we do just right here. Salt call and the important part is this, state of high state. What that does is Salt is going to create Saltmaster, which then looks in a Git repository on the master branch and applies a configuration on your current system, which in this case is a root FS, so we're inside a shoot right here. And then we undo our hack, we install some kernel modules, we generate our initRT using drag cuts, and then we create a squash FS with everything that we've done just here. Okay, so this setup actually has a few problems. First it's not reproducible for a very simple reason. Every time we start an image build, we don't know what packages are going to be installed because well, Arch Linux is a rolling release, so they just release packages whenever they want. So we would just install what's latest in the Arch Linux repositories. We could actually pin all those packages to a version, but there would just be too much maintenance for not much result. So we just hope that every time we rebuild an image, everything just goes fine. Salt is a pain because of what we've seen just before. It's also very hard to test the changes we do on these images. Right now we have two ways of testing changes. The first one is to push on a branch on our soltrapestory, build the image. We usually have an image called Arch Linux test for this purpose. So once the image is built, which takes in somewhere between 20 to 60 minutes, we can actually start a machine using this image, downloading the image depending on where you are in the school, how good is your connection can take between 2 minutes to 15 minutes. So you're in for about 30 to an hour and 30 minutes before you can actually test your changes. So it's very hard to iterate on any configuration changes that you are doing. The second way we have to test changes is actually to commit on our soltrapestory on master because of course, and then go on an already booted up machine and run solt called state.highstates, which is going to query the configuration it should have from its solt master and then apply it on the system. And then we can test it, which is pretty fine, but again pushing on master not great. And you can test anything if you make any changes to a system D unit that gets loaded on startup, you can test that, for example. CI takes forever as I said, and we also don't have any packages cache, which is probably why the CI takes forever, because we have no time to implement it. As far as I know, it's not very much trivial. So there's quite a running gag in our team that says, if we switch to NixOS, it's going to solve all our problems and everything is going to be fine. Everything is going to be fine just right after that. And what we, well, I actually decided to go and find out. So I mainly got inspired from the netboots configuration in Nix packages. And then what I did is basically hack on it. You can find everything here. The two things I needed to add on top of the netboots configuration was torrent downloading, which is basically putting aria2 in the initrd and then launching the command here. And add boot cache support, which is basically checking if a partition exists. I do right here. If it does, you mount it. If it fails, we just mount a tmpfs so the image can still seed while the torrent file. Right. You might wonder, how do I get the init file? In a standard NixOS installation, you would have a kernel command line argument called init that equals to something like slash nix slash tor slash hash, nixOS system slash init. But the thing is to add that in our current setup, I would have to have an API route on our service that manages images to upload, to update this path. And I just couldn't be bothered. So I did something that's a bit of a hack and frankly not quite satisfying. What I do when I'm creating my squashfs, so the closure info, that's basically everything that has to go in the image store contents is a list that contains config.system.build.topLevel. So I have that, I have to put the nix path registration also in my squashfs. And then when the system boots up, I have to run nix store loaddb and everything is in there. And then I echo config.system.build.topLevel slash init. That's going to evaluate to something like slash nix slash tor slash hash dash nixOS slash system dash whatever slash init which I put here. And I echo that in a stage2 init file. This stage2 init files end up in the squashfs. And then in my init.rd when I am done mounting everything, which means I have my nix store mounted and just right before I switch to stage2, I actually export this variable stage2init which is the variable that's usually set when the stage1 parses the kernel command line arguments. If there's init, it will set stage2init to whatever value is provided. And here I just set it to what would be in slash nix slash tor slash stage2init which comes from here. There's a better way of doing this actually. I would have to generate an IPXE script like that's what's done in the netbook module in Nix packages which is actually the right way of doing it. And then upload that IPXE script also to our S3 where things get downloaded from. And our service that manages images instead of providing a kernel and install.j to download it should just chain load on that script in our S3. I have to test it but it should work and it probably will be a better solution than this. So let's go over the list of the problems we had before and see if we solved some. Not reproducible. Well, Nix helps a lot with that since we can pin our packages which is the main problem we had before. However, my images are not yet fully reproducible. I noticed that when doing some tests to see if I could just upload an image on S3 even if it hadn't been updated. And some bits actually change between the two images I built at separate times. I haven't had time to investigate it but the only thing it adds up is that we don't have to have the manual step of going into our GitLab UI and start the job that uploads the images to the S3 which we want to have anyway so we just don't care about this. So we're pinning Nix packages using Flakes so we are only upgrading packages when we want. Salt is no longer paying because we are now only using it to actually run commands on several machines across a computer room and not using it to build a system anymore. So that's better. It's not hard to test changes anymore because you can just run a Nix build, config.system.build.vm and launch a VM which takes about a minute which is much better than the R we would take you before and you can iterate on hacking configuration pretty fast using that method. The VM is actually a bit broken right now because I won't accept my netboot configuration but I just have to override that output and disable the netboot. But detail. We now have a CI which only builds the top level configuration and not what I call the netboot top level which includes the squashFS which takes some time. So the CI is pretty fast because it's only building a Nix-VS configuration. Some packages does not take a very long time. It allows you to have some feedback when you push on the repository if your configuration is valid and builds. The squashFS part of the build which is what takes most of the time of the build is separated into another job that's launched manually as I said before and also uploads the images to our S3. And we also now have a binary cache that we set up in a few minutes so we can cache our packages and our configuration. Now let's go over what that provides to our students because what we've seen right now until then is what it helps us to do. Let's see what it allows our students to do. First they can reuse the configuration. For example our Kerberos and OpenFS configuration is exported as part of a Nix-VS module in our Flake which means they can import it in their configuration and use it from there. So they don't have to reconfigure all the servers and everything from their configuration. They can just use ours and if we update it they will be updated up to date with it. Students can also install the same packages we have at school which means they'll have the same environment as they would have at school which means it's also the same environment that's used in the test when we automatically test their code to grade it. And then they can, so on the commutators at school install new packages. Right now the only way they have is to download the source and compile it from there. Now because Nix is a multi-user they can actually install packages. So what's done, as I said, Netboot works. Kerberos and LDAP configuration and thus OpenFS support also works. I already have included some packages needed for development by the students. I think there's only Python right now. I have to add a bunch more but that's for later. And the CI for the images deployment works too. What's left to do is some extensive testing. I've never used them before, I have to see how they work, will they integrate with our workflow and that's some more work left to do. I have to write some introductory blog posts for the students that already worked one. Explaining from the very basics what's a package manager, what's its role, what does it do, how does it do it for a classic one and then how Nix does it. I have to do some team training because currently in our team we only have two people proficient with Nix and NixOS. But if you want to use that in production as a long-term solution, well, we have to train the whole team so they get on board and then know how it works. And then I have to write a small script to live update the images. So with Sol you could do a state.high state which you can't do anymore. So I have to find a way to update the images while they're running which is pretty simple, I guess. And then what's next for this project? It will be on a one-year trial so the student can test it, they can provide us with feedback and we'll see how it goes and maybe we'll do a full switch to NixOS. And then the last item here is just some blue sky insanity for me. Maybe I would be able to somehow share the Nix store between all the machines in a room. I don't know how I could do it, maybe using IPFS, I don't know how that works. So if one student installed a package and another one installed the same package, they could just download it from the other computer instead of downloading it from cache.nixos.org. All right, I guess that's all from me. If there are any questions, I'll be happy to answer them. If you let me just switch to the JITSEE room. They just rebooted. Yep. Live streaming is on. Okay, everyone, sorry about that. We had a little slight technical difficulty with the stream and the JITSEE room being cancelled but we ironed that out. So this now enters into the Q&A portion of this talk and we do have, I think, three questions to which are actually from the speaker who's going to be speaking right up next. So first one is, have you tried to bring NixOS to the routers? Is there a plan to do so? Sorry, to the router, you said? Yeah, to the routers. Okay, so we don't have actually any plans for that. The only plan right now is to use it for the computers used by the students. Okay, gotcha. And what kind of network setup does EP to have? Or runs? Well, what do you mean network? Like, okay, so, well, it's quite complicated. Basically, it's a mess. We have a computer room has a net. Our server room also has a subnet. The book I spoke of at the beginning of my talk actually routes them for us. So I don't exactly what goes on behind the scenes on their infrastructure because I'm not working. And then the only network we have to do is the one in our server room. So that's pretty basic. Even though we have some weird stuff going on for adding between the LAN and all that stuff. I'm not sure that is a question. I have my eye on the, oh yeah, okay. I have my next con QA channel if you want to. Okay. So, yes, there is isolation. Probably, but again, that's not what we do. That's the job of the book out to do that. Right. And someone just said you should not tell that to strangers. Okay. The next question we have is what and if there is CI, what is it? Is it being used? So we're using GitLab's CI because that's what was used up until then. We're not really planning to use Hercules or Hydra because, well, that just doesn't fit in our flow currently. I'm not quite sure there's a way for them to automatically upload the outputs of builds. Like not the path in the next store, but actually it's in it. So I don't want to, you know, when you next build you have a results sim link. I don't want to upload what's behind that result in link, but I want to upload only results slash, let's say, nixOS-test.nitrg and nixOS-test.squashfs and nixOS-test.colonel. I want to upload these three files fully to three with the same name. So I actually change my image. So we're not currently planning on any nix-focused CI system. Right. I've used Cache 6 for some things like that. I'm not sure if it's entirely possible to do what you're doing, though. It might be worth looking into maybe momentarily. Okay. So I don't think I see any more questions in the channel. Right. Yeah. I think that concludes the Q&A session. Thank you so much. Thanks to you.
will start with a current state of the learning infrastructure running on Arch Linux and the signs of migrating to next to us and what has already been done. So I believe the school that believe is pronounced EPI TA, a French school of engineers in computer science. And it's bringing presentation sort of showing the challenges of maintaining infrastructure of more than 800 machines used by students. So they started using