Nixery - A Nix-backed container registry
Nixery builds container images on-demand via Nix and serves them via the standard Docker registry protocol. In this talk we look at how it works, which implementation challenges came up and how it is useful in the so-called "real world".
 I actually didn't count. Hi. Right. I'm going to be doing a little bit of dual microphoneing. I've noticed kind of throughout the course of the conference that very few people are doing demos. So I'm a little bit afraid that the demo gods might actually be striking down all of their vengeance upon me today because I have four. So we'll see how that goes. Right. Hello, everyone. My name is Vincent. I think I know probably half the people in the room already. Test one, two, three. Okay. It's a good start. Right. Let's do it this way. My name is Vincent. I think I know about half of the people in this room. Test gen is my username. That's probably the name by which people know me better than my real name. So yeah. It's the same username on GitHub, Twitter, IOC, whatever. So that's how you can reach out to me. Cool. So I'm here to talk to you about Nixery, my fresh and healthy way of building container images. And to kind of kick that off, I want to give you a little bit of an overview over what the container ecosystem looks like at the moment on a very high level. So quick show of hands. Who has never used Docker or anything like it? Okay. Very few people. Cool. I'm not going to go into what containers are and so on, but I'll say a little bit about how the tooling works because that's kind of all the background that you're going to need. So Docker has a concept of images that are basically like root file systems of a distribution that are made up of a bunch of different tar balls called layers that are distributed over a registry protocol. The details of how this works under the hood don't really matter. But the important thing is that there are different layers that get merged together. So if you have a sequential list of tar balls that contain non-overlapping paths, then they will be squashed into one file system. If you have overlapping paths, then the ones from the latter ones in the order of the image manifest appearance are the ones that take precedence. So this syntax over here is what's called a Docker file. This is the default method of building Docker images. You specify the base image that you have at the top, which in this case is Ubuntu, and then you run individual commands. And there's a bunch of additional instructions such as N which sets environment variables and so on. What happens when you execute this is that Docker goes and it takes every single instruction in this Docker file and it executes it and it takes the resulting diff of file systems and turns it into a layer. What this means is that this is inherently extremely stateful because there is no care taken to make sure that things like timestamps on files are in any way normalized. So things that we deal with in Nix on a very fundamental level just kind of don't exist conceptually in this world. So I actually took this particular Docker file and I ran it twice earlier today. And here are the hashes resulting from the two image builds. As you can see they have nothing to do with each other at all. This was kind of my reaction. And a while back I already started thinking about whether or not there might be a better way to do this. Those of you who talked to me at the Nixcon in 2018 probably heard my ideas about the Kubernetes controller for Nix. I will get to that in a second. But also there have been other people working on very similar things for a while. There was an initial attempt, actually not an attempt at work to build a Docker image via Nix called just build image. That's in docker tools dot build image. And what that does, the way I understand it is it spins up a QMU, installs the things that you want in it and then takes a snapshot of that file system plus some extra information. That was the first idea for building Docker images in Nix and actually worked reasonably well. And then Graham C. Is he here? Somewhere he is. Maybe not in this room right now. Anyways, he wrote a thing called build layered image that I will be referencing a couple of times throughout the talk, which takes a more pure Nix approach to the construction of the layers. So instead of spinning up virtual machines and snapshotting their disk, it actually uses Nix to construct the tower walls themselves. Right. So about a year and a half ago I had this idea for what would the world look like if we didn't actually have to build these container images explicitly. And we just have some sort of content specification that tells us what we want in an image and the image is built based on that. So here's a mockup of something that I kind of prototyped in the beginning where we added an extra resource to the Kubernetes API. That's something that is supported in Kubernetes. You can see that up here. And then we had this simple YAML specification where each of the keys referred to a top level entry in Nix packages. And you kind of submitted this into your cluster and then a demon kind of ran in the back of the cluster, built an image out of this, pushed it into your registry and did this whole normal stateful kind of dance, but abstracted away for the user by just creating an image with this name. After building this, an interesting thing happened. So I tried to demo this to people. And it turns out that when you're demoing this, there's a lot of moving parts in it. You need to write a YAML file. You need to submit the YAML file to a Kubernetes cluster. Then a thing needs to run in the background and build the image and push it into a registry. And then Kubernetes launches pods that pull the image from there. So you've got six, seven things that can go wrong and they all take a little bit of time. So it's the kind of demo where you try to show it to someone. You make a few typos and they're kind of half falling asleep. And like, trust me, this is a valuable concept that I'm trying to get across. So I actually invented on the fly at some point this idea of a demo mode for the controller. So the idea here was that instead of having a specification pre-hoc of what goes into the image, we just use the components of the image to describe what should go into it. So there are kind of two main parts here. Just the host name, which is just the address at which Docker should look up the image that you're downloading. Then we've got a section that describes the packages. And the first ones in here are so-called meta packages, which is kind of a shorthand for sets of packages. So Shell, for example, gives you a bash interactive, IANA, Etsy, and like all these kinds of things that you probably want if you want a shell environment. And then you can specify paths separated other packages that should go into this image. So I built this demo mode, and then I realized that there's actually a much better idea than the first one I had. And then I kind of scrapped the actual controller, and I was just like, this is the way forward. Right. Before we talk about this slide, I'm going to start doing my first little demo just to show you what this thing actually looks like in practice. Okay. Microphone number two. So here's the public instance of Nixery. It's available at nixery.dev. And this is kind of like a public service that you can go and play with. So just to remember the... I can, I can. You don't actually need to read this. I'm just trying to show the logo as often as I can because I spent countless hours designing this, as you can see. So let's do a little demo of this. Here's the standard Docker CLI. What this command says. Is that readable enough? No. No? Okay. Excellent. What this says is we're going to run a container. We're going to attach a TTY interactively. We're going to remove the container after we're done. And then we will pull the image from nixery.dev and we want an image containing shell. So as per the previous slide, this is going to give us an image that just contains a basic bash environment in a second. So I need to actually tell it what to run. So here we go. Here's the bash. There are four utils and all sorts of things installed, but no additional programs. If I tried to run Htop, which I didn't specify in the image, it's not going to be included. So what I can do is I can exit out of this and I can edit it to include additional programs. Again these are just referencing keys in nix packages. And then an image is built on the fly and kind of sent to me. And now I have a shell in which I can actually run Htop. So that's the basic mode of operation. In this public instance that's running at nixery.dev, I track whichever is the latest nix-res channel and I do updates for this roughly once or twice a week. But it turns out that what's actually a much more interesting use case than just having a thing that can give you these arbitrary images at Hoc is the ability to add your own private services into it and see how this can be useful for CI and organizations. So there's something I have to show. I personally am a monorep of personal and here is the default of nix of my personal infrastructure repository. It contains stuff like my blog and so on. And this is mostly standard nix stuff. So there's nothing like super interesting to show here. I import nix packages from a pin commit and then I have an overlay that overlays my standard packages on top of it. If you want to know how this works in detail and have discussions about pinning channels and so on, we can do that after the talk because it's a slightly different thing. Anyways, in here I have a thing that I built which is the nix-con demo using nmateers. Where is he? Also somewhere. Excellent Nash which builds Rust based on lock file specifications. Huge fan of that. And I wrote a tiny, that's the wrong one, I wrote a tiny demo which just looks like this. This can either run as a web server showing me this nice message. I hope for the Czech people here that I spelled that correctly, trusting the internet or run with the CLI argument. It doesn't really matter what this program does. I just want to show you that there's an actual thing going on here. So I'm importing that into my nix package set and then I have it available in my overlay of packages. So down here I have a shell in which DRNV has just, thanks to Zimba, has just loaded some environment variables for me. And one of them is the one called nix-ery packages path. So I support multiple different sources for importing package sets into nix-ery. And one of them is using git repositories and one of them is using local file paths. In this case I'm just using the file path of the checkout of my repositories. I'm telling nix-ery that the package set from which I want you to build images is the one I have at this place on my disk. So what I can do now is I can go and spin up a nix-ery in here, which looks like that. And then over here on the other side, I can do the same thing I was doing before with my shell. But instead of pulling from nix-ery.dev, I'm pulling from localhost. So this is the local instance I have running. So this one wasn't particularly interesting because it was cached already, I think. But yeah, I just want to show that there is nothing in here right now. So there's no program called nix-condemo. Now I'm going to remove that image just to prove to you that I'm not going to try and hit any caches here. I don't actually, ah, what help if I could spell? I don't have it even. So then we can go and run this command again and add this one to the image. Here we go. Could not find nix package. Thanks. Here we go. Okay. So what was supposed to happen is that I pull an image down now and it contains the binary that I've added here. Quick show of hands, how much time should I spend working on getting that demo up and running versus not? Okay, I'll give it 20 seconds. So that looks correct to me. And then here at the top somewhere. Yep. Ah, okay. It's just called nix-con. Here we go. Right. And now I have my nix-con demo thing in here built from the package set, overlaid into my local configuration. Cool. So that works. It actually turns into kind of interesting way of deploying services because in the configuration mode where I let you specify git repository instead of a file path, I can translate tags into git commits, which we'll see in a bit. And then you kind of start getting the ability to build CI pipelines out of this. Before I show that, I want to talk about something that I spent way too much time on. As a sort of premature optimization, but it was an interesting thing to go through. And this is something that Graham kind of kicked off when you started building build layered image. For historical reasons, Docker has a maximum number of layers that you can have in an image. It's around 125 for some reason. And it turns out that even using up all of those is not a very good idea because you share this layer restriction with your user. So if you have users that want to pull an image and extend it by adding additional files, you count if there's already 125 layers in the image. So Graham had this initial idea of creating a graph of all the dependencies that a given image needs and then counting the popularity of individual things inside of that graph. So if you have more than one derivation that references something like G-Lip C, then G-Lip C becomes more popular and eventually you sort by this popularity and kind of pick the most important ones from the top and hope that those are going to match. But in this case, you have a maximum of one derivation or one closure per image layer. But there are many situations in which some kind of thing depends on, let's say we have a program called foo and it depends on foo data. There's never really a good reason to split foo and foo data apart from each other. We want to actually put them in the same layer and make sure that we get kind of optimal caching here during the builds. So I'm going to walk you quickly through how that works and there's a more detailed write up for those of you that like graphs. Right. I suspect that some of this is not going to be readable all the way from the back, but I'm going to very quickly talk through this. So I had to find something that doesn't have a whole lot of dependencies to actually demo this. My initial attempt was to use git, but git depends on literally the entire universe and every single Perl package that exists. So that's a lot of packages. What we have here is you see a blue node at the top that's called the image root. And then there's a few arrows that go from that to the top level of this graph, which are images, which are packages that the user requested. Specifically here, there's htop and there's nano and there's a shell somewhere and that's kind of it for this image. Actually nano is part of the shell alias. And then if we look at this, there's a few interesting things in here. So for example, Perl is only referenced once by the more utils from up there, but Perl itself actually turns out to be a very popular package. There's a lot of things that depend on Perl. So we would like to use this information and kind of make sure that Perl ends up not bundled together with more utils. And similarly we can see that glip see is extremely popular. It's got a lot of arrows going into it. So what I started doing at this point is together with a few people in IOC and also at work, we devised an algorithm for counting the popularity of all packages and NICs packages by just checking how many runtime references there are to any given derivation and then creating a kind of popularity metric by multiplying that with the size of the closure so that we can get kind of the best hash hit performance in terms of data transferred and whatnot. So the way this works is that the red ones here have been identified by NICS-R-E as popular packages and we go and we draw extra edges from the root to those popular packages which in the case of Perl, for example, now means that there are two paths going into Perl. That's the red one up here. And then we create what's called a dominator tree of this, which is a relay outing of a graph into a tree in which every node is preceded by the ones that all paths to it must pass through. If that was confusing, the Wikipedia article will explain it to you. That ends up looking like this and suddenly we have a very ordered graph of how these things look. So we have the image root and then we have a top layer which actually matches very well with the images, the image layer so we would like to stick into the same table. And then stuff like more utils which depends on some of these not very popular Perl libraries and relatively speaking to the rest of the ecosystem anyways. And we can bundle these together into the same layout. So once we get to this point, we can calculate the ideal layer layout which looks like this and then based on the budget we have available for layers, we can start merging the least popular things together. And then we actually get pretty good build caching out of it. Cool. Does this make sense so far? Does anybody would like to inject a question at this point? Right, let's move on. So what about image tags? One component of image names I didn't mention before is this yellow one here at the end. It's the image tag. For Docker this is commonly versions that people add to things. So they will have something like Ubuntu, colon and then an Ubuntu release version or whatever. But I realized that we can actually use this information and map it onto something else, specifically Git commits. So if you're pointing nixery at the public nix packages wrapper, you could use this to get arbitrary references like specific channels or specific commits. If you're pointing it at a private repository, you can start having your CI substitute these parameters and then you can use this as your deployment strategy. So to quickly show that, I will show you a demo involving Kubernetes and this is going to be the one that will definitely fail. Wish me luck. All right. What I basically want to show first of all is the deployment manifest for my blog. So my blog, which most of you haven't read and I don't really post to it, so whatever. But it's an interesting experimental playground for new technology. I have this Kubernetes manifest that deploys the blog. For those of you not familiar with Kubernetes, most of the fields in here don't really matter. It's just a kind of verbose thing. The primary line in here is this one here. It specifies the image to run. I'm going to zoom in a little bit just to make sure that people can see this. Where what's basically going on is that I'm pulling an image from nixery.local, which is a private DNS name pointing to a nixery instance running inside of my cluster. How to do this is something I'm currently writing documentation for. So people will be able to start doing that on their own very soon. And then I want the attribute task gen.blog, which is the derivation for my blog. And I specify the version as githead. So this syntax here is from a templating tool that I use called contemplate, which basically just inserts the current git commit of the repository that you're in. Right. So what I can show you at this point is I can describe the deployment for my blog in Kubernetes. I'm live speaking to the Kubernetes cluster in which I deploy my personal things. And we can see that the image is currently some commit in this git repository. I can take this commit and look at what it is exactly. And I know that this is the one on which this particular image was built. So to do something a little interactive now, I have spun up the nixery inside of the cluster. Here's the live.log stream from nixery. And I'm going to, first of all, run a container inside of Kubernetes and attach to it interactively. And then we will do something more interesting. So the way this works here is syntactically very similar to Docker. So if you saw the previous command and understood it, it's pretty much the same thing. We're running a container. We're attaching a TTY. We're doing this interactively. We're giving it some randomly generated name because they don't want to conflict. We're not restarting it. And we're pulling the image from nixery.local slash shell. So it's just going to give me the standard bash environment. If I run this down here, I will see that a demo container is now creating. And after a couple of seconds, I get the shell out of it. Cool. So there's nothing in here as before. There's no extra commands. I'm not going to demo Htop again. Instead, I'm just going to let you know that this nixery instance is pointing at the repository I was showing you earlier. So I can actually go and pull the nixcon demo that I was showing up for a second into this instance and deploy a container with it. I'm just going to go ahead and do that for talking too much about it. And now we should start seeing messages from nixery here. So let me just make sure that this stream is not dead. Yeah, here we go. So there's now a build happening. And if domain is doing its job well and cache access up and running, then this should hit the binary cache. If it doesn't, it's going to be interesting because this is a Rust thing and the Rust compiler is not the fastest one. And then I'm probably going to end up in a situation like yesterday with the Haskell demo. So let's give it a moment and see what happens. Actually on the topic of caches in this case, what I found to be a very effective way of working with this is having a CI set up on your system that populates the binary cache, something like cache x on every push so that all the things are already pre-built when nixery tries to go and fetch them from a binary cache. Nixery can in theory go and start building everything from scratch, but you probably don't want it to be spending a couple of hours building your Haskell services. So it's nice to have that done already when you get to the point of deployment. What we're interested in here is not necessarily having everything pre-cached on an image level because we want to be able to dynamically modify what goes into an image, for example, by adding debugging tools, but it's still nice to actually have offloaded the caching of stuff like your actual service builds to nix. Okay, here we go. The nix build is done. A bunch of stuff happened under the hood, and now we have a command prompt here that should actually give me the nixcon demo. And now if I forward... Let's see if we can get this to work. This is the one. If we forward something like port 42, 42 into this container, then I should be able to... Right, that demo worked. I'm surprised. The trivial first one. That's how it always goes. Cool. So Kubernetes is pretty much one of the ideal ways of using this in production, and it's kind of based of the origin of the idea where this whole thing came from. And the deployment of Nixery into a cluster is relatively simple. Like I was saying, this is going to be documented on my page. You basically just run a nixery instance and add some environment variables to configure it. There's nothing really special about it. The one restriction that is in place at the moment, simply because I don't have time to implement support for everything, is that this only runs on Google Cloud Platform right now. There is a mode coming up for just being able to use a local file system instead of serving image layers from our storage system. And once that happens, you can actually run this anywhere. Also contributions welcome if somebody wants to implement that. The issue tracker describes how it should work. And once you have that up and running, you can create these kinds of things to solidify the deployment infrastructure using this service. Right, that's kind of the primary thing I wanted to say. Here are my contact details and where to find source code and all that kind of thing. Yes. And I want to say thanks to Sarah and Ida for coming up with the broccoli alias, which we kind of picked up late yesterday evening. Right, questions. So, two questions. First, in the URL, you specify the path and you specify packages or the versions using slash. Yep. Did you try to, or did you consider using NIC's expressions there? Yes. The primary issue is that there are restrictions in the Docker registry protocol in which characters are allowed and it's quite restrictive. So for example, an interesting fact, there's an issue on the issue tracker about this. You cannot have uppercase characters in there. And NIC's packages has things like Haskell packages, which is an uppercase thing, but you might want to refer to packages inside of that that are actually binary. So what I did is I tried to do a first lookup with the exact name that the user specified. And if that doesn't work, I normalize the casing of all attributes and then do a lookup with that. And some of you are probably guessing this right now. Yes, there are name clashes based on casing inside of NIC's packages. Fortunately, at the top level, this only affects things that are actually aliases for each other, but inside of Haskell packages, there are actually different pieces of software with names that are only different in the casing. So for those, I just don't have a solution at the moment. If you need one of those, I don't know, make a private wrapper on aliases, it's something sane. With the original Kubernetes controller idea, I also considered adding a field just to straight add a NICs expression in there, but since that idea has kind of been dropped, I thought that it's probably more reasonable to actually keep those in the NICs repository rather than inside of the deployment manifest. Okay, thank you. And the second question will be, how do you set up private Git repository with it? Lastly, I'm interested with the keys. So I'm using the built-in fetch Git functionality to fetch the repo, and built-in fetch Git uses the SSH configuration of your environment. So it's impure, which means that as long as the environment in which Nixery is running is configured with appropriate SSH credentials, then everything will just kind of work. Thank you. Cheers. All right, anything else? Yes, I've missed that bit, but I wonder, Dockerfile has a lot more to express like mount points, X-X, whatever. Is that handled with Nixery, or do you have to build some kind of command line on that? So at the moment, most of those things, because they're kind of only metadata outsourced to Kubernetes, so you specify stuff like the entry point inside of the deployment manifest. There is an effort in progress in Nix packages right now to add metadata to every derivation that contains binaries that tells us what is the primary binary in this package. And once that is in place, we can actually automatically generate entry points. But then there's also a discussion of if somebody writes shell slash engine X slash n grog, which one do they actually want to run? So at some point, you still have to defer to the user. As for environment variables and that kind of thing, in theory, the Nix builder that we have to do this kind of stuff supports them, but there is currently no surface in the API where you can actually attach that. We might come up with something later. There's a few issues about related things on the tracker. Feel free to give some input. Other examples of other meta packages besides shell? And what can you do with those meta packages? So shell is the only one that the public instance supports at the moment. The next one that I'm currently working on is RM64 to serve your RM64 binaries. The thing with that is that the Docker registry protocol has sort of support for switching architectures, but it does it in the form of serving you one manifest that contains hashes for all architectures. So if you were to use that functionality, you would have to build all things for all architectures on every request, which is probably not what the user wants. So I'm thinking that meta packages as the first element are kind of a more sane way of toggling that behavior. All right. Thank you very much. Thank you.
 I actually didn't count.  Hi.  Right.  I'm going to be doing a little bit of dual microphoneing.  I've noticed kind of throughout the course of the conference that very few people are  doing demos.  So I'm a little bit afraid that the demo gods might actually be striking down all of their  vengeance upon me today because I have four.  So we'll see how that goes.  Right.  Hello, everyone.  My name is Vincent.  I think I know probably half the people in the room already.  Test one, two, three.  Okay.  It's a good start.  Right.  Let's do it this way.  My name is Vincent.  I think I know about half of the people in this room.  Test gen is my username.  That's probably the name by which people know me better than my real name.  So yeah.  It's the same username on GitHub, Twitter, IOC, whatever.  So that's how you can reach out to me.  Cool.  So I'm here to talk to you about Nixery, my fresh and healthy way of building container  images.  And to kind of kick that off, I want to give you a little bit of an overview over what  the container ecosystem looks like at the moment on a very high level.  So quick show of hands.  Who has never used Docker or anything like it?  Okay.  Very few people.  Cool.  I'm not going to go into what containers are and so on, but I'll say a little bit about  how the tooling works because that's kind of all the background that you're going to  need.  So Docker has a concept of images that are basically like root file systems of a distribution  that are made up of a bunch of different tar balls called layers that are distributed  over a registry protocol.  The details of how this works under the hood don't really matter.  But the important thing is that there are different layers that get merged together.  So if you have a sequential list of tar balls that contain non-overlapping paths, then they  will be squashed into one file system.  If you have overlapping paths, then the ones from the latter ones in the order of the image  manifest appearance are the ones that take precedence.  So this syntax over here is what's called a Docker file.  This is the default method of building Docker images.  You specify the base image that you have at the top, which in this case is Ubuntu, and  then you run individual commands.  And there's a bunch of additional instructions such as N which sets environment variables  and so on.  What happens when you execute this is that Docker goes and it takes every single instruction  in this Docker file and it executes it and it takes the resulting diff of file systems  and turns it into a layer.  What this means is that this is inherently extremely stateful because there is no care  taken to make sure that things like timestamps on files are in any way normalized.  So things that we deal with in Nix on a very fundamental level just kind of don't exist  conceptually in this world.  So I actually took this particular Docker file and I ran it twice earlier today.  And here are the hashes resulting from the two image builds.  As you can see they have nothing to do with each other at all.  This was kind of my reaction.  And a while back I already started thinking about whether or not there might be a better  way to do this.  Those of you who talked to me at the Nixcon in 2018 probably heard my ideas about the  Kubernetes controller for Nix.  I will get to that in a second.  But also there have been other people working on very similar things for a while.  There was an initial attempt, actually not an attempt at work to build a Docker image  via Nix called just build image.  That's in docker tools dot build image.  And what that does, the way I understand it is it spins up a QMU, installs the things  that you want in it and then takes a snapshot of that file system plus some extra information.  That was the first idea for building Docker images in Nix and actually worked reasonably  well.  And then Graham C. Is he here?  Somewhere he is.  Maybe not in this room right now.  Anyways, he wrote a thing called build layered image that I will be referencing a couple  of times throughout the talk, which takes a more pure Nix approach to the construction  of the layers.  So instead of spinning up virtual machines and snapshotting their disk, it actually uses  Nix to construct the tower walls themselves.  Right.  So about a year and a half ago I had this idea for what would the world look like if  we didn't actually have to build these container images explicitly.  And we just have some sort of content specification that tells us what we want in an image and  the image is built based on that.  So here's a mockup of something that I kind of prototyped in the beginning where we added  an extra resource to the Kubernetes API.  That's something that is supported in Kubernetes.  You can see that up here.  And then we had this simple YAML specification where each of the keys referred to a top level  entry in Nix packages.  And you kind of submitted this into your cluster and then a demon kind of ran in the back of  the cluster, built an image out of this, pushed it into your registry and did this whole normal  stateful kind of dance, but abstracted away for the user by just creating an image with  this name.  After building this, an interesting thing happened.  So I tried to demo this to people.  And it turns out that when you're demoing this, there's a lot of moving parts in it.  You need to write a YAML file.  You need to submit the YAML file to a Kubernetes cluster.  Then a thing needs to run in the background and build the image and push it into a registry.  And then Kubernetes launches pods that pull the image from there.  So you've got six, seven things that can go wrong and they all take a little bit of time.  So it's the kind of demo where you try to show it to someone.  You make a few typos and they're kind of half falling asleep.  And like, trust me, this is a valuable concept that I'm trying to get across.  So I actually invented on the fly at some point this idea of a demo mode for the controller.  So the idea here was that instead of having a specification pre-hoc of what goes into  the image, we just use the components of the image to describe what should go into it.  So there are kind of two main parts here.  Just the host name, which is just the address at which Docker should look up the image that  you're downloading.  Then we've got a section that describes the packages.  And the first ones in here are so-called meta packages, which is kind of a shorthand for  sets of packages.  So Shell, for example, gives you a bash interactive, IANA, Etsy, and like all these kinds of things  that you probably want if you want a shell environment.  And then you can specify paths separated other packages that should go into this image.  So I built this demo mode, and then I realized that there's actually a much better idea  than the first one I had.  And then I kind of scrapped the actual controller, and I was just like, this is the way forward.  Right.  Before we talk about this slide, I'm going to start doing my first little demo just to  show you what this thing actually looks like in practice.  Okay.  Microphone number two.  So here's the public instance of Nixery.  It's available at nixery.dev.  And this is kind of like a public service that you can go and play with.  So just to remember the...  I can, I can.  You don't actually need to read this.  I'm just trying to show the logo as often as I can because I spent countless hours designing  this, as you can see.  So let's do a little demo of this.  Here's the standard Docker CLI.  What this command says.  Is that readable enough?  No.  No?  Okay.  Excellent.  What this says is we're going to run a container.  We're going to attach a TTY interactively.  We're going to remove the container after we're done.  And then we will pull the image from nixery.dev and we want an image containing shell.  So as per the previous slide, this is going to give us an image that just contains a basic  bash environment in a second.  So I need to actually tell it what to run.  So here we go.  Here's the bash.  There are four utils and all sorts of things installed, but no additional programs.  If I tried to run Htop, which I didn't specify in the image, it's not going to be included.  So what I can do is I can exit out of this and I can edit it to include additional programs.  Again these are just referencing keys in nix packages.  And then an image is built on the fly and kind of sent to me.  And now I have a shell in which I can actually run Htop.  So that's the basic mode of operation.  In this public instance that's running at nixery.dev, I track whichever is the latest  nix-res channel and I do updates for this roughly once or twice a week.  But it turns out that what's actually a much more interesting use case than just having  a thing that can give you these arbitrary images at Hoc is the ability to add your own  private services into it and see how this can be useful for CI and organizations.  So there's something I have to show.  I personally am a monorep of personal and here is the default of nix of my personal infrastructure  repository.  It contains stuff like my blog and so on.  And this is mostly standard nix stuff.  So there's nothing like super interesting to show here.  I import nix packages from a pin commit and then I have an overlay that overlays my standard  packages on top of it.  If you want to know how this works in detail and have discussions about pinning channels  and so on, we can do that after the talk because it's a slightly different thing.  Anyways, in here I have a thing that I built which is the nix-con demo using nmateers.  Where is he?  Also somewhere.  Excellent Nash which builds Rust based on lock file specifications.  Huge fan of that.  And I wrote a tiny, that's the wrong one, I wrote a tiny demo which just looks like this.  This can either run as a web server showing me this nice message.  I hope for the Czech people here that I spelled that correctly, trusting the internet or run  with the CLI argument.  It doesn't really matter what this program does.  I just want to show you that there's an actual thing going on here.  So I'm importing that into my nix package set and then I have it available in my overlay  of packages.  So down here I have a shell in which DRNV has just, thanks to Zimba, has just loaded  some environment variables for me.  And one of them is the one called nix-ery packages path.  So I support multiple different sources for importing package sets into nix-ery.  And one of them is using git repositories and one of them is using local file paths.  In this case I'm just using the file path of the checkout of my repositories.  I'm telling nix-ery that the package set from which I want you to build images is the one  I have at this place on my disk.  So what I can do now is I can go and spin up a nix-ery in here, which looks like that.  And then over here on the other side, I can do the same thing I was doing before with  my shell.  But instead of pulling from nix-ery.dev, I'm pulling from localhost.  So this is the local instance I have running.  So this one wasn't particularly interesting because it was cached already, I think.  But yeah, I just want to show that there is nothing in here right now.  So there's no program called nix-condemo.  Now I'm going to remove that image just to prove to you that I'm not going to try and  hit any caches here.  I don't actually, ah, what help if I could spell?  I don't have it even.  So then we can go and run this command again and add this one to the image.  Here we go.  Could not find nix package.  Thanks.  Here we go.  Okay.  So what was supposed to happen is that I pull an image down now and it contains the binary  that I've added here.  Quick show of hands, how much time should I spend working on getting that demo up and  running versus not?  Okay, I'll give it 20 seconds.  So that looks correct to me.  And then here at the top somewhere.  Yep.  Ah, okay.  It's just called nix-con.  Here we go.  Right.  And now I have my nix-con demo thing in here built from the package set, overlaid into  my local configuration.  Cool.  So that works.  It actually turns into kind of interesting way of deploying services because in the configuration  mode where I let you specify git repository instead of a file path, I can translate tags  into git commits, which we'll see in a bit.  And then you kind of start getting the ability to build CI pipelines out of this.  Before I show that, I want to talk about something that I spent way too much time on.  As a sort of premature optimization, but it was an interesting thing to go through.  And this is something that Graham kind of kicked off when you started building build  layered image.  For historical reasons, Docker has a maximum number of layers that you can have in an image.  It's around 125 for some reason.  And it turns out that even using up all of those is not a very good idea because you  share this layer restriction with your user.  So if you have users that want to pull an image and extend it by adding additional files,  you count if there's already 125 layers in the image.  So Graham had this initial idea of creating a graph of all the dependencies that a given  image needs and then counting the popularity of individual things inside of that graph.  So if you have more than one derivation that references something like G-Lip C, then G-Lip  C becomes more popular and eventually you sort by this popularity and kind of pick the  most important ones from the top and hope that those are going to match.  But in this case, you have a maximum of one derivation or one closure per image layer.  But there are many situations in which some kind of thing depends on, let's say we have  a program called foo and it depends on foo data.  There's never really a good reason to split foo and foo data apart from each other.  We want to actually put them in the same layer and make sure that we get kind of optimal  caching here during the builds.  So I'm going to walk you quickly through how that works and there's a more detailed  write up for those of you that like graphs.  Right.  I suspect that some of this is not going to be readable all the way from the back, but  I'm going to very quickly talk through this.  So I had to find something that doesn't have a whole lot of dependencies to actually demo  this.  My initial attempt was to use git, but git depends on literally the entire universe and  every single Perl package that exists.  So that's a lot of packages.  What we have here is you see a blue node at the top that's called the image root.  And then there's a few arrows that go from that to the top level of this graph, which  are images, which are packages that the user requested.  Specifically here, there's htop and there's nano and there's a shell somewhere and that's  kind of it for this image.  Actually nano is part of the shell alias.  And then if we look at this, there's a few interesting things in here.  So for example, Perl is only referenced once by the more utils from up there, but Perl  itself actually turns out to be a very popular package.  There's a lot of things that depend on Perl.  So we would like to use this information and kind of make sure that Perl ends up not bundled  together with more utils.  And similarly we can see that glip see is extremely popular.  It's got a lot of arrows going into it.  So what I started doing at this point is together with a few people in IOC and also at work,  we devised an algorithm for counting the popularity of all packages and NICs packages by just  checking how many runtime references there are to any given derivation and then creating  a kind of popularity metric by multiplying that with the size of the closure so that  we can get kind of the best hash hit performance in terms of data transferred and whatnot.  So the way this works is that the red ones here have been identified by NICS-R-E as popular  packages and we go and we draw extra edges from the root to those popular packages which  in the case of Perl, for example, now means that there are two paths going into Perl.  That's the red one up here.  And then we create what's called a dominator tree of this, which is a relay outing of a  graph into a tree in which every node is preceded by the ones that all paths to it must pass  through.  If that was confusing, the Wikipedia article will explain it to you.  That ends up looking like this and suddenly we have a very ordered graph of how these  things look.  So we have the image root and then we have a top layer which actually matches very well  with the images, the image layer so we would like to stick into the same table.  And then stuff like more utils which depends on some of these not very popular Perl libraries  and relatively speaking to the rest of the ecosystem anyways.  And we can bundle these together into the same layout.  So once we get to this point, we can calculate the ideal layer layout which looks like this  and then based on the budget we have available for layers, we can start merging the least  popular things together.  And then we actually get pretty good build caching out of it.  Cool.  Does this make sense so far?  Does anybody would like to inject a question at this point?  Right, let's move on.  So what about image tags?  One component of image names I didn't mention before is this yellow one here at the end.  It's the image tag.  For Docker this is commonly versions that people add to things.  So they will have something like Ubuntu, colon and then an Ubuntu release version or whatever.  But I realized that we can actually use this information and map it onto something else,  specifically Git commits.  So if you're pointing nixery at the public nix packages wrapper, you could use this to  get arbitrary references like specific channels or specific commits.  If you're pointing it at a private repository, you can start having your CI substitute these  parameters and then you can use this as your deployment strategy.  So to quickly show that, I will show you a demo involving Kubernetes and this is going  to be the one that will definitely fail.  Wish me luck.  All right.  What I basically want to show first of all is the deployment manifest for my blog.  So my blog, which most of you haven't read and I don't really post to it, so whatever.  But it's an interesting experimental playground for new technology.  I have this Kubernetes manifest that deploys the blog.  For those of you not familiar with Kubernetes, most of the fields in here don't really matter.  It's just a kind of verbose thing.  The primary line in here is this one here.  It specifies the image to run.  I'm going to zoom in a little bit just to make sure that people can see this.  Where what's basically going on is that I'm pulling an image from nixery.local, which is  a private DNS name pointing to a nixery instance running inside of my cluster.  How to do this is something I'm currently writing documentation for.  So people will be able to start doing that on their own very soon.  And then I want the attribute task gen.blog, which is the derivation for my blog.  And I specify the version as githead.  So this syntax here is from a templating tool that I use called contemplate, which basically  just inserts the current git commit of the repository that you're in.  Right.  So what I can show you at this point is I can describe the deployment for my blog in  Kubernetes.  I'm live speaking to the Kubernetes cluster in which I deploy my personal things.  And we can see that the image is currently some commit in this git repository.  I can take this commit and look at what it is exactly.  And I know that this is the one on which this particular image was built.  So to do something a little interactive now, I have spun up the nixery inside of the cluster.  Here's the live.log stream from nixery.  And I'm going to, first of all, run a container inside of Kubernetes and attach to it interactively.  And then we will do something more interesting.  So the way this works here is syntactically very similar to Docker.  So if you saw the previous command and understood it, it's pretty much the same thing.  We're running a container.  We're attaching a TTY.  We're doing this interactively.  We're giving it some randomly generated name because they don't want to conflict.  We're not restarting it.  And we're pulling the image from nixery.local slash shell.  So it's just going to give me the standard bash environment.  If I run this down here, I will see that a demo container is now creating.  And after a couple of seconds, I get the shell out of it.  Cool.  So there's nothing in here as before.  There's no extra commands.  I'm not going to demo Htop again.  Instead, I'm just going to let you know that this nixery instance is pointing at the repository  I was showing you earlier.  So I can actually go and pull the nixcon demo that I was showing up for a second into this  instance and deploy a container with it.  I'm just going to go ahead and do that for talking too much about it.  And now we should start seeing messages from nixery here.  So let me just make sure that this stream is not dead.  Yeah, here we go.  So there's now a build happening.  And if domain is doing its job well and cache access up and running, then this should hit  the binary cache.  If it doesn't, it's going to be interesting because this is a Rust thing and the Rust  compiler is not the fastest one.  And then I'm probably going to end up in a situation like yesterday with the Haskell  demo.  So let's give it a moment and see what happens.  Actually on the topic of caches in this case, what I found to be a very effective way of  working with this is having a CI set up on your system that populates the binary cache,  something like cache x on every push so that all the things are already pre-built when  nixery tries to go and fetch them from a binary cache.  Nixery can in theory go and start building everything from scratch, but you probably  don't want it to be spending a couple of hours building your Haskell services.  So it's nice to have that done already when you get to the point of deployment.  What we're interested in here is not necessarily having everything pre-cached on an image level  because we want to be able to dynamically modify what goes into an image, for example,  by adding debugging tools, but it's still nice to actually have offloaded the caching  of stuff like your actual service builds to nix.  Okay, here we go.  The nix build is done.  A bunch of stuff happened under the hood, and now we have a command prompt here that  should actually give me the nixcon demo.  And now if I forward...  Let's see if we can get this to work.  This is the one.  If we forward something like port 42, 42 into this container, then I should be able to...  Right, that demo worked.  I'm surprised.  The trivial first one.  That's how it always goes.  Cool.  So Kubernetes is pretty much one of the ideal ways of using this in production, and it's  kind of based of the origin of the idea where this whole thing came from.  And the deployment of Nixery into a cluster is relatively simple.  Like I was saying, this is going to be documented on my page.  You basically just run a nixery instance and add some environment variables to configure  it.  There's nothing really special about it.  The one restriction that is in place at the moment, simply because I don't have time to  implement support for everything, is that this only runs on Google Cloud Platform right  now.  There is a mode coming up for just being able to use a local file system instead of serving  image layers from our storage system.  And once that happens, you can actually run this anywhere.  Also contributions welcome if somebody wants to implement that.  The issue tracker describes how it should work.  And once you have that up and running, you can create these kinds of things to solidify  the deployment infrastructure using this service.  Right, that's kind of the primary thing I wanted to say.  Here are my contact details and where to find source code and all that kind of thing.  Yes.  And I want to say thanks to Sarah and Ida for coming up with the broccoli alias, which  we kind of picked up late yesterday evening.  Right, questions.  So, two questions.  First, in the URL, you specify the path and you specify packages or the versions using  slash.  Yep.  Did you try to, or did you consider using NIC's expressions there?  Yes.  The primary issue is that there are restrictions in the Docker registry protocol in which characters  are allowed and it's quite restrictive.  So for example, an interesting fact, there's an issue on the issue tracker about this.  You cannot have uppercase characters in there.  And NIC's packages has things like Haskell packages, which is an uppercase thing, but  you might want to refer to packages inside of that that are actually binary.  So what I did is I tried to do a first lookup with the exact name that the user specified.  And if that doesn't work, I normalize the casing of all attributes and then do a lookup  with that.  And some of you are probably guessing this right now.  Yes, there are name clashes based on casing inside of NIC's packages.  Fortunately, at the top level, this only affects things that are actually aliases for each  other, but inside of Haskell packages, there are actually different pieces of software  with names that are only different in the casing.  So for those, I just don't have a solution at the moment.  If you need one of those, I don't know, make a private wrapper on aliases, it's something  sane.  With the original Kubernetes controller idea, I also considered adding a field just to straight  add a NICs expression in there, but since that idea has kind of been dropped, I thought  that it's probably more reasonable to actually keep those in the NICs repository rather than  inside of the deployment manifest.  Okay, thank you.  And the second question will be, how do you set up private Git repository with it?  Lastly, I'm interested with the keys.  So I'm using the built-in fetch Git functionality to fetch the repo, and built-in fetch Git  uses the SSH configuration of your environment.  So it's impure, which means that as long as the environment in which Nixery is running  is configured with appropriate SSH credentials, then everything will just kind of work.  Thank you.  Cheers.  All right, anything else?  Yes, I've missed that bit, but I wonder, Dockerfile has a lot more to express like mount points,  X-X, whatever.  Is that handled with Nixery, or do you have to build some kind of command line on that?  So at the moment, most of those things, because they're kind of only metadata outsourced to  Kubernetes, so you specify stuff like the entry point inside of the deployment manifest.  There is an effort in progress in Nix packages right now to add metadata to every derivation  that contains binaries that tells us what is the primary binary in this package.  And once that is in place, we can actually automatically generate entry points.  But then there's also a discussion of if somebody writes shell slash engine X slash  n grog, which one do they actually want to run?  So at some point, you still have to defer to the user.  As for environment variables and that kind of thing, in theory, the Nix builder that  we have to do this kind of stuff supports them, but there is currently no surface in  the API where you can actually attach that.  We might come up with something later.  There's a few issues about related things on the tracker.  Feel free to give some input.  Other examples of other meta packages besides shell?  And what can you do with those meta packages?  So shell is the only one that the public instance supports at the moment.  The next one that I'm currently working on is RM64 to serve your RM64 binaries.  The thing with that is that the Docker registry protocol has sort of support for switching  architectures, but it does it in the form of serving you one manifest that contains  hashes for all architectures.  So if you were to use that functionality, you would have to build all things for all  architectures on every request, which is probably not what the user wants.  So I'm thinking that meta packages as the first element are kind of a more sane way  of toggling that behavior.  All right.  Thank you very much.  Thank you.
Nixery_-_A_Nix-backed_container_registry_1_0_1871.png Nixery_-_A_Nix-backed_container_registry_1_1872_11843.png Nixery_-_A_Nix-backed_container_registry_1_11844_11891.png Nixery_-_A_Nix-backed_container_registry_1_11892_11951.png Nixery_-_A_Nix-backed_container_registry_1_11952_12011.png Nixery_-_A_Nix-backed_container_registry_1_12012_12071.png Nixery_-_A_Nix-backed_container_registry_1_12072_12275.png Nixery_-_A_Nix-backed_container_registry_1_12276_12551.png Nixery_-_A_Nix-backed_container_registry_1_12552_12671.png Nixery_-_A_Nix-backed_container_registry_1_12672_13139.png Nixery_-_A_Nix-backed_container_registry_1_13140_14159.png Nixery_-_A_Nix-backed_container_registry_1_14160_15851.png Nixery_-_A_Nix-backed_container_registry_1_15852_17063.png Nixery_-_A_Nix-backed_container_registry_1_17064_17111.png Nixery_-_A_Nix-backed_container_registry_1_17112_17159.png Nixery_-_A_Nix-backed_container_registry_1_17160_17207.png Nixery_-_A_Nix-backed_container_registry_1_17208_17267.png Nixery_-_A_Nix-backed_container_registry_1_17268_17315.png Nixery_-_A_Nix-backed_container_registry_1_17316_17639.png Nixery_-_A_Nix-backed_container_registry_1_17640_19835.png Nixery_-_A_Nix-backed_container_registry_1_19836_20507.png Nixery_-_A_Nix-backed_container_registry_1_20508_29495.png Nixery_-_A_Nix-backed_container_registry_1_29496_29555.png Nixery_-_A_Nix-backed_container_registry_1_29556_30851.png Nixery_-_A_Nix-backed_container_registry_1_30852_31295.png Nixery_-_A_Nix-backed_container_registry_1_31296_36539.png Nixery_-_A_Nix-backed_container_registry_1_36540_37319.png Nixery_-_A_Nix-backed_container_registry_1_37320_39143.png Nixery_-_A_Nix-backed_container_registry_1_39144_41423.png Nixery_-_A_Nix-backed_container_registry_1_41424_42299.png Nixery_-_A_Nix-backed_container_registry_1_42300_45966.png
I actually didn't count. Hi. Right. I'm going to be doing a little bit of dual microphoneing. I've noticed kind of throughout the course of the conference that very few people are doing demos. So I'm a little bit afraid that the demo gods might actually be striking down all of their vengeance upon me today because I have four. So we'll see how that goes. Right. Hello, everyone. My name is Vincent. I think I know probably half the people in the room already. Test one, two, three. Okay. It's a good start. Right. Let's do it this way. My name is Vincent. I think I know about half of the people in this room. Test gen is my username. That's probably the name by which people know me better than my real name. So yeah. It's the same username on GitHub, Twitter, IOC, whatever. So that's how you can reach out to me. Cool. So I'm here to talk to you about Nixery, my fresh and healthy way of building container images. And to kind of kick that off, I want to give you a little bit of an overview over what the container ecosystem looks like at the moment on a very high level. So quick show of hands. Who has never used Docker or anything like it? Okay. Very few people. Cool. I'm not going to go into what containers are and so on, but I'll say a little bit about how the tooling works because that's kind of all the background that you're going to need. So Docker has a concept of images that are basically like root file systems of a distribution that are made up of a bunch of different tar balls called layers that are distributed over a registry protocol. The details of how this works under the hood don't really matter. But the important thing is that there are different layers that get merged together. So if you have a sequential list of tar balls that contain non-overlapping paths, then they will be squashed into one file system. If you have overlapping paths, then the ones from the latter ones in the order of the image manifest appearance are the ones that take precedence. So this syntax over here is what's called a Docker file. This is the default method of building Docker images. You specify the base image that you have at the top, which in this case is Ubuntu, and then you run individual commands. And there's a bunch of additional instructions such as N which sets environment variables and so on. What happens when you execute this is that Docker goes and it takes every single instruction in this Docker file and it executes it and it takes the resulting diff of file systems and turns it into a layer. What this means is that this is inherently extremely stateful because there is no care taken to make sure that things like timestamps on files are in any way normalized. So things that we deal with in Nix on a very fundamental level just kind of don't exist conceptually in this world. So I actually took this particular Docker file and I ran it twice earlier today. And here are the hashes resulting from the two image builds. As you can see they have nothing to do with each other at all. This was kind of my reaction. And a while back I already started thinking about whether or not there might be a better way to do this. Those of you who talked to me at the Nixcon in 2018 probably heard my ideas about the Kubernetes controller for Nix. I will get to that in a second. But also there have been other people working on very similar things for a while. There was an initial attempt, actually not an attempt at work to build a Docker image via Nix called just build image. That's in docker tools dot build image. And what that does, the way I understand it is it spins up a QMU, installs the things that you want in it and then takes a snapshot of that file system plus some extra information. That was the first idea for building Docker images in Nix and actually worked reasonably well. And then Graham C. Is he here? Somewhere he is. Maybe not in this room right now. Anyways, he wrote a thing called build layered image that I will be referencing a couple of times throughout the talk, which takes a more pure Nix approach to the construction of the layers. So instead of spinning up virtual machines and snapshotting their disk, it actually uses Nix to construct the tower walls themselves. Right. So about a year and a half ago I had this idea for what would the world look like if we didn't actually have to build these container images explicitly. And we just have some sort of content specification that tells us what we want in an image and the image is built based on that. So here's a mockup of something that I kind of prototyped in the beginning where we added an extra resource to the Kubernetes API. That's something that is supported in Kubernetes. You can see that up here. And then we had this simple YAML specification where each of the keys referred to a top level entry in Nix packages. And you kind of submitted this into your cluster and then a demon kind of ran in the back of the cluster, built an image out of this, pushed it into your registry and did this whole normal stateful kind of dance, but abstracted away for the user by just creating an image with this name. After building this, an interesting thing happened. So I tried to demo this to people. And it turns out that when you're demoing this, there's a lot of moving parts in it. You need to write a YAML file. You need to submit the YAML file to a Kubernetes cluster. Then a thing needs to run in the background and build the image and push it into a registry. And then Kubernetes launches pods that pull the image from there. So you've got six, seven things that can go wrong and they all take a little bit of time. So it's the kind of demo where you try to show it to someone. You make a few typos and they're kind of half falling asleep. And like, trust me, this is a valuable concept that I'm trying to get across. So I actually invented on the fly at some point this idea of a demo mode for the controller. So the idea here was that instead of having a specification pre-hoc of what goes into the image, we just use the components of the image to describe what should go into it. So there are kind of two main parts here. Just the host name, which is just the address at which Docker should look up the image that you're downloading. Then we've got a section that describes the packages. And the first ones in here are so-called meta packages, which is kind of a shorthand for sets of packages. So Shell, for example, gives you a bash interactive, IANA, Etsy, and like all these kinds of things that you probably want if you want a shell environment. And then you can specify paths separated other packages that should go into this image. So I built this demo mode, and then I realized that there's actually a much better idea than the first one I had. And then I kind of scrapped the actual controller, and I was just like, this is the way forward. Right. Before we talk about this slide, I'm going to start doing my first little demo just to show you what this thing actually looks like in practice. Okay. Microphone number two. So here's the public instance of Nixery. It's available at nixery.dev. And this is kind of like a public service that you can go and play with. So just to remember the... I can, I can. You don't actually need to read this. I'm just trying to show the logo as often as I can because I spent countless hours designing this, as you can see. So let's do a little demo of this. Here's the standard Docker CLI. What this command says. Is that readable enough? No. No? Okay. Excellent. What this says is we're going to run a container. We're going to attach a TTY interactively. We're going to remove the container after we're done. And then we will pull the image from nixery.dev and we want an image containing shell. So as per the previous slide, this is going to give us an image that just contains a basic bash environment in a second. So I need to actually tell it what to run. So here we go. Here's the bash. There are four utils and all sorts of things installed, but no additional programs. If I tried to run Htop, which I didn't specify in the image, it's not going to be included. So what I can do is I can exit out of this and I can edit it to include additional programs. Again these are just referencing keys in nix packages. And then an image is built on the fly and kind of sent to me. And now I have a shell in which I can actually run Htop. So that's the basic mode of operation. In this public instance that's running at nixery.dev, I track whichever is the latest nix-res channel and I do updates for this roughly once or twice a week. But it turns out that what's actually a much more interesting use case than just having a thing that can give you these arbitrary images at Hoc is the ability to add your own private services into it and see how this can be useful for CI and organizations. So there's something I have to show. I personally am a monorep of personal and here is the default of nix of my personal infrastructure repository. It contains stuff like my blog and so on. And this is mostly standard nix stuff. So there's nothing like super interesting to show here. I import nix packages from a pin commit and then I have an overlay that overlays my standard packages on top of it. If you want to know how this works in detail and have discussions about pinning channels and so on, we can do that after the talk because it's a slightly different thing. Anyways, in here I have a thing that I built which is the nix-con demo using nmateers. Where is he? Also somewhere. Excellent Nash which builds Rust based on lock file specifications. Huge fan of that. And I wrote a tiny, that's the wrong one, I wrote a tiny demo which just looks like this. This can either run as a web server showing me this nice message. I hope for the Czech people here that I spelled that correctly, trusting the internet or run with the CLI argument. It doesn't really matter what this program does. I just want to show you that there's an actual thing going on here. So I'm importing that into my nix package set and then I have it available in my overlay of packages. So down here I have a shell in which DRNV has just, thanks to Zimba, has just loaded some environment variables for me. And one of them is the one called nix-ery packages path. So I support multiple different sources for importing package sets into nix-ery. And one of them is using git repositories and one of them is using local file paths. In this case I'm just using the file path of the checkout of my repositories. I'm telling nix-ery that the package set from which I want you to build images is the one I have at this place on my disk. So what I can do now is I can go and spin up a nix-ery in here, which looks like that. And then over here on the other side, I can do the same thing I was doing before with my shell. But instead of pulling from nix-ery.dev, I'm pulling from localhost. So this is the local instance I have running. So this one wasn't particularly interesting because it was cached already, I think. But yeah, I just want to show that there is nothing in here right now. So there's no program called nix-condemo. Now I'm going to remove that image just to prove to you that I'm not going to try and hit any caches here. I don't actually, ah, what help if I could spell? I don't have it even. So then we can go and run this command again and add this one to the image. Here we go. Could not find nix package. Thanks. Here we go. Okay. So what was supposed to happen is that I pull an image down now and it contains the binary that I've added here. Quick show of hands, how much time should I spend working on getting that demo up and running versus not? Okay, I'll give it 20 seconds. So that looks correct to me. And then here at the top somewhere. Yep. Ah, okay. It's just called nix-con. Here we go. Right. And now I have my nix-con demo thing in here built from the package set, overlaid into my local configuration. Cool. So that works. It actually turns into kind of interesting way of deploying services because in the configuration mode where I let you specify git repository instead of a file path, I can translate tags into git commits, which we'll see in a bit. And then you kind of start getting the ability to build CI pipelines out of this. Before I show that, I want to talk about something that I spent way too much time on. As a sort of premature optimization, but it was an interesting thing to go through. And this is something that Graham kind of kicked off when you started building build layered image. For historical reasons, Docker has a maximum number of layers that you can have in an image. It's around 125 for some reason. And it turns out that even using up all of those is not a very good idea because you share this layer restriction with your user. So if you have users that want to pull an image and extend it by adding additional files, you count if there's already 125 layers in the image. So Graham had this initial idea of creating a graph of all the dependencies that a given image needs and then counting the popularity of individual things inside of that graph. So if you have more than one derivation that references something like G-Lip C, then G-Lip C becomes more popular and eventually you sort by this popularity and kind of pick the most important ones from the top and hope that those are going to match. But in this case, you have a maximum of one derivation or one closure per image layer. But there are many situations in which some kind of thing depends on, let's say we have a program called foo and it depends on foo data. There's never really a good reason to split foo and foo data apart from each other. We want to actually put them in the same layer and make sure that we get kind of optimal caching here during the builds. So I'm going to walk you quickly through how that works and there's a more detailed write up for those of you that like graphs. Right. I suspect that some of this is not going to be readable all the way from the back, but I'm going to very quickly talk through this. So I had to find something that doesn't have a whole lot of dependencies to actually demo this. My initial attempt was to use git, but git depends on literally the entire universe and every single Perl package that exists. So that's a lot of packages. What we have here is you see a blue node at the top that's called the image root. And then there's a few arrows that go from that to the top level of this graph, which are images, which are packages that the user requested. Specifically here, there's htop and there's nano and there's a shell somewhere and that's kind of it for this image. Actually nano is part of the shell alias. And then if we look at this, there's a few interesting things in here. So for example, Perl is only referenced once by the more utils from up there, but Perl itself actually turns out to be a very popular package. There's a lot of things that depend on Perl. So we would like to use this information and kind of make sure that Perl ends up not bundled together with more utils. And similarly we can see that glip see is extremely popular. It's got a lot of arrows going into it. So what I started doing at this point is together with a few people in IOC and also at work, we devised an algorithm for counting the popularity of all packages and NICs packages by just checking how many runtime references there are to any given derivation and then creating a kind of popularity metric by multiplying that with the size of the closure so that we can get kind of the best hash hit performance in terms of data transferred and whatnot. So the way this works is that the red ones here have been identified by NICS-R-E as popular packages and we go and we draw extra edges from the root to those popular packages which in the case of Perl, for example, now means that there are two paths going into Perl. That's the red one up here. And then we create what's called a dominator tree of this, which is a relay outing of a graph into a tree in which every node is preceded by the ones that all paths to it must pass through. If that was confusing, the Wikipedia article will explain it to you. That ends up looking like this and suddenly we have a very ordered graph of how these things look. So we have the image root and then we have a top layer which actually matches very well with the images, the image layer so we would like to stick into the same table. And then stuff like more utils which depends on some of these not very popular Perl libraries and relatively speaking to the rest of the ecosystem anyways. And we can bundle these together into the same layout. So once we get to this point, we can calculate the ideal layer layout which looks like this and then based on the budget we have available for layers, we can start merging the least popular things together. And then we actually get pretty good build caching out of it. Cool. Does this make sense so far? Does anybody would like to inject a question at this point? Right, let's move on. So what about image tags? One component of image names I didn't mention before is this yellow one here at the end. It's the image tag. For Docker this is commonly versions that people add to things. So they will have something like Ubuntu, colon and then an Ubuntu release version or whatever. But I realized that we can actually use this information and map it onto something else, specifically Git commits. So if you're pointing nixery at the public nix packages wrapper, you could use this to get arbitrary references like specific channels or specific commits. If you're pointing it at a private repository, you can start having your CI substitute these parameters and then you can use this as your deployment strategy. So to quickly show that, I will show you a demo involving Kubernetes and this is going to be the one that will definitely fail. Wish me luck. All right. What I basically want to show first of all is the deployment manifest for my blog. So my blog, which most of you haven't read and I don't really post to it, so whatever. But it's an interesting experimental playground for new technology. I have this Kubernetes manifest that deploys the blog. For those of you not familiar with Kubernetes, most of the fields in here don't really matter. It's just a kind of verbose thing. The primary line in here is this one here. It specifies the image to run. I'm going to zoom in a little bit just to make sure that people can see this. Where what's basically going on is that I'm pulling an image from nixery.local, which is a private DNS name pointing to a nixery instance running inside of my cluster. How to do this is something I'm currently writing documentation for. So people will be able to start doing that on their own very soon. And then I want the attribute task gen.blog, which is the derivation for my blog. And I specify the version as githead. So this syntax here is from a templating tool that I use called contemplate, which basically just inserts the current git commit of the repository that you're in. Right. So what I can show you at this point is I can describe the deployment for my blog in Kubernetes. I'm live speaking to the Kubernetes cluster in which I deploy my personal things. And we can see that the image is currently some commit in this git repository. I can take this commit and look at what it is exactly. And I know that this is the one on which this particular image was built. So to do something a little interactive now, I have spun up the nixery inside of the cluster. Here's the live.log stream from nixery. And I'm going to, first of all, run a container inside of Kubernetes and attach to it interactively. And then we will do something more interesting. So the way this works here is syntactically very similar to Docker. So if you saw the previous command and understood it, it's pretty much the same thing. We're running a container. We're attaching a TTY. We're doing this interactively. We're giving it some randomly generated name because they don't want to conflict. We're not restarting it. And we're pulling the image from nixery.local slash shell. So it's just going to give me the standard bash environment. If I run this down here, I will see that a demo container is now creating. And after a couple of seconds, I get the shell out of it. Cool. So there's nothing in here as before. There's no extra commands. I'm not going to demo Htop again. Instead, I'm just going to let you know that this nixery instance is pointing at the repository I was showing you earlier. So I can actually go and pull the nixcon demo that I was showing up for a second into this instance and deploy a container with it. I'm just going to go ahead and do that for talking too much about it. And now we should start seeing messages from nixery here. So let me just make sure that this stream is not dead. Yeah, here we go. So there's now a build happening. And if domain is doing its job well and cache access up and running, then this should hit the binary cache. If it doesn't, it's going to be interesting because this is a Rust thing and the Rust compiler is not the fastest one. And then I'm probably going to end up in a situation like yesterday with the Haskell demo. So let's give it a moment and see what happens. Actually on the topic of caches in this case, what I found to be a very effective way of working with this is having a CI set up on your system that populates the binary cache, something like cache x on every push so that all the things are already pre-built when nixery tries to go and fetch them from a binary cache. Nixery can in theory go and start building everything from scratch, but you probably don't want it to be spending a couple of hours building your Haskell services. So it's nice to have that done already when you get to the point of deployment. What we're interested in here is not necessarily having everything pre-cached on an image level because we want to be able to dynamically modify what goes into an image, for example, by adding debugging tools, but it's still nice to actually have offloaded the caching of stuff like your actual service builds to nix. Okay, here we go. The nix build is done. A bunch of stuff happened under the hood, and now we have a command prompt here that should actually give me the nixcon demo. And now if I forward... Let's see if we can get this to work. This is the one. If we forward something like port 42, 42 into this container, then I should be able to... Right, that demo worked. I'm surprised. The trivial first one. That's how it always goes. Cool. So Kubernetes is pretty much one of the ideal ways of using this in production, and it's kind of based of the origin of the idea where this whole thing came from. And the deployment of Nixery into a cluster is relatively simple. Like I was saying, this is going to be documented on my page. You basically just run a nixery instance and add some environment variables to configure it. There's nothing really special about it. The one restriction that is in place at the moment, simply because I don't have time to implement support for everything, is that this only runs on Google Cloud Platform right now. There is a mode coming up for just being able to use a local file system instead of serving image layers from our storage system. And once that happens, you can actually run this anywhere. Also contributions welcome if somebody wants to implement that. The issue tracker describes how it should work. And once you have that up and running, you can create these kinds of things to solidify the deployment infrastructure using this service. Right, that's kind of the primary thing I wanted to say. Here are my contact details and where to find source code and all that kind of thing. Yes. And I want to say thanks to Sarah and Ida for coming up with the broccoli alias, which we kind of picked up late yesterday evening. Right, questions. So, two questions. First, in the URL, you specify the path and you specify packages or the versions using slash. Yep. Did you try to, or did you consider using NIC's expressions there? Yes. The primary issue is that there are restrictions in the Docker registry protocol in which characters are allowed and it's quite restrictive. So for example, an interesting fact, there's an issue on the issue tracker about this. You cannot have uppercase characters in there. And NIC's packages has things like Haskell packages, which is an uppercase thing, but you might want to refer to packages inside of that that are actually binary. So what I did is I tried to do a first lookup with the exact name that the user specified. And if that doesn't work, I normalize the casing of all attributes and then do a lookup with that. And some of you are probably guessing this right now. Yes, there are name clashes based on casing inside of NIC's packages. Fortunately, at the top level, this only affects things that are actually aliases for each other, but inside of Haskell packages, there are actually different pieces of software with names that are only different in the casing. So for those, I just don't have a solution at the moment. If you need one of those, I don't know, make a private wrapper on aliases, it's something sane. With the original Kubernetes controller idea, I also considered adding a field just to straight add a NICs expression in there, but since that idea has kind of been dropped, I thought that it's probably more reasonable to actually keep those in the NICs repository rather than inside of the deployment manifest. Okay, thank you. And the second question will be, how do you set up private Git repository with it? Lastly, I'm interested with the keys. So I'm using the built-in fetch Git functionality to fetch the repo, and built-in fetch Git uses the SSH configuration of your environment. So it's impure, which means that as long as the environment in which Nixery is running is configured with appropriate SSH credentials, then everything will just kind of work. Thank you. Cheers. All right, anything else? Yes, I've missed that bit, but I wonder, Dockerfile has a lot more to express like mount points, X-X, whatever. Is that handled with Nixery, or do you have to build some kind of command line on that? So at the moment, most of those things, because they're kind of only metadata outsourced to Kubernetes, so you specify stuff like the entry point inside of the deployment manifest. There is an effort in progress in Nix packages right now to add metadata to every derivation that contains binaries that tells us what is the primary binary in this package. And once that is in place, we can actually automatically generate entry points. But then there's also a discussion of if somebody writes shell slash engine X slash n grog, which one do they actually want to run? So at some point, you still have to defer to the user. As for environment variables and that kind of thing, in theory, the Nix builder that we have to do this kind of stuff supports them, but there is currently no surface in the API where you can actually attach that. We might come up with something later. There's a few issues about related things on the tracker. Feel free to give some input. Other examples of other meta packages besides shell? And what can you do with those meta packages? So shell is the only one that the public instance supports at the moment. The next one that I'm currently working on is RM64 to serve your RM64 binaries. The thing with that is that the Docker registry protocol has sort of support for switching architectures, but it does it in the form of serving you one manifest that contains hashes for all architectures. So if you were to use that functionality, you would have to build all things for all architectures on every request, which is probably not what the user wants. So I'm thinking that meta packages as the first element are kind of a more sane way of toggling that behavior. All right. Thank you very much. Thank you.
understand it is it spins up a QMU, installs the Docker images in Nix and actually worked a little demo of this. Here's the standard Docker then if we look at this, there's a few the Docker registry protocol in which characters