Nix modules: Improving Nix's discoverability and usability
Dolstra, Eelco
Nix's configuration language is quite powerful, but suffers from a lack of discoverability, usability and consistency. In this talk, I'll describe an experimental Nix module system that provides a consistent, discoverable mechanism to write configurations such as packages and NixOS systems, and show how this enables a better user experience for both new and advanced users. Nix's configuration language is quite powerful, but suffers from a lack of discoverability, usability and consistency. To name just a few examples: There is no easy way to find out from the command line or from the REPL what arguments are supported by functions like stdenv.mkDerivation or buildPythonPackage. Mechanisms like the .override attribute provide an almost unlimited ability to customize packages, but the only way to figure out what you can override is to read the source of the Nix package, and writing overrides is often black magic. NixOS has a nice self-documenting module system, but Nix packages are written in a completely different functional style. The Nix CLI doesn't know anything about package functions, .override and .overrideDerivation, the NixOS module system, the Nixpkgs config attribute set, Nixpkgs overlays, or any other customization mechanisms that have emerged over the years. The syntax and semantics of Nix expressions are often an obstacle to new users and have a steep learning curve. In this talk, I'll show an experimental Nix module system, similar to the NixOS module system, to replace the "functional" package style using in Nixpkgs. This means that functions like mkDerivation or buildPythonPackage as well as packages become modules that can build on each other. For instance, the "GNU Hello" package is a module that inherits from the unixPackage module, which in turn inherits from other modules like derivation. Package customization is done in the same way: by inheriting a module. These modules, just like in NixOS, have types and documentation. As a result, everything becomes discoverable and modifiable from the command line. For instance, there is a command nix list options that shows everything that can be customized in a package. It also provides a standard for documentation: the command nix doc generates HTML documentation for the modules in a flake.
 Okay, we have our first talk here. This talk is called Nix Modules, improving Nix's discoverability and usability. So our speaker today is none other than Ilkel Dostra. He I think he's honestly one who wouldn't really need an introduction because he is the creator of Nix. So let me go over the topic of this talk. This talk is about Nix's configuration language, but in its very powerful but suffers from a lack of discoverability, usability and consistency. In this talk, Ilkel is going to describe an experimental Nix module system that provides a consistent discoverable mechanism to write configurations such as packages and Nix with systems and show how this enables the bear user experience circles new and advanced users. And I will say Ilkel's current work setup is that he is a senior engineer at Tugayo and he joined that in 2018. Okay, great. Ilkel, take it away. All right. Can you hear me? Okay. Okay, great. Yeah. Let me start my screen sharing. So unfortunately, it appears that in Jitsi you cannot share a screen and video at the same time, at least not without the video becoming horribly blurry. So sadly, you won't be seeing me for the rest of the talk. Okay, so in any case, thank you all for that. Thank you, World of Peace, for the introduction and a big thank you to all the organizers. So yeah, of course, a slight downside compared to a physical Nix con is that I have no idea who I'm talking to, whether I might be talking to a void, but I'll just imagine that all of you are here. Yeah, so this talk is about... Ilkel, just to interrupt you one moment. It seems that at least with our infra sort of status on your slides and your screen share, I cannot actually see it. So do you think you could talk about that on and off? I'm sorry, everyone watching. Yeah. Okay, so tell me when you toggle that again and I will get a confirmation that they can see that. Okay, give me one second. I have toggled it. Can you please check the second slide? Yeah. Talk to the shows on the live stream. So now on the second slide. Okay, I believe that your screen sharing is working. You can go to the first slide and begin your talk. I'm sorry about that. It just doesn't seem to appear to me, but it is working a live stream. Okay, proceed. Okay, well, I'll monitor the comments here. So yeah, just let me know if it stops working. Okay. So yeah, this talk is about a bunch of sort of brainstorm sessions we've had, or I have had together with a couple of other people in particular, rock garboss on how to make Nix more beginner friendly and more user friendly. Now so the warning, this is quite a lot of vapor wearing this talk. So most of the things I'm describing actually do exist. There is a sort of proof of concept of the things I'm going to talk about, but it's very far from being ready for actual use, let alone merging. So this is all kind of an exploration of some ideas on how to make Nix more user friendly in various ways. But I mean, it's very far from being in a state that you can actually use. Okay, so with that having said, let's talk about what the problems are with Nix currently. So Nix is of course, it's awesome, but it's not very beginner friendly. There's a very steep learning curve. So you have to learn all these new concepts like immutability and Nix store and Nix expressions in particular. That's the big wonders is how kind of strange language that you need to learn, especially if you're not familiar with functional programming. And yeah, that's a steep learning curve. And probably a lot of people never get over that curve. So they just go away and give up. So maybe we can improve that. The second issue is that even if you are over the hurdle, you're an advanced user. There are actually a lot of things about Nix that are just not that great to use in particular. And Nix is wonderful in how possible it makes to adapt packages and configurations. So there's probably no other package manager where it's even possible to just say, I'm going to rebuild my entire system with a patched version of GCC. Or I'm going to set up a developer shell with these packages, but in some random combination with some patches applied and compiled with these versions of compiler. So I mean, that's really great about Nix. But those configuration mechanisms are for the most part not easily discoverable. So you need to basically Google about them to find out that they exist, how you use them. Yeah, another problem is there are actually a bunch of configuration mechanisms and they all work in slightly different ways. And they're often not very easy to use. So what can we do about this? So in this talk, I will discuss adding a more uniform, sorry, that was a comma. So yeah, so we're going to try to provide a more uniform and discoverable configuration mechanism, make things more discoverable from the comment line. And maybe in this gets a bit controversial, provide a simpler configuration language like Toml for simple projects. So that would make the hurdle a bit less big for beginners. So a quick look at the configuration mechanism in Nix packages and what the problem are. And so you have function arguments, the override, config, overlays, NixOS modules. So all these things have sort of grown organically over time and they're all slightly, well, they serve different use cases. But it's not clear, for instance, why for NixOS we use these modules and for packages we use this function argument style of configuration. In particular, the function argument style. That's this style where you write packages as functions that take a bunch of arguments like the dependencies and user configurable things like CUDA support. And so this is the Nix expression for blender, which you can enable CUDA support. So if you build this thing, yeah, so if you want to have blender with CUDA support, there's actually no way to find out that this option exists except by reading the source of this Nix expression. So it's not discoverable at all. And once you have discovered that this option exists, maybe by Googling about it, how do I enable CUDA in blender, you can't actually use it from the comment line. So you might think that something like Nix builds, blender, arc, CUDA support through does the right thing, but it doesn't because that argument doesn't get propagated all the way down. Other problems, there's no type checking. It's kind of ugly that it mixes user-facing configuration with dependencies. So this is not a very great configuration mechanism. Now we have dot override. I won't go into details, but dot override is another of these things that grew kind of organically. It has all sorts of problems. It's inefficient because you need to do under the hood, it calls a function twice just to get this dot override function. And it leaks memory. So if you're wondering why Nix evaluation takes multiple gigabytes, well, this is one of the reasons. Yeah, then there is the Nix packages configuration file. So you can put things like this inside a config file. But again, it's not discoverable. There's no type checking. It's not obvious how you use it from the comment line. So yeah. Now, then finally, there's overlays. Overlays are great, but the syntax is kind of mysterious. And it's also not obvious how things like nested overwrites work. So if you want to override something at top level, it's easy. If you want to override something deep down like in Python packages, it's not obvious how it works. And then there are NixOS modules. And NixOS modules are great. They're awesome. So they basically tick off most of the boxes. They're discoverable. You can just type manconfiguration.nix. They have documentation. They have types. I mean, there are some problems. They're entirely a library feature. So Nix, the comment line tool, doesn't know anything about them. So you can't easily query NixOS options from the Nix comment line or set options or do things like that. So yeah. So OK. So let's, as a solution, since NixOS modules are so great, maybe we can just use NixOS modules everywhere. So turn NixOS modules into a language feature, maybe make some improvements, like clean up the syntax and semantics and then use it everywhere. So instead of, for instance, having functions that create packages, we can have modules that create packages. So here's an example of what that looks like. So this is a module that builds Hello World. So if you know NixOS modules, then the syntax is probably familiar. Probably we would actually want to improve this syntax if we really turn this into a language feature because then you're no longer confined by having to have these weird constructs like having config as a function argument and as an attribute name. But apart from that, it should look familiar. So what this module does is it extends a bunch of other modules like standard nf. So in this modular world, we no longer have a function called standard nf.make derivation. That standard nf is a module that you can extend or inherit in your own modules. And that should, in theory, make it easier to combine modules. So for instance, if you have a package that both includes Rust code and Python code, you could just include the module for building Rust and the module for building Python. So what this module does is it says Hello World by default, but the text of the greeting is configurable at compile time. So it has a who option that you can set at build time. And that option is used in the configuration of this module. So the configuration sets a bunch of options that it inherited from the standard nf module and the package module, like the package name, the version, the build script. And that build script then generates a Hello World binary. All right. So how would you use this? Well, basically the same as how you use Nix now. And so by the way, I should mention that this is intended to be part of a flake. So modules.hello would be a flake output. So just as a flake can have packages now, it can have modules. So if you say nixrun.hello, that will build the Hello module from the flake in the current directory and run it. And that's actually equivalent to building and running modules.hello.final.derivation. And derivation is an option inherited from one of these modules that we're extending, like standard nf, that evaluates to a derivation graph. So just like in NixOS, you have a option called system.build.topLevel, which builds your entire system. Here we have an output option called derivation, which returns the derivation for your package. So the whole point was to make things discoverable. And now we have that because we have a standard structure and we have options that have descriptions and types. Actually, we don't have types yet, but we could have them just listen to NixOS module system. So yeah, we can discover via command nix list options, and that will show us all the options in this package. So for instance, this is how we discovered that this thing has a who option. And then we can override this from the comment line. So we can, for instance, say, Nix run hello, and we can override who to Nixcon. And that will build a new derivation and run it. Now you might be thinking aren't flakes supposed to be hermetic? So you can't actually override something inside a flake. So what this actually does is it on the fly, it constructs a new flake in a temporary directory that imports the flake in the current directory. So basically it generates something that looks like this. And this is what you would write yourself if you're trying to extend the module. So you could have a flake with a module named my hello that extends the hello module and sets Nixcon to who. So this is basically the replacement for the dot override or dot override derivation mechanisms. Another nice thing to mention is that we now have a documentation mechanism, a standardized documentation mechanism. So there is a command called Nix doc that given a flake generates the documentation for that flake just like cargo doc. It actually uses MD book, which is also what cargo uses. So it takes all these markdown doc strings of all the options and modules and generates documentation out of them. So the goal here is to have a standard uniform documentation mechanism for all flakes. So in the future you should be able to say Nix doc on any flake and get documentation out of them. All right, final thing to mention is the beginner friendliness. So if I go back of two slides, so this Nix expression, there's a lot of mysterious code in here, like this whole self hello stuff, this config thing over here. So that might be daunting for new users. And in fact, it's kind of unnecessary because with modules, modules are really just name value pairs like who, Nix, gone. So you don't need the whole Nix expression language just to customize a module. So the idea is you can just write a tunnel file, you still have to declare your flake dependencies, but then you can say who is Nix gone. And of course we're extending a certain module and that's it. So yeah, you could have a more user friendly syntax. And another advantage of a simpler syntax like tunnel is that this is easier to generate so we can have a command line mechanism to modify a flake. So we can just provide a command to set options or add dependencies to a Nix shell or things like that. And that's less error prone than telling new users that they need to edit a Nix expression and get the syntax exactly right. So that's the idea. All right, so that's it actually. So the next steps that we're interested in doing is, well, actually try to improve the syntax and semantics of this toy module system because right now it's an extremely bare copy of the module system that NixOS has and it has no types or merge functions or stuff like that so it's extremely bare. But yeah, so we'd like to come up with a better syntax and semantics and improve on the NixOS module system. For instance, it would be great to actually have sort of proper scoping. So for instance, a module should not be able to access options that it hasn't explicitly inherited from another module. And of course, we'd need to do a lot of experiments to see whether any new mechanism actually meets all the current use cases for the currently existing configuration mechanisms. I mean, so stuff like overlays or package sets. Yeah, we need to see whether we can actually do that. Okay, so yeah, there is a proof of concept implementation, but so it contains all these commands like NixDoc and NixListOptions. So you can actually play with it, but it's very bare at this point. And there are actually some examples that you can check out. Here is a document somewhere that has more ideas about these language changes. All right, that's it for this talk. I understand I can now answer questions, so I need to figure out where I can actually see the questions. Hi, so I will just read those off to you. You are running early, so your Q&A will actually have two extra minutes if you want. Okay, let me read the first one we have in the pad. So this is from Andy on FreeNode. It says, what is the cost of having more features that other language already built into Nix? Sorry, can you repeat that? I didn't quite catch it. Okay, I can repeat that. What is the cost of having more features that other languages already built into Nix? What is the cost? Well, obviously there is a cost in terms of we have to implement it. It makes the language more complex because of course the existing language features don't go away. But I mean, there's also the cost of not having this. So if we don't improve things like discoverability, we might lose a lot of potential users. So I would more see it from that perspective. All of this is motivated by thinking in terms of use cases. So if a user wants to set up a Nix shell, what do they need to write? And currently, if they for instance want to write a flake for their project, they have to write a lot of boilerplate codes just to get that to work. So they need to think about what would an ideal Nix expression look like. Or maybe you don't want to write an Nix expression at all and you want some other syntax for that. So yeah, I guess that's not a very good answer, but I don't actually know how to quantify the cost very well. Right. I do like that you're considering what an optimal Nix expression would look like. So the next question we have is from David AK and it is about why was TOML chosen? Do you think you can answer that? Yeah, so I mean, I'm not very married to Toml, but it exists. It's well known. So yeah, a lot of tools have support for it, editors have support for it. And I mean, you could go for Jason, that would be the simplest choice, but Jason is maybe a little bit too primitive because you can't even have things like comments in it, at least not in a nice way. But yeah, so Jason, Toml, Yammel, those would be possibilities. And then Yammel, I mean, Yammel is maybe a bit too complex. But yeah, so Toml is kind of, it's not too bare bones, so it seems like a fairly decent choice. Okay. Yeah, I think it's a fairly decent choice as well, actually. I think of those three options you mentioned, you picked the test one, but that's just my opinion. So another question from Andy, and I hope this is an okay question to ask, you can tell me if it isn't, but will I also need a Nix integration test at some point? I wonder what's wrong with just Nix build the attribute flag and then doc. Ah, okay. So that actually works. Sorry, does it work? No. So ideally at some point it would just work. So you would just have a doc option that generates the documentation for your module. But well, no, actually, that would be quite hard to do. No. Yeah, so the Nix doc command internally actually, it actually just builds a Nix expression that takes the flag as an argument. So there's actually not a lot of mysterious going on there. So you could actually do it with a Nix build and probably with an argument. But I mean, that's not very user friendly. I mean, you have to, again, we want to get avoid magic in incantations. So just saying Nix build or Nix doc is a lot easier than figuring out the correct Nix builds attribute name and so on. I hope that answers the question. Yep, I think that does. Let me see if there's any more questions. I actually do not think there's any more questions and I believe that ends the Q&A portion. So yeah, we are on time for that. So I guess this concludes your talk today. So thank you so much. I also, I would like to mention something just briefly before we sign off. I really do like your avatar on GitHub as Dexter. Is that some sort of inside sort of thing? Like, because I do notice that Dexter is sort of like a precocious little genius that has like a laboratory in his room and he's always working on experiments. And I've noticed for the past Nix con that you've also just like talked about your experiments. So you're both sort of like scientists in your own right. Yeah, I mean, it's not really any deep thought that went into that. But I was a Dexter fan in the 90s. And we both wear glasses. So I guess that's close enough. Okay, I guess that signs off for today. Thank you so much. Everyone in the chat, please put clapping emojis or you know, loud clapping sounds and you know, we're live and direct. Can you also like out now, we talk about binaries, put ones in the chat like go crazy. Come on. I want to see the chat go crazy.
 Okay, we have our first talk here.  This talk is called Nix Modules, improving Nix's discoverability and usability.  So our speaker today is none other than Ilkel Dostra.  He I think he's honestly one who wouldn't really need an introduction because he is  the creator of Nix.  So let me go over the topic of this talk.  This talk is about Nix's configuration language, but in its very powerful but suffers from  a lack of discoverability, usability and consistency.  In this talk, Ilkel is going to describe an experimental Nix module system that provides  a consistent discoverable mechanism to write configurations such as packages and Nix with  systems and show how this enables the bear user experience circles new and advanced users.  And I will say Ilkel's current work setup is that he is a senior engineer at Tugayo  and he joined that in 2018.  Okay, great.  Ilkel, take it away.  All right.  Can you hear me?  Okay.  Okay, great.  Yeah.  Let me start my screen sharing.  So unfortunately, it appears that in Jitsi you cannot share a screen and video at the  same time, at least not without the video becoming horribly blurry.  So sadly, you won't be seeing me for the rest of the talk.  Okay, so in any case, thank you all for that.  Thank you, World of Peace, for the introduction and a big thank you to all the organizers.  So yeah, of course, a slight downside compared to a physical Nix con is that I have no idea  who I'm talking to, whether I might be talking to a void, but I'll just imagine that all  of you are here.  Yeah, so this talk is about...  Ilkel, just to interrupt you one moment.  It seems that at least with our infra sort of status on your slides and your screen share,  I cannot actually see it.  So do you think you could talk about that on and off?  I'm sorry, everyone watching.  Yeah.  Okay, so tell me when you toggle that again and I will get a confirmation that they can  see that.  Okay, give me one second.  I have toggled it.  Can you please check the second slide?  Yeah.  Talk to the shows on the live stream.  So now on the second slide.  Okay, I believe that your screen sharing is working.  You can go to the first slide and begin your talk.  I'm sorry about that.  It just doesn't seem to appear to me, but it is working a live stream.  Okay, proceed.  Okay, well, I'll monitor the comments here.  So yeah, just let me know if it stops working.  Okay.  So yeah, this talk is about a bunch of sort of brainstorm sessions we've had, or I have  had together with a couple of other people in particular, rock garboss on how to make  Nix more beginner friendly and more user friendly.  Now so the warning, this is quite a lot of vapor wearing this talk.  So most of the things I'm describing actually do exist.  There is a sort of proof of concept of the things I'm going to talk about, but it's very  far from being ready for actual use, let alone merging.  So this is all kind of an exploration of some ideas on how to make Nix more user friendly  in various ways.  But I mean, it's very far from being in a state that you can actually use.  Okay, so with that having said, let's talk about what the problems are with Nix currently.  So Nix is of course, it's awesome, but it's not very beginner friendly.  There's a very steep learning curve.  So you have to learn all these new concepts like immutability and Nix store and Nix expressions  in particular.  That's the big wonders is how kind of strange language that you need to learn, especially  if you're not familiar with functional programming.  And yeah, that's a steep learning curve.  And probably a lot of people never get over that curve.  So they just go away and give up.  So maybe we can improve that.  The second issue is that even if you are over the hurdle, you're an advanced user.  There are actually a lot of things about Nix that are just not that great to use in particular.  And Nix is wonderful in how possible it makes to adapt packages and configurations.  So there's probably no other package manager where it's even possible to just say, I'm  going to rebuild my entire system with a patched version of GCC.  Or I'm going to set up a developer shell with these packages, but in some random combination  with some patches applied and compiled with these versions of compiler.  So I mean, that's really great about Nix.  But those configuration mechanisms are for the most part not easily discoverable.  So you need to basically Google about them to find out that they exist, how you use them.  Yeah, another problem is there are actually a bunch of configuration mechanisms and they  all work in slightly different ways.  And they're often not very easy to use.  So what can we do about this?  So in this talk, I will discuss adding a more uniform, sorry, that was a comma.  So yeah, so we're going to try to provide a more uniform and discoverable configuration  mechanism, make things more discoverable from the comment line.  And maybe in this gets a bit controversial, provide a simpler configuration language like  Toml for simple projects.  So that would make the hurdle a bit less big for beginners.  So a quick look at the configuration mechanism in Nix packages and what the problem are.  And so you have function arguments, the override, config, overlays, NixOS modules.  So all these things have sort of grown organically over time and they're all slightly, well,  they serve different use cases.  But it's not clear, for instance, why for NixOS we use these modules and for packages  we use this function argument style of configuration.  In particular, the function argument style.  That's this style where you write packages as functions that take a bunch of arguments  like the dependencies and user configurable things like CUDA support.  And so this is the Nix expression for blender, which you can enable CUDA support.  So if you build this thing, yeah, so if you want to have blender with CUDA support, there's  actually no way to find out that this option exists except by reading the source of this  Nix expression.  So it's not discoverable at all.  And once you have discovered that this option exists, maybe by Googling about it, how do  I enable CUDA in blender, you can't actually use it from the comment line.  So you might think that something like Nix builds, blender, arc, CUDA support through  does the right thing, but it doesn't because that argument doesn't get propagated all the  way down.  Other problems, there's no type checking.  It's kind of ugly that it mixes user-facing configuration with dependencies.  So this is not a very great configuration mechanism.  Now we have dot override.  I won't go into details, but dot override is another of these things that grew kind of  organically.  It has all sorts of problems.  It's inefficient because you need to do under the hood, it calls a function twice just to  get this dot override function.  And it leaks memory.  So if you're wondering why Nix evaluation takes multiple gigabytes, well, this is one  of the reasons.  Yeah, then there is the Nix packages configuration file.  So you can put things like this inside a config file.  But again, it's not discoverable.  There's no type checking.  It's not obvious how you use it from the comment line.  So yeah.  Now, then finally, there's overlays.  Overlays are great, but the syntax is kind of mysterious.  And it's also not obvious how things like nested overwrites work.  So if you want to override something at top level, it's easy.  If you want to override something deep down like in Python packages, it's not obvious  how it works.  And then there are NixOS modules.  And NixOS modules are great.  They're awesome.  So they basically tick off most of the boxes.  They're discoverable.  You can just type manconfiguration.nix.  They have documentation.  They have types.  I mean, there are some problems.  They're entirely a library feature.  So Nix, the comment line tool, doesn't know anything about them.  So you can't easily query NixOS options from the Nix comment line or set options or do  things like that.  So yeah.  So OK.  So let's, as a solution, since NixOS modules are so great, maybe we can just use NixOS modules  everywhere.  So turn NixOS modules into a language feature, maybe make some improvements, like clean up  the syntax and semantics and then use it everywhere.  So instead of, for instance, having functions that create packages, we can have modules  that create packages.  So here's an example of what that looks like.  So this is a module that builds Hello World.  So if you know NixOS modules, then the syntax is probably familiar.  Probably we would actually want to improve this syntax if we really turn this into a  language feature because then you're no longer confined by having to have these weird constructs  like having config as a function argument and as an attribute name.  But apart from that, it should look familiar.  So what this module does is it extends a bunch of other modules like standard nf.  So in this modular world, we no longer have a function called standard nf.make derivation.  That standard nf is a module that you can extend or inherit in your own modules.  And that should, in theory, make it easier to combine modules.  So for instance, if you have a package that both includes Rust code and Python code, you  could just include the module for building Rust and the module for building Python.  So what this module does is it says Hello World by default, but the text of the greeting  is configurable at compile time.  So it has a who option that you can set at build time.  And that option is used in the configuration of this module.  So the configuration sets a bunch of options that it inherited from the standard nf module  and the package module, like the package name, the version, the build script.  And that build script then generates a Hello World binary.  All right.  So how would you use this?  Well, basically the same as how you use Nix now.  And so by the way, I should mention that this is intended to be part of a flake.  So modules.hello would be a flake output.  So just as a flake can have packages now, it can have modules.  So if you say nixrun.hello, that will build the Hello module from the flake in the current  directory and run it.  And that's actually equivalent to building and running modules.hello.final.derivation.  And derivation is an option inherited from one of these modules that we're extending,  like standard nf, that evaluates to a derivation graph.  So just like in NixOS, you have a option called system.build.topLevel, which builds your entire  system.  Here we have an output option called derivation, which returns the derivation for your package.  So the whole point was to make things discoverable.  And now we have that because we have a standard structure and we have options that have descriptions  and types.  Actually, we don't have types yet, but we could have them just listen to NixOS module  system.  So yeah, we can discover via command nix list options, and that will show us all the options  in this package.  So for instance, this is how we discovered that this thing has a who option.  And then we can override this from the comment line.  So we can, for instance, say, Nix run hello, and we can override who to Nixcon.  And that will build a new derivation and run it.  Now you might be thinking aren't flakes supposed to be hermetic?  So you can't actually override something inside a flake.  So what this actually does is it on the fly, it constructs a new flake in a temporary directory  that imports the flake in the current directory.  So basically it generates something that looks like this.  And this is what you would write yourself if you're trying to extend the module.  So you could have a flake with a module named my hello that extends the hello module and  sets Nixcon to who.  So this is basically the replacement for the dot override or dot override derivation mechanisms.  Another nice thing to mention is that we now have a documentation mechanism, a standardized  documentation mechanism.  So there is a command called Nix doc that given a flake generates the documentation for that  flake just like cargo doc.  It actually uses MD book, which is also what cargo uses.  So it takes all these markdown doc strings of all the options and modules and generates  documentation out of them.  So the goal here is to have a standard uniform documentation mechanism for all flakes.  So in the future you should be able to say Nix doc on any flake and get documentation  out of them.  All right, final thing to mention is the beginner friendliness.  So if I go back of two slides, so this Nix expression, there's a lot of mysterious code  in here, like this whole self hello stuff, this config thing over here.  So that might be daunting for new users.  And in fact, it's kind of unnecessary because with modules, modules are really just name  value pairs like who, Nix, gone.  So you don't need the whole Nix expression language just to customize a module.  So the idea is you can just write a tunnel file, you still have to declare your flake  dependencies, but then you can say who is Nix gone.  And of course we're extending a certain module and that's it.  So yeah, you could have a more user friendly syntax.  And another advantage of a simpler syntax like tunnel is that this is easier to generate  so we can have a command line mechanism to modify a flake.  So we can just provide a command to set options or add dependencies to a Nix shell or things  like that.  And that's less error prone than telling new users that they need to edit a Nix expression  and get the syntax exactly right.  So that's the idea.  All right, so that's it actually.  So the next steps that we're interested in doing is, well, actually try to improve the  syntax and semantics of this toy module system because right now it's an extremely bare copy  of the module system that NixOS has and it has no types or merge functions or stuff like  that so it's extremely bare.  But yeah, so we'd like to come up with a better syntax and semantics and improve on the NixOS  module system.  For instance, it would be great to actually have sort of proper scoping.  So for instance, a module should not be able to access options that it hasn't explicitly  inherited from another module.  And of course, we'd need to do a lot of experiments to see whether any new mechanism actually meets  all the current use cases for the currently existing configuration mechanisms.  I mean, so stuff like overlays or package sets.  Yeah, we need to see whether we can actually do that.  Okay, so yeah, there is a proof of concept implementation, but so it contains all these  commands like NixDoc and NixListOptions.  So you can actually play with it, but it's very bare at this point.  And there are actually some examples that you can check out.  Here is a document somewhere that has more ideas about these language changes.  All right, that's it for this talk.  I understand I can now answer questions, so I need to figure out where I can actually  see the questions.  Hi, so I will just read those off to you.  You are running early, so your Q&A will actually have two extra minutes if you want.  Okay, let me read the first one we have in the pad.  So this is from Andy on FreeNode.  It says, what is the cost of having more features that other language already built  into Nix?  Sorry, can you repeat that?  I didn't quite catch it.  Okay, I can repeat that.  What is the cost of having more features that other languages already built into Nix?  What is the cost?  Well, obviously there is a cost in terms of we have to implement it.  It makes the language more complex because of course the existing language features don't  go away.  But I mean, there's also the cost of not having this.  So if we don't improve things like discoverability, we might lose a lot of potential users.  So I would more see it from that perspective.  All of this is motivated by thinking in terms of use cases.  So if a user wants to set up a Nix shell, what do they need to write?  And currently, if they for instance want to write a flake for their project, they have  to write a lot of boilerplate codes just to get that to work.  So they need to think about what would an ideal Nix expression look like.  Or maybe you don't want to write an Nix expression at all and you want some other syntax for  that.  So yeah, I guess that's not a very good answer, but I don't actually know how to quantify  the cost very well.  Right.  I do like that you're considering what an optimal Nix expression would look like.  So the next question we have is from David AK and it is about why was TOML chosen?  Do you think you can answer that?  Yeah, so I mean, I'm not very married to Toml, but it exists.  It's well known.  So yeah, a lot of tools have support for it, editors have support for it.  And I mean, you could go for Jason, that would be the simplest choice, but Jason is maybe  a little bit too primitive because you can't even have things like comments in it, at least  not in a nice way.  But yeah, so Jason, Toml, Yammel, those would be possibilities.  And then Yammel, I mean, Yammel is maybe a bit too complex.  But yeah, so Toml is kind of, it's not too bare bones, so it seems like a fairly decent  choice.  Okay.  Yeah, I think it's a fairly decent choice as well, actually.  I think of those three options you mentioned, you picked the test one, but that's just my  opinion.  So another question from Andy, and I hope this is an okay question to ask, you can tell  me if it isn't, but will I also need a Nix integration test at some point?  I wonder what's wrong with just Nix build the attribute flag and then doc.  Ah, okay.  So that actually works.  Sorry, does it work?  No.  So ideally at some point it would just work.  So you would just have a doc option that generates the documentation for your module.  But well, no, actually, that would be quite hard to do.  No.  Yeah, so the Nix doc command internally actually, it actually just builds a Nix expression that  takes the flag as an argument.  So there's actually not a lot of mysterious going on there.  So you could actually do it with a Nix build and probably with an argument.  But I mean, that's not very user friendly.  I mean, you have to, again, we want to get avoid magic in incantations.  So just saying Nix build or Nix doc is a lot easier than figuring out the correct Nix builds  attribute name and so on.  I hope that answers the question.  Yep, I think that does.  Let me see if there's any more questions.  I actually do not think there's any more questions and I believe that ends the Q&A portion.  So yeah, we are on time for that.  So I guess this concludes your talk today.  So thank you so much.  I also, I would like to mention something just briefly before we sign off.  I really do like your avatar on GitHub as Dexter.  Is that some sort of inside sort of thing?  Like, because I do notice that Dexter is sort of like a precocious little genius that has  like a laboratory in his room and he's always working on experiments.  And I've noticed for the past Nix con that you've also just like talked about your experiments.  So you're both sort of like scientists in your own right.  Yeah, I mean, it's not really any deep thought that went into that.  But I was a Dexter fan in the 90s.  And we both wear glasses.  So I guess that's close enough.  Okay, I guess that signs off for today.  Thank you so much.  Everyone in the chat, please put clapping emojis or you know, loud clapping sounds and you  know, we're live and direct.  Can you also like out now, we talk about binaries, put ones in the chat like go crazy.  Come on.  I want to see the chat go crazy.
NixCon_2020_Vid_2_0_59.png NixCon_2020_Vid_2_60_239.png NixCon_2020_Vid_2_240_287.png NixCon_2020_Vid_2_288_359.png NixCon_2020_Vid_2_360_491.png NixCon_2020_Vid_2_492_1127.png NixCon_2020_Vid_2_1128_1235.png NixCon_2020_Vid_2_1236_1487.png NixCon_2020_Vid_2_1488_1535.png NixCon_2020_Vid_2_1536_1655.png NixCon_2020_Vid_2_1656_2195.png NixCon_2020_Vid_2_2196_3419.png NixCon_2020_Vid_2_3420_3587.png NixCon_2020_Vid_2_3588_3731.png NixCon_2020_Vid_2_3732_3803.png NixCon_2020_Vid_2_3804_4043.png NixCon_2020_Vid_2_4044_4163.png NixCon_2020_Vid_2_4164_4727.png NixCon_2020_Vid_2_4728_5279.png NixCon_2020_Vid_2_5280_6407.png NixCon_2020_Vid_2_6408_7715.png NixCon_2020_Vid_2_7716_10931.png NixCon_2020_Vid_2_10932_11771.png NixCon_2020_Vid_2_11772_12827.png NixCon_2020_Vid_2_12828_13883.png NixCon_2020_Vid_2_13884_14759.png NixCon_2020_Vid_2_14760_15227.png NixCon_2020_Vid_2_15228_15851.png NixCon_2020_Vid_2_15852_16799.png NixCon_2020_Vid_2_16800_21707.png NixCon_2020_Vid_2_21708_21779.png NixCon_2020_Vid_2_21780_22331.png NixCon_2020_Vid_2_22332_22619.png NixCon_2020_Vid_2_22620_23543.png NixCon_2020_Vid_2_23544_23735.png NixCon_2020_Vid_2_23736_24515.png NixCon_2020_Vid_2_24516_28283.png NixCon_2020_Vid_2_28284_28331.png NixCon_2020_Vid_2_28332_28823.png NixCon_2020_Vid_2_28824_29591.png NixCon_2020_Vid_2_29592_42590.png
Okay, we have our first talk here. This talk is called Nix Modules, improving Nix's discoverability and usability. So our speaker today is none other than Ilkel Dostra. He I think he's honestly one who wouldn't really need an introduction because he is the creator of Nix. So let me go over the topic of this talk. This talk is about Nix's configuration language, but in its very powerful but suffers from a lack of discoverability, usability and consistency. In this talk, Ilkel is going to describe an experimental Nix module system that provides a consistent discoverable mechanism to write configurations such as packages and Nix with systems and show how this enables the bear user experience circles new and advanced users. And I will say Ilkel's current work setup is that he is a senior engineer at Tugayo and he joined that in 2018. Okay, great. Ilkel, take it away. All right. Can you hear me? Okay. Okay, great. Yeah. Let me start my screen sharing. So unfortunately, it appears that in Jitsi you cannot share a screen and video at the same time, at least not without the video becoming horribly blurry. So sadly, you won't be seeing me for the rest of the talk. Okay, so in any case, thank you all for that. Thank you, World of Peace, for the introduction and a big thank you to all the organizers. So yeah, of course, a slight downside compared to a physical Nix con is that I have no idea who I'm talking to, whether I might be talking to a void, but I'll just imagine that all of you are here. Yeah, so this talk is about... Ilkel, just to interrupt you one moment. It seems that at least with our infra sort of status on your slides and your screen share, I cannot actually see it. So do you think you could talk about that on and off? I'm sorry, everyone watching. Yeah. Okay, so tell me when you toggle that again and I will get a confirmation that they can see that. Okay, give me one second. I have toggled it. Can you please check the second slide? Yeah. Talk to the shows on the live stream. So now on the second slide. Okay, I believe that your screen sharing is working. You can go to the first slide and begin your talk. I'm sorry about that. It just doesn't seem to appear to me, but it is working a live stream. Okay, proceed. Okay, well, I'll monitor the comments here. So yeah, just let me know if it stops working. Okay. So yeah, this talk is about a bunch of sort of brainstorm sessions we've had, or I have had together with a couple of other people in particular, rock garboss on how to make Nix more beginner friendly and more user friendly. Now so the warning, this is quite a lot of vapor wearing this talk. So most of the things I'm describing actually do exist. There is a sort of proof of concept of the things I'm going to talk about, but it's very far from being ready for actual use, let alone merging. So this is all kind of an exploration of some ideas on how to make Nix more user friendly in various ways. But I mean, it's very far from being in a state that you can actually use. Okay, so with that having said, let's talk about what the problems are with Nix currently. So Nix is of course, it's awesome, but it's not very beginner friendly. There's a very steep learning curve. So you have to learn all these new concepts like immutability and Nix store and Nix expressions in particular. That's the big wonders is how kind of strange language that you need to learn, especially if you're not familiar with functional programming. And yeah, that's a steep learning curve. And probably a lot of people never get over that curve. So they just go away and give up. So maybe we can improve that. The second issue is that even if you are over the hurdle, you're an advanced user. There are actually a lot of things about Nix that are just not that great to use in particular. And Nix is wonderful in how possible it makes to adapt packages and configurations. So there's probably no other package manager where it's even possible to just say, I'm going to rebuild my entire system with a patched version of GCC. Or I'm going to set up a developer shell with these packages, but in some random combination with some patches applied and compiled with these versions of compiler. So I mean, that's really great about Nix. But those configuration mechanisms are for the most part not easily discoverable. So you need to basically Google about them to find out that they exist, how you use them. Yeah, another problem is there are actually a bunch of configuration mechanisms and they all work in slightly different ways. And they're often not very easy to use. So what can we do about this? So in this talk, I will discuss adding a more uniform, sorry, that was a comma. So yeah, so we're going to try to provide a more uniform and discoverable configuration mechanism, make things more discoverable from the comment line. And maybe in this gets a bit controversial, provide a simpler configuration language like Toml for simple projects. So that would make the hurdle a bit less big for beginners. So a quick look at the configuration mechanism in Nix packages and what the problem are. And so you have function arguments, the override, config, overlays, NixOS modules. So all these things have sort of grown organically over time and they're all slightly, well, they serve different use cases. But it's not clear, for instance, why for NixOS we use these modules and for packages we use this function argument style of configuration. In particular, the function argument style. That's this style where you write packages as functions that take a bunch of arguments like the dependencies and user configurable things like CUDA support. And so this is the Nix expression for blender, which you can enable CUDA support. So if you build this thing, yeah, so if you want to have blender with CUDA support, there's actually no way to find out that this option exists except by reading the source of this Nix expression. So it's not discoverable at all. And once you have discovered that this option exists, maybe by Googling about it, how do I enable CUDA in blender, you can't actually use it from the comment line. So you might think that something like Nix builds, blender, arc, CUDA support through does the right thing, but it doesn't because that argument doesn't get propagated all the way down. Other problems, there's no type checking. It's kind of ugly that it mixes user-facing configuration with dependencies. So this is not a very great configuration mechanism. Now we have dot override. I won't go into details, but dot override is another of these things that grew kind of organically. It has all sorts of problems. It's inefficient because you need to do under the hood, it calls a function twice just to get this dot override function. And it leaks memory. So if you're wondering why Nix evaluation takes multiple gigabytes, well, this is one of the reasons. Yeah, then there is the Nix packages configuration file. So you can put things like this inside a config file. But again, it's not discoverable. There's no type checking. It's not obvious how you use it from the comment line. So yeah. Now, then finally, there's overlays. Overlays are great, but the syntax is kind of mysterious. And it's also not obvious how things like nested overwrites work. So if you want to override something at top level, it's easy. If you want to override something deep down like in Python packages, it's not obvious how it works. And then there are NixOS modules. And NixOS modules are great. They're awesome. So they basically tick off most of the boxes. They're discoverable. You can just type manconfiguration.nix. They have documentation. They have types. I mean, there are some problems. They're entirely a library feature. So Nix, the comment line tool, doesn't know anything about them. So you can't easily query NixOS options from the Nix comment line or set options or do things like that. So yeah. So OK. So let's, as a solution, since NixOS modules are so great, maybe we can just use NixOS modules everywhere. So turn NixOS modules into a language feature, maybe make some improvements, like clean up the syntax and semantics and then use it everywhere. So instead of, for instance, having functions that create packages, we can have modules that create packages. So here's an example of what that looks like. So this is a module that builds Hello World. So if you know NixOS modules, then the syntax is probably familiar. Probably we would actually want to improve this syntax if we really turn this into a language feature because then you're no longer confined by having to have these weird constructs like having config as a function argument and as an attribute name. But apart from that, it should look familiar. So what this module does is it extends a bunch of other modules like standard nf. So in this modular world, we no longer have a function called standard nf.make derivation. That standard nf is a module that you can extend or inherit in your own modules. And that should, in theory, make it easier to combine modules. So for instance, if you have a package that both includes Rust code and Python code, you could just include the module for building Rust and the module for building Python. So what this module does is it says Hello World by default, but the text of the greeting is configurable at compile time. So it has a who option that you can set at build time. And that option is used in the configuration of this module. So the configuration sets a bunch of options that it inherited from the standard nf module and the package module, like the package name, the version, the build script. And that build script then generates a Hello World binary. All right. So how would you use this? Well, basically the same as how you use Nix now. And so by the way, I should mention that this is intended to be part of a flake. So modules.hello would be a flake output. So just as a flake can have packages now, it can have modules. So if you say nixrun.hello, that will build the Hello module from the flake in the current directory and run it. And that's actually equivalent to building and running modules.hello.final.derivation. And derivation is an option inherited from one of these modules that we're extending, like standard nf, that evaluates to a derivation graph. So just like in NixOS, you have a option called system.build.topLevel, which builds your entire system. Here we have an output option called derivation, which returns the derivation for your package. So the whole point was to make things discoverable. And now we have that because we have a standard structure and we have options that have descriptions and types. Actually, we don't have types yet, but we could have them just listen to NixOS module system. So yeah, we can discover via command nix list options, and that will show us all the options in this package. So for instance, this is how we discovered that this thing has a who option. And then we can override this from the comment line. So we can, for instance, say, Nix run hello, and we can override who to Nixcon. And that will build a new derivation and run it. Now you might be thinking aren't flakes supposed to be hermetic? So you can't actually override something inside a flake. So what this actually does is it on the fly, it constructs a new flake in a temporary directory that imports the flake in the current directory. So basically it generates something that looks like this. And this is what you would write yourself if you're trying to extend the module. So you could have a flake with a module named my hello that extends the hello module and sets Nixcon to who. So this is basically the replacement for the dot override or dot override derivation mechanisms. Another nice thing to mention is that we now have a documentation mechanism, a standardized documentation mechanism. So there is a command called Nix doc that given a flake generates the documentation for that flake just like cargo doc. It actually uses MD book, which is also what cargo uses. So it takes all these markdown doc strings of all the options and modules and generates documentation out of them. So the goal here is to have a standard uniform documentation mechanism for all flakes. So in the future you should be able to say Nix doc on any flake and get documentation out of them. All right, final thing to mention is the beginner friendliness. So if I go back of two slides, so this Nix expression, there's a lot of mysterious code in here, like this whole self hello stuff, this config thing over here. So that might be daunting for new users. And in fact, it's kind of unnecessary because with modules, modules are really just name value pairs like who, Nix, gone. So you don't need the whole Nix expression language just to customize a module. So the idea is you can just write a tunnel file, you still have to declare your flake dependencies, but then you can say who is Nix gone. And of course we're extending a certain module and that's it. So yeah, you could have a more user friendly syntax. And another advantage of a simpler syntax like tunnel is that this is easier to generate so we can have a command line mechanism to modify a flake. So we can just provide a command to set options or add dependencies to a Nix shell or things like that. And that's less error prone than telling new users that they need to edit a Nix expression and get the syntax exactly right. So that's the idea. All right, so that's it actually. So the next steps that we're interested in doing is, well, actually try to improve the syntax and semantics of this toy module system because right now it's an extremely bare copy of the module system that NixOS has and it has no types or merge functions or stuff like that so it's extremely bare. But yeah, so we'd like to come up with a better syntax and semantics and improve on the NixOS module system. For instance, it would be great to actually have sort of proper scoping. So for instance, a module should not be able to access options that it hasn't explicitly inherited from another module. And of course, we'd need to do a lot of experiments to see whether any new mechanism actually meets all the current use cases for the currently existing configuration mechanisms. I mean, so stuff like overlays or package sets. Yeah, we need to see whether we can actually do that. Okay, so yeah, there is a proof of concept implementation, but so it contains all these commands like NixDoc and NixListOptions. So you can actually play with it, but it's very bare at this point. And there are actually some examples that you can check out. Here is a document somewhere that has more ideas about these language changes. All right, that's it for this talk. I understand I can now answer questions, so I need to figure out where I can actually see the questions. Hi, so I will just read those off to you. You are running early, so your Q&A will actually have two extra minutes if you want. Okay, let me read the first one we have in the pad. So this is from Andy on FreeNode. It says, what is the cost of having more features that other language already built into Nix? Sorry, can you repeat that? I didn't quite catch it. Okay, I can repeat that. What is the cost of having more features that other languages already built into Nix? What is the cost? Well, obviously there is a cost in terms of we have to implement it. It makes the language more complex because of course the existing language features don't go away. But I mean, there's also the cost of not having this. So if we don't improve things like discoverability, we might lose a lot of potential users. So I would more see it from that perspective. All of this is motivated by thinking in terms of use cases. So if a user wants to set up a Nix shell, what do they need to write? And currently, if they for instance want to write a flake for their project, they have to write a lot of boilerplate codes just to get that to work. So they need to think about what would an ideal Nix expression look like. Or maybe you don't want to write an Nix expression at all and you want some other syntax for that. So yeah, I guess that's not a very good answer, but I don't actually know how to quantify the cost very well. Right. I do like that you're considering what an optimal Nix expression would look like. So the next question we have is from David AK and it is about why was TOML chosen? Do you think you can answer that? Yeah, so I mean, I'm not very married to Toml, but it exists. It's well known. So yeah, a lot of tools have support for it, editors have support for it. And I mean, you could go for Jason, that would be the simplest choice, but Jason is maybe a little bit too primitive because you can't even have things like comments in it, at least not in a nice way. But yeah, so Jason, Toml, Yammel, those would be possibilities. And then Yammel, I mean, Yammel is maybe a bit too complex. But yeah, so Toml is kind of, it's not too bare bones, so it seems like a fairly decent choice. Okay. Yeah, I think it's a fairly decent choice as well, actually. I think of those three options you mentioned, you picked the test one, but that's just my opinion. So another question from Andy, and I hope this is an okay question to ask, you can tell me if it isn't, but will I also need a Nix integration test at some point? I wonder what's wrong with just Nix build the attribute flag and then doc. Ah, okay. So that actually works. Sorry, does it work? No. So ideally at some point it would just work. So you would just have a doc option that generates the documentation for your module. But well, no, actually, that would be quite hard to do. No. Yeah, so the Nix doc command internally actually, it actually just builds a Nix expression that takes the flag as an argument. So there's actually not a lot of mysterious going on there. So you could actually do it with a Nix build and probably with an argument. But I mean, that's not very user friendly. I mean, you have to, again, we want to get avoid magic in incantations. So just saying Nix build or Nix doc is a lot easier than figuring out the correct Nix builds attribute name and so on. I hope that answers the question. Yep, I think that does. Let me see if there's any more questions. I actually do not think there's any more questions and I believe that ends the Q&A portion. So yeah, we are on time for that. So I guess this concludes your talk today. So thank you so much. I also, I would like to mention something just briefly before we sign off. I really do like your avatar on GitHub as Dexter. Is that some sort of inside sort of thing? Like, because I do notice that Dexter is sort of like a precocious little genius that has like a laboratory in his room and he's always working on experiments. And I've noticed for the past Nix con that you've also just like talked about your experiments. So you're both sort of like scientists in your own right. Yeah, I mean, it's not really any deep thought that went into that. But I was a Dexter fan in the 90s. And we both wear glasses. So I guess that's close enough. Okay, I guess that signs off for today. Thank you so much. Everyone in the chat, please put clapping emojis or you know, loud clapping sounds and you know, we're live and direct. Can you also like out now, we talk about binaries, put ones in the chat like go crazy. Come on. I want to see the chat go crazy.
is about Nix's configuration language, but in its very powerful but suffers from a lack of discoverability, usability and consistency. In this talk, Ilkel is going to describe an experimental Nix module system that provides a consistent discoverable mechanism to write configurations such as packages and Nix with systems and show how this enables the bear user experience circles new and advanced users. And I very steep learning curve. So you have to learn over the hurdle, you're an advanced user. There they're often not very easy to use. So what can provide a simpler configuration language like have function arguments, the override, config, use this function argument style of CUDA support, there's actually no way to find out source of this Nix expression. So it's not comment line tool, doesn't know anything about from the Nix comment line or set options or do argument and as an attribute name. But apart from says Hello World by default, but the text of the module and the package module, like the package discover via command nix list options, and that derivation mechanisms. Another nice thing to mechanism. So there is a command called Nix doc for that flake just like cargo doc. It actually the options and modules and generates have a standard uniform documentation mechanism to customize a module. So the idea is you can syntax and semantics of this toy module system the module system that NixOS has and it has no come up with a better syntax and semantics and improve on the NixOS module system. For instance, documentation for your module. But well, no,