Nix roadmap
Dolstra, Eelco
Our BDFL will be talking about the Nix roadmap - Bio: Eelco is a senior software engineer at Tweag I/O. He obtained a PhD in Computer Science from Utrecht University in 2006 and was a postdoctoral researcher at Utrecht University and Delft University. As part of his PhD research project at Utrecht University, he developed Nix, the purely functional package manager, which forms the basis of the NixOS Linux distribution. He previously worked at LogicBlox and Infor.
 Okay, just a second. All right. So first up is the man who doesn't need an introduction but gets one anyway. So first up we've got Elko and for those who maybe don't know yet because I did see a couple of fans of people attending Nixcon for the first time. Elko is the person that we have to thank for the initiation of Nix basically because he worked on that during his PhD research and today is however not the time to look back but today Elko is going to tell us about what lies ahead for Nix, the roadmap of the future of Nix basically, right? So please give him a hand. Elko, thank you. So I guess, does this work? Can you hear me? Okay, great. Okay, so first of all thanks to the organizers and apart from that let me say that talk about external monitors being hard to configure on a Nix so as it's fake news I just plugged it in and it worked, it's magic. Yeah, so this talk was originally called a Nix roadmap but there is no Nix roadmap, it's towards a Nix roadmap because this should really be a community effort so this is sort of a starting point towards a roadmap. So people have been saying for years that we should have a roadmap. So lately I've been doing a lot of Rust programming, I've been really drinking the Rust Kool-Aid so whenever I have a problem now I ask myself what would Rust do? So it turns out Rust, they have an RFC for everything, they have a beautiful process for everything. So yeah, so why should you have a roadmap to begin with? Well, so they answer that so the main thing is that it allows the world to see what are sort of the long-term plans and the strategic priorities and it allows all the developers to hopefully get behind that and so everybody is hopefully... So hopefully then everybody is kind of pointing in the same direction and another thing they mentioned is that so they have a rapid release cycle but it turns out that that kind of meant that sort of bigger features as sort of long-term projects were sort of falling by the wayside because they don't really fit into that rapid release cycle. So establishing annual goals, so in 2018 we're going to make the compiler fast, that's a way to really get people behind that and ensure that they spend time on that so that at the end of the year they don't have to say we fail to reach our goals. So the process that they have and I'm not necessarily saying that we should follow that but so they have an annual roadmap and so they have a process for creating that roadmap so they write an RFC where they gather problems that the community has and then from that they extract a bunch of goals so that sounds like a reasonable thing to do and then they have a whole plan for the year so in say February they start planning how to reach those goals and then they start implementing them so I'm not saying that that necessarily makes a lot of sense for us but at least having a roadmap for say 2019 saying these are the goals that we want for NICs, that sounds like a very valuable thing to have. So in the rest of the talk I'm just going to do a brain dump of some things that I think are problems with NICs and from that it follows that there are some goals that we should implement but this is just sort of my ideas and so I would like to do kind of this RFC process and get everybody's input and from that hopefully we can get a roadmap for 2019. Okay, so some problem statements so these are just some things that are currently problematic with NICs so for example or things that I would like to do but I can't at the moment so one is I would like to use NICs as a make replacement or a basal replacement so NICs is all these nice features, a purely functional language, reproducible builds, isolation and we do this for sort of large things like packages and for very small things like configuration files in NICsOS so it seems like it should be a perfect build tool as well so for building things like C source files or whatever other language you want to build but there are a bunch of reasons that currently you can't really do that at the moment so I'll come back to that so that's one problem. Another NICs package options are not easily discoverable or configurable so right now NICs packages have all sorts of options so for example a package function might have an enable foo argument but this is completely undiscoverable except by reading the NICs packages source code and it's also not configurable via NICs N for any other tool so this is not very good UX. Another problem is that this is an increasingly big problem is that NICs packages and NICsOS evaluation is slow and it's getting slower all the time because the sort of the level of abstractions that are used in NICs packages are increasing and it uses a lot of RAM so and actually it turns out that this problem is kind of related to the previous problem. Another one that comes up a lot is that NICs currently has no way to handle secrets so things like passwords or keys you don't want to store those in the NICs store because then they're world readable which is bad so you need some way to deal with them or so another problem as an unprivileged so right now if you want to pull something from a binary cache from an arbitrary binary cache so you cannot do that as an unprivileged user. Things need to be signed by a key configured by the administrator so if you just want to pull something from some arbitrary cache you cannot do that as an arbitrary user. Another one so closure bloat so this is a fairly big issue so in NICs it's very easy to end up with a package that has way more runtime dependencies than it actually needs so I'll also give an example of that later. So yeah these are just some random problems so probably many of you have other problems so we're interested to hear them but so here are some goals, I skipped something, yeah so here are some goals that you can extract from those problem statements so these are more at a technical level so by the way if you look at the Rust goals there are also a lot of non-technical goals there like improve documentation, evangelize in certain communities, make the community more diverse so that's all great as well but here I'm more focusing on technical stuff but we should definitely not restrict ourselves to that in the roadmap. So make NICs a compelling build tool so a compelling replacement for NICs or Bazel, for make or Bazel or something that can actually complement those tools. Make the NICs store content addressable so that's a very technical goal but it's kind of related to all the others. Make NICs packages discoverable and configurable, improve the evaluation efficiency, provide mechanisms to prevent closure bloats, provide a way to store secrets in the NIC store so yeah these are just some goals that I would like to work on in the next year and to some extent I have been working on them. So the rest of this talk is just some random brain dump on how these goals could be achieved. So yeah, so first the goal NICs as a build tool so what do we need to get to that? So in a way you can already do this, in fact you could do that ten years ago, in fact there is a NICs make repository somewhere which has a bunch of functions for building C or C++ projects and that works fine but the problem is so now you have your project and you're using these NICs make functions so you can run NICs build to have incremental builds for your project and that's all great. So now you want to package this thing and put it in NICs packages so you make a tar ball containing your source code and your NICs expression and now you want to write a NICs expression in NICs packages that extracts this tar ball and builds it and there you run into the problem that you need to be able to call NICs from inside a NICs build because you're using a NICs expression to build your project so instead of a make file, so previously you would call make right but now you have a NICs expression that builds your project so you need to be able to call NICs build but you're inside a NICs build already and so a NICs derivation doesn't actually have arbitrary write access to the NICs store, in fact it only has write access to its outputs so this doesn't work. So now you have a very embarrassing situation so you have a package that's written that has a build system written in NICs but you can't actually put it in NICs packages. You could put it in Debian probably but you can't put it in NICs packages so this is not good. So you need recursive NICs so that's kind of a required feature, another not essential but very nice to have feature is content addressability which I'll come back to and caching of copying files to the store so a tool like that so if you have your project which might consist of thousands of source files so now every time you run NICs build it has to read all those source files and copy them to the NICs store or at least check whether they already are in the NICs store so that's a lot of IO and that's slow so a tool like make, prevent, avoids that by only checking time stamps and even that can get slow for very large projects but yeah NICs needs to hash all these files so it actually needs to read all of them so you want to have some kind of caching for that and maybe something like an I notify demon to efficiently notice when files change but this is in the nice to have category. So the content addressability this is kind of been a holy grail for many years so this is the property that I should step back for a second so if you remember so a NICs store path contains a cryptographic hash but that cryptographic hash is a hash of the derivation that built that path it's not actually a hash of the content of that path and this is why you need signatures on binary caches because you need to trust that some store path was actually produced by the derivation that it claims to be built by and that could be a lie so somebody could set up a binary cache where so you get a legitimate NICs expression so that's for example built Firefox and then you pull a binary from the cache that actually contains something completely different like a trojan version of Firefox so that's why you need signatures. So in a content addressable store the store path so the hash in the store path is actually a hash of the contents of that path so you no longer need to trust anything so you can just verify that for example in a path like this NIC stores hash you just check that the cryptographic hash of the contents of this path is this so a path basically contains its own proof of integrity so if you have this then yeah unprivileged users can install things from arbitrary binary caches. Another very big advantage is that you get deduplication so for example if you say you make an irrelevant change to something in the dependency graph like you make a wide space change to G-Lib C so currently that causes the entire system to be rebuilt which is bad and actually yeah not just rebuilt but duplicated in the NIC store so you need twice the storage space now whereas with a content addressable store so because this change is irrelevant it doesn't actually change anything to the output of a build it ends up being stored in the same location so that's much nicer and in fact it does prevent rebuilds because if for instance so you make that change to G-Lib C you still need to recompile G-Lib C to discover that that change doesn't matter but after that you don't need to rebuild anything that depends on it because you've already discovered that yeah this G-Lib C is actually the same so it sort of acts as a barrier in the dependency graph. Yeah so this is why content addressability would be a great feature to have so a few interesting things about content addressability to make this work properly it really needs perfect binary reproducibility so that's currently not the case with NIC packages so if you build a package twice you might actually end up with slightly different results so for example if a binary stores a timestamp somewhere but yeah a lot of people are at work to improve that so for example there's a whole reproducible builds.org project that's basically improving all sorts of packages and build systems to eliminate sources of binary impurity. Okay so for the other thing so yeah prevent oh sorry prevent enclosure bloat so this is something of an obsession of mine sorry about that so in NICs because of the way it detects it finds dependencies it's very easy to have an accidental dependency so this is not the case in say had Debian where you specify dependencies so you don't end up with an accidental runtime dependency on say the C compiler so here for example there was a situation where Thunderbird was storing its build configuration so you can do about config in the URL bar and it will show you the path to the C compiler used to compile it which is of course a kind of a useless thing but it does yeah adds 1200 megabytes of blow to the closure and yeah so this can happen very accidentally you don't get any errors if you do that so yeah we need better tools to detect when this happens so we already have some attributes that you can specify in NICs expressions for example you can say disallowed requisites to say that something should not have a runtime reference to say the C compiler but this is very limited for example you cannot do any pattern matching you would like to say this thing should not have any references to developer outputs and it should be per output because for example your dev output probably should be allowed to have references to other dev outputs and you might want to have size checks so for example if say the NICS or ISO suddenly gets a gigabyte bigger than would like to get some error so what I recently implemented partially so not all of this works yet is that you can specify per output check so for example you can say the output the output should not have a closure bigger than 256 megabytes it should not be should not reference the C compiler or any dev output but the dev output itself can reference anything but it should not be larger than 128 kilobytes as a random example so yeah so we can start putting these things in NICS packages it could even be a generic thing so for example the rule that things should not allow should not be allowed to reference dev outputs is something you could actually put in standard NF so as a general policy so yeah that would be very nice how am I doing on time actually I can just check okay great so yeah now I come to the really wild and vaporware part of the talk so so yeah really a big issue is discoverability and efficiency like I mentioned so NICS packages have basically no discoverability well I mean you can discover that they exist sometimes I mean NICS NF doesn't necessarily recurs into everything but so you can see the packages exist we can see what options they have and customizing packages is also very ad hoc it's sort of evolved it's not really a properly designed thing so if these things like dot override dot override derivation config and in fact this this whole dot override thing is kind of disastrous for performance so it's really one of them it's one of the two main reasons why NICS evaluation takes a lot of memory so so dot override basically destroys the ability of the garbage collector at runtime to actually collect any garbage because you call a function so you pass it some arguments which can be very big because there are arbitrary dependencies so large graph and and then the output just contains the inputs so the inputs can never be garbage collected so and this is this was kind of a bad idea but we don't really have anything better so it's kind of a meta issue here of all these things so somewhere along the way we forgot that NICS is intended as a domain specific language for specifying build graphs and configurations like like NICS systems but as a DSL it's not really doing a great job so for example it has no concept of a package or an option or a configuration or things like plugins so any sort of features you or concepts you might expect in a DSL intended for doing these things so so maybe we need to get back to well we need to improve NICS as a DSL so so one thing that I've been thinking about is so essentially turning the NICS as module system or an improved version of it into a language feature into something called a configuration which can really think of as an attribute set an extensible attribute set which is really what a NICS configuration is it's a it's a bunch of attributes that you can change and so if you change one attribute it can trigger other attributes to change yeah so so a configuration is attribute set which contains attributes called options that can be set and they can be overwritten later but unlike attribute set and like NICS or as options they can have types and documentations and merge functions and that's the thing that gives you discoverability so things like package options can be expressed in this way and because they have things like a description and a type tools could can discover them and present them to the user and allow them to be changed program and programmatically so so the sketchy design for this language feature is a bit like this so a configuration looks a lot like an attribute set only it uses angled brackets might change don't get too angry or or enthusiastic about the syntax but yeah so it's you could think of it as an attribute set so if an attribute foo an attribute bar and an attribute ABC that actually refers to foo and bar so it's a recursive it's like a rec attribute set so if from this thing you select the ABC attribute you would get a value one two three because bar is true so if if true then one two three right but what you can now do is take that configuration module and apply a new module to it that sets bar to false and so now if from this module you select ABC then it will return a false then it goes to one two three times two so it will return two four six so this is pretty much exactly the behavior of the the Nixar's module system yeah and then the idea is that you can have some sort of syntax which I'm not sure about but to to attach fields or annotations to those to those options like documentation type default value and so on merge functions priorities yeah all that sort of thing basically all the things you have in the Nixar's module system right but now the idea is that we can apply this to building packages so rather than having packages as functions which have the problem that well there's no override mechanism I mean no good override mechanism there's no documentation and so on we can basically treat packages in the same way as the Nixar's module system treats system configurations so you you build a package in sort of a modular way by combining a bunch of modules so for example you could have a very sort of at the bottom you have a module that captures the concept of a derivation so what is a derivation well a derivation Nixar's a name and a version and it has a builder and it has arguments and it has an environment and if you set those things then you can evaluate a derivation attribute which produces a low level derivation so so this is very low level but now you can build higher level modules on top of that so for example this thing basically expresses the basic standard environment so the concept of phases and dependencies on other packages so this thing adds a an option called build inputs and an option called phases and it it implements this on top of the lower level derivation module by setting builder and arcs and and that causes a derivation to be computed yeah that uses these things so just to continue this a bit so you could have a module that captures the concept of a package so package has a description and a home page and so on and all these things have had descriptions and they have types so they're they're discoverable and and you get error messages so for example if if you use this previous so currently with derivations if for example you you misspell build inputs you're not going to get an error because nix derivation is basically just a bunch of environments variable bindings so yeah so there's no checking whatsoever there but here if you set an option that hasn't been declared then you get an error message just like in the in the module system yeah so you can build higher and higher level abstractions on this so for example you can extend the sort of the generic standard environment with the concept of a unix package which for example has a configure phase and because it runs a configure script yeah and then finally you can define a package so a package is something that extends the unix package module with something that sets a name description source but it also has its own option namely enable GUI so in this fictional example hello world has a GTK support so now you can say build inputs is if enable GUI then use GTK and this thing is now discoverable so you could have say a nix query package command which will show that this thing has an enable GUI option and they could have a nix install command that that allows you to set that option so yeah and you could override things using the xx same module system so that's just like in xos so that's about it so there are lots of other things you could imagine for the roadmap I'll skip that so yeah so what I should do is create a sort of a roadmap issue and where everybody can go wild with ideas and suggestions and then we should try to synthesize something workable from that so yeah that's it thank you. So for the configurations idea of using doing overrides how does that solve the memory issue don't you still need to hold on to the references to all the inputs? No because as soon as you evaluate the.drv attribute you don't need anything else anymore you can after that you can just discard everything that went into it so it's sort of like in nixOS if you evaluate system.build.top level you get a derivation out of that and at that point you can garbage collect all the inputs to that thing. So like when you're passing something as an input to something else you're passing the.drv not the composable thing? So presumably this is all fiction so here this GTK thing would trigger an evaluation of GTK.drv implicitly. On the slide about the nixos build tool you have been talking about avoiding rebuilds and similar properties have you looked into the recent paper build systems a la carte by Neil Mitchell, Simon Pinat-Jones and Andrey Markov they're analyzing various build systems there and nix is one of them and nix ticks off almost all features of the ultimate build system. Was that the I-Shift-P paper? Yes. Okay yeah it's on my to-do list. Okay, please take a look at this it gives very good names, good glossary to talk about these properties it's a great paper thank you. So the configuration options it excites the type system in me. I won't go there. Is there also so the nixos module system also supports the notion of overlays is this should is there a version of this also in this or is this then kind of extension? Yeah probably so that's sort of a high level thing that I haven't figured out yet so how you actually put these things together so I but you need some way to do that and yeah so I don't know yet. Okay yeah I think we've got to move maybe one more. Last one. It's more of a policy question I've seen a lot of commits like fixing Thunderbird like you showed and there has been commits for every separate packages. Why don't you enforce this kind of like do not reference GCC for all the package by default and then if some package really needs GCC then you could enable it. Right well that's kind of the point so we don't have a way really to enforce that yet so right now it's really only if people sort of notice that suddenly a closure has become much bigger so you can use these disallowed requisites attributes but very few packages do that. But you could use that by default like add that to MKE derivation and then by default just do not reference GCC there is no reason for most packages to do so. So probably for GCC that would work but you really want to say things like it should not reference any death output and that doesn't work at the moment because death output should be allowed to reference death output so you can't use the existing attributes for that. But yeah so I would definitely like to have in NixOS for example that we say that all the NixOS VM tests could just check that their closures don't have any death outputs in them and no GCC or clang so that would already help quite a bit. Okay all righty so thank you again for your talk.
 Okay, just a second.  All right.  So first up is the man who doesn't need an introduction but gets one anyway.  So first up we've got Elko and for those who maybe don't know yet because I did see a couple  of fans of people attending Nixcon for the first time.  Elko is the person that we have to thank for the initiation of Nix basically because he  worked on that during his PhD research and today is however not the time to look back  but today Elko is going to tell us about what lies ahead for Nix, the roadmap of the future  of Nix basically, right?  So please give him a hand.  Elko, thank you.  So I guess, does this work?  Can you hear me?  Okay, great.  Okay, so first of all thanks to the organizers and apart from that let me say that talk about  external monitors being hard to configure on a Nix so as it's fake news I just plugged  it in and it worked, it's magic.  Yeah, so this talk was originally called a Nix roadmap but there is no Nix roadmap,  it's towards a Nix roadmap because this should really be a community effort so this is sort  of a starting point towards a roadmap.  So people have been saying for years that we should have a roadmap.  So lately I've been doing a lot of Rust programming, I've been really drinking the Rust Kool-Aid  so whenever I have a problem now I ask myself what would Rust do?  So it turns out Rust, they have an RFC for everything, they have a beautiful process  for everything.  So yeah, so why should you have a roadmap to begin with?  Well, so they answer that so the main thing is that it allows the world to see what are  sort of the long-term plans and the strategic priorities and it allows all the developers  to hopefully get behind that and so everybody is hopefully...  So hopefully then everybody is kind of pointing in the same direction and another thing they  mentioned is that so they have a rapid release cycle but it turns out that that kind of meant  that sort of bigger features as sort of long-term projects were sort of falling by the wayside  because they don't really fit into that rapid release cycle.  So establishing annual goals, so in 2018 we're going to make the compiler fast, that's a  way to really get people behind that and ensure that they spend time on that so that at the  end of the year they don't have to say we fail to reach our goals.  So the process that they have and I'm not necessarily saying that we should follow that  but so they have an annual roadmap and so they have a process for creating that roadmap  so they write an RFC where they gather problems that the community has and then from that  they extract a bunch of goals so that sounds like a reasonable thing to do and then they  have a whole plan for the year so in say February they start planning how to reach those goals  and then they start implementing them so I'm not saying that that necessarily makes a lot  of sense for us but at least having a roadmap for say 2019 saying these are the goals that  we want for NICs, that sounds like a very valuable thing to have.  So in the rest of the talk I'm just going to do a brain dump of some things that I think  are problems with NICs and from that it follows that there are some goals that we should implement  but this is just sort of my ideas and so I would like to do kind of this RFC process  and get everybody's input and from that hopefully we can get a roadmap for 2019.  Okay, so some problem statements so these are just some things that are currently problematic  with NICs so for example or things that I would like to do but I can't at the moment  so one is I would like to use NICs as a make replacement or a basal replacement so NICs  is all these nice features, a purely functional language, reproducible builds, isolation and  we do this for sort of large things like packages and for very small things like configuration  files in NICsOS so it seems like it should be a perfect build tool as well so for building  things like C source files or whatever other language you want to build but there are a  bunch of reasons that currently you can't really do that at the moment so I'll come back to  that so that's one problem.  Another NICs package options are not easily discoverable or configurable so right now  NICs packages have all sorts of options so for example a package function might have  an enable foo argument but this is completely undiscoverable except by reading the NICs  packages source code and it's also not configurable via NICs N for any other tool so this is not  very good UX.  Another problem is that this is an increasingly big problem is that NICs packages and NICsOS  evaluation is slow and it's getting slower all the time because the sort of the level  of abstractions that are used in NICs packages are increasing and it uses a lot of RAM so  and actually it turns out that this problem is kind of related to the previous problem.  Another one that comes up a lot is that NICs currently has no way to handle secrets so  things like passwords or keys you don't want to store those in the NICs store because then  they're world readable which is bad so you need some way to deal with them or so another  problem as an unprivileged so right now if you want to pull something from a binary cache  from an arbitrary binary cache so you cannot do that as an unprivileged user.  Things need to be signed by a key configured by the administrator so if you just want to  pull something from some arbitrary cache you cannot do that as an arbitrary user.  Another one so closure bloat so this is a fairly big issue so in NICs it's very easy  to end up with a package that has way more runtime dependencies than it actually needs  so I'll also give an example of that later.  So yeah these are just some random problems so probably many of you have other problems  so we're interested to hear them but so here are some goals, I skipped something, yeah  so here are some goals that you can extract from those problem statements so these are  more at a technical level so by the way if you look at the Rust goals there are also  a lot of non-technical goals there like improve documentation, evangelize in certain communities,  make the community more diverse so that's all great as well but here I'm more focusing  on technical stuff but we should definitely not restrict ourselves to that in the roadmap.  So make NICs a compelling build tool so a compelling replacement for NICs or Bazel,  for make or Bazel or something that can actually complement those tools.  Make the NICs store content addressable so that's a very technical goal but it's kind  of related to all the others.  Make NICs packages discoverable and configurable, improve the evaluation efficiency, provide  mechanisms to prevent closure bloats, provide a way to store secrets in the NIC store so  yeah these are just some goals that I would like to work on in the next year and to some  extent I have been working on them.  So the rest of this talk is just some random brain dump on how these goals could be achieved.  So yeah, so first the goal NICs as a build tool so what do we need to get to that?  So in a way you can already do this, in fact you could do that ten years ago, in fact there  is a NICs make repository somewhere which has a bunch of functions for building C or C++  projects and that works fine but the problem is so now you have your project and you're  using these NICs make functions so you can run NICs build to have incremental builds  for your project and that's all great.  So now you want to package this thing and put it in NICs packages so you make a tar  ball containing your source code and your NICs expression and now you want to write  a NICs expression in NICs packages that extracts this tar ball and builds it and there you  run into the problem that you need to be able to call NICs from inside a NICs build because  you're using a NICs expression to build your project so instead of a make file, so previously  you would call make right but now you have a NICs expression that builds your project  so you need to be able to call NICs build but you're inside a NICs build already and  so a NICs derivation doesn't actually have arbitrary write access to the NICs store,  in fact it only has write access to its outputs so this doesn't work.  So now you have a very embarrassing situation so you have a package that's written that  has a build system written in NICs but you can't actually put it in NICs packages.  You could put it in Debian probably but you can't put it in NICs packages so this is not  good.  So you need recursive NICs so that's kind of a required feature, another not essential  but very nice to have feature is content addressability which I'll come back to and caching of copying  files to the store so a tool like that so if you have your project which might consist  of thousands of source files so now every time you run NICs build it has to read all  those source files and copy them to the NICs store or at least check whether they already  are in the NICs store so that's a lot of IO and that's slow so a tool like make, prevent,  avoids that by only checking time stamps and even that can get slow for very large projects  but yeah NICs needs to hash all these files so it actually needs to read all of them so  you want to have some kind of caching for that and maybe something like an I notify  demon to efficiently notice when files change but this is in the nice to have category.  So the content addressability this is kind of been a holy grail for many years so this  is the property that I should step back for a second so if you remember so a NICs store  path contains a cryptographic hash but that cryptographic hash is a hash of the derivation  that built that path it's not actually a hash of the content of that path and this is why  you need signatures on binary caches because you need to trust that some store path was  actually produced by the derivation that it claims to be built by and that could be a  lie so somebody could set up a binary cache where so you get a legitimate NICs expression  so that's for example built Firefox and then you pull a binary from the cache that actually  contains something completely different like a trojan version of Firefox so that's why  you need signatures.  So in a content addressable store the store path so the hash in the store path is actually  a hash of the contents of that path so you no longer need to trust anything so you can  just verify that for example in a path like this NIC stores hash you just check that the  cryptographic hash of the contents of this path is this so a path basically contains  its own proof of integrity so if you have this then yeah unprivileged users can install  things from arbitrary binary caches. Another very big advantage is that you get deduplication  so for example if you say you make an irrelevant change to something in the dependency graph  like you make a wide space change to G-Lib C so currently that causes the entire system  to be rebuilt which is bad and actually yeah not just rebuilt but duplicated in the NIC  store so you need twice the storage space now whereas with a content addressable store  so because this change is irrelevant it doesn't actually change anything to the output of a  build it ends up being stored in the same location so that's much nicer and in fact it does prevent  rebuilds because if for instance so you make that change to G-Lib C you still need to recompile  G-Lib C to discover that that change doesn't matter but after that you don't need to rebuild  anything that depends on it because you've already discovered that yeah this G-Lib C  is actually the same so it sort of acts as a barrier in the dependency graph. Yeah so  this is why content addressability would be a great feature to have so a few interesting  things about content addressability to make this work properly it really needs perfect  binary reproducibility so that's currently not the case with NIC packages so if you build  a package twice you might actually end up with slightly different results so for example  if a binary stores a timestamp somewhere but yeah a lot of people are at work to improve  that so for example there's a whole reproducible builds.org project that's basically improving  all sorts of packages and build systems to eliminate sources of binary impurity. Okay  so for the other thing so yeah prevent oh sorry prevent enclosure bloat so this is something  of an obsession of mine sorry about that so in NICs because of the way it detects it  finds dependencies it's very easy to have an accidental dependency so this is not the  case in say had Debian where you specify dependencies so you don't end up with an accidental runtime  dependency on say the C compiler so here for example there was a situation where Thunderbird  was storing its build configuration so you can do about config in the URL bar and it  will show you the path to the C compiler used to compile it which is of course a kind of  a useless thing but it does yeah adds 1200 megabytes of blow to the closure and yeah so  this can happen very accidentally you don't get any errors if you do that so yeah we need  better tools to detect when this happens so we already have some attributes that you can  specify in NICs expressions for example you can say disallowed requisites to say that something  should not have a runtime reference to say the C compiler but this is very limited for  example you cannot do any pattern matching you would like to say this thing should not  have any references to developer outputs and it should be per output because for example  your dev output probably should be allowed to have references to other dev outputs and  you might want to have size checks so for example if say the NICS or ISO suddenly gets a gigabyte  bigger than would like to get some error so what I recently implemented partially so not  all of this works yet is that you can specify per output check so for example you can say  the output the output should not have a closure bigger than 256 megabytes it should not be  should not reference the C compiler or any dev output but the dev output itself can reference  anything but it should not be larger than 128 kilobytes as a random example so yeah so we  can start putting these things in NICS packages it could even be a generic thing so for example  the rule that things should not allow should not be allowed to reference dev outputs is  something you could actually put in standard NF so as a general policy so yeah that would  be very nice how am I doing on time actually I can just check okay great so yeah now I  come to the really wild and vaporware part of the talk so so yeah really a big issue  is discoverability and efficiency like I mentioned so NICS packages have basically no discoverability  well I mean you can discover that they exist sometimes I mean NICS NF doesn't necessarily  recurs into everything but so you can see the packages exist we can see what options they  have and customizing packages is also very ad hoc it's sort of evolved it's not really  a properly designed thing so if these things like dot override dot override derivation config  and in fact this this whole dot override thing is kind of disastrous for performance so it's  really one of them it's one of the two main reasons why NICS evaluation takes a lot of  memory so so dot override basically destroys the ability of the garbage collector at runtime  to actually collect any garbage because you call a function so you pass it some arguments  which can be very big because there are arbitrary dependencies so large graph and and then the  output just contains the inputs so the inputs can never be garbage collected so and this  is this was kind of a bad idea but we don't really have anything better so it's kind of  a meta issue here of all these things so somewhere along the way we forgot that NICS is intended  as a domain specific language for specifying build graphs and configurations like like  NICS systems but as a DSL it's not really doing a great job so for example it has no  concept of a package or an option or a configuration or things like plugins so any sort of features  you or concepts you might expect in a DSL intended for doing these things so so maybe  we need to get back to well we need to improve NICS as a DSL so so one thing that I've been  thinking about is so essentially turning the NICS as module system or an improved version  of it into a language feature into something called a configuration which can really think  of as an attribute set an extensible attribute set which is really what a NICS configuration  is it's a it's a bunch of attributes that you can change and so if you change one attribute  it can trigger other attributes to change yeah so so a configuration is attribute set which  contains attributes called options that can be set and they can be overwritten later  but unlike attribute set and like NICS or as options they can have types and documentations  and merge functions and that's the thing that gives you discoverability so things like package  options can be expressed in this way and because they have things like a description and a  type tools could can discover them and present them to the user and allow them to be changed  program and programmatically so so the sketchy design for this language feature is a bit  like this so a configuration looks a lot like an attribute set only it uses angled brackets  might change don't get too angry or or enthusiastic about the syntax but yeah so it's you could  think of it as an attribute set so if an attribute foo an attribute bar and an attribute ABC  that actually refers to foo and bar so it's a recursive it's like a rec attribute set  so if from this thing you select the ABC attribute you would get a value one two three because  bar is true so if if true then one two three right but what you can now do is take that  configuration module and apply a new module to it that sets bar to false and so now if  from this module you select ABC then it will return a false then it goes to one two three  times two so it will return two four six so this is pretty much exactly the behavior  of the the Nixar's module system yeah and then the idea is that you can have some sort  of syntax which I'm not sure about but to to attach fields or annotations to those to  those options like documentation type default value and so on merge functions priorities  yeah all that sort of thing basically all the things you have in the Nixar's module  system right but now the idea is that we can apply this to building packages so rather  than having packages as functions which have the problem that well there's no override  mechanism I mean no good override mechanism there's no documentation and so on we can  basically treat packages in the same way as the Nixar's module system treats system configurations  so you you build a package in sort of a modular way by combining a bunch of modules so for  example you could have a very sort of at the bottom you have a module that captures the  concept of a derivation so what is a derivation well a derivation Nixar's a name and a version  and it has a builder and it has arguments and it has an environment and if you set those  things then you can evaluate a derivation attribute which produces a low level derivation  so so this is very low level but now you can build higher level modules on top of that  so for example this thing basically expresses the basic standard environment so the concept  of phases and dependencies on other packages so this thing adds a an option called build  inputs and an option called phases and it it implements this on top of the lower level  derivation module by setting builder and arcs and and that causes a derivation to be computed  yeah that uses these things so just to continue this a bit so you could have a module that  captures the concept of a package so package has a description and a home page and so on  and all these things have had descriptions and they have types so they're they're discoverable  and and you get error messages so for example if if you use this previous so currently with  derivations if for example you you misspell build inputs you're not going to get an error  because nix derivation is basically just a bunch of environments variable bindings so  yeah so there's no checking whatsoever there but here if you set an option that hasn't  been declared then you get an error message just like in the in the module system yeah  so you can build higher and higher level abstractions on this so for example you can extend the  sort of the generic standard environment with the concept of a unix package which for example  has a configure phase and because it runs a configure script yeah and then finally you  can define a package so a package is something that extends the unix package module with  something that sets a name description source but it also has its own option namely enable  GUI so in this fictional example hello world has a GTK support so now you can say build  inputs is if enable GUI then use GTK and this thing is now discoverable so you could have  say a nix query package command which will show that this thing has an enable GUI option  and they could have a nix install command that that allows you to set that option so yeah  and you could override things using the xx same module system so that's just like in  xos so that's about it so there are lots of other things you could imagine for the roadmap  I'll skip that so yeah so what I should do is create a sort of a roadmap issue and where  everybody can go wild with ideas and suggestions and then we should try to synthesize something  workable from that so yeah that's it thank you.  So for the configurations idea of using doing overrides how does that solve the memory issue  don't you still need to hold on to the references to all the inputs?  No because as soon as you evaluate the.drv attribute you don't need anything else anymore  you can after that you can just discard everything that went into it so it's sort of like in  nixOS if you evaluate system.build.top level you get a derivation out of that and at that  point you can garbage collect all the inputs to that thing.  So like when you're passing something as an input to something else you're passing the .drv not the composable thing?  So presumably this is all fiction so here this GTK thing would trigger an evaluation  of GTK.drv implicitly.  On the slide about the nixos build tool you have been talking about avoiding rebuilds  and similar properties have you looked into the recent paper build systems a la carte  by Neil Mitchell, Simon Pinat-Jones and Andrey Markov they're analyzing various build systems  there and nix is one of them and nix ticks off almost all features of the ultimate build  system.  Was that the I-Shift-P paper?  Yes.  Okay yeah it's on my to-do list.  Okay, please take a look at this it gives very good names, good glossary to talk about  these properties it's a great paper thank you.  So the configuration options it excites the type system in me.  I won't go there.  Is there also so the nixos module system also supports the notion of overlays is this should  is there a version of this also in this or is this then kind of extension?  Yeah probably so that's sort of a high level thing that I haven't figured out yet so how  you actually put these things together so I but you need some way to do that and yeah  so I don't know yet.  Okay yeah I think we've got to move maybe one more.  Last one.  It's more of a policy question I've seen a lot of commits like fixing Thunderbird like  you showed and there has been commits for every separate packages.  Why don't you enforce this kind of like do not reference GCC for all the package by default  and then if some package really needs GCC then you could enable it.  Right well that's kind of the point so we don't have a way really to enforce that yet  so right now it's really only if people sort of notice that suddenly a closure has become  much bigger so you can use these disallowed requisites attributes but very few packages  do that.  But you could use that by default like add that to MKE derivation and then by default  just do not reference GCC there is no reason for most packages to do so.  So probably for GCC that would work but you really want to say things like it should not  reference any death output and that doesn't work at the moment because death output should  be allowed to reference death output so you can't use the existing attributes for that.  But yeah so I would definitely like to have in NixOS for example that we say that all  the NixOS VM tests could just check that their closures don't have any death outputs  in them and no GCC or clang so that would already help quite a bit.  Okay all righty so thank you again for your talk.
Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_0_155.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_156_203.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_204_371.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_372_419.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_420_563.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_564_611.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_612_851.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_852_1007.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_1008_1163.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_1164_1271.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_1272_2663.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_2664_2771.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_2772_4091.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_4092_4967.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_4968_5363.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_5364_6539.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_6540_8015.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_8016_9407.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_9408_16451.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_16452_18239.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_18240_18479.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_18480_19883.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_19884_22367.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_22368_22811.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_22812_23507.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_23508_23651.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_23652_30515.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_30516_32723.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_32724_33275.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_33276_34379.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_34380_35099.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_35100_35399.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_35400_35447.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_35448_37367.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_37368_37595.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_37596_38531.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_38532_42251.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_42252_42695.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_42696_43475.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_43476_44795.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_44796_45299.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_45300_46319.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_46320_47231.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_47232_52991.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_52992_53591.png Eelco_Dolstra_-_the_Nix_roadmap_8M6yvJC00J4_53592_53777.png
Okay, just a second. All right. So first up is the man who doesn't need an introduction but gets one anyway. So first up we've got Elko and for those who maybe don't know yet because I did see a couple of fans of people attending Nixcon for the first time. Elko is the person that we have to thank for the initiation of Nix basically because he worked on that during his PhD research and today is however not the time to look back but today Elko is going to tell us about what lies ahead for Nix, the roadmap of the future of Nix basically, right? So please give him a hand. Elko, thank you. So I guess, does this work? Can you hear me? Okay, great. Okay, so first of all thanks to the organizers and apart from that let me say that talk about external monitors being hard to configure on a Nix so as it's fake news I just plugged it in and it worked, it's magic. Yeah, so this talk was originally called a Nix roadmap but there is no Nix roadmap, it's towards a Nix roadmap because this should really be a community effort so this is sort of a starting point towards a roadmap. So people have been saying for years that we should have a roadmap. So lately I've been doing a lot of Rust programming, I've been really drinking the Rust Kool-Aid so whenever I have a problem now I ask myself what would Rust do? So it turns out Rust, they have an RFC for everything, they have a beautiful process for everything. So yeah, so why should you have a roadmap to begin with? Well, so they answer that so the main thing is that it allows the world to see what are sort of the long-term plans and the strategic priorities and it allows all the developers to hopefully get behind that and so everybody is hopefully... So hopefully then everybody is kind of pointing in the same direction and another thing they mentioned is that so they have a rapid release cycle but it turns out that that kind of meant that sort of bigger features as sort of long-term projects were sort of falling by the wayside because they don't really fit into that rapid release cycle. So establishing annual goals, so in 2018 we're going to make the compiler fast, that's a way to really get people behind that and ensure that they spend time on that so that at the end of the year they don't have to say we fail to reach our goals. So the process that they have and I'm not necessarily saying that we should follow that but so they have an annual roadmap and so they have a process for creating that roadmap so they write an RFC where they gather problems that the community has and then from that they extract a bunch of goals so that sounds like a reasonable thing to do and then they have a whole plan for the year so in say February they start planning how to reach those goals and then they start implementing them so I'm not saying that that necessarily makes a lot of sense for us but at least having a roadmap for say 2019 saying these are the goals that we want for NICs, that sounds like a very valuable thing to have. So in the rest of the talk I'm just going to do a brain dump of some things that I think are problems with NICs and from that it follows that there are some goals that we should implement but this is just sort of my ideas and so I would like to do kind of this RFC process and get everybody's input and from that hopefully we can get a roadmap for 2019. Okay, so some problem statements so these are just some things that are currently problematic with NICs so for example or things that I would like to do but I can't at the moment so one is I would like to use NICs as a make replacement or a basal replacement so NICs is all these nice features, a purely functional language, reproducible builds, isolation and we do this for sort of large things like packages and for very small things like configuration files in NICsOS so it seems like it should be a perfect build tool as well so for building things like C source files or whatever other language you want to build but there are a bunch of reasons that currently you can't really do that at the moment so I'll come back to that so that's one problem. Another NICs package options are not easily discoverable or configurable so right now NICs packages have all sorts of options so for example a package function might have an enable foo argument but this is completely undiscoverable except by reading the NICs packages source code and it's also not configurable via NICs N for any other tool so this is not very good UX. Another problem is that this is an increasingly big problem is that NICs packages and NICsOS evaluation is slow and it's getting slower all the time because the sort of the level of abstractions that are used in NICs packages are increasing and it uses a lot of RAM so and actually it turns out that this problem is kind of related to the previous problem. Another one that comes up a lot is that NICs currently has no way to handle secrets so things like passwords or keys you don't want to store those in the NICs store because then they're world readable which is bad so you need some way to deal with them or so another problem as an unprivileged so right now if you want to pull something from a binary cache from an arbitrary binary cache so you cannot do that as an unprivileged user. Things need to be signed by a key configured by the administrator so if you just want to pull something from some arbitrary cache you cannot do that as an arbitrary user. Another one so closure bloat so this is a fairly big issue so in NICs it's very easy to end up with a package that has way more runtime dependencies than it actually needs so I'll also give an example of that later. So yeah these are just some random problems so probably many of you have other problems so we're interested to hear them but so here are some goals, I skipped something, yeah so here are some goals that you can extract from those problem statements so these are more at a technical level so by the way if you look at the Rust goals there are also a lot of non-technical goals there like improve documentation, evangelize in certain communities, make the community more diverse so that's all great as well but here I'm more focusing on technical stuff but we should definitely not restrict ourselves to that in the roadmap. So make NICs a compelling build tool so a compelling replacement for NICs or Bazel, for make or Bazel or something that can actually complement those tools. Make the NICs store content addressable so that's a very technical goal but it's kind of related to all the others. Make NICs packages discoverable and configurable, improve the evaluation efficiency, provide mechanisms to prevent closure bloats, provide a way to store secrets in the NIC store so yeah these are just some goals that I would like to work on in the next year and to some extent I have been working on them. So the rest of this talk is just some random brain dump on how these goals could be achieved. So yeah, so first the goal NICs as a build tool so what do we need to get to that? So in a way you can already do this, in fact you could do that ten years ago, in fact there is a NICs make repository somewhere which has a bunch of functions for building C or C++ projects and that works fine but the problem is so now you have your project and you're using these NICs make functions so you can run NICs build to have incremental builds for your project and that's all great. So now you want to package this thing and put it in NICs packages so you make a tar ball containing your source code and your NICs expression and now you want to write a NICs expression in NICs packages that extracts this tar ball and builds it and there you run into the problem that you need to be able to call NICs from inside a NICs build because you're using a NICs expression to build your project so instead of a make file, so previously you would call make right but now you have a NICs expression that builds your project so you need to be able to call NICs build but you're inside a NICs build already and so a NICs derivation doesn't actually have arbitrary write access to the NICs store, in fact it only has write access to its outputs so this doesn't work. So now you have a very embarrassing situation so you have a package that's written that has a build system written in NICs but you can't actually put it in NICs packages. You could put it in Debian probably but you can't put it in NICs packages so this is not good. So you need recursive NICs so that's kind of a required feature, another not essential but very nice to have feature is content addressability which I'll come back to and caching of copying files to the store so a tool like that so if you have your project which might consist of thousands of source files so now every time you run NICs build it has to read all those source files and copy them to the NICs store or at least check whether they already are in the NICs store so that's a lot of IO and that's slow so a tool like make, prevent, avoids that by only checking time stamps and even that can get slow for very large projects but yeah NICs needs to hash all these files so it actually needs to read all of them so you want to have some kind of caching for that and maybe something like an I notify demon to efficiently notice when files change but this is in the nice to have category. So the content addressability this is kind of been a holy grail for many years so this is the property that I should step back for a second so if you remember so a NICs store path contains a cryptographic hash but that cryptographic hash is a hash of the derivation that built that path it's not actually a hash of the content of that path and this is why you need signatures on binary caches because you need to trust that some store path was actually produced by the derivation that it claims to be built by and that could be a lie so somebody could set up a binary cache where so you get a legitimate NICs expression so that's for example built Firefox and then you pull a binary from the cache that actually contains something completely different like a trojan version of Firefox so that's why you need signatures. So in a content addressable store the store path so the hash in the store path is actually a hash of the contents of that path so you no longer need to trust anything so you can just verify that for example in a path like this NIC stores hash you just check that the cryptographic hash of the contents of this path is this so a path basically contains its own proof of integrity so if you have this then yeah unprivileged users can install things from arbitrary binary caches. Another very big advantage is that you get deduplication so for example if you say you make an irrelevant change to something in the dependency graph like you make a wide space change to G-Lib C so currently that causes the entire system to be rebuilt which is bad and actually yeah not just rebuilt but duplicated in the NIC store so you need twice the storage space now whereas with a content addressable store so because this change is irrelevant it doesn't actually change anything to the output of a build it ends up being stored in the same location so that's much nicer and in fact it does prevent rebuilds because if for instance so you make that change to G-Lib C you still need to recompile G-Lib C to discover that that change doesn't matter but after that you don't need to rebuild anything that depends on it because you've already discovered that yeah this G-Lib C is actually the same so it sort of acts as a barrier in the dependency graph. Yeah so this is why content addressability would be a great feature to have so a few interesting things about content addressability to make this work properly it really needs perfect binary reproducibility so that's currently not the case with NIC packages so if you build a package twice you might actually end up with slightly different results so for example if a binary stores a timestamp somewhere but yeah a lot of people are at work to improve that so for example there's a whole reproducible builds.org project that's basically improving all sorts of packages and build systems to eliminate sources of binary impurity. Okay so for the other thing so yeah prevent oh sorry prevent enclosure bloat so this is something of an obsession of mine sorry about that so in NICs because of the way it detects it finds dependencies it's very easy to have an accidental dependency so this is not the case in say had Debian where you specify dependencies so you don't end up with an accidental runtime dependency on say the C compiler so here for example there was a situation where Thunderbird was storing its build configuration so you can do about config in the URL bar and it will show you the path to the C compiler used to compile it which is of course a kind of a useless thing but it does yeah adds 1200 megabytes of blow to the closure and yeah so this can happen very accidentally you don't get any errors if you do that so yeah we need better tools to detect when this happens so we already have some attributes that you can specify in NICs expressions for example you can say disallowed requisites to say that something should not have a runtime reference to say the C compiler but this is very limited for example you cannot do any pattern matching you would like to say this thing should not have any references to developer outputs and it should be per output because for example your dev output probably should be allowed to have references to other dev outputs and you might want to have size checks so for example if say the NICS or ISO suddenly gets a gigabyte bigger than would like to get some error so what I recently implemented partially so not all of this works yet is that you can specify per output check so for example you can say the output the output should not have a closure bigger than 256 megabytes it should not be should not reference the C compiler or any dev output but the dev output itself can reference anything but it should not be larger than 128 kilobytes as a random example so yeah so we can start putting these things in NICS packages it could even be a generic thing so for example the rule that things should not allow should not be allowed to reference dev outputs is something you could actually put in standard NF so as a general policy so yeah that would be very nice how am I doing on time actually I can just check okay great so yeah now I come to the really wild and vaporware part of the talk so so yeah really a big issue is discoverability and efficiency like I mentioned so NICS packages have basically no discoverability well I mean you can discover that they exist sometimes I mean NICS NF doesn't necessarily recurs into everything but so you can see the packages exist we can see what options they have and customizing packages is also very ad hoc it's sort of evolved it's not really a properly designed thing so if these things like dot override dot override derivation config and in fact this this whole dot override thing is kind of disastrous for performance so it's really one of them it's one of the two main reasons why NICS evaluation takes a lot of memory so so dot override basically destroys the ability of the garbage collector at runtime to actually collect any garbage because you call a function so you pass it some arguments which can be very big because there are arbitrary dependencies so large graph and and then the output just contains the inputs so the inputs can never be garbage collected so and this is this was kind of a bad idea but we don't really have anything better so it's kind of a meta issue here of all these things so somewhere along the way we forgot that NICS is intended as a domain specific language for specifying build graphs and configurations like like NICS systems but as a DSL it's not really doing a great job so for example it has no concept of a package or an option or a configuration or things like plugins so any sort of features you or concepts you might expect in a DSL intended for doing these things so so maybe we need to get back to well we need to improve NICS as a DSL so so one thing that I've been thinking about is so essentially turning the NICS as module system or an improved version of it into a language feature into something called a configuration which can really think of as an attribute set an extensible attribute set which is really what a NICS configuration is it's a it's a bunch of attributes that you can change and so if you change one attribute it can trigger other attributes to change yeah so so a configuration is attribute set which contains attributes called options that can be set and they can be overwritten later but unlike attribute set and like NICS or as options they can have types and documentations and merge functions and that's the thing that gives you discoverability so things like package options can be expressed in this way and because they have things like a description and a type tools could can discover them and present them to the user and allow them to be changed program and programmatically so so the sketchy design for this language feature is a bit like this so a configuration looks a lot like an attribute set only it uses angled brackets might change don't get too angry or or enthusiastic about the syntax but yeah so it's you could think of it as an attribute set so if an attribute foo an attribute bar and an attribute ABC that actually refers to foo and bar so it's a recursive it's like a rec attribute set so if from this thing you select the ABC attribute you would get a value one two three because bar is true so if if true then one two three right but what you can now do is take that configuration module and apply a new module to it that sets bar to false and so now if from this module you select ABC then it will return a false then it goes to one two three times two so it will return two four six so this is pretty much exactly the behavior of the the Nixar's module system yeah and then the idea is that you can have some sort of syntax which I'm not sure about but to to attach fields or annotations to those to those options like documentation type default value and so on merge functions priorities yeah all that sort of thing basically all the things you have in the Nixar's module system right but now the idea is that we can apply this to building packages so rather than having packages as functions which have the problem that well there's no override mechanism I mean no good override mechanism there's no documentation and so on we can basically treat packages in the same way as the Nixar's module system treats system configurations so you you build a package in sort of a modular way by combining a bunch of modules so for example you could have a very sort of at the bottom you have a module that captures the concept of a derivation so what is a derivation well a derivation Nixar's a name and a version and it has a builder and it has arguments and it has an environment and if you set those things then you can evaluate a derivation attribute which produces a low level derivation so so this is very low level but now you can build higher level modules on top of that so for example this thing basically expresses the basic standard environment so the concept of phases and dependencies on other packages so this thing adds a an option called build inputs and an option called phases and it it implements this on top of the lower level derivation module by setting builder and arcs and and that causes a derivation to be computed yeah that uses these things so just to continue this a bit so you could have a module that captures the concept of a package so package has a description and a home page and so on and all these things have had descriptions and they have types so they're they're discoverable and and you get error messages so for example if if you use this previous so currently with derivations if for example you you misspell build inputs you're not going to get an error because nix derivation is basically just a bunch of environments variable bindings so yeah so there's no checking whatsoever there but here if you set an option that hasn't been declared then you get an error message just like in the in the module system yeah so you can build higher and higher level abstractions on this so for example you can extend the sort of the generic standard environment with the concept of a unix package which for example has a configure phase and because it runs a configure script yeah and then finally you can define a package so a package is something that extends the unix package module with something that sets a name description source but it also has its own option namely enable GUI so in this fictional example hello world has a GTK support so now you can say build inputs is if enable GUI then use GTK and this thing is now discoverable so you could have say a nix query package command which will show that this thing has an enable GUI option and they could have a nix install command that that allows you to set that option so yeah and you could override things using the xx same module system so that's just like in xos so that's about it so there are lots of other things you could imagine for the roadmap I'll skip that so yeah so what I should do is create a sort of a roadmap issue and where everybody can go wild with ideas and suggestions and then we should try to synthesize something workable from that so yeah that's it thank you. So for the configurations idea of using doing overrides how does that solve the memory issue don't you still need to hold on to the references to all the inputs? No because as soon as you evaluate the.drv attribute you don't need anything else anymore you can after that you can just discard everything that went into it so it's sort of like in nixOS if you evaluate system.build.top level you get a derivation out of that and at that point you can garbage collect all the inputs to that thing. So like when you're passing something as an input to something else you're passing the.drv not the composable thing? So presumably this is all fiction so here this GTK thing would trigger an evaluation of GTK.drv implicitly. On the slide about the nixos build tool you have been talking about avoiding rebuilds and similar properties have you looked into the recent paper build systems a la carte by Neil Mitchell, Simon Pinat-Jones and Andrey Markov they're analyzing various build systems there and nix is one of them and nix ticks off almost all features of the ultimate build system. Was that the I-Shift-P paper? Yes. Okay yeah it's on my to-do list. Okay, please take a look at this it gives very good names, good glossary to talk about these properties it's a great paper thank you. So the configuration options it excites the type system in me. I won't go there. Is there also so the nixos module system also supports the notion of overlays is this should is there a version of this also in this or is this then kind of extension? Yeah probably so that's sort of a high level thing that I haven't figured out yet so how you actually put these things together so I but you need some way to do that and yeah so I don't know yet. Okay yeah I think we've got to move maybe one more. Last one. It's more of a policy question I've seen a lot of commits like fixing Thunderbird like you showed and there has been commits for every separate packages. Why don't you enforce this kind of like do not reference GCC for all the package by default and then if some package really needs GCC then you could enable it. Right well that's kind of the point so we don't have a way really to enforce that yet so right now it's really only if people sort of notice that suddenly a closure has become much bigger so you can use these disallowed requisites attributes but very few packages do that. But you could use that by default like add that to MKE derivation and then by default just do not reference GCC there is no reason for most packages to do so. So probably for GCC that would work but you really want to say things like it should not reference any death output and that doesn't work at the moment because death output should be allowed to reference death output so you can't use the existing attributes for that. But yeah so I would definitely like to have in NixOS for example that we say that all the NixOS VM tests could just check that their closures don't have any death outputs in them and no GCC or clang so that would already help quite a bit. Okay all righty so thank you again for your talk.
because he worked on that during his PhD research a Nix roadmap because this should really be a functional language, reproducible builds, incremental builds for your project and that's example there was a situation where Thunderbird vaporware part of the talk so so yeah really a derivation so what is a derivation well a nix query package command which will show that tool you have been talking about avoiding definitely like to have in NixOS for example that