require.nix: dependency management for your favorite
Levy, Shea
Nix is a fantastic tool for managing the dependencies of your development projects, but ironically the Nix language itself has very limited facilities for modular code reuse and composition. In this talk, I will present on my recent experiments with a system, inspired by node.js's "require" mechanism, to define packages of nix language code that can depend on other packages. Together with conventions encouraging detailed documentation, good error messages, and clean interfaces, I hope this project can form the seed for a robust ecosystem of libraries and tools that can help move us past the monolith of nixpkgs and enable us to more easily benefit from each others' work. - Bio: Shea is a lead engineer at Target's data science and optimization group, whose production stack is built with Nix. He has been part of the NixOS community since 2010, has been working professionally with Nix since 2012, and is part of the Nix core team.
 Alrighty, so next up is Shay. Shay is working at Target, one of our sponsors. And Shay is also on the next core team that was created not too long ago. And Shay is going to take us on some sort of inception kind of ride, talking about the dependency manager for the dependency manager. I'm looking forward to it, give him a round of applause. Hi, thank you. As I said, I'm Shay, work at Target, been working with Nix for quite a while, and Required.Nix is a project that's come out of problems I've seen using Nix, especially in professional settings for managing development and deployment life cycles. So before I get to what to work with Required.Nix is, I'm going to go through some of the problems I've seen that inspired it. So first, status quo of what exists in the Nix world today is where a community of monolithic entry points. So almost every Nix expression in the world outside of Nix packages has this line at the top, and all of the useful, interesting functionality is pulled from there. Or maybe if you're a little bit more sophisticated, you'll have builtins.fetchkit, or your company will have its own Nix packages fork, or sometimes you'll have a company Nix repo, or if your company has a Mono repo, then it'll have a big Mono repo. But either way, this is really, for the most part, your Nix projects will all revolve around some big central repository of all Nix functionality. Why this is a problem, just about to get political, I'm going to make a stand on Mono repo versus Poly repo for our context in this world, not in general. Mono, our current workflow, especially Nix packages, but even outside of it, leads to very tightly coupled development. Everything within Nix packages is updated at once atomically, which makes sense because you want everything to work together, but it means the interfaces between our components are very blurred. If you make a standard end change, you can just change all of the packages that breaks in one commit, and then you never have to think about, was that a good interface to begin with, and is this change going to be usable for external users, etc. It's hard to compose in the sense of we have many different functionalities within Nix packages, and I can't easily say I want the Haskell builder from 1803, but within the package set of 1809, that's certainly not possible. But even the Haskell builder of a few weeks ago probably won't work, and it's also technically very difficult to actually plug it in. We also accumulate a bunch of responsibilities within Nix packages, so we do experiments, and we also have our stable package set, and we have a lot of different functionalities and different components, as I was saying about, and they all exist within one file system tree, one set of process for managing the system, for stable updates, for pull requests, for governance, all of that. Everything is kind of bound by the same system when we have this sort of mix of things. So, for example, I don't know if you guys have been following John Erickson's work on improving cross compilation in standard N, a big difficulty he's run into that I've seen is he has to kind of shove the work he's doing into the flow of Nix packages without breaking too many things and without causing too much disruption at once, because there's no good way for him to iterate on standard N independently of Nix packages. And kind of back to what Rock was talking about earlier in his talk, well, maybe he's yesterday, I can't even remember, where in the Nix world, we can get the benefits of a monorepo without actually having a monorepo by tying everything together at the top. In principle, we could do that same kind of thing with Nix packages, where Nix packages remains, where everything is integrated in together in the actual sort of trusted set of the world, but the individual components are built independently on their own time frame. So, this is the main thing that pushed me, but there are a few other things that I think can be solved with require.nix, which I also have not yet described. First, annotations, and by annotations I mean documentation, types, what are the flags, how do I, given some data, what's the metadata associated with that, whether that's a function definition or whether that's a package definition. We have some conventions for, I mean, especially for metadata, but for package metadata, but we don't have general conventions for this and we don't have discoverability around those annotations or discoverability for what functionality is even available. It's pretty hard to say, does this package exist unless you know enough about how Nix packages is laid out to already know most of the answer. You know, for, if you go through the attributes in the top level packages set, how do you know which of those is a package versus a sub package set, versus a set of library functionality, versus some combination of all of the above, we don't have that available to us within Nix packages very easily. And finally, open sourcing is actually very difficult, at least in my experience, because again we have these monolithic company Nix repos and they combine functionality that we would love to share that is completely generic, but it's also like all tied up with functionality like our private package sets and all of that. And if we open sourced it, we would have to break out that functionality and we wouldn't use the thing we open sourced because we have no way to combine it in with our big monorepo. So this is a problem that I kind of was looking at and sat down and did some design sessions and I think the answer, what we need in Nix, which we don't have, is packages and modules. And not packages and modules in the sense that Nix means them, but I mean in the sense of normal programming language packages and modules like Python has a module and package exports those modules and you import that and you get your definitions from that. And so required at Nix is a system to provide packages and modules for the Nix language. So there's some kind of core technical capabilities like a package specification saying, you know, this is the required at Nix package itself. It's got a description. It's got versions. These are the packages I depend on with version ranges and things like that. I can define my, what modules I export and kind of some formatting and just a heads up for all of these things. These are like the first thing I could think of that would cover exactly what I needed and nothing more. So I'm not saying this format is great. I'm saying we have a format. It gives metadata. It gives you everything you need to know to consume the package. Similarly, we could have lock files to say, okay, you depend on the base library. This, for this repo, use this version of base from this revision of GitHub. And, you know, just a note about how I've kind of built the lock files so far. They not only need to contain your direct dependencies, they need to contain your whole dependency set. But kind of going back to what Adam was talking about earlier, it still fails if you try to import a module not in your dependency list. So you don't have that problem that Node has, but you still have a top level kind of tying everything together. These are the specific versions I'm using in my package. And it also happens to allow for circular dependencies, but I don't think anybody's going to use that. And if they did, that's on their head, in my opinion. And the implementation I have written also allows for local lock files. It's expected you add them to your Gitignore so you can kind of override locally. And also we use these lock, we use a separate lock file for bootstrapping require.nix itself, which hopefully will go away when we get some like custom tooling. And then here's how we actually use it, and this is where the name comes from, inspired by Node.js' require. A module within the require.nix world is a function that takes this require function. And then you can call require to import a module. And if you ignore the module, if you omit the module here, it'll import like the top level module. So I'm importing the trivial module from base, and then I'm, and think of this as like the, which identifiers you're importing, it's just an inherit. But I think this kind of pattern will probably be common. So I'm saying, okay, get me the identity function from base, and then in my module is going to define identity functions specialized to lists. So that's really the core technical innovation of require.nix. But as I was saying, there's also, this is an opportunity to kind of improve on conventions around how we build our projects. If we're going to be building this new, if we're going to be building new libraries in this new system, we might as well take advantage of that opportunity to do things more standardized in a kind of more conventional way. So the first convention that I like, and I think this will probably be the most controversial of my conventions, is going for declarative or like plain old data domain specific types for everything we're representing. So here this is, this is an instance of a type I call source specification. So instead of doing everything in paths and doing like builtins.fetchget this, and then I'm going to get the subdirectory here, and then I'm going to filter that. You represent what your path is at a, in a declarative way, and then only at the very edges when you need to go to a nix built in or something else. I have a timer on my, I have the timer on my camera. If going to a nix built in or something else, that you actually need to convert it to a path. So you, you, you represent all of your, your data in the, in the domain that you want to think about it in. So there's, this is for sources. You could also have it for packages like a cabal package is different from a C++ package. And in today's world, what we do to express that difference is we have a different function we call. But if instead we just passed around like these are the, these are the fields that define my cabal package. These are the fields that define my C++ package. And only when I actually need to consume it in a derivation generic way do I call the function that converts it into a derivation. So it's sort of related to some of what Ilco is doing, whether the DRV field of the modules only, it's only consumed at the end, but you're operating over, you're operating over this kind of generic, this domain specific representation rather. So one of the, one of the reasons I like this kind of approach is it lets you sort of efficiently compose and, and, and efficiently and sanely compose or transform or query your data. So like in, in Nix packages, some of you may know there's this way to do composable filter source and it's sort of ad hoc and it's only for one thing. If you, if you actually just try to do filter source multiple times, you'll add to store multiple times. And actually it won't even work because you have to discard context and all of that. But even if you omit that part, you still are adding the path to the store over and over and over again. If, but in, in practice what you often have in some of these, these systems is you have a source, you have a, the representation of your project source and then you're going to call Kabbal to Nix. So I want a filter source that just has the Kabbal file in it, but I don't, but then I also want the top level source thing and then maybe I actually have a source with multiple projects in it and so I want to recurse into each one. And if, if each time I changed the path, I was re-adding it to the store, that'd be very inefficient. If I make all of, do all of my transformations on this representation, then only, and then only at the end when I actually needed to consume it, kind of close the loop back to a Nix primitive, then you get things are much more efficient and it's also much easier to look at a path and see what it's doing and what it means. So this is, this is for paths, but I think we should be doing it for everything. I think, you know, the, the attribute sets that we, so using make derivation and then using dot override, or maybe it's Kabbal dot make derivation and dot override for that instead and trying to kind of keep in mind those things instead of just having a standard of this is what the types look like. I think is, would really help kind of people use Nix packages, especially people who, and one of the things I really want to support is developers owning their own Nix expressions, even if they're not people into packaging and giving them a language that is as close to how they want to think about things as possible is a huge way to get there. And then also, this, again, going to what Ilko was talking about yesterday, this could eventually lead into tooling that lets you discover what are the flags for this package because you'd have like a simple flags field of like what it is and what they mean with metadata that you want to do it. And then you can say, okay, I want to install less with the secure flag. Great. I, my, my, my install tool knows how to parse out the flags from a package and call that in. So that's one thing. Another thing that I think everybody will be on board with is documentation. And here, documentation structured in a way. So the resolve function is from the source specifications library. And so the module, which is exporting resolve will also have a metadata field and you can have metadata like module level description metadata, but also annotations per whatever is exported there. So this is just a very, you know, the description here is intentionally long and detailed and the idea is you should be able to look at it and kind of understand what it's doing without needing to kind of jump, jump a lot. I mean, sometimes you're going to have to cross reference, but I think this documentation in terms of length is something we want to look at. I think having every identifier, having every exported top level identifier have its own description. This could have, you know, this could be Doc book or some kind of markup. I don't have strong opinions on that. I think we should just choose something and standardize it. And then of course, given if we're going to break things out into separate projects and separate libraries, the libraries themselves should have really good top level documentation of like just like you would have in any other project like this is what this library is for. This is how it's organized. This is how you should think about it. These are the top level concepts, etc. Next thing that I know a lot of people are, have been interested in is type annotations. So in addition to human readable descriptions, this is a machine readable description of the type of a function. And so I've come up with a way to encode kind of very generally encode types with as Nix values. So, you know, this is saying resolve is a function. The input type is a source specification and source specification is sort of defined in some other types module. And the output is a result type and result is like if you're familiar with Rust result type or it's an either where you've got an error condition or a success condition and say, okay, if this succeeded, then you will give back a path. If it failed, we'll give back some structured error type describing what the failure was. And then the structured error, the reason I gave this example is I also wanted to say structured errors as opposed to just throwing is often a better way to go because then you can pass information higher up the chain until the error message actually shows the context that the user is going to want. So this, the type definition I've got, the type language I came up with is rich enough to have like sums and record types. It also, as you can see, has function applications and function parameters. It's way too general for what we're actually want to do. But I just kind of, again, started with the basic thing that would work. So, and then the last thing that we can do is build tooling on top of all of this. So, again, all that documentation stuff and the types could be rendered in some nice way. We could have, you know, some hosted documentation for all of our libraries. We could expose configurations of tools as I mentioned earlier. If your package has some flags, then your install script could use those flags to do things. And we could build tooling that's sort of native to require.next. Right now, to like resolve that package description in the lock file that I was talking about earlier, right now to do that I have some like janky bootstrap code that reads it all in. And then a small subset of the stuff that require.next itself uses is implemented twice, once within require.next's repo. So it can read the file enough to fetch it, its real dependencies and then bootstrap itself. But if we had that, the only reason I do that is so that you can just write a default on Nix and use Nix instantiator Nix build like you normally do. But if we had required that Nix native tooling, it could do the job of parsing these package definitions and these lock files. It could be smarter about how it fetches the dependencies and shares them. And it can also give you a nice repel experience where for any given identifier you can easily find, okay, give me the doc string for this or show me the type of this. And all of that. So, yeah. So now I want to kind of go into, so this is the idea. This is where I want to go. And the big question is what's coming next? And really the reason what I don't have, I have stuff locally on my machine that works and it's proven the concept for me. The reason I haven't gone much further with it is this is not a viable project unless there's buy-in from the community. And I personally am not a sort of language packaging expert. I don't know what the right convention should be. I don't know what the right field should be. So I have a number of things that I would love for people to be looking at in the near future, possibly even the hackathon if you guys are here. And things that I also want to be looking at, but I'm mostly interested in following where there's actually community interest. So one set of things we can do is make some libraries. So we could have language specific libraries. We've got the Haskell infrastructure. We've got the Perl infrastructure. We've got the Rust infrastructure. They all have their own builders. Many of them have their own kind of ways to generate expressions. They have their own package sets. Maybe we could break those out. And then we can have a look at the opinions on how that should be done. I kind of disagree with the love for the Yarn to Nix approach. I think Cabal to Nix is actually closer to the one true path. But ask me later if you want to hear more about that. And just as a little teaser, this is something that we're using within Target. And yeah, these are probably proprietary names I shouldn't be sharing, but oh well. Where we have a single repo with a bunch of packages defined in a Cabal.project. And we just pass this list and a given tag for that repo and it just pulls them all in and we have them available in our package set. And this is something we would love to open source. This functionality is not target specific at all. But again, we don't have a nice Haskell library to plug it into and it would be kind of weird within Nix packages proper. Also just Nix packages library, all of the kind of like basic stuff that's in there could be broken out in particular kind of like the basic trivial stuff could easily be broken out into its own base module of just this is the kind of stuff you want to do if you're doing basic stuff with Nix. Standard end like the core concept of what is part of the standard environment and what is the interface to make derivation and all of those things could be separated out and iterated and experiments it upon. And Nix packages could do the actual bootstrapping and definition of what the standard end is for Nix packages now but you could come along and plug your own in or plug your own in just for some packages and kind of explore with that. And the Nix OS module system is another significant candidate for, you know, it's not really Nix packages specific, it's not really Nix OS specific as we learned earlier today. It's a complete standalone Nix language functionality that could be in its own place and be experimented on and developed on its own path. The other thing and this is I think where I need more help on is all those standardizations I talked about earlier, I don't know what they should be. I don't know what the right answers on these things. I really what I want to do is import best practices from other communities and get the people who are expert to care about these things to kind of help us figure out what to do. So what should the package format be? What should the lock format be? What should our conventions be? And in particular with conventions one thing I have completely not gotten to is tests and test suites. Like how do we want to represent those? Do those go in the package format? Are they some kind of separate thing? What kind of tooling would we need to run these tests? And all of that. And then of course the one last project I want to do is publish these slides and announce that this is happening. But again that depends on if people are interested and want to make it happen. So that's all I had. Thank you guys and we've got about 10 minutes it looks like for questions. So. So are there any questions or is everything already entirely clear and. Hi thanks for the talk. I have a couple of questions or I'll ask just one and maybe I'll raise my hand again. So this is all nice and it's a great idea. Pros and cons as you mentioned with Monoribo and libraries. But do you or have you thought about like a migration path because we all have the next packages as they are and we're using as they are and it's a lot of work that's been put into it. So you know and you want to don't want to break backwards compatibility and so on. Yeah so I mean I think for migration path one one thing that I've thought of to start with is start with this kind of. Functionality doesn't really make sense in next packages to begin with like it's not core to how you use Nick so S. But it maybe it's how you use Nixon development. So like this like this nice stuff we have for building Haskell package sets right. It doesn't really need to be a next package is next packages has its Haskell package set it's fine. So we could start with those libraries validate validate the concepts validate the tooling go with what works. The one thing I'm hesitant to say on that though is I'd rather just go with where people are interested and willing to do the implementation. And so I'm not going to say no to people who if there's interest in kind of jumping straight to standard M. In terms of migration like once we're ready to do that I think that's going to be kind of it's going to need a community wide consideration because it's kind of fun going to fundamentally change what happens when you import next packages and how it's all tied together. The one nice thing though is that from a on a surface level at least this will all be opaque to the user because the default next will still work. It'll just kind of import it'll have some library dependencies and in its package with the default next will handle the bootstrapping for you and pull it all together. So it's only once you get to the point of people want to override it or developers how they want to use it and I don't have good thoughts there on what exactly that path is again that depends on buy in and interest. Okay let's take a question from the Internet. Why do the lock files not contain the hash of the next file that is being imported. I mean in principle they could accept this first of all it's not a next file that's being imported its entire its entire package right. So let's go back to the where is it. So basically this this kind of gets back to the question of there's a trade off in usability versus kind of power I guess not really power but this this is a format that kind of I trust as at a high level. I trust that if I'm referencing a tag that it's referentially transparent in the meeting in a meaningful way like this. I trust the owner of this repo not to update the tag and so this gives a unique name. Now it's true that this is sort of cheating. This is this is not good enough for Nix but it's good enough for me. The question is if we force hashes if we force people to calculate all these things in practice I have genuinely seen people give up Nix over little issues like this like they have to update the hash every time. And so the question and I'm not I guess I'll say I'm not kind of wed to this completely but my my assumption my current gut feel is if we can define things at a domain specific level in a way that lets people that that that it that we the concept of purity makes sense for that domain. We'll get a huge win on usability and the loss on purity will be slim to none. One more question. How much time of any Nix command depends on fetching requires from the network. I mean so basically the way the way the way it currently works is they just kind of fetches everything in the lock up front. So you know it's I guess that that's a a question. So we're I'm currently using the fetch get from the targets Nix fetches repo which has some pretty good caching. So once you've already fetched it once it's actually it's actually pretty fast to kind of validate that. Oh it's already there and I'm fine. Nix the built-in fetch get also does that if you provide a revision but since we're just writing tags that sometimes has to refetch. But so it's not too long. I haven't actually timed the whole thing and I but I do think this is a place where standalone require dot Nix native tooling would help because we could do smart things about fetching and knowing when it needs to fetch and and how it all handles that outside of a Nix evaluation. Yeah just a small point on what was just discussed in terms of putting hashes there for tags for example so I've seen people push tags to a different version. So I would be a big proponent of saying OK do put the shots in there but I think what would solve the problem much more nicely is that if you have something like this where you have a thing where you say OK I understand what a what did you call it source. Source specification is right just put a little bit of tooling into the next tool that says your hash is different. Press yes to update right and it writes your thing it's much easier and I think it gets us both. So but the other. So the other thing is what I will say here is this kind of format also has a mode for specifying a branch and a get revision. And so as long as you trust gets revisions you can just use those. It still doesn't require a hash of the file system contents though. But yeah we could also we could do that. The discussions are already starting was yes let's do that. No let's not let's not. Keep it going. It's good to see someone else thinking about these sort of workflows because I have a couple of similar things where I want to use certain library functions out of Nix packages but not the actual packages themselves. And in my case I've actually found what works really well is at the top to have a let binding that says something like let licenses equals built in stuff fetched tarball Nix packages by shot two fifty six dot lib dot licenses in which case I fetched a small sub component of Nix packages in a deterministic way from the monorepo creating in effect my own library without busting up the monorepo and introducing version dependencies for upstream testing and discoverability issues. Have you tried this approach and is it working. Yeah I mean so the one thing that it doesn't support is when I want to push when I like I if I want to push changes that I in my opinion don't really make sense in Nix packages except for the fact that Nix packages is kind of the central place doesn't really help that and it doesn't help with a couple deployment development aspect of the monorepo actually causes problems. And one thing I actually wanted to say that I didn't mention at the time is of course all of these things I'm talking about are about kind of practice and policy and in principle you can implement policies that don't have these problems within a monorepo but I do believe in my experience has been that the monorepo that we have in the policy practices we follow encourage this kind of lack of clean interfaces between components and that sort of thing. Any more questions from someone who hasn't already asked a question. We're at two minutes. It's more of a statement. I think this is a brilliant idea and something that hasn't been mentioned is that if it gets adopted packages could provide their required at Nix file and then as Nix packages contributor we could just import the derivation directly and we don't have to do the packaging. Yeah exactly. You could you could provide a required at Nix file within your package or a package set that like gives a list of overlays so it could either import its own Nix packages or it could give an overlay list that you can apply to your own Nix packages and that sort of thing. Any more sentences with question marks. So another similar point that I've seen a handful of people in the community struggle with is if you have a package that you don't intend to upstream to Nix packages. There's some duplication between the default dot Nix in the repository of the package the shell dot Nix the release dot Nix and potentially your company's monorepo or potentially Nix packages monorepo. Have you thought a little bit about solving this particular problem or do you have any. I haven't. I mean I have thought about that problem. I don't know that this is a full solution to that but I do think having more abilities to kind of reference standalone libraries you'd have one place where the actual expression lives. Have you also worked on a say practical example in which you can demonstrate what value this modellization approach provides because I think I also agree with you we need we need way better modularization but I think it would help a lot. If there's something you can show something really practical for which it makes sense. Have you thought about that. I mean yeah so I've thought about that and that's to me that's the next step to see it wasn't clear to me before I'm talking about it this weekend is whether anybody else was on board. Now it seems like it is so yes my next step is polish up the Haskell the sort of library of like nice Haskell functionality that we're using we're using some of it within targets some of what I've used at previous companies and just kind of make it a nice clean standalone piece of functionality. That's that's my next project unless again there's some big push for some other thing that I should help out with. Any. Oh we're at not negative 20 seconds anyway. Are you are you up for answering one more question. Yeah absolutely. Then let's go for it. Talk all night. Yeah so I mean I think this sounds like a really great initiative. One question I have around says I think I agree it's generally. Too difficult to get as a metadata of stuff and says messes up the user experience and also the tooling like for example Nick's and query also basically was a clusterfuck and Nick's sort of improves it but it needs a caching mechanism. And it seems to me a lot of those difficulties could be ameliorated if we just use JSON or whatever to specify the non well the actual metadata stuff like the description as a shas because you don't actually want to compute some and having some as some static information. So normal tooling can query and update automatically also I think would improve a lot so one question I have is how bad it is idea of doing stuff is Nick's expression. I suppose so did you did you consider that you consider doing something like your own. I meant to mention you'll notice both of these are completely representable as Jason that's not an accident. So yes I'm with you possibly I'm open to the idea of saying the metadata for the packages themselves should be Jason so easier make tooling. Or just use hNix or RNix or the Nix evaluator and C++ and import it that way. Anything else. Cool. OK already. Thank you. Thank you.
 Alrighty, so next up is Shay. Shay is working at Target, one of our sponsors.  And Shay is also on the next core team that was created not too long ago.  And Shay is going to take us on some sort of inception kind of ride, talking about the dependency manager for the dependency manager.  I'm looking forward to it, give him a round of applause.  Hi, thank you. As I said, I'm Shay, work at Target, been working with Nix for quite a while, and Required.Nix is a project that's come out of problems I've seen using Nix, especially in professional settings for managing development and deployment life cycles.  So before I get to what to work with Required.Nix is, I'm going to go through some of the problems I've seen that inspired it.  So first, status quo of what exists in the Nix world today is where a community of monolithic entry points.  So almost every Nix expression in the world outside of Nix packages has this line at the top, and all of the useful, interesting functionality is pulled from there.  Or maybe if you're a little bit more sophisticated, you'll have builtins.fetchkit, or your company will have its own Nix packages fork, or sometimes you'll have a company Nix repo, or if your company has a Mono repo, then it'll have a big Mono repo.  But either way, this is really, for the most part, your Nix projects will all revolve around some big central repository of all Nix functionality.  Why this is a problem, just about to get political, I'm going to make a stand on Mono repo versus Poly repo for our context in this world, not in general.  Mono, our current workflow, especially Nix packages, but even outside of it, leads to very tightly coupled development.  Everything within Nix packages is updated at once atomically, which makes sense because you want everything to work together, but it means the interfaces between our components are very blurred.  If you make a standard end change, you can just change all of the packages that breaks in one commit, and then you never have to think about, was that a good interface to begin with, and is this change going to be usable for external users, etc.  It's hard to compose in the sense of we have many different functionalities within Nix packages, and I can't easily say I want the Haskell builder from 1803, but within the package set of 1809, that's certainly not possible.  But even the Haskell builder of a few weeks ago probably won't work, and it's also technically very difficult to actually plug it in.  We also accumulate a bunch of responsibilities within Nix packages, so we do experiments, and we also have our stable package set, and we have a lot of different functionalities and different components, as I was saying about, and they all exist within one file system tree,  one set of process for managing the system, for stable updates, for pull requests, for governance, all of that. Everything is kind of bound by the same system when we have this sort of mix of things.  So, for example, I don't know if you guys have been following John Erickson's work on improving cross compilation in standard N, a big difficulty he's run into that I've seen is he has to kind of shove the work he's doing into the flow of Nix packages without breaking too many things and without causing too much disruption at once,  because there's no good way for him to iterate on standard N independently of Nix packages. And kind of back to what Rock was talking about earlier in his talk, well, maybe he's yesterday, I can't even remember, where in the Nix world, we can get the benefits of a monorepo without actually having a monorepo by tying everything together at the top.  In principle, we could do that same kind of thing with Nix packages, where Nix packages remains, where everything is integrated in together in the actual sort of trusted set of the world, but the individual components are built independently on their own time frame.  So, this is the main thing that pushed me, but there are a few other things that I think can be solved with require.nix, which I also have not yet described.  First, annotations, and by annotations I mean documentation, types, what are the flags, how do I, given some data, what's the metadata associated with that, whether that's a function definition or whether that's a package definition.  We have some conventions for, I mean, especially for metadata, but for package metadata, but we don't have general conventions for this and we don't have discoverability around those annotations or discoverability for what functionality is even available.  It's pretty hard to say, does this package exist unless you know enough about how Nix packages is laid out to already know most of the answer.  You know, for, if you go through the attributes in the top level packages set, how do you know which of those is a package versus a sub package set, versus a set of library functionality, versus some combination of all of the above, we don't have that available to us within Nix packages very easily.  And finally, open sourcing is actually very difficult, at least in my experience, because again we have these monolithic company Nix repos and they combine functionality that we would love to share that is completely generic, but it's also like all tied up with functionality like our private package sets and all of that.  And if we open sourced it, we would have to break out that functionality and we wouldn't use the thing we open sourced because we have no way to combine it in with our big monorepo.  So this is a problem that I kind of was looking at and sat down and did some design sessions and I think the answer, what we need in Nix, which we don't have, is packages and modules.  And not packages and modules in the sense that Nix means them, but I mean in the sense of normal programming language packages and modules like Python has a module and package exports those modules and you import that and you get your definitions from that.  And so required at Nix is a system to provide packages and modules for the Nix language.  So there's some kind of core technical capabilities like a package specification saying, you know, this is the required at Nix package itself.  It's got a description. It's got versions. These are the packages I depend on with version ranges and things like that.  I can define my, what modules I export and kind of some formatting and just a heads up for all of these things. These are like the first thing I could think of that would cover exactly what I needed and nothing more.  So I'm not saying this format is great. I'm saying we have a format. It gives metadata. It gives you everything you need to know to consume the package.  Similarly, we could have lock files to say, okay, you depend on the base library. This, for this repo, use this version of base from this revision of GitHub.  And, you know, just a note about how I've kind of built the lock files so far. They not only need to contain your direct dependencies, they need to contain your whole dependency set.  But kind of going back to what Adam was talking about earlier, it still fails if you try to import a module not in your dependency list.  So you don't have that problem that Node has, but you still have a top level kind of tying everything together. These are the specific versions I'm using in my package.  And it also happens to allow for circular dependencies, but I don't think anybody's going to use that. And if they did, that's on their head, in my opinion.  And the implementation I have written also allows for local lock files. It's expected you add them to your Gitignore so you can kind of override locally.  And also we use these lock, we use a separate lock file for bootstrapping require.nix itself, which hopefully will go away when we get some like custom tooling.  And then here's how we actually use it, and this is where the name comes from, inspired by Node.js' require. A module within the require.nix world is a function that takes this require function.  And then you can call require to import a module. And if you ignore the module, if you omit the module here, it'll import like the top level module.  So I'm importing the trivial module from base, and then I'm, and think of this as like the, which identifiers you're importing, it's just an inherit.  But I think this kind of pattern will probably be common. So I'm saying, okay, get me the identity function from base, and then in my module is going to define identity functions specialized to lists.  So that's really the core technical innovation of require.nix. But as I was saying, there's also, this is an opportunity to kind of improve on conventions around how we build our projects.  If we're going to be building this new, if we're going to be building new libraries in this new system, we might as well take advantage of that opportunity to do things more standardized in a kind of more conventional way.  So the first convention that I like, and I think this will probably be the most controversial of my conventions, is going for declarative or like plain old data domain specific types for everything we're representing.  So here this is, this is an instance of a type I call source specification.  So instead of doing everything in paths and doing like builtins.fetchget this, and then I'm going to get the subdirectory here, and then I'm going to filter that.  You represent what your path is at a, in a declarative way, and then only at the very edges when you need to go to a nix built in or something else.  I have a timer on my, I have the timer on my camera.  If going to a nix built in or something else, that you actually need to convert it to a path. So you, you, you represent all of your, your data in the, in the domain that you want to think about it in.  So there's, this is for sources. You could also have it for packages like a cabal package is different from a C++ package.  And in today's world, what we do to express that difference is we have a different function we call.  But if instead we just passed around like these are the, these are the fields that define my cabal package.  These are the fields that define my C++ package.  And only when I actually need to consume it in a derivation generic way do I call the function that converts it into a derivation.  So it's sort of related to some of what Ilco is doing, whether the DRV field of the modules only, it's only consumed at the end, but you're operating over, you're operating over this kind of generic, this domain specific representation rather.  So one of the, one of the reasons I like this kind of approach is it lets you sort of efficiently compose and, and, and efficiently and sanely compose or transform or query your data.  So like in, in Nix packages, some of you may know there's this way to do composable filter source and it's sort of ad hoc and it's only for one thing.  If you, if you actually just try to do filter source multiple times, you'll add to store multiple times.  And actually it won't even work because you have to discard context and all of that.  But even if you omit that part, you still are adding the path to the store over and over and over again.  If, but in, in practice what you often have in some of these, these systems is you have a source, you have a, the representation of your project source and then you're going to call Kabbal to Nix.  So I want a filter source that just has the Kabbal file in it, but I don't, but then I also want the top level source thing and then maybe I actually have a source with multiple projects in it and so I want to recurse into each one.  And if, if each time I changed the path, I was re-adding it to the store, that'd be very inefficient.  If I make all of, do all of my transformations on this representation, then only, and then only at the end when I actually needed to consume it, kind of close the loop back to a Nix primitive, then you get things are much more efficient and it's also much easier to look at a path and see what it's doing and what it means.  So this is, this is for paths, but I think we should be doing it for everything.  I think, you know, the, the attribute sets that we, so using make derivation and then using dot override, or maybe it's Kabbal dot make derivation and dot override for that instead and trying to kind of keep in mind those things instead of just having a standard of this is what the types look like.  I think is, would really help kind of people use Nix packages, especially people who, and one of the things I really want to support is developers owning their own Nix expressions, even if they're not people into packaging and giving them a language that is as close to how they want to think about things as possible is a huge way to get there.  And then also, this, again, going to what Ilko was talking about yesterday, this could eventually lead into tooling that lets you discover what are the flags for this package because you'd have like a simple flags field of like what it is and what they mean with metadata that you want to do it.  And then you can say, okay, I want to install less with the secure flag. Great. I, my, my, my install tool knows how to parse out the flags from a package and call that in.  So that's one thing. Another thing that I think everybody will be on board with is documentation. And here, documentation structured in a way.  So the resolve function is from the source specifications library. And so the module, which is exporting resolve will also have a metadata field and you can have metadata like module level description metadata, but also annotations per whatever is exported there.  So this is just a very, you know, the description here is intentionally long and detailed and the idea is you should be able to look at it and kind of understand what it's doing without needing to kind of jump, jump a lot.  I mean, sometimes you're going to have to cross reference, but I think this documentation in terms of length is something we want to look at.  I think having every identifier, having every exported top level identifier have its own description. This could have, you know, this could be Doc book or some kind of markup.  I don't have strong opinions on that. I think we should just choose something and standardize it.  And then of course, given if we're going to break things out into separate projects and separate libraries, the libraries themselves should have really good top level documentation of like just like you would have in any other project like this is what this library is for.  This is how it's organized. This is how you should think about it. These are the top level concepts, etc.  Next thing that I know a lot of people are, have been interested in is type annotations.  So in addition to human readable descriptions, this is a machine readable description of the type of a function.  And so I've come up with a way to encode kind of very generally encode types with as Nix values.  So, you know, this is saying resolve is a function. The input type is a source specification and source specification is sort of defined in some other types module.  And the output is a result type and result is like if you're familiar with Rust result type or it's an either where you've got an error condition or a success condition and say, okay, if this succeeded, then you will give back a path.  If it failed, we'll give back some structured error type describing what the failure was.  And then the structured error, the reason I gave this example is I also wanted to say structured errors as opposed to just throwing is often a better way to go because then you can pass information higher up the chain until the error message actually shows the context that the user is going to want.  So this, the type definition I've got, the type language I came up with is rich enough to have like sums and record types. It also, as you can see, has function applications and function parameters.  It's way too general for what we're actually want to do. But I just kind of, again, started with the basic thing that would work.  So, and then the last thing that we can do is build tooling on top of all of this.  So, again, all that documentation stuff and the types could be rendered in some nice way.  We could have, you know, some hosted documentation for all of our libraries.  We could expose configurations of tools as I mentioned earlier.  If your package has some flags, then your install script could use those flags to do things.  And we could build tooling that's sort of native to require.next.  Right now, to like resolve that package description in the lock file that I was talking about earlier, right now to do that I have some like janky bootstrap code that reads it all in.  And then a small subset of the stuff that require.next itself uses is implemented twice, once within require.next's repo.  So it can read the file enough to fetch it, its real dependencies and then bootstrap itself.  But if we had that, the only reason I do that is so that you can just write a default on Nix and use Nix instantiator Nix build like you normally do.  But if we had required that Nix native tooling, it could do the job of parsing these package definitions and these lock files.  It could be smarter about how it fetches the dependencies and shares them.  And it can also give you a nice repel experience where for any given identifier you can easily find, okay, give me the doc string for this or show me the type of this.  And all of that.  So, yeah.  So now I want to kind of go into, so this is the idea.  This is where I want to go.  And the big question is what's coming next?  And really the reason what I don't have, I have stuff locally on my machine that works and it's proven the concept for me.  The reason I haven't gone much further with it is this is not a viable project unless there's buy-in from the community.  And I personally am not a sort of language packaging expert.  I don't know what the right convention should be.  I don't know what the right field should be.  So I have a number of things that I would love for people to be looking at in the near future, possibly even the hackathon if you guys are here.  And things that I also want to be looking at, but I'm mostly interested in following where there's actually community interest.  So one set of things we can do is make some libraries.  So we could have language specific libraries.  We've got the Haskell infrastructure.  We've got the Perl infrastructure.  We've got the Rust infrastructure.  They all have their own builders.  Many of them have their own kind of ways to generate expressions.  They have their own package sets.  Maybe we could break those out.  And then we can have a look at the opinions on how that should be done.  I kind of disagree with the love for the Yarn to Nix approach.  I think Cabal to Nix is actually closer to the one true path.  But ask me later if you want to hear more about that.  And just as a little teaser, this is something that we're using within Target.  And yeah, these are probably proprietary names I shouldn't be sharing, but oh well.  Where we have a single repo with a bunch of packages defined in a Cabal.project.  And we just pass this list and a given tag for that repo and it just pulls them all in  and we have them available in our package set.  And this is something we would love to open source.  This functionality is not target specific at all.  But again, we don't have a nice Haskell library to plug it into  and it would be kind of weird within Nix packages proper.  Also just Nix packages library, all of the kind of like basic stuff that's in there  could be broken out in particular kind of like the basic trivial stuff  could easily be broken out into its own base module of just this is the kind of stuff  you want to do if you're doing basic stuff with Nix.  Standard end like the core concept of what is part of the standard environment  and what is the interface to make derivation and all of those things could be separated out  and iterated and experiments it upon.  And Nix packages could do the actual bootstrapping and definition of what the standard end is  for Nix packages now but you could come along and plug your own in or plug your own in  just for some packages and kind of explore with that.  And the Nix OS module system is another significant candidate for, you know,  it's not really Nix packages specific, it's not really Nix OS specific as we learned earlier today.  It's a complete standalone Nix language functionality that could be in its own place  and be experimented on and developed on its own path.  The other thing and this is I think where I need more help on is all those standardizations  I talked about earlier, I don't know what they should be.  I don't know what the right answers on these things.  I really what I want to do is import best practices from other communities  and get the people who are expert to care about these things to kind of help us figure out what to do.  So what should the package format be?  What should the lock format be?  What should our conventions be?  And in particular with conventions one thing I have completely not gotten to is tests and test suites.  Like how do we want to represent those?  Do those go in the package format?  Are they some kind of separate thing?  What kind of tooling would we need to run these tests?  And all of that.  And then of course the one last project I want to do is publish these slides and announce that this is happening.  But again that depends on if people are interested and want to make it happen.  So that's all I had.  Thank you guys and we've got about 10 minutes it looks like for questions.  So.  So are there any questions or is everything already entirely clear and.  Hi thanks for the talk.  I have a couple of questions or I'll ask just one and maybe I'll raise my hand again.  So this is all nice and it's a great idea.  Pros and cons as you mentioned with Monoribo and libraries.  But do you or have you thought about like a migration path because we all have the next packages as they are and we're using as they are and it's a lot of work that's been put into it.  So you know and you want to don't want to break backwards compatibility and so on.  Yeah so I mean I think for migration path one one thing that I've thought of to start with is start with this kind of.  Functionality doesn't really make sense in next packages to begin with like it's not core to how you use Nick so S.  But it maybe it's how you use Nixon development.  So like this like this nice stuff we have for building Haskell package sets right.  It doesn't really need to be a next package is next packages has its Haskell package set it's fine.  So we could start with those libraries validate validate the concepts validate the tooling go with what works.  The one thing I'm hesitant to say on that though is I'd rather just go with where people are interested and willing to do the implementation.  And so I'm not going to say no to people who if there's interest in kind of jumping straight to standard M.  In terms of migration like once we're ready to do that I think that's going to be kind of it's going to need a community wide consideration because it's kind of fun going to fundamentally change what happens when you import next packages and how it's all tied together.  The one nice thing though is that from a on a surface level at least this will all be opaque to the user because the default next will still work.  It'll just kind of import it'll have some library dependencies and in its package with the default next will handle the bootstrapping for you and pull it all together.  So it's only once you get to the point of people want to override it or developers how they want to use it and I don't have good thoughts there on what exactly that path is again that depends on buy in and interest.  Okay let's take a question from the Internet.  Why do the lock files not contain the hash of the next file that is being imported.  I mean in principle they could accept this first of all it's not a next file that's being imported its entire its entire package right.  So let's go back to the where is it.  So basically this this kind of gets back to the question of there's a trade off in usability versus kind of power I guess not really power but this this is a format that kind of I trust as at a high level.  I trust that if I'm referencing a tag that it's referentially transparent in the meeting in a meaningful way like this.  I trust the owner of this repo not to update the tag and so this gives a unique name.  Now it's true that this is sort of cheating.  This is this is not good enough for Nix but it's good enough for me.  The question is if we force hashes if we force people to calculate all these things in practice I have genuinely seen people give up Nix over little issues like this like they have to update the hash every time.  And so the question and I'm not I guess I'll say I'm not kind of wed to this completely but my my assumption my current gut feel is if we can define things at a domain specific level in a way that lets people that that that it that we the concept of purity makes sense for that domain.  We'll get a huge win on usability and the loss on purity will be slim to none.  One more question.  How much time of any Nix command depends on fetching requires from the network.  I mean so basically the way the way the way it currently works is they just kind of fetches everything in the lock up front.  So you know it's I guess that that's a a question.  So we're I'm currently using the fetch get from the targets Nix fetches repo which has some pretty good caching.  So once you've already fetched it once it's actually it's actually pretty fast to kind of validate that.  Oh it's already there and I'm fine.  Nix the built-in fetch get also does that if you provide a revision but since we're just writing tags that sometimes has to refetch.  But so it's not too long.  I haven't actually timed the whole thing and I but I do think this is a place where standalone require dot Nix native tooling would help because we could do smart things about fetching and knowing when it needs to fetch and and how it all handles that outside of a Nix evaluation.  Yeah just a small point on what was just discussed in terms of putting hashes there for tags for example so I've seen people push tags to a different version.  So I would be a big proponent of saying OK do put the shots in there but I think what would solve the problem much more nicely is that if you have something like this where you have a thing where you say OK I understand what a what did you call it source.  Source specification is right just put a little bit of tooling into the next tool that says your hash is different. Press yes to update right and it writes your thing it's much easier and I think it gets us both.  So but the other.  So the other thing is what I will say here is this kind of format also has a mode for specifying a branch and a get revision.  And so as long as you trust gets revisions you can just use those. It still doesn't require a hash of the file system contents though.  But yeah we could also we could do that.  The discussions are already starting was yes let's do that. No let's not let's not.  Keep it going.  It's good to see someone else thinking about these sort of workflows because I have a couple of similar things where I want to use certain library functions out of Nix packages but not the actual packages themselves.  And in my case I've actually found what works really well is at the top to have a let binding that says something like let licenses equals built in stuff fetched tarball Nix packages by shot two fifty six dot lib dot licenses in which case I fetched a small sub component of Nix packages in a deterministic way from the  monorepo creating in effect my own library without busting up the monorepo and introducing version dependencies for upstream testing and discoverability issues. Have you tried this approach and is it working.  Yeah I mean so the one thing that it doesn't support is when I want to push when I like I if I want to push changes that I in my opinion don't really make sense in Nix packages except for the fact that Nix packages is kind of the central place doesn't really help that and it doesn't help with a couple deployment development aspect of the monorepo actually causes problems.  And one thing I actually wanted to say that I didn't mention at the time is of course all of these things I'm talking about are about kind of practice and policy and in principle you can implement policies that don't have these problems within a monorepo but I do believe in my experience has been that the monorepo that we have in the  policy practices we follow encourage this kind of lack of clean interfaces between components and that sort of thing. Any more questions from someone who hasn't already asked a question.  We're at two minutes.  It's more of a statement.  I think this is a brilliant idea and something that hasn't been mentioned is that if it gets adopted packages could provide their required at Nix file and then as Nix packages contributor we could just import the derivation directly and we don't have to do the packaging.  Yeah exactly. You could you could provide a required at Nix file within your package or a package set that like gives a list of overlays so it could either import its own Nix packages or it could give an overlay list that you can apply to your own Nix packages and that sort of thing.  Any more sentences with question marks.  So another similar point that I've seen a handful of people in the community struggle with is if you have a package that you don't intend to upstream to Nix packages.  There's some duplication between the default dot Nix in the repository of the package the shell dot Nix the release dot Nix and potentially your company's monorepo or potentially Nix packages monorepo.  Have you thought a little bit about solving this particular problem or do you have any.  I haven't. I mean I have thought about that problem. I don't know that this is a full solution to that but I do think having more abilities to kind of reference standalone libraries you'd have one place where the actual expression lives.  Have you also worked on a say practical example in which you can demonstrate what value this modellization approach provides because I think I also agree with you we need we need way better modularization but I think it would help a lot.  If there's something you can show something really practical for which it makes sense. Have you thought about that.  I mean yeah so I've thought about that and that's to me that's the next step to see it wasn't clear to me before I'm talking about it this weekend is whether anybody else was on board.  Now it seems like it is so yes my next step is polish up the Haskell the sort of library of like nice Haskell functionality that we're using we're using some of it within targets some of what I've used at previous companies and just kind of make it a nice clean standalone piece of functionality.  That's that's my next project unless again there's some big push for some other thing that I should help out with.  Any.  Oh we're at not negative 20 seconds anyway.  Are you are you up for answering one more question. Yeah absolutely. Then let's go for it.  Talk all night. Yeah so I mean I think this sounds like a really great initiative. One question I have around says I think I agree it's generally.  Too difficult to get as a metadata of stuff and says messes up the user experience and also the tooling like for example Nick's and query also basically was a clusterfuck and Nick's sort of improves it but it needs a caching mechanism.  And it seems to me a lot of those difficulties could be ameliorated if we just use JSON or whatever to specify the non well the actual metadata stuff like the description as a shas because you don't actually want to compute some and having some as some static information.  So normal tooling can query and update automatically also I think would improve a lot so one question I have is how bad it is idea of doing stuff is Nick's expression.  I suppose so did you did you consider that you consider doing something like your own. I meant to mention you'll notice both of these are completely representable as Jason that's not an accident.  So yes I'm with you possibly I'm open to the idea of saying the metadata for the packages themselves should be Jason so easier make tooling.  Or just use hNix or RNix or the Nix evaluator and C++ and import it that way.  Anything else.  Cool.  OK already.  Thank you.  Thank you.
Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_0_59.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_60_959.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_960_2435.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_2436_2735.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_2736_3251.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_3252_4163.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_4164_4979.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_4980_5459.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_5460_6467.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_6468_6635.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_6636_8291.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_8292_8831.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_8832_8939.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_8940_8987.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_8988_9059.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_9060_10451.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_10452_10523.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_10524_10571.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_10572_11159.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_11160_12011.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_12012_15923.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_15924_16043.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_16044_17207.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_17208_17999.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_18000_18611.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_18612_24251.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_24252_25115.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_25116_25271.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_25272_27251.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_27252_27359.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_27360_27539.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_27540_27971.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_27972_28319.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_28320_28547.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_28548_28595.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_28596_28931.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_28932_29591.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_29592_30719.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_30720_30911.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_30912_31871.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_31872_32051.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_32052_32171.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_32172_33107.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_33108_35807.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_35808_37799.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_37800_40931.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_40932_43139.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_43140_45623.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_45624_49619.png Shea_Levy_-_require_nix__dependency_management_for_your_favorite_DHOLjsyXPtM_49620_49691.png
Alrighty, so next up is Shay. Shay is working at Target, one of our sponsors. And Shay is also on the next core team that was created not too long ago. And Shay is going to take us on some sort of inception kind of ride, talking about the dependency manager for the dependency manager. I'm looking forward to it, give him a round of applause. Hi, thank you. As I said, I'm Shay, work at Target, been working with Nix for quite a while, and Required.Nix is a project that's come out of problems I've seen using Nix, especially in professional settings for managing development and deployment life cycles. So before I get to what to work with Required.Nix is, I'm going to go through some of the problems I've seen that inspired it. So first, status quo of what exists in the Nix world today is where a community of monolithic entry points. So almost every Nix expression in the world outside of Nix packages has this line at the top, and all of the useful, interesting functionality is pulled from there. Or maybe if you're a little bit more sophisticated, you'll have builtins.fetchkit, or your company will have its own Nix packages fork, or sometimes you'll have a company Nix repo, or if your company has a Mono repo, then it'll have a big Mono repo. But either way, this is really, for the most part, your Nix projects will all revolve around some big central repository of all Nix functionality. Why this is a problem, just about to get political, I'm going to make a stand on Mono repo versus Poly repo for our context in this world, not in general. Mono, our current workflow, especially Nix packages, but even outside of it, leads to very tightly coupled development. Everything within Nix packages is updated at once atomically, which makes sense because you want everything to work together, but it means the interfaces between our components are very blurred. If you make a standard end change, you can just change all of the packages that breaks in one commit, and then you never have to think about, was that a good interface to begin with, and is this change going to be usable for external users, etc. It's hard to compose in the sense of we have many different functionalities within Nix packages, and I can't easily say I want the Haskell builder from 1803, but within the package set of 1809, that's certainly not possible. But even the Haskell builder of a few weeks ago probably won't work, and it's also technically very difficult to actually plug it in. We also accumulate a bunch of responsibilities within Nix packages, so we do experiments, and we also have our stable package set, and we have a lot of different functionalities and different components, as I was saying about, and they all exist within one file system tree, one set of process for managing the system, for stable updates, for pull requests, for governance, all of that. Everything is kind of bound by the same system when we have this sort of mix of things. So, for example, I don't know if you guys have been following John Erickson's work on improving cross compilation in standard N, a big difficulty he's run into that I've seen is he has to kind of shove the work he's doing into the flow of Nix packages without breaking too many things and without causing too much disruption at once, because there's no good way for him to iterate on standard N independently of Nix packages. And kind of back to what Rock was talking about earlier in his talk, well, maybe he's yesterday, I can't even remember, where in the Nix world, we can get the benefits of a monorepo without actually having a monorepo by tying everything together at the top. In principle, we could do that same kind of thing with Nix packages, where Nix packages remains, where everything is integrated in together in the actual sort of trusted set of the world, but the individual components are built independently on their own time frame. So, this is the main thing that pushed me, but there are a few other things that I think can be solved with require.nix, which I also have not yet described. First, annotations, and by annotations I mean documentation, types, what are the flags, how do I, given some data, what's the metadata associated with that, whether that's a function definition or whether that's a package definition. We have some conventions for, I mean, especially for metadata, but for package metadata, but we don't have general conventions for this and we don't have discoverability around those annotations or discoverability for what functionality is even available. It's pretty hard to say, does this package exist unless you know enough about how Nix packages is laid out to already know most of the answer. You know, for, if you go through the attributes in the top level packages set, how do you know which of those is a package versus a sub package set, versus a set of library functionality, versus some combination of all of the above, we don't have that available to us within Nix packages very easily. And finally, open sourcing is actually very difficult, at least in my experience, because again we have these monolithic company Nix repos and they combine functionality that we would love to share that is completely generic, but it's also like all tied up with functionality like our private package sets and all of that. And if we open sourced it, we would have to break out that functionality and we wouldn't use the thing we open sourced because we have no way to combine it in with our big monorepo. So this is a problem that I kind of was looking at and sat down and did some design sessions and I think the answer, what we need in Nix, which we don't have, is packages and modules. And not packages and modules in the sense that Nix means them, but I mean in the sense of normal programming language packages and modules like Python has a module and package exports those modules and you import that and you get your definitions from that. And so required at Nix is a system to provide packages and modules for the Nix language. So there's some kind of core technical capabilities like a package specification saying, you know, this is the required at Nix package itself. It's got a description. It's got versions. These are the packages I depend on with version ranges and things like that. I can define my, what modules I export and kind of some formatting and just a heads up for all of these things. These are like the first thing I could think of that would cover exactly what I needed and nothing more. So I'm not saying this format is great. I'm saying we have a format. It gives metadata. It gives you everything you need to know to consume the package. Similarly, we could have lock files to say, okay, you depend on the base library. This, for this repo, use this version of base from this revision of GitHub. And, you know, just a note about how I've kind of built the lock files so far. They not only need to contain your direct dependencies, they need to contain your whole dependency set. But kind of going back to what Adam was talking about earlier, it still fails if you try to import a module not in your dependency list. So you don't have that problem that Node has, but you still have a top level kind of tying everything together. These are the specific versions I'm using in my package. And it also happens to allow for circular dependencies, but I don't think anybody's going to use that. And if they did, that's on their head, in my opinion. And the implementation I have written also allows for local lock files. It's expected you add them to your Gitignore so you can kind of override locally. And also we use these lock, we use a separate lock file for bootstrapping require.nix itself, which hopefully will go away when we get some like custom tooling. And then here's how we actually use it, and this is where the name comes from, inspired by Node.js' require. A module within the require.nix world is a function that takes this require function. And then you can call require to import a module. And if you ignore the module, if you omit the module here, it'll import like the top level module. So I'm importing the trivial module from base, and then I'm, and think of this as like the, which identifiers you're importing, it's just an inherit. But I think this kind of pattern will probably be common. So I'm saying, okay, get me the identity function from base, and then in my module is going to define identity functions specialized to lists. So that's really the core technical innovation of require.nix. But as I was saying, there's also, this is an opportunity to kind of improve on conventions around how we build our projects. If we're going to be building this new, if we're going to be building new libraries in this new system, we might as well take advantage of that opportunity to do things more standardized in a kind of more conventional way. So the first convention that I like, and I think this will probably be the most controversial of my conventions, is going for declarative or like plain old data domain specific types for everything we're representing. So here this is, this is an instance of a type I call source specification. So instead of doing everything in paths and doing like builtins.fetchget this, and then I'm going to get the subdirectory here, and then I'm going to filter that. You represent what your path is at a, in a declarative way, and then only at the very edges when you need to go to a nix built in or something else. I have a timer on my, I have the timer on my camera. If going to a nix built in or something else, that you actually need to convert it to a path. So you, you, you represent all of your, your data in the, in the domain that you want to think about it in. So there's, this is for sources. You could also have it for packages like a cabal package is different from a C++ package. And in today's world, what we do to express that difference is we have a different function we call. But if instead we just passed around like these are the, these are the fields that define my cabal package. These are the fields that define my C++ package. And only when I actually need to consume it in a derivation generic way do I call the function that converts it into a derivation. So it's sort of related to some of what Ilco is doing, whether the DRV field of the modules only, it's only consumed at the end, but you're operating over, you're operating over this kind of generic, this domain specific representation rather. So one of the, one of the reasons I like this kind of approach is it lets you sort of efficiently compose and, and, and efficiently and sanely compose or transform or query your data. So like in, in Nix packages, some of you may know there's this way to do composable filter source and it's sort of ad hoc and it's only for one thing. If you, if you actually just try to do filter source multiple times, you'll add to store multiple times. And actually it won't even work because you have to discard context and all of that. But even if you omit that part, you still are adding the path to the store over and over and over again. If, but in, in practice what you often have in some of these, these systems is you have a source, you have a, the representation of your project source and then you're going to call Kabbal to Nix. So I want a filter source that just has the Kabbal file in it, but I don't, but then I also want the top level source thing and then maybe I actually have a source with multiple projects in it and so I want to recurse into each one. And if, if each time I changed the path, I was re-adding it to the store, that'd be very inefficient. If I make all of, do all of my transformations on this representation, then only, and then only at the end when I actually needed to consume it, kind of close the loop back to a Nix primitive, then you get things are much more efficient and it's also much easier to look at a path and see what it's doing and what it means. So this is, this is for paths, but I think we should be doing it for everything. I think, you know, the, the attribute sets that we, so using make derivation and then using dot override, or maybe it's Kabbal dot make derivation and dot override for that instead and trying to kind of keep in mind those things instead of just having a standard of this is what the types look like. I think is, would really help kind of people use Nix packages, especially people who, and one of the things I really want to support is developers owning their own Nix expressions, even if they're not people into packaging and giving them a language that is as close to how they want to think about things as possible is a huge way to get there. And then also, this, again, going to what Ilko was talking about yesterday, this could eventually lead into tooling that lets you discover what are the flags for this package because you'd have like a simple flags field of like what it is and what they mean with metadata that you want to do it. And then you can say, okay, I want to install less with the secure flag. Great. I, my, my, my install tool knows how to parse out the flags from a package and call that in. So that's one thing. Another thing that I think everybody will be on board with is documentation. And here, documentation structured in a way. So the resolve function is from the source specifications library. And so the module, which is exporting resolve will also have a metadata field and you can have metadata like module level description metadata, but also annotations per whatever is exported there. So this is just a very, you know, the description here is intentionally long and detailed and the idea is you should be able to look at it and kind of understand what it's doing without needing to kind of jump, jump a lot. I mean, sometimes you're going to have to cross reference, but I think this documentation in terms of length is something we want to look at. I think having every identifier, having every exported top level identifier have its own description. This could have, you know, this could be Doc book or some kind of markup. I don't have strong opinions on that. I think we should just choose something and standardize it. And then of course, given if we're going to break things out into separate projects and separate libraries, the libraries themselves should have really good top level documentation of like just like you would have in any other project like this is what this library is for. This is how it's organized. This is how you should think about it. These are the top level concepts, etc. Next thing that I know a lot of people are, have been interested in is type annotations. So in addition to human readable descriptions, this is a machine readable description of the type of a function. And so I've come up with a way to encode kind of very generally encode types with as Nix values. So, you know, this is saying resolve is a function. The input type is a source specification and source specification is sort of defined in some other types module. And the output is a result type and result is like if you're familiar with Rust result type or it's an either where you've got an error condition or a success condition and say, okay, if this succeeded, then you will give back a path. If it failed, we'll give back some structured error type describing what the failure was. And then the structured error, the reason I gave this example is I also wanted to say structured errors as opposed to just throwing is often a better way to go because then you can pass information higher up the chain until the error message actually shows the context that the user is going to want. So this, the type definition I've got, the type language I came up with is rich enough to have like sums and record types. It also, as you can see, has function applications and function parameters. It's way too general for what we're actually want to do. But I just kind of, again, started with the basic thing that would work. So, and then the last thing that we can do is build tooling on top of all of this. So, again, all that documentation stuff and the types could be rendered in some nice way. We could have, you know, some hosted documentation for all of our libraries. We could expose configurations of tools as I mentioned earlier. If your package has some flags, then your install script could use those flags to do things. And we could build tooling that's sort of native to require.next. Right now, to like resolve that package description in the lock file that I was talking about earlier, right now to do that I have some like janky bootstrap code that reads it all in. And then a small subset of the stuff that require.next itself uses is implemented twice, once within require.next's repo. So it can read the file enough to fetch it, its real dependencies and then bootstrap itself. But if we had that, the only reason I do that is so that you can just write a default on Nix and use Nix instantiator Nix build like you normally do. But if we had required that Nix native tooling, it could do the job of parsing these package definitions and these lock files. It could be smarter about how it fetches the dependencies and shares them. And it can also give you a nice repel experience where for any given identifier you can easily find, okay, give me the doc string for this or show me the type of this. And all of that. So, yeah. So now I want to kind of go into, so this is the idea. This is where I want to go. And the big question is what's coming next? And really the reason what I don't have, I have stuff locally on my machine that works and it's proven the concept for me. The reason I haven't gone much further with it is this is not a viable project unless there's buy-in from the community. And I personally am not a sort of language packaging expert. I don't know what the right convention should be. I don't know what the right field should be. So I have a number of things that I would love for people to be looking at in the near future, possibly even the hackathon if you guys are here. And things that I also want to be looking at, but I'm mostly interested in following where there's actually community interest. So one set of things we can do is make some libraries. So we could have language specific libraries. We've got the Haskell infrastructure. We've got the Perl infrastructure. We've got the Rust infrastructure. They all have their own builders. Many of them have their own kind of ways to generate expressions. They have their own package sets. Maybe we could break those out. And then we can have a look at the opinions on how that should be done. I kind of disagree with the love for the Yarn to Nix approach. I think Cabal to Nix is actually closer to the one true path. But ask me later if you want to hear more about that. And just as a little teaser, this is something that we're using within Target. And yeah, these are probably proprietary names I shouldn't be sharing, but oh well. Where we have a single repo with a bunch of packages defined in a Cabal.project. And we just pass this list and a given tag for that repo and it just pulls them all in and we have them available in our package set. And this is something we would love to open source. This functionality is not target specific at all. But again, we don't have a nice Haskell library to plug it into and it would be kind of weird within Nix packages proper. Also just Nix packages library, all of the kind of like basic stuff that's in there could be broken out in particular kind of like the basic trivial stuff could easily be broken out into its own base module of just this is the kind of stuff you want to do if you're doing basic stuff with Nix. Standard end like the core concept of what is part of the standard environment and what is the interface to make derivation and all of those things could be separated out and iterated and experiments it upon. And Nix packages could do the actual bootstrapping and definition of what the standard end is for Nix packages now but you could come along and plug your own in or plug your own in just for some packages and kind of explore with that. And the Nix OS module system is another significant candidate for, you know, it's not really Nix packages specific, it's not really Nix OS specific as we learned earlier today. It's a complete standalone Nix language functionality that could be in its own place and be experimented on and developed on its own path. The other thing and this is I think where I need more help on is all those standardizations I talked about earlier, I don't know what they should be. I don't know what the right answers on these things. I really what I want to do is import best practices from other communities and get the people who are expert to care about these things to kind of help us figure out what to do. So what should the package format be? What should the lock format be? What should our conventions be? And in particular with conventions one thing I have completely not gotten to is tests and test suites. Like how do we want to represent those? Do those go in the package format? Are they some kind of separate thing? What kind of tooling would we need to run these tests? And all of that. And then of course the one last project I want to do is publish these slides and announce that this is happening. But again that depends on if people are interested and want to make it happen. So that's all I had. Thank you guys and we've got about 10 minutes it looks like for questions. So. So are there any questions or is everything already entirely clear and. Hi thanks for the talk. I have a couple of questions or I'll ask just one and maybe I'll raise my hand again. So this is all nice and it's a great idea. Pros and cons as you mentioned with Monoribo and libraries. But do you or have you thought about like a migration path because we all have the next packages as they are and we're using as they are and it's a lot of work that's been put into it. So you know and you want to don't want to break backwards compatibility and so on. Yeah so I mean I think for migration path one one thing that I've thought of to start with is start with this kind of. Functionality doesn't really make sense in next packages to begin with like it's not core to how you use Nick so S. But it maybe it's how you use Nixon development. So like this like this nice stuff we have for building Haskell package sets right. It doesn't really need to be a next package is next packages has its Haskell package set it's fine. So we could start with those libraries validate validate the concepts validate the tooling go with what works. The one thing I'm hesitant to say on that though is I'd rather just go with where people are interested and willing to do the implementation. And so I'm not going to say no to people who if there's interest in kind of jumping straight to standard M. In terms of migration like once we're ready to do that I think that's going to be kind of it's going to need a community wide consideration because it's kind of fun going to fundamentally change what happens when you import next packages and how it's all tied together. The one nice thing though is that from a on a surface level at least this will all be opaque to the user because the default next will still work. It'll just kind of import it'll have some library dependencies and in its package with the default next will handle the bootstrapping for you and pull it all together. So it's only once you get to the point of people want to override it or developers how they want to use it and I don't have good thoughts there on what exactly that path is again that depends on buy in and interest. Okay let's take a question from the Internet. Why do the lock files not contain the hash of the next file that is being imported. I mean in principle they could accept this first of all it's not a next file that's being imported its entire its entire package right. So let's go back to the where is it. So basically this this kind of gets back to the question of there's a trade off in usability versus kind of power I guess not really power but this this is a format that kind of I trust as at a high level. I trust that if I'm referencing a tag that it's referentially transparent in the meeting in a meaningful way like this. I trust the owner of this repo not to update the tag and so this gives a unique name. Now it's true that this is sort of cheating. This is this is not good enough for Nix but it's good enough for me. The question is if we force hashes if we force people to calculate all these things in practice I have genuinely seen people give up Nix over little issues like this like they have to update the hash every time. And so the question and I'm not I guess I'll say I'm not kind of wed to this completely but my my assumption my current gut feel is if we can define things at a domain specific level in a way that lets people that that that it that we the concept of purity makes sense for that domain. We'll get a huge win on usability and the loss on purity will be slim to none. One more question. How much time of any Nix command depends on fetching requires from the network. I mean so basically the way the way the way it currently works is they just kind of fetches everything in the lock up front. So you know it's I guess that that's a a question. So we're I'm currently using the fetch get from the targets Nix fetches repo which has some pretty good caching. So once you've already fetched it once it's actually it's actually pretty fast to kind of validate that. Oh it's already there and I'm fine. Nix the built-in fetch get also does that if you provide a revision but since we're just writing tags that sometimes has to refetch. But so it's not too long. I haven't actually timed the whole thing and I but I do think this is a place where standalone require dot Nix native tooling would help because we could do smart things about fetching and knowing when it needs to fetch and and how it all handles that outside of a Nix evaluation. Yeah just a small point on what was just discussed in terms of putting hashes there for tags for example so I've seen people push tags to a different version. So I would be a big proponent of saying OK do put the shots in there but I think what would solve the problem much more nicely is that if you have something like this where you have a thing where you say OK I understand what a what did you call it source. Source specification is right just put a little bit of tooling into the next tool that says your hash is different. Press yes to update right and it writes your thing it's much easier and I think it gets us both. So but the other. So the other thing is what I will say here is this kind of format also has a mode for specifying a branch and a get revision. And so as long as you trust gets revisions you can just use those. It still doesn't require a hash of the file system contents though. But yeah we could also we could do that. The discussions are already starting was yes let's do that. No let's not let's not. Keep it going. It's good to see someone else thinking about these sort of workflows because I have a couple of similar things where I want to use certain library functions out of Nix packages but not the actual packages themselves. And in my case I've actually found what works really well is at the top to have a let binding that says something like let licenses equals built in stuff fetched tarball Nix packages by shot two fifty six dot lib dot licenses in which case I fetched a small sub component of Nix packages in a deterministic way from the monorepo creating in effect my own library without busting up the monorepo and introducing version dependencies for upstream testing and discoverability issues. Have you tried this approach and is it working. Yeah I mean so the one thing that it doesn't support is when I want to push when I like I if I want to push changes that I in my opinion don't really make sense in Nix packages except for the fact that Nix packages is kind of the central place doesn't really help that and it doesn't help with a couple deployment development aspect of the monorepo actually causes problems. And one thing I actually wanted to say that I didn't mention at the time is of course all of these things I'm talking about are about kind of practice and policy and in principle you can implement policies that don't have these problems within a monorepo but I do believe in my experience has been that the monorepo that we have in the policy practices we follow encourage this kind of lack of clean interfaces between components and that sort of thing. Any more questions from someone who hasn't already asked a question. We're at two minutes. It's more of a statement. I think this is a brilliant idea and something that hasn't been mentioned is that if it gets adopted packages could provide their required at Nix file and then as Nix packages contributor we could just import the derivation directly and we don't have to do the packaging. Yeah exactly. You could you could provide a required at Nix file within your package or a package set that like gives a list of overlays so it could either import its own Nix packages or it could give an overlay list that you can apply to your own Nix packages and that sort of thing. Any more sentences with question marks. So another similar point that I've seen a handful of people in the community struggle with is if you have a package that you don't intend to upstream to Nix packages. There's some duplication between the default dot Nix in the repository of the package the shell dot Nix the release dot Nix and potentially your company's monorepo or potentially Nix packages monorepo. Have you thought a little bit about solving this particular problem or do you have any. I haven't. I mean I have thought about that problem. I don't know that this is a full solution to that but I do think having more abilities to kind of reference standalone libraries you'd have one place where the actual expression lives. Have you also worked on a say practical example in which you can demonstrate what value this modellization approach provides because I think I also agree with you we need we need way better modularization but I think it would help a lot. If there's something you can show something really practical for which it makes sense. Have you thought about that. I mean yeah so I've thought about that and that's to me that's the next step to see it wasn't clear to me before I'm talking about it this weekend is whether anybody else was on board. Now it seems like it is so yes my next step is polish up the Haskell the sort of library of like nice Haskell functionality that we're using we're using some of it within targets some of what I've used at previous companies and just kind of make it a nice clean standalone piece of functionality. That's that's my next project unless again there's some big push for some other thing that I should help out with. Any. Oh we're at not negative 20 seconds anyway. Are you are you up for answering one more question. Yeah absolutely. Then let's go for it. Talk all night. Yeah so I mean I think this sounds like a really great initiative. One question I have around says I think I agree it's generally. Too difficult to get as a metadata of stuff and says messes up the user experience and also the tooling like for example Nick's and query also basically was a clusterfuck and Nick's sort of improves it but it needs a caching mechanism. And it seems to me a lot of those difficulties could be ameliorated if we just use JSON or whatever to specify the non well the actual metadata stuff like the description as a shas because you don't actually want to compute some and having some as some static information. So normal tooling can query and update automatically also I think would improve a lot so one question I have is how bad it is idea of doing stuff is Nick's expression. I suppose so did you did you consider that you consider doing something like your own. I meant to mention you'll notice both of these are completely representable as Jason that's not an accident. So yes I'm with you possibly I'm open to the idea of saying the metadata for the packages themselves should be Jason so easier make tooling. Or just use hNix or RNix or the Nix evaluator and C++ and import it that way. Anything else. Cool. OK already. Thank you. Thank you.
the next core team that was created not too long work at Target, been working with Nix for quite a in professional settings for managing development for the most part, your Nix projects will all us within Nix packages very easily. And finally, required at Nix is a system to provide packages and modules for the Nix language. So there's some packages I depend on with version ranges and from, inspired by Node.js' require. A module module is going to define identity functions the timer on my camera. If going to a nix built all of your, your data in the, in the domain that reference, but I think this documentation in you can see, has function applications and project unless there's buy-in from the community. part of the standard environment and what is the more help on is all those standardizations I be? And in particular with conventions one thing the other thing is what I will say here is this of clean interfaces between components and that