Buil outside of the (sand)box.
Maudoux, Guillaume (layus)
This talk will present experiments at building packages incrementally by relying on an external cache outside the nix sandbox. We will show how tools such as distcc and bazel can benefit from impure information during the build, and discuss how this impacts purity and reproducibility.
 So, the last talk of this year's Nixcon is going to be about building outside of the sandbox. Bye, Laos. Like this one is working? Perfect. Okay, so hello everyone. I have this strange pleasure to be the last one to talk at this conference, but there is the old that's precisely made for you to think a bit. I have no real message for you or rather would love to challenge you and think about like what we do a bit differently. So I am known as Laos. I am currently doing a PhD on build systems. We can have a chat about that later on. And this is an experiment that comes from some presentation that I made two years ago. So it will go in very simply with three different examples. I want to build outside of the Nix sandbox and I will use different tools to do that like CKH, SCKH, RCC. You will get explanations of that. And then we have a small discussion about what does it mean, where do we go and final conclusion. So two years ago I was here precisely in front of you and I had this video. We didn't work at the time, but I think it's fixed now. Like yeah. So this video basically shows what happens when you are trying to build a long package like Firefox. For me it's often Firefox, but possibly you are building GTK or anything like that. And then you reach the top, nearly all of the compilation is done and then for some reason at some point, well, there is a fix up thing that doesn't work and bam, you are done. You are just back at the start. And if you want to fix the issue, then you have to pay all of the compilation of the full package. Well, so what's the situation now, two years after that? Well, we cannot build the next package incrementally of course and that's because of the sandbox. So it means that for Firefox, GTK, Libreface, QTE, GCC, well GCC is specific, all of these things you have to build them from scratch every time you make a change to derivation. Possibly that's not something that happens often for you, but if you try to hack into the next package, especially if you want to refactor the standard environment, something that you should never do, but if you work on that, then you will pay that you have to build everything. So my first attempt to escape this sandbox is CK. CK, I guess everyone knows that, no need to explain. It's quite an old tool, but it's still heavily used in some places. And the basic idea with all of these cases is that, well, you already asked me to compile this file with these options and with these headers, so I already know the answer. I can just fetch it from the cache. A small picture, right? So just query the cache and the cache says, well, no, no, it's not there yet. Okay, so you can compile locally and then you upload it to the cache for the next iteration and then you get your results, right? Which means that on the next time, when there is a cache hit, you can just recover the result from the cache and that's it. It should be quite fast. Okay. The funny thing is that I say, well, I want to try that and then I discover that it already exists. It's already feasible in these packages. There is a CKH STDN. That's a standard environment modified that allows you to use CKH. There is a lot of wrapping going on there. We can have a small chat about what that means, but basically if you replace the standard environment with that one, you get CKH working. Well, you get CKH working, but you still have the sandbox, right? So we'll see how to get out of that. But in Nix OS, that's also already provided and then it comes with all the defaults that you need to change into Nix, meaning that you need to build by allowing an extra folder, the cache, to be mounted inside the sandbox. And Nix OS, we do that for you. If you are just using Nix, then you have to use options to do that. They made it in a, I have no idea what they did. That's why the way I should check, but they made it in a very smart way in a sense that you can specify a list of packages that will have this CKH enabled so that you're not breaking the sandbox for everything, but only for these very heavy packages that you will compile often. So if in your company you have one big package like that, you can add it to the list. Okay, let's try the demo then. So the demo will work. I've taken i3 just because it's an average size C++ project. And we will build that normally and then we see CKH to see what it gives. Okay, let's go. Yeah. Okay, so if I want to build i3 normally, I just need to build this i3 stuff and we will try to get that into, we will try to filter the outputs. So it's been modified a bit. Just checking the delivery. So it's been modified a bit because I've added some kind of debugging to it so you can just see the time that each phase takes. So we get this unpack phase, well, that's way below the second, the patch phase way below the second too. I expect something about 10 seconds to build. So you have 8 seconds to configure and 10 seconds to build i3. That's with the sandbox, that's everything's normal except of this extra logging at it. Now, if we start with another tool, we can next build a i3 CKH. Yeah, CKH. If I do it like that, then the sandbox is still active so it won't change anything, right? So you need to add an option. That's an extra sandbox path. With that, I can build and mount the CKH inside the sandbox. Well I don't want to do that at the moment because this cache, we will try to clean it before. So just to be sure that there is nothing in there on the first try. We can start compiling it. And of course I'm still missing that. I'm not sure if you have an idea of what's to be expected but in this case, we are, there is nothing in the cache. So we need to compile everything so we'll have to take the same time. And also we have to pay an extra overhead for serializing and inserting everything into the folder. So this should be a bit slower actually. So now it's 10 seconds and 15 so it matches what we had before. We are from 8 to 10 and from 10 to 15 just because we paid the overhead of initializing everything. So if you want to start with an, we need to compile it again. So I made a special trick for you that you can just say again to the package and well I can't explain you how it works but it's nothing very magic. The nice part is that it starts the exact same compilation with some random stuff added to the derivation so that's the next thing that needs to be done again. And this time it's supposed to be faster. Modulo demo effects. Well, okay, that's not exactly what I used to have but it makes some sense. So what's not, what's, okay, right. So we have all of these. If you don't mind I will take the old results that I had. They are basically the same, 7 and 6, right? That's what we had, 7 and 6. So it's like the rounded numbers are the same. So well nothing unexpected here except that's, well you see that there is this 6 seconds that's below the previous build times and in a sense I3 is not a tool that uses heavy GCC compilations. There are a lot of manual generation documentation generating a lot of random stuff that are not cached by C cache. So of course on a bigger project that uses more C compiling you would have a better speed up here but possibly we will also have a higher penalty on the first time that you use it. Makes sense. Well, is this a good thing? Yeah, the problem is that when you start to make holes in the sandbox like that, well you know you don't have any guarantee of what's happening, right? A small hole in the sandbox and that's over. But at the same time, and we'll have a discussion at the end, and I think it can be a good thing at least in some well-defined situations. Anyway, let's go to the second attempt. So like I wanted to use, well, C cache just good but no people do better things at that. What if I want to use the same cache as my colleague or someone else? We should put that on the network and that's basically the idea of SCC which is shared or shared C cache developed by Mozilla. So it's Mozilla so they made it for C and C++ and then they also added the rest because well, it's Mozilla. And if you look at the diagram, it doesn't change much. Just like basically it's exactly the same, the only change is the name of the cache. It's not a folder, now it's a network machine. Which means that you are not mounting a folder inside the sandbox. You are making a completely different type of hole in there, you need network access. Well is that possible? No. No, it's not possible, it's not supported. You cannot make a small hole and say, okay, I want this machine, I want this port and that server, it doesn't work. So at the moment and for the experiments, what I do is just disable the sandbox. So we are entering strange countries but technically it should be feasible to have like a restricted network access in the sandbox if you want to keep some of the properties but not all of them. Yeah, also discovered that the sandbox is not, disabling the sandbox is not possible when you are building in different stores than your main stores. So all of these demos are just changing my main stores, possibly it breaks things but we'll see. Yeah, okay, disclosure here. I won't do it because for reasons that we don't have much time and also because it's not that different, just disable the sandbox, configure the remote and that's it. So I will go directly to the third attempt which is RECC and RECC, we keep the same ID except that we have the remote execution. That's a tool that has been developed at Bloomberg and they do it mainly because well, they love Bazel, they would like to have all these remote execution and things but they can't move at the moment like migrating all of your builds to Bazel, that's a huge investment so they don't want to do that and they develop this very small replacement that's instead of changing all of the build system, you just wrap each GCC invocation with it and then at that moment everything is sent to the server. Well, you get the better explanation with diagram but what we do in this case is that we remove the local compilation and everything is remote. The main advantage is that you can parallelize a lot more. If you have like 100 workers and a huge workload then you can spread the workload on 100 workers. That's the main advantage of not executing the compilation locally of course. So this is the example that may not work perfectly. I discovered that you need of course a server that allows this remote execution. So in this case it's build grid. I could have used another one but I discovered different issues. Not sure if it's NICs related or my VM related or something but I cannot work in parallel so I can only compile one thing at a time because the server won't take many items at the same time which is a bit stupid. It's not a limitation of the tool. It's used perfectly well in a lot of companies so it's most probably an issue with my setup but anyway. So it's single-threaded and then the worker is quite slow. So if I had to do the demo for you, well that's the time it takes. It takes about three to four minutes to compile now. Well there is definitely some issues. Nevertheless I think this is mostly like bad configuration from my part but I really want to show you what kind of strange things you can discover when you do that. So let's start again. So if we... This is completely stupid. Like it's a build grid server but it's called build form. Well we have to leave with that. Now on this server everything is already compiled so we will remove what's in there. I removed the content that I stored on the server. I removed the database so there is just basically a clean server anyway. And I start the server. I also need to start a worker of course. Yeah that's initialization. And then because I'm lazy I did not specify any kind of docker, worker, any environment. I just like use whatever is on this machine. Well that's it. And now can we make this smaller? Okay right. So what we want to do now is build exactly the same thing as we did before. Yes. So it's Nix build. Okay we want this i3 RCC. I will take the RCC small whatever. This one should be built already. Let's try. Option sandbox false. This is what allows you to access the network. Yeah it's already built. That's good. And the next is already built I guess. Yep. I need to go over and over again. Yeah okay that's working. Very useful when you have to build other things like that. And well no I won't say you. Let's see what we get. It fails. It fails and that's normal. That's what you get on the first attempt. If you're like there is nothing on this remote host you need all of the build inputs. All of the dependencies of your derivation needs to be there in a way. And this is of course not automated. It doesn't work at the moment but then I think there is some potential to make integration and to make Nix catch the network access and wrap it in some way or we should be able to find a way to make this work. At the moment what we have to do is like upload the inputs. So oh crap. So I just try to collect all of the build inputs of my derivation and then I send that to the server. This is a trick. There is always a trick. The network is not that good that I can upload everything but I just removed one of the missing files and this is the one that has just been updated. So if we start again and build this again, again, again thing what do we expect? Does it work now? Well it doesn't work. The tool is still missing. I was at possible we just added the tool that was missing. And in this case it's like one of the most frustrating parts of this is that well the failure is already in the cache. So we are just fetching that. It says previous time it fails. So I can fail again. If you want to work around that then you need to kill everything. In this case I have no idea of removing one entry so just let's be efficient. Okay, everything started clean from scratch so now I can build it. And well of course it takes like three minutes so we won't wait for the end of that. But this time it should work. Let's wait. By the way this VCC dash small is configured in a way that's the cache. VCC is not used in the configure phase because the configure phase just drops a lot of the outputs and so you don't see the error that we had before. That's the reason for the dash small. So it's working. It's compiling one thing at a time. Well it will compile for three minutes. I don't really need it because I have everything in there so let's drop that. Okay. It makes sense. So like you've discovered we need to have all the inputs on the server. It does not exist at the moment but the fact that it failed is also a good thing. It shows that if you are missing the exact compiler that's required by that version on the worker it will say I don't have it. It's not trying to use another thing. It's not trying to use GCC that's installed on that machine. So Nix by design with all of these strange paths and ashes makes it so that you do not pick random stuff. We've discovered that like these impure errors are cached. The fact that there was something missing in the state of the worker like this is part of this is there forever. That's really annoying. So well that's something to care about. But I think we could really automate that. And well the gain is potentially really huge because in Nix if we can have the real exact set of inputs we could share this information through everyone that would be like an either cache for very small C file builds. Well now there is a question of like well this is not pure right. We've broken everything. The balloon has exploded. The sandbox is no more. What remains of the all the good properties that we had with that sandbox. And so like well is it still reproducible? Is it still pure? Is it still hermetic? And all of these nice words that we are used to and that we absolutely want in our builds. And for that I have a small answer that's based on Nix experimental. What I call Nix experimental is like it's an experimental branch from Elko where you can transform a derivation into, that's difficult to say. Well the problem with most of our derivations is that we have plenty of them that look the same, they have the same content but because they come from a different.dv they are stored in a different store path and then that store path is inside the binaries, it's inside a lot of places and if you compare the bits are different. But we can do something that's funny. So let's find the i3. So this is the one that we built before so it's there. If you go to Nix, of course it's a local checkout of Nix and another one because I had to build a different branch. Okay, let's hope that it works correctly. Yeah, you have a lot of outputs in this case. See if I reduce the size, that's still readable for everyone. Well in this case you have a lot of random stuff, let's just cheat a bit. There is a mismatch between the D-man and the Nix that's running and they don't agree on what we are trying to do but you see here that this is the path that we received and this is what it looks like when it's been content addressable. So basically all the closure has been rewritten and the name of the path has been changed so that it matches the actual content of the path. That's the meaning of content addressable. Of course there is a problem because the path, they reference themselves and you need to compute the hash on something and you would never hand. So basically you just remove self-references, compute the hash and then reinsert them and so it's not perfect content addressable, so it's like content address module reference. What's interesting in there is that if everything is working correctly, this I3C cache, the thing that I built, it's still cached, okay, it has a different hash but if I make it content addressable, then you see that you get the exact same hash here and this means, and we can check that but it doesn't make any sense because it's the exact same store location but if you try to diff these two things, you will see that there is no difference. Well, that's obvious but you can try if you want. And of course the one that remains to be tested is this RCC, yeah, small, whatever. And that one should also give us the same output. Yes, that's working. Well, in this case you may not believe me because it was already in the cache before but trust me, you don't want to wait five minutes just for that to build. So that's kind of part and answer. We have broken everything. There is no sandbox at all and we still manage to get the exact same output, right? So this sandbox is not like, it's not totally needed. It's there to guard you against making mistakes. It avoids typical issues. It avoids the network if something is trying and you don't want it to try but if you configure everything properly, then it should build the same outside of the sandbox or inside the sandbox. And in this case, you have different vision, different contents before and then this is the image that was there if there was some issue with the demo but I guess you got the idea. So what we have now, it's a bit like before but well, you know, you compile Firefox, it takes a lot of time, you see that it's making progress, you've done most of the rest and then C++ is over and you will enter the fix up phase. And then, oh, it doesn't work. But then you are not back from scratch, right? You can recover some of the work that has already been done. So that's better. So that's a few messages. Well, this presentation is not about sending messages. It's more like challenging so I would like to get feedback if it gives you an idea that's perfect but like the main message is that you can bridge a sandbox and it does not mean that you are doing something wrong but well, that you should know what you are doing mostly. It's really nice. And when I see that, I'm like, yeah, it's going to work. We really get these incremental builds inside Nix packages and that's something that I really want. I think it will also be really useful to get fast CI answers. So if you want to make a pull request to Nix packages, then you could get the answer within like a few minutes and not a few hours. And also, like, it also means that we can trust other tools. Like RCC is doing its best to make it reproducible, hermetic and all of these nice properties. C cache is doing its best but it's very old so it doesn't know everything. And then these tools, they are just C compiler abstractions but they behave exactly the same as RCC behaves exactly as Bazel, for example. So it's just that RCC manages only one C compile but Bazel manages a set of files. From the outside, meaning from the Nix point of view, it's exactly the same. It's a tool that's trying to execute something remotely and get the results. So if we go that way, we could consider trusting Bazel, for example, for doing things purely and correctly and allow it to have access to a cache and a remote execution engine somewhere. So that's it for me. I have questions for you if you don't have any. So think about it. What about verifying some compiler cache instruments for reproducibility so that they can be really trusted and not just relying on limited amount of experiments? Well, I guess that's always the same thing with reproducibility that you have to test it. And every time it's reproducible, that's good. But it's not a proof that it will never be reproducible. So we can do that with these tools too. We can just start to include them and then compile, compile a few times and then once in a while just make a perfectly aromatic sandbox build from scratch and see if we are still building the same thing. And that would give us more confidence that the tool is working correctly. I mean, maybe the problem is that when these tools had been designed, there was no thought about such thing as reproducibility. And it might depend on different things like system clock, for example. Well, truth is I'm not aware of all the details of all of these tools. So that's something that we have to explore. It's like, look, it's feasible. Does someone want to work with me on that? So one idea came to my mind when you showed mounting the file inside. Did you try to mount like a unique socket, maybe, I don't know, Ccash and SCcash very well, but maybe you can try networking that way? I tried and then I had no more time. So I think that theoretically it could work, but I don't know enough about all the technology details. On the outside looks trivial and then when you try to set up everything with the namespace and stuff, well, it breaks easily. So that's something we can try tomorrow. If you think that it can be done in a few hours, then I would really like to try. Cheers. Hey, first I wanted to say I really appreciate because I've been many times in the situation where I wanted to package something large like Ceph, for example, we're also iterating on bugs, building Ceph takes one hour. I had to build it over 80 times so you can imagine that took a while. So I really appreciate any effort going in that direction. Same thing with Haskell, which when you work on GHC, for example, or iterate on a low-level library and rebuild NICs packages can also take a very long time. So I think this is really good. I wanted to also point out another approach that is a little bit orthogonal to what you've presented that we just discussed yesterday here at the table with, I think, Edef Pi and me, so NH2, which was that you can also do the type of caching that Ccache, for example, does specifically for like C things on the syscall level. So we could, if we wanted to, P trace the entire build and then see that if, let's say, an execve produces a certain output, we can cache that and then wrap any execve call that happens again to produce exactly the same output again. So that is, of course, somewhat in the further future, but that might be something that can be very generic across multiple things. I think practically I will first go with exactly this approach because you have now shown that it works and it's great. And I just need to see how one can get the same thing for GHC Haskell, for example. But in the long term, that might be something that if we agree on enough, bringing enough hackiness into this thing as a kind of like quick developer thing we might do as well. Well, it's not really a question, but I have still two answers for you. Well, it's funny because I had the exact same discussion with NBP like last year and the year before, like, well, we could do that on the syscall level. That would be fun. Through this, it's not that easy. You have lots of cache entries and et cetera, so you need to be really, really efficient. It's a cool idea, but I'm not sure it will turn out as practiced, but I'm always available to talk about that idea. Thank you. Oh. Thank you. Thank you. Thank you. Thank you.
 So, the last talk of this year's Nixcon is going to be about building outside of the  sandbox.  Bye, Laos.  Like this one is working?  Perfect.  Okay, so hello everyone.  I have this strange pleasure to be the last one to talk at this conference, but there is  the old that's precisely made for you to think a bit.  I have no real message for you or rather would love to challenge you and think about  like what we do a bit differently.  So I am known as Laos.  I am currently doing a PhD on build systems.  We can have a chat about that later on.  And this is an experiment that comes from some presentation that I made two years ago.  So it will go in very simply with three different examples.  I want to build outside of the Nix sandbox and I will use different tools to do that  like CKH, SCKH, RCC.  You will get explanations of that.  And then we have a small discussion about what does it mean, where do we go and final  conclusion.  So two years ago I was here precisely in front of you and I had this video.  We didn't work at the time, but I think it's fixed now.  Like yeah.  So this video basically shows what happens when you are trying to build a long package  like Firefox.  For me it's often Firefox, but possibly you are building GTK or anything like that.  And then you reach the top, nearly all of the compilation is done and then for some  reason at some point, well, there is a fix up thing that doesn't work and bam, you are  done.  You are just back at the start.  And if you want to fix the issue, then you have to pay all of the compilation of the  full package.  Well, so what's the situation now, two years after that?  Well, we cannot build the next package incrementally of course and that's because of the sandbox.  So it means that for Firefox, GTK, Libreface, QTE, GCC, well GCC is specific, all of these  things you have to build them from scratch every time you make a change to derivation.  Possibly that's not something that happens often for you, but if you try to hack into  the next package, especially if you want to refactor the standard environment, something  that you should never do, but if you work on that, then you will pay that you have to  build everything.  So my first attempt to escape this sandbox is CK.  CK, I guess everyone knows that, no need to explain.  It's quite an old tool, but it's still heavily used in some places.  And the basic idea with all of these cases is that, well, you already asked me to compile  this file with these options and with these headers, so I already know the answer.  I can just fetch it from the cache.  A small picture, right?  So just query the cache and the cache says, well, no, no, it's not there yet.  Okay, so you can compile locally and then you upload it to the cache for the next iteration  and then you get your results, right?  Which means that on the next time, when there is a cache hit, you can just recover the result  from the cache and that's it.  It should be quite fast.  Okay.  The funny thing is that I say, well, I want to try that and then I discover that it already  exists.  It's already feasible in these packages.  There is a CKH STDN.  That's a standard environment modified that allows you to use CKH.  There is a lot of wrapping going on there.  We can have a small chat about what that means, but basically if you replace the standard environment  with that one, you get CKH working.  Well, you get CKH working, but you still have the sandbox, right?  So we'll see how to get out of that.  But in Nix OS, that's also already provided and then it comes with all the defaults that  you need to change into Nix, meaning that you need to build by allowing an extra folder,  the cache, to be mounted inside the sandbox.  And Nix OS, we do that for you.  If you are just using Nix, then you have to use options to do that.  They made it in a, I have no idea what they did.  That's why the way I should check, but they made it in a very smart way in a sense that  you can specify a list of packages that will have this CKH enabled so that you're not breaking  the sandbox for everything, but only for these very heavy packages that you will compile  often.  So if in your company you have one big package like that, you can add it to the list.  Okay, let's try the demo then.  So the demo will work.  I've taken i3 just because it's an average size C++ project.  And we will build that normally and then we see CKH to see what it gives.  Okay, let's go.  Yeah.  Okay, so if I want to build i3 normally, I just need to build this i3 stuff and we will  try to get that into, we will try to filter the outputs.  So it's been modified a bit.  Just checking the delivery.  So it's been modified a bit because I've added some kind of debugging to it so you can just  see the time that each phase takes.  So we get this unpack phase, well, that's way below the second, the patch phase way below  the second too.  I expect something about 10 seconds to build.  So you have 8 seconds to configure and 10 seconds to build i3.  That's with the sandbox, that's everything's normal except of this extra logging at it.  Now, if we start with another tool, we can next build a i3 CKH.  Yeah, CKH.  If I do it like that, then the sandbox is still active so it won't change anything, right?  So you need to add an option.  That's an extra sandbox path.  With that, I can build and mount the CKH inside the sandbox.  Well I don't want to do that at the moment because this cache, we will try to clean it  before.  So just to be sure that there is nothing in there on the first try.  We can start compiling it.  And of course I'm still missing that.  I'm not sure if you have an idea of what's to be expected but in this case, we are, there  is nothing in the cache.  So we need to compile everything so we'll have to take the same time.  And also we have to pay an extra overhead for serializing and inserting everything into  the folder.  So this should be a bit slower actually.  So now it's 10 seconds and 15 so it matches what we had before.  We are from 8 to 10 and from 10 to 15 just because we paid the overhead of initializing  everything.  So if you want to start with an, we need to compile it again.  So I made a special trick for you that you can just say again to the package and well  I can't explain you how it works but it's nothing very magic.  The nice part is that it starts the exact same compilation with some random stuff added  to the derivation so that's the next thing that needs to be done again.  And this time it's supposed to be faster.  Modulo demo effects.  Well, okay, that's not exactly what I used to have but it makes some sense.  So what's not, what's, okay, right.  So we have all of these.  If you don't mind I will take the old results that I had.  They are basically the same, 7 and 6, right?  That's what we had, 7 and 6.  So it's like the rounded numbers are the same.  So well nothing unexpected here except that's, well you see that there is this 6 seconds  that's below the previous build times and in a sense I3 is not a tool that uses heavy  GCC compilations.  There are a lot of manual generation documentation generating a lot of random stuff that are not  cached by C cache.  So of course on a bigger project that uses more C compiling you would have a better speed  up here but possibly we will also have a higher penalty on the first time that you use it.  Makes sense.  Well, is this a good thing?  Yeah, the problem is that when you start to make holes in the sandbox like that, well  you know you don't have any guarantee of what's happening, right?  A small hole in the sandbox and that's over.  But at the same time, and we'll have a discussion at the end, and I think it can be a good thing  at least in some well-defined situations.  Anyway, let's go to the second attempt.  So like I wanted to use, well, C cache just good but no people do better things at that.  What if I want to use the same cache as my colleague or someone else?  We should put that on the network and that's basically the idea of SCC which is shared  or shared C cache developed by Mozilla.  So it's Mozilla so they made it for C and C++ and then they also added the rest because  well, it's Mozilla.  And if you look at the diagram, it doesn't change much.  Just like basically it's exactly the same, the only change is the name of the cache.  It's not a folder, now it's a network machine.  Which means that you are not mounting a folder inside the sandbox.  You are making a completely different type of hole in there, you need network access.  Well is that possible?  No.  No, it's not possible, it's not supported.  You cannot make a small hole and say, okay, I want this machine, I want this port and  that server, it doesn't work.  So at the moment and for the experiments, what I do is just disable the sandbox.  So we are entering strange countries but technically it should be feasible to have like a restricted  network access in the sandbox if you want to keep some of the properties but not all  of them.  Yeah, also discovered that the sandbox is not, disabling the sandbox is not possible  when you are building in different stores than your main stores.  So all of these demos are just changing my main stores, possibly it breaks things but  we'll see.  Yeah, okay, disclosure here.  I won't do it because for reasons that we don't have much time and also because it's  not that different, just disable the sandbox, configure the remote and that's it.  So I will go directly to the third attempt which is RECC and RECC, we keep the same ID  except that we have the remote execution.  That's a tool that has been developed at Bloomberg and they do it mainly because well, they love  Bazel, they would like to have all these remote execution and things but they can't move at  the moment like migrating all of your builds to Bazel, that's a huge investment so they  don't want to do that and they develop this very small replacement that's instead of changing  all of the build system, you just wrap each GCC invocation with it and then at that moment  everything is sent to the server.  Well, you get the better explanation with diagram but what we do in this case is that  we remove the local compilation and everything is remote.  The main advantage is that you can parallelize a lot more.  If you have like 100 workers and a huge workload then you can spread the workload on 100 workers.  That's the main advantage of not executing the compilation locally of course.  So this is the example that may not work perfectly.  I discovered that you need of course a server that allows this remote execution.  So in this case it's build grid.  I could have used another one but I discovered different issues.  Not sure if it's NICs related or my VM related or something but I cannot work in parallel  so I can only compile one thing at a time because the server won't take many items at  the same time which is a bit stupid.  It's not a limitation of the tool.  It's used perfectly well in a lot of companies so it's most probably an issue with my setup  but anyway.  So it's single-threaded and then the worker is quite slow.  So if I had to do the demo for you, well that's the time it takes.  It takes about three to four minutes to compile now.  Well there is definitely some issues.  Nevertheless I think this is mostly like bad configuration from my part but I really want  to show you what kind of strange things you can discover when you do that.  So let's start again.  So if we...  This is completely stupid.  Like it's a build grid server but it's called build form.  Well we have to leave with that.  Now on this server everything is already compiled so we will remove what's in there.  I removed the content that I stored on the server.  I removed the database so there is just basically a clean server anyway.  And I start the server.  I also need to start a worker of course.  Yeah that's initialization.  And then because I'm lazy I did not specify any kind of docker, worker, any environment.  I just like use whatever is on this machine.  Well that's it.  And now can we make this smaller?  Okay right.  So what we want to do now is build exactly the same thing as we did before.  Yes.  So it's Nix build.  Okay we want this i3 RCC.  I will take the RCC small whatever.  This one should be built already.  Let's try.  Option sandbox false.  This is what allows you to access the network.  Yeah it's already built.  That's good.  And the next is already built I guess.  Yep.  I need to go over and over again.  Yeah okay that's working.  Very useful when you have to build other things like that.  And well no I won't say you.  Let's see what we get.  It fails.  It fails and that's normal.  That's what you get on the first attempt.  If you're like there is nothing on this remote host you need all of the build inputs.  All of the dependencies of your derivation needs to be there in a way.  And this is of course not automated.  It doesn't work at the moment but then I think there is some potential to make integration  and to make Nix catch the network access and wrap it in some way or we should be able to  find a way to make this work.  At the moment what we have to do is like upload the inputs.  So oh crap.  So I just try to collect all of the build inputs of my derivation and then I send that  to the server.  This is a trick.  There is always a trick.  The network is not that good that I can upload everything but I just removed one of the missing  files and this is the one that has just been updated.  So if we start again and build this again, again, again thing what do we expect?  Does it work now?  Well it doesn't work.  The tool is still missing.  I was at possible we just added the tool that was missing.  And in this case it's like one of the most frustrating parts of this is that well the  failure is already in the cache.  So we are just fetching that.  It says previous time it fails.  So I can fail again.  If you want to work around that then you need to kill everything.  In this case I have no idea of removing one entry so just let's be efficient.  Okay, everything started clean from scratch so now I can build it.  And well of course it takes like three minutes so we won't wait for the end of that.  But this time it should work.  Let's wait.  By the way this VCC dash small is configured in a way that's the cache.  VCC is not used in the configure phase because the configure phase just drops a lot of the  outputs and so you don't see the error that we had before.  That's the reason for the dash small.  So it's working.  It's compiling one thing at a time.  Well it will compile for three minutes.  I don't really need it because I have everything in there so let's drop that.  Okay.  It makes sense.  So like you've discovered we need to have all the inputs on the server.  It does not exist at the moment but the fact that it failed is also a good thing.  It shows that if you are missing the exact compiler that's required by that version on  the worker it will say I don't have it.  It's not trying to use another thing.  It's not trying to use GCC that's installed on that machine.  So Nix by design with all of these strange paths and ashes makes it so that you do not  pick random stuff.  We've discovered that like these impure errors are cached.  The fact that there was something missing in the state of the worker like this is part  of this is there forever.  That's really annoying.  So well that's something to care about.  But I think we could really automate that.  And well the gain is potentially really huge because in Nix if we can have the real exact  set of inputs we could share this information through everyone that would be like an either  cache for very small C file builds.  Well now there is a question of like well this is not pure right.  We've broken everything.  The balloon has exploded.  The sandbox is no more.  What remains of the all the good properties that we had with that sandbox.  And so like well is it still reproducible?  Is it still pure?  Is it still hermetic?  And all of these nice words that we are used to and that we absolutely want in our builds.  And for that I have a small answer that's based on Nix experimental.  What I call Nix experimental is like it's an experimental branch from Elko where you  can transform a derivation into, that's difficult to say.  Well the problem with most of our derivations is that we have plenty of them that look the  same, they have the same content but because they come from a different.dv they are stored  in a different store path and then that store path is inside the binaries, it's inside a  lot of places and if you compare the bits are different.  But we can do something that's funny.  So let's find the i3.  So this is the one that we built before so it's there.  If you go to Nix, of course it's a local checkout of Nix and another one because I had to build  a different branch.  Okay, let's hope that it works correctly.  Yeah, you have a lot of outputs in this case.  See if I reduce the size, that's still readable for everyone.  Well in this case you have a lot of random stuff, let's just cheat a bit.  There is a mismatch between the D-man and the Nix that's running and they don't agree  on what we are trying to do but you see here that this is the path that we received and  this is what it looks like when it's been content addressable.  So basically all the closure has been rewritten and the name of the path has been changed  so that it matches the actual content of the path.  That's the meaning of content addressable.  Of course there is a problem because the path, they reference themselves and you need to  compute the hash on something and you would never hand.  So basically you just remove self-references, compute the hash and then reinsert them and  so it's not perfect content addressable, so it's like content address module reference.  What's interesting in there is that if everything is working correctly, this I3C cache, the thing  that I built, it's still cached, okay, it has a different hash but if I make it content  addressable, then you see that you get the exact same hash here and this means, and we  can check that but it doesn't make any sense because it's the exact same store location  but if you try to diff these two things, you will see that there is no difference.  Well, that's obvious but you can try if you want.  And of course the one that remains to be tested is this RCC, yeah, small, whatever.  And that one should also give us the same output.  Yes, that's working.  Well, in this case you may not believe me because it was already in the cache before  but trust me, you don't want to wait five minutes just for that to build.  So that's kind of part and answer.  We have broken everything.  There is no sandbox at all and we still manage to get the exact same output, right?  So this sandbox is not like, it's not totally needed.  It's there to guard you against making mistakes.  It avoids typical issues.  It avoids the network if something is trying and you don't want it to try but if you configure  everything properly, then it should build the same outside of the sandbox or inside  the sandbox.  And in this case, you have different vision, different contents before and then this is  the image that was there if there was some issue with the demo but I guess you got the  idea.  So what we have now, it's a bit like before but well, you know, you compile Firefox, it  takes a lot of time, you see that it's making progress, you've done most of the rest and  then C++ is over and you will enter the fix up phase.  And then, oh, it doesn't work.  But then you are not back from scratch, right?  You can recover some of the work that has already been done.  So that's better.  So that's a few messages.  Well, this presentation is not about sending messages.  It's more like challenging so I would like to get feedback if it gives you an idea that's  perfect but like the main message is that you can bridge a sandbox and it does not mean  that you are doing something wrong but well, that you should know what you are doing mostly.  It's really nice.  And when I see that, I'm like, yeah, it's going to work.  We really get these incremental builds inside Nix packages and that's something that I really  want.  I think it will also be really useful to get fast CI answers.  So if you want to make a pull request to Nix packages, then you could get the answer within  like a few minutes and not a few hours.  And also, like, it also means that we can trust other tools.  Like RCC is doing its best to make it reproducible, hermetic and all of these nice properties .  C cache is doing its best but it's very old so it doesn't know everything.  And then these tools, they are just C compiler abstractions but they behave exactly the same  as RCC behaves exactly as Bazel, for example.  So it's just that RCC manages only one C compile but Bazel manages a set of files.  From the outside, meaning from the Nix point of view, it's exactly the same.  It's a tool that's trying to execute something remotely and get the results.  So if we go that way, we could consider trusting Bazel, for example, for doing things purely  and correctly and allow it to have access to a cache and a remote execution engine somewhere.  So that's it for me.  I have questions for you if you don't have any.  So think about it.  What about verifying some compiler cache instruments for reproducibility so that they can be really  trusted and not just relying on limited amount of experiments?  Well, I guess that's always the same thing with reproducibility that you have to test  it.  And every time it's reproducible, that's good.  But it's not a proof that it will never be reproducible.  So we can do that with these tools too.  We can just start to include them and then compile, compile a few times and then once  in a while just make a perfectly aromatic sandbox build from scratch and see if we are  still building the same thing.  And that would give us more confidence that the tool is working correctly.  I mean, maybe the problem is that when these tools had been designed, there was no thought  about such thing as reproducibility.  And it might depend on different things like system clock, for example.  Well, truth is I'm not aware of all the details of all of these tools.  So that's something that we have to explore.  It's like, look, it's feasible.  Does someone want to work with me on that?  So one idea came to my mind when you showed mounting the file inside.  Did you try to mount like a unique socket, maybe, I don't know, Ccash and SCcash very  well, but maybe you can try networking that way?  I tried and then I had no more time.  So I think that theoretically it could work, but I don't know enough about all the technology  details.  On the outside looks trivial and then when you try to set up everything with the namespace  and stuff, well, it breaks easily.  So that's something we can try tomorrow.  If you think that it can be done in a few hours, then I would really like to try.  Cheers.  Hey, first I wanted to say I really appreciate because I've been many times in the situation  where I wanted to package something large like Ceph, for example, we're also iterating  on bugs, building Ceph takes one hour.  I had to build it over 80 times so you can imagine that took a while.  So I really appreciate any effort going in that direction.  Same thing with Haskell, which when you work on GHC, for example, or iterate on a low-level  library and rebuild NICs packages can also take a very long time.  So I think this is really good.  I wanted to also point out another approach that is a little bit orthogonal to what you've  presented that we just discussed yesterday here at the table with, I think, Edef Pi and  me, so NH2, which was that you can also do the type of caching that Ccache, for example,  does specifically for like C things on the syscall level.  So we could, if we wanted to, P trace the entire build and then see that if, let's say, an  execve produces a certain output, we can cache that and then wrap any execve call that happens  again to produce exactly the same output again.  So that is, of course, somewhat in the further future, but that might be something that can  be very generic across multiple things.  I think practically I will first go with exactly this approach because you have now shown that  it works and it's great.  And I just need to see how one can get the same thing for GHC Haskell, for example.  But in the long term, that might be something that if we agree on enough, bringing enough  hackiness into this thing as a kind of like quick developer thing we might do as well.  Well, it's not really a question, but I have still two answers for you.  Well, it's funny because I had the exact same discussion with NBP like last year and the  year before, like, well, we could do that on the syscall level.  That would be fun.  Through this, it's not that easy.  You have lots of cache entries and et cetera, so you need to be really, really efficient.  It's a cool idea, but I'm not sure it will turn out as practiced, but I'm always available  to talk about that idea.  Thank you.  Oh.  Thank you.  Thank you.  Thank you.  Thank you.
Build_outside_of_the__sand_box__NixCon_2019__0_575.png Build_outside_of_the__sand_box__NixCon_2019__576_2495.png Build_outside_of_the__sand_box__NixCon_2019__2496_4751.png Build_outside_of_the__sand_box__NixCon_2019__4752_8363.png Build_outside_of_the__sand_box__NixCon_2019__8364_8687.png Build_outside_of_the__sand_box__NixCon_2019__8688_9335.png Build_outside_of_the__sand_box__NixCon_2019__9336_11015.png Build_outside_of_the__sand_box__NixCon_2019__11016_11087.png Build_outside_of_the__sand_box__NixCon_2019__11088_11939.png Build_outside_of_the__sand_box__NixCon_2019__11940_12143.png Build_outside_of_the__sand_box__NixCon_2019__12144_12695.png Build_outside_of_the__sand_box__NixCon_2019__12696_12959.png Build_outside_of_the__sand_box__NixCon_2019__12960_13043.png Build_outside_of_the__sand_box__NixCon_2019__13044_13307.png Build_outside_of_the__sand_box__NixCon_2019__13308_13475.png Build_outside_of_the__sand_box__NixCon_2019__13476_13811.png Build_outside_of_the__sand_box__NixCon_2019__13812_13895.png Build_outside_of_the__sand_box__NixCon_2019__13896_14291.png Build_outside_of_the__sand_box__NixCon_2019__14292_14459.png Build_outside_of_the__sand_box__NixCon_2019__14460_14795.png Build_outside_of_the__sand_box__NixCon_2019__14796_16367.png Build_outside_of_the__sand_box__NixCon_2019__16368_17387.png Build_outside_of_the__sand_box__NixCon_2019__17388_17519.png Build_outside_of_the__sand_box__NixCon_2019__17520_23303.png Build_outside_of_the__sand_box__NixCon_2019__23304_24683.png Build_outside_of_the__sand_box__NixCon_2019__24684_24791.png Build_outside_of_the__sand_box__NixCon_2019__24792_25199.png Build_outside_of_the__sand_box__NixCon_2019__25200_25451.png Build_outside_of_the__sand_box__NixCon_2019__25452_26303.png Build_outside_of_the__sand_box__NixCon_2019__26304_26399.png Build_outside_of_the__sand_box__NixCon_2019__26400_28415.png Build_outside_of_the__sand_box__NixCon_2019__28416_29063.png Build_outside_of_the__sand_box__NixCon_2019__29064_29111.png Build_outside_of_the__sand_box__NixCon_2019__29112_29519.png Build_outside_of_the__sand_box__NixCon_2019__29520_29567.png Build_outside_of_the__sand_box__NixCon_2019__29568_29615.png Build_outside_of_the__sand_box__NixCon_2019__29616_30875.png Build_outside_of_the__sand_box__NixCon_2019__30876_32759.png Build_outside_of_the__sand_box__NixCon_2019__32760_33563.png Build_outside_of_the__sand_box__NixCon_2019__33564_33623.png Build_outside_of_the__sand_box__NixCon_2019__33624_34931.png Build_outside_of_the__sand_box__NixCon_2019__34932_35771.png Build_outside_of_the__sand_box__NixCon_2019__35772_35975.png Build_outside_of_the__sand_box__NixCon_2019__35976_37343.png Build_outside_of_the__sand_box__NixCon_2019__37344_37427.png Build_outside_of_the__sand_box__NixCon_2019__37428_40175.png Build_outside_of_the__sand_box__NixCon_2019__40176_40283.png Build_outside_of_the__sand_box__NixCon_2019__40284_43451.png Build_outside_of_the__sand_box__NixCon_2019__43452_51201.png
So, the last talk of this year's Nixcon is going to be about building outside of the sandbox. Bye, Laos. Like this one is working? Perfect. Okay, so hello everyone. I have this strange pleasure to be the last one to talk at this conference, but there is the old that's precisely made for you to think a bit. I have no real message for you or rather would love to challenge you and think about like what we do a bit differently. So I am known as Laos. I am currently doing a PhD on build systems. We can have a chat about that later on. And this is an experiment that comes from some presentation that I made two years ago. So it will go in very simply with three different examples. I want to build outside of the Nix sandbox and I will use different tools to do that like CKH, SCKH, RCC. You will get explanations of that. And then we have a small discussion about what does it mean, where do we go and final conclusion. So two years ago I was here precisely in front of you and I had this video. We didn't work at the time, but I think it's fixed now. Like yeah. So this video basically shows what happens when you are trying to build a long package like Firefox. For me it's often Firefox, but possibly you are building GTK or anything like that. And then you reach the top, nearly all of the compilation is done and then for some reason at some point, well, there is a fix up thing that doesn't work and bam, you are done. You are just back at the start. And if you want to fix the issue, then you have to pay all of the compilation of the full package. Well, so what's the situation now, two years after that? Well, we cannot build the next package incrementally of course and that's because of the sandbox. So it means that for Firefox, GTK, Libreface, QTE, GCC, well GCC is specific, all of these things you have to build them from scratch every time you make a change to derivation. Possibly that's not something that happens often for you, but if you try to hack into the next package, especially if you want to refactor the standard environment, something that you should never do, but if you work on that, then you will pay that you have to build everything. So my first attempt to escape this sandbox is CK. CK, I guess everyone knows that, no need to explain. It's quite an old tool, but it's still heavily used in some places. And the basic idea with all of these cases is that, well, you already asked me to compile this file with these options and with these headers, so I already know the answer. I can just fetch it from the cache. A small picture, right? So just query the cache and the cache says, well, no, no, it's not there yet. Okay, so you can compile locally and then you upload it to the cache for the next iteration and then you get your results, right? Which means that on the next time, when there is a cache hit, you can just recover the result from the cache and that's it. It should be quite fast. Okay. The funny thing is that I say, well, I want to try that and then I discover that it already exists. It's already feasible in these packages. There is a CKH STDN. That's a standard environment modified that allows you to use CKH. There is a lot of wrapping going on there. We can have a small chat about what that means, but basically if you replace the standard environment with that one, you get CKH working. Well, you get CKH working, but you still have the sandbox, right? So we'll see how to get out of that. But in Nix OS, that's also already provided and then it comes with all the defaults that you need to change into Nix, meaning that you need to build by allowing an extra folder, the cache, to be mounted inside the sandbox. And Nix OS, we do that for you. If you are just using Nix, then you have to use options to do that. They made it in a, I have no idea what they did. That's why the way I should check, but they made it in a very smart way in a sense that you can specify a list of packages that will have this CKH enabled so that you're not breaking the sandbox for everything, but only for these very heavy packages that you will compile often. So if in your company you have one big package like that, you can add it to the list. Okay, let's try the demo then. So the demo will work. I've taken i3 just because it's an average size C++ project. And we will build that normally and then we see CKH to see what it gives. Okay, let's go. Yeah. Okay, so if I want to build i3 normally, I just need to build this i3 stuff and we will try to get that into, we will try to filter the outputs. So it's been modified a bit. Just checking the delivery. So it's been modified a bit because I've added some kind of debugging to it so you can just see the time that each phase takes. So we get this unpack phase, well, that's way below the second, the patch phase way below the second too. I expect something about 10 seconds to build. So you have 8 seconds to configure and 10 seconds to build i3. That's with the sandbox, that's everything's normal except of this extra logging at it. Now, if we start with another tool, we can next build a i3 CKH. Yeah, CKH. If I do it like that, then the sandbox is still active so it won't change anything, right? So you need to add an option. That's an extra sandbox path. With that, I can build and mount the CKH inside the sandbox. Well I don't want to do that at the moment because this cache, we will try to clean it before. So just to be sure that there is nothing in there on the first try. We can start compiling it. And of course I'm still missing that. I'm not sure if you have an idea of what's to be expected but in this case, we are, there is nothing in the cache. So we need to compile everything so we'll have to take the same time. And also we have to pay an extra overhead for serializing and inserting everything into the folder. So this should be a bit slower actually. So now it's 10 seconds and 15 so it matches what we had before. We are from 8 to 10 and from 10 to 15 just because we paid the overhead of initializing everything. So if you want to start with an, we need to compile it again. So I made a special trick for you that you can just say again to the package and well I can't explain you how it works but it's nothing very magic. The nice part is that it starts the exact same compilation with some random stuff added to the derivation so that's the next thing that needs to be done again. And this time it's supposed to be faster. Modulo demo effects. Well, okay, that's not exactly what I used to have but it makes some sense. So what's not, what's, okay, right. So we have all of these. If you don't mind I will take the old results that I had. They are basically the same, 7 and 6, right? That's what we had, 7 and 6. So it's like the rounded numbers are the same. So well nothing unexpected here except that's, well you see that there is this 6 seconds that's below the previous build times and in a sense I3 is not a tool that uses heavy GCC compilations. There are a lot of manual generation documentation generating a lot of random stuff that are not cached by C cache. So of course on a bigger project that uses more C compiling you would have a better speed up here but possibly we will also have a higher penalty on the first time that you use it. Makes sense. Well, is this a good thing? Yeah, the problem is that when you start to make holes in the sandbox like that, well you know you don't have any guarantee of what's happening, right? A small hole in the sandbox and that's over. But at the same time, and we'll have a discussion at the end, and I think it can be a good thing at least in some well-defined situations. Anyway, let's go to the second attempt. So like I wanted to use, well, C cache just good but no people do better things at that. What if I want to use the same cache as my colleague or someone else? We should put that on the network and that's basically the idea of SCC which is shared or shared C cache developed by Mozilla. So it's Mozilla so they made it for C and C++ and then they also added the rest because well, it's Mozilla. And if you look at the diagram, it doesn't change much. Just like basically it's exactly the same, the only change is the name of the cache. It's not a folder, now it's a network machine. Which means that you are not mounting a folder inside the sandbox. You are making a completely different type of hole in there, you need network access. Well is that possible? No. No, it's not possible, it's not supported. You cannot make a small hole and say, okay, I want this machine, I want this port and that server, it doesn't work. So at the moment and for the experiments, what I do is just disable the sandbox. So we are entering strange countries but technically it should be feasible to have like a restricted network access in the sandbox if you want to keep some of the properties but not all of them. Yeah, also discovered that the sandbox is not, disabling the sandbox is not possible when you are building in different stores than your main stores. So all of these demos are just changing my main stores, possibly it breaks things but we'll see. Yeah, okay, disclosure here. I won't do it because for reasons that we don't have much time and also because it's not that different, just disable the sandbox, configure the remote and that's it. So I will go directly to the third attempt which is RECC and RECC, we keep the same ID except that we have the remote execution. That's a tool that has been developed at Bloomberg and they do it mainly because well, they love Bazel, they would like to have all these remote execution and things but they can't move at the moment like migrating all of your builds to Bazel, that's a huge investment so they don't want to do that and they develop this very small replacement that's instead of changing all of the build system, you just wrap each GCC invocation with it and then at that moment everything is sent to the server. Well, you get the better explanation with diagram but what we do in this case is that we remove the local compilation and everything is remote. The main advantage is that you can parallelize a lot more. If you have like 100 workers and a huge workload then you can spread the workload on 100 workers. That's the main advantage of not executing the compilation locally of course. So this is the example that may not work perfectly. I discovered that you need of course a server that allows this remote execution. So in this case it's build grid. I could have used another one but I discovered different issues. Not sure if it's NICs related or my VM related or something but I cannot work in parallel so I can only compile one thing at a time because the server won't take many items at the same time which is a bit stupid. It's not a limitation of the tool. It's used perfectly well in a lot of companies so it's most probably an issue with my setup but anyway. So it's single-threaded and then the worker is quite slow. So if I had to do the demo for you, well that's the time it takes. It takes about three to four minutes to compile now. Well there is definitely some issues. Nevertheless I think this is mostly like bad configuration from my part but I really want to show you what kind of strange things you can discover when you do that. So let's start again. So if we... This is completely stupid. Like it's a build grid server but it's called build form. Well we have to leave with that. Now on this server everything is already compiled so we will remove what's in there. I removed the content that I stored on the server. I removed the database so there is just basically a clean server anyway. And I start the server. I also need to start a worker of course. Yeah that's initialization. And then because I'm lazy I did not specify any kind of docker, worker, any environment. I just like use whatever is on this machine. Well that's it. And now can we make this smaller? Okay right. So what we want to do now is build exactly the same thing as we did before. Yes. So it's Nix build. Okay we want this i3 RCC. I will take the RCC small whatever. This one should be built already. Let's try. Option sandbox false. This is what allows you to access the network. Yeah it's already built. That's good. And the next is already built I guess. Yep. I need to go over and over again. Yeah okay that's working. Very useful when you have to build other things like that. And well no I won't say you. Let's see what we get. It fails. It fails and that's normal. That's what you get on the first attempt. If you're like there is nothing on this remote host you need all of the build inputs. All of the dependencies of your derivation needs to be there in a way. And this is of course not automated. It doesn't work at the moment but then I think there is some potential to make integration and to make Nix catch the network access and wrap it in some way or we should be able to find a way to make this work. At the moment what we have to do is like upload the inputs. So oh crap. So I just try to collect all of the build inputs of my derivation and then I send that to the server. This is a trick. There is always a trick. The network is not that good that I can upload everything but I just removed one of the missing files and this is the one that has just been updated. So if we start again and build this again, again, again thing what do we expect? Does it work now? Well it doesn't work. The tool is still missing. I was at possible we just added the tool that was missing. And in this case it's like one of the most frustrating parts of this is that well the failure is already in the cache. So we are just fetching that. It says previous time it fails. So I can fail again. If you want to work around that then you need to kill everything. In this case I have no idea of removing one entry so just let's be efficient. Okay, everything started clean from scratch so now I can build it. And well of course it takes like three minutes so we won't wait for the end of that. But this time it should work. Let's wait. By the way this VCC dash small is configured in a way that's the cache. VCC is not used in the configure phase because the configure phase just drops a lot of the outputs and so you don't see the error that we had before. That's the reason for the dash small. So it's working. It's compiling one thing at a time. Well it will compile for three minutes. I don't really need it because I have everything in there so let's drop that. Okay. It makes sense. So like you've discovered we need to have all the inputs on the server. It does not exist at the moment but the fact that it failed is also a good thing. It shows that if you are missing the exact compiler that's required by that version on the worker it will say I don't have it. It's not trying to use another thing. It's not trying to use GCC that's installed on that machine. So Nix by design with all of these strange paths and ashes makes it so that you do not pick random stuff. We've discovered that like these impure errors are cached. The fact that there was something missing in the state of the worker like this is part of this is there forever. That's really annoying. So well that's something to care about. But I think we could really automate that. And well the gain is potentially really huge because in Nix if we can have the real exact set of inputs we could share this information through everyone that would be like an either cache for very small C file builds. Well now there is a question of like well this is not pure right. We've broken everything. The balloon has exploded. The sandbox is no more. What remains of the all the good properties that we had with that sandbox. And so like well is it still reproducible? Is it still pure? Is it still hermetic? And all of these nice words that we are used to and that we absolutely want in our builds. And for that I have a small answer that's based on Nix experimental. What I call Nix experimental is like it's an experimental branch from Elko where you can transform a derivation into, that's difficult to say. Well the problem with most of our derivations is that we have plenty of them that look the same, they have the same content but because they come from a different.dv they are stored in a different store path and then that store path is inside the binaries, it's inside a lot of places and if you compare the bits are different. But we can do something that's funny. So let's find the i3. So this is the one that we built before so it's there. If you go to Nix, of course it's a local checkout of Nix and another one because I had to build a different branch. Okay, let's hope that it works correctly. Yeah, you have a lot of outputs in this case. See if I reduce the size, that's still readable for everyone. Well in this case you have a lot of random stuff, let's just cheat a bit. There is a mismatch between the D-man and the Nix that's running and they don't agree on what we are trying to do but you see here that this is the path that we received and this is what it looks like when it's been content addressable. So basically all the closure has been rewritten and the name of the path has been changed so that it matches the actual content of the path. That's the meaning of content addressable. Of course there is a problem because the path, they reference themselves and you need to compute the hash on something and you would never hand. So basically you just remove self-references, compute the hash and then reinsert them and so it's not perfect content addressable, so it's like content address module reference. What's interesting in there is that if everything is working correctly, this I3C cache, the thing that I built, it's still cached, okay, it has a different hash but if I make it content addressable, then you see that you get the exact same hash here and this means, and we can check that but it doesn't make any sense because it's the exact same store location but if you try to diff these two things, you will see that there is no difference. Well, that's obvious but you can try if you want. And of course the one that remains to be tested is this RCC, yeah, small, whatever. And that one should also give us the same output. Yes, that's working. Well, in this case you may not believe me because it was already in the cache before but trust me, you don't want to wait five minutes just for that to build. So that's kind of part and answer. We have broken everything. There is no sandbox at all and we still manage to get the exact same output, right? So this sandbox is not like, it's not totally needed. It's there to guard you against making mistakes. It avoids typical issues. It avoids the network if something is trying and you don't want it to try but if you configure everything properly, then it should build the same outside of the sandbox or inside the sandbox. And in this case, you have different vision, different contents before and then this is the image that was there if there was some issue with the demo but I guess you got the idea. So what we have now, it's a bit like before but well, you know, you compile Firefox, it takes a lot of time, you see that it's making progress, you've done most of the rest and then C++ is over and you will enter the fix up phase. And then, oh, it doesn't work. But then you are not back from scratch, right? You can recover some of the work that has already been done. So that's better. So that's a few messages. Well, this presentation is not about sending messages. It's more like challenging so I would like to get feedback if it gives you an idea that's perfect but like the main message is that you can bridge a sandbox and it does not mean that you are doing something wrong but well, that you should know what you are doing mostly. It's really nice. And when I see that, I'm like, yeah, it's going to work. We really get these incremental builds inside Nix packages and that's something that I really want. I think it will also be really useful to get fast CI answers. So if you want to make a pull request to Nix packages, then you could get the answer within like a few minutes and not a few hours. And also, like, it also means that we can trust other tools. Like RCC is doing its best to make it reproducible, hermetic and all of these nice properties. C cache is doing its best but it's very old so it doesn't know everything. And then these tools, they are just C compiler abstractions but they behave exactly the same as RCC behaves exactly as Bazel, for example. So it's just that RCC manages only one C compile but Bazel manages a set of files. From the outside, meaning from the Nix point of view, it's exactly the same. It's a tool that's trying to execute something remotely and get the results. So if we go that way, we could consider trusting Bazel, for example, for doing things purely and correctly and allow it to have access to a cache and a remote execution engine somewhere. So that's it for me. I have questions for you if you don't have any. So think about it. What about verifying some compiler cache instruments for reproducibility so that they can be really trusted and not just relying on limited amount of experiments? Well, I guess that's always the same thing with reproducibility that you have to test it. And every time it's reproducible, that's good. But it's not a proof that it will never be reproducible. So we can do that with these tools too. We can just start to include them and then compile, compile a few times and then once in a while just make a perfectly aromatic sandbox build from scratch and see if we are still building the same thing. And that would give us more confidence that the tool is working correctly. I mean, maybe the problem is that when these tools had been designed, there was no thought about such thing as reproducibility. And it might depend on different things like system clock, for example. Well, truth is I'm not aware of all the details of all of these tools. So that's something that we have to explore. It's like, look, it's feasible. Does someone want to work with me on that? So one idea came to my mind when you showed mounting the file inside. Did you try to mount like a unique socket, maybe, I don't know, Ccash and SCcash very well, but maybe you can try networking that way? I tried and then I had no more time. So I think that theoretically it could work, but I don't know enough about all the technology details. On the outside looks trivial and then when you try to set up everything with the namespace and stuff, well, it breaks easily. So that's something we can try tomorrow. If you think that it can be done in a few hours, then I would really like to try. Cheers. Hey, first I wanted to say I really appreciate because I've been many times in the situation where I wanted to package something large like Ceph, for example, we're also iterating on bugs, building Ceph takes one hour. I had to build it over 80 times so you can imagine that took a while. So I really appreciate any effort going in that direction. Same thing with Haskell, which when you work on GHC, for example, or iterate on a low-level library and rebuild NICs packages can also take a very long time. So I think this is really good. I wanted to also point out another approach that is a little bit orthogonal to what you've presented that we just discussed yesterday here at the table with, I think, Edef Pi and me, so NH2, which was that you can also do the type of caching that Ccache, for example, does specifically for like C things on the syscall level. So we could, if we wanted to, P trace the entire build and then see that if, let's say, an execve produces a certain output, we can cache that and then wrap any execve call that happens again to produce exactly the same output again. So that is, of course, somewhat in the further future, but that might be something that can be very generic across multiple things. I think practically I will first go with exactly this approach because you have now shown that it works and it's great. And I just need to see how one can get the same thing for GHC Haskell, for example. But in the long term, that might be something that if we agree on enough, bringing enough hackiness into this thing as a kind of like quick developer thing we might do as well. Well, it's not really a question, but I have still two answers for you. Well, it's funny because I had the exact same discussion with NBP like last year and the year before, like, well, we could do that on the syscall level. That would be fun. Through this, it's not that easy. You have lots of cache entries and et cetera, so you need to be really, really efficient. It's a cool idea, but I'm not sure it will turn out as practiced, but I'm always available to talk about that idea. Thank you. Oh. Thank you. Thank you. Thank you. Thank you.
to be about building outside of the sandbox. Bye, build systems. We can have a chat about that by allowing an extra folder, the cache, to be of packages that will have this CKH enabled so be really trusted and not just relying on limited was no thought about such thing as