pidfds: Process file descriptors on Linux
Brauner, Christian
Traditionally processes are identified globally via process identifiers (PIDs). Due to how pid allocation works the kernel is free to recycle PIDs once a process has been reaped. As such, PIDs do not allow another process to maintain a private, stable reference on a process. On systems under pressure it is thus possible that a PID is recycled without other (non-parent) processes being aware of it. This becomes rather problematic when (non-parent) processes are in charge of managing other processes as is the case for system managers or userspace implementations of OOM killers. Over the last months we have been working on solving these and other problems by introducing pidfds – process file descriptors. Among other nice properties, the allow callers to maintain a private, stable reference on a process. In this talk we will look at challenges we faced and the different approaches people pushed for. We will see what already has been implement and pushed upstream, look into various implementation details and outline what we have planned for the future.
 Next up, Mr. Christian Browner. Hey. So I just came back from Paris yesterday evening and the first thing I saw on Twitter about this conference is that Leonard is going to steal our home directories. It was great news. No joking, you've been talking about this for the last couple of weeks and months already. Right. I'm Christian. I write code. I work for Canonical right now. I mostly do upstream kernel work. I also maintain Lexi and Lexi container runs at times you might know about. And we do a lot of work in the upstream kernel for new container features, security, crossing a bunch of subsystems, also maintaining a few bits and pieces. And this is work we've been doing over the last couple of kernel releases. I've spoken about this for the first time, I think, in depth at Linux Plumbers a couple of weeks ago. And this is a concept which we call pitfds. Who has heard of that, by the way? Some people read LWN. Okay. So in short, what is a pitfd? The idea is it's a file descriptor referring to a process. Don't worry, I'm going to go into details what other systems had that before and so on. It's not a super novel idea. So a file descriptor referring to a process and specifically right now it's a file descriptor that refers to a threat group leader. If you have any questions about this or stuff, you can just yell right away. Right. Which refers to a threat group leader. So right now, but we don't exclude the possibility to do this in the future, is to make it possible. It's not possible to make a file descriptor refer to a single threat. Reason for this being that at the time when we started doing this work, there was no real reason to do it. Nobody really wanted it. Or yelled we wanted. And also it would have made the whole code a lot more complicated. So threat group leaders in the kernel are really horrible in general. So it's a stable private handle. So the file descriptor, a pit file descriptor guarantees that you maintain a reference to the same process as long as you hold that fd open. And it works in a very specific way. I will go into this in a little more detail in a little bit. And pitfds use a pre-existing stable process handle that the kernel already knows about, which is struct pit. So it's already used in proc to pin a process. So all the proc pit directories stash away a reference to struct pit. So the first question, if you know a little bit about the kernel, you might ask why we not using a task struct. Ideas, takers. So struct pit in the kernel is a way, it's the kernel's idea. If you read the comment in the code, it's the kernel's way of maintaining a stable process handle but without having to pin task struct. Why is that a problem? If you look at task struct in the kernel, it's like this chunk of code. And there's lists in there and pointers and probably arrays and all that kind of stuff. And so struct pit is a cheap way of getting around the problem of pinning, having to pin a lot of memory in the kernel for a long time. Because sometimes there are various code paths in the kernel that take references on struct pit to keep it alive because they need to look at some information or need to look who's the threat group leader or who's the session leader or and so on. So struct pit has a bunch of members. You already see something interesting in there which we added, which is wait queue for pit of denotification. We will get to this in a bit. And the idea is from a struct pit you can get to all of the interesting task structs which reference a type of pit that you're interested in. So the kernel makes difference between a threat group leader, a process group leader, and a session ID leader. And you can, if you reference struct pit, there's an array in there which is, you can get from a struct pit, right, with hlist head, tasks, pit type max. You can get to all the pits that are used. Yes, you can get to all the task structs that are used by this struct pit for a specific process type. You can get at the process group leader, you can get a session ID leader or the threat group leader depending on what information you need. So that's a pit of D. Pit of D stash is a reference to struct pit. It doesn't go away. And why do this in the first place? This is usually the first question. I would have been happy to just write it for my own entertainment, but we actually had a bunch of use cases. The first one that is pretty obvious and it comes up, even though it's always heavily debated whether or not this is a real issue and there are other ways to fix it, but it's pit recycling. So avoid pit falls of pit recycling on high pressure systems. Pit allocation, pit number allocation, I should be precise, in the kernel works cyclically. That means the kernel keeps on ramping up the pit number until it hits the maximum number of pits on the system and then it wraps around and takes the next free pit number. So if you have a lot of processes exiting and so on, then you can get in a situation where you can have a process whose pit has been recycled while you're still operating under the assumption that it's actually the prior process that you're operating on. This is useful, this is basically, you can use this to have timing-based attacks of which we actually had quite a few. So there is one against, which Jan found, against Android's get-pitcon. So you could trick it into operating on the wrong security context if I remember it correctly. There are actually two bugs related to this or two CVEs related to this, the first two. So you can have a look at that if you're interested. There are a bunch of pit-based Mac exploits, which I didn't know about. So Mac OS doesn't have, as far as I know, doesn't have the concept of a process file descriptor. There is a bunch of stuff in Qt that had problems with this. The last ones are actually hardware service manager arbitrary. The last ones are the get-pit exploits. I think the first one might actually be for Polkit, which you could attack with this as well. So this is really an issue. One thing you can prevent this to make it at least more, less likely to run into this issue is bump the maximum number of pits to a 4 million, I think, which system you started doing at some point. You could also probably get around this problem by using UUIDs and not file descriptors. There was a lot of discussions going on how exactly we should solve this problem. We went with PIT of these. And I'm going to explain why, I think, which is a good reason why. So pit recycling, that was one of the issues we really, really wanted to get around. And why do this in the first place? Again, there were a bunch of other reasons. One thing that came up repeatedly was shared libraries. I want to allow, I want to spawn of invisible helper processes. What does this mean? So an exit notification on Linux works in the following way. Oh, by the way, if anyone knows more than me, please yell as well. So you get a sick child signal, usually, on process excerpt, right? So that's how you get informed. But for example, if you have a main loop running, like a large main loop running where you have a lot of callbacks, and one of your callbacks is there to reap helper processes. Now this callback calls wait, wait, pit one, wait ID one, minus one, sorry, which means wait for all my children, or wait ID, P all, wait pit minus one, wait ID all. So it wants to wait for all children, specifically use case that probably in its systems want to have or want to support. But now you can end up in a situation where any other callback in your main loop could have spawned of a helper process and also relies on sick child and exit signals to be received. So now the wrong helper wakes up in your loop, gets a sick child signal and calls wait ID P all, and it's like, yes, I'm going to reap all of my children now, and then accidentally reap someone else's child. Other process now gets confused as to where the hell is my child. So this is really not a nice situation. So it's fogging off invisible helper processes. With some work you can do it, but then you run into issues with, then you run into issues with threat safety and signal handlers. There's a long blog post, I think, from Tiago out there. Who is one of the cute project maintainers that want to make use of this feature? So PITFDs, we'll see how hopefully, make it possible to spawn of invisible helper process pretty nicely. They also allow you to get notifications for process exit as a non-parent process in a clean way. And process management delegation in general, a handle to a non-parent process for a bunch of operations that you want to perform, which you cannot safely do right now. I mean, you can pass a PIT, but apart from, if you're the parent, you usually know that this is your child and you can be sure that this is still your child. If you're a non-parent and you have more issues to figure this out, you need to pass to PROC, look at start times, all kinds of hacks. Creeu has had issues with this for a long time, for example. Yes, so hand of a handle to a non-parent process for waiting, signaling, whatever, which if you have an FD, is a stable private handle on a process, that problem should go away. Another reason why we did go with FDs, the ubiquity of FDs, which sounds like a trivial argument, but it's actually not, I think. There are common patterns everywhere in code bases that make use of FDs. Most people have an E-Poll loop to listen for events on file descriptors. Most people have parsing logic to parse out FD info from PROC self, FD, FD number, and then FD info or something, and have logic for sending around file descriptors via SCM writes and so on. There is not a lot of adapting that you have to do. If you would have to build UUIDs on top of everything now, then it would have been annoying for the kernel to generate them and handing them out to user space. I know Leonard still wants them very much. You're not going to get them. So FDs seem to be quite the obvious solution. And also, here's where we a little bit get into this part. There was prior art for this as well, or similar art before. So ubiquity of these is a pretty good argument for doing it this way. And last but not least, user space really care about this feature. So nowadays, when you try to bring something into the kernel, it's usually not that you just get to do it. You usually have to say, like, this is a problem, and people really care about this. So we need a justification for why we wanted to do this, even though it seems obvious that it makes a problem go away, that it is a pretty big deal. Yes, user space really cares about this. Some of those have talked to me before. Some of those have written mails after. And some of them I just figured out by pure chance, by people pointing out, hey, they're making use of this feature. So Endless List keeps growing. One is Dbust. Dbust has an issue open where they want to switch from doing pit-based authentication to a pit-of-D-based authentication, because they have issues with pit recycling as well, or at least are afraid of running into issues with this. Cued wants to use it for sub-process management, so it's forking off invisible helper processes, which I mentioned before. The system need the only issue that I currently know about that is open is using it to reliably kill processes per C-group. They probably have other use cases for this as well in the future. Creew, which uses it to detect pit reuse. So it has a function that is called detect pit reuse, which is a hack, correct me if I'm wrong. And we can switch out this function for pit-of-Ds, using it to reliably track processes. Android, a low memory killer demon, is using pit-of-Ds as well. So we're actually one of the first that got really excited about this. It all derived back from a debate. So parts of that, I had an argument with a discussion with Case Cook, and I think David Howell is a while back at Linux security summit somewhere in Edinburgh or something where we talked about various things that we should do or could do and how we should do it, which is where the pit-of-D stuff started. Then there was also in parallel a discussion that started on the mailing list where people started to hack around in proc to make it at least possible to send signals via files and so on. So there was a lot happening at the same time, and this is ultimately the approach that came to fruition. So the Android guys, you had to give it to Joel, who works for Google as a kernel engineer who helped with this work, and who also did a lot of polling work that we'll see in a bit. So no memory killer demon is using pit-of-Ds to reliably track processes and kill them. This basically runs on Android 10 already. Oh, no, not pit-of-Ds. Sorry, they will be in Android 11. They back ported it to all of the LTS kernels, but they're definitely going to use it to get around issues where they have to make sure that the process that they're killing is really the process that they want to kill and so on. And BPF trace, once the switch to pit-of-Ds as well, they have an issue open as well. Don't just trust me, click on the links to verify it. Maybe they're all wrong. And there's prior art. This goes back to a former slide, to an earlier slide. Why do this? In this specific way, well, there is precedence in other systems, and I usually think it's not a good idea if you keep deviating too much between different operating system implementations because it makes it horrible for user space that at least try to be compatible across different operating systems to write working code. So I always was under the impression, I have to admit I did something which is not very smart from the perspective of a kernel guy. I didn't look at other kernels before actually starting this work. I did it later. And we were kind of lucky that we didn't get run in a lot of issues today originally, run into, but that was just by pure chance and having a lot of smart people yell at our work. But it also means I was falsely under the impression, for example, that Solaris had pit-of-Ds, which is wrong. They actually have it not, don't have it. At least the Lumos, the open source implementation of Solaris, only has a pure user space emulation of stable process handles, proc open, proc run, proc close, and proc free, but they have the same problem essentially. Open BSD and net BSD don't have it. Free BSD is the only system other than Linux that has it. They have a concept called proc file descriptor, or proc descriptor, sorry. And they have three system calls, PD fork, PD get, and PD kill, and they have gone with the different decisions or with which they have taken slightly different decisions than we have on Linux, parts of which are implementation-based, and most of which are implementation-based. And if you have questions about this, I can go into detail, but probably not going to be enough time. PD fork gives you a back-of-pit file descriptor, essentially, a proc file descriptor. And PD get-pit allows you to translate it to a pit, and PD kill is used to send a signal through one of those file descriptors. And on Linux, there are multiple approaches to get this into the kernel at once, 450, which is, I think, was originally done by Tiago as well from Qt, or at least was one of his suggestions. And then there was another patch set, which is called Clonavd. None of those made it. And I think one of the reasons, the patches were fine. They were interesting, had interesting new concepts. But for example, Clonavd tried to do a lot of things at the same time. So they mixed auto-reping semantics with file descriptors for processes and so on. So a lot of contention going on on how to actually do this correctly. And I think ultimately it didn't go in because it tried to do as much, too many things at once. Maybe there was another reason. I didn't see it from the thread, but there was a lot of stuff in that patch set. So okay. So we started building a new API around process management. And I want to start with this right away, which I also did at LPC. My intention has never been to say we have PITFTs and we have PITs, and they are totally separate worlds, and you either use the one, but then you can't use the other, which I think is the wrong way to think about this. PITFTs get around a very specific problem that you have in multiple, but rather specific situations. So you probably want the way to cross between PITs and PITFTs and use it both at the same time. So it's not like we're deprecating the PIT API and only going to PITFTs in the future. That's probably not what's going to happen. It may still, it may be the case, and this is what I expect, that a lot of new interesting features that people care about can be built upon PITFTs just by being a stable process handle, which you couldn't build upon PITs. So the first thing that we did in 5.1 was implement a new syscall, which is called PITFT send signal, which allows you to send a signal through a PITFT. This was the really obvious piece, because it's PIT recycling is usually concerned with sending signals, right? You're operating on something that is not yours. So it's pretty obvious to make the argument, look, this solves an issue. This lets us cleanly solve an issue. It's clearly something that user space has run into. There are a bunch of people who point this out that this is an issue for them. So we should do this. And it's actually in a lot of codes you can see here. Oh, there's a bit more to it down below, but the whole FD handling part is encapsulated in what you see here in the top. The first controversy that we had about this was what exactly is a PITFT going to be. People had very strong opinions about this, which also derives from the fact that it's an obvious problem in the sense, oh, it solves something really obvious. Here's my opinion and here's how we should do it. I'm not going to back down. So we kept yelling each other for a long while, which is usually what happens. And the first idea was to use a slash proc slash PIT to dear of these as PITFTs because they already pinstruck PID. They're pretty easy to get by. And it's a nice shortcut. You open, you call open on slash proc slash PID for the process you're interested in ignoring for now that this also has PIT recycling issues. Then you have an FD. It can't be stolen from you. Then you stuff it into PITFT sense signal and you send a signal to it. And if the process has exited behind your back and it's not around anymore and you send a signal, you get ESR which is Colonel Speak 4, no such process. So this brings me to another point. We don't pin PIDs. We don't pin PID numbers. It doesn't mean that when you hold a PID of D that now your PID is not going to recycle. Your PID is going to be recycled. We don't care about this. DFT is your stable handle. PID can be recycled. We're not stopping the Colonel from doing PID allocation or something. So right, we use proc PID as a shortcut of these, a sort of shortcut that is really handy for user space. But then we started, or we already had thought about this. We were faced with implementing the part where you return a file descriptor from one of those nice forklown functions that we have on Linux. And here is where we ran into real interesting problems. So this is work. So Jan and I started discussing about this because he had good input on this. And some people have the opinion, clone should just return file descriptors from slash proc slash PID. Sounds straightforward if you ignore all of the security issues. Like for example, there is a net directory in proc PID which allows you to snoop on the traffic of another process. It's also really horrible in terms of how file systems, especially the proc file system works in the Colonel. So if you want to return a proc PID file descriptor, you have to preallocate a dentry. You have to preallocate something that the Colonel uses internally to refer to a file. And then later on splice it into proc. Believe me, it was really nasty code. So what we did was we showed our preferred implementation and we showed the implementation that some people preferred and wrote both implementations, which was a lot of work. The one implementation showed slash proc PID file descriptor uses PID of these and the other one showed our implementation. And that was like really, if you compare it as like this much code and then you have this much code. And so people were like, yes, let's go with the implementation that you wanted, which was lucky for us. I think it's safe. There's a lot of headache and I would have been very unhappy. Actually, I considered if we go with proc PID directories, if I'm going to abandon this. But we didn't. So we got lucky. So in 5.2, this is where we landed support for returning file descriptors from the Clone Function. We were always under the impression that all of the Clone Flex were gone. Ha! No, there was one left. Which no one knew. I mean, we only saw this because Linus pointed it out. No, there is one flag bit left. I always assume we're out. But okay. So we added a new flag called clone PID of these, which creates PID of these at process creation time to completely let you get rid of the race where your PID can be recycled. They have a bunch of interesting properties. I'm going to talk about this a little bit. This is more or less, there are a bunch of more places that we had to touch or that I had to touch. But overall, this is the code that you see in the kernel internals clone function or fork function in this case. You know the comment for now, but if the flag is set, you allocate a new file descriptor and your return is user space. It stashes away a reference to a struct PID, which is the kernel internals notion of a stable reference on a process file descriptor. And you see these are a non-inode file descriptors. If you use a timer FD, if you use an event FD, if you use a signal FD, if you use an FD for the new amount API, and there is probably a bunch more I'm forgetting, the second notify FD, they're all anonymous inode based, which is just a single inode in the kernel that gets allocated when the kernel boots up and then you can get a new file from this. So the inode number is the same for all of them. FD means you don't need to allocate an inode and that's why it's very cheap. It doesn't waste memory, it doesn't waste allocation time and so on. So this is ideal more or less. You stash away a bunch of operations you want to allow in that file descriptor. So this is the PID of the FOP stuff. PID is the struct PID that I talked about. So it's pretty simple code overall. And one of the things that we also did is we made PID of these close on exec by default. So if you get a PID of D back, you really want to be sure that it doesn't leak into the child process when it execs, for example. And yeah, so this is what we did. And I think if you think about any new file descriptor type that you bring into the kernel, please make it close on exec by default. I tried to convince people to do this for the mount API, didn't fly well. But it really helps user space. It's one of the major pain points actually that you get file descriptors that stay open after you exec. And close on exec is really easy to set. Like, this is all it takes. So we'll read, write, occloexec and then you get an occloexec file descriptor back. Really makes life easier in user space. And another property is that we added an FD info file, so the FD info file will contain the PID of the process as seen from the PID namespace with which your proc mount was mounted. So any proc mount, especially in containers, is attached to the container's PID namespace, if you remount it at least. And in a new PID namespace, the PID that the process has will be different from the one that it has in one of the ancestor PID namespaces. And so we write it in there. So if you parse it, you will get the PID of that process in your proc instance, which is, for example, helpful if you send around to PIDFD and it was created in another PID namespace. And this is how you get the PID. But we also made it such, and this was Alec's idea, Alec Nesterhoff's idea. Originally we had it implemented in a way that when you set clone PIDFD, you got a file descriptor instead of a PID, which is problematic in multiple ways because file descriptors start at zero. And zero is obviously used to differentiate between the child and the parent. So you cannot really return zero as a valid file descriptor. So PIDFDs would have started at one, which is not nice. So for legacy clone, we made it such that we abuse one of those return arguments it has where usually the TID of the parent process, the TID of the child process is placed to return a PIDFD. So if you set clone PIDFD, you get the PID back and you get the PIDFD back at the same time. So you have no disconnect, you know both at the same time, which I think is pretty nice. It's even a little bit nicer, I think, than FreeBSD's PD fork. And then in 5.3, I think this is really exciting. We added Joel added polling support. This is something that they really wanted for the low memory killer demon so that you can get exit notification for non-parents. And in a more complex sense, it allows you to turn off the exit signal, which means that when a process exits, you can tell the kernel already today that I don't want a sick child. I don't want a sick child signal. I'm not going to explicitly ignore it because then you would order it, but I just don't want a signal when the process exits, which was a bit problematic because then how you know when the child is exited and so on. And the polling support will allow you to do this because what it essentially does is as soon as the process exits, so it's a threat group leader exits and the threat group is empty, then you get an exit notification saying, I'm ready. I have exited. So if you hand up one of those pit of these to a non-parent process, you get reliable exit notification and you don't need to rely on sick child signals and so on, which is pretty nifty. Also, pretty small code. It's not a lot to do. This is actually in two different files. It doesn't matter. You can grab for it if you're really interested in it. It's the polling implementation. So yeah, the one caveat that we have poll is only when the whole threat group exits, if the threat group leader exits before a lot of threats in the group, then polls should block similar to the Wade family. That's actually a problem you can run into and that's why threat group leaders are not really nice. But yeah, polling support is pretty exciting for process management. And in 5.3, we added another syscall, right, another syscall, PTIFTI is without code, PTIFTI open, the idea being that when you have forked the process, you sometimes still want to create a PTIFTI, especially if you want to watch a bunch of other processes, PIDs, and you can't rely on them using clone PTIFTI. PTIFTI open will allow you to do just that. It gets you a new PTIFTI. It also verifies that you give it a threat group leader so you can't get a PTIFTI for a threat. Yeah, that's 5.3 and 5.4. Excellent. That was proposed. That's actually no longer true. And it's pulled it from me, so we have that in. You can now wait on processes through PTIFTIs. So you can pass, wait ID has gained a new flag, PTIFTI, you pass it a PTIFTI instead of a PID, and then it retrieves it and then it waits on the PTIFTI, which I think is pretty neat. And we have a bunch of work plans. There are some work that I've, there's a lot of work that, or ideas that I have, I'm just going to speak about two of them. This at least came up as one of the original ideas, or the first ideas that we had was to make it possible similar to what FreeBSD has. So on FreeBSD, if you have a PID file descriptor, if you have a process file descriptor, it is a key line closed by default, which means if you close the last file descriptor, which has a reference to the corresponding struct file in the kernel, then it will kill the process that's, it killed the process, which is pretty neat. I wanted to do it the other way around, so per default, that's not the default. So if you close the last PIDFD, then still things are still fine. But we could add a flag that is set at process creation time, you can take it away afterwards. So no PR cuddles, or the PR cuddles, or that nonsense. And then you kill the process when the file descriptor is closed. There is a problem though that is, has been bothering me and that I've been thinking about this. So on FreeBSD, closing file descriptor is synchronous, so that means when closed returns, and that FD was the last reference to the struct file inside of the kernel, then by the time closed returns, you can be guaranteed that all of the cleanup operations have finished. Linux is smarter in some ways, or let's say complicated maybe, in the sense that closed can return without the corresponding release or cleanup method that belongs to the struct file that the last FD referred to has been run. It adds it to a work queue, or a K thread. And then at some point when the kernel thinks, ah, it's fine, I have some time to run this right now, then it cleans up everything. So that means if you close the last FD, closed returns, you are not guaranteed that the process is dead, which is not ideal, but usually the kernel cleans it up really quickly, or calls the release method for the corresponding file fairly quickly. It should only delay it when there's a lot of memory pressure, for example, in which case you screwed anyway. And another thing which is for the shared library case is exclusive waiting. So right now, anyone can still wait on the process that you've forked off, either via PID or via PIDFD. There is no differentiation there right now, and I think that just makes sense because then you have this connection between the PIDFD and the PID API. But there should be a way to say, I'm now going to separate this connection. And exclusive waiting would allow you to do this. So you have something like clone wait PID, flag name, open for discussion, which hides the process from generic wait requests. This is similar to what FreeBSD has as well, but I actually would like to make it stronger, which derives back from a discussion I had with Eric a while back. So you would only be able to wait on a process through a PIDFD, as long as there is a PIDFD referring to it, and when the last PIDFD is gone, the process gets auto-reaped. You know auto-reaping semantics? Thank you. So auto-reaping semantics, really, it's not, you explicitly, you set SIG child, you ignore SIG child explicitly. You say, I don't want SIG child, but I explicitly tell you that I don't want SIG child. At which point the kernelist says, OK, then I'm going to clean up that process for you if it exits, then it exits. And it's gone, which is really neat. It's different, for example, on FreeBSD, which is why they have chosen to implement proc file descriptors a little bit differently. If you explicitly ignore SIG child on FreeBSD, then the process doesn't get auto-cleaned up by the kernel. It gets re-parented to PID1, and PID1 then gets a signal for that process, which is basically saying, like, I'm done, you take care of it. Whereas Linux really cleans it up. So it would be really nice if you have clone weight PID. As long as you have a PID of D, you can wait on the process explicitly. If you close the last PID of D, telling the kernel, I don't care about this process anymore. If this process exits, just clean it up. The problem is the implementation, usually. It's pretty tricky to get right, I think. But I might put a patch set up there soonish. There's a bunch of other ideas, but I could keep talking, but I probably shouldn't. So, yeah, this is what we built over the last couple of kernel releases. We were at 5.4. We've obviously also been fixing a lot of bugs along the way. So this wasn't a whole bug-free process. But overall, I think it was pretty good. It was also a pretty good collaboration effort. A lot of people took part in the discussions, brought in really good ideas, gave reviews and helped with this. So it could probably give a shout out to a lot of people here. But yeah, that's about it. So if you have questions, go at it. Yes. So what about integration with CM credentials and other places where you send a PID to someone else? What do you mean? Could we have a flag where a CM credentials contains a PID FD instead of a PID? Can you just send it as a regular FD? Why you want a special flag? Because it gets sent implicitly. You set some flags on the socket and then the kernel does the job for you. And it is possible that by the time the consumer looks at this data, the PID could be not valid anymore. So you're saying the PID is sent implicitly, but instead of a PID, do you now want a PID FD? Well, I don't want to, from the top of my head, I don't necessarily see a problem with this. Yes? I guess the problem is, for instance, like a process sends a log message to Giorno and dies. So at the time, Giorno wants to look at the process to figure out which C group is running, which services. I remember this. Okay. Right. I'm trying to think if, yeah, we should probably talk about this. It shouldn't be something that is, I wouldn't put this off the table. It sounds useful if there is a really good use case for this and it doesn't really complicate in kernel code too much. So I have reason to, like, a good cost to justify this. It should be fine. I have a similar question, which is we now have PIDFDs and namespace FDs, but for the namespace SD, we have to go through the file system, procFS, to get them. Is there a way to derive namespace FD from a PIDFD? Maybe I have plans for that. So it's official request. Can you do that? One of the things that has always bothered me is if you do a set an S into a namespace, you have to do it iteratively, right? So it actually iteratively in two stages. You have to call open like seven times nowadays. And then you have to call set an S seven times and often in the correct order. And that's obviously a problem. Well, I see it as a problem. Maybe some people don't see it as a problem. I think this is the wrong approach. Ideally we could change. I had one set the idea, played with the idea. I may have mentioned on the mailing list at some point, two is if set an S would take a PIDFD and interpret the type argument that it has right now as a flag argument so you could specify the namespaces that you want to attach to. And then it derives it from the PIDFD in kernel and gets you into all of those namespaces, which would make attaching for containers and so on way nicer. Yes, that's definitely something which has been on my mind. It's usually that now it becomes a battle of the maintainers that I would like to call it. Because then it's, you know, we need to agree with does Eric think this is a good idea? How exactly does the API should look like and so on? But overall, yes, that's something I definitely have thought about. I also have thought about just recently forking into namespaces. Actually, I shouldn't claim this completely for me. David Howell suggested this once in a discussion. So ideally at clone time you say, not just create me a set of new namespaces, but create me a process in this set of namespaces. But there needs to be strong justification for why we would need this. Like is there some security issue or something when you create a process and then you do all of this stuff on it and so on. Yeah, that's definitely something we can think about. Yes. Hello. A question on kill on close feature. Yeah. So is it dangerous if the process and shall process exit a set you ID program on become more privileged and then the parent can kill it? Can the parent then kill it? Yes. In the naive implementation that I prototyped, yes, because it's an internal signal, right? So there is no security disconnect. So yes. So the question is the answer is yes, it would just kill the set you ID program that you spawned, even if it runs with more privileges for now. Even haven't thought about this, but I haven't worked on this in a lot more detail. If you have specific concerns about this, we should probably talk. No, that's not important. Just curious. So one of the things that I really want and is important I think is we have a tendency on Linux and this may be a good thing in some situations to good things. In some situations it's a bad thing where we, for example, we create, especially with a specific property and then later on we add a PR cuddle, and then we do a PR cuddle and that takes that property away from the process, which is horribly annoying. If I as a parent say I'm creating a new process with these properties set, then this property needs to stick to this process. It can't be taken away anymore after the fact, which is the thing I want to do with the clone weight pit, for example, the close on kill flex. It's a property that sticks to the process as long as it's alive and if it's gone, it's gone. I don't want to end up in a scenario where suddenly you can change all bits and pieces and flags again on processes. So I want sticky properties, essentially. I would like if it's useful at least. So Christopher Browner, thank you very much. Okay, so if you create a child process and then you send a PPD to that process to some other process, does the parent then get a sick child notification if the other process wants an exit notification that you just added in 5.3? So do both processes get notified when the process exits one over PPD and the parent over normal sick child? If you have said sick child, yes. If you want that, but you can explicitly turn it off. Okay, I wanted an parent. Okay, thank you. Yes. Anyone? Yes, so these are just a few remarks while doing this work. I forgot that slide. Okay, thank you very much.
 Next up, Mr. Christian Browner.  Hey.  So I just came back from Paris yesterday evening and the first thing I saw on Twitter about  this conference is that Leonard is going to steal our home directories.  It was great news.  No joking, you've been talking about this for the last couple of weeks and months already.  Right.  I'm Christian.  I write code.  I work for Canonical right now.  I mostly do upstream kernel work.  I also maintain Lexi and Lexi container runs at times you might know about.  And we do a lot of work in the upstream kernel for new container features, security, crossing  a bunch of subsystems, also maintaining a few bits and pieces.  And this is work we've been doing over the last couple of kernel releases.  I've spoken about this for the first time, I think, in depth at Linux Plumbers a couple  of weeks ago.  And this is a concept which we call pitfds.  Who has heard of that, by the way?  Some people read LWN.  Okay.  So in short, what is a pitfd?  The idea is it's a file descriptor referring to a process.  Don't worry, I'm going to go into details what other systems had that before and so on.  It's not a super novel idea.  So a file descriptor referring to a process and specifically right now it's a file descriptor  that refers to a threat group leader.  If you have any questions about this or stuff, you can just yell right away.  Right.  Which refers to a threat group leader.  So right now, but we don't exclude the possibility to do this in the future, is to make it possible.  It's not possible to make a file descriptor refer to a single threat.  Reason for this being that at the time when we started doing this work, there was no real  reason to do it.  Nobody really wanted it.  Or yelled we wanted.  And also it would have made the whole code a lot more complicated.  So threat group leaders in the kernel are really horrible in general.  So it's a stable private handle.  So the file descriptor, a pit file descriptor guarantees that you maintain a reference to  the same process as long as you hold that fd open.  And it works in a very specific way.  I will go into this in a little more detail in a little bit.  And pitfds use a pre-existing stable process handle that the kernel already knows about,  which is struct pit.  So it's already used in proc to pin a process.  So all the proc pit directories stash away a reference to struct pit.  So the first question, if you know a little bit about the kernel, you might ask why we  not using a task struct.  Ideas, takers.  So struct pit in the kernel is a way, it's the kernel's idea.  If you read the comment in the code, it's the kernel's way of maintaining a stable process  handle but without having to pin task struct.  Why is that a problem?  If you look at task struct in the kernel, it's like this chunk of code.  And there's lists in there and pointers and probably arrays and all that kind of stuff.  And so struct pit is a cheap way of getting around the problem of pinning, having to pin  a lot of memory in the kernel for a long time.  Because sometimes there are various code paths in the kernel that take references on struct  pit to keep it alive because they need to look at some information or need to look who's  the threat group leader or who's the session leader or and so on.  So struct pit has a bunch of members.  You already see something interesting in there which we added, which is wait queue for pit  of denotification.  We will get to this in a bit.  And the idea is from a struct pit you can get to all of the interesting task structs  which reference a type of pit that you're interested in.  So the kernel makes difference between a threat group leader, a process group leader, and  a session ID leader.  And you can, if you reference struct pit, there's an array in there which is, you can  get from a struct pit, right, with hlist head, tasks, pit type max.  You can get to all the pits that are used.  Yes, you can get to all the task structs that are used by this struct pit for a specific  process type.  You can get at the process group leader, you can get a session ID leader or the threat  group leader depending on what information you need.  So that's a pit of D. Pit of D stash is a reference to struct pit.  It doesn't go away.  And why do this in the first place?  This is usually the first question.  I would have been happy to just write it for my own entertainment, but we actually had  a bunch of use cases.  The first one that is pretty obvious and it comes up, even though it's always heavily  debated whether or not this is a real issue and there are other ways to fix it, but it's  pit recycling.  So avoid pit falls of pit recycling on high pressure systems.  Pit allocation, pit number allocation, I should be precise, in the kernel works cyclically.  That means the kernel keeps on ramping up the pit number until it hits the maximum number  of pits on the system and then it wraps around and takes the next free pit number.  So if you have a lot of processes exiting and so on, then you can get in a situation  where you can have a process whose pit has been recycled while you're still operating  under the assumption that it's actually the prior process that you're operating on.  This is useful, this is basically, you can use this to have timing-based attacks of which  we actually had quite a few.  So there is one against, which Jan found, against Android's get-pitcon.  So you could trick it into operating on the wrong security context if I remember it correctly.  There are actually two bugs related to this or two CVEs related to this, the first two.  So you can have a look at that if you're interested.  There are a bunch of pit-based Mac exploits, which I didn't know about.  So Mac OS doesn't have, as far as I know, doesn't have the concept of a process file  descriptor.  There is a bunch of stuff in Qt that had problems with this.  The last ones are actually hardware service manager arbitrary.  The last ones are the get-pit exploits.  I think the first one might actually be for Polkit, which you could attack with this as  well.  So this is really an issue.  One thing you can prevent this to make it at least more, less likely to run into this  issue is bump the maximum number of pits to a 4 million, I think, which system you started  doing at some point.  You could also probably get around this problem by using UUIDs and not file descriptors.  There was a lot of discussions going on how exactly we should solve this problem.  We went with PIT of these.  And I'm going to explain why, I think, which is a good reason why.  So pit recycling, that was one of the issues we really, really wanted to get around.  And why do this in the first place?  Again, there were a bunch of other reasons.  One thing that came up repeatedly was shared libraries.  I want to allow, I want to spawn of invisible helper processes.  What does this mean?  So an exit notification on Linux works in the following way.  Oh, by the way, if anyone knows more than me, please yell as well.  So you get a sick child signal, usually, on process excerpt, right?  So that's how you get informed.  But for example, if you have a main loop running, like a large main loop running where you have  a lot of callbacks, and one of your callbacks is there to reap helper processes.  Now this callback calls wait, wait, pit one, wait ID one, minus one, sorry, which means  wait for all my children, or wait ID, P all, wait pit minus one, wait ID all.  So it wants to wait for all children, specifically use case that probably in its systems want  to have or want to support.  But now you can end up in a situation where any other callback in your main loop could  have spawned of a helper process and also relies on sick child and exit signals to be  received.  So now the wrong helper wakes up in your loop, gets a sick child signal and calls wait ID  P all, and it's like, yes, I'm going to reap all of my children now, and then accidentally  reap someone else's child.  Other process now gets confused as to where the hell is my child.  So this is really not a nice situation.  So it's fogging off invisible helper processes.  With some work you can do it, but then you run into issues with, then you run into issues  with threat safety and signal handlers.  There's a long blog post, I think, from Tiago out there.  Who is one of the cute project maintainers that want to make use of this feature?  So PITFDs, we'll see how hopefully, make it possible to spawn of invisible helper process  pretty nicely.  They also allow you to get notifications for process exit as a non-parent process in a  clean way.  And process management delegation in general, a handle to a non-parent process for a bunch  of operations that you want to perform, which you cannot safely do right now.  I mean, you can pass a PIT, but apart from, if you're the parent, you usually know that  this is your child and you can be sure that this is still your child.  If you're a non-parent and you have more issues to figure this out, you need to pass  to PROC, look at start times, all kinds of hacks.  Creeu has had issues with this for a long time, for example.  Yes, so hand of a handle to a non-parent process for waiting, signaling, whatever, which if  you have an FD, is a stable private handle on a process, that problem should go away.  Another reason why we did go with FDs, the ubiquity of FDs, which sounds like a trivial  argument, but it's actually not, I think.  There are common patterns everywhere in code bases that make use of FDs.  Most people have an E-Poll loop to listen for events on file descriptors.  Most people have parsing logic to parse out FD info from PROC self, FD, FD number, and  then FD info or something, and have logic for sending around file descriptors via SCM  writes and so on.  There is not a lot of adapting that you have to do.  If you would have to build UUIDs on top of everything now, then it would have been annoying  for the kernel to generate them and handing them out to user space.  I know Leonard still wants them very much.  You're not going to get them.  So FDs seem to be quite the obvious solution.  And also, here's where we a little bit get into this part.  There was prior art for this as well, or similar art before.  So ubiquity of these is a pretty good argument for doing it this way.  And last but not least, user space really care about this feature.  So nowadays, when you try to bring something into the kernel, it's usually not that you  just get to do it.  You usually have to say, like, this is a problem, and people really care about this.  So we need a justification for why we wanted to do this, even though it seems obvious that  it makes a problem go away, that it is a pretty big deal.  Yes, user space really cares about this.  Some of those have talked to me before.  Some of those have written mails after.  And some of them I just figured out by pure chance, by people pointing out, hey, they're  making use of this feature.  So Endless List keeps growing.  One is Dbust.  Dbust has an issue open where they want to switch from doing pit-based authentication  to a pit-of-D-based authentication, because they have issues with pit recycling as well,  or at least are afraid of running into issues with this.  Cued wants to use it for sub-process management, so it's forking off invisible helper processes,  which I mentioned before.  The system need the only issue that I currently know about that is open is using it to reliably  kill processes per C-group.  They probably have other use cases for this as well in the future.  Creew, which uses it to detect pit reuse.  So it has a function that is called detect pit reuse, which is a hack, correct me if  I'm wrong.  And we can switch out this function for pit-of-Ds, using it to reliably track processes.  Android, a low memory killer demon, is using pit-of-Ds as well.  So we're actually one of the first that got really excited about this.  It all derived back from a debate.  So parts of that, I had an argument with a discussion with Case Cook, and I think David  Howell is a while back at Linux security summit somewhere in Edinburgh or something where  we talked about various things that we should do or could do and how we should do it, which  is where the pit-of-D stuff started.  Then there was also in parallel a discussion that started on the mailing list where people  started to hack around in proc to make it at least possible to send signals via files  and so on.  So there was a lot happening at the same time, and this is ultimately the approach that came  to fruition.  So the Android guys, you had to give it to Joel, who works for Google as a kernel engineer  who helped with this work, and who also did a lot of polling work that we'll see in a  bit.  So no memory killer demon is using pit-of-Ds to reliably track processes and kill them.  This basically runs on Android 10 already.  Oh, no, not pit-of-Ds.  Sorry, they will be in Android 11.  They back ported it to all of the LTS kernels, but they're definitely going to use it to  get around issues where they have to make sure that the process that they're killing  is really the process that they want to kill and so on.  And BPF trace, once the switch to pit-of-Ds as well, they have an issue open as well.  Don't just trust me, click on the links to verify it.  Maybe they're all wrong.  And there's prior art.  This goes back to a former slide, to an earlier slide.  Why do this?  In this specific way, well, there is precedence in other systems, and I usually think it's  not a good idea if you keep deviating too much between different operating system implementations  because it makes it horrible for user space that at least try to be compatible across  different operating systems to write working code.  So I always was under the impression, I have to admit I did something which is not very  smart from the perspective of a kernel guy.  I didn't look at other kernels before actually starting this work.  I did it later.  And we were kind of lucky that we didn't get run in a lot of issues today originally, run  into, but that was just by pure chance and having a lot of smart people yell at our work.  But it also means I was falsely under the impression, for example, that Solaris had  pit-of-Ds, which is wrong.  They actually have it not, don't have it.  At least the Lumos, the open source implementation of Solaris, only has a pure user space emulation  of stable process handles, proc open, proc run, proc close, and proc free, but they have  the same problem essentially.  Open BSD and net BSD don't have it.  Free BSD is the only system other than Linux that has it.  They have a concept called proc file descriptor, or proc descriptor, sorry.  And they have three system calls, PD fork, PD get, and PD kill, and they have gone with  the different decisions or with which they have taken slightly different decisions than  we have on Linux, parts of which are implementation-based, and most of which are implementation-based.  And if you have questions about this, I can go into detail, but probably not going to  be enough time.  PD fork gives you a back-of-pit file descriptor, essentially, a proc file descriptor.  And PD get-pit allows you to translate it to a pit, and PD kill is used to send a signal  through one of those file descriptors.  And on Linux, there are multiple approaches to get this into the kernel at once, 450,  which is, I think, was originally done by Tiago as well from Qt, or at least was one  of his suggestions.  And then there was another patch set, which is called Clonavd.  None of those made it.  And I think one of the reasons, the patches were fine.  They were interesting, had interesting new concepts.  But for example, Clonavd tried to do a lot of things at the same time.  So they mixed auto-reping semantics with file descriptors for processes and so on.  So a lot of contention going on on how to actually do this correctly.  And I think ultimately it didn't go in because it tried to do as much, too many things at  once.  Maybe there was another reason.  I didn't see it from the thread, but there was a lot of stuff in that patch set.  So okay.  So we started building a new API around process management.  And I want to start with this right away, which I also did at LPC.  My intention has never been to say we have PITFTs and we have PITs, and they are totally  separate worlds, and you either use the one, but then you can't use the other, which I  think is the wrong way to think about this.  PITFTs get around a very specific problem that you have in multiple, but rather specific  situations.  So you probably want the way to cross between PITs and PITFTs and use it both at the same  time.  So it's not like we're deprecating the PIT API and only going to PITFTs in the future.  That's probably not what's going to happen.  It may still, it may be the case, and this is what I expect, that a lot of new interesting  features that people care about can be built upon PITFTs just by being a stable process  handle, which you couldn't build upon PITs.  So the first thing that we did in 5.1 was implement a new syscall, which is called PITFT  send signal, which allows you to send a signal through a PITFT.  This was the really obvious piece, because it's PIT recycling is usually concerned with  sending signals, right?  You're operating on something that is not yours.  So it's pretty obvious to make the argument, look, this solves an issue.  This lets us cleanly solve an issue.  It's clearly something that user space has run into.  There are a bunch of people who point this out that this is an issue for them.  So we should do this.  And it's actually in a lot of codes you can see here.  Oh, there's a bit more to it down below, but the whole FD handling part is encapsulated  in what you see here in the top.  The first controversy that we had about this was what exactly is a PITFT going to be.  People had very strong opinions about this, which also derives from the fact that it's  an obvious problem in the sense, oh, it solves something really obvious.  Here's my opinion and here's how we should do it.  I'm not going to back down.  So we kept yelling each other for a long while, which is usually what happens.  And the first idea was to use a slash proc slash PIT to dear of these as PITFTs because  they already pinstruck PID.  They're pretty easy to get by.  And it's a nice shortcut.  You open, you call open on slash proc slash PID for the process you're interested in  ignoring for now that this also has PIT recycling issues.  Then you have an FD.  It can't be stolen from you.  Then you stuff it into PITFT sense signal and you send a signal to it.  And if the process has exited behind your back and it's not around anymore and you  send a signal, you get ESR which is Colonel Speak 4, no such process.  So this brings me to another point.  We don't pin PIDs.  We don't pin PID numbers.  It doesn't mean that when you hold a PID of D that now your PID is not going to recycle.  Your PID is going to be recycled.  We don't care about this.  DFT is your stable handle.  PID can be recycled.  We're not stopping the Colonel from doing PID allocation or something.  So right, we use proc PID as a shortcut of these, a sort of shortcut that is really handy  for user space.  But then we started, or we already had thought about this.  We were faced with implementing the part where you return a file descriptor from one of those  nice forklown functions that we have on Linux.  And here is where we ran into real interesting problems.  So this is work.  So Jan and I started discussing about this because he had good input on this.  And some people have the opinion, clone should just return file descriptors from slash proc  slash PID.  Sounds straightforward if you ignore all of the security issues.  Like for example, there is a net directory in proc PID which allows you to snoop on the  traffic of another process.  It's also really horrible in terms of how file systems, especially the proc file system  works in the Colonel.  So if you want to return a proc PID file descriptor, you have to preallocate a dentry.  You have to preallocate something that the Colonel uses internally to refer to a file.  And then later on splice it into proc.  Believe me, it was really nasty code.  So what we did was we showed our preferred implementation and we showed the implementation  that some people preferred and wrote both implementations, which was a lot of work.  The one implementation showed slash proc PID file descriptor uses PID of these and the other  one showed our implementation.  And that was like really, if you compare it as like this much code and then you have this  much code.  And so people were like, yes, let's go with the implementation that you wanted, which  was lucky for us.  I think it's safe.  There's a lot of headache and I would have been very unhappy.  Actually, I considered if we go with proc PID directories, if I'm going to abandon this.  But we didn't.  So we got lucky.  So in 5.2, this is where we landed support for returning file descriptors from the Clone  Function.  We were always under the impression that all of the Clone Flex were gone.  Ha!  No, there was one left.  Which no one knew.  I mean, we only saw this because Linus pointed it out.  No, there is one flag bit left.  I always assume we're out.  But okay.  So we added a new flag called clone PID of these, which creates PID of these at process  creation time to completely let you get rid of the race where your PID can be recycled.  They have a bunch of interesting properties.  I'm going to talk about this a little bit.  This is more or less, there are a bunch of more places that we had to touch or that I  had to touch.  But overall, this is the code that you see in the kernel internals clone function or  fork function in this case.  You know the comment for now, but if the flag is set, you allocate a new file descriptor  and your return is user space.  It stashes away a reference to a struct PID, which is the kernel internals notion of a  stable reference on a process file descriptor.  And you see these are a non-inode file descriptors.  If you use a timer FD, if you use an event FD, if you use a signal FD, if you use an  FD for the new amount API, and there is probably a bunch more I'm forgetting, the second notify  FD, they're all anonymous inode based, which is just a single inode in the kernel that  gets allocated when the kernel boots up and then you can get a new file from this.  So the inode number is the same for all of them.  FD means you don't need to allocate an inode and that's why it's very cheap.  It doesn't waste memory, it doesn't waste allocation time and so on.  So this is ideal more or less.  You stash away a bunch of operations you want to allow in that file descriptor.  So this is the PID of the FOP stuff.  PID is the struct PID that I talked about.  So it's pretty simple code overall.  And one of the things that we also did is we made PID of these close on exec by default.  So if you get a PID of D back, you really want to be sure that it doesn't leak into  the child process when it execs, for example.  And yeah, so this is what we did.  And I think if you think about any new file descriptor type that you bring into the kernel,  please make it close on exec by default.  I tried to convince people to do this for the mount API, didn't fly well.  But it really helps user space.  It's one of the major pain points actually that you get file descriptors that stay open  after you exec.  And close on exec is really easy to set.  Like, this is all it takes.  So we'll read, write, occloexec and then you get an occloexec file descriptor back.  Really makes life easier in user space.  And another property is that we added an FD info file, so the FD info file will contain  the PID of the process as seen from the PID namespace with which your proc mount was mounted.  So any proc mount, especially in containers, is attached to the container's PID namespace,  if you remount it at least.  And in a new PID namespace, the PID that the process has will be different from the one  that it has in one of the ancestor PID namespaces.  And so we write it in there.  So if you parse it, you will get the PID of that process in your proc instance, which  is, for example, helpful if you send around to PIDFD and it was created in another PID  namespace.  And this is how you get the PID.  But we also made it such, and this was Alec's idea, Alec Nesterhoff's idea.  Originally we had it implemented in a way that when you set clone PIDFD, you got a file  descriptor instead of a PID, which is problematic in multiple ways because file descriptors start  at zero.  And zero is obviously used to differentiate between the child and the parent.  So you cannot really return zero as a valid file descriptor.  So PIDFDs would have started at one, which is not nice.  So for legacy clone, we made it such that we abuse one of those return arguments it  has where usually the TID of the parent process, the TID of the child process is placed to  return a PIDFD.  So if you set clone PIDFD, you get the PID back and you get the PIDFD back at the same  time.  So you have no disconnect, you know both at the same time, which I think is pretty nice.  It's even a little bit nicer, I think, than FreeBSD's PD fork.  And then in 5.3, I think this is really exciting.  We added Joel added polling support.  This is something that they really wanted for the low memory killer demon so that you  can get exit notification for non-parents.  And in a more complex sense, it allows you to turn off the exit signal, which means  that when a process exits, you can tell the kernel already today that I don't want a  sick child.  I don't want a sick child signal.  I'm not going to explicitly ignore it because then you would order it, but I just don't want  a signal when the process exits, which was a bit problematic because then how you know  when the child is exited and so on.  And the polling support will allow you to do this because what it essentially does is  as soon as the process exits, so it's a threat group leader exits and the threat group is  empty, then you get an exit notification saying, I'm ready.  I have exited.  So if you hand up one of those pit of these to a non-parent process, you get reliable  exit notification and you don't need to rely on sick child signals and so on, which is  pretty nifty.  Also, pretty small code.  It's not a lot to do.  This is actually in two different files.  It doesn't matter.  You can grab for it if you're really interested in it.  It's the polling implementation.  So yeah, the one caveat that we have poll is only when the whole threat group exits, if  the threat group leader exits before a lot of threats in the group, then polls should  block similar to the Wade family.  That's actually a problem you can run into and that's why threat group leaders are not  really nice.  But yeah, polling support is pretty exciting for process management.  And in 5.3, we added another syscall, right, another syscall, PTIFTI is without code, PTIFTI  open, the idea being that when you have forked the process, you sometimes still want to create  a PTIFTI, especially if you want to watch a bunch of other processes, PIDs, and you  can't rely on them using clone PTIFTI.  PTIFTI open will allow you to do just that.  It gets you a new PTIFTI.  It also verifies that you give it a threat group leader so you can't get a PTIFTI for  a threat.  Yeah, that's 5.3 and 5.4.  Excellent.  That was proposed.  That's actually no longer true.  And it's pulled it from me, so we have that in.  You can now wait on processes through PTIFTIs.  So you can pass, wait ID has gained a new flag, PTIFTI, you pass it a PTIFTI instead of a  PID, and then it retrieves it and then it waits on the PTIFTI, which I think is pretty  neat.  And we have a bunch of work plans.  There are some work that I've, there's a lot of work that, or ideas that I have, I'm  just going to speak about two of them.  This at least came up as one of the original ideas, or the first ideas that we had was  to make it possible similar to what FreeBSD has.  So on FreeBSD, if you have a PID file descriptor, if you have a process file descriptor, it is  a key line closed by default, which means if you close the last file descriptor, which  has a reference to the corresponding struct file in the kernel, then it will kill the  process that's, it killed the process, which is pretty neat.  I wanted to do it the other way around, so per default, that's not the default.  So if you close the last PIDFD, then still things are still fine.  But we could add a flag that is set at process creation time, you can take it away afterwards.  So no PR cuddles, or the PR cuddles, or that nonsense.  And then you kill the process when the file descriptor is closed.  There is a problem though that is, has been bothering me and that I've been thinking about  this.  So on FreeBSD, closing file descriptor is synchronous, so that means when closed returns,  and that FD was the last reference to the struct file inside of the kernel, then by  the time closed returns, you can be guaranteed that all of the cleanup operations have finished.  Linux is smarter in some ways, or let's say complicated maybe, in the sense that closed  can return without the corresponding release or cleanup method that belongs to the struct  file that the last FD referred to has been run.  It adds it to a work queue, or a K thread.  And then at some point when the kernel thinks, ah, it's fine, I have some time to run this  right now, then it cleans up everything.  So that means if you close the last FD, closed returns, you are not guaranteed that the process  is dead, which is not ideal, but usually the kernel cleans it up really quickly, or calls  the release method for the corresponding file fairly quickly.  It should only delay it when there's a lot of memory pressure, for example, in which  case you screwed anyway.  And another thing which is for the shared library case is exclusive waiting.  So right now, anyone can still wait on the process that you've forked off, either via  PID or via PIDFD.  There is no differentiation there right now, and I think that just makes sense because then  you have this connection between the PIDFD and the PID API.  But there should be a way to say, I'm now going to separate this connection.  And exclusive waiting would allow you to do this.  So you have something like clone wait PID, flag name, open for discussion, which hides  the process from generic wait requests.  This is similar to what FreeBSD has as well, but I actually would like to make it stronger,  which derives back from a discussion I had with Eric a while back.  So you would only be able to wait on a process through a PIDFD, as long as there is a PIDFD  referring to it, and when the last PIDFD is gone, the process gets auto-reaped.  You know auto-reaping semantics?  Thank you.  So auto-reaping semantics, really, it's not, you explicitly, you set SIG child, you ignore  SIG child explicitly.  You say, I don't want SIG child, but I explicitly tell you that I don't want SIG child.  At which point the kernelist says, OK, then I'm going to clean up that process for you  if it exits, then it exits.  And it's gone, which is really neat.  It's different, for example, on FreeBSD, which is why they have chosen to implement proc  file descriptors a little bit differently.  If you explicitly ignore SIG child on FreeBSD, then the process doesn't get auto-cleaned up  by the kernel.  It gets re-parented to PID1, and PID1 then gets a signal for that process, which is basically  saying, like, I'm done, you take care of it.  Whereas Linux really cleans it up.  So it would be really nice if you have clone weight PID.  As long as you have a PID of D, you can wait on the process explicitly.  If you close the last PID of D, telling the kernel, I don't care about this process anymore.  If this process exits, just clean it up.  The problem is the implementation, usually.  It's pretty tricky to get right, I think.  But I might put a patch set up there soonish.  There's a bunch of other ideas, but I could keep talking, but I probably shouldn't.  So, yeah, this is what we built over the last couple of kernel releases.  We were at 5.4.  We've obviously also been fixing a lot of bugs along the way.  So this wasn't a whole bug-free process.  But overall, I think it was pretty good.  It was also a pretty good collaboration effort.  A lot of people took part in the discussions, brought in really good ideas, gave reviews  and helped with this.  So it could probably give a shout out to a lot of people here.  But yeah, that's about it.  So if you have questions, go at it.  Yes.  So what about integration with CM credentials and other places where you send a PID to someone  else?  What do you mean?  Could we have a flag where a CM credentials contains a PID FD instead of a PID?  Can you just send it as a regular FD?  Why you want a special flag?  Because it gets sent implicitly.  You set some flags on the socket and then the kernel does the job for you.  And it is possible that by the time the consumer looks at this data, the PID could be not valid  anymore.  So you're saying the PID is sent implicitly, but instead of a PID, do you now want a PID  FD?  Well, I don't want to, from the top of my head, I don't necessarily see a problem with  this.  Yes?  I guess the problem is, for instance, like a process sends a log message to Giorno and  dies.  So at the time, Giorno wants to look at the process to figure out which C group is running,  which services.  I remember this.  Okay.  Right.  I'm trying to think if, yeah, we should probably talk about this.  It shouldn't be something that is, I wouldn't put this off the table.  It sounds useful if there is a really good use case for this and it doesn't really complicate  in kernel code too much.  So I have reason to, like, a good cost to justify this.  It should be fine.  I have a similar question, which is we now have PIDFDs and namespace FDs, but for the  namespace SD, we have to go through the file system, procFS, to get them.  Is there a way to derive namespace FD from a PIDFD?  Maybe I have plans for that.  So it's official request.  Can you do that?  One of the things that has always bothered me is if you do a set an S into a namespace,  you have to do it iteratively, right?  So it actually iteratively in two stages.  You have to call open like seven times nowadays.  And then you have to call set an S seven times and often in the correct order.  And that's obviously a problem.  Well, I see it as a problem.  Maybe some people don't see it as a problem.  I think this is the wrong approach.  Ideally we could change.  I had one set the idea, played with the idea.  I may have mentioned on the mailing list at some point, two is if set an S would take  a PIDFD and interpret the type argument that it has right now as a flag argument so you  could specify the namespaces that you want to attach to.  And then it derives it from the PIDFD in kernel and gets you into all of those namespaces,  which would make attaching for containers and so on way nicer.  Yes, that's definitely something which has been on my mind.  It's usually that now it becomes a battle of the maintainers that I would like to call  it.  Because then it's, you know, we need to agree with does Eric think this is a good idea?  How exactly does the API should look like and so on?  But overall, yes, that's something I definitely have thought about.  I also have thought about just recently forking into namespaces.  Actually, I shouldn't claim this completely for me.  David Howell suggested this once in a discussion.  So ideally at clone time you say, not just create me a set of new namespaces, but create  me a process in this set of namespaces.  But there needs to be strong justification for why we would need this.  Like is there some security issue or something when you create a process and then you do  all of this stuff on it and so on.  Yeah, that's definitely something we can think about.  Yes.  Hello.  A question on kill on close feature.  Yeah.  So is it dangerous if the process and shall process exit a set you ID program on become  more privileged and then the parent can kill it?  Can the parent then kill it?  Yes.  In the naive implementation that I prototyped, yes, because it's an internal signal, right?  So there is no security disconnect.  So yes.  So the question is the answer is yes, it would just kill the set you ID program that  you spawned, even if it runs with more privileges for now.  Even haven't thought about this, but I haven't worked on this in a lot more detail.  If you have specific concerns about this, we should probably talk.  No, that's not important.  Just curious.  So one of the things that I really want and is important I think is we have a tendency  on Linux and this may be a good thing in some situations to good things.  In some situations it's a bad thing where we, for example, we create, especially with  a specific property and then later on we add a PR cuddle, and then we do a PR cuddle and  that takes that property away from the process, which is horribly annoying.  If I as a parent say I'm creating a new process with these properties set, then this property  needs to stick to this process.  It can't be taken away anymore after the fact, which is the thing I want to do with the clone  weight pit, for example, the close on kill flex.  It's a property that sticks to the process as long as it's alive and if it's gone, it's  gone.  I don't want to end up in a scenario where suddenly you can change all bits and pieces  and flags again on processes.  So I want sticky properties, essentially.  I would like if it's useful at least.  So Christopher Browner, thank you very much.  Okay, so if you create a child process and then you send a PPD to that process to some  other process, does the parent then get a sick child notification if the other process  wants an exit notification that you just added in 5.3?  So do both processes get notified when the process exits one over PPD and the parent  over normal sick child?  If you have said sick child, yes.  If you want that, but you can explicitly turn it off.  Okay, I wanted an parent.  Okay, thank you.  Yes.  Anyone?  Yes, so these are just a few remarks while doing this work.  I forgot that slide.  Okay, thank you very much.
pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_0_131.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_132_7667.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_7668_7919.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_7920_17291.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_17292_23171.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_23172_23423.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_23424_24347.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_24348_24599.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_24600_25607.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_25608_26615.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_26616_27563.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_27564_27827.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_27828_35879.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_35880_35975.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_35976_36371.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_36372_39179.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_39180_43967.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_43968_46463.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_46464_46595.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_46596_49415.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_49416_51431.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_51432_53423.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_53424_53507.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_53508_53579.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_53580_53627.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_53628_53831.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_53832_53951.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_53952_54023.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_54024_54071.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_54072_54155.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_54156_54263.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_54264_55043.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_55044_55091.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_55092_55139.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_55140_55319.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_55320_55619.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_55620_56075.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_56076_56135.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_56136_56219.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_56220_56327.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_56328_56567.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_56568_56771.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_56772_56855.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_56856_57047.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_57048_57203.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_57204_58451.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_58452_58571.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_58572_59147.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_59148_59303.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_59304_59603.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_59604_59807.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_59808_60191.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_60192_60239.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_60240_60695.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_60696_61295.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_61296_61847.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_61848_62111.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_62112_62303.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_62304_62747.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_62748_62855.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_62856_62999.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_63000_63395.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_63396_63587.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_63588_63959.png pidfds__Process_file_descriptors_on_Linux_ZKSUjt9VjWE_63960_64364.png
Next up, Mr. Christian Browner. Hey. So I just came back from Paris yesterday evening and the first thing I saw on Twitter about this conference is that Leonard is going to steal our home directories. It was great news. No joking, you've been talking about this for the last couple of weeks and months already. Right. I'm Christian. I write code. I work for Canonical right now. I mostly do upstream kernel work. I also maintain Lexi and Lexi container runs at times you might know about. And we do a lot of work in the upstream kernel for new container features, security, crossing a bunch of subsystems, also maintaining a few bits and pieces. And this is work we've been doing over the last couple of kernel releases. I've spoken about this for the first time, I think, in depth at Linux Plumbers a couple of weeks ago. And this is a concept which we call pitfds. Who has heard of that, by the way? Some people read LWN. Okay. So in short, what is a pitfd? The idea is it's a file descriptor referring to a process. Don't worry, I'm going to go into details what other systems had that before and so on. It's not a super novel idea. So a file descriptor referring to a process and specifically right now it's a file descriptor that refers to a threat group leader. If you have any questions about this or stuff, you can just yell right away. Right. Which refers to a threat group leader. So right now, but we don't exclude the possibility to do this in the future, is to make it possible. It's not possible to make a file descriptor refer to a single threat. Reason for this being that at the time when we started doing this work, there was no real reason to do it. Nobody really wanted it. Or yelled we wanted. And also it would have made the whole code a lot more complicated. So threat group leaders in the kernel are really horrible in general. So it's a stable private handle. So the file descriptor, a pit file descriptor guarantees that you maintain a reference to the same process as long as you hold that fd open. And it works in a very specific way. I will go into this in a little more detail in a little bit. And pitfds use a pre-existing stable process handle that the kernel already knows about, which is struct pit. So it's already used in proc to pin a process. So all the proc pit directories stash away a reference to struct pit. So the first question, if you know a little bit about the kernel, you might ask why we not using a task struct. Ideas, takers. So struct pit in the kernel is a way, it's the kernel's idea. If you read the comment in the code, it's the kernel's way of maintaining a stable process handle but without having to pin task struct. Why is that a problem? If you look at task struct in the kernel, it's like this chunk of code. And there's lists in there and pointers and probably arrays and all that kind of stuff. And so struct pit is a cheap way of getting around the problem of pinning, having to pin a lot of memory in the kernel for a long time. Because sometimes there are various code paths in the kernel that take references on struct pit to keep it alive because they need to look at some information or need to look who's the threat group leader or who's the session leader or and so on. So struct pit has a bunch of members. You already see something interesting in there which we added, which is wait queue for pit of denotification. We will get to this in a bit. And the idea is from a struct pit you can get to all of the interesting task structs which reference a type of pit that you're interested in. So the kernel makes difference between a threat group leader, a process group leader, and a session ID leader. And you can, if you reference struct pit, there's an array in there which is, you can get from a struct pit, right, with hlist head, tasks, pit type max. You can get to all the pits that are used. Yes, you can get to all the task structs that are used by this struct pit for a specific process type. You can get at the process group leader, you can get a session ID leader or the threat group leader depending on what information you need. So that's a pit of D. Pit of D stash is a reference to struct pit. It doesn't go away. And why do this in the first place? This is usually the first question. I would have been happy to just write it for my own entertainment, but we actually had a bunch of use cases. The first one that is pretty obvious and it comes up, even though it's always heavily debated whether or not this is a real issue and there are other ways to fix it, but it's pit recycling. So avoid pit falls of pit recycling on high pressure systems. Pit allocation, pit number allocation, I should be precise, in the kernel works cyclically. That means the kernel keeps on ramping up the pit number until it hits the maximum number of pits on the system and then it wraps around and takes the next free pit number. So if you have a lot of processes exiting and so on, then you can get in a situation where you can have a process whose pit has been recycled while you're still operating under the assumption that it's actually the prior process that you're operating on. This is useful, this is basically, you can use this to have timing-based attacks of which we actually had quite a few. So there is one against, which Jan found, against Android's get-pitcon. So you could trick it into operating on the wrong security context if I remember it correctly. There are actually two bugs related to this or two CVEs related to this, the first two. So you can have a look at that if you're interested. There are a bunch of pit-based Mac exploits, which I didn't know about. So Mac OS doesn't have, as far as I know, doesn't have the concept of a process file descriptor. There is a bunch of stuff in Qt that had problems with this. The last ones are actually hardware service manager arbitrary. The last ones are the get-pit exploits. I think the first one might actually be for Polkit, which you could attack with this as well. So this is really an issue. One thing you can prevent this to make it at least more, less likely to run into this issue is bump the maximum number of pits to a 4 million, I think, which system you started doing at some point. You could also probably get around this problem by using UUIDs and not file descriptors. There was a lot of discussions going on how exactly we should solve this problem. We went with PIT of these. And I'm going to explain why, I think, which is a good reason why. So pit recycling, that was one of the issues we really, really wanted to get around. And why do this in the first place? Again, there were a bunch of other reasons. One thing that came up repeatedly was shared libraries. I want to allow, I want to spawn of invisible helper processes. What does this mean? So an exit notification on Linux works in the following way. Oh, by the way, if anyone knows more than me, please yell as well. So you get a sick child signal, usually, on process excerpt, right? So that's how you get informed. But for example, if you have a main loop running, like a large main loop running where you have a lot of callbacks, and one of your callbacks is there to reap helper processes. Now this callback calls wait, wait, pit one, wait ID one, minus one, sorry, which means wait for all my children, or wait ID, P all, wait pit minus one, wait ID all. So it wants to wait for all children, specifically use case that probably in its systems want to have or want to support. But now you can end up in a situation where any other callback in your main loop could have spawned of a helper process and also relies on sick child and exit signals to be received. So now the wrong helper wakes up in your loop, gets a sick child signal and calls wait ID P all, and it's like, yes, I'm going to reap all of my children now, and then accidentally reap someone else's child. Other process now gets confused as to where the hell is my child. So this is really not a nice situation. So it's fogging off invisible helper processes. With some work you can do it, but then you run into issues with, then you run into issues with threat safety and signal handlers. There's a long blog post, I think, from Tiago out there. Who is one of the cute project maintainers that want to make use of this feature? So PITFDs, we'll see how hopefully, make it possible to spawn of invisible helper process pretty nicely. They also allow you to get notifications for process exit as a non-parent process in a clean way. And process management delegation in general, a handle to a non-parent process for a bunch of operations that you want to perform, which you cannot safely do right now. I mean, you can pass a PIT, but apart from, if you're the parent, you usually know that this is your child and you can be sure that this is still your child. If you're a non-parent and you have more issues to figure this out, you need to pass to PROC, look at start times, all kinds of hacks. Creeu has had issues with this for a long time, for example. Yes, so hand of a handle to a non-parent process for waiting, signaling, whatever, which if you have an FD, is a stable private handle on a process, that problem should go away. Another reason why we did go with FDs, the ubiquity of FDs, which sounds like a trivial argument, but it's actually not, I think. There are common patterns everywhere in code bases that make use of FDs. Most people have an E-Poll loop to listen for events on file descriptors. Most people have parsing logic to parse out FD info from PROC self, FD, FD number, and then FD info or something, and have logic for sending around file descriptors via SCM writes and so on. There is not a lot of adapting that you have to do. If you would have to build UUIDs on top of everything now, then it would have been annoying for the kernel to generate them and handing them out to user space. I know Leonard still wants them very much. You're not going to get them. So FDs seem to be quite the obvious solution. And also, here's where we a little bit get into this part. There was prior art for this as well, or similar art before. So ubiquity of these is a pretty good argument for doing it this way. And last but not least, user space really care about this feature. So nowadays, when you try to bring something into the kernel, it's usually not that you just get to do it. You usually have to say, like, this is a problem, and people really care about this. So we need a justification for why we wanted to do this, even though it seems obvious that it makes a problem go away, that it is a pretty big deal. Yes, user space really cares about this. Some of those have talked to me before. Some of those have written mails after. And some of them I just figured out by pure chance, by people pointing out, hey, they're making use of this feature. So Endless List keeps growing. One is Dbust. Dbust has an issue open where they want to switch from doing pit-based authentication to a pit-of-D-based authentication, because they have issues with pit recycling as well, or at least are afraid of running into issues with this. Cued wants to use it for sub-process management, so it's forking off invisible helper processes, which I mentioned before. The system need the only issue that I currently know about that is open is using it to reliably kill processes per C-group. They probably have other use cases for this as well in the future. Creew, which uses it to detect pit reuse. So it has a function that is called detect pit reuse, which is a hack, correct me if I'm wrong. And we can switch out this function for pit-of-Ds, using it to reliably track processes. Android, a low memory killer demon, is using pit-of-Ds as well. So we're actually one of the first that got really excited about this. It all derived back from a debate. So parts of that, I had an argument with a discussion with Case Cook, and I think David Howell is a while back at Linux security summit somewhere in Edinburgh or something where we talked about various things that we should do or could do and how we should do it, which is where the pit-of-D stuff started. Then there was also in parallel a discussion that started on the mailing list where people started to hack around in proc to make it at least possible to send signals via files and so on. So there was a lot happening at the same time, and this is ultimately the approach that came to fruition. So the Android guys, you had to give it to Joel, who works for Google as a kernel engineer who helped with this work, and who also did a lot of polling work that we'll see in a bit. So no memory killer demon is using pit-of-Ds to reliably track processes and kill them. This basically runs on Android 10 already. Oh, no, not pit-of-Ds. Sorry, they will be in Android 11. They back ported it to all of the LTS kernels, but they're definitely going to use it to get around issues where they have to make sure that the process that they're killing is really the process that they want to kill and so on. And BPF trace, once the switch to pit-of-Ds as well, they have an issue open as well. Don't just trust me, click on the links to verify it. Maybe they're all wrong. And there's prior art. This goes back to a former slide, to an earlier slide. Why do this? In this specific way, well, there is precedence in other systems, and I usually think it's not a good idea if you keep deviating too much between different operating system implementations because it makes it horrible for user space that at least try to be compatible across different operating systems to write working code. So I always was under the impression, I have to admit I did something which is not very smart from the perspective of a kernel guy. I didn't look at other kernels before actually starting this work. I did it later. And we were kind of lucky that we didn't get run in a lot of issues today originally, run into, but that was just by pure chance and having a lot of smart people yell at our work. But it also means I was falsely under the impression, for example, that Solaris had pit-of-Ds, which is wrong. They actually have it not, don't have it. At least the Lumos, the open source implementation of Solaris, only has a pure user space emulation of stable process handles, proc open, proc run, proc close, and proc free, but they have the same problem essentially. Open BSD and net BSD don't have it. Free BSD is the only system other than Linux that has it. They have a concept called proc file descriptor, or proc descriptor, sorry. And they have three system calls, PD fork, PD get, and PD kill, and they have gone with the different decisions or with which they have taken slightly different decisions than we have on Linux, parts of which are implementation-based, and most of which are implementation-based. And if you have questions about this, I can go into detail, but probably not going to be enough time. PD fork gives you a back-of-pit file descriptor, essentially, a proc file descriptor. And PD get-pit allows you to translate it to a pit, and PD kill is used to send a signal through one of those file descriptors. And on Linux, there are multiple approaches to get this into the kernel at once, 450, which is, I think, was originally done by Tiago as well from Qt, or at least was one of his suggestions. And then there was another patch set, which is called Clonavd. None of those made it. And I think one of the reasons, the patches were fine. They were interesting, had interesting new concepts. But for example, Clonavd tried to do a lot of things at the same time. So they mixed auto-reping semantics with file descriptors for processes and so on. So a lot of contention going on on how to actually do this correctly. And I think ultimately it didn't go in because it tried to do as much, too many things at once. Maybe there was another reason. I didn't see it from the thread, but there was a lot of stuff in that patch set. So okay. So we started building a new API around process management. And I want to start with this right away, which I also did at LPC. My intention has never been to say we have PITFTs and we have PITs, and they are totally separate worlds, and you either use the one, but then you can't use the other, which I think is the wrong way to think about this. PITFTs get around a very specific problem that you have in multiple, but rather specific situations. So you probably want the way to cross between PITs and PITFTs and use it both at the same time. So it's not like we're deprecating the PIT API and only going to PITFTs in the future. That's probably not what's going to happen. It may still, it may be the case, and this is what I expect, that a lot of new interesting features that people care about can be built upon PITFTs just by being a stable process handle, which you couldn't build upon PITs. So the first thing that we did in 5.1 was implement a new syscall, which is called PITFT send signal, which allows you to send a signal through a PITFT. This was the really obvious piece, because it's PIT recycling is usually concerned with sending signals, right? You're operating on something that is not yours. So it's pretty obvious to make the argument, look, this solves an issue. This lets us cleanly solve an issue. It's clearly something that user space has run into. There are a bunch of people who point this out that this is an issue for them. So we should do this. And it's actually in a lot of codes you can see here. Oh, there's a bit more to it down below, but the whole FD handling part is encapsulated in what you see here in the top. The first controversy that we had about this was what exactly is a PITFT going to be. People had very strong opinions about this, which also derives from the fact that it's an obvious problem in the sense, oh, it solves something really obvious. Here's my opinion and here's how we should do it. I'm not going to back down. So we kept yelling each other for a long while, which is usually what happens. And the first idea was to use a slash proc slash PIT to dear of these as PITFTs because they already pinstruck PID. They're pretty easy to get by. And it's a nice shortcut. You open, you call open on slash proc slash PID for the process you're interested in ignoring for now that this also has PIT recycling issues. Then you have an FD. It can't be stolen from you. Then you stuff it into PITFT sense signal and you send a signal to it. And if the process has exited behind your back and it's not around anymore and you send a signal, you get ESR which is Colonel Speak 4, no such process. So this brings me to another point. We don't pin PIDs. We don't pin PID numbers. It doesn't mean that when you hold a PID of D that now your PID is not going to recycle. Your PID is going to be recycled. We don't care about this. DFT is your stable handle. PID can be recycled. We're not stopping the Colonel from doing PID allocation or something. So right, we use proc PID as a shortcut of these, a sort of shortcut that is really handy for user space. But then we started, or we already had thought about this. We were faced with implementing the part where you return a file descriptor from one of those nice forklown functions that we have on Linux. And here is where we ran into real interesting problems. So this is work. So Jan and I started discussing about this because he had good input on this. And some people have the opinion, clone should just return file descriptors from slash proc slash PID. Sounds straightforward if you ignore all of the security issues. Like for example, there is a net directory in proc PID which allows you to snoop on the traffic of another process. It's also really horrible in terms of how file systems, especially the proc file system works in the Colonel. So if you want to return a proc PID file descriptor, you have to preallocate a dentry. You have to preallocate something that the Colonel uses internally to refer to a file. And then later on splice it into proc. Believe me, it was really nasty code. So what we did was we showed our preferred implementation and we showed the implementation that some people preferred and wrote both implementations, which was a lot of work. The one implementation showed slash proc PID file descriptor uses PID of these and the other one showed our implementation. And that was like really, if you compare it as like this much code and then you have this much code. And so people were like, yes, let's go with the implementation that you wanted, which was lucky for us. I think it's safe. There's a lot of headache and I would have been very unhappy. Actually, I considered if we go with proc PID directories, if I'm going to abandon this. But we didn't. So we got lucky. So in 5.2, this is where we landed support for returning file descriptors from the Clone Function. We were always under the impression that all of the Clone Flex were gone. Ha! No, there was one left. Which no one knew. I mean, we only saw this because Linus pointed it out. No, there is one flag bit left. I always assume we're out. But okay. So we added a new flag called clone PID of these, which creates PID of these at process creation time to completely let you get rid of the race where your PID can be recycled. They have a bunch of interesting properties. I'm going to talk about this a little bit. This is more or less, there are a bunch of more places that we had to touch or that I had to touch. But overall, this is the code that you see in the kernel internals clone function or fork function in this case. You know the comment for now, but if the flag is set, you allocate a new file descriptor and your return is user space. It stashes away a reference to a struct PID, which is the kernel internals notion of a stable reference on a process file descriptor. And you see these are a non-inode file descriptors. If you use a timer FD, if you use an event FD, if you use a signal FD, if you use an FD for the new amount API, and there is probably a bunch more I'm forgetting, the second notify FD, they're all anonymous inode based, which is just a single inode in the kernel that gets allocated when the kernel boots up and then you can get a new file from this. So the inode number is the same for all of them. FD means you don't need to allocate an inode and that's why it's very cheap. It doesn't waste memory, it doesn't waste allocation time and so on. So this is ideal more or less. You stash away a bunch of operations you want to allow in that file descriptor. So this is the PID of the FOP stuff. PID is the struct PID that I talked about. So it's pretty simple code overall. And one of the things that we also did is we made PID of these close on exec by default. So if you get a PID of D back, you really want to be sure that it doesn't leak into the child process when it execs, for example. And yeah, so this is what we did. And I think if you think about any new file descriptor type that you bring into the kernel, please make it close on exec by default. I tried to convince people to do this for the mount API, didn't fly well. But it really helps user space. It's one of the major pain points actually that you get file descriptors that stay open after you exec. And close on exec is really easy to set. Like, this is all it takes. So we'll read, write, occloexec and then you get an occloexec file descriptor back. Really makes life easier in user space. And another property is that we added an FD info file, so the FD info file will contain the PID of the process as seen from the PID namespace with which your proc mount was mounted. So any proc mount, especially in containers, is attached to the container's PID namespace, if you remount it at least. And in a new PID namespace, the PID that the process has will be different from the one that it has in one of the ancestor PID namespaces. And so we write it in there. So if you parse it, you will get the PID of that process in your proc instance, which is, for example, helpful if you send around to PIDFD and it was created in another PID namespace. And this is how you get the PID. But we also made it such, and this was Alec's idea, Alec Nesterhoff's idea. Originally we had it implemented in a way that when you set clone PIDFD, you got a file descriptor instead of a PID, which is problematic in multiple ways because file descriptors start at zero. And zero is obviously used to differentiate between the child and the parent. So you cannot really return zero as a valid file descriptor. So PIDFDs would have started at one, which is not nice. So for legacy clone, we made it such that we abuse one of those return arguments it has where usually the TID of the parent process, the TID of the child process is placed to return a PIDFD. So if you set clone PIDFD, you get the PID back and you get the PIDFD back at the same time. So you have no disconnect, you know both at the same time, which I think is pretty nice. It's even a little bit nicer, I think, than FreeBSD's PD fork. And then in 5.3, I think this is really exciting. We added Joel added polling support. This is something that they really wanted for the low memory killer demon so that you can get exit notification for non-parents. And in a more complex sense, it allows you to turn off the exit signal, which means that when a process exits, you can tell the kernel already today that I don't want a sick child. I don't want a sick child signal. I'm not going to explicitly ignore it because then you would order it, but I just don't want a signal when the process exits, which was a bit problematic because then how you know when the child is exited and so on. And the polling support will allow you to do this because what it essentially does is as soon as the process exits, so it's a threat group leader exits and the threat group is empty, then you get an exit notification saying, I'm ready. I have exited. So if you hand up one of those pit of these to a non-parent process, you get reliable exit notification and you don't need to rely on sick child signals and so on, which is pretty nifty. Also, pretty small code. It's not a lot to do. This is actually in two different files. It doesn't matter. You can grab for it if you're really interested in it. It's the polling implementation. So yeah, the one caveat that we have poll is only when the whole threat group exits, if the threat group leader exits before a lot of threats in the group, then polls should block similar to the Wade family. That's actually a problem you can run into and that's why threat group leaders are not really nice. But yeah, polling support is pretty exciting for process management. And in 5.3, we added another syscall, right, another syscall, PTIFTI is without code, PTIFTI open, the idea being that when you have forked the process, you sometimes still want to create a PTIFTI, especially if you want to watch a bunch of other processes, PIDs, and you can't rely on them using clone PTIFTI. PTIFTI open will allow you to do just that. It gets you a new PTIFTI. It also verifies that you give it a threat group leader so you can't get a PTIFTI for a threat. Yeah, that's 5.3 and 5.4. Excellent. That was proposed. That's actually no longer true. And it's pulled it from me, so we have that in. You can now wait on processes through PTIFTIs. So you can pass, wait ID has gained a new flag, PTIFTI, you pass it a PTIFTI instead of a PID, and then it retrieves it and then it waits on the PTIFTI, which I think is pretty neat. And we have a bunch of work plans. There are some work that I've, there's a lot of work that, or ideas that I have, I'm just going to speak about two of them. This at least came up as one of the original ideas, or the first ideas that we had was to make it possible similar to what FreeBSD has. So on FreeBSD, if you have a PID file descriptor, if you have a process file descriptor, it is a key line closed by default, which means if you close the last file descriptor, which has a reference to the corresponding struct file in the kernel, then it will kill the process that's, it killed the process, which is pretty neat. I wanted to do it the other way around, so per default, that's not the default. So if you close the last PIDFD, then still things are still fine. But we could add a flag that is set at process creation time, you can take it away afterwards. So no PR cuddles, or the PR cuddles, or that nonsense. And then you kill the process when the file descriptor is closed. There is a problem though that is, has been bothering me and that I've been thinking about this. So on FreeBSD, closing file descriptor is synchronous, so that means when closed returns, and that FD was the last reference to the struct file inside of the kernel, then by the time closed returns, you can be guaranteed that all of the cleanup operations have finished. Linux is smarter in some ways, or let's say complicated maybe, in the sense that closed can return without the corresponding release or cleanup method that belongs to the struct file that the last FD referred to has been run. It adds it to a work queue, or a K thread. And then at some point when the kernel thinks, ah, it's fine, I have some time to run this right now, then it cleans up everything. So that means if you close the last FD, closed returns, you are not guaranteed that the process is dead, which is not ideal, but usually the kernel cleans it up really quickly, or calls the release method for the corresponding file fairly quickly. It should only delay it when there's a lot of memory pressure, for example, in which case you screwed anyway. And another thing which is for the shared library case is exclusive waiting. So right now, anyone can still wait on the process that you've forked off, either via PID or via PIDFD. There is no differentiation there right now, and I think that just makes sense because then you have this connection between the PIDFD and the PID API. But there should be a way to say, I'm now going to separate this connection. And exclusive waiting would allow you to do this. So you have something like clone wait PID, flag name, open for discussion, which hides the process from generic wait requests. This is similar to what FreeBSD has as well, but I actually would like to make it stronger, which derives back from a discussion I had with Eric a while back. So you would only be able to wait on a process through a PIDFD, as long as there is a PIDFD referring to it, and when the last PIDFD is gone, the process gets auto-reaped. You know auto-reaping semantics? Thank you. So auto-reaping semantics, really, it's not, you explicitly, you set SIG child, you ignore SIG child explicitly. You say, I don't want SIG child, but I explicitly tell you that I don't want SIG child. At which point the kernelist says, OK, then I'm going to clean up that process for you if it exits, then it exits. And it's gone, which is really neat. It's different, for example, on FreeBSD, which is why they have chosen to implement proc file descriptors a little bit differently. If you explicitly ignore SIG child on FreeBSD, then the process doesn't get auto-cleaned up by the kernel. It gets re-parented to PID1, and PID1 then gets a signal for that process, which is basically saying, like, I'm done, you take care of it. Whereas Linux really cleans it up. So it would be really nice if you have clone weight PID. As long as you have a PID of D, you can wait on the process explicitly. If you close the last PID of D, telling the kernel, I don't care about this process anymore. If this process exits, just clean it up. The problem is the implementation, usually. It's pretty tricky to get right, I think. But I might put a patch set up there soonish. There's a bunch of other ideas, but I could keep talking, but I probably shouldn't. So, yeah, this is what we built over the last couple of kernel releases. We were at 5.4. We've obviously also been fixing a lot of bugs along the way. So this wasn't a whole bug-free process. But overall, I think it was pretty good. It was also a pretty good collaboration effort. A lot of people took part in the discussions, brought in really good ideas, gave reviews and helped with this. So it could probably give a shout out to a lot of people here. But yeah, that's about it. So if you have questions, go at it. Yes. So what about integration with CM credentials and other places where you send a PID to someone else? What do you mean? Could we have a flag where a CM credentials contains a PID FD instead of a PID? Can you just send it as a regular FD? Why you want a special flag? Because it gets sent implicitly. You set some flags on the socket and then the kernel does the job for you. And it is possible that by the time the consumer looks at this data, the PID could be not valid anymore. So you're saying the PID is sent implicitly, but instead of a PID, do you now want a PID FD? Well, I don't want to, from the top of my head, I don't necessarily see a problem with this. Yes? I guess the problem is, for instance, like a process sends a log message to Giorno and dies. So at the time, Giorno wants to look at the process to figure out which C group is running, which services. I remember this. Okay. Right. I'm trying to think if, yeah, we should probably talk about this. It shouldn't be something that is, I wouldn't put this off the table. It sounds useful if there is a really good use case for this and it doesn't really complicate in kernel code too much. So I have reason to, like, a good cost to justify this. It should be fine. I have a similar question, which is we now have PIDFDs and namespace FDs, but for the namespace SD, we have to go through the file system, procFS, to get them. Is there a way to derive namespace FD from a PIDFD? Maybe I have plans for that. So it's official request. Can you do that? One of the things that has always bothered me is if you do a set an S into a namespace, you have to do it iteratively, right? So it actually iteratively in two stages. You have to call open like seven times nowadays. And then you have to call set an S seven times and often in the correct order. And that's obviously a problem. Well, I see it as a problem. Maybe some people don't see it as a problem. I think this is the wrong approach. Ideally we could change. I had one set the idea, played with the idea. I may have mentioned on the mailing list at some point, two is if set an S would take a PIDFD and interpret the type argument that it has right now as a flag argument so you could specify the namespaces that you want to attach to. And then it derives it from the PIDFD in kernel and gets you into all of those namespaces, which would make attaching for containers and so on way nicer. Yes, that's definitely something which has been on my mind. It's usually that now it becomes a battle of the maintainers that I would like to call it. Because then it's, you know, we need to agree with does Eric think this is a good idea? How exactly does the API should look like and so on? But overall, yes, that's something I definitely have thought about. I also have thought about just recently forking into namespaces. Actually, I shouldn't claim this completely for me. David Howell suggested this once in a discussion. So ideally at clone time you say, not just create me a set of new namespaces, but create me a process in this set of namespaces. But there needs to be strong justification for why we would need this. Like is there some security issue or something when you create a process and then you do all of this stuff on it and so on. Yeah, that's definitely something we can think about. Yes. Hello. A question on kill on close feature. Yeah. So is it dangerous if the process and shall process exit a set you ID program on become more privileged and then the parent can kill it? Can the parent then kill it? Yes. In the naive implementation that I prototyped, yes, because it's an internal signal, right? So there is no security disconnect. So yes. So the question is the answer is yes, it would just kill the set you ID program that you spawned, even if it runs with more privileges for now. Even haven't thought about this, but I haven't worked on this in a lot more detail. If you have specific concerns about this, we should probably talk. No, that's not important. Just curious. So one of the things that I really want and is important I think is we have a tendency on Linux and this may be a good thing in some situations to good things. In some situations it's a bad thing where we, for example, we create, especially with a specific property and then later on we add a PR cuddle, and then we do a PR cuddle and that takes that property away from the process, which is horribly annoying. If I as a parent say I'm creating a new process with these properties set, then this property needs to stick to this process. It can't be taken away anymore after the fact, which is the thing I want to do with the clone weight pit, for example, the close on kill flex. It's a property that sticks to the process as long as it's alive and if it's gone, it's gone. I don't want to end up in a scenario where suddenly you can change all bits and pieces and flags again on processes. So I want sticky properties, essentially. I would like if it's useful at least. So Christopher Browner, thank you very much. Okay, so if you create a child process and then you send a PPD to that process to some other process, does the parent then get a sick child notification if the other process wants an exit notification that you just added in 5.3? So do both processes get notified when the process exits one over PPD and the parent over normal sick child? If you have said sick child, yes. If you want that, but you can explicitly turn it off. Okay, I wanted an parent. Okay, thank you. Yes. Anyone? Yes, so these are just a few remarks while doing this work. I forgot that slide. Okay, thank you very much.
you've been talking about this for the last couple of weeks and months already. Right. I'm worry, I'm going to go into details what other in the future, is to make it possible. It's not guarantees that you maintain a reference to the that we should do or could do and how we should recycle. Your PID is going to be recycled. We snoop on the traffic of another process. It's systems, especially the proc file system works in showed the implementation that some people and the other one showed our implementation. And properties. I'm going to talk about this a little is the kernel internals notion of a stable reference on a process file descriptor. And you see these are a non-inode file descriptors. If allow in that file descriptor. So this is the PID will be different from the one that it has in one PTIFTI open, the idea being that when you have a bunch of other processes, PIDs, and you can't pressure, for example, in which case you screwed PID. As long as you have a PID of D, you can wait possible that by the time the consumer looks at time, Giorno wants to look at the process to