Transactionally Protected Package Management
Johnson, Jeff
The talk will describe current development efforts @rpm5.org to add ACID properties to RPM package management. All operations involved in installing a package, not only the package metadata, but also the system calls, content, and scripts run by RPM will be logged within a transaction using a two-phase apply -> commit/abort so that all RPM operations can become stateless and invertible. The talk will specifically focus on and overview of the log management tools and the rpmdb schema changes necessary to use Berkeley DB ACID and transactional log extensions to add ACID properties to RPM package management. Connections with the parallel Mancoosi WP3 efforts to model/simulate package script behavior using a Domain Specific Language will also be described, time permitting.
 I'm Jeff Johnson and I'm the lead developer at rpm5.org. And also, part of the Mankuzi project, you just saw me meet many of the other members, you know, essentially for the first time. I'm a remotey since I live in the U.S. But Mankuzi is the best research I've seen on package management, you know, since forever. And with actually analyzing and modeling and looking at the details of package management and trying to scale package management into, you know, hundreds of repositories and thousands of packages and trying to make sure that the fundamental goal of package management of software installation is actually functional. Okay, so I recommend Mankuzi not just because, you know, I'm part of it, but it's the best research I'm aware of. And they're the only attempt that I know to approach Debian and RPM packaging, you know, neutrally. All right, for the last decade there's been an insane amount of effort wasted just because of packaging war differences. Okay, both of them are successful, RPM and D package, and they're just different, okay? And they're more alike than different these days. You know, the same problems are being solved. It's just one person says banana and the other person says tomato, but it's the same intrinsic problem. In any case, my talk is pretty much ad hoc. You know, I have no slides and so on. You know, as a developer I tend to have too many notes and not enough music, and I promise certain members that I wouldn't be boring. This is my third talk here at Fostom. My first one I made the mistake of providing too many details and I put a whole room to sleep. But my second talk I found that it was easier to find essentially three to five talking points and try and convey an overview of a very complex subject. The subject here is transactually protected package management. And those highfalutin words basically just mean trying to add acid behavior to make packaging more reliable. Acid is well known and if you are adding transactions to a database, the essential components are well analyzed. Acid, for those who don't know, stands for atomicity, consistency, isolation, and durability. And atomicity is usually achieved by locks. Almost everything has it. Consistency, if you are inconsistent, well, you die fairly quickly. So those are the rather easy aspects of acid. The parts that need to be added to package management are isolation and durability. Durability is perhaps the easier one because it just adds persistence and a persistent state machine on beyond just the action of an install. Package management is different from archivers because it tries to manage software over its entire life cycle. From building into distribution into installation into retiring it and legacy compatibility. So durability is trying to add a persistence to the store of the metadata on the installed machine so that it's known reliable. And it can be maintained in a consistent fashion. So isolation is rather more complicated because of the mixed problem case that a package manager has. There are basically three different types of information that need to be preserved when a package is installed. The easiest one, because it's an already solved problem, is the database acid. Databases have acid. We all know this. The second one is file systems. And file systems is where isolation becomes very tricky. File system by its nature is shared. I mean, there is no file system that really has any credible approach to isolation. What isolation is, is two processes shall not know that they're both about each other. There shall be no side effects that are shared. But that isn't how file systems work. File systems try and provide a consistent viewpoint onto a data store on a file system. And so another entity will see a file as soon as it's created. Since a package manager is installing on a file system, isolation is rather tricky to achieve. There have been several attempts to do this with, let's say, file snapshotting or copy on write and aspects like this. But acid will never be fully achieved on a file system until the file systems themselves start providing some sort of transactional protection. So it's actually parked, and this is probably an appropriate place to point it out, as part of doing due diligence on whether transactionally protected package management was feasible. One of the things I looked at is the two attempts that I'm aware of to try and do transactionally protected system calls. And one of them is the Valor file system, and this is the third implementation that is starting to perform nearly as well as EXT3. It's only three times slower. But when you add the logging necessary for transactions and transactional rollbacks and two phase commits, you're going to write more information. And so actually a factor of three slower is not too bad. But the Valor file system has transactions built within a kernel driver to, and the benefit for a file system is all sorts of race conditions are avoided if you can provide the I in acid isolation. All right. The other approach, which I'll just mention offhand is I think it's out of the University of Texas, and it tries to provide a begin and commit on a transaction around sets of operations. That's a second approach to essentially the same thing. Unlike a file system, a package manager can, speed is not as critical. RPM is not going to be compared to EXT3. It will be compared to D package. Okay. But D package is slow enough that it's possible to try and provide some isolation that on file system operations, all right, which is the second component. If I recover all my parentheticals, you know, there's acid for databases. There's acid for file systems. And the third part is the part that John Thompson talked about this morning, which is acid for scriptlets. And that's a really hard problem to solve. But the second component where isolation is very tricky is how to preserve all the system calls that are done by a package manager as it's installing content on a file system, all right. And just to summarize, you know, a rather complicated and roundabout way of talking about it, until file system provides isolation, a pack, RPM can't, all right. RPM does do things like installing files in a temp name and then renaming them in place. If the temp name is unique enough, it's unlikely that it will be, that there will be an interaction between two interacting processes. But that's about the best you can do, you know, with transactionally protected package management, is install into a unique name and then try and move it into place as easily as possible. So the three components for transactionally protected package management are database acid, system call acid, and scriptlet acid. The underlying mechanism is quite complicated, but it's not hard to understand. If you want a transactional log, you have to write a log record before you perform any operation. And if you have a log record, then you can also put a marker in the log that indicates that you completed successfully. If there is no marker, you can infer, go back to previous one. And that's essentially what happens with a rollback. Once you have that basic state machine for transactional logging, then the rest of it's rather easy. It's just moving between different, you know, consistent points. Anything in between can be inverted and discarded, which removes side effects. This will be one of the major benefits. I know it was one of the very first things I was asked, can RPM5 truly, if there's two headers in there, will the other one disappear? And this is one of the side effects, if there's an abnormal exit in the package manager, then there will be residual state that's left. And this often shows up, if you do a query, you'll have both the old and the new because you failed before the old could be removed. That will now be transactionally protected, and if that install a race pair does not achieve a commit, then the intervening information can be discarded reliably. So that the underlying store is known to be moving through time by definite checkpoints with known good behavior. Most of the rest of what I'll talk about, and the part that's complete now, I believe, although the really hard part about transactionally protected package management is to know that you have full coverage over all possible cases. That will only come with sufficient testing and usage. But what is there now is using Berkeley DB. RPM already uses Berkeley DB since all that transactionally protected package management relies on is acid behavior any database would do, but Berkeley DB already has all this. And so there's no reason to look at alternative forms, but any SQL database these days provides some form of acid behavior. The choice at RPM 5 was to use Berkeley DB. The underlying reason for this was Berkeley DB already has sample code that shows how to extend the database logs into other abstract events. Other abstract events, the sample case that's in there is if you have two system calls, makedir and removedir, they form an invertible pair. While you're installing, you make a directory, but if you want to roll it back, you remove a directory. And so it's very important that every action has its own inverse, but with sample code and an awkscript that generates all the glue layers, it's possible to extend Berkeley DB to handle system calls. RPM itself during package installation performs about 20 system call operations. And there are things like makedir and removedir and of course writing a file to extend Berkeley DB acid to file system acid. When if you're going to remove a file as part of installing a package, what needs to be done is to write the previous content into the log and then remove the file. So if you have to put it back, the content can be put directly back out of the log. Nothing else is needed except the log and something which can read the log. So my goal with transactionally protected package management, and I'm not there yet because there's a large number of pieces that still need to be implemented and tested, is to take a machine, take the log off it, take the executable which reads the log, remove everything else and put the machine, the entire machine, all state on the machine. Every single file exactly the same using only the log. That's the goal of this because RPM is expected to run in empty change routes and on bare metal machines and there is no stronger test of it, if you have a test of any implementation, then can you recreate anything, everything that's on the system and pass a diff test to make sure all the information is identical. I think that that is feasible, but I won't know until I actually get there. Yes. Sure. For the camera, the goal is remarkable. There is a lot of good behind what everything you're saying, but in order to do all this protection, how much overhead do I put over my file system? I'm installing 20,000 packages that takes 40 gigabyte and then I need to go and restore from the logs. It means that I need to have at least 40 gigabyte of logs depending on how many times I've been updating in between. So there must be some kind of optimization you have in mind to avoid that kind of overhead on the local machine. You give me a perfect entry point to describe what I've been working on for the last six months. I've had a number of problems with the schema that has been used by RPM for years. I've never really been able to tune Berkeley DB performance to try and minimize the overhead that you're mentioning. So a large part of the work was preparing the infrastructure to take in a Berkeley DB transactional machine. But in order to succeed at that, I have to be no slower than I was before. So I had to fix a lot of the flaws in the current schema. I've achieved at least a tenfold improvement in performance, which since for logging operations you essentially have to do everything twice right once the log and once under the file system, well, 10 is bigger than 2. The way that this has been done is to, Berkeley DB will use an M-map to move information from the database. RPM also does M-map.io. So the second copy is not that expensive. Almost all of the overhead of reading a database once the resource cap is changed to permit, up to let's say a quarter of the machine to be memory mapped, the I-O overhead on a database largely disappears. Writing to a second location from a memory map is also not that significant in overhead. There is still the cost of the log itself. And writing content in the log, if you actually try and preserve those logs over time, it becomes quite significant. At the moment, what I am doing is every successful RPM operation creates a checkpoint. By creating a checkpoint, I'm establishing a base that determines how far back in history do I want to go. And clearly nobody cares what was done on a machine one year ago. I mean, people care, but not most users. And you don't need gigabytes of logs going back forever in most cases. The cases where you do care, I mean, it's important and you'll allocate resources to maintain those logs, but not for an end user machine. What I'm seeing is that by paying attention and changing the schema, and I'll describe some of the details in a moment, because that is the most important point. Things like rebuildDB are not the right thing to do if you have a transactionally protected database and are no longer necessary either. What I'm actually measuring is that it's quite feasible to switch from the concurrent access model that RPM is using now to a transactionally protected module if the schema changes at the same time. And there's measurable increased lock overhead, but the overall wall clock measurements are about the same. And the saving the previous transaction is probably a tolerable saving, so that if you have an interrupted action that you can restore the metadata store and take out the side effects of a broken install is probably worth the effort. But that's not for me to say. This is my opinion, okay? I mean, and I do have deep mixed feelings about logging and transactional protection. There's basically two approaches to trying to improve reliability. One of them is what I'm talking about, where you have a log on the local machine. The other one is called, I call it provisioning, and that's basically you take the state off the machine and put it someplace. This is basically what Red Hat Network does. This is the sort of Google model, and I call it provisioning because you can always recreate the machine. And provisioning as an approach to the same problem has two major benefits. First of all, it's always off the machine, so you never have to worry about hardware failures and things like that. And second, it's usually centrally located, which minimizes the maintenance cost of it. The disadvantages of a provisioning approach is you often don't have the sufficient number of details to track all the detailed state. This is better done on a target machine, on a client machine, and with a log. There's a balance point someplace between those two, all right? But RPM and package management in general is a client only. It's not tied to a server, so I'm solving that problem, but the other one exists, and there's nothing better or worse with it. Does that answer the question? Yes, absolutely. Okay. Since I just described the two sort of general alternatives, provisioning versus transactional protection, I probably should say, and describe some of the other approaches. One of the approaches has been from Nexento with ZFS, and YUM in Fedora is attempting to use BitterFS and FileSystem snapshots. These are definitely viable approaches. The problem is that if you're going to use something like a ZFS or BitterFS snapshot, you first are stuck with some of the artifacts of being a FileSystem. ZFS can perform snapshots per mount point. A package can install on several mount points, and so you have this snarl of package side effects across multiple mount points that makes it more complex than otherwise. The intrinsic flaw is if you're going to use a FileSystem approach to, let's say, minimize the number of writes or overhead or something like that, is that you must have that FileSystem in the kernel, and neither BitterFS or ZFS are sufficiently widely deployed. RPM5 runs on every Unix in site. It's not in Mac OSX and lots of other places, and I'm finding myself talking to people about even doing RPM on Windows. You can't bet on things like BitterFS or ZFS as a general approach. That's part of the reason why I decided to just use BerkeleyDB rather than to rely on a snapshot in a FileSystem. A snapshot is basically a cheap, it's essentially a checkpoint on a FileSystem with a copy on write attached. So at the checkpoint, the current state is known, and anything that changes from that point is tracked separately with some sort of overlay, if you will. That's the way I pictured a FileSystem snapshot. A log internal to RPM is essentially the same operation. The real difference in the performance measurement is basically what you choose to count, how you choose to count the resource usage, like a clock ticks or a number of blocks of I.O., but all logging approaches are going to be the same. You have to write a copy before actually performing some operation. But the point I was trying to make is that Transactually Protected Package Management is not the same as a FileSystem snapshot. These are complementary approaches to essentially the same problem. Another thing that Transactually Protected Package Management is not about is, let's say, a representation of the log. If you're trying something like the modeling that Mancusi is doing, you often have to try and come up with a reasonable format. CUDF is one. I'm quite sure that there's another one coming from Mancusi Work Project 3 that describes the elements of a rollback. Each one of these is a different format. Each one of these is a different representation of the same information. But describing things differently is, in a different type of markup, for different usage cases is not exactly what the... It's a different meaning of log, and that's not what I'm talking about with Transactually Protected Package Management. The state machine for rolling back a transaction is actually quite different than a package state machine, which does erase the old package and update with the newer content. So the log is pretty much internal, and the representation can always be retrofitted later. Since I'm reaching about 15 minutes, let me go through some of the details that have already been implemented with database rather shortly. But one of the important elements is that there are tools that are going to be needed that will be external to RPM. The most important, if you bring nothing else away from this talk, if you're trying to use a transaction model, RebuildDB won't help you, but the alternative is DBRecover minus EV. Since most of the world is fixing RPM side effects by attempting RebuildDB, it won't hurt you, but it's not going to save you. It's going to save you. So the most important element of RebuildDB is the log management, which is the most important element at reducing log size. The tool needed is DBCheckPoint, and that puts in a boundary, which in time, that says this is the last known good, but the end of older logs, the tool is DB Archive, and you will need tools to manage your database. I haven't yet even considered the question of compatibility with older versions of RPM. Compatibility is going to be rather tricky because essentially lots of the elements that used to be in RPM no longer exist. Most specifically is there are no join keys. These are integer identifiers for packages that were monotonically increasing, and since those don't exist, I'm not sure how compatibility can be meaningfully defined. I hope to do it by providing conversion. You know, conversion from one to the other is an acceptable way of achieving compatibility, but that's not the same thing as interoperable compatibility. All integers are now saved in network order. This means cross-platform is now meaningful. Previously, RPM saved in native order all the time. Network order permits binary strings and integers to be dealt with equivalently using a mem compare, which is important for B tree access, and that also is a performance increase by switching to network order. One of the other incompatibilities that's been introduced is that BerkeleyDB provides a counter that does, that monotonically increases. Previously, this was done with a weird hack by using record number zero in the primary store as if it were an instance counter. By using a persistent number, there is now a identifier that is durable over time. Some of the other performance improvements is I'm now able to do pattern retrieves on keys, which means this is much higher performing than what RPM has done in the past, which is a secondary lookup with a header load and marshalling and signature checking that the actual database side effects of the retrieval. This is where most of the performance improvements that I've achieved reside. One of the other sort of deep fundamental infrastructure changes is that signatures and digest are no longer verified. There hasn't been a problem that has been detected with a signature or digest check on a database blob for quite some time. What I'm doing instead now is using M-Map and Prott-Read to protect the blob when it's read. There's pretty clearly no reason to verify the digest and signature on a blob of binary data that is hardware protected. That removed a very large amount of overhead reading headers from a database. It's a big chunk of the tenfold performance improvement that I've gotten. That's about because of time. I'm much more interested in questions. If you want me to continue, I can go into some of the way that Scriplet acid will be achieved. Having heard John Thompson's talk this morning, that rapidly got changed since 10 o'clock this morning, but I can try and flesh out some of the details that I think were the connection between the work that Jim Thompson and Paolo and Kaisha Magica are doing is going to fit into this. Or I can also just try and field some questions. It's up to you. Are there questions? Yes. I have another question. How important it is for you, backward comparability? You mentioned that the new version you are developing now of RPM breaks compared to the old one. You need to provide a method to transition from one to another. Isn't it sometimes necessary to just break and say, now we start clean, so to speak, and move on? Well, compatibility depends on what's being talked about. And as does break. I'm not trying to break things without reason. Most of the problems with the RPM schema have known for a long time, and I've lived with that. But yes, there's times to break. I've just been notified five minutes. Any other questions? Okay. There you go. I might be a bit off here because I'm not really involved with RPM distribution. But as I understand it, it's also an option within RPM to do scriptlets. An installation. Maybe you covered this. How do you plan to roll back those scriptlets? They may be creating files. The talk by Jim Thompson, John Thompson this morning was about using a DSL to break a scriptlet into smaller pieces. The fundamental problem with scripts in packaging is that they're too coarse-grained and they're opaque. The general case is you have side effects. And the side effects need to be captured somehow. The benefit that I see from the DSL is that it will be possible not just to look at the exit code from a script, which is all the information that is being returned, but it will be possible. It was even in one of John's examples that he illustrated that there were two operations within a single scriptlet. And that's fundamentally the problem. If you have a configuration change and a Daemon restart and other side effects, it's possible to have one of them fail and it being unimportant while the other ones are absolutely essential to the operation. So by having finer grain, then it becomes possible to come up with partial failures and partially invertible failures. The other benefit will be in simulation and modeling. Because scriptlets are often in shell, it's painful to do anything other than de facto testing of the scripts by simulating and modeling other properties such as ensuring that prerequisites that aren't specified in the ordering of the package are actually met. And what I hope to do is to use that DSL, breaking it up into smaller pieces, and map those pieces into a log as a starting event and a failure event. The other more traditional approach is scriptlets run in an environment, and that environment needs to be captured in the log. What the environment variables were, what the prerequisites are to some extent that provides a level of acid and invertibility. The problem is that this is a general programmatic language, and so things like typos cannot be tolerated. But these are not very interesting problems. They are important problems that have to be solved, but it is a very hard problem to get acid underneath scriptlet behavior. Does that answer the question? Almost. There's just one more thing, actually, what I basically meant is if one of those scriptlets creates a file or updates a configuration file or whatever, and you figure out later that you need to undo the upgrade for some reason, then you should basically maybe also undo what the scriptlet did. Sure. Scriptlets need to be invertible. Yeah. There needs to be an undo. The same invertibility happens when you install a file. If you're writing content or you're removing a file. Yeah, but there's probably some way for the scriptlet then to communicate to RPM that, hey, I've modified this file, or is that way? A better communication will be the means by which acid is achieved under scriptlet behavior. All right. And it's the same basic effect that you log any operation before it's performed. The complexity comes solely from using BornShell, which is painful. Right. And embedding is another approach that could be used, like either in embedded Lua, and then RPM is more aware of the actual operations are there. But the general approach is just to use the 55 operations in Dev Helper and to parameterize those. And that's a perfectly sound starting point because that's the order of magnitude of the side effects of package management. And this isn't a large corpus. Okay. Got you. Any other questions? Yes. What do you think about the NixOS package management system, which is a purely functional way, and which avoid most of the side effects? I like it a lot. I've been talking to Ilco. I started about a year ago. And from a package manager point of view, the single biggest failure mode at the moment in packaging is scripts. Buggy scripts, you know, that it's not only buggy, but scripts that aren't written generally enough. All right. And my private view is that I need to remove all scripts from RPM. If I say that in public and I didn't say it here, okay, if I actually remove all the scripts, I'm going to also appear to remove all programmatic means to change packaging. Okay. This is the same decision point that everybody faces, switching from to a functional model. You know, you have to think about the problem space very different. All right. From a package manager point of view, I have two alternatives. I go to the NixOS route and I say, no scripts and everybody hates me. And then I retrofit the same actions, okay, through other means. Or I take a slow and gradual approach to a functional, okay, where the first step is to use Dev Helper and then to break Dev Helper into smaller pieces. And, you know, moving gradually towards, you know, refactoring code, fewer side effects, and eventually achieve a functional model. I don't know what the answer is, but I like NixOS a lot. Is it something that you are going to try in Monkusi project to tend to? I can't speak for Monkusi, but I have tried to talk to the NixOS presentation. You know, I haven't yet found it, but, and I have considered this, you know, the approach that I will probably take will be to embed Haskell and try and do NixOS package management directly from RPM and to see whether that approach is viable. All right. But I'm well aware of what NixOS is, and I like functional programming. I don't do it, okay, but because it's just too hard to switch from one to the other. You know, I live in the real world, and I'm a C programmer. Okay. Any other questions? Time. How do I do? Thank you.
 I'm Jeff Johnson and I'm the lead developer at rpm5.org.  And also, part of the Mankuzi project, you just saw me meet many of the other members,  you know, essentially for the first time.  I'm a remotey since I live in the U.S.  But Mankuzi is the best research I've seen on package management, you know, since forever.  And with actually analyzing and modeling and looking at the details of package management  and trying to scale package management into, you know, hundreds of repositories and thousands of packages  and trying to make sure that the fundamental goal of package management of software installation  is actually functional.  Okay, so I recommend Mankuzi not just because, you know, I'm part of it, but it's the best research I'm aware of.  And they're the only attempt that I know to approach Debian and RPM packaging, you know, neutrally.  All right, for the last decade there's been an insane amount of effort wasted just because of packaging war differences.  Okay, both of them are successful, RPM and D package, and they're just different, okay?  And they're more alike than different these days.  You know, the same problems are being solved.  It's just one person says banana and the other person says tomato, but it's the same intrinsic problem.  In any case, my talk is pretty much ad hoc.  You know, I have no slides and so on.  You know, as a developer I tend to have too many notes and not enough music,  and I promise certain members that I wouldn't be boring.  This is my third talk here at Fostom.  My first one I made the mistake of providing too many details and I put a whole room to sleep.  But my second talk I found that it was easier to find essentially three to five talking points and try and convey an overview of a very complex subject.  The subject here is transactually protected package management.  And those highfalutin words basically just mean trying to add acid behavior to make packaging more reliable.  Acid is well known and if you are adding transactions to a database, the essential components are well analyzed.  Acid, for those who don't know, stands for atomicity, consistency, isolation, and durability.  And atomicity is usually achieved by locks. Almost everything has it. Consistency, if you are inconsistent, well, you die fairly quickly.  So those are the rather easy aspects of acid.  The parts that need to be added to package management are isolation and durability.  Durability is perhaps the easier one because it just adds persistence and a persistent state machine on beyond just the action of an install.  Package management is different from archivers because it tries to manage software over its entire life cycle.  From building into distribution into installation into retiring it and legacy compatibility.  So durability is trying to add a persistence to the store of the metadata on the installed machine so that it's known reliable.  And it can be maintained in a consistent fashion.  So isolation is rather more complicated because of the mixed problem case that a package manager has.  There are basically three different types of information that need to be preserved when a package is installed.  The easiest one, because it's an already solved problem, is the database acid. Databases have acid. We all know this.  The second one is file systems. And file systems is where isolation becomes very tricky.  File system by its nature is shared.  I mean, there is no file system that really has any credible approach to isolation.  What isolation is, is two processes shall not know that they're both about each other.  There shall be no side effects that are shared.  But that isn't how file systems work.  File systems try and provide a consistent viewpoint onto a data store on a file system.  And so another entity will see a file as soon as it's created.  Since a package manager is installing on a file system, isolation is rather tricky to achieve.  There have been several attempts to do this with, let's say, file snapshotting or copy on write and aspects like this.  But acid will never be fully achieved on a file system until the file systems themselves start providing some sort of transactional protection.  So it's actually parked, and this is probably an appropriate place to point it out,  as part of doing due diligence on whether transactionally protected package management was feasible.  One of the things I looked at is the two attempts that I'm aware of to try and do transactionally protected system calls.  And one of them is the Valor file system, and this is the third implementation that is starting to perform nearly as well as EXT3.  It's only three times slower.  But when you add the logging necessary for transactions and transactional rollbacks and two phase commits,  you're going to write more information.  And so actually a factor of three slower is not too bad.  But the Valor file system has transactions built within a kernel driver to,  and the benefit for a file system is all sorts of race conditions are avoided if you can provide the I in acid isolation.  All right.  The other approach, which I'll just mention offhand is I think it's out of the University of Texas,  and it tries to provide a begin and commit on a transaction around sets of operations.  That's a second approach to essentially the same thing.  Unlike a file system, a package manager can, speed is not as critical.  RPM is not going to be compared to EXT3. It will be compared to D package.  Okay.  But D package is slow enough that it's possible to try and provide some isolation that on file system operations,  all right, which is the second component.  If I recover all my parentheticals, you know, there's acid for databases.  There's acid for file systems.  And the third part is the part that John Thompson talked about this morning, which is acid for scriptlets.  And that's a really hard problem to solve.  But the second component where isolation is very tricky is how to preserve all the system calls that are done by a package manager  as it's installing content on a file system, all right.  And just to summarize, you know, a rather complicated and roundabout way of talking about it,  until file system provides isolation, a pack, RPM can't, all right.  RPM does do things like installing files in a temp name and then renaming them in place.  If the temp name is unique enough, it's unlikely that it will be, that there will be an interaction between two interacting processes.  But that's about the best you can do, you know, with transactionally protected package management,  is install into a unique name and then try and move it into place as easily as possible.  So the three components for transactionally protected package management are database acid, system call acid, and scriptlet acid.  The underlying mechanism is quite complicated, but it's not hard to understand.  If you want a transactional log, you have to write a log record before you perform any operation.  And if you have a log record, then you can also put a marker in the log that indicates that you completed successfully.  If there is no marker, you can infer, go back to previous one.  And that's essentially what happens with a rollback.  Once you have that basic state machine for transactional logging, then the rest of it's rather easy.  It's just moving between different, you know, consistent points.  Anything in between can be inverted and discarded, which removes side effects.  This will be one of the major benefits.  I know it was one of the very first things I was asked, can RPM5 truly, if there's two headers in there, will the other one disappear?  And this is one of the side effects, if there's an abnormal exit in the package manager, then there will be residual state that's left.  And this often shows up, if you do a query, you'll have both the old and the new because you failed before the old could be removed.  That will now be transactionally protected, and if that install a race pair does not achieve a commit, then the intervening information can be discarded reliably.  So that the underlying store is known to be moving through time by definite checkpoints with known good behavior.  Most of the rest of what I'll talk about, and the part that's complete now, I believe, although the really hard part about transactionally protected package management is to know that you have full coverage over all possible cases.  That will only come with sufficient testing and usage.  But what is there now is using Berkeley DB.  RPM already uses Berkeley DB since all that transactionally protected package management relies on is acid behavior any database would do, but Berkeley DB already has all this.  And so there's no reason to look at alternative forms, but any SQL database these days provides some form of acid behavior.  The choice at RPM 5 was to use Berkeley DB.  The underlying reason for this was Berkeley DB already has sample code that shows how to extend the database logs into other abstract events.  Other abstract events, the sample case that's in there is if you have two system calls, makedir and removedir, they form an invertible pair.  While you're installing, you make a directory, but if you want to roll it back, you remove a directory.  And so it's very important that every action has its own inverse, but with sample code and an awkscript that generates all the glue layers, it's possible to extend Berkeley DB to handle system calls.  RPM itself during package installation performs about 20 system call operations.  And there are things like makedir and removedir and of course writing a file to extend Berkeley DB acid to file system acid.  When if you're going to remove a file as part of installing a package, what needs to be done is to write the previous content into the log and then remove the file.  So if you have to put it back, the content can be put directly back out of the log.  Nothing else is needed except the log and something which can read the log.  So my goal with transactionally protected package management, and I'm not there yet because there's a large number of pieces that still need to be implemented and tested, is to take a machine,  take the log off it, take the executable which reads the log, remove everything else and put the machine, the entire machine, all state on the machine.  Every single file exactly the same using only the log.  That's the goal of this because RPM is expected to run in empty change routes and on bare metal machines and there is no stronger test of it,  if you have a test of any implementation, then can you recreate anything, everything that's on the system and pass a diff test to make sure all the information is identical.  I think that that is feasible, but I won't know until I actually get there.  Yes.  Sure.  For the camera, the goal is remarkable.  There is a lot of good behind what everything you're saying, but in order to do all this protection, how much overhead do I put over my file system?  I'm installing 20,000 packages that takes 40 gigabyte and then I need to go and restore from the logs. It means that I need to have at least 40 gigabyte of logs depending on how many times I've been updating in between.  So there must be some kind of optimization you have in mind to avoid that kind of overhead on the local machine.  You give me a perfect entry point to describe what I've been working on for the last six months.  I've had a number of problems with the schema that has been used by RPM for years.  I've never really been able to tune Berkeley DB performance to try and minimize the overhead that you're mentioning.  So a large part of the work was preparing the infrastructure to take in a Berkeley DB transactional machine.  But in order to succeed at that, I have to be no slower than I was before.  So I had to fix a lot of the flaws in the current schema. I've achieved at least a tenfold improvement in performance,  which since for logging operations you essentially have to do everything twice right once the log and once under the file system,  well, 10 is bigger than 2. The way that this has been done is to,  Berkeley DB will use an M-map to move information from the database.  RPM also does M-map.io. So the second copy is not that expensive.  Almost all of the overhead of reading a database once the resource cap is changed to permit,  up to let's say a quarter of the machine to be memory mapped, the I-O overhead on a database largely disappears.  Writing to a second location from a memory map is also not that significant in overhead.  There is still the cost of the log itself.  And writing content in the log, if you actually try and preserve those logs over time, it becomes quite significant.  At the moment, what I am doing is every successful RPM operation creates a checkpoint.  By creating a checkpoint, I'm establishing a base that determines how far back in history do I want to go.  And clearly nobody cares what was done on a machine one year ago.  I mean, people care, but not most users.  And you don't need gigabytes of logs going back forever in most cases.  The cases where you do care, I mean, it's important and you'll allocate resources to maintain those logs,  but not for an end user machine.  What I'm seeing is that by paying attention and changing the schema,  and I'll describe some of the details in a moment, because that is the most important point.  Things like rebuildDB are not the right thing to do if you have a transactionally protected database and are no longer necessary either.  What I'm actually measuring is that it's quite feasible to switch from the concurrent access model that RPM is using now  to a transactionally protected module if the schema changes at the same time.  And there's measurable increased lock overhead, but the overall wall clock measurements are about the same.  And the saving the previous transaction is probably a tolerable saving,  so that if you have an interrupted action that you can restore the metadata store  and take out the side effects of a broken install is probably worth the effort.  But that's not for me to say. This is my opinion, okay?  I mean, and I do have deep mixed feelings about logging and transactional protection.  There's basically two approaches to trying to improve reliability.  One of them is what I'm talking about, where you have a log on the local machine.  The other one is called, I call it provisioning, and that's basically you take the state off the machine and put it someplace.  This is basically what Red Hat Network does.  This is the sort of Google model, and I call it provisioning because you can always recreate the machine.  And provisioning as an approach to the same problem has two major benefits.  First of all, it's always off the machine, so you never have to worry about hardware failures and things like that.  And second, it's usually centrally located, which minimizes the maintenance cost of it.  The disadvantages of a provisioning approach is you often don't have the sufficient number of details to track all the detailed state.  This is better done on a target machine, on a client machine, and with a log.  There's a balance point someplace between those two, all right?  But RPM and package management in general is a client only.  It's not tied to a server, so I'm solving that problem, but the other one exists, and there's nothing better or worse with it.  Does that answer the question?  Yes, absolutely.  Okay.  Since I just described the two sort of general alternatives, provisioning versus transactional protection,  I probably should say, and describe some of the other approaches.  One of the approaches has been from Nexento with ZFS,  and YUM in Fedora is attempting to use BitterFS and FileSystem snapshots.  These are definitely viable approaches.  The problem is that if you're going to use something like a ZFS or BitterFS snapshot,  you first are stuck with some of the artifacts of being a FileSystem.  ZFS can perform snapshots per mount point.  A package can install on several mount points, and so you have this snarl of package side effects across multiple mount points  that makes it more complex than otherwise.  The intrinsic flaw is if you're going to use a FileSystem approach to, let's say, minimize the number of writes or overhead or something like that,  is that you must have that FileSystem in the kernel, and neither BitterFS or ZFS are sufficiently widely deployed.  RPM5 runs on every Unix in site. It's not in Mac OSX and lots of other places, and I'm finding myself talking to people about even doing RPM on Windows.  You can't bet on things like BitterFS or ZFS as a general approach.  That's part of the reason why I decided to just use BerkeleyDB rather than to rely on a snapshot in a FileSystem.  A snapshot is basically a cheap, it's essentially a checkpoint on a FileSystem with a copy on write attached.  So at the checkpoint, the current state is known, and anything that changes from that point is tracked separately with some sort of overlay, if you will.  That's the way I pictured a FileSystem snapshot.  A log internal to RPM is essentially the same operation.  The real difference in the performance measurement is basically what you choose to count, how you choose to count the resource usage,  like a clock ticks or a number of blocks of I.O., but all logging approaches are going to be the same.  You have to write a copy before actually performing some operation.  But the point I was trying to make is that Transactually Protected Package Management is not the same as a FileSystem snapshot.  These are complementary approaches to essentially the same problem.  Another thing that Transactually Protected Package Management is not about is, let's say, a representation of the log.  If you're trying something like the modeling that Mancusi is doing, you often have to try and come up with a reasonable format.  CUDF is one.  I'm quite sure that there's another one coming from Mancusi Work Project 3 that describes the elements of a rollback.  Each one of these is a different format. Each one of these is a different representation of the same information.  But describing things differently is, in a different type of markup, for different usage cases is not exactly what the...  It's a different meaning of log, and that's not what I'm talking about with Transactually Protected Package Management.  The state machine for rolling back a transaction is actually quite different than a package state machine,  which does erase the old package and update with the newer content.  So the log is pretty much internal, and the representation can always be retrofitted later.  Since I'm reaching about 15 minutes, let me go through some of the details that have already been implemented with database rather shortly.  But one of the important elements is that there are tools that are going to be needed that will be external to RPM.  The most important, if you bring nothing else away from this talk, if you're trying to use a transaction model,  RebuildDB won't help you, but the alternative is DBRecover minus EV.  Since most of the world is fixing RPM side effects by attempting RebuildDB, it won't hurt you, but it's not going to save you.  It's going to save you.  So the most important element of RebuildDB is the log management, which is the most important element at reducing log size.  The tool needed is DBCheckPoint, and that puts in a boundary, which in time, that says this is the last known good,  but the end of older logs, the tool is DB Archive, and you will need tools to manage your database.  I haven't yet even considered the question of compatibility with older versions of RPM.  Compatibility is going to be rather tricky because essentially lots of the elements that used to be in RPM no longer exist.  Most specifically is there are no join keys.  These are integer identifiers for packages that were monotonically increasing, and since those don't exist,  I'm not sure how compatibility can be meaningfully defined.  I hope to do it by providing conversion.  You know, conversion from one to the other is an acceptable way of achieving compatibility,  but that's not the same thing as interoperable compatibility.  All integers are now saved in network order.  This means cross-platform is now meaningful.  Previously, RPM saved in native order all the time.  Network order permits binary strings and integers to be dealt with equivalently using a mem compare,  which is important for B tree access, and that also is a performance increase by switching to network order.  One of the other incompatibilities that's been introduced is that BerkeleyDB provides a counter that does,  that monotonically increases.  Previously, this was done with a weird hack by using record number zero in the primary store as if it were an instance counter.  By using a persistent number, there is now a identifier that is durable over time.  Some of the other performance improvements is I'm now able to do pattern retrieves on keys,  which means this is much higher performing than what RPM has done in the past, which is a secondary lookup  with a header load and marshalling and signature checking that the actual database side effects of the retrieval.  This is where most of the performance improvements that I've achieved reside.  One of the other sort of deep fundamental infrastructure changes is that signatures and digest are no longer verified.  There hasn't been a problem that has been detected with a signature or digest check on a database blob for quite some time.  What I'm doing instead now is using M-Map and Prott-Read to protect the blob when it's read.  There's pretty clearly no reason to verify the digest and signature on a blob of binary data that is hardware protected.  That removed a very large amount of overhead reading headers from a database.  It's a big chunk of the tenfold performance improvement that I've gotten.  That's about because of time.  I'm much more interested in questions. If you want me to continue, I can go into some of the way that Scriplet acid will be achieved.  Having heard John Thompson's talk this morning, that rapidly got changed since 10 o'clock this morning,  but I can try and flesh out some of the details that I think were the connection between the work that Jim Thompson and Paolo and Kaisha Magica are doing  is going to fit into this.  Or I can also just try and field some questions.  It's up to you. Are there questions?  Yes.  I have another question. How important it is for you, backward comparability?  You mentioned that the new version you are developing now of RPM breaks compared to the old one.  You need to provide a method to transition from one to another.  Isn't it sometimes necessary to just break and say, now we start clean, so to speak, and move on?  Well, compatibility depends on what's being talked about.  And as does break. I'm not trying to break things without reason.  Most of the problems with the RPM schema have known for a long time, and I've lived with that.  But yes, there's times to break.  I've just been notified five minutes. Any other questions?  Okay. There you go.  I might be a bit off here because I'm not really involved with RPM distribution.  But as I understand it, it's also an option within RPM to do scriptlets.  An installation. Maybe you covered this.  How do you plan to roll back those scriptlets?  They may be creating files.  The talk by Jim Thompson, John Thompson this morning was about using a DSL to break a scriptlet into smaller pieces.  The fundamental problem with scripts in packaging is that they're too coarse-grained and they're opaque.  The general case is you have side effects. And the side effects need to be captured somehow.  The benefit that I see from the DSL is that it will be possible not just to look at the exit code from a script,  which is all the information that is being returned, but it will be possible.  It was even in one of John's examples that he illustrated that there were two operations within a single scriptlet.  And that's fundamentally the problem.  If you have a configuration change and a Daemon restart and other side effects,  it's possible to have one of them fail and it being unimportant while the other ones are absolutely essential to the operation.  So by having finer grain, then it becomes possible to come up with partial failures and partially invertible failures.  The other benefit will be in simulation and modeling.  Because scriptlets are often in shell, it's painful to do anything other than de facto testing of the scripts  by simulating and modeling other properties such as ensuring that prerequisites that aren't specified in the ordering of the package are actually met.  And what I hope to do is to use that DSL, breaking it up into smaller pieces,  and map those pieces into a log as a starting event and a failure event.  The other more traditional approach is scriptlets run in an environment,  and that environment needs to be captured in the log.  What the environment variables were, what the prerequisites are to some extent that provides a level of acid and invertibility.  The problem is that this is a general programmatic language, and so things like typos cannot be tolerated.  But these are not very interesting problems.  They are important problems that have to be solved, but it is a very hard problem to get acid underneath scriptlet behavior.  Does that answer the question?  Almost. There's just one more thing, actually, what I basically meant is if one of those scriptlets creates a file or updates a configuration file or whatever,  and you figure out later that you need to undo the upgrade for some reason, then you should basically maybe also undo what the scriptlet did.  Sure. Scriptlets need to be invertible.  Yeah. There needs to be an undo.  The same invertibility happens when you install a file. If you're writing content or you're removing a file.  Yeah, but there's probably some way for the scriptlet then to communicate to RPM that, hey, I've modified this file, or is that way?  A better communication will be the means by which acid is achieved under scriptlet behavior.  All right. And it's the same basic effect that you log any operation before it's performed.  The complexity comes solely from using BornShell, which is painful.  Right.  And embedding is another approach that could be used, like either in embedded Lua, and then RPM is more aware of the actual operations are there.  But the general approach is just to use the 55 operations in Dev Helper and to parameterize those.  And that's a perfectly sound starting point because that's the order of magnitude of the side effects of package management.  And this isn't a large corpus.  Okay.  Got you.  Any other questions?  Yes.  What do you think about the NixOS package management system, which is a purely functional way, and which avoid most of the side effects?  I like it a lot.  I've been talking to Ilco.  I started about a year ago.  And from a package manager point of view, the single biggest failure mode at the moment in packaging is scripts.  Buggy scripts, you know, that it's not only buggy, but scripts that aren't written generally enough.  All right.  And my private view is that I need to remove all scripts from RPM.  If I say that in public and I didn't say it here, okay, if I actually remove all the scripts, I'm going to also appear to remove all programmatic means to change packaging.  Okay.  This is the same decision point that everybody faces, switching from to a functional model.  You know, you have to think about the problem space very different.  All right.  From a package manager point of view, I have two alternatives.  I go to the NixOS route and I say, no scripts and everybody hates me.  And then I retrofit the same actions, okay, through other means.  Or I take a slow and gradual approach to a functional, okay, where the first step is to use Dev Helper and then to break Dev Helper into smaller pieces.  And, you know, moving gradually towards, you know, refactoring code, fewer side effects, and eventually achieve a functional model.  I don't know what the answer is, but I like NixOS a lot.  Is it something that you are going to try in Monkusi project to tend to?  I can't speak for Monkusi, but I have tried to talk to the NixOS presentation.  You know, I haven't yet found it, but, and I have considered this, you know, the approach that I will probably take will be to embed Haskell and try and do NixOS package management directly from RPM  and to see whether that approach is viable.  All right.  But I'm well aware of what NixOS is, and I like functional programming.  I don't do it, okay, but because it's just too hard to switch from one to the other.  You know, I live in the real world, and I'm a C programmer.  Okay.  Any other questions?  Time.  How do I do?  Thank you.
Transactionally_Protected_Package_Management_wEZ9JKWt328_0_803.png Transactionally_Protected_Package_Management_wEZ9JKWt328_804_1823.png Transactionally_Protected_Package_Management_wEZ9JKWt328_1824_3563.png Transactionally_Protected_Package_Management_wEZ9JKWt328_3564_4043.png Transactionally_Protected_Package_Management_wEZ9JKWt328_4044_4907.png Transactionally_Protected_Package_Management_wEZ9JKWt328_4908_6431.png Transactionally_Protected_Package_Management_wEZ9JKWt328_6432_7235.png Transactionally_Protected_Package_Management_wEZ9JKWt328_7236_7943.png Transactionally_Protected_Package_Management_wEZ9JKWt328_7944_8939.png Transactionally_Protected_Package_Management_wEZ9JKWt328_8940_12107.png Transactionally_Protected_Package_Management_wEZ9JKWt328_12108_17639.png Transactionally_Protected_Package_Management_wEZ9JKWt328_17640_53399.png Transactionally_Protected_Package_Management_wEZ9JKWt328_53400_58811.png Transactionally_Protected_Package_Management_wEZ9JKWt328_58812_60743.png Transactionally_Protected_Package_Management_wEZ9JKWt328_60744_61187.png Transactionally_Protected_Package_Management_wEZ9JKWt328_61188_61247.png Transactionally_Protected_Package_Management_wEZ9JKWt328_61248_61343.png Transactionally_Protected_Package_Management_wEZ9JKWt328_61344_62447.png Transactionally_Protected_Package_Management_wEZ9JKWt328_62448_62567.png Transactionally_Protected_Package_Management_wEZ9JKWt328_62568_63851.png Transactionally_Protected_Package_Management_wEZ9JKWt328_63852_64391.png Transactionally_Protected_Package_Management_wEZ9JKWt328_64392_64607.png Transactionally_Protected_Package_Management_wEZ9JKWt328_64608_64715.png Transactionally_Protected_Package_Management_wEZ9JKWt328_64716_65051.png Transactionally_Protected_Package_Management_wEZ9JKWt328_65052_65524.png
I'm Jeff Johnson and I'm the lead developer at rpm5.org. And also, part of the Mankuzi project, you just saw me meet many of the other members, you know, essentially for the first time. I'm a remotey since I live in the U.S. But Mankuzi is the best research I've seen on package management, you know, since forever. And with actually analyzing and modeling and looking at the details of package management and trying to scale package management into, you know, hundreds of repositories and thousands of packages and trying to make sure that the fundamental goal of package management of software installation is actually functional. Okay, so I recommend Mankuzi not just because, you know, I'm part of it, but it's the best research I'm aware of. And they're the only attempt that I know to approach Debian and RPM packaging, you know, neutrally. All right, for the last decade there's been an insane amount of effort wasted just because of packaging war differences. Okay, both of them are successful, RPM and D package, and they're just different, okay? And they're more alike than different these days. You know, the same problems are being solved. It's just one person says banana and the other person says tomato, but it's the same intrinsic problem. In any case, my talk is pretty much ad hoc. You know, I have no slides and so on. You know, as a developer I tend to have too many notes and not enough music, and I promise certain members that I wouldn't be boring. This is my third talk here at Fostom. My first one I made the mistake of providing too many details and I put a whole room to sleep. But my second talk I found that it was easier to find essentially three to five talking points and try and convey an overview of a very complex subject. The subject here is transactually protected package management. And those highfalutin words basically just mean trying to add acid behavior to make packaging more reliable. Acid is well known and if you are adding transactions to a database, the essential components are well analyzed. Acid, for those who don't know, stands for atomicity, consistency, isolation, and durability. And atomicity is usually achieved by locks. Almost everything has it. Consistency, if you are inconsistent, well, you die fairly quickly. So those are the rather easy aspects of acid. The parts that need to be added to package management are isolation and durability. Durability is perhaps the easier one because it just adds persistence and a persistent state machine on beyond just the action of an install. Package management is different from archivers because it tries to manage software over its entire life cycle. From building into distribution into installation into retiring it and legacy compatibility. So durability is trying to add a persistence to the store of the metadata on the installed machine so that it's known reliable. And it can be maintained in a consistent fashion. So isolation is rather more complicated because of the mixed problem case that a package manager has. There are basically three different types of information that need to be preserved when a package is installed. The easiest one, because it's an already solved problem, is the database acid. Databases have acid. We all know this. The second one is file systems. And file systems is where isolation becomes very tricky. File system by its nature is shared. I mean, there is no file system that really has any credible approach to isolation. What isolation is, is two processes shall not know that they're both about each other. There shall be no side effects that are shared. But that isn't how file systems work. File systems try and provide a consistent viewpoint onto a data store on a file system. And so another entity will see a file as soon as it's created. Since a package manager is installing on a file system, isolation is rather tricky to achieve. There have been several attempts to do this with, let's say, file snapshotting or copy on write and aspects like this. But acid will never be fully achieved on a file system until the file systems themselves start providing some sort of transactional protection. So it's actually parked, and this is probably an appropriate place to point it out, as part of doing due diligence on whether transactionally protected package management was feasible. One of the things I looked at is the two attempts that I'm aware of to try and do transactionally protected system calls. And one of them is the Valor file system, and this is the third implementation that is starting to perform nearly as well as EXT3. It's only three times slower. But when you add the logging necessary for transactions and transactional rollbacks and two phase commits, you're going to write more information. And so actually a factor of three slower is not too bad. But the Valor file system has transactions built within a kernel driver to, and the benefit for a file system is all sorts of race conditions are avoided if you can provide the I in acid isolation. All right. The other approach, which I'll just mention offhand is I think it's out of the University of Texas, and it tries to provide a begin and commit on a transaction around sets of operations. That's a second approach to essentially the same thing. Unlike a file system, a package manager can, speed is not as critical. RPM is not going to be compared to EXT3. It will be compared to D package. Okay. But D package is slow enough that it's possible to try and provide some isolation that on file system operations, all right, which is the second component. If I recover all my parentheticals, you know, there's acid for databases. There's acid for file systems. And the third part is the part that John Thompson talked about this morning, which is acid for scriptlets. And that's a really hard problem to solve. But the second component where isolation is very tricky is how to preserve all the system calls that are done by a package manager as it's installing content on a file system, all right. And just to summarize, you know, a rather complicated and roundabout way of talking about it, until file system provides isolation, a pack, RPM can't, all right. RPM does do things like installing files in a temp name and then renaming them in place. If the temp name is unique enough, it's unlikely that it will be, that there will be an interaction between two interacting processes. But that's about the best you can do, you know, with transactionally protected package management, is install into a unique name and then try and move it into place as easily as possible. So the three components for transactionally protected package management are database acid, system call acid, and scriptlet acid. The underlying mechanism is quite complicated, but it's not hard to understand. If you want a transactional log, you have to write a log record before you perform any operation. And if you have a log record, then you can also put a marker in the log that indicates that you completed successfully. If there is no marker, you can infer, go back to previous one. And that's essentially what happens with a rollback. Once you have that basic state machine for transactional logging, then the rest of it's rather easy. It's just moving between different, you know, consistent points. Anything in between can be inverted and discarded, which removes side effects. This will be one of the major benefits. I know it was one of the very first things I was asked, can RPM5 truly, if there's two headers in there, will the other one disappear? And this is one of the side effects, if there's an abnormal exit in the package manager, then there will be residual state that's left. And this often shows up, if you do a query, you'll have both the old and the new because you failed before the old could be removed. That will now be transactionally protected, and if that install a race pair does not achieve a commit, then the intervening information can be discarded reliably. So that the underlying store is known to be moving through time by definite checkpoints with known good behavior. Most of the rest of what I'll talk about, and the part that's complete now, I believe, although the really hard part about transactionally protected package management is to know that you have full coverage over all possible cases. That will only come with sufficient testing and usage. But what is there now is using Berkeley DB. RPM already uses Berkeley DB since all that transactionally protected package management relies on is acid behavior any database would do, but Berkeley DB already has all this. And so there's no reason to look at alternative forms, but any SQL database these days provides some form of acid behavior. The choice at RPM 5 was to use Berkeley DB. The underlying reason for this was Berkeley DB already has sample code that shows how to extend the database logs into other abstract events. Other abstract events, the sample case that's in there is if you have two system calls, makedir and removedir, they form an invertible pair. While you're installing, you make a directory, but if you want to roll it back, you remove a directory. And so it's very important that every action has its own inverse, but with sample code and an awkscript that generates all the glue layers, it's possible to extend Berkeley DB to handle system calls. RPM itself during package installation performs about 20 system call operations. And there are things like makedir and removedir and of course writing a file to extend Berkeley DB acid to file system acid. When if you're going to remove a file as part of installing a package, what needs to be done is to write the previous content into the log and then remove the file. So if you have to put it back, the content can be put directly back out of the log. Nothing else is needed except the log and something which can read the log. So my goal with transactionally protected package management, and I'm not there yet because there's a large number of pieces that still need to be implemented and tested, is to take a machine, take the log off it, take the executable which reads the log, remove everything else and put the machine, the entire machine, all state on the machine. Every single file exactly the same using only the log. That's the goal of this because RPM is expected to run in empty change routes and on bare metal machines and there is no stronger test of it, if you have a test of any implementation, then can you recreate anything, everything that's on the system and pass a diff test to make sure all the information is identical. I think that that is feasible, but I won't know until I actually get there. Yes. Sure. For the camera, the goal is remarkable. There is a lot of good behind what everything you're saying, but in order to do all this protection, how much overhead do I put over my file system? I'm installing 20,000 packages that takes 40 gigabyte and then I need to go and restore from the logs. It means that I need to have at least 40 gigabyte of logs depending on how many times I've been updating in between. So there must be some kind of optimization you have in mind to avoid that kind of overhead on the local machine. You give me a perfect entry point to describe what I've been working on for the last six months. I've had a number of problems with the schema that has been used by RPM for years. I've never really been able to tune Berkeley DB performance to try and minimize the overhead that you're mentioning. So a large part of the work was preparing the infrastructure to take in a Berkeley DB transactional machine. But in order to succeed at that, I have to be no slower than I was before. So I had to fix a lot of the flaws in the current schema. I've achieved at least a tenfold improvement in performance, which since for logging operations you essentially have to do everything twice right once the log and once under the file system, well, 10 is bigger than 2. The way that this has been done is to, Berkeley DB will use an M-map to move information from the database. RPM also does M-map.io. So the second copy is not that expensive. Almost all of the overhead of reading a database once the resource cap is changed to permit, up to let's say a quarter of the machine to be memory mapped, the I-O overhead on a database largely disappears. Writing to a second location from a memory map is also not that significant in overhead. There is still the cost of the log itself. And writing content in the log, if you actually try and preserve those logs over time, it becomes quite significant. At the moment, what I am doing is every successful RPM operation creates a checkpoint. By creating a checkpoint, I'm establishing a base that determines how far back in history do I want to go. And clearly nobody cares what was done on a machine one year ago. I mean, people care, but not most users. And you don't need gigabytes of logs going back forever in most cases. The cases where you do care, I mean, it's important and you'll allocate resources to maintain those logs, but not for an end user machine. What I'm seeing is that by paying attention and changing the schema, and I'll describe some of the details in a moment, because that is the most important point. Things like rebuildDB are not the right thing to do if you have a transactionally protected database and are no longer necessary either. What I'm actually measuring is that it's quite feasible to switch from the concurrent access model that RPM is using now to a transactionally protected module if the schema changes at the same time. And there's measurable increased lock overhead, but the overall wall clock measurements are about the same. And the saving the previous transaction is probably a tolerable saving, so that if you have an interrupted action that you can restore the metadata store and take out the side effects of a broken install is probably worth the effort. But that's not for me to say. This is my opinion, okay? I mean, and I do have deep mixed feelings about logging and transactional protection. There's basically two approaches to trying to improve reliability. One of them is what I'm talking about, where you have a log on the local machine. The other one is called, I call it provisioning, and that's basically you take the state off the machine and put it someplace. This is basically what Red Hat Network does. This is the sort of Google model, and I call it provisioning because you can always recreate the machine. And provisioning as an approach to the same problem has two major benefits. First of all, it's always off the machine, so you never have to worry about hardware failures and things like that. And second, it's usually centrally located, which minimizes the maintenance cost of it. The disadvantages of a provisioning approach is you often don't have the sufficient number of details to track all the detailed state. This is better done on a target machine, on a client machine, and with a log. There's a balance point someplace between those two, all right? But RPM and package management in general is a client only. It's not tied to a server, so I'm solving that problem, but the other one exists, and there's nothing better or worse with it. Does that answer the question? Yes, absolutely. Okay. Since I just described the two sort of general alternatives, provisioning versus transactional protection, I probably should say, and describe some of the other approaches. One of the approaches has been from Nexento with ZFS, and YUM in Fedora is attempting to use BitterFS and FileSystem snapshots. These are definitely viable approaches. The problem is that if you're going to use something like a ZFS or BitterFS snapshot, you first are stuck with some of the artifacts of being a FileSystem. ZFS can perform snapshots per mount point. A package can install on several mount points, and so you have this snarl of package side effects across multiple mount points that makes it more complex than otherwise. The intrinsic flaw is if you're going to use a FileSystem approach to, let's say, minimize the number of writes or overhead or something like that, is that you must have that FileSystem in the kernel, and neither BitterFS or ZFS are sufficiently widely deployed. RPM5 runs on every Unix in site. It's not in Mac OSX and lots of other places, and I'm finding myself talking to people about even doing RPM on Windows. You can't bet on things like BitterFS or ZFS as a general approach. That's part of the reason why I decided to just use BerkeleyDB rather than to rely on a snapshot in a FileSystem. A snapshot is basically a cheap, it's essentially a checkpoint on a FileSystem with a copy on write attached. So at the checkpoint, the current state is known, and anything that changes from that point is tracked separately with some sort of overlay, if you will. That's the way I pictured a FileSystem snapshot. A log internal to RPM is essentially the same operation. The real difference in the performance measurement is basically what you choose to count, how you choose to count the resource usage, like a clock ticks or a number of blocks of I.O., but all logging approaches are going to be the same. You have to write a copy before actually performing some operation. But the point I was trying to make is that Transactually Protected Package Management is not the same as a FileSystem snapshot. These are complementary approaches to essentially the same problem. Another thing that Transactually Protected Package Management is not about is, let's say, a representation of the log. If you're trying something like the modeling that Mancusi is doing, you often have to try and come up with a reasonable format. CUDF is one. I'm quite sure that there's another one coming from Mancusi Work Project 3 that describes the elements of a rollback. Each one of these is a different format. Each one of these is a different representation of the same information. But describing things differently is, in a different type of markup, for different usage cases is not exactly what the... It's a different meaning of log, and that's not what I'm talking about with Transactually Protected Package Management. The state machine for rolling back a transaction is actually quite different than a package state machine, which does erase the old package and update with the newer content. So the log is pretty much internal, and the representation can always be retrofitted later. Since I'm reaching about 15 minutes, let me go through some of the details that have already been implemented with database rather shortly. But one of the important elements is that there are tools that are going to be needed that will be external to RPM. The most important, if you bring nothing else away from this talk, if you're trying to use a transaction model, RebuildDB won't help you, but the alternative is DBRecover minus EV. Since most of the world is fixing RPM side effects by attempting RebuildDB, it won't hurt you, but it's not going to save you. It's going to save you. So the most important element of RebuildDB is the log management, which is the most important element at reducing log size. The tool needed is DBCheckPoint, and that puts in a boundary, which in time, that says this is the last known good, but the end of older logs, the tool is DB Archive, and you will need tools to manage your database. I haven't yet even considered the question of compatibility with older versions of RPM. Compatibility is going to be rather tricky because essentially lots of the elements that used to be in RPM no longer exist. Most specifically is there are no join keys. These are integer identifiers for packages that were monotonically increasing, and since those don't exist, I'm not sure how compatibility can be meaningfully defined. I hope to do it by providing conversion. You know, conversion from one to the other is an acceptable way of achieving compatibility, but that's not the same thing as interoperable compatibility. All integers are now saved in network order. This means cross-platform is now meaningful. Previously, RPM saved in native order all the time. Network order permits binary strings and integers to be dealt with equivalently using a mem compare, which is important for B tree access, and that also is a performance increase by switching to network order. One of the other incompatibilities that's been introduced is that BerkeleyDB provides a counter that does, that monotonically increases. Previously, this was done with a weird hack by using record number zero in the primary store as if it were an instance counter. By using a persistent number, there is now a identifier that is durable over time. Some of the other performance improvements is I'm now able to do pattern retrieves on keys, which means this is much higher performing than what RPM has done in the past, which is a secondary lookup with a header load and marshalling and signature checking that the actual database side effects of the retrieval. This is where most of the performance improvements that I've achieved reside. One of the other sort of deep fundamental infrastructure changes is that signatures and digest are no longer verified. There hasn't been a problem that has been detected with a signature or digest check on a database blob for quite some time. What I'm doing instead now is using M-Map and Prott-Read to protect the blob when it's read. There's pretty clearly no reason to verify the digest and signature on a blob of binary data that is hardware protected. That removed a very large amount of overhead reading headers from a database. It's a big chunk of the tenfold performance improvement that I've gotten. That's about because of time. I'm much more interested in questions. If you want me to continue, I can go into some of the way that Scriplet acid will be achieved. Having heard John Thompson's talk this morning, that rapidly got changed since 10 o'clock this morning, but I can try and flesh out some of the details that I think were the connection between the work that Jim Thompson and Paolo and Kaisha Magica are doing is going to fit into this. Or I can also just try and field some questions. It's up to you. Are there questions? Yes. I have another question. How important it is for you, backward comparability? You mentioned that the new version you are developing now of RPM breaks compared to the old one. You need to provide a method to transition from one to another. Isn't it sometimes necessary to just break and say, now we start clean, so to speak, and move on? Well, compatibility depends on what's being talked about. And as does break. I'm not trying to break things without reason. Most of the problems with the RPM schema have known for a long time, and I've lived with that. But yes, there's times to break. I've just been notified five minutes. Any other questions? Okay. There you go. I might be a bit off here because I'm not really involved with RPM distribution. But as I understand it, it's also an option within RPM to do scriptlets. An installation. Maybe you covered this. How do you plan to roll back those scriptlets? They may be creating files. The talk by Jim Thompson, John Thompson this morning was about using a DSL to break a scriptlet into smaller pieces. The fundamental problem with scripts in packaging is that they're too coarse-grained and they're opaque. The general case is you have side effects. And the side effects need to be captured somehow. The benefit that I see from the DSL is that it will be possible not just to look at the exit code from a script, which is all the information that is being returned, but it will be possible. It was even in one of John's examples that he illustrated that there were two operations within a single scriptlet. And that's fundamentally the problem. If you have a configuration change and a Daemon restart and other side effects, it's possible to have one of them fail and it being unimportant while the other ones are absolutely essential to the operation. So by having finer grain, then it becomes possible to come up with partial failures and partially invertible failures. The other benefit will be in simulation and modeling. Because scriptlets are often in shell, it's painful to do anything other than de facto testing of the scripts by simulating and modeling other properties such as ensuring that prerequisites that aren't specified in the ordering of the package are actually met. And what I hope to do is to use that DSL, breaking it up into smaller pieces, and map those pieces into a log as a starting event and a failure event. The other more traditional approach is scriptlets run in an environment, and that environment needs to be captured in the log. What the environment variables were, what the prerequisites are to some extent that provides a level of acid and invertibility. The problem is that this is a general programmatic language, and so things like typos cannot be tolerated. But these are not very interesting problems. They are important problems that have to be solved, but it is a very hard problem to get acid underneath scriptlet behavior. Does that answer the question? Almost. There's just one more thing, actually, what I basically meant is if one of those scriptlets creates a file or updates a configuration file or whatever, and you figure out later that you need to undo the upgrade for some reason, then you should basically maybe also undo what the scriptlet did. Sure. Scriptlets need to be invertible. Yeah. There needs to be an undo. The same invertibility happens when you install a file. If you're writing content or you're removing a file. Yeah, but there's probably some way for the scriptlet then to communicate to RPM that, hey, I've modified this file, or is that way? A better communication will be the means by which acid is achieved under scriptlet behavior. All right. And it's the same basic effect that you log any operation before it's performed. The complexity comes solely from using BornShell, which is painful. Right. And embedding is another approach that could be used, like either in embedded Lua, and then RPM is more aware of the actual operations are there. But the general approach is just to use the 55 operations in Dev Helper and to parameterize those. And that's a perfectly sound starting point because that's the order of magnitude of the side effects of package management. And this isn't a large corpus. Okay. Got you. Any other questions? Yes. What do you think about the NixOS package management system, which is a purely functional way, and which avoid most of the side effects? I like it a lot. I've been talking to Ilco. I started about a year ago. And from a package manager point of view, the single biggest failure mode at the moment in packaging is scripts. Buggy scripts, you know, that it's not only buggy, but scripts that aren't written generally enough. All right. And my private view is that I need to remove all scripts from RPM. If I say that in public and I didn't say it here, okay, if I actually remove all the scripts, I'm going to also appear to remove all programmatic means to change packaging. Okay. This is the same decision point that everybody faces, switching from to a functional model. You know, you have to think about the problem space very different. All right. From a package manager point of view, I have two alternatives. I go to the NixOS route and I say, no scripts and everybody hates me. And then I retrofit the same actions, okay, through other means. Or I take a slow and gradual approach to a functional, okay, where the first step is to use Dev Helper and then to break Dev Helper into smaller pieces. And, you know, moving gradually towards, you know, refactoring code, fewer side effects, and eventually achieve a functional model. I don't know what the answer is, but I like NixOS a lot. Is it something that you are going to try in Monkusi project to tend to? I can't speak for Monkusi, but I have tried to talk to the NixOS presentation. You know, I haven't yet found it, but, and I have considered this, you know, the approach that I will probably take will be to embed Haskell and try and do NixOS package management directly from RPM and to see whether that approach is viable. All right. But I'm well aware of what NixOS is, and I like functional programming. I don't do it, okay, but because it's just too hard to switch from one to the other. You know, I live in the real world, and I'm a C programmer. Okay. Any other questions? Time. How do I do? Thank you.
rpm5.org. And also, part of the Mankuzi project, the details of package management and trying to to add a persistence to the store of the metadata on the installed machine so that it's known logging necessary for transactions and within a kernel driver to, and the benefit for a the system calls that are done by a package The choice at RPM 5 was to use Berkeley DB. The installing a package, what needs to be done is to schema, and I'll describe some of the details in changes at the same time. And there's measurable about logging and transactional protection. state machine for rolling back a transaction is update with the newer content. So the log is to be needed that will be external to RPM. The integers to be dealt with equivalently using a that it's not only buggy, but scripts that aren't