Introducing Rust into a Legacy Embedded System
Walter, Steven
Discuss the what to do, what not to do, and the “we’re still not sure if this is a good idea” parts of using Rust in an existing code base that is both resource-constrained and littered with history. Specific topics will include building Rust projects inside the bitbake build system, as well as balancing different deployment methods against code size constraints.
 So, yeah, I'll be talking about my experiences introducing Rust into an existing embedded code base, a little bit of background about my experience. So I work for Lexmark. We manufacture laser printers. I've been there about 12 years now, so I actually started on the kernel team, which did occasionally work on the kernel, but mostly that was the team that got to deal with the problems that were too hard for other teams to debug. So thread A, stomp thread B stack, and team B is really confused what happened to their stack, so that goes to the kernel team. We help unravel it. So that was fun, but we got to deal with a lot of really memory on safety issues on that team, also performance or memory issues where somebody ran the system out of memory. Then spent some time on the network team and got to do a lot of protocol interpretation there, a lot of security sensitive operations there. And then spent about a year, 18 months on the build system team, where I helped to move the firmware team away from an old proprietary in-house build system over to a standard open source tool we now use BitBake from the Octo project. I'm also interested outside of work. I like retro computing, especially anything 6502 based Commodore 64, and when I get frustrated with computers, I like to go fly around. So our embedded systems are laser printers, and in particular for Lexmark, that means it's an ARM computer running Linux with some motors and lasers attached to it, which is kind of cool if you think about it that way. We do use a lot of free and open source software, so we've got Apache, some of our products use Android, we use SystemD to coordinate our boot, and Python. So if you're wondering what kind of an embedded system we have, that can scale from really big to really small, we're probably in the middle, we actually span a fairly decent range, so we are multi-core on all of our devices, either two-core or four-core, ARM A53s. On the low end, we'll have as little as 256 megabytes of RAM, and then that scales all the way up to four gigs of RAM on the big boxes, and similarly for code, as little as 128 megs of NAND, which is not that much for a Octo project in particular, all the way up to four gigabytes. If that seems too big and too much horsepower for you, then we are actually looking at using Rust on a 256-kilobyte RAM system, running the Zephyr OS, and I'll touch back on that here in a little bit. So in addition to the nice open source parts, there are unfortunately a lot of proprietary printer-specific parts, so that would be things like PostScript and PDF interpretation that actually describes the images that are going to be printed. We have a graphics engine for formatting and resizing and doing half toning and screening, and then there's the actual code that runs the lasers and does the mechanism control and all of that fun stuff. Some of this code is extremely old, and so it dates back to the late 80s, so it's written in C and C++ of course, but it's not just C, it's really scary bad C, the kind of C code where you pass devoid star all the time and then just cast back on the other side because you know what it is, or we actually still have some functions that are K and R style prototype where the compiler will literally let you pass whatever you want to anything. So as you might expect, we have a lot of the issues that such a code base would naturally have, memory leak, security issues, it's not a great situation. So the question was how can we start digging our way out of this technical debt, and we all know that Rust solves a lot of these problems, but the question is how do we introduce Rust without breaking the code that we have, and not only that, without breaking the people that we have, right? We have a lot of C developers that are comfortable with C, don't necessarily want to learn any new language, especially one with a reputation as being difficult to get used to. But you might also ask if you're not very familiar with the laser printer industry, why do we even need an operating system? You're just putting some toner dots on a piece of paper and running some motors, that seems to be easy. Well, here's the list of things that we had to deal with, I just filled up the slide until, came up with a thing until I filled up the slide, it didn't take very long, but we have to authenticate against Active Directory, Windows Domain, so that users can log into the printer. We have to be able to pull files from Windows file shares and push to Windows file shares. Some of our devices actually have a Java virtual machine where we run third-party code that's been developed so that you can have custom workflows and do lots of whatever third-party things want to do. RFID Badge Readers, again, for authentication, we even support the CAC standard, which is used by Department of Defense, it's a smart card standard, so lots of code goes into that. We are an Internet of Thing device, we were doing that really before it had a name, but we're your DHCP Ethernet Wi-Fi, we even have printers that can be directly connected with fiber, I don't know who wants to do that, but apparently somebody did. We have an embedded web server, so like I mentioned, we're running Apache, so you can hit port 80 or port 443 and get a control panel and settings and debug tools and all sorts of fun stuff. Airprint, Google Cloud Print, we can send emails, we can do optical character recognition on the printer, we have to deal with security compliance for government, so we have to do audit logging. So lots and lots of stuff, clearly we need an operating system to help us to manage all of these things. Even though this code is newer, it's not all legacy like the low-level printing stuff, unfortunately it's not all that much better. It's still written in C, it still uses pointers, it still misuses pointers. You might think that C++ 14 and 17, they do add some new features that are helpful, so like smart pointers and move semantics, the new features allow you to write better code, but unfortunately C++ doesn't require you to write better code, so people still just kind of write code the way they're used to. So our approach to get Rust into the system was to just do it. We didn't ask anybody for permission, we just kind of picked a project that was off to the side that wasn't very important and said, all right, we're going to write this in Rust, and we kind of did that for two reasons. One is that if management came along and said, no, you can't use this, then it would only take us a couple days to rewrite it and it wouldn't be that big of a deal. But also so that most people didn't have to know or care that it was there, it just kind of built and if you weren't touching that particular component, you could continue to live your life as a C developer and pretend that nothing had changed. We also wanted to keep Rust in its own processes in the early days, primarily because of the problems I mentioned earlier where people are writing through old pointers and validating random memory, we didn't want some busted C code to corrupt Rust data structures and cause the Rust thread to crash and kind of give Rust a bad name for no fault of its own. So we wanted to keep it nicely isolated on its own. Unfortunately, that experiment did go very well. As people found out about it, nobody really cared because it worked and we explained the benefits of it and people kind of bought into that and said, okay, that makes sense, keep doing that. And so that kind of gave us the confidence to go to more complex components. So we still didn't want things that had a lot of dependencies, we didn't really go to the core of the system. Being a legacy embedded system, you get a lot of the everything depends on everything effect. So we did try to stay at the edges and that was primarily so that we didn't have to deal with FFI, so we didn't want to have to generate stuff from our headers, we didn't want to have to generate headers for C code and there's extra build system complexity there as well. Unfortunately that has still gone really well and we actually have probably about a dozen people in the company now that are competent in Rust, four or five of us that are pretty passionate about it. And so we kind of now treat anything as fair game. We've done some fairly large projects based on Rust now. We particularly try to push the use of Rust if it's something that's security sensitive where memory safety really gives you a lot of benefits. If it's something that deals with protocol parsing, CERDE is really good with that, there's other good crates in the Rust ecosystem for that but we've used CERDE in multiple places and been really happy with it. Similarly for the client service stuff, I mentioned that I was on the network team and saw that we had lots of opportunity to use things like Tokyo and Futures which we've been using a little bit and we're very excited about East Inc. await coming to the standard library and looking forward to doing that. Kind of the one area that we still try to shy away from in Rust is if it's anything that requires unsafe code or if people just think they need to use unsafe, we really try to discourage that. I've joked that since I run the Git servers, if I have to, I'll put in a hook that will reject anything that adds unsafe to a.rs file. Fortunately, I've not had to do that yet. Again I know that not everyone here is an embedded person so I wanted to give a little bit of background about what is BitBake, what is this build tool that we use. The point of BitBake is to let you build your own custom Linux distribution. You can choose what architecture, you can choose the compile flags you want, you can choose how big or how small it is. It's kind of analogous to build root, that's another open source tool that does a similar thing. It's part of the Octo project which has quite a bit of industry backing, Intel's behind it, a lot of the single board computers and other systems support that. We've really gotten the benefit of the open source community and the support that's gone into it. As far as the history of the code base, it actually goes back to Gen2's eMERGE. If you've ever used Gen2, ever looked at their recipes, the syntax that they use, you'll be pretty at home with how BitBake builds things. The point of these recipes is you'll have one recipe for each software component in your computer system. You'll generate an RPM, so you'll build up a whole bunch of RPM packages and then you'll have a recipe for an image that just says, here's all the different packages that I want to pull in and it takes care of handling the dependencies and generating the root file system, putting in a cram file system, whatever type of transport that you need to actually then program it down to the device. Here's a little bit of an example recipe. You don't really have to understand most of this, but there are a couple of interesting things that you can deduce from it. One of them is that BitBake wants to fetch the code for you. It wants to know where the code comes from and it wants to do the fetch itself. That's not really the way Cargo works. Similarly, it wants to know all about your dependencies. It wants to know what components do you depend on so that if they change, BitBake knows to rebuild you. Already, we can see where BitBake and Cargo are going to be playing in the same area and might be some tension there. The way that from a very practical perspective, the way we're building our Rust code into our system is we're using a project called MetaRust and full disclosure, I am one of the co-maintainers now. I didn't start it, but I am involved in that now. What that project gives you are the BitBake recipes for the compiler itself, for the standard library, and the rules for being able to easily cross-compile stuff. It's not too hard to build Rust in a tool chain for native compilation, but if you're building it for ARM or some other CPU that you're not running on, it gets a little more difficult. MetaRust takes care of all of that component for you. Then there's another tool called Cargo BitBake. What that is is a helper program that you can run from your own crate, and it will generate for you a BitBake recipe. You can go drop that into your recipes repository, and now BitBake will be able to build your Rust code for you. In particular, it looks at the Cargo lock file and figures out what all of your direct and indirect dependencies are. This allows us to hook into BitBake's ability to fetch code. Now we can play nicely with source mirrors and caching, and really just try to play along with BitBake's model of the world as much as possible. So far, our experiences have been really positive. One of the things, I'm not the first person to have said this, but it's another vote for if it builds, there's a very good chance that it'll work. We've seen that in lots of cases where, yeah, we may have had to fight with a borrower checker a little bit, but once we got through there, it did the right thing. It worked, and not only did it work right away, it continued to work. We didn't push it out into the testers, and then they found a bunch of corner cases and holes. We wrote it, got it working, and then we're able to move on to the next thing without a lot of downstream work later on in the development cycle. The corollary, for me at least, is anytime I try to write on safe code, it will crash. I'm thinking I'm 100% on that. It's especially kind of appealing for C developers, you might think, oh, well, I'm used to writing C code. C is unsafe, therefore unsafe as Rust, and Rust can't be that much worse. But actually it is, and the reason for that is that the Rust compiler will make a lot more assumptions about the system and your code than a C compiler will. So you actually have to maintain a lot more guarantees, even in the unsafe parts of Rust that you might not be used to coming from a C background. So one of the good things is that as far as runtime memory use goes, it's fairly low, it's for our purposes low enough, and it's pretty predictable. The memory model with borrowing and putting most things onto the stack instead of the heap makes it fairly predictable, whereas in an equivalent C++ program you're more likely to rely more on dynamic memory management, which makes it a lot harder to profile and predict how that code is going to work. So really the main bad side, unfortunately there are some bad sides, are around the code size. So as I mentioned earlier, on the very low end, we have products that fit into 128 megs of NAND. And even without Rust code, stripped out about as much as you can, just the kernel and libc and systemd and a couple of very other low-level things gets you into 30 or 40 megs, and you're not even into your application code yet. So there's not a whole lot of room to play with. We did start introducing Rust when we were doing the high-end devices, so the way our product cycles go is we'll do a few high-end devices, a few low-end devices, and kind of bounce back and forth. So everything was great on the high-end devices, but then we really had to look at how efficient we were being with code size and how we were spinning that on Rust. So now normally, from a C background, I would think, okay, these multiple Rust binaries that we have, there's a lot of duplicated code between them, so we can just use a shared library, move all the code that's shared into the shared object. Now there's only one copy of it. Everything will be great. Well, MetaRust supports that a little bit. It does have the ability for libstandard to be built as a shared object, and that helps for at least that part of it, but it really doesn't help you for your intermediate crates, things like futures and Tokyo and Saturday, and all of those other guys that are going to be used in a lot of different projects. Those are still going to end up duplicated. And really, at the end of the day, Rust shared library support is not all that mature. There's no standardized ABI, so that's one of the reasons why the devs kind of discouraged that use. That's not a big problem for BitBake because it builds everything in a single pass and kind of knows when it needs to rebuild stuff to make sure everything's consistent. But then even when you do use shared libraries, there's still going to be code for generic code, stuff that gets generated. That's going to end up in the individual binaries. So for example, I'm not sure if this particular example is true, but say if you use a VEC of U8 in multiple places and multiple programs, there's going to be code that the compiler generates and that's not necessarily implemented in the standard library, so that may end up in your multiple executables as well. And this is kind of understandable because the use case for dilib and Rust is not really to reduce code size. They're not really worried about my particular case. They just want for you to be able to make a shared object that you can link against or that you can deal open for FFI purposes. So given that use case, it makes sense that they really want to put as much code as possible into that shared library so that it's standalone and self-sufficient, but that doesn't really help the case when you have multiple of them and the code size implications of that. So the first pass that we took at trying to solve this problem didn't work, but just so that no one else wants to go and try it. Our thought was, well, cargo really wants to build lots of static binaries. It wants to do the full pass all by itself. So we'll just not use cargo. We'll invoke Rust directly. We'll write a recipe for every individual crate. We'll write a recipe for regex and we'll write a recipe for memcur and all of those guys, several dozen of them, and we'll just invoke Rust C directly and generate shared objects for each one of those. And while that is bad, it wasn't as bad as it would be in other languages. At least the highly regular structure of Rust projects meant that that was kind of a tractable approach. You always have a lib.rs that you can point Rusty at and it knows how to find all the other source files based on that. If you were in C or C++, every project is completely different and there's no regularity. It's really not tractable at all. But even though you can kind of do it, it doesn't scale very well, especially when you start getting into the more complex crates like Tokyo and futures. And really what completely kills this is build.rs. If it's a crate that wants to build and run Rust code to then figure out how to build and run the rest of the Rust code, we really just have no way of being able to support that. So the current solution that we're using is, well, if having multiple Rust binaries waste a lot of code, what if we just only have a single binary? We'll take all of our Rust in the system and we'll link it into a single large object. And we call that uranium. So it's kind of a super binary, kind of like Busybox if you're familiar with that. So what Busybox does is it takes a bunch of common embedded tools, including like the shell, LS test, grep, more, all of those guys, links them into one program. So this is kind of the same idea, but with our Rust code. So the advantages of that are now that we only have a single Rust executable, we don't need shared objects, not even for libSTD. And so the nice thing about that is we no longer have the parts of libSTD that we don't use. With static linking, if you don't use it, it doesn't get pulled in. So you're not paying a code penalty for that. So that's nice. That also extends to all of our intermediate components. So now Futures and Tokyo and Surde and all of those guys, again, they're only getting linked in one time. The parts we don't use, we don't have to pay for. And similarly for any generic code that gets generated, the link time optimizer is smart enough to figure out. These are the same functions for the same purpose, even though they're being used by multiple downstream dependencies, so I only have to include them once. And from a theoretical perspective, this is almost the best you can possibly do short of the compiler just generating less code. We have only the code that we need, and we only have one single copy of it. So just to go into a little more nitty-gritty of how we implemented this in case this is an approach that might be helpful for someone else, there are downsides to it. But basically we went into all of our Rust programs and converted them from instead of being bins to being RLibs. And then we used Git submodules to kind of pull them all into a single mono repo so that BitBake can check out all of our Rust code in the system all at once. And then we can use cargo to build it all in a single pass. Then the real main function, the uranium main function, just looks at argv0 to figure out, okay, which program did you actually want to run? And then it dispatches to the correct main function and the correct crate. So like I said, that'll build everything in a single pass of cargo, which that itself has pros and cons. But it does avoid one of the downsides of this approach from C. So as a C person, if you told someone you were going to link all of the code that was completely unrelated into a single binary, you would give pause to that. And one of the reasons that you would not want to do that is that you can grow undesired, implicit dependencies from unrelated parts of your code. Because C symbol visibility is basically everything can see everything, you actually can call a function from a completely unrelated piece of code and the compiler or linker will be okay with that because at the end of the day, they're all in the executable, they're all visible, and that'll work. So Rust at least does protect against that particular software engineering danger because unless you list the dependency in your cargo tumble, you're not going to be able to see it. You won't let you use it even though you do all get linked together in the end. And so just to give a little more color to this, there's actually are still two components, the LibEranium shared object, which is where the bulk of the code is, and then just a tiny executable that all it does is thunk over to the main function and the other guy. And we do that so that in the few places where we do want to do FFI, we can directly link against this without having to have another copy and start incurring those code penalties again and also let's just do DL open. Oh, the first pass of that, we actually just had an executable built as a position independent executable and we thought, okay, well, let's just DL open that or link against that. It's technically the same thing or almost the same thing as a shared library. That doesn't work. Don't do that. So there are some downsides to this. One of the bigger ones is the build time. So because all of your Rust code is getting built in a single passive cargo, that means if any of your Rust code anywhere changes, you have to rebuild all of it. So the caching model that BitBake uses is per recipe. So if anything that is in that recipe changes, then you're going to have to rebuild that whole recipe. You don't really get to benefit a whole lot from incremental compilation, especially not for system builds. Local developers might get it a little bit, but for any kind of a system build, it's going to be a clean build from scratch and it takes a fair amount of time. It also adds some extra hurdles for developers as they're trying to integrate their code. So in addition to actually checking your code into the Rust repo that you care about, you're now also going to have to go into our Mono repo and update the get sub module over there and then finally go into the meta layer where the BitBake recipe lives and change the source rev as well. So while it is maybe just one extra step that is a 50% overhead, it's not great. And another big problem is if you have lots of different subsets of this code that you use. So in our use today, we pretty much use all of the Rust code everywhere. We just kind of have one configuration, if you will, of Rust programs. But if you wanted to have on one image, say, applets A, B, and C, and on the second image, B, C, and D, et cetera, that does not scale well at all. It's like an in factorial scaling or something. So if you have a lot of different flavors, then you're much worse off than you are in the other model that I talked about where you have a single recipe per Rust crate. So we do know about some things we can look at into the future that might make this better. And again, if you have other ideas for how we can improve this, definitely find me in the hallways or afterwards to talk about this. But one thought we've had is that we can use kind of a private registry, and that will get around some of the downsides of using the get submodule approach at least. So rather than using get submodules to track that, we can actually have a registry push version releases into our registry and consume them that way. That does require that you version your code correctly, which we're not always great about. But we think that that's probably a little more of a Rust-like way to handle the problem. But I think what would really benefit us the best is if cargo had better capability to do its work not all at once. If we were able to run cargo in one place and use the output of that run of cargo into the next place so that we could still have multiple Rust recipes, each running cargo, and each benefiting from the output of previous runs of it. I'm not a compiler guy or a cargo guy. I don't know how easy or difficult that is. But that's kind of the model that works with us because that's kind of the model that bit bake and a lot of other build tools, similar build tools, NixOS and other guys have similar problems. So I did want to circle back to and plug the Zephyr Rust project just a little bit. So that's actually primarily Tyler Hall if he wants to wave the people back there. If you're curious about this, you can talk to him afterwards. But basically this allows us to run a Rust app on the Zephyr kernel. I should probably should have said what Zephyr is. It's a real time operating system for really small, primarily Internet of Things devices. Usually it's order hundreds of K of RAM and devices that don't even have a memory management unit. So even if you wanted to run Linux on there, you really can't. So today it has the capability to call any Zephyr Syscall you want as an unsafe call. And it also has safe wrappers for a lot of those. We do have some initial support for being able to do async await for certain types of IO and not network but UART and things like that can be used with async await. And in particular it does not require no STD. A lot of the projects that target these kind of bare metal systems require that you not use the standard library at all. So the entire standard library does compile and anything without OS specific so that you're not using STD Sys, it will build. It may or may not run because a lot of it is stubbed out. But that at least makes it a lot easier to get your existing code running in this environment. It's available on GitHub and that's the link. And that's all I've got. Thank you very much.
 So, yeah, I'll be talking about my experiences introducing Rust into an existing embedded  code base, a little bit of background about my experience.  So I work for Lexmark.  We manufacture laser printers.  I've been there about 12 years now, so I actually started on the kernel team, which did occasionally  work on the kernel, but mostly that was the team that got to deal with the problems that  were too hard for other teams to debug.  So thread A, stomp thread B stack, and team B is really confused what happened to their  stack, so that goes to the kernel team.  We help unravel it.  So that was fun, but we got to deal with a lot of really memory on safety issues on that  team, also performance or memory issues where somebody ran the system out of memory.  Then spent some time on the network team and got to do a lot of protocol interpretation  there, a lot of security sensitive operations there.  And then spent about a year, 18 months on the build system team, where I helped to move  the firmware team away from an old proprietary in-house build system over to a standard open  source tool we now use BitBake from the Octo project.  I'm also interested outside of work.  I like retro computing, especially anything 6502 based Commodore 64, and when I get frustrated  with computers, I like to go fly around.  So our embedded systems are laser printers, and in particular for Lexmark, that means  it's an ARM computer running Linux with some motors and lasers attached to it, which is  kind of cool if you think about it that way.  We do use a lot of free and open source software, so we've got Apache, some of our products  use Android, we use SystemD to coordinate our boot, and Python.  So if you're wondering what kind of an embedded system we have, that can scale from really  big to really small, we're probably in the middle, we actually span a fairly decent range,  so we are multi-core on all of our devices, either two-core or four-core, ARM A53s.  On the low end, we'll have as little as 256 megabytes of RAM, and then that scales all  the way up to four gigs of RAM on the big boxes, and similarly for code, as little as  128 megs of NAND, which is not that much for a Octo project in particular, all the way  up to four gigabytes.  If that seems too big and too much horsepower for you, then we are actually looking at using  Rust on a 256-kilobyte RAM system, running the Zephyr OS, and I'll touch back on that  here in a little bit.  So in addition to the nice open source parts, there are unfortunately a lot of proprietary  printer-specific parts, so that would be things like PostScript and PDF interpretation that  actually describes the images that are going to be printed.  We have a graphics engine for formatting and resizing and doing half toning and screening,  and then there's the actual code that runs the lasers and does the mechanism control and  all of that fun stuff.  Some of this code is extremely old, and so it dates back to the late 80s, so it's written  in C and C++ of course, but it's not just C, it's really scary bad C, the kind of C  code where you pass devoid star all the time and then just cast back on the other side  because you know what it is, or we actually still have some functions that are K and R  style prototype where the compiler will literally let you pass whatever you want to anything.  So as you might expect, we have a lot of the issues that such a code base would naturally  have, memory leak, security issues, it's not a great situation.  So the question was how can we start digging our way out of this technical debt, and we  all know that Rust solves a lot of these problems, but the question is how do we introduce Rust  without breaking the code that we have, and not only that, without breaking the people  that we have, right?  We have a lot of C developers that are comfortable with C, don't necessarily want to learn any  new language, especially one with a reputation as being difficult to get used to.  But you might also ask if you're not very familiar with the laser printer industry, why  do we even need an operating system?  You're just putting some toner dots on a piece of paper and running some motors, that  seems to be easy.  Well, here's the list of things that we had to deal with, I just filled up the slide  until, came up with a thing until I filled up the slide, it didn't take very long, but  we have to authenticate against Active Directory, Windows Domain, so that users can log into  the printer.  We have to be able to pull files from Windows file shares and push to Windows file shares.  Some of our devices actually have a Java virtual machine where we run third-party code that's  been developed so that you can have custom workflows and do lots of whatever third-party  things want to do.  RFID Badge Readers, again, for authentication, we even support the CAC standard, which is  used by Department of Defense, it's a smart card standard, so lots of code goes into that.  We are an Internet of Thing device, we were doing that really before it had a name, but  we're your DHCP Ethernet Wi-Fi, we even have printers that can be directly connected with  fiber, I don't know who wants to do that, but apparently somebody did.  We have an embedded web server, so like I mentioned, we're running Apache, so you can  hit port 80 or port 443 and get a control panel and settings and debug tools and all  sorts of fun stuff.  Airprint, Google Cloud Print, we can send emails, we can do optical character recognition  on the printer, we have to deal with security compliance for government, so we have to do  audit logging.  So lots and lots of stuff, clearly we need an operating system to help us to manage all  of these things.  Even though this code is newer, it's not all legacy like the low-level printing stuff,  unfortunately it's not all that much better.  It's still written in C, it still uses pointers, it still misuses pointers.  You might think that C++ 14 and 17, they do add some new features that are helpful,  so like smart pointers and move semantics, the new features allow you to write better  code, but unfortunately C++ doesn't require you to write better code, so people still  just kind of write code the way they're used to.  So our approach to get Rust into the system was to just do it.  We didn't ask anybody for permission, we just kind of picked a project that was off to the  side that wasn't very important and said, all right, we're going to write this in Rust,  and we kind of did that for two reasons.  One is that if management came along and said, no, you can't use this, then it would only  take us a couple days to rewrite it and it wouldn't be that big of a deal.  But also so that most people didn't have to know or care that it was there, it just kind  of built and if you weren't touching that particular component, you could continue to live your  life as a C developer and pretend that nothing had changed.  We also wanted to keep Rust in its own processes in the early days, primarily because of the  problems I mentioned earlier where people are writing through old pointers and validating  random memory, we didn't want some busted C code to corrupt Rust data structures and  cause the Rust thread to crash and kind of give Rust a bad name for no fault of its  own.  So we wanted to keep it nicely isolated on its own.  Unfortunately, that experiment did go very well.  As people found out about it, nobody really cared because it worked and we explained the  benefits of it and people kind of bought into that and said, okay, that makes sense, keep  doing that.  And so that kind of gave us the confidence to go to more complex components.  So we still didn't want things that had a lot of dependencies, we didn't really go to  the core of the system.  Being a legacy embedded system, you get a lot of the everything depends on everything  effect.  So we did try to stay at the edges and that was primarily so that we didn't have to deal  with FFI, so we didn't want to have to generate stuff from our headers, we didn't want to  have to generate headers for C code and there's extra build system complexity there as well.  Unfortunately that has still gone really well and we actually have probably about a dozen  people in the company now that are competent in Rust, four or five of us that are pretty  passionate about it.  And so we kind of now treat anything as fair game.  We've done some fairly large projects based on Rust now.  We particularly try to push the use of Rust if it's something that's security sensitive  where memory safety really gives you a lot of benefits.  If it's something that deals with protocol parsing, CERDE is really good with that, there's  other good crates in the Rust ecosystem for that but we've used CERDE in multiple places  and been really happy with it.  Similarly for the client service stuff, I mentioned that I was on the network team and  saw that we had lots of opportunity to use things like Tokyo and Futures which we've  been using a little bit and we're very excited about East Inc. await coming to the standard  library and looking forward to doing that.  Kind of the one area that we still try to shy away from in Rust is if it's anything  that requires unsafe code or if people just think they need to use unsafe, we really try  to discourage that.  I've joked that since I run the Git servers, if I have to, I'll put in a hook that will  reject anything that adds unsafe to a.rs file.  Fortunately, I've not had to do that yet.  Again I know that not everyone here is an embedded person so I wanted to give a little  bit of background about what is BitBake, what is this build tool that we use.  The point of BitBake is to let you build your own custom Linux distribution.  You can choose what architecture, you can choose the compile flags you want, you can  choose how big or how small it is.  It's kind of analogous to build root, that's another open source tool that does a similar  thing.  It's part of the Octo project which has quite a bit of industry backing, Intel's behind  it, a lot of the single board computers and other systems support that.  We've really gotten the benefit of the open source community and the support that's gone  into it.  As far as the history of the code base, it actually goes back to Gen2's eMERGE.  If you've ever used Gen2, ever looked at their recipes, the syntax that they use, you'll  be pretty at home with how BitBake builds things.  The point of these recipes is you'll have one recipe for each software component in your  computer system.  You'll generate an RPM, so you'll build up a whole bunch of RPM packages and then you'll  have a recipe for an image that just says, here's all the different packages that I want  to pull in and it takes care of handling the dependencies and generating the root file  system, putting in a cram file system, whatever type of transport that you need to actually  then program it down to the device.  Here's a little bit of an example recipe.  You don't really have to understand most of this, but there are a couple of interesting  things that you can deduce from it.  One of them is that BitBake wants to fetch the code for you.  It wants to know where the code comes from and it wants to do the fetch itself.  That's not really the way Cargo works.  Similarly, it wants to know all about your dependencies.  It wants to know what components do you depend on so that if they change, BitBake knows to  rebuild you.  Already, we can see where BitBake and Cargo are going to be playing in the same area and  might be some tension there.  The way that from a very practical perspective, the way we're building our Rust code into  our system is we're using a project called MetaRust and full disclosure, I am one of the  co-maintainers now.  I didn't start it, but I am involved in that now.  What that project gives you are the BitBake recipes for the compiler itself, for the standard  library, and the rules for being able to easily cross-compile stuff.  It's not too hard to build Rust in a tool chain for native compilation, but if you're  building it for ARM or some other CPU that you're not running on, it gets a little more  difficult.  MetaRust takes care of all of that component for you.  Then there's another tool called Cargo BitBake.  What that is is a helper program that you can run from your own crate, and it will generate  for you a BitBake recipe.  You can go drop that into your recipes repository, and now BitBake will be able to build your  Rust code for you.  In particular, it looks at the Cargo lock file and figures out what all of your direct  and indirect dependencies are.  This allows us to hook into BitBake's ability to fetch code.  Now we can play nicely with source mirrors and caching, and really just try to play along  with BitBake's model of the world as much as possible.  So far, our experiences have been really positive.  One of the things, I'm not the first person to have said this, but it's another vote for  if it builds, there's a very good chance that it'll work.  We've seen that in lots of cases where, yeah, we may have had to fight with a borrower checker  a little bit, but once we got through there, it did the right thing.  It worked, and not only did it work right away, it continued to work.  We didn't push it out into the testers, and then they found a bunch of corner cases and  holes.  We wrote it, got it working, and then we're able to move on to the next thing without  a lot of downstream work later on in the development cycle.  The corollary, for me at least, is anytime I try to write on safe code, it will crash.  I'm thinking I'm 100% on that.  It's especially kind of appealing for C developers, you might think, oh, well, I'm used to writing  C code.  C is unsafe, therefore unsafe as Rust, and Rust can't be that much worse.  But actually it is, and the reason for that is that the Rust compiler will make a lot more  assumptions about the system and your code than a C compiler will.  So you actually have to maintain a lot more guarantees, even in the unsafe parts of Rust  that you might not be used to coming from a C background.  So one of the good things is that as far as runtime memory use goes, it's fairly low,  it's for our purposes low enough, and it's pretty predictable.  The memory model with borrowing and putting most things onto the stack instead of the  heap makes it fairly predictable, whereas in an equivalent C++ program you're more likely  to rely more on dynamic memory management, which makes it a lot harder to profile and  predict how that code is going to work.  So really the main bad side, unfortunately there are some bad sides, are around the code  size.  So as I mentioned earlier, on the very low end, we have products that fit into 128 megs  of NAND.  And even without Rust code, stripped out about as much as you can, just the kernel and libc  and systemd and a couple of very other low-level things gets you into 30 or 40 megs, and you're  not even into your application code yet.  So there's not a whole lot of room to play with.  We did start introducing Rust when we were doing the high-end devices, so the way our  product cycles go is we'll do a few high-end devices, a few low-end devices, and kind of  bounce back and forth.  So everything was great on the high-end devices, but then we really had to look at how efficient  we were being with code size and how we were spinning that on Rust.  So now normally, from a C background, I would think, okay, these multiple Rust binaries  that we have, there's a lot of duplicated code between them, so we can just use a shared  library, move all the code that's shared into the shared object.  Now there's only one copy of it.  Everything will be great.  Well, MetaRust supports that a little bit.  It does have the ability for libstandard to be built as a shared object, and that helps  for at least that part of it, but it really doesn't help you for your intermediate crates,  things like futures and Tokyo and Saturday, and all of those other guys that are going  to be used in a lot of different projects.  Those are still going to end up duplicated.  And really, at the end of the day, Rust shared library support is not all that mature.  There's no standardized ABI, so that's one of the reasons why the devs kind of discouraged  that use.  That's not a big problem for BitBake because it builds everything in a single pass and kind  of knows when it needs to rebuild stuff to make sure everything's consistent.  But then even when you do use shared libraries, there's still going to be code for generic  code, stuff that gets generated.  That's going to end up in the individual binaries.  So for example, I'm not sure if this particular example is true, but say if you use a VEC of  U8 in multiple places and multiple programs, there's going to be code that the compiler  generates and that's not necessarily implemented in the standard library, so that may end up  in your multiple executables as well.  And this is kind of understandable because the use case for dilib and Rust is not really  to reduce code size.  They're not really worried about my particular case.  They just want for you to be able to make a shared object that you can link against or  that you can deal open for FFI purposes.  So given that use case, it makes sense that they really want to put as much code as possible  into that shared library so that it's standalone and self-sufficient, but that doesn't really  help the case when you have multiple of them and the code size implications of that.  So the first pass that we took at trying to solve this problem didn't work, but just so  that no one else wants to go and try it.  Our thought was, well, cargo really wants to build lots of static binaries.  It wants to do the full pass all by itself.  So we'll just not use cargo.  We'll invoke Rust directly.  We'll write a recipe for every individual crate.  We'll write a recipe for regex and we'll write a recipe for memcur and all of those guys,  several dozen of them, and we'll just invoke Rust C directly and generate shared objects  for each one of those.  And while that is bad, it wasn't as bad as it would be in other languages.  At least the highly regular structure of Rust projects meant that that was kind of a tractable  approach.  You always have a lib.rs that you can point Rusty at and it knows how to find all the  other source files based on that.  If you were in C or C++, every project is completely different and there's no regularity.  It's really not tractable at all.  But even though you can kind of do it, it doesn't scale very well, especially when you  start getting into the more complex crates like Tokyo and futures.  And really what completely kills this is build.rs.  If it's a crate that wants to build and run Rust code to then figure out how to build  and run the rest of the Rust code, we really just have no way of being able to support  that.  So the current solution that we're using is, well, if having multiple Rust binaries  waste a lot of code, what if we just only have a single binary?  We'll take all of our Rust in the system and we'll link it into a single large object.  And we call that uranium.  So it's kind of a super binary, kind of like Busybox if you're familiar with that.  So what Busybox does is it takes a bunch of common embedded tools, including like the  shell, LS test, grep, more, all of those guys, links them into one program.  So this is kind of the same idea, but with our Rust code.  So the advantages of that are now that we only have a single Rust executable, we don't  need shared objects, not even for libSTD.  And so the nice thing about that is we no longer have the parts of libSTD that we don't  use.  With static linking, if you don't use it, it doesn't get pulled in.  So you're not paying a code penalty for that.  So that's nice.  That also extends to all of our intermediate components.  So now Futures and Tokyo and Surde and all of those guys, again, they're only getting  linked in one time.  The parts we don't use, we don't have to pay for.  And similarly for any generic code that gets generated, the link time optimizer is smart  enough to figure out.  These are the same functions for the same purpose, even though they're being used by  multiple downstream dependencies, so I only have to include them once.  And from a theoretical perspective, this is almost the best you can possibly do short  of the compiler just generating less code.  We have only the code that we need, and we only have one single copy of it.  So just to go into a little more nitty-gritty of how we implemented this in case this is  an approach that might be helpful for someone else, there are downsides to it.  But basically we went into all of our Rust programs and converted them from instead of  being bins to being RLibs.  And then we used Git submodules to kind of pull them all into a single mono repo so that  BitBake can check out all of our Rust code in the system all at once.  And then we can use cargo to build it all in a single pass.  Then the real main function, the uranium main function, just looks at argv0 to figure out,  okay, which program did you actually want to run?  And then it dispatches to the correct main function and the correct crate.  So like I said, that'll build everything in a single pass of cargo, which that itself  has pros and cons.  But it does avoid one of the downsides of this approach from C. So as a C person, if  you told someone you were going to link all of the code that was completely unrelated  into a single binary, you would give pause to that.  And one of the reasons that you would not want to do that is that you can grow undesired,  implicit dependencies from unrelated parts of your code.  Because C symbol visibility is basically everything can see everything, you actually  can call a function from a completely unrelated piece of code and the compiler or linker will  be okay with that because at the end of the day, they're all in the executable, they're  all visible, and that'll work.  So Rust at least does protect against that particular software engineering danger because  unless you list the dependency in your cargo tumble, you're not going to be able to see  it.  You won't let you use it even though you do all get linked together in the end.  And so just to give a little more color to this, there's actually are still two components,  the LibEranium shared object, which is where the bulk of the code is, and then just a tiny  executable that all it does is thunk over to the main function and the other guy.  And we do that so that in the few places where we do want to do FFI, we can directly link  against this without having to have another copy and start incurring those code penalties  again and also let's just do DL open.  Oh, the first pass of that, we actually just had an executable built as a position independent  executable and we thought, okay, well, let's just DL open that or link against that.  It's technically the same thing or almost the same thing as a shared library.  That doesn't work.  Don't do that.  So there are some downsides to this.  One of the bigger ones is the build time.  So because all of your Rust code is getting built in a single passive cargo, that means  if any of your Rust code anywhere changes, you have to rebuild all of it.  So the caching model that BitBake uses is per recipe.  So if anything that is in that recipe changes, then you're going to have to rebuild that  whole recipe.  You don't really get to benefit a whole lot from incremental compilation, especially  not for system builds.  Local developers might get it a little bit, but for any kind of a system build, it's going  to be a clean build from scratch and it takes a fair amount of time.  It also adds some extra hurdles for developers as they're trying to integrate their code.  So in addition to actually checking your code into the Rust repo that you care about, you're  now also going to have to go into our Mono repo and update the get sub module over there  and then finally go into the meta layer where the BitBake recipe lives and change the source  rev as well.  So while it is maybe just one extra step that is a 50% overhead, it's not great.  And another big problem is if you have lots of different subsets of this code that you  use.  So in our use today, we pretty much use all of the Rust code everywhere.  We just kind of have one configuration, if you will, of Rust programs.  But if you wanted to have on one image, say, applets A, B, and C, and on the second image,  B, C, and D, et cetera, that does not scale well at all.  It's like an in factorial scaling or something.  So if you have a lot of different flavors, then you're much worse off than you are in  the other model that I talked about where you have a single recipe per Rust crate.  So we do know about some things we can look at into the future that might make this better.  And again, if you have other ideas for how we can improve this, definitely find me in  the hallways or afterwards to talk about this.  But one thought we've had is that we can use kind of a private registry, and that will  get around some of the downsides of using the get submodule approach at least.  So rather than using get submodules to track that, we can actually have a registry push  version releases into our registry and consume them that way.  That does require that you version your code correctly, which we're not always great about.  But we think that that's probably a little more of a Rust-like way to handle the problem.  But I think what would really benefit us the best is if cargo had better capability to  do its work not all at once.  If we were able to run cargo in one place and use the output of that run of cargo into  the next place so that we could still have multiple Rust recipes, each running cargo,  and each benefiting from the output of previous runs of it.  I'm not a compiler guy or a cargo guy.  I don't know how easy or difficult that is.  But that's kind of the model that works with us because that's kind of the model that bit  bake and a lot of other build tools, similar build tools, NixOS and other guys have similar  problems.  So I did want to circle back to and plug the Zephyr Rust project just a little bit.  So that's actually primarily Tyler Hall if he wants to wave the people back there.  If you're curious about this, you can talk to him afterwards.  But basically this allows us to run a Rust app on the Zephyr kernel.  I should probably should have said what Zephyr is.  It's a real time operating system for really small, primarily Internet of Things devices.  Usually it's order hundreds of K of RAM and devices that don't even have a memory management  unit.  So even if you wanted to run Linux on there, you really can't.  So today it has the capability to call any Zephyr Syscall you want as an unsafe call.  And it also has safe wrappers for a lot of those.  We do have some initial support for being able to do async await for certain types of  IO and not network but UART and things like that can be used with async await.  And in particular it does not require no STD.  A lot of the projects that target these kind of bare metal systems require that you not  use the standard library at all.  So the entire standard library does compile and anything without OS specific so that you're  not using STD Sys, it will build.  It may or may not run because a lot of it is stubbed out.  But that at least makes it a lot easier to get your existing code running in this environment.  It's available on GitHub and that's the link.  And that's all I've got.  Thank you very much.
Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_0_143.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_144_203.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_204_275.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_276_1895.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_1896_5675.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_5676_5771.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_5772_5867.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_5868_5915.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_5916_8615.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_8616_8963.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_8964_9059.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_9060_9263.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_9264_9347.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_9348_9683.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_9684_9731.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_9732_9827.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_9828_10067.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_10068_10259.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_10260_18395.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_18396_21167.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_21168_26135.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_26136_28307.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_28308_28367.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_28368_28751.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_28752_28931.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_28932_31331.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_31332_31703.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_31704_36179.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_36180_38207.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_38208_38291.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_38292_38387.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_38388_38459.png Introducing_Rust_into_a_Legacy_Embedded_System_-_S_EoV94cg_Tug_38460_38513.png
So, yeah, I'll be talking about my experiences introducing Rust into an existing embedded code base, a little bit of background about my experience. So I work for Lexmark. We manufacture laser printers. I've been there about 12 years now, so I actually started on the kernel team, which did occasionally work on the kernel, but mostly that was the team that got to deal with the problems that were too hard for other teams to debug. So thread A, stomp thread B stack, and team B is really confused what happened to their stack, so that goes to the kernel team. We help unravel it. So that was fun, but we got to deal with a lot of really memory on safety issues on that team, also performance or memory issues where somebody ran the system out of memory. Then spent some time on the network team and got to do a lot of protocol interpretation there, a lot of security sensitive operations there. And then spent about a year, 18 months on the build system team, where I helped to move the firmware team away from an old proprietary in-house build system over to a standard open source tool we now use BitBake from the Octo project. I'm also interested outside of work. I like retro computing, especially anything 6502 based Commodore 64, and when I get frustrated with computers, I like to go fly around. So our embedded systems are laser printers, and in particular for Lexmark, that means it's an ARM computer running Linux with some motors and lasers attached to it, which is kind of cool if you think about it that way. We do use a lot of free and open source software, so we've got Apache, some of our products use Android, we use SystemD to coordinate our boot, and Python. So if you're wondering what kind of an embedded system we have, that can scale from really big to really small, we're probably in the middle, we actually span a fairly decent range, so we are multi-core on all of our devices, either two-core or four-core, ARM A53s. On the low end, we'll have as little as 256 megabytes of RAM, and then that scales all the way up to four gigs of RAM on the big boxes, and similarly for code, as little as 128 megs of NAND, which is not that much for a Octo project in particular, all the way up to four gigabytes. If that seems too big and too much horsepower for you, then we are actually looking at using Rust on a 256-kilobyte RAM system, running the Zephyr OS, and I'll touch back on that here in a little bit. So in addition to the nice open source parts, there are unfortunately a lot of proprietary printer-specific parts, so that would be things like PostScript and PDF interpretation that actually describes the images that are going to be printed. We have a graphics engine for formatting and resizing and doing half toning and screening, and then there's the actual code that runs the lasers and does the mechanism control and all of that fun stuff. Some of this code is extremely old, and so it dates back to the late 80s, so it's written in C and C++ of course, but it's not just C, it's really scary bad C, the kind of C code where you pass devoid star all the time and then just cast back on the other side because you know what it is, or we actually still have some functions that are K and R style prototype where the compiler will literally let you pass whatever you want to anything. So as you might expect, we have a lot of the issues that such a code base would naturally have, memory leak, security issues, it's not a great situation. So the question was how can we start digging our way out of this technical debt, and we all know that Rust solves a lot of these problems, but the question is how do we introduce Rust without breaking the code that we have, and not only that, without breaking the people that we have, right? We have a lot of C developers that are comfortable with C, don't necessarily want to learn any new language, especially one with a reputation as being difficult to get used to. But you might also ask if you're not very familiar with the laser printer industry, why do we even need an operating system? You're just putting some toner dots on a piece of paper and running some motors, that seems to be easy. Well, here's the list of things that we had to deal with, I just filled up the slide until, came up with a thing until I filled up the slide, it didn't take very long, but we have to authenticate against Active Directory, Windows Domain, so that users can log into the printer. We have to be able to pull files from Windows file shares and push to Windows file shares. Some of our devices actually have a Java virtual machine where we run third-party code that's been developed so that you can have custom workflows and do lots of whatever third-party things want to do. RFID Badge Readers, again, for authentication, we even support the CAC standard, which is used by Department of Defense, it's a smart card standard, so lots of code goes into that. We are an Internet of Thing device, we were doing that really before it had a name, but we're your DHCP Ethernet Wi-Fi, we even have printers that can be directly connected with fiber, I don't know who wants to do that, but apparently somebody did. We have an embedded web server, so like I mentioned, we're running Apache, so you can hit port 80 or port 443 and get a control panel and settings and debug tools and all sorts of fun stuff. Airprint, Google Cloud Print, we can send emails, we can do optical character recognition on the printer, we have to deal with security compliance for government, so we have to do audit logging. So lots and lots of stuff, clearly we need an operating system to help us to manage all of these things. Even though this code is newer, it's not all legacy like the low-level printing stuff, unfortunately it's not all that much better. It's still written in C, it still uses pointers, it still misuses pointers. You might think that C++ 14 and 17, they do add some new features that are helpful, so like smart pointers and move semantics, the new features allow you to write better code, but unfortunately C++ doesn't require you to write better code, so people still just kind of write code the way they're used to. So our approach to get Rust into the system was to just do it. We didn't ask anybody for permission, we just kind of picked a project that was off to the side that wasn't very important and said, all right, we're going to write this in Rust, and we kind of did that for two reasons. One is that if management came along and said, no, you can't use this, then it would only take us a couple days to rewrite it and it wouldn't be that big of a deal. But also so that most people didn't have to know or care that it was there, it just kind of built and if you weren't touching that particular component, you could continue to live your life as a C developer and pretend that nothing had changed. We also wanted to keep Rust in its own processes in the early days, primarily because of the problems I mentioned earlier where people are writing through old pointers and validating random memory, we didn't want some busted C code to corrupt Rust data structures and cause the Rust thread to crash and kind of give Rust a bad name for no fault of its own. So we wanted to keep it nicely isolated on its own. Unfortunately, that experiment did go very well. As people found out about it, nobody really cared because it worked and we explained the benefits of it and people kind of bought into that and said, okay, that makes sense, keep doing that. And so that kind of gave us the confidence to go to more complex components. So we still didn't want things that had a lot of dependencies, we didn't really go to the core of the system. Being a legacy embedded system, you get a lot of the everything depends on everything effect. So we did try to stay at the edges and that was primarily so that we didn't have to deal with FFI, so we didn't want to have to generate stuff from our headers, we didn't want to have to generate headers for C code and there's extra build system complexity there as well. Unfortunately that has still gone really well and we actually have probably about a dozen people in the company now that are competent in Rust, four or five of us that are pretty passionate about it. And so we kind of now treat anything as fair game. We've done some fairly large projects based on Rust now. We particularly try to push the use of Rust if it's something that's security sensitive where memory safety really gives you a lot of benefits. If it's something that deals with protocol parsing, CERDE is really good with that, there's other good crates in the Rust ecosystem for that but we've used CERDE in multiple places and been really happy with it. Similarly for the client service stuff, I mentioned that I was on the network team and saw that we had lots of opportunity to use things like Tokyo and Futures which we've been using a little bit and we're very excited about East Inc. await coming to the standard library and looking forward to doing that. Kind of the one area that we still try to shy away from in Rust is if it's anything that requires unsafe code or if people just think they need to use unsafe, we really try to discourage that. I've joked that since I run the Git servers, if I have to, I'll put in a hook that will reject anything that adds unsafe to a.rs file. Fortunately, I've not had to do that yet. Again I know that not everyone here is an embedded person so I wanted to give a little bit of background about what is BitBake, what is this build tool that we use. The point of BitBake is to let you build your own custom Linux distribution. You can choose what architecture, you can choose the compile flags you want, you can choose how big or how small it is. It's kind of analogous to build root, that's another open source tool that does a similar thing. It's part of the Octo project which has quite a bit of industry backing, Intel's behind it, a lot of the single board computers and other systems support that. We've really gotten the benefit of the open source community and the support that's gone into it. As far as the history of the code base, it actually goes back to Gen2's eMERGE. If you've ever used Gen2, ever looked at their recipes, the syntax that they use, you'll be pretty at home with how BitBake builds things. The point of these recipes is you'll have one recipe for each software component in your computer system. You'll generate an RPM, so you'll build up a whole bunch of RPM packages and then you'll have a recipe for an image that just says, here's all the different packages that I want to pull in and it takes care of handling the dependencies and generating the root file system, putting in a cram file system, whatever type of transport that you need to actually then program it down to the device. Here's a little bit of an example recipe. You don't really have to understand most of this, but there are a couple of interesting things that you can deduce from it. One of them is that BitBake wants to fetch the code for you. It wants to know where the code comes from and it wants to do the fetch itself. That's not really the way Cargo works. Similarly, it wants to know all about your dependencies. It wants to know what components do you depend on so that if they change, BitBake knows to rebuild you. Already, we can see where BitBake and Cargo are going to be playing in the same area and might be some tension there. The way that from a very practical perspective, the way we're building our Rust code into our system is we're using a project called MetaRust and full disclosure, I am one of the co-maintainers now. I didn't start it, but I am involved in that now. What that project gives you are the BitBake recipes for the compiler itself, for the standard library, and the rules for being able to easily cross-compile stuff. It's not too hard to build Rust in a tool chain for native compilation, but if you're building it for ARM or some other CPU that you're not running on, it gets a little more difficult. MetaRust takes care of all of that component for you. Then there's another tool called Cargo BitBake. What that is is a helper program that you can run from your own crate, and it will generate for you a BitBake recipe. You can go drop that into your recipes repository, and now BitBake will be able to build your Rust code for you. In particular, it looks at the Cargo lock file and figures out what all of your direct and indirect dependencies are. This allows us to hook into BitBake's ability to fetch code. Now we can play nicely with source mirrors and caching, and really just try to play along with BitBake's model of the world as much as possible. So far, our experiences have been really positive. One of the things, I'm not the first person to have said this, but it's another vote for if it builds, there's a very good chance that it'll work. We've seen that in lots of cases where, yeah, we may have had to fight with a borrower checker a little bit, but once we got through there, it did the right thing. It worked, and not only did it work right away, it continued to work. We didn't push it out into the testers, and then they found a bunch of corner cases and holes. We wrote it, got it working, and then we're able to move on to the next thing without a lot of downstream work later on in the development cycle. The corollary, for me at least, is anytime I try to write on safe code, it will crash. I'm thinking I'm 100% on that. It's especially kind of appealing for C developers, you might think, oh, well, I'm used to writing C code. C is unsafe, therefore unsafe as Rust, and Rust can't be that much worse. But actually it is, and the reason for that is that the Rust compiler will make a lot more assumptions about the system and your code than a C compiler will. So you actually have to maintain a lot more guarantees, even in the unsafe parts of Rust that you might not be used to coming from a C background. So one of the good things is that as far as runtime memory use goes, it's fairly low, it's for our purposes low enough, and it's pretty predictable. The memory model with borrowing and putting most things onto the stack instead of the heap makes it fairly predictable, whereas in an equivalent C++ program you're more likely to rely more on dynamic memory management, which makes it a lot harder to profile and predict how that code is going to work. So really the main bad side, unfortunately there are some bad sides, are around the code size. So as I mentioned earlier, on the very low end, we have products that fit into 128 megs of NAND. And even without Rust code, stripped out about as much as you can, just the kernel and libc and systemd and a couple of very other low-level things gets you into 30 or 40 megs, and you're not even into your application code yet. So there's not a whole lot of room to play with. We did start introducing Rust when we were doing the high-end devices, so the way our product cycles go is we'll do a few high-end devices, a few low-end devices, and kind of bounce back and forth. So everything was great on the high-end devices, but then we really had to look at how efficient we were being with code size and how we were spinning that on Rust. So now normally, from a C background, I would think, okay, these multiple Rust binaries that we have, there's a lot of duplicated code between them, so we can just use a shared library, move all the code that's shared into the shared object. Now there's only one copy of it. Everything will be great. Well, MetaRust supports that a little bit. It does have the ability for libstandard to be built as a shared object, and that helps for at least that part of it, but it really doesn't help you for your intermediate crates, things like futures and Tokyo and Saturday, and all of those other guys that are going to be used in a lot of different projects. Those are still going to end up duplicated. And really, at the end of the day, Rust shared library support is not all that mature. There's no standardized ABI, so that's one of the reasons why the devs kind of discouraged that use. That's not a big problem for BitBake because it builds everything in a single pass and kind of knows when it needs to rebuild stuff to make sure everything's consistent. But then even when you do use shared libraries, there's still going to be code for generic code, stuff that gets generated. That's going to end up in the individual binaries. So for example, I'm not sure if this particular example is true, but say if you use a VEC of U8 in multiple places and multiple programs, there's going to be code that the compiler generates and that's not necessarily implemented in the standard library, so that may end up in your multiple executables as well. And this is kind of understandable because the use case for dilib and Rust is not really to reduce code size. They're not really worried about my particular case. They just want for you to be able to make a shared object that you can link against or that you can deal open for FFI purposes. So given that use case, it makes sense that they really want to put as much code as possible into that shared library so that it's standalone and self-sufficient, but that doesn't really help the case when you have multiple of them and the code size implications of that. So the first pass that we took at trying to solve this problem didn't work, but just so that no one else wants to go and try it. Our thought was, well, cargo really wants to build lots of static binaries. It wants to do the full pass all by itself. So we'll just not use cargo. We'll invoke Rust directly. We'll write a recipe for every individual crate. We'll write a recipe for regex and we'll write a recipe for memcur and all of those guys, several dozen of them, and we'll just invoke Rust C directly and generate shared objects for each one of those. And while that is bad, it wasn't as bad as it would be in other languages. At least the highly regular structure of Rust projects meant that that was kind of a tractable approach. You always have a lib.rs that you can point Rusty at and it knows how to find all the other source files based on that. If you were in C or C++, every project is completely different and there's no regularity. It's really not tractable at all. But even though you can kind of do it, it doesn't scale very well, especially when you start getting into the more complex crates like Tokyo and futures. And really what completely kills this is build.rs. If it's a crate that wants to build and run Rust code to then figure out how to build and run the rest of the Rust code, we really just have no way of being able to support that. So the current solution that we're using is, well, if having multiple Rust binaries waste a lot of code, what if we just only have a single binary? We'll take all of our Rust in the system and we'll link it into a single large object. And we call that uranium. So it's kind of a super binary, kind of like Busybox if you're familiar with that. So what Busybox does is it takes a bunch of common embedded tools, including like the shell, LS test, grep, more, all of those guys, links them into one program. So this is kind of the same idea, but with our Rust code. So the advantages of that are now that we only have a single Rust executable, we don't need shared objects, not even for libSTD. And so the nice thing about that is we no longer have the parts of libSTD that we don't use. With static linking, if you don't use it, it doesn't get pulled in. So you're not paying a code penalty for that. So that's nice. That also extends to all of our intermediate components. So now Futures and Tokyo and Surde and all of those guys, again, they're only getting linked in one time. The parts we don't use, we don't have to pay for. And similarly for any generic code that gets generated, the link time optimizer is smart enough to figure out. These are the same functions for the same purpose, even though they're being used by multiple downstream dependencies, so I only have to include them once. And from a theoretical perspective, this is almost the best you can possibly do short of the compiler just generating less code. We have only the code that we need, and we only have one single copy of it. So just to go into a little more nitty-gritty of how we implemented this in case this is an approach that might be helpful for someone else, there are downsides to it. But basically we went into all of our Rust programs and converted them from instead of being bins to being RLibs. And then we used Git submodules to kind of pull them all into a single mono repo so that BitBake can check out all of our Rust code in the system all at once. And then we can use cargo to build it all in a single pass. Then the real main function, the uranium main function, just looks at argv0 to figure out, okay, which program did you actually want to run? And then it dispatches to the correct main function and the correct crate. So like I said, that'll build everything in a single pass of cargo, which that itself has pros and cons. But it does avoid one of the downsides of this approach from C. So as a C person, if you told someone you were going to link all of the code that was completely unrelated into a single binary, you would give pause to that. And one of the reasons that you would not want to do that is that you can grow undesired, implicit dependencies from unrelated parts of your code. Because C symbol visibility is basically everything can see everything, you actually can call a function from a completely unrelated piece of code and the compiler or linker will be okay with that because at the end of the day, they're all in the executable, they're all visible, and that'll work. So Rust at least does protect against that particular software engineering danger because unless you list the dependency in your cargo tumble, you're not going to be able to see it. You won't let you use it even though you do all get linked together in the end. And so just to give a little more color to this, there's actually are still two components, the LibEranium shared object, which is where the bulk of the code is, and then just a tiny executable that all it does is thunk over to the main function and the other guy. And we do that so that in the few places where we do want to do FFI, we can directly link against this without having to have another copy and start incurring those code penalties again and also let's just do DL open. Oh, the first pass of that, we actually just had an executable built as a position independent executable and we thought, okay, well, let's just DL open that or link against that. It's technically the same thing or almost the same thing as a shared library. That doesn't work. Don't do that. So there are some downsides to this. One of the bigger ones is the build time. So because all of your Rust code is getting built in a single passive cargo, that means if any of your Rust code anywhere changes, you have to rebuild all of it. So the caching model that BitBake uses is per recipe. So if anything that is in that recipe changes, then you're going to have to rebuild that whole recipe. You don't really get to benefit a whole lot from incremental compilation, especially not for system builds. Local developers might get it a little bit, but for any kind of a system build, it's going to be a clean build from scratch and it takes a fair amount of time. It also adds some extra hurdles for developers as they're trying to integrate their code. So in addition to actually checking your code into the Rust repo that you care about, you're now also going to have to go into our Mono repo and update the get sub module over there and then finally go into the meta layer where the BitBake recipe lives and change the source rev as well. So while it is maybe just one extra step that is a 50% overhead, it's not great. And another big problem is if you have lots of different subsets of this code that you use. So in our use today, we pretty much use all of the Rust code everywhere. We just kind of have one configuration, if you will, of Rust programs. But if you wanted to have on one image, say, applets A, B, and C, and on the second image, B, C, and D, et cetera, that does not scale well at all. It's like an in factorial scaling or something. So if you have a lot of different flavors, then you're much worse off than you are in the other model that I talked about where you have a single recipe per Rust crate. So we do know about some things we can look at into the future that might make this better. And again, if you have other ideas for how we can improve this, definitely find me in the hallways or afterwards to talk about this. But one thought we've had is that we can use kind of a private registry, and that will get around some of the downsides of using the get submodule approach at least. So rather than using get submodules to track that, we can actually have a registry push version releases into our registry and consume them that way. That does require that you version your code correctly, which we're not always great about. But we think that that's probably a little more of a Rust-like way to handle the problem. But I think what would really benefit us the best is if cargo had better capability to do its work not all at once. If we were able to run cargo in one place and use the output of that run of cargo into the next place so that we could still have multiple Rust recipes, each running cargo, and each benefiting from the output of previous runs of it. I'm not a compiler guy or a cargo guy. I don't know how easy or difficult that is. But that's kind of the model that works with us because that's kind of the model that bit bake and a lot of other build tools, similar build tools, NixOS and other guys have similar problems. So I did want to circle back to and plug the Zephyr Rust project just a little bit. So that's actually primarily Tyler Hall if he wants to wave the people back there. If you're curious about this, you can talk to him afterwards. But basically this allows us to run a Rust app on the Zephyr kernel. I should probably should have said what Zephyr is. It's a real time operating system for really small, primarily Internet of Things devices. Usually it's order hundreds of K of RAM and devices that don't even have a memory management unit. So even if you wanted to run Linux on there, you really can't. So today it has the capability to call any Zephyr Syscall you want as an unsafe call. And it also has safe wrappers for a lot of those. We do have some initial support for being able to do async await for certain types of IO and not network but UART and things like that can be used with async await. And in particular it does not require no STD. A lot of the projects that target these kind of bare metal systems require that you not use the standard library at all. So the entire standard library does compile and anything without OS specific so that you're not using STD Sys, it will build. It may or may not run because a lot of it is stubbed out. But that at least makes it a lot easier to get your existing code running in this environment. It's available on GitHub and that's the link. And that's all I've got. Thank you very much.
introducing Rust into an existing embedded code it. As far as the history of the code base, it into our system is we're using a project called guarantees, even in the unsafe parts of Rust that for example, I'm not sure if this particular multiple executables as well. And this is kind of multiple of them and the code size implications the reasons that you would not want to do that is