The FreeBSD Tool Chain
Maste, Ed
As a BSD distribution FreeBSD has the concept of a base system, an integrated kernel, and core userland, which are developed, tested, and released together by a common team. The userland includes the C language runtime, the build toolchain, basic system utilities, and some third-party libraries and applications. The toolchain includes the compiler and linker, which translate source code into executable objects, the debugger, and related utilities for inspecting or modifying those objects. In this talk you will learn about the history of FreeBSD's toolchain, why certain choices were made when choosing toolchain components, the current state of the toolchain, and plans for the future. For most of its history FreeBSD relied on the GNU toolchain: the GNU Compiler Collection (GCC), the GNU binutils linker and binary inspection tools, the GNU debugger GDB. This was the case between 1993 and 2007, and the copy of the toolchain included in the FreeBSD tree was regularly updated throughout that period. In 2007 the Free Software Foundation migrated to version 3 of the GNU Public License (GPLv3), which included new restrictions that some FreeBSD users and developers found objectionable. Broad updates to the GNU toolchain ceased, and it quickly became outdated. A migration to permissively licensed toolchain components is in progress. FreeBSD now uses the Clang compiler for Tier-1 architectures, with ongoing effort to switch to Clang for the remainder. Work is ongoing to use LLVM's LLD linker, and it is currently usable on amd64. Binary utilities such as objcopy and readelf are obtained from the ELF Tool Chain project on all supported FreeBSD architectures. Runtime libraries largely come from the LLVM project. This includes the libc++ C++ runtime, the compiler-rt library used by the compiler, and sanitizer runtimes used for diagnostic and debugging tools. One library, libcxxrt, comes from Pathscale. A migration to LLVM's LLDB debugger is also underway.
 Okay, good morning. I'm Ed Mast. I'm going to talk a bit about the FreeBSD toolchain. I was thinking about doing a last minute rewrite of my presentation and giving a presentation on LibTrue instead, but I think it's a little bit too exciting, so I'm going to stick with the original topic. I'll explain very briefly what I mean by the toolchain in FreeBSD, and so it's anything that transforms source code into some sort of derived binary artifact or derived artifact of that earlier object, and this can be a progression through multiple stages, source code to object file to executable, for example. Here we have, say, a C++ source file that's transformed into a executable that you might run, or we can have C source and C++ source that are combined by a linker into a single executable or a shared object, and we might have assembly source that's processed by an assembler and combined with those other components. Then we have system libraries, the C library, and other parts of the system that are linked into that executable you might produce, and we might have tools that take the binary or shared object and process it in some way, take an alpha object and do something else with it. For example, we can produce a shared object or a binary and then split it into two components, so we might have debug information in one piece and the original binary or shared object itself in another piece so that the two pieces can be managed separately. You can ship just the binary and keep the debug info for use elsewhere, installing only if you need it, that sort of thing. Then there's also ancillary tools like the debugger, which we will use for tracing program execution, single stepping, or post-mortem analysis if you have a crash, and then tools like editors are broadly speaking part of the tool chain but not part of my presentation. I'm talking about really the tools that we ship with the base system in FreeBSD to be able to make it self-host, be able to reproduce itself, and basic ancillary tools, so the debugger is included in that. In FreeBSD, from the beginning, we were using the GNU tool chain, GCC, Binutils, GDB, and the GNU folks would release new versions on a regular basis, and we would bring those versions into FreeBSD. The light blue, cyan, upper or gray, upper bar represents release dates from GCC and GDB components, and then the purple, magenta bar below represents the point at which we imported it into FreeBSD. You can see that for the most part, within a month or maybe a little bit longer, we would import a new version of those components, tracking most upstream releases. There are a few exceptions. Sometimes it took a little while longer to import into FreeBSD, but we followed along fairly closely. The FSF kept releasing new versions, and GCC at least, we were bringing new versions in. You notice that in 2005, it was the last time we imported a GDB debugger, and there are a couple of different reasons for that, but basically, every time a new upstream release would come out, it might need changes to adapt to newer versions of FreeBSD, and over time, there were growing difficulties collaborating within the project and with upstream, and so it became more and more difficult to bring new versions of GDB into the FreeBSD-based system. There's some history there that some folks in the project can share over a period. Largely predates my direct involvement in these things, but in any case, it doesn't really matter. The point is that since more than a decade ago, we haven't imported a new version of the debugger, and we also stopped importing new versions of GCC. Why is that? Well, in the middle of 2007, the Free Software Foundation released version three of the GPL, and it had some restrictions that were unpalatable to a number of FreeBSD developers and FreeBSD users. We basically, as a project, decided, and I think this is true for other BSDs in general, decided we much prefer permissively licensed toolchain components, and in FreeBSD, we just stopped importing new versions of GDB and GCC. Some other BSDs have split it so that they have a GPLv2 and a GPLv3 toolchain in the tree. OpenBSD, I think, doesn't have a GPLv3 tree at all. NetBSD, I think, has a GPLv3 compiler for architectures that require it, but FreeBSD and OpenBSD, we just said we don't want GPLv3. A search came on for a new toolchain, and in FreeBSD, we've got quite a lot of architectures. This is kind of a high-level map of everything from relating FreeBSD architecture support, FreeBSD versions, and the specific component, the compiler, assembler, et cetera, and the source of that toolchain component for that architecture and that version. I don't expect you to actually be able to read it. My point in showing all of this is just to really demonstrate from a high-level, we have a multitude of different components coming from different sources in the FreeBSD toolchain on different architectures. One thing that is interesting that you can see from this high-level view, so if you look at the columns here, which are FreeBSD versions from left to right, FreeBSD 9.x is on the left, and earlier versions basically would follow the same sources, just different versions. You can see that in FreeBSD 9 and earlier, it's all read. We basically used a single source of toolchain components leading up to FreeBSD 9. I'll explain the colors in just a moment, but we basically have a single cohesive toolchain across all architectures. Starting in FreeBSD 10, we switched to using Clang on a number of architectures and still had GCC here on a number of them. You can see that over time, as we progressed, we get more of this sort of mix. In some places, we have this light yellow color showing up. We're in a world right now in FreeBSD where we have three main sources of components leading into the toolchain. It's not a great place to be, but as we go sort of into the future, you can see that we're looking to get back to one or two, converging back onto one or two. With a color scheme, basically, the red were tools that are in the FreeBSD tree, basically the GPL GNU tools, and are outdated at this point. The green ones are maintained tools in the FreeBSD tree, and so this is now Clang and Elf toolchain and other similar tools. The new category that's in here relatively recently is this yellow one that's tools that are coming out of the FreeBSD ports tree, and I'll explain a bit more about that in a moment. These basically document the state of the world as it exists today. Clearly, the tools weren't outdated obsolete tools at the time we imported them. They were just the GNU tools that we imported, and they were up to date, and they just haven't been updated in a long time. And specifically at the compiler, the CNC++ compiler, as I said, we had GCC across all architectures. In FreeBSD 10, in January 2014, we switched to Clang for basically our tier one architectures. And as we go across from left to right, basically FreeBSD 10, 10.0 was the first release with Clang, 10.x represents stable 10 as it exists in the tree today. FreeBSD 11 was the FreeBSD 11 release, and then 11.x is stable 11 as it exists today. What I'm calling soon here is either what's in FreeBSD head, or maybe is a work in progress branch or there's patches in review. It's either in the tree right now, or it's sort of right on the horizon, and later represents maybe what we hope to do for 12.0 or after 12.0. So we still used, even after we switched to Clang, we still used the old GCC for the tier two and lower architectures MIPS PowerPC Spark 64. And one thing that's interesting, and it's sort of, we almost sort of needed another column which is actually right now and very soon. But RISC-5 represents the beginning of an experiment for us where we rely, we have an architecture in the tree that relies on external tool chain. So this diverges from the historical BSD model of having everything in the tree and have it self-hosting. This basically was a way of allowing us to develop and work with RISC-5 without being stuck on having a tool chain available. It makes releases tricky and it makes development somewhat tricky, but it's still a much more convenient user workflow than alternatives. Warner? So what did we do for the tool chain though? How did we actually ship it? We didn't do releases until we had the tool chain in the tree. So for BSD, we started off with I386 as the primary architecture, added alpha in the 3.x time frame and over time, a couple of architectures have been added and, or lots of architectures have been added, a few architectures have been removed to get us to where we are today. So as we did with GCC early on, we've been tracking updates to Clang on an ongoing basis. So each time there was a Clang update, 3.3, 3.4, 3.5, 3.9, 4.0, and then after 4.0 comes 5.0 and then next 6.0. So the numbering scheme has changed, but as new releases came out from Clang and LLVM, we imported them into FreeBSD and we do that for many reasons. Bugfix is new C++ and C standard support. So 3.3, when we switched to it, it had full C++11 support, 3.4 added full C++14 support, and C++1Z features are kind of scattered throughout later versions so that Clang should have full C++1Z support when it actually becomes a final standard. And as we bring in new compilers, existing code might fail to build. So either the base system or ports, we may find that the source code that used to build just fine with the older compiler no longer does for a number of different reasons. And so we've started the last couple of releases of Clang. We've switched to a model where to stage an import, we're bringing Clang into a projects branch as, so in this case, it would be Clang 500-import to do the work of updating support Clang 5 in FreeBSD. And so that allows us to iterate on bringing the base system up to correcting the base system source code to work with Clang and then also it allows us to more easily test the ports tree with the updated compiler. So in the ports collection, it's about 27,000 different pieces of software in the FreeBSD ports tree and this is what we use to build packages from and individual end users can install or build that software using the ports tree for their own systems. So we have this concept called an Xbrun, which is an experimental build of the entire 27,000 pieces of software in the FreeBSD ports tree with some patch applied to the base system source tree or to the ports source repository. And then we collect the results and see what which ports newly fail as a result of the changes that we've introduced. So we've been doing this for the last couple of imports. This is the request for an Xbrun for the Clang 500-import branch. And with the Clang 500 branch, as it exists today, there's about 40 failing ports, which is actually quite small, under the 27,000. And 75 ports are skipped, which means those ports have a dependency on one of the ones that is now failing. And so we have a number of different kinds of cases as to why things fail when we update to a new compiler. And here's an example of imports out of MongoDB. So this is a fairly common idiom that exists in many third-party source trees where a pointer is compared with an integer, with a not an equality comparison, but either greater than zero or similar cases. It's just not valid. And Clang 5 refuses to build that. And so we'll need to either work around. We need to get a fix for these sorts of things into the upstream software eventually, but we have work arounds we can apply in the short term to continue moving on. And one thing I also want to mention is the Clang 500 branch, Clang 5 isn't yet released. It's the Clang version that's going to be the next release of Clang. So we are tracking the development that's happening in Clang's subversion repository and eventually Clang's Git in our repository. So last update we did in FreeBSD was about five or six days ago. So the Clang 500 branch in FreeBSD is up to date within, we keep it up to date within weeks of what's happening in Clang so that we can find issues long before the compiler gets released. And one of the things that that's good for is that we also find errors in Clang when we do this because we have a corpus of 27,000 pieces of software we're trying to build. And so in this case, there's an assertion that we tripped over in Clang and we'll generate a bug report and send it to the LLVM development community. The FreeBSD developers who work on the tool chain internally, generally speaking, also work in the upstream LLVM community. So Dmitri Andrik who does a lot of the, most of the work on bringing Clang versions in isn't a committer in Clang and LLVM. I'm also a committer in Clang and LLVM as well as FreeBSD. So we work very closely to make sure that we can say reduce a test case from a failure when we try and build some very large piece of software to something that's small that we can send upstream and have people iterate on to fix there. And this is something that provides an incredible amount of value to both FreeBSD and to the LLVM community and to the upstream software communities. So if we look at Clang usage in ports over time, the darker blue kind of bars show the number of make files in the ports tree for the ports make files. And you can see we're about 27,000 right now. And the orangey red color represents on the right axis here, represents the percentage of ports that are tagged with use underscore GCC which is forcing those ports to build with GCC. So that's one of the ways that we can work around problems with Clang. If some piece of software doesn't compile with Clang, either because it relies on GCC extensions or it relies on some sort of undefined behavior that GCC limits in a certain way, we can force it to use GCC and force it to use a specific version of GCC. So some of these, it's been in there for a very long time and some of the early uses were to force specific versions of GCC. You can see that right now we're under a percent or so of the ports that are forcing that use underscore GCC setting. And there's other ways that ports might end up using GCC. For example, any port right now that is tagged as using open MP gets compiled with GCC and there's a few other cases. But I'm just trying to illustrate the general trend here. The interesting thing is that you can see that in 2013 is when FreeBSD 10 came out and we made the switch to Clang for our tier one architectures. And there was a fairly significant increase in the use of forcing GCC. And then over time, it's been decreasing as we reinvestigate why some of those changes, why some of those ports were overridden to use GCC and as Clang implements functionality that supports the kinds of needs we had. So next I'll talk briefly about the assembler. So the assembler takes assembly source and turns it into an object file. And as with other things, we started off with a version of Binutils. 2.17.50 is the last Binutils version released under GPLv2. And it's the version that's used in lots of projects that have a desire not to move beyond GPLv2 tools. And of course, it was consistent across all of our architectures and has been becoming increasingly stale over time. Unlike Clang though, we didn't really have a compelling alternative for quite a while. The only exception here is ARM64, 64-bit ARM architecture where we actually, this was actually a slightly earlier experiment with using external tool chain. So for ARM64, when we released in 11.0, we shipped with the entry Clang being able to compile for ARM64, but we still relied on an external Binutils to be able to link and assemble. But the interesting thing is that now, as of FreeBSD, as in head right now, and as of FreeBSD, in FreeBSD 11 stable, what will become 11.1, we've switched to using Clang's integrated assembler. And I'll talk a little bit more about that in just a moment. Andy? Can you speak into the integrated assembler for everything that wasn't, what files? Right. So Andy points out we were using the integrated assembler here. And I'll explain a little bit more about that in just a moment as to why we use, where we use either the integrated assembler or a standalone assembler. But the interesting point is that very shortly, I expect, we'll be able to switch to using the integrated assembler for all of our architectures and switch to relying on the, a port's tree version of Binutils for the tier two and lower architectures. So the interesting thing about the assembler is that very little actually requires a user BinAS. It's not in POSIX. It's not sort of expected to be there by any standard. And so I asked for an Xbrun to try and build ports without user BinAS existing. AMD64 fared well. The WorldEnd kernel built just fine. And then I think there were, so there were 11 ports that failed on AMD64 when user BinAS didn't exist. And it was QMU, some other emulator, seven programming languages, so things like MIT Scheme and OCaml and some multimedia library didn't build. It's a relatively small set and then quite a bit of downstream fallout of skipped ports that depended on one of these that didn't build. And the reason that it gets, that we use either the integrated assembler or AS in different ways. We have assembly files that have a capital S and assembly files that have a lowercase s in the source tree. And for a lot of people, this was just sort of a mystery. The way it works is that capital S assembly files are processed by the C preprocessor and lowercase assembly files are sent to the assembler directly. And so this is across all architectures. We have 750 capital S assembly files on all of our architectures and for AMD64, that was at the time that the Xbrun was performed, that was all of the assembly files used on AMD64. In the time since the Xbrun, we ended up introducing a single assembly file that is passed directly to the assembler. And it can't be assembled by a Clang's integrated assembler because it uses GCC's macro support. Gas is macro support, which is not supported by the integrated assembler or the other alternative assemblers that are available. So things like YASM, none of them support the full set at least of macro use that they cast us. So we're a little bit stuck, but since it is only one assembly file that has this problem, there's a number of alternatives we can do. We can rewrite this single file to avoid use of the macro directive and expand it to expand it out in line. We need to do a lot of testing to make sure that we haven't broken it in doing that. We could also do something like assemble it with the GNU assembler and then disassemble it and take the disassembly and turn that back into what we're going to feed back into an assembler that doesn't support macro. None of these are particularly nice, but it's a very tractable problem because it is only one case. For I386, in a lot of cases, we can just rename the lowercase s files to uppercase s and let Clang IAS take care of it, but again, it needs a lot of testing to do it. But that's my goal for moving forward with those. Do you know what part is missing? Because the dot macro is implemented in MC. Yeah, I'm not sure what specifically. That's a good point. Another alternative is that we can work to extend the macro support in the integrated assembler to implement what we need. Next, we have the linker. And again, we're using a relatively ancient BFD linker on all architectures still until freeBSD11. And as with the assembler, sort of similar case, this is the reason we really needed to bring in the binutils port for ARM64 builds because we needed a linker. And now in freeBSD12 and in freeBSD11.1, when it comes out, we'll be using LLVM's LLD linker for ARM64. And then LLD works today for AMD64. My laptop is running freeBSD built about five days ago with LLD as the linker, and it generally works just fine. I386 has a few problems. We can't build kernel modules on I386 with LLD today. And 32-bit ARM has another problem linking libc. So they're fairly close for the other tier one or close to tier one architectures, but not fully there yet. And the goal is to get rid of the ancient binutils and the ancient GCC in time for freeBSD12. So the likely path is going to be to migrate to requiring external binutils ports to build architectures that don't have entry tool chain support. I just wanted to talk briefly about LLVM's LLD because it's quite an interesting case. LLD existed in the LLVM project for quite a while. It was based on the Mako-Adam model and had a port to ELF of that functionality building on top of infrastructure and libraries that were shared with the Adam Mako linker. But it was a fairly poor match. It didn't really work very well. So in 2015, Rui Yamma from Google implemented a new cough linker from Scratch in LLD. And it was basically two linkers in one. So it had one code path that handled Mako and ELF and old cough and one code path that was a brand new, completely decoupled cough linker. And shortly after, Michael Spencer took that design and implemented a new ELF linker. So we had three linkers in one. We had a standalone cough linker, a standalone ELF linker, and then all the rest. But the interesting thing is that about half a year later, LLD could self-host on freeBSD. So the ELF support progressed significantly faster when done this way rather than trying to build it on top of what was the common infrastructure that didn't really match. So in 2016, there was some really significant progress on LLD. Many people upstream worked on LLD, but I've listed a few folks here who worked specifically to improve freeBSD support in LLD or fixed generic LLD problems that were identified as a result of freeBSD trying to start using it. And some of the features that were implemented in 2016 that were important for us, probably the linker script and symbol versioning support were the two key things that didn't exist elsewhere and were very important for freeBSD's linker use. And I basically sort of found issues. I ran lots of tests trying to use, trying to build world and build kernel with LLD and submitted bugs and worked with the upstream folks to get those in and have a lot of sort of minor improvements, bug fixes across the tree. And then once we had a linker that was getting close, we get asked for an Xbrun with LLD installed as user bin LD. And in this case, it was actually quite surprising, only 270 ports failed when trying to use LLD and about 1,000 were skipped because they depended on one of the ones that failed, which I think for swapping out a linker with one that's only been around for a relatively short period of time is quite impressive. So we got about 94% or so of the ports tree to build with LLD. And there's one key issue that's preventing a lot of them, that's responsible for a lot of those failures. A lot of software ends up trying to access a protected symbol from a shared library and a symbol defined with a protected visibility. With some hackish patches to work around that in the ports tree, basically, it's fairly straightforward, although hackish to get individual ports to build and avoid this issue, we get to all but around 1% or 2% of the ports building with LLD. So my expectation for moving forward on this will be very similar to what we did with Clang, or with the introduction of Clang. We'll add some annotation in individual ports make files to say that they don't work with LLD or that they require binutils. And over time, as either the upstream port gets fixed to avoid this problem or as LLD gains new functionality, we'll strip those back out and allow those ports to build with the entry linker. So my expectation is that we will make that change before for BSD12. So this graph here, I think, really shows why LLD is interesting to a lot of developers. So this is the time to, the real time for linking a Clang release build. And the BFD linker that we have in base, GNU-LD, takes over 40 seconds of real time. New versions of BFD linker from ports, we're down to 3.7 seconds, so a pretty significant beat up. Gold is about a second and a half, and then LLD threaded or non-threaded is about half a second. And the difference between 3.7 seconds, 1.4, 0.6, 0.5, that's not a showstopper. But when we look at linking something even larger, so a debug build of Clang in this case, it was almost five minutes for BFD to link that. A minute and a half for the BFD linker from ports. Gold was over half a minute, and then LLD, almost 19 seconds without threads and 11.6 seconds with threads. And this is on some arbitrary desktop machine that I tested on. Which file system? Pardon me? Which file system? On UFS. But we achieved a 25% speed up over the BFD linker in the base system. There's some really interesting performance numbers in a more scientific benchmarking perspective, running the link on a memory file system. My goal here was to just kind of see for the usual free BFD developer workflow of building the base system, what kind of effects are we going to expect to see as a direct end user, as a direct developer. And it's pretty significant. And LLD also has been intended to support threaded operation from the beginning. And this is just a close up view of the gold versus LLD unthreaded versus LLD threaded performance from that last graph. So we're about two times unthreaded gold, and about three times when LLD is threaded. In this case, gold was not threaded. This was the default configuration in the ports tree. So in the ports tree, gold could achieve a speed up by turning on threads. But I expect that there's some reason that hasn't been done, either because it wasn't stable at the time, or I'm not sure, possibly some other reasons. So next steps for the linker. There's a few simple things and a few more exciting things that we can do when we have LLD available. One of them, the fairly simple ones, is to enable build ID. So this is a scheme that takes a hash of at least all of the significant components of the shared object or executable. It could be the entire thing, but it doesn't have to be. At least all of the loadable significant parts of that executable generates a unique identifier for that binary and stores it in an elf section. And the reason this is interesting is it allows us to do things like locate the original binary for core file. So the build ID gets stored into core files and the original binary can be found, located automatically if you have the core file. It can be used to tie together a binary and its debug objects. And so either to locate the debug file for a binary, and also to confirm that they're a proper match. LTO is also a much more significant improvement that's going to come out of using LLD. So LLTO is link time optimization. Its whole program optimization performed at link time. And in brief, what happens is that all of the intermediate objects that are combined into an executable or shared object at final link time are stored as LLVM IR and only converted to machine instructions when that final link happens. And this allows LLVM's optimizer to run over the entire program and either eliminate unused code or perform any of the sort of standard optimizations that LLVM can do across the entire program, not just a single object at a time. We'll add a mechanism to override the linker for individual ports, as I alluded to earlier. And once I think that as soon as that's done, we'll be able to switch AMD64 over to LLD. Jim, you had a question? Jim points out turning on L03 or L04 loses explicit B0. So I think, yeah, specific cases like that, we will definitely need to ensure that regressions aren't introduced. The LTO work specifically is a little bit more experimental in general at this point. It is possible with what's in the FreeBSD tree today to start experimenting with LTO builds, but there's a few things that are a little bit hacky and it's not really something we can do in production just yet. And then following that, we need effort on some of the slightly less popular architectures. At this point, 32-bit ARM and X86 fall into that category. They're fairly close with LLD, but not quite there. MIPS and PowerPC are also coming along, but need some significant effort. And then working through the backlog of issues in the ports tree. Basically, all of the cases where we've used an override to say use the binutils LD explicitly, we'll want to go through and take the root cause why we had to enable that and fix that in LLD or in the upstream port. We have quite a few assorted small binary utilities in FreeBSD. As with the assembler and the linker mentioned earlier in the original case, these typically came from the binutils project. And so we had binutils 2.17.50 up until 10.x. In FreeBSD 11, we switched to the ELF tool chain project as a source for these small tools with the exception of ARM64, which had the external binutils in addition. So although we really only needed the linker from binutils for ARM64, we actually ended up with two copies of NM and two copies of strings and all those sorts of tools, the internal ones provided by FreeBSD and the ones from the external port. So ELF tool chain is a project that was started by Joseph Kashi and Kai Wang, who were FreeBSD developers, and took a number of bespoke tools from FreeBSD, so things like AR and ELF dump, moved them into an external repository, and tried to develop them there with a goal of bringing other people outside of the FreeBSD project into working on a shared non-Ginu tool chain. It's unclear how successful that was. The project has attracted contributions from other than the FreeBSD community and other than the original developers, but not significantly so yet. But in any case, they started with a few tools obtained from FreeBSD and then went through and implemented new versions or additional tools in the collection. So this is a map of sort of the native, POSIX and historic native tools and their relationship to ELF tool chain. So you can see that for things like C++ Filt, NM, size, strings, and strip, we now get out of ELF tool chain instead of their historical sources. ELF tool chain provides a very rough implementation of a linker. It's not something that we're looking at using. And there's a number of tools that were introduced by GNU that ELF tool chain provides implementations of as well. So read ELF I think is a pretty common one. ELF copy is the tool for manipulating ELF objects and converting them from one format to another or stripping sections or doing that sort of thing. It's in GNU Benutils, it's called Obstump, and the ELF tool chain people called it Elftump originally because it operated only on ELF files. It has since grown to be able to read and write binary files and output SRAC and hex and other formats. So in Fribsd we install it as Obstcopy and it's command line equivalent and basically functionally equivalent to what GNU Obstcopy provides. And ELF tool chain has a number of libraries included with it as well. LibElftc and LibPE are internal libraries to support the rest of the tools, but LibElft for reading and writing ELF files and LibDwarf for manipulating dwarf debug info. Elftool chain provides a separate implementation of those. So LibMC is their planned equivalent to BFD basically. It's a machine code library. It would be used for the assembler and the disassembler and that sort of thing. So we have a collection of tools that are still bespoke in Fribsd. I would like to move these to come from the same sources as others. Right now we have AR for example or ELF dump. We have some changes in Fribsd that have happened since the ELF tool chain split off and we've had changes in the ELF tool chain project since they started. So it would be a regression for us to just drop what we have in the base system and switch to these, but I would like to rationalize them and bring them all from a single source. And the one question we have down here is obj dump, which is it's also not a required tool and we could just say anyone who needs to use obj dump, install it from the ports. It really is more important to have obj dump as a developer. It's generally not a good idea for scripts and things to rely on obj dump. But in any case, LLVM has an LLVM obj dump, which is broadly compatible except many of the same command line arguments. The output format is a bit different, but it does provide the use cases that most people use obj dump for. So disassembling object files or mixed source and disassembly LLVM obj dump works reasonably well for that. What are the changes to AR? One thing we've done in AR in FreeBSD for sure that isn't in Alpha Toolchains copy is added CAPSCOM compartmentalization support to it. There's a few other minor changes. It's not a lot of work, I think, to rationalize the two. It's just that the AR that we have in the base system basically is completely usable for us. So there's not a huge benefit. We're not getting any new features by switching to migrating to the other one. It really is just to avoid having conflicting tools. The one reason, though, that I do want to try and reduce the number of individual sources for tools like this is that for LTO, we want to have an AR that can deal with LLVM IR in object files. The one thing, FreeBSD also uses the least bit, extremely small. That's going to allow the electric possession to slow the operation of the archives and to link the kind of files. So there may be performance reasons as well. LLVM has an LLVM AR that we should look at using as well. But all that will happen as we start trying to flesh out LTO plans. What I would say is that for small tools that you would expect to install on an end target system, so things like strings or size, you may well want one of those installed on your target even if you don't have a full tool chain, strings especially. And so for those tools, I expect that we'll continue to use a small version out of ELF tool chain even if we did start importing heavier weight LLVM versions of the tools in general. A few other miscellaneous tools. We have a device tree compiler. So this is originated sort of in open firmware. And so device tree in general is a scheme for describing hardware and is typically of interest in free BSD for embedded ARM devices. And David Chisnell implemented a bespoke version of the device tree compiler. The upstream device tree compiler, the compiler itself is GPLed. There's a corresponding library that's dual BSD and GPL licensed that can parse and operate on device tree files. But the compiler itself that turns device tree source files into device tree binaries is GPLed. David Chisnell has an implementation that we have in the source tree and is under a build time knob to turn it on. It was on for some period of time. An update to the DTS files in the free BSD tree basically made it stop working. It didn't support some features that were added in newer DTS files. It now is fully functional and implements all of the support that we need. So it's a matter of testing and verification to make sure that all of the platforms we care about continue to work if we want to switch it back on. Warner? Overlight. Overlight work, yes. Really? Yeah. Yeah. So I... It's a little bit important. Last time I tried it, that was the impediment. When did you try it? Well, like January? I believe I've imported it since then. I'll try again. Yeah. Maybe by the end of the talk you can let me know. So I have built the entire free BSD universe using the device... The BSD license device tree compiler set as the default. So the debugger, it's not required for self-hosting, but it is a pretty important component of a traditional BSD-based system. And again, we had a outdated version of new GDB across all architectures. And in FreeBSD 11, we imported LLVM's LLDB. So this is a project that originated initially in Apple, and it's the debugger that went with Xcode. Now Apple and Google are putting pretty significant amounts of effort into it. It still has a ways to go both upstream and in FreeBSD. And it's not completely compatible and doesn't cover all cases. I'll mention a bit about that in a moment. So I expected our path forward for the debugger in FreeBSD will be a combination of using LLDB in the tree and first class GDB port support. So one of the things that's happened since 10 years ago plus when we had the outdated GDB is John Baldwin has put a lot of effort into updating the GDB ports and getting all of the architecture support that existed only in FreeBSD into upstream GDB port. And so for people who have finger memory who are really familiar with GDB, the GDB ports on FreeBSD now work really well and support kernel debugging. I'll mention that in a second. LLDB is built as a set of reusable components as with the sort of general LLVM philosophy and it builds on components that come from Clang or LLVM. So one of the interesting things about that is that the expression parser in LLDB is actually just Clang. So if you say print some sort of complicated C++ expression, it's Clang turning that internally into LLVM IR and executing it. So it provides a very high fidelity expression parser. And the other thing that's interesting about it is it is usable as a component from a front end or from a scripting language. So there's the traditional LLDB command line that we provide in FreeBSD and it uses the LLDB library. There's a Python. You can use LLDB as a Python module. So this is not invoking Python scripts from within the debugger but importing the debugger into a Python script. And then it can function as the support for various IDE. So Xcode is the usual example. There's some open source ones that can make use of it as well. Next steps for the debugger. We need to add AVX register support for AMD64 to make it a fully usable userland debugger. It's a fairly straightforward project. The debug server is a small stub that's used typically for remote debugging. So it will run on a target system that's separate from where you're running the debugger and executes the debuggy under that stub. And we don't have one available for FreeBSD today. There's a few different implementations that might work. But one of the interesting things about LLDB is that it actually uses, on other architectures, so not on FreeBSD and not on the original import of the Linux support but on the Linux support is now built this way. LLDB actually uses the debug server even for local debugging. So the interesting thing about that is it means that you know your remote debugging case is always going to work properly because remote debugging and local debugging is the same. And it also simplifies the debugger internals a little bit because it doesn't have to have this other process attached to it, to the process that's driving your UI and is heavily threaded and things like that. We need to add kernel debug support. So both crash dump handling and live debugging against DevMem. And fork following is a nice feature to have to make it a fully usable userland debugger. We didn't have that in GDB 6.1. It's not at all a hard requirement but it is something that improves usability quite a bit. We also have detrace related tools. CTF is the compact format for debug information that's used to drive detrace. And the way we generate it right now is using two tools that imported from OpenSolaris slash Alumos. So we first compile an object file with dwarf. CTF convert runs on that dwarf information in the object file and converts it to CTF. And then CTF merge takes the CTF from a number of object files and combines it into one. It's a little bit awkward. OpenBSD developers are working on a BSD licensed implementation rather than the CDDL CTF convert and CTF merge that do it as a single step. And so we're looking at that work that's going on. And then longer term, it would be really nice to integrate this natively and transparently into the tool chain. So the Clang compiler driver and linker can just produce the output that we want directly. Have you? And finally, the last sort of component of the tool chain that we're going to talk about is all of the runtime support libraries. And there's quite a lot to go over in this very brief remaining time. So we've got the compiler runtime that supports optimized routines and implements functionality that the compiler doesn't want to implement in line. So say floating point math conversions, byte swapping routines, assembly optimized versions of things like that. The compiler, if there's anything that's too complex for the compiler to generate directly, it'll emit a call to one of these support routines. In the GCC world, that's libGCC. In the LLVM world, the project is called compiler RT. We've sort of done an incremental migration to using the LLVM versions over time because we have cases where we're building some architectures with GCC and some architectures with Clang, or during the transition we had a build knob. And so you might build sometimes with Clang and sometimes with GCC on the same architecture. So quite a while ago, long before we even switched to Clang, we used LLVM's implementation of the compiler built-ins for the static linking case at least, libGCC.a. And much more recently, so only at the end of last year, we've introduced libGCC.s, the shared version of the support routines being built from LLVM's implementation. There's a little bit more about that in a second. libGCC, it is the core piece of runtime support on any BSD. It has existed in some form or another basically longer than I've been alive. But recently we have a couple of updates to support C++11 in libGCC. C++ requires a couple of different runtime support libraries. There's very low level support, which provides foundation building blocks for exceptions type info, a demangler so that C++mangle names can be converted back into something representable to a user. In the GCC world, this is called libSup C++. In 2011, the previous D Foundation and that BSD Foundation worked together to obtain a BSD licensed C++ runtime from PathScale. That's called libCXXRT. And subsequently there's also been an open-sourced version in the LLVM project called libCXXABI. It didn't exist when we first did this work, but it does now. The standard library in C++, the collection of classes and functions for containers and operations on those containers like sorting and strings and streams and things like that. The libSpaner C++ that we have in the tree comes along with the old GCC that we had and it predates C++11. So we have now libC++, well, for six years we have libC++ that is following along and updated with all of ongoing C standard newer C++ standards. And libC++ is C++11 code itself and can't be compiled with the old ancient GCC that we have in the tree. So if you build, say, MIPS with the old GCC, that build will include lib standard C++. If you build MIPS with a out-of-tree C++11 GCC compiler, it will use libC++ as well. And the reason we want to use libC++ on all of the architectures, even if GCC is the normal compiler so that we can share the same C++ library in general and use, avoid ABI incompatibility. So along with, I really for C++ is exception unwinding, but it's used by C code as well for cases like forced P thread cleanup. This ties into the libGCC that I mentioned earlier and it's a little bit awkward because you want only one copy of the exception unwinder in any one process, which is why there's this combination of libGCC.a, libGCC.s.so, and libGCC.eh.so. And that also prevented us originally from switching the shared versions of these to use the LLVM provided ones because there wasn't a replacement unwinder for us. We're now using LLVM's lib unwind. This is, it came out of Apple as well. It was, they realized there was one piece missing from a fully licensed runtime stack. So they released it in October 2013 and it wasn't fully usable yet, but it went through a lot of development upstream and we've since brought it into FreeBSD and it now, as of Octoberish, provides, it's used as the unwinder in libGCC.s. We have a bunch of standardizers that come from LLVM. These exist upstream. In FreeBSD right now, we support only the, out of the box in the tree, we support only the hydro sanitizer and undefined behavior sanitizer. So these are used, these can be used as a replacement for tools like Valgrand to find errors in your program at runtime. And finally, OpenMP. So OpenMP is a shared memory API for parallel programming and in the GCC world, it's provided, the runtime is provided by libGOMP. In FreeBSD, we had Clang, if you asked for OpenMP support, it would link against libGOMP, but it actually didn't do anything, so it was relatively useless. It now will link against LLVM's libOMP, which was initially donated by Intel and called libIOMP at the time. So the Clang in the base system will link against and it calls for libOMP for OpenMP support and we provide a port called develop.openMP to provide that runtime library. The goal is to include the OpenMP runtime in the base system for exactly this reason, for many reasons, but this is kind of the highlight of the problem that we face. We want the tool chain to be able to be usable for actual end user cases and this comes up fairly regularly that when you build arbitrary software using the defaults out of the box on FreeBSD, OpenMP support doesn't work and things don't perform as well. And it's relatively straightforward to build and install the OpenMP runtime, but we have all the rest of the tool chain in the base system. There's not really a good reason for us to say we don't want to provide the runtime and you have to do the run through, jump to some extra hoops to get your code to run quickly. And finally here's a list of the folks who have been heavily involved in the tool chain over time so I just want to put them up here and call them out for all the work that's gone into this over the last seven or eight years. Okay, thank you. So I notice on your time frame slides there was one similarity in all of them that was Spark 6.4 is, can you touch on that a bit? So there's a faction of FreeBSD developers and users who wish to eject Spark 6.4 from the tree. I don't consider myself among that group, but my take on it is I'm perfectly happy to leave it there as long as someone wants to maintain or keep it building. If no one else has an interest in doing so then it clearly won't be sticking around. It won't remain in the tree if it becomes unbuildable and no one takes an interest in it. One of the problems we have in FreeBSD with Spark 6.4 is we haven't updated, we don't support any recent Spark 6.4 hardware. So the only thing we support is hardware that has been obsolete for sort of more than a decade. So it's not a particularly interesting platform for us. So are there any issues on the LLVM or toolchain side or is it primarily a FreeBSD interest? So the Spark 6.4 support that exists in LLVM and Clang is exceedingly basic and minimal and I don't know, Rafael, do you know if that's going to be even maintained? There's more videos I think they use Spark 4 so they added whatever they needed to work in but that's a different basis. I don't think there's any interest in doing so. I think Spark 6.4 support in upstream LLVM is really going to depend on you guys and OpenBSD guys and NetBSD guys pursuing it if that is really something you want to do. For us, I think we're more than happy to say for as long as Spark 6.4 support remains in FreeBSD, it's going to require an external toolchain. Anyone else? Sure. The question was how does OpenBSD handle user-defined probes in the toolchain that they're working on and Baptiste doesn't know. Mark? Could you explain that? So, when I work with not done, it's one of the things I try to figure out if it's 100% the maximum. So, it should be okay. I can give you, I'm just editing them to the latest updates, I can give you a link if you want to, just to explain what they are with them. Yeah, very welcome. And do you know if they're using helpful changes or different? No, they, so the, what are you saying? Well, I'll give you the backwards compatible with the new remote debugging protocols. The question is, will LDB be backwards compatible with the new remote debugging protocols? Yes, it already is. It uses the same basic protocol as GDB does when GDB talks to its remote stub. There are a couple of extensions that have been added for LLVM. So, when LLVM is talking to its own debug server, it uses a few things that don't exist in the regular GDB protocol. But broadly speaking, it should be compatible and can be, you know, if it isn't, it can probably be considered a bug and fix as opposed to just an architectural mismatch. Okay, no more questions? Thank you very much. So, for those who didn't catch it, I don't just, well, just very much at none left, there is a ZFS session, well, a bit of further session, we're in 11.10. And here you're going to have my shark, I think, who's going to host BSE user development session. Well, both sessions. You have time to grab a bite if you want to come back. Otherwise, everybody has to be back in the session for 1.30. Thanks. Close there. Keep the wallet. Sure, leave it in the middle.
 Okay, good morning. I'm Ed Mast. I'm going to talk a bit about the FreeBSD toolchain.  I was thinking about doing a last minute rewrite of my presentation and giving a presentation  on LibTrue instead, but I think it's a little bit too exciting, so I'm going to stick with  the original topic.  I'll explain very briefly what I mean by the toolchain in FreeBSD, and so it's anything  that transforms source code into some sort of derived binary artifact or derived artifact  of that earlier object, and this can be a progression through multiple stages, source  code to object file to executable, for example.  Here we have, say, a C++ source file that's transformed into a executable that you might  run, or we can have C source and C++ source that are combined by a linker into a single  executable or a shared object, and we might have assembly source that's processed by an  assembler and combined with those other components.  Then we have system libraries, the C library, and other parts of the system that are linked  into that executable you might produce, and we might have tools that take the binary or  shared object and process it in some way, take an alpha object and do something else  with it.  For example, we can produce a shared object or a binary and then split it into two components,  so we might have debug information in one piece and the original binary or shared object  itself in another piece so that the two pieces can be managed separately.  You can ship just the binary and keep the debug info for use elsewhere, installing  only if you need it, that sort of thing.  Then there's also ancillary tools like the debugger, which we will use for tracing program  execution, single stepping, or post-mortem analysis if you have a crash, and then tools  like editors are broadly speaking part of the tool chain but not part of my presentation.  I'm talking about really the tools that we ship with the base system in FreeBSD to be  able to make it self-host, be able to reproduce itself, and basic ancillary tools, so the  debugger is included in that.  In FreeBSD, from the beginning, we were using the GNU tool chain, GCC, Binutils, GDB, and  the GNU folks would release new versions on a regular basis, and we would bring those  versions into FreeBSD.  The light blue, cyan, upper or gray, upper bar represents release dates from GCC and  GDB components, and then the purple, magenta bar below represents the point at which we  imported it into FreeBSD.  You can see that for the most part, within a month or maybe a little bit longer, we would  import a new version of those components, tracking most upstream releases.  There are a few exceptions.  Sometimes it took a little while longer to import into FreeBSD, but we followed along  fairly closely.  The FSF kept releasing new versions, and GCC at least, we were bringing new versions  in.  You notice that in 2005, it was the last time we imported a GDB debugger, and there are  a couple of different reasons for that, but basically, every time a new upstream release  would come out, it might need changes to adapt to newer versions of FreeBSD, and over time,  there were growing difficulties collaborating within the project and with upstream, and  so it became more and more difficult to bring new versions of GDB into the FreeBSD-based  system.  There's some history there that some folks in the project can share over a period.  Largely predates my direct involvement in these things, but in any case, it doesn't  really matter.  The point is that since more than a decade ago, we haven't imported a new version of  the debugger, and we also stopped importing new versions of GCC.  Why is that?  Well, in the middle of 2007, the Free Software Foundation released version three of the GPL,  and it had some restrictions that were unpalatable to a number of FreeBSD developers and FreeBSD  users.  We basically, as a project, decided, and I think this is true for other BSDs in general,  decided we much prefer permissively licensed toolchain components, and in FreeBSD, we just  stopped importing new versions of GDB and GCC.  Some other BSDs have split it so that they have a GPLv2 and a GPLv3 toolchain in the  tree.  OpenBSD, I think, doesn't have a GPLv3 tree at all.  NetBSD, I think, has a GPLv3 compiler for architectures that require it, but FreeBSD  and OpenBSD, we just said we don't want GPLv3.  A search came on for a new toolchain, and in FreeBSD, we've got quite a lot of architectures.  This is kind of a high-level map of everything from relating FreeBSD architecture support,  FreeBSD versions, and the specific component, the compiler, assembler, et cetera, and the  source of that toolchain component for that architecture and that version.  I don't expect you to actually be able to read it.  My point in showing all of this is just to really demonstrate from a high-level, we have  a multitude of different components coming from different sources in the FreeBSD toolchain  on different architectures.  One thing that is interesting that you can see from this high-level view, so if you look  at the columns here, which are FreeBSD versions from left to right, FreeBSD 9.x is on the  left, and earlier versions basically would follow the same sources, just different versions.  You can see that in FreeBSD 9 and earlier, it's all read.  We basically used a single source of toolchain components leading up to FreeBSD 9.  I'll explain the colors in just a moment, but we basically have a single cohesive toolchain  across all architectures.  Starting in FreeBSD 10, we switched to using Clang on a number of architectures and still  had GCC here on a number of them.  You can see that over time, as we progressed, we get more of this sort of mix.  In some places, we have this light yellow color showing up.  We're in a world right now in FreeBSD where we have three main sources of components leading  into the toolchain.  It's not a great place to be, but as we go sort of into the future, you can see that  we're looking to get back to one or two, converging back onto one or two.  With a color scheme, basically, the red were tools that are in the FreeBSD tree, basically  the GPL GNU tools, and are outdated at this point.  The green ones are maintained tools in the FreeBSD tree, and so this is now Clang and  Elf toolchain and other similar tools.  The new category that's in here relatively recently is this yellow one that's tools  that are coming out of the FreeBSD ports tree, and I'll explain a bit more about that in  a moment.  These basically document the state of the world as it exists today.  Clearly, the tools weren't outdated obsolete tools at the time we imported them.  They were just the GNU tools that we imported, and they were up to date, and they just haven't  been updated in a long time.  And specifically at the compiler, the CNC++ compiler, as I said, we had GCC across all  architectures.  In FreeBSD 10, in January 2014, we switched to Clang for basically our tier one architectures.  And as we go across from left to right, basically FreeBSD 10, 10.0 was the first release with  Clang, 10.x represents stable 10 as it exists in the tree today.  FreeBSD 11 was the FreeBSD 11 release, and then 11.x is stable 11 as it exists today.  What I'm calling soon here is either what's in FreeBSD head, or maybe is a work in progress  branch or there's patches in review.  It's either in the tree right now, or it's sort of right on the horizon, and later represents  maybe what we hope to do for 12.0 or after 12.0.  So we still used, even after we switched to Clang, we still used the old GCC for the tier  two and lower architectures MIPS PowerPC Spark 64.  And one thing that's interesting, and it's sort of, we almost sort of needed another  column which is actually right now and very soon.  But RISC-5 represents the beginning of an experiment for us where we rely, we have an  architecture in the tree that relies on external tool chain.  So this diverges from the historical BSD model of having everything in the tree and  have it self-hosting.  This basically was a way of allowing us to develop and work with RISC-5 without being  stuck on having a tool chain available.  It makes releases tricky and it makes development somewhat tricky, but it's still a much more  convenient user workflow than alternatives.  Warner?  So what did we do for the tool chain though?  How did we actually ship it?  We didn't do releases until we had the tool chain in the tree.  So for BSD, we started off with I386 as the primary architecture, added alpha in the 3.x  time frame and over time, a couple of architectures have been added and, or lots of architectures  have been added, a few architectures have been removed to get us to where we are today.  So as we did with GCC early on, we've been tracking updates to Clang on an ongoing basis.  So each time there was a Clang update, 3.3, 3.4, 3.5, 3.9, 4.0, and then after 4.0 comes  5.0 and then next 6.0.  So the numbering scheme has changed, but as new releases came out from Clang and LLVM,  we imported them into FreeBSD and we do that for many reasons.  Bugfix is new C++ and C standard support.  So 3.3, when we switched to it, it had full C++11 support, 3.4 added full C++14 support,  and C++1Z features are kind of scattered throughout later versions so that Clang should have full  C++1Z support when it actually becomes a final standard.  And as we bring in new compilers, existing code might fail to build.  So either the base system or ports, we may find that the source code that used to build  just fine with the older compiler no longer does for a number of different reasons.  And so we've started the last couple of releases of Clang.  We've switched to a model where to stage an import, we're bringing Clang into a projects  branch as, so in this case, it would be Clang 500-import to do the work of updating support  Clang 5 in FreeBSD.  And so that allows us to iterate on bringing the base system up to correcting the base  system source code to work with Clang and then also it allows us to more easily test  the ports tree with the updated compiler.  So in the ports collection, it's about 27,000 different pieces of software in the FreeBSD  ports tree and this is what we use to build packages from and individual end users can  install or build that software using the ports tree for their own systems.  So we have this concept called an Xbrun, which is an experimental build of the entire 27,000  pieces of software in the FreeBSD ports tree with some patch applied to the base system  source tree or to the ports source repository.  And then we collect the results and see what which ports newly fail as a result of the  changes that we've introduced.  So we've been doing this for the last couple of imports.  This is the request for an Xbrun for the Clang 500-import branch.  And with the Clang 500 branch, as it exists today, there's about 40 failing ports, which  is actually quite small, under the 27,000.  And 75 ports are skipped, which means those ports have a dependency on one of the ones  that is now failing.  And so we have a number of different kinds of cases as to why things fail when we update  to a new compiler.  And here's an example of imports out of MongoDB.  So this is a fairly common idiom that exists in many third-party source trees where a pointer  is compared with an integer, with a not an equality comparison, but either greater than  zero or similar cases.  It's just not valid.  And Clang 5 refuses to build that.  And so we'll need to either work around.  We need to get a fix for these sorts of things into the upstream software eventually, but  we have work arounds we can apply in the short term to continue moving on.  And one thing I also want to mention is the Clang 500 branch, Clang 5 isn't yet released.  It's the Clang version that's going to be the next release of Clang.  So we are tracking the development that's happening in Clang's subversion repository  and eventually Clang's Git in our repository.  So last update we did in FreeBSD was about five or six days ago.  So the Clang 500 branch in FreeBSD is up to date within, we keep it up to date within  weeks of what's happening in Clang so that we can find issues long before the compiler  gets released.  And one of the things that that's good for is that we also find errors in Clang when  we do this because we have a corpus of 27,000 pieces of software we're trying to build.  And so in this case, there's an assertion that we tripped over in Clang and we'll generate  a bug report and send it to the LLVM development community.  The FreeBSD developers who work on the tool chain internally, generally speaking, also  work in the upstream LLVM community.  So Dmitri Andrik who does a lot of the, most of the work on bringing Clang versions in  isn't a committer in Clang and LLVM.  I'm also a committer in Clang and LLVM as well as FreeBSD.  So we work very closely to make sure that we can say reduce a test case from a failure  when we try and build some very large piece of software to something that's small that  we can send upstream and have people iterate on to fix there.  And this is something that provides an incredible amount of value to both FreeBSD and to the  LLVM community and to the upstream software communities.  So if we look at Clang usage in ports over time, the darker blue kind of bars show the  number of make files in the ports tree for the ports make files.  And you can see we're about 27,000 right now.  And the orangey red color represents on the right axis here, represents the percentage  of ports that are tagged with use underscore GCC which is forcing those ports to build  with GCC.  So that's one of the ways that we can work around problems with Clang.  If some piece of software doesn't compile with Clang, either because it relies on GCC  extensions or it relies on some sort of undefined behavior that GCC limits in a certain way,  we can force it to use GCC and force it to use a specific version of GCC.  So some of these, it's been in there for a very long time and some of the early uses  were to force specific versions of GCC.  You can see that right now we're under a percent or so of the ports that are forcing  that use underscore GCC setting.  And there's other ways that ports might end up using GCC.  For example, any port right now that is tagged as using open MP gets compiled with GCC and  there's a few other cases.  But I'm just trying to illustrate the general trend here.  The interesting thing is that you can see that in 2013 is when FreeBSD 10 came out and  we made the switch to Clang for our tier one architectures.  And there was a fairly significant increase in the use of forcing GCC.  And then over time, it's been decreasing as we reinvestigate why some of those changes,  why some of those ports were overridden to use GCC and as Clang implements functionality  that supports the kinds of needs we had.  So next I'll talk briefly about the assembler.  So the assembler takes assembly source and turns it into an object file.  And as with other things, we started off with a version of Binutils.  2.17.50 is the last Binutils version released under GPLv2.  And it's the version that's used in lots of projects that have a desire not to move beyond  GPLv2 tools.  And of course, it was consistent across all of our architectures and has been becoming  increasingly stale over time.  Unlike Clang though, we didn't really have a compelling alternative for quite a while.  The only exception here is ARM64, 64-bit ARM architecture where we actually, this was  actually a slightly earlier experiment with using external tool chain.  So for ARM64, when we released in 11.0, we shipped with the entry Clang being able to  compile for ARM64, but we still relied on an external Binutils to be able to link and  assemble.  But the interesting thing is that now, as of FreeBSD, as in head right now, and as of  FreeBSD, in FreeBSD 11 stable, what will become 11.1, we've switched to using Clang's  integrated assembler.  And I'll talk a little bit more about that in just a moment.  Andy?  Can you speak into the integrated assembler for everything that wasn't, what files?  Right.  So Andy points out we were using the integrated assembler here.  And I'll explain a little bit more about that in just a moment as to why we use, where we  use either the integrated assembler or a standalone assembler.  But the interesting point is that very shortly, I expect, we'll be able to switch to using  the integrated assembler for all of our architectures and switch to relying on the, a port's tree  version of Binutils for the tier two and lower architectures.  So the interesting thing about the assembler is that very little actually requires a user  BinAS.  It's not in POSIX.  It's not sort of expected to be there by any standard.  And so I asked for an Xbrun to try and build ports without user BinAS existing.  AMD64 fared well.  The WorldEnd kernel built just fine.  And then I think there were, so there were 11 ports that failed on AMD64 when user BinAS  didn't exist.  And it was QMU, some other emulator, seven programming languages, so things like MIT Scheme  and OCaml and some multimedia library didn't build.  It's a relatively small set and then quite a bit of downstream fallout of skipped ports  that depended on one of these that didn't build.  And the reason that it gets, that we use either the integrated assembler or AS in different  ways.  We have assembly files that have a capital S and assembly files that have a lowercase  s in the source tree.  And for a lot of people, this was just sort of a mystery.  The way it works is that capital S assembly files are processed by the C preprocessor  and lowercase assembly files are sent to the assembler directly.  And so this is across all architectures.  We have 750 capital S assembly files on all of our architectures and for AMD64, that was  at the time that the Xbrun was performed, that was all of the assembly files used on  AMD64.  In the time since the Xbrun, we ended up introducing a single assembly file that is passed directly  to the assembler.  And it can't be assembled by a Clang's integrated assembler because it uses GCC's macro support.  Gas is macro support, which is not supported by the integrated assembler or the other alternative  assemblers that are available.  So things like YASM, none of them support the full set at least of macro use that they  cast us.  So we're a little bit stuck, but since it is only one assembly file that has this problem,  there's a number of alternatives we can do.  We can rewrite this single file to avoid use of the macro directive and expand it to expand  it out in line.  We need to do a lot of testing to make sure that we haven't broken it in doing that.  We could also do something like assemble it with the GNU assembler and then disassemble  it and take the disassembly and turn that back into what we're going to feed back into  an assembler that doesn't support macro.  None of these are particularly nice, but it's a very tractable problem because it is only  one case.  For I386, in a lot of cases, we can just rename the lowercase s files to uppercase s and let  Clang IAS take care of it, but again, it needs a lot of testing to do it.  But that's my goal for moving forward with those.  Do you know what part is missing?  Because the dot macro is implemented in MC.  Yeah, I'm not sure what specifically.  That's a good point.  Another alternative is that we can work to extend the macro support in the integrated  assembler to implement what we need.  Next, we have the linker.  And again, we're using a relatively ancient BFD linker on all architectures still until  freeBSD11.  And as with the assembler, sort of similar case, this is the reason we really needed  to bring in the binutils port for ARM64 builds because we needed a linker.  And now in freeBSD12 and in freeBSD11.1, when it comes out, we'll be using LLVM's  LLD linker for ARM64.  And then LLD works today for AMD64.  My laptop is running freeBSD built about five days ago with LLD as the linker, and it generally  works just fine.  I386 has a few problems.  We can't build kernel modules on I386 with LLD today.  And 32-bit ARM has another problem linking libc.  So they're fairly close for the other tier one or close to tier one architectures, but  not fully there yet.  And the goal is to get rid of the ancient binutils and the ancient GCC in time for  freeBSD12.  So the likely path is going to be to migrate to requiring external binutils ports to build  architectures that don't have entry tool chain support.  I just wanted to talk briefly about LLVM's LLD because it's quite an interesting case.  LLD existed in the LLVM project for quite a while.  It was based on the Mako-Adam model and had a port to ELF of that functionality building  on top of infrastructure and libraries that were shared with the Adam Mako linker.  But it was a fairly poor match.  It didn't really work very well.  So in 2015, Rui Yamma from Google implemented a new cough linker from Scratch in LLD.  And it was basically two linkers in one.  So it had one code path that handled Mako and ELF and old cough and one code path that  was a brand new, completely decoupled cough linker.  And shortly after, Michael Spencer took that design and implemented a new ELF linker.  So we had three linkers in one.  We had a standalone cough linker, a standalone ELF linker, and then all the rest.  But the interesting thing is that about half a year later, LLD could self-host on freeBSD.  So the ELF support progressed significantly faster when done this way rather than trying  to build it on top of what was the common infrastructure that didn't really match.  So in 2016, there was some really significant progress on LLD.  Many people upstream worked on LLD, but I've listed a few folks here who worked specifically  to improve freeBSD support in LLD or fixed generic LLD problems that were identified  as a result of freeBSD trying to start using it.  And some of the features that were implemented in 2016 that were important for us, probably  the linker script and symbol versioning support were the two key things that didn't exist  elsewhere and were very important for freeBSD's linker use.  And I basically sort of found issues.  I ran lots of tests trying to use, trying to build world and build kernel with LLD and  submitted bugs and worked with the upstream folks to get those in and have a lot of sort  of minor improvements, bug fixes across the tree.  And then once we had a linker that was getting close, we get asked for an Xbrun with LLD  installed as user bin LD.  And in this case, it was actually quite surprising, only 270 ports failed when trying to use LLD  and about 1,000 were skipped because they depended on one of the ones that failed, which I think  for swapping out a linker with one that's only been around for a relatively short period  of time is quite impressive.  So we got about 94% or so of the ports tree to build with LLD.  And there's one key issue that's preventing a lot of them, that's responsible for a lot  of those failures.  A lot of software ends up trying to access a protected symbol from a shared library and  a symbol defined with a protected visibility.  With some hackish patches to work around that in the ports tree, basically, it's fairly  straightforward, although hackish to get individual ports to build and avoid this issue, we get  to all but around 1% or 2% of the ports building with LLD.  So my expectation for moving forward on this will be very similar to what we did with Clang,  or with the introduction of Clang.  We'll add some annotation in individual ports make files to say that they don't work with  LLD or that they require binutils.  And over time, as either the upstream port gets fixed to avoid this problem or as LLD  gains new functionality, we'll strip those back out and allow those ports to build with  the entry linker.  So my expectation is that we will make that change before for BSD12.  So this graph here, I think, really shows why LLD is interesting to a lot of developers.  So this is the time to, the real time for linking a Clang release build.  And the BFD linker that we have in base, GNU-LD, takes over 40 seconds of real time.  New versions of BFD linker from ports, we're down to 3.7 seconds, so a pretty significant  beat up.  Gold is about a second and a half, and then LLD threaded or non-threaded is about half  a second.  And the difference between 3.7 seconds, 1.4, 0.6, 0.5, that's not a showstopper.  But when we look at linking something even larger, so a debug build of Clang in this  case, it was almost five minutes for BFD to link that.  A minute and a half for the BFD linker from ports.  Gold was over half a minute, and then LLD, almost 19 seconds without threads and 11.6  seconds with threads.  And this is on some arbitrary desktop machine that I tested on.  Which file system?  Pardon me?  Which file system?  On UFS.  But we achieved a 25% speed up over the BFD linker in the base system.  There's some really interesting performance numbers in a more scientific benchmarking  perspective, running the link on a memory file system.  My goal here was to just kind of see for the usual free BFD developer workflow of building  the base system, what kind of effects are we going to expect to see as a direct end  user, as a direct developer.  And it's pretty significant.  And LLD also has been intended to support threaded operation from the beginning.  And this is just a close up view of the gold versus LLD unthreaded versus LLD threaded  performance from that last graph.  So we're about two times unthreaded gold, and about three times when LLD is threaded.  In this case, gold was not threaded.  This was the default configuration in the ports tree.  So in the ports tree, gold could achieve a speed up by turning on threads.  But I expect that there's some reason that hasn't been done, either because it wasn't  stable at the time, or I'm not sure, possibly some other reasons.  So next steps for the linker.  There's a few simple things and a few more exciting things that we can do when we have  LLD available.  One of them, the fairly simple ones, is to enable build ID.  So this is a scheme that takes a hash of at least all of the significant components of  the shared object or executable.  It could be the entire thing, but it doesn't have to be.  At least all of the loadable significant parts of that executable generates a unique identifier  for that binary and stores it in an elf section.  And the reason this is interesting is it allows us to do things like locate the original binary  for core file.  So the build ID gets stored into core files and the original binary can be found, located  automatically if you have the core file.  It can be used to tie together a binary and its debug objects.  And so either to locate the debug file for a binary, and also to confirm that they're  a proper match.  LTO is also a much more significant improvement that's going to come out of using LLD.  So LLTO is link time optimization.  Its whole program optimization performed at link time.  And in brief, what happens is that all of the intermediate objects that are combined  into an executable or shared object at final link time are stored as LLVM IR and only converted  to machine instructions when that final link happens.  And this allows LLVM's optimizer to run over the entire program and either eliminate unused  code or perform any of the sort of standard optimizations that LLVM can do across the  entire program, not just a single object at a time.  We'll add a mechanism to override the linker for individual ports, as I alluded to earlier.  And once I think that as soon as that's done, we'll be able to switch AMD64 over to LLD.  Jim, you had a question?  Jim points out turning on L03 or L04 loses explicit B0.  So I think, yeah, specific cases like that, we will definitely need to ensure that regressions  aren't introduced.  The LTO work specifically is a little bit more experimental in general at this point.  It is possible with what's in the FreeBSD tree today to start experimenting with LTO  builds, but there's a few things that are a little bit hacky and it's not really something  we can do in production just yet.  And then following that, we need effort on some of the slightly less popular architectures.  At this point, 32-bit ARM and X86 fall into that category.  They're fairly close with LLD, but not quite there.  MIPS and PowerPC are also coming along, but need some significant effort.  And then working through the backlog of issues in the ports tree.  Basically, all of the cases where we've used an override to say use the binutils LD explicitly,  we'll want to go through and take the root cause why we had to enable that and fix that  in LLD or in the upstream port.  We have quite a few assorted small binary utilities in FreeBSD.  As with the assembler and the linker mentioned earlier in the original case, these typically  came from the binutils project.  And so we had binutils 2.17.50 up until 10.x.  In FreeBSD 11, we switched to the ELF tool chain project as a source for these small  tools with the exception of ARM64, which had the external binutils in addition.  So although we really only needed the linker from binutils for ARM64, we actually ended  up with two copies of NM and two copies of strings and all those sorts of tools, the  internal ones provided by FreeBSD and the ones from the external port.  So ELF tool chain is a project that was started by Joseph Kashi and Kai Wang, who were FreeBSD  developers, and took a number of bespoke tools from FreeBSD, so things like AR and  ELF dump, moved them into an external repository, and tried to develop them there with a goal  of bringing other people outside of the FreeBSD project into working on a shared non-Ginu  tool chain.  It's unclear how successful that was.  The project has attracted contributions from other than the FreeBSD community and other  than the original developers, but not significantly so yet.  But in any case, they started with a few tools obtained from FreeBSD and then went through  and implemented new versions or additional tools in the collection.  So this is a map of sort of the native, POSIX and historic native tools and their relationship  to ELF tool chain.  So you can see that for things like C++ Filt, NM, size, strings, and strip, we now get out  of ELF tool chain instead of their historical sources.  ELF tool chain provides a very rough implementation of a linker.  It's not something that we're looking at using.  And there's a number of tools that were introduced by GNU that ELF tool chain provides implementations  of as well.  So read ELF I think is a pretty common one.  ELF copy is the tool for manipulating ELF objects and converting them from one format  to another or stripping sections or doing that sort of thing.  It's in GNU Benutils, it's called Obstump, and the ELF tool chain people called it Elftump  originally because it operated only on ELF files.  It has since grown to be able to read and write binary files and output SRAC and hex  and other formats.  So in Fribsd we install it as Obstcopy and it's command line equivalent and basically  functionally equivalent to what GNU Obstcopy provides.  And ELF tool chain has a number of libraries included with it as well.  LibElftc and LibPE are internal libraries to support the rest of the tools, but LibElft  for reading and writing ELF files and LibDwarf for manipulating dwarf debug info.  Elftool chain provides a separate implementation of those.  So LibMC is their planned equivalent to BFD basically.  It's a machine code library.  It would be used for the assembler and the disassembler and that sort of thing.  So we have a collection of tools that are still bespoke in Fribsd.  I would like to move these to come from the same sources as others.  Right now we have AR for example or ELF dump.  We have some changes in Fribsd that have happened since the ELF tool chain split off  and we've had changes in the ELF tool chain project since they started.  So it would be a regression for us to just drop what we have in the base system and switch  to these, but I would like to rationalize them and bring them all from a single source.  And the one question we have down here is obj dump, which is it's also not a required  tool and we could just say anyone who needs to use obj dump, install it from the ports.  It really is more important to have obj dump as a developer.  It's generally not a good idea for scripts and things to rely on obj dump.  But in any case, LLVM has an LLVM obj dump, which is broadly compatible except many of  the same command line arguments.  The output format is a bit different, but it does provide the use cases that most people  use obj dump for.  So disassembling object files or mixed source and disassembly LLVM obj dump works reasonably  well for that.  What are the changes to AR?  One thing we've done in AR in FreeBSD for sure that isn't in Alpha Toolchains copy is  added CAPSCOM compartmentalization support to it.  There's a few other minor changes.  It's not a lot of work, I think, to rationalize the two.  It's just that the AR that we have in the base system basically is completely usable  for us.  So there's not a huge benefit.  We're not getting any new features by switching to migrating to the other one.  It really is just to avoid having conflicting tools.  The one reason, though, that I do want to try and reduce the number of individual sources  for tools like this is that for LTO, we want to have an AR that can deal with LLVM IR in  object files.  The one thing, FreeBSD also uses the least bit, extremely small.  That's going to allow the electric possession to slow the operation of the archives and  to link the kind of files.  So there may be performance reasons as well.  LLVM has an LLVM AR that we should look at using as well.  But all that will happen as we start trying to flesh out LTO plans.  What I would say is that for small tools that you would expect to install on an end target  system, so things like strings or size, you may well want one of those installed on your  target even if you don't have a full tool chain, strings especially.  And so for those tools, I expect that we'll continue to use a small version out of ELF  tool chain even if we did start importing heavier weight LLVM versions of the tools  in general.  A few other miscellaneous tools.  We have a device tree compiler.  So this is originated sort of in open firmware.  And so device tree in general is a scheme for describing hardware and is typically of interest  in free BSD for embedded ARM devices.  And David Chisnell implemented a bespoke version of the device tree compiler.  The upstream device tree compiler, the compiler itself is GPLed.  There's a corresponding library that's dual BSD and GPL licensed that can parse and operate  on device tree files.  But the compiler itself that turns device tree source files into device tree binaries  is GPLed.  David Chisnell has an implementation that we have in the source tree and is under a build  time knob to turn it on.  It was on for some period of time.  An update to the DTS files in the free BSD tree basically made it stop working.  It didn't support some features that were added in newer DTS files.  It now is fully functional and implements all of the support that we need.  So it's a matter of testing and verification to make sure that all of the platforms we  care about continue to work if we want to switch it back on.  Warner?  Overlight.  Overlight work, yes.  Really?  Yeah.  Yeah.  So I...  It's a little bit important.  Last time I tried it, that was the impediment.  When did you try it?  Well, like January?  I believe I've imported it since then.  I'll try again.  Yeah.  Maybe by the end of the talk you can let me know.  So I have built the entire free BSD universe using the device...  The BSD license device tree compiler set as the default.  So the debugger, it's not required for self-hosting, but it is a pretty important component of  a traditional BSD-based system.  And again, we had a outdated version of new GDB across all architectures.  And in FreeBSD 11, we imported LLVM's LLDB.  So this is a project that originated initially in Apple, and it's the debugger that went  with Xcode.  Now Apple and Google are putting pretty significant amounts of effort into it.  It still has a ways to go both upstream and in FreeBSD.  And it's not completely compatible and doesn't cover all cases.  I'll mention a bit about that in a moment.  So I expected our path forward for the debugger in FreeBSD will be a combination of using  LLDB in the tree and first class GDB port support.  So one of the things that's happened since 10 years ago plus when we had the outdated  GDB is John Baldwin has put a lot of effort into updating the GDB ports and getting all  of the architecture support that existed only in FreeBSD into upstream GDB port.  And so for people who have finger memory who are really familiar with GDB, the GDB ports  on FreeBSD now work really well and support kernel debugging.  I'll mention that in a second.  LLDB is built as a set of reusable components as with the sort of general LLVM philosophy  and it builds on components that come from Clang or LLVM.  So one of the interesting things about that is that the expression parser in LLDB is actually  just Clang.  So if you say print some sort of complicated C++ expression, it's Clang turning that internally  into LLVM IR and executing it.  So it provides a very high fidelity expression parser.  And the other thing that's interesting about it is it is usable as a component from a  front end or from a scripting language.  So there's the traditional LLDB command line that we provide in FreeBSD and it uses the  LLDB library.  There's a Python.  You can use LLDB as a Python module.  So this is not invoking Python scripts from within the debugger but importing the debugger  into a Python script.  And then it can function as the support for various IDE.  So Xcode is the usual example.  There's some open source ones that can make use of it as well.  Next steps for the debugger.  We need to add AVX register support for AMD64 to make it a fully usable userland debugger.  It's a fairly straightforward project.  The debug server is a small stub that's used typically for remote debugging.  So it will run on a target system that's separate from where you're running the debugger and  executes the debuggy under that stub.  And we don't have one available for FreeBSD today.  There's a few different implementations that might work.  But one of the interesting things about LLDB is that it actually uses, on other architectures,  so not on FreeBSD and not on the original import of the Linux support but on the Linux  support is now built this way.  LLDB actually uses the debug server even for local debugging.  So the interesting thing about that is it means that you know your remote debugging case is  always going to work properly because remote debugging and local debugging is the same.  And it also simplifies the debugger internals a little bit because it doesn't have to have  this other process attached to it, to the process that's driving your UI and is heavily  threaded and things like that.  We need to add kernel debug support.  So both crash dump handling and live debugging against DevMem.  And fork following is a nice feature to have to make it a fully usable userland debugger.  We didn't have that in GDB 6.1.  It's not at all a hard requirement but it is something that improves usability quite  a bit.  We also have detrace related tools.  CTF is the compact format for debug information that's used to drive detrace.  And the way we generate it right now is using two tools that imported from OpenSolaris slash  Alumos.  So we first compile an object file with dwarf.  CTF convert runs on that dwarf information in the object file and converts it to CTF.  And then CTF merge takes the CTF from a number of object files and combines it into one.  It's a little bit awkward.  OpenBSD developers are working on a BSD licensed implementation rather than the CDDL CTF convert  and CTF merge that do it as a single step.  And so we're looking at that work that's going on.  And then longer term, it would be really nice to integrate this natively and transparently  into the tool chain.  So the Clang compiler driver and linker can just produce the output that we want directly.  Have you?  And finally, the last sort of component of the tool chain that we're going to talk about  is all of the runtime support libraries.  And there's quite a lot to go over in this very brief remaining time.  So we've got the compiler runtime that supports optimized routines and implements functionality  that the compiler doesn't want to implement in line.  So say floating point math conversions, byte swapping routines, assembly optimized versions  of things like that.  The compiler, if there's anything that's too complex for the compiler to generate directly,  it'll emit a call to one of these support routines.  In the GCC world, that's libGCC.  In the LLVM world, the project is called compiler RT.  We've sort of done an incremental migration to using the LLVM versions over time because  we have cases where we're building some architectures with GCC and some architectures with Clang,  or during the transition we had a build knob.  And so you might build sometimes with Clang and sometimes with GCC on the same architecture.  So quite a while ago, long before we even switched to Clang, we used LLVM's implementation  of the compiler built-ins for the static linking case at least, libGCC.a.  And much more recently, so only at the end of last year, we've introduced libGCC.s, the  shared version of the support routines being built from LLVM's implementation.  There's a little bit more about that in a second.  libGCC, it is the core piece of runtime support on any BSD.  It has existed in some form or another basically longer than I've been alive.  But recently we have a couple of updates to support C++11 in libGCC.  C++ requires a couple of different runtime support libraries.  There's very low level support, which provides foundation building blocks for exceptions type  info, a demangler so that C++mangle names can be converted back into something representable  to a user.  In the GCC world, this is called libSup C++.  In 2011, the previous D Foundation and that BSD Foundation worked together to obtain a  BSD licensed C++ runtime from PathScale.  That's called libCXXRT.  And subsequently there's also been an open-sourced version in the LLVM project called libCXXABI.  It didn't exist when we first did this work, but it does now.  The standard library in C++, the collection of classes and functions for containers and  operations on those containers like sorting and strings and streams and things like that.  The libSpaner C++ that we have in the tree comes along with the old GCC that we had and  it predates C++11.  So we have now libC++, well, for six years we have libC++ that is following along and  updated with all of ongoing C standard newer C++ standards.  And libC++ is C++11 code itself and can't be compiled with the old ancient GCC that  we have in the tree.  So if you build, say, MIPS with the old GCC, that build will include lib standard C++.  If you build MIPS with a out-of-tree C++11 GCC compiler, it will use libC++ as well.  And the reason we want to use libC++ on all of the architectures, even if GCC is the normal  compiler so that we can share the same C++ library in general and use, avoid ABI incompatibility.  So along with, I really for C++ is exception unwinding, but it's used by C code as well  for cases like forced P thread cleanup.  This ties into the libGCC that I mentioned earlier and it's a little bit awkward because  you want only one copy of the exception unwinder in any one process, which is why there's this  combination of libGCC.a, libGCC.s.so, and libGCC.eh.so.  And that also prevented us originally from switching the shared versions of these to  use the LLVM provided ones because there wasn't a replacement unwinder for us.  We're now using LLVM's lib unwind.  This is, it came out of Apple as well.  It was, they realized there was one piece missing from a fully licensed runtime stack.  So they released it in October 2013 and it wasn't fully usable yet, but it went through  a lot of development upstream and we've since brought it into FreeBSD and it now, as of  Octoberish, provides, it's used as the unwinder in libGCC.s.  We have a bunch of standardizers that come from LLVM.  These exist upstream.  In FreeBSD right now, we support only the, out of the box in the tree, we support only  the hydro sanitizer and undefined behavior sanitizer.  So these are used, these can be used as a replacement for tools like Valgrand to find  errors in your program at runtime.  And finally, OpenMP.  So OpenMP is a shared memory API for parallel programming and in the GCC world, it's provided,  the runtime is provided by libGOMP.  In FreeBSD, we had Clang, if you asked for OpenMP support, it would link against libGOMP,  but it actually didn't do anything, so it was relatively useless.  It now will link against LLVM's libOMP, which was initially donated by Intel and called  libIOMP at the time.  So the Clang in the base system will link against and it calls for libOMP for OpenMP  support and we provide a port called develop.openMP to provide that runtime library.  The goal is to include the OpenMP runtime in the base system for exactly this reason,  for many reasons, but this is kind of the highlight of the problem that we face.  We want the tool chain to be able to be usable for actual end user cases and this comes up  fairly regularly that when you build arbitrary software using the defaults out of the box  on FreeBSD, OpenMP support doesn't work and things don't perform as well.  And it's relatively straightforward to build and install the OpenMP runtime, but we have  all the rest of the tool chain in the base system.  There's not really a good reason for us to say we don't want to provide the runtime and  you have to do the run through, jump to some extra hoops to get your code to run quickly.  And finally here's a list of the folks who have been heavily involved in the tool chain  over time so I just want to put them up here and call them out for all the work that's  gone into this over the last seven or eight years.  Okay, thank you.  So I notice on your time frame slides there was one similarity in all of them that was  Spark 6.4 is, can you touch on that a bit?  So there's a faction of FreeBSD developers and users who wish to eject Spark 6.4 from  the tree.  I don't consider myself among that group, but my take on it is I'm perfectly happy to  leave it there as long as someone wants to maintain or keep it building.  If no one else has an interest in doing so then it clearly won't be sticking around.  It won't remain in the tree if it becomes unbuildable and no one takes an interest in  it.  One of the problems we have in FreeBSD with Spark 6.4 is we haven't updated, we don't  support any recent Spark 6.4 hardware.  So the only thing we support is hardware that has been obsolete for sort of more than a  decade.  So it's not a particularly interesting platform for us.  So are there any issues on the LLVM or toolchain side or is it primarily a FreeBSD interest?  So the Spark 6.4 support that exists in LLVM and Clang is exceedingly basic and minimal  and I don't know, Rafael, do you know if that's going to be even maintained?  There's more videos I think they use Spark 4 so they added whatever they needed to work  in but that's a different basis.  I don't think there's any interest in doing so.  I think Spark 6.4 support in upstream LLVM is really going to depend on you guys and  OpenBSD guys and NetBSD guys pursuing it if that is really something you want to do.  For us, I think we're more than happy to say for as long as Spark 6.4 support remains in  FreeBSD, it's going to require an external toolchain.  Anyone else?  Sure.  The question was how does OpenBSD handle user-defined probes in the toolchain that they're working  on and Baptiste doesn't know.  Mark?  Could you explain that?  So, when I work with not done, it's one of the things I try to figure out if it's 100%  the maximum.  So, it should be okay.  I can give you, I'm just editing them to the latest updates, I can give you a link if you  want to, just to explain what they are with them.  Yeah, very welcome.  And do you know if they're using helpful changes or different?  No, they, so the, what are you saying?  Well, I'll give you the backwards compatible with the new remote debugging protocols.  The question is, will LDB be backwards compatible with the new remote debugging protocols?  Yes, it already is.  It uses the same basic protocol as GDB does when GDB talks to its remote stub.  There are a couple of extensions that have been added for LLVM.  So, when LLVM is talking to its own debug server, it uses a few things that don't exist  in the regular GDB protocol.  But broadly speaking, it should be compatible and can be, you know, if it isn't, it can  probably be considered a bug and fix as opposed to just an architectural mismatch.  Okay, no more questions?  Thank you very much.  So, for those who didn't catch it, I don't just, well, just very much at none left, there  is a ZFS session, well, a bit of further session, we're in 11.10.  And here you're going to have my shark, I think, who's going to host BSE user development  session.  Well, both sessions.  You have time to grab a bite if you want to come back.  Otherwise, everybody has to be back in the session for 1.30.  Thanks.  Close there.  Keep the wallet.  Sure, leave it in the middle.
Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_0_311.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_312_911.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_912_1139.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_1140_1583.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_1584_1919.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_1920_3851.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_3852_11459.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_11460_11855.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_11856_12155.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_12156_13427.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_13428_14783.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_14784_16475.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_16476_17387.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_17388_17447.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_17448_18251.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_18252_18539.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_18540_18659.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_18660_19211.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_19212_19307.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_19308_19547.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_19548_19727.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_19728_19847.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_19848_19979.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_19980_20075.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_20076_22523.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_22524_22571.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_22572_23711.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_23712_24251.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_24252_29327.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_29328_29627.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_29628_30491.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_30492_41075.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_41076_41267.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_41268_41411.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_41412_42635.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_42636_42947.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_42948_43079.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_43080_44579.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_44580_51719.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_51720_53339.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_53340_56111.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_56112_56411.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_56412_57143.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_57144_57311.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_57312_58115.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_58116_61451.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_61452_61715.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_61716_62087.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_62088_62255.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_62256_62807.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_62808_65399.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_65400_65579.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_65580_66911.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_66912_67715.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_67716_68807.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_68808_70511.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_70512_75299.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_75300_76991.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_76992_78227.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_78228_78275.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_78276_79451.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_79452_79523.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_79524_79991.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_79992_80231.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_80232_80363.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_80364_80627.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_80628_84719.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_84720_85439.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_85440_86459.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_86460_86807.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_86808_88103.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_88104_88259.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_88260_90227.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_90228_92507.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_92508_93035.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_93036_93167.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_93168_93455.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_93456_93731.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_93732_95207.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_95208_99443.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_99444_99791.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_99792_100883.png Ed_Maste_-_FreeBSD_Tool_Chain_z04tZ_CDVMA_100884_101520.png
Okay, good morning. I'm Ed Mast. I'm going to talk a bit about the FreeBSD toolchain. I was thinking about doing a last minute rewrite of my presentation and giving a presentation on LibTrue instead, but I think it's a little bit too exciting, so I'm going to stick with the original topic. I'll explain very briefly what I mean by the toolchain in FreeBSD, and so it's anything that transforms source code into some sort of derived binary artifact or derived artifact of that earlier object, and this can be a progression through multiple stages, source code to object file to executable, for example. Here we have, say, a C++ source file that's transformed into a executable that you might run, or we can have C source and C++ source that are combined by a linker into a single executable or a shared object, and we might have assembly source that's processed by an assembler and combined with those other components. Then we have system libraries, the C library, and other parts of the system that are linked into that executable you might produce, and we might have tools that take the binary or shared object and process it in some way, take an alpha object and do something else with it. For example, we can produce a shared object or a binary and then split it into two components, so we might have debug information in one piece and the original binary or shared object itself in another piece so that the two pieces can be managed separately. You can ship just the binary and keep the debug info for use elsewhere, installing only if you need it, that sort of thing. Then there's also ancillary tools like the debugger, which we will use for tracing program execution, single stepping, or post-mortem analysis if you have a crash, and then tools like editors are broadly speaking part of the tool chain but not part of my presentation. I'm talking about really the tools that we ship with the base system in FreeBSD to be able to make it self-host, be able to reproduce itself, and basic ancillary tools, so the debugger is included in that. In FreeBSD, from the beginning, we were using the GNU tool chain, GCC, Binutils, GDB, and the GNU folks would release new versions on a regular basis, and we would bring those versions into FreeBSD. The light blue, cyan, upper or gray, upper bar represents release dates from GCC and GDB components, and then the purple, magenta bar below represents the point at which we imported it into FreeBSD. You can see that for the most part, within a month or maybe a little bit longer, we would import a new version of those components, tracking most upstream releases. There are a few exceptions. Sometimes it took a little while longer to import into FreeBSD, but we followed along fairly closely. The FSF kept releasing new versions, and GCC at least, we were bringing new versions in. You notice that in 2005, it was the last time we imported a GDB debugger, and there are a couple of different reasons for that, but basically, every time a new upstream release would come out, it might need changes to adapt to newer versions of FreeBSD, and over time, there were growing difficulties collaborating within the project and with upstream, and so it became more and more difficult to bring new versions of GDB into the FreeBSD-based system. There's some history there that some folks in the project can share over a period. Largely predates my direct involvement in these things, but in any case, it doesn't really matter. The point is that since more than a decade ago, we haven't imported a new version of the debugger, and we also stopped importing new versions of GCC. Why is that? Well, in the middle of 2007, the Free Software Foundation released version three of the GPL, and it had some restrictions that were unpalatable to a number of FreeBSD developers and FreeBSD users. We basically, as a project, decided, and I think this is true for other BSDs in general, decided we much prefer permissively licensed toolchain components, and in FreeBSD, we just stopped importing new versions of GDB and GCC. Some other BSDs have split it so that they have a GPLv2 and a GPLv3 toolchain in the tree. OpenBSD, I think, doesn't have a GPLv3 tree at all. NetBSD, I think, has a GPLv3 compiler for architectures that require it, but FreeBSD and OpenBSD, we just said we don't want GPLv3. A search came on for a new toolchain, and in FreeBSD, we've got quite a lot of architectures. This is kind of a high-level map of everything from relating FreeBSD architecture support, FreeBSD versions, and the specific component, the compiler, assembler, et cetera, and the source of that toolchain component for that architecture and that version. I don't expect you to actually be able to read it. My point in showing all of this is just to really demonstrate from a high-level, we have a multitude of different components coming from different sources in the FreeBSD toolchain on different architectures. One thing that is interesting that you can see from this high-level view, so if you look at the columns here, which are FreeBSD versions from left to right, FreeBSD 9.x is on the left, and earlier versions basically would follow the same sources, just different versions. You can see that in FreeBSD 9 and earlier, it's all read. We basically used a single source of toolchain components leading up to FreeBSD 9. I'll explain the colors in just a moment, but we basically have a single cohesive toolchain across all architectures. Starting in FreeBSD 10, we switched to using Clang on a number of architectures and still had GCC here on a number of them. You can see that over time, as we progressed, we get more of this sort of mix. In some places, we have this light yellow color showing up. We're in a world right now in FreeBSD where we have three main sources of components leading into the toolchain. It's not a great place to be, but as we go sort of into the future, you can see that we're looking to get back to one or two, converging back onto one or two. With a color scheme, basically, the red were tools that are in the FreeBSD tree, basically the GPL GNU tools, and are outdated at this point. The green ones are maintained tools in the FreeBSD tree, and so this is now Clang and Elf toolchain and other similar tools. The new category that's in here relatively recently is this yellow one that's tools that are coming out of the FreeBSD ports tree, and I'll explain a bit more about that in a moment. These basically document the state of the world as it exists today. Clearly, the tools weren't outdated obsolete tools at the time we imported them. They were just the GNU tools that we imported, and they were up to date, and they just haven't been updated in a long time. And specifically at the compiler, the CNC++ compiler, as I said, we had GCC across all architectures. In FreeBSD 10, in January 2014, we switched to Clang for basically our tier one architectures. And as we go across from left to right, basically FreeBSD 10, 10.0 was the first release with Clang, 10.x represents stable 10 as it exists in the tree today. FreeBSD 11 was the FreeBSD 11 release, and then 11.x is stable 11 as it exists today. What I'm calling soon here is either what's in FreeBSD head, or maybe is a work in progress branch or there's patches in review. It's either in the tree right now, or it's sort of right on the horizon, and later represents maybe what we hope to do for 12.0 or after 12.0. So we still used, even after we switched to Clang, we still used the old GCC for the tier two and lower architectures MIPS PowerPC Spark 64. And one thing that's interesting, and it's sort of, we almost sort of needed another column which is actually right now and very soon. But RISC-5 represents the beginning of an experiment for us where we rely, we have an architecture in the tree that relies on external tool chain. So this diverges from the historical BSD model of having everything in the tree and have it self-hosting. This basically was a way of allowing us to develop and work with RISC-5 without being stuck on having a tool chain available. It makes releases tricky and it makes development somewhat tricky, but it's still a much more convenient user workflow than alternatives. Warner? So what did we do for the tool chain though? How did we actually ship it? We didn't do releases until we had the tool chain in the tree. So for BSD, we started off with I386 as the primary architecture, added alpha in the 3.x time frame and over time, a couple of architectures have been added and, or lots of architectures have been added, a few architectures have been removed to get us to where we are today. So as we did with GCC early on, we've been tracking updates to Clang on an ongoing basis. So each time there was a Clang update, 3.3, 3.4, 3.5, 3.9, 4.0, and then after 4.0 comes 5.0 and then next 6.0. So the numbering scheme has changed, but as new releases came out from Clang and LLVM, we imported them into FreeBSD and we do that for many reasons. Bugfix is new C++ and C standard support. So 3.3, when we switched to it, it had full C++11 support, 3.4 added full C++14 support, and C++1Z features are kind of scattered throughout later versions so that Clang should have full C++1Z support when it actually becomes a final standard. And as we bring in new compilers, existing code might fail to build. So either the base system or ports, we may find that the source code that used to build just fine with the older compiler no longer does for a number of different reasons. And so we've started the last couple of releases of Clang. We've switched to a model where to stage an import, we're bringing Clang into a projects branch as, so in this case, it would be Clang 500-import to do the work of updating support Clang 5 in FreeBSD. And so that allows us to iterate on bringing the base system up to correcting the base system source code to work with Clang and then also it allows us to more easily test the ports tree with the updated compiler. So in the ports collection, it's about 27,000 different pieces of software in the FreeBSD ports tree and this is what we use to build packages from and individual end users can install or build that software using the ports tree for their own systems. So we have this concept called an Xbrun, which is an experimental build of the entire 27,000 pieces of software in the FreeBSD ports tree with some patch applied to the base system source tree or to the ports source repository. And then we collect the results and see what which ports newly fail as a result of the changes that we've introduced. So we've been doing this for the last couple of imports. This is the request for an Xbrun for the Clang 500-import branch. And with the Clang 500 branch, as it exists today, there's about 40 failing ports, which is actually quite small, under the 27,000. And 75 ports are skipped, which means those ports have a dependency on one of the ones that is now failing. And so we have a number of different kinds of cases as to why things fail when we update to a new compiler. And here's an example of imports out of MongoDB. So this is a fairly common idiom that exists in many third-party source trees where a pointer is compared with an integer, with a not an equality comparison, but either greater than zero or similar cases. It's just not valid. And Clang 5 refuses to build that. And so we'll need to either work around. We need to get a fix for these sorts of things into the upstream software eventually, but we have work arounds we can apply in the short term to continue moving on. And one thing I also want to mention is the Clang 500 branch, Clang 5 isn't yet released. It's the Clang version that's going to be the next release of Clang. So we are tracking the development that's happening in Clang's subversion repository and eventually Clang's Git in our repository. So last update we did in FreeBSD was about five or six days ago. So the Clang 500 branch in FreeBSD is up to date within, we keep it up to date within weeks of what's happening in Clang so that we can find issues long before the compiler gets released. And one of the things that that's good for is that we also find errors in Clang when we do this because we have a corpus of 27,000 pieces of software we're trying to build. And so in this case, there's an assertion that we tripped over in Clang and we'll generate a bug report and send it to the LLVM development community. The FreeBSD developers who work on the tool chain internally, generally speaking, also work in the upstream LLVM community. So Dmitri Andrik who does a lot of the, most of the work on bringing Clang versions in isn't a committer in Clang and LLVM. I'm also a committer in Clang and LLVM as well as FreeBSD. So we work very closely to make sure that we can say reduce a test case from a failure when we try and build some very large piece of software to something that's small that we can send upstream and have people iterate on to fix there. And this is something that provides an incredible amount of value to both FreeBSD and to the LLVM community and to the upstream software communities. So if we look at Clang usage in ports over time, the darker blue kind of bars show the number of make files in the ports tree for the ports make files. And you can see we're about 27,000 right now. And the orangey red color represents on the right axis here, represents the percentage of ports that are tagged with use underscore GCC which is forcing those ports to build with GCC. So that's one of the ways that we can work around problems with Clang. If some piece of software doesn't compile with Clang, either because it relies on GCC extensions or it relies on some sort of undefined behavior that GCC limits in a certain way, we can force it to use GCC and force it to use a specific version of GCC. So some of these, it's been in there for a very long time and some of the early uses were to force specific versions of GCC. You can see that right now we're under a percent or so of the ports that are forcing that use underscore GCC setting. And there's other ways that ports might end up using GCC. For example, any port right now that is tagged as using open MP gets compiled with GCC and there's a few other cases. But I'm just trying to illustrate the general trend here. The interesting thing is that you can see that in 2013 is when FreeBSD 10 came out and we made the switch to Clang for our tier one architectures. And there was a fairly significant increase in the use of forcing GCC. And then over time, it's been decreasing as we reinvestigate why some of those changes, why some of those ports were overridden to use GCC and as Clang implements functionality that supports the kinds of needs we had. So next I'll talk briefly about the assembler. So the assembler takes assembly source and turns it into an object file. And as with other things, we started off with a version of Binutils. 2.17.50 is the last Binutils version released under GPLv2. And it's the version that's used in lots of projects that have a desire not to move beyond GPLv2 tools. And of course, it was consistent across all of our architectures and has been becoming increasingly stale over time. Unlike Clang though, we didn't really have a compelling alternative for quite a while. The only exception here is ARM64, 64-bit ARM architecture where we actually, this was actually a slightly earlier experiment with using external tool chain. So for ARM64, when we released in 11.0, we shipped with the entry Clang being able to compile for ARM64, but we still relied on an external Binutils to be able to link and assemble. But the interesting thing is that now, as of FreeBSD, as in head right now, and as of FreeBSD, in FreeBSD 11 stable, what will become 11.1, we've switched to using Clang's integrated assembler. And I'll talk a little bit more about that in just a moment. Andy? Can you speak into the integrated assembler for everything that wasn't, what files? Right. So Andy points out we were using the integrated assembler here. And I'll explain a little bit more about that in just a moment as to why we use, where we use either the integrated assembler or a standalone assembler. But the interesting point is that very shortly, I expect, we'll be able to switch to using the integrated assembler for all of our architectures and switch to relying on the, a port's tree version of Binutils for the tier two and lower architectures. So the interesting thing about the assembler is that very little actually requires a user BinAS. It's not in POSIX. It's not sort of expected to be there by any standard. And so I asked for an Xbrun to try and build ports without user BinAS existing. AMD64 fared well. The WorldEnd kernel built just fine. And then I think there were, so there were 11 ports that failed on AMD64 when user BinAS didn't exist. And it was QMU, some other emulator, seven programming languages, so things like MIT Scheme and OCaml and some multimedia library didn't build. It's a relatively small set and then quite a bit of downstream fallout of skipped ports that depended on one of these that didn't build. And the reason that it gets, that we use either the integrated assembler or AS in different ways. We have assembly files that have a capital S and assembly files that have a lowercase s in the source tree. And for a lot of people, this was just sort of a mystery. The way it works is that capital S assembly files are processed by the C preprocessor and lowercase assembly files are sent to the assembler directly. And so this is across all architectures. We have 750 capital S assembly files on all of our architectures and for AMD64, that was at the time that the Xbrun was performed, that was all of the assembly files used on AMD64. In the time since the Xbrun, we ended up introducing a single assembly file that is passed directly to the assembler. And it can't be assembled by a Clang's integrated assembler because it uses GCC's macro support. Gas is macro support, which is not supported by the integrated assembler or the other alternative assemblers that are available. So things like YASM, none of them support the full set at least of macro use that they cast us. So we're a little bit stuck, but since it is only one assembly file that has this problem, there's a number of alternatives we can do. We can rewrite this single file to avoid use of the macro directive and expand it to expand it out in line. We need to do a lot of testing to make sure that we haven't broken it in doing that. We could also do something like assemble it with the GNU assembler and then disassemble it and take the disassembly and turn that back into what we're going to feed back into an assembler that doesn't support macro. None of these are particularly nice, but it's a very tractable problem because it is only one case. For I386, in a lot of cases, we can just rename the lowercase s files to uppercase s and let Clang IAS take care of it, but again, it needs a lot of testing to do it. But that's my goal for moving forward with those. Do you know what part is missing? Because the dot macro is implemented in MC. Yeah, I'm not sure what specifically. That's a good point. Another alternative is that we can work to extend the macro support in the integrated assembler to implement what we need. Next, we have the linker. And again, we're using a relatively ancient BFD linker on all architectures still until freeBSD11. And as with the assembler, sort of similar case, this is the reason we really needed to bring in the binutils port for ARM64 builds because we needed a linker. And now in freeBSD12 and in freeBSD11.1, when it comes out, we'll be using LLVM's LLD linker for ARM64. And then LLD works today for AMD64. My laptop is running freeBSD built about five days ago with LLD as the linker, and it generally works just fine. I386 has a few problems. We can't build kernel modules on I386 with LLD today. And 32-bit ARM has another problem linking libc. So they're fairly close for the other tier one or close to tier one architectures, but not fully there yet. And the goal is to get rid of the ancient binutils and the ancient GCC in time for freeBSD12. So the likely path is going to be to migrate to requiring external binutils ports to build architectures that don't have entry tool chain support. I just wanted to talk briefly about LLVM's LLD because it's quite an interesting case. LLD existed in the LLVM project for quite a while. It was based on the Mako-Adam model and had a port to ELF of that functionality building on top of infrastructure and libraries that were shared with the Adam Mako linker. But it was a fairly poor match. It didn't really work very well. So in 2015, Rui Yamma from Google implemented a new cough linker from Scratch in LLD. And it was basically two linkers in one. So it had one code path that handled Mako and ELF and old cough and one code path that was a brand new, completely decoupled cough linker. And shortly after, Michael Spencer took that design and implemented a new ELF linker. So we had three linkers in one. We had a standalone cough linker, a standalone ELF linker, and then all the rest. But the interesting thing is that about half a year later, LLD could self-host on freeBSD. So the ELF support progressed significantly faster when done this way rather than trying to build it on top of what was the common infrastructure that didn't really match. So in 2016, there was some really significant progress on LLD. Many people upstream worked on LLD, but I've listed a few folks here who worked specifically to improve freeBSD support in LLD or fixed generic LLD problems that were identified as a result of freeBSD trying to start using it. And some of the features that were implemented in 2016 that were important for us, probably the linker script and symbol versioning support were the two key things that didn't exist elsewhere and were very important for freeBSD's linker use. And I basically sort of found issues. I ran lots of tests trying to use, trying to build world and build kernel with LLD and submitted bugs and worked with the upstream folks to get those in and have a lot of sort of minor improvements, bug fixes across the tree. And then once we had a linker that was getting close, we get asked for an Xbrun with LLD installed as user bin LD. And in this case, it was actually quite surprising, only 270 ports failed when trying to use LLD and about 1,000 were skipped because they depended on one of the ones that failed, which I think for swapping out a linker with one that's only been around for a relatively short period of time is quite impressive. So we got about 94% or so of the ports tree to build with LLD. And there's one key issue that's preventing a lot of them, that's responsible for a lot of those failures. A lot of software ends up trying to access a protected symbol from a shared library and a symbol defined with a protected visibility. With some hackish patches to work around that in the ports tree, basically, it's fairly straightforward, although hackish to get individual ports to build and avoid this issue, we get to all but around 1% or 2% of the ports building with LLD. So my expectation for moving forward on this will be very similar to what we did with Clang, or with the introduction of Clang. We'll add some annotation in individual ports make files to say that they don't work with LLD or that they require binutils. And over time, as either the upstream port gets fixed to avoid this problem or as LLD gains new functionality, we'll strip those back out and allow those ports to build with the entry linker. So my expectation is that we will make that change before for BSD12. So this graph here, I think, really shows why LLD is interesting to a lot of developers. So this is the time to, the real time for linking a Clang release build. And the BFD linker that we have in base, GNU-LD, takes over 40 seconds of real time. New versions of BFD linker from ports, we're down to 3.7 seconds, so a pretty significant beat up. Gold is about a second and a half, and then LLD threaded or non-threaded is about half a second. And the difference between 3.7 seconds, 1.4, 0.6, 0.5, that's not a showstopper. But when we look at linking something even larger, so a debug build of Clang in this case, it was almost five minutes for BFD to link that. A minute and a half for the BFD linker from ports. Gold was over half a minute, and then LLD, almost 19 seconds without threads and 11.6 seconds with threads. And this is on some arbitrary desktop machine that I tested on. Which file system? Pardon me? Which file system? On UFS. But we achieved a 25% speed up over the BFD linker in the base system. There's some really interesting performance numbers in a more scientific benchmarking perspective, running the link on a memory file system. My goal here was to just kind of see for the usual free BFD developer workflow of building the base system, what kind of effects are we going to expect to see as a direct end user, as a direct developer. And it's pretty significant. And LLD also has been intended to support threaded operation from the beginning. And this is just a close up view of the gold versus LLD unthreaded versus LLD threaded performance from that last graph. So we're about two times unthreaded gold, and about three times when LLD is threaded. In this case, gold was not threaded. This was the default configuration in the ports tree. So in the ports tree, gold could achieve a speed up by turning on threads. But I expect that there's some reason that hasn't been done, either because it wasn't stable at the time, or I'm not sure, possibly some other reasons. So next steps for the linker. There's a few simple things and a few more exciting things that we can do when we have LLD available. One of them, the fairly simple ones, is to enable build ID. So this is a scheme that takes a hash of at least all of the significant components of the shared object or executable. It could be the entire thing, but it doesn't have to be. At least all of the loadable significant parts of that executable generates a unique identifier for that binary and stores it in an elf section. And the reason this is interesting is it allows us to do things like locate the original binary for core file. So the build ID gets stored into core files and the original binary can be found, located automatically if you have the core file. It can be used to tie together a binary and its debug objects. And so either to locate the debug file for a binary, and also to confirm that they're a proper match. LTO is also a much more significant improvement that's going to come out of using LLD. So LLTO is link time optimization. Its whole program optimization performed at link time. And in brief, what happens is that all of the intermediate objects that are combined into an executable or shared object at final link time are stored as LLVM IR and only converted to machine instructions when that final link happens. And this allows LLVM's optimizer to run over the entire program and either eliminate unused code or perform any of the sort of standard optimizations that LLVM can do across the entire program, not just a single object at a time. We'll add a mechanism to override the linker for individual ports, as I alluded to earlier. And once I think that as soon as that's done, we'll be able to switch AMD64 over to LLD. Jim, you had a question? Jim points out turning on L03 or L04 loses explicit B0. So I think, yeah, specific cases like that, we will definitely need to ensure that regressions aren't introduced. The LTO work specifically is a little bit more experimental in general at this point. It is possible with what's in the FreeBSD tree today to start experimenting with LTO builds, but there's a few things that are a little bit hacky and it's not really something we can do in production just yet. And then following that, we need effort on some of the slightly less popular architectures. At this point, 32-bit ARM and X86 fall into that category. They're fairly close with LLD, but not quite there. MIPS and PowerPC are also coming along, but need some significant effort. And then working through the backlog of issues in the ports tree. Basically, all of the cases where we've used an override to say use the binutils LD explicitly, we'll want to go through and take the root cause why we had to enable that and fix that in LLD or in the upstream port. We have quite a few assorted small binary utilities in FreeBSD. As with the assembler and the linker mentioned earlier in the original case, these typically came from the binutils project. And so we had binutils 2.17.50 up until 10.x. In FreeBSD 11, we switched to the ELF tool chain project as a source for these small tools with the exception of ARM64, which had the external binutils in addition. So although we really only needed the linker from binutils for ARM64, we actually ended up with two copies of NM and two copies of strings and all those sorts of tools, the internal ones provided by FreeBSD and the ones from the external port. So ELF tool chain is a project that was started by Joseph Kashi and Kai Wang, who were FreeBSD developers, and took a number of bespoke tools from FreeBSD, so things like AR and ELF dump, moved them into an external repository, and tried to develop them there with a goal of bringing other people outside of the FreeBSD project into working on a shared non-Ginu tool chain. It's unclear how successful that was. The project has attracted contributions from other than the FreeBSD community and other than the original developers, but not significantly so yet. But in any case, they started with a few tools obtained from FreeBSD and then went through and implemented new versions or additional tools in the collection. So this is a map of sort of the native, POSIX and historic native tools and their relationship to ELF tool chain. So you can see that for things like C++ Filt, NM, size, strings, and strip, we now get out of ELF tool chain instead of their historical sources. ELF tool chain provides a very rough implementation of a linker. It's not something that we're looking at using. And there's a number of tools that were introduced by GNU that ELF tool chain provides implementations of as well. So read ELF I think is a pretty common one. ELF copy is the tool for manipulating ELF objects and converting them from one format to another or stripping sections or doing that sort of thing. It's in GNU Benutils, it's called Obstump, and the ELF tool chain people called it Elftump originally because it operated only on ELF files. It has since grown to be able to read and write binary files and output SRAC and hex and other formats. So in Fribsd we install it as Obstcopy and it's command line equivalent and basically functionally equivalent to what GNU Obstcopy provides. And ELF tool chain has a number of libraries included with it as well. LibElftc and LibPE are internal libraries to support the rest of the tools, but LibElft for reading and writing ELF files and LibDwarf for manipulating dwarf debug info. Elftool chain provides a separate implementation of those. So LibMC is their planned equivalent to BFD basically. It's a machine code library. It would be used for the assembler and the disassembler and that sort of thing. So we have a collection of tools that are still bespoke in Fribsd. I would like to move these to come from the same sources as others. Right now we have AR for example or ELF dump. We have some changes in Fribsd that have happened since the ELF tool chain split off and we've had changes in the ELF tool chain project since they started. So it would be a regression for us to just drop what we have in the base system and switch to these, but I would like to rationalize them and bring them all from a single source. And the one question we have down here is obj dump, which is it's also not a required tool and we could just say anyone who needs to use obj dump, install it from the ports. It really is more important to have obj dump as a developer. It's generally not a good idea for scripts and things to rely on obj dump. But in any case, LLVM has an LLVM obj dump, which is broadly compatible except many of the same command line arguments. The output format is a bit different, but it does provide the use cases that most people use obj dump for. So disassembling object files or mixed source and disassembly LLVM obj dump works reasonably well for that. What are the changes to AR? One thing we've done in AR in FreeBSD for sure that isn't in Alpha Toolchains copy is added CAPSCOM compartmentalization support to it. There's a few other minor changes. It's not a lot of work, I think, to rationalize the two. It's just that the AR that we have in the base system basically is completely usable for us. So there's not a huge benefit. We're not getting any new features by switching to migrating to the other one. It really is just to avoid having conflicting tools. The one reason, though, that I do want to try and reduce the number of individual sources for tools like this is that for LTO, we want to have an AR that can deal with LLVM IR in object files. The one thing, FreeBSD also uses the least bit, extremely small. That's going to allow the electric possession to slow the operation of the archives and to link the kind of files. So there may be performance reasons as well. LLVM has an LLVM AR that we should look at using as well. But all that will happen as we start trying to flesh out LTO plans. What I would say is that for small tools that you would expect to install on an end target system, so things like strings or size, you may well want one of those installed on your target even if you don't have a full tool chain, strings especially. And so for those tools, I expect that we'll continue to use a small version out of ELF tool chain even if we did start importing heavier weight LLVM versions of the tools in general. A few other miscellaneous tools. We have a device tree compiler. So this is originated sort of in open firmware. And so device tree in general is a scheme for describing hardware and is typically of interest in free BSD for embedded ARM devices. And David Chisnell implemented a bespoke version of the device tree compiler. The upstream device tree compiler, the compiler itself is GPLed. There's a corresponding library that's dual BSD and GPL licensed that can parse and operate on device tree files. But the compiler itself that turns device tree source files into device tree binaries is GPLed. David Chisnell has an implementation that we have in the source tree and is under a build time knob to turn it on. It was on for some period of time. An update to the DTS files in the free BSD tree basically made it stop working. It didn't support some features that were added in newer DTS files. It now is fully functional and implements all of the support that we need. So it's a matter of testing and verification to make sure that all of the platforms we care about continue to work if we want to switch it back on. Warner? Overlight. Overlight work, yes. Really? Yeah. Yeah. So I... It's a little bit important. Last time I tried it, that was the impediment. When did you try it? Well, like January? I believe I've imported it since then. I'll try again. Yeah. Maybe by the end of the talk you can let me know. So I have built the entire free BSD universe using the device... The BSD license device tree compiler set as the default. So the debugger, it's not required for self-hosting, but it is a pretty important component of a traditional BSD-based system. And again, we had a outdated version of new GDB across all architectures. And in FreeBSD 11, we imported LLVM's LLDB. So this is a project that originated initially in Apple, and it's the debugger that went with Xcode. Now Apple and Google are putting pretty significant amounts of effort into it. It still has a ways to go both upstream and in FreeBSD. And it's not completely compatible and doesn't cover all cases. I'll mention a bit about that in a moment. So I expected our path forward for the debugger in FreeBSD will be a combination of using LLDB in the tree and first class GDB port support. So one of the things that's happened since 10 years ago plus when we had the outdated GDB is John Baldwin has put a lot of effort into updating the GDB ports and getting all of the architecture support that existed only in FreeBSD into upstream GDB port. And so for people who have finger memory who are really familiar with GDB, the GDB ports on FreeBSD now work really well and support kernel debugging. I'll mention that in a second. LLDB is built as a set of reusable components as with the sort of general LLVM philosophy and it builds on components that come from Clang or LLVM. So one of the interesting things about that is that the expression parser in LLDB is actually just Clang. So if you say print some sort of complicated C++ expression, it's Clang turning that internally into LLVM IR and executing it. So it provides a very high fidelity expression parser. And the other thing that's interesting about it is it is usable as a component from a front end or from a scripting language. So there's the traditional LLDB command line that we provide in FreeBSD and it uses the LLDB library. There's a Python. You can use LLDB as a Python module. So this is not invoking Python scripts from within the debugger but importing the debugger into a Python script. And then it can function as the support for various IDE. So Xcode is the usual example. There's some open source ones that can make use of it as well. Next steps for the debugger. We need to add AVX register support for AMD64 to make it a fully usable userland debugger. It's a fairly straightforward project. The debug server is a small stub that's used typically for remote debugging. So it will run on a target system that's separate from where you're running the debugger and executes the debuggy under that stub. And we don't have one available for FreeBSD today. There's a few different implementations that might work. But one of the interesting things about LLDB is that it actually uses, on other architectures, so not on FreeBSD and not on the original import of the Linux support but on the Linux support is now built this way. LLDB actually uses the debug server even for local debugging. So the interesting thing about that is it means that you know your remote debugging case is always going to work properly because remote debugging and local debugging is the same. And it also simplifies the debugger internals a little bit because it doesn't have to have this other process attached to it, to the process that's driving your UI and is heavily threaded and things like that. We need to add kernel debug support. So both crash dump handling and live debugging against DevMem. And fork following is a nice feature to have to make it a fully usable userland debugger. We didn't have that in GDB 6.1. It's not at all a hard requirement but it is something that improves usability quite a bit. We also have detrace related tools. CTF is the compact format for debug information that's used to drive detrace. And the way we generate it right now is using two tools that imported from OpenSolaris slash Alumos. So we first compile an object file with dwarf. CTF convert runs on that dwarf information in the object file and converts it to CTF. And then CTF merge takes the CTF from a number of object files and combines it into one. It's a little bit awkward. OpenBSD developers are working on a BSD licensed implementation rather than the CDDL CTF convert and CTF merge that do it as a single step. And so we're looking at that work that's going on. And then longer term, it would be really nice to integrate this natively and transparently into the tool chain. So the Clang compiler driver and linker can just produce the output that we want directly. Have you? And finally, the last sort of component of the tool chain that we're going to talk about is all of the runtime support libraries. And there's quite a lot to go over in this very brief remaining time. So we've got the compiler runtime that supports optimized routines and implements functionality that the compiler doesn't want to implement in line. So say floating point math conversions, byte swapping routines, assembly optimized versions of things like that. The compiler, if there's anything that's too complex for the compiler to generate directly, it'll emit a call to one of these support routines. In the GCC world, that's libGCC. In the LLVM world, the project is called compiler RT. We've sort of done an incremental migration to using the LLVM versions over time because we have cases where we're building some architectures with GCC and some architectures with Clang, or during the transition we had a build knob. And so you might build sometimes with Clang and sometimes with GCC on the same architecture. So quite a while ago, long before we even switched to Clang, we used LLVM's implementation of the compiler built-ins for the static linking case at least, libGCC.a. And much more recently, so only at the end of last year, we've introduced libGCC.s, the shared version of the support routines being built from LLVM's implementation. There's a little bit more about that in a second. libGCC, it is the core piece of runtime support on any BSD. It has existed in some form or another basically longer than I've been alive. But recently we have a couple of updates to support C++11 in libGCC. C++ requires a couple of different runtime support libraries. There's very low level support, which provides foundation building blocks for exceptions type info, a demangler so that C++mangle names can be converted back into something representable to a user. In the GCC world, this is called libSup C++. In 2011, the previous D Foundation and that BSD Foundation worked together to obtain a BSD licensed C++ runtime from PathScale. That's called libCXXRT. And subsequently there's also been an open-sourced version in the LLVM project called libCXXABI. It didn't exist when we first did this work, but it does now. The standard library in C++, the collection of classes and functions for containers and operations on those containers like sorting and strings and streams and things like that. The libSpaner C++ that we have in the tree comes along with the old GCC that we had and it predates C++11. So we have now libC++, well, for six years we have libC++ that is following along and updated with all of ongoing C standard newer C++ standards. And libC++ is C++11 code itself and can't be compiled with the old ancient GCC that we have in the tree. So if you build, say, MIPS with the old GCC, that build will include lib standard C++. If you build MIPS with a out-of-tree C++11 GCC compiler, it will use libC++ as well. And the reason we want to use libC++ on all of the architectures, even if GCC is the normal compiler so that we can share the same C++ library in general and use, avoid ABI incompatibility. So along with, I really for C++ is exception unwinding, but it's used by C code as well for cases like forced P thread cleanup. This ties into the libGCC that I mentioned earlier and it's a little bit awkward because you want only one copy of the exception unwinder in any one process, which is why there's this combination of libGCC.a, libGCC.s.so, and libGCC.eh.so. And that also prevented us originally from switching the shared versions of these to use the LLVM provided ones because there wasn't a replacement unwinder for us. We're now using LLVM's lib unwind. This is, it came out of Apple as well. It was, they realized there was one piece missing from a fully licensed runtime stack. So they released it in October 2013 and it wasn't fully usable yet, but it went through a lot of development upstream and we've since brought it into FreeBSD and it now, as of Octoberish, provides, it's used as the unwinder in libGCC.s. We have a bunch of standardizers that come from LLVM. These exist upstream. In FreeBSD right now, we support only the, out of the box in the tree, we support only the hydro sanitizer and undefined behavior sanitizer. So these are used, these can be used as a replacement for tools like Valgrand to find errors in your program at runtime. And finally, OpenMP. So OpenMP is a shared memory API for parallel programming and in the GCC world, it's provided, the runtime is provided by libGOMP. In FreeBSD, we had Clang, if you asked for OpenMP support, it would link against libGOMP, but it actually didn't do anything, so it was relatively useless. It now will link against LLVM's libOMP, which was initially donated by Intel and called libIOMP at the time. So the Clang in the base system will link against and it calls for libOMP for OpenMP support and we provide a port called develop.openMP to provide that runtime library. The goal is to include the OpenMP runtime in the base system for exactly this reason, for many reasons, but this is kind of the highlight of the problem that we face. We want the tool chain to be able to be usable for actual end user cases and this comes up fairly regularly that when you build arbitrary software using the defaults out of the box on FreeBSD, OpenMP support doesn't work and things don't perform as well. And it's relatively straightforward to build and install the OpenMP runtime, but we have all the rest of the tool chain in the base system. There's not really a good reason for us to say we don't want to provide the runtime and you have to do the run through, jump to some extra hoops to get your code to run quickly. And finally here's a list of the folks who have been heavily involved in the tool chain over time so I just want to put them up here and call them out for all the work that's gone into this over the last seven or eight years. Okay, thank you. So I notice on your time frame slides there was one similarity in all of them that was Spark 6.4 is, can you touch on that a bit? So there's a faction of FreeBSD developers and users who wish to eject Spark 6.4 from the tree. I don't consider myself among that group, but my take on it is I'm perfectly happy to leave it there as long as someone wants to maintain or keep it building. If no one else has an interest in doing so then it clearly won't be sticking around. It won't remain in the tree if it becomes unbuildable and no one takes an interest in it. One of the problems we have in FreeBSD with Spark 6.4 is we haven't updated, we don't support any recent Spark 6.4 hardware. So the only thing we support is hardware that has been obsolete for sort of more than a decade. So it's not a particularly interesting platform for us. So are there any issues on the LLVM or toolchain side or is it primarily a FreeBSD interest? So the Spark 6.4 support that exists in LLVM and Clang is exceedingly basic and minimal and I don't know, Rafael, do you know if that's going to be even maintained? There's more videos I think they use Spark 4 so they added whatever they needed to work in but that's a different basis. I don't think there's any interest in doing so. I think Spark 6.4 support in upstream LLVM is really going to depend on you guys and OpenBSD guys and NetBSD guys pursuing it if that is really something you want to do. For us, I think we're more than happy to say for as long as Spark 6.4 support remains in FreeBSD, it's going to require an external toolchain. Anyone else? Sure. The question was how does OpenBSD handle user-defined probes in the toolchain that they're working on and Baptiste doesn't know. Mark? Could you explain that? So, when I work with not done, it's one of the things I try to figure out if it's 100% the maximum. So, it should be okay. I can give you, I'm just editing them to the latest updates, I can give you a link if you want to, just to explain what they are with them. Yeah, very welcome. And do you know if they're using helpful changes or different? No, they, so the, what are you saying? Well, I'll give you the backwards compatible with the new remote debugging protocols. The question is, will LDB be backwards compatible with the new remote debugging protocols? Yes, it already is. It uses the same basic protocol as GDB does when GDB talks to its remote stub. There are a couple of extensions that have been added for LLVM. So, when LLVM is talking to its own debug server, it uses a few things that don't exist in the regular GDB protocol. But broadly speaking, it should be compatible and can be, you know, if it isn't, it can probably be considered a bug and fix as opposed to just an architectural mismatch. Okay, no more questions? Thank you very much. So, for those who didn't catch it, I don't just, well, just very much at none left, there is a ZFS session, well, a bit of further session, we're in 11.10. And here you're going to have my shark, I think, who's going to host BSE user development session. Well, both sessions. You have time to grab a bite if you want to come back. Otherwise, everybody has to be back in the session for 1.30. Thanks. Close there. Keep the wallet. Sure, leave it in the middle.
that transforms source code into some sort of combined by a linker into a single executable or have system libraries, the C library, and other so the debugger is included in that. In FreeBSD, chain, GCC, Binutils, GDB, and the GNU folks of 2007, the Free Software Foundation released restrictions that were unpalatable to a number of FreeBSD developers and FreeBSD users. We we much prefer permissively licensed toolchain think, has a GPLv3 compiler for architectures and the specific component, the compiler, tools that are in the FreeBSD tree, basically the toolchain and other similar tools. The new document the state of the world as it exists on, we've been tracking updates to Clang on an the base system source tree or to the ports of the, most of the work on bringing Clang switch to Clang for our tier one architectures. those changes, why some of those ports were support, which is not supported by the integrated the assembler, sort of similar case, this is the comes out, we'll be using LLVM's LLD linker for tests trying to use, trying to build world and This was the default configuration in the ports be used to tie together a binary and its debug intermediate objects that are combined into an in FreeBSD. As with the assembler and the linker typically came from the binutils project. And so I think is a pretty common one. ELF copy is the would be used for the assembler and the builds on components that come from Clang or about it is it is usable as a component from a provide in FreeBSD and it uses the LLDB library. used typically for remote debugging. So it will Linux support but on the Linux support is now also have detrace related tools. CTF is the developers are working on a BSD licensed the tool chain. So the Clang compiler driver and talk about is all of the runtime support architectures with GCC and some architectures been an open-sourced version in the LLVM project unwinding, but it's used by C code as well for and this comes up fairly regularly that when you