Building an LLVM-based tool
Denisov, Alex
In this talk, I want to share my experience in building an LLVM-based tool. For the last three years, I work on a tool for mutation testing. Currently, it works on Linux, macOS, and FreeBSD and the source code is compatible with any LLVM version between 3.8 and 7.0. Anything that can run in parallel - runs in parallel. I will cover the following topics: How to support multiple LLVM versions How to support different operating systems (Linux, macOS, FreeBSD) How to build against precompiled LLVM and sources at the same time How to parallelize everything How to test things Different ways to get bitcode from any project
 Okay, so hello everyone, thanks for coming and thanks for having me here. Today I want to share some experience about building a LLVM-based tools. Yeah, just a few words about myself. So my name is Alex, I work at a company called PT Scientist JMBH. You can find me on the internet if you want to, like my blog or Twitter. So yeah, this is the agenda for today's talk. So basically these are just like sets of some, somehow like random, but generally enough tips and tricks that might be helpful if you want to build something based on LLVM. So the tool I'm talking about is called MAL. I build it with some friends and colleagues like for several years, like three years, something. The source code is available on GitHub. If you want to know more about the tool itself, then I encourage you to watch my talk from Fosdam two years ago. Yeah, besides that, I'm not going to talk about the tool. So everything will be kind of general. So a few interesting things. In my opinion that the tool works on Linux, for BSD and MacOS. It may work on Windows, but just no one has access to Windows machine. And the nice part also that it can be compiled against any LLVM version between, like, yeah, in between of three, nine and seven. Yeah, and another interesting part in my opinion is I think it has quite, it provides quite nice user experience. So here I have just normal program. Yeah, it runs like some tests in this case, but then I can also do the following. Oops, sorry. Yeah, I just run my tool against that binary and it works. And it operates on the bitcode level. So I find it quite nice comparing to some other tools that I've seen. But let's get back to the slides and begin. So just to have some framework in mind, I defined LLVM-based tool as something that operates on LLVM bitcode and it does the following things. It parses or loads the bitcode files in memory, like creating LLVM modules. Then it does some analysis and then some transformation potentially. And in the end, it does some processing. So the tool can be something like a compiler or a foscaretor or like in our case, tool for mutation testing. So I think this approach is general enough. Of course, there might be examples where it doesn't fit, but I just want to have some kind of structure. So the first thing that you should consider, of course, is the build system. And I think lots of articles and documentation, like suggest to use LLVM config, it may work if you just start something fast or some prototype, but in the long run, it's not really good. And there are a few quirks, so this is how it usually looks like. So you just use a clank, then you call LLVM config, you ask for C++ flags and linker flags and so on. So the first problem, the C++ flags. So these are likely not the flags you want, but these are the flags that LLVM was compiled with. So here, the first one, just some part of it, just an example, include yours valid, you need it, obviously, but for example, this dash W error, I think it won't compile with GCC because it's some clank-specific thing, and it may also not compile with older versions of clank. So yeah, another point that it's, unless you compile LLVM yourself, you will likely get optimized versions. So if you build your tool using this C++ flags, then it will be compiled in optimized way all the time. So yeah, you should just be aware of that. Another thing that the LLVM config for libraries and linker flags, it works really good, really well, but there are some other issue. If let's say you have on your machine installed more than one version of LLVM and you have flip LLVM dilip, then eventually you may get this problem. So if you compile against, if you link against some library, specific library like LLVM support, for example, and you run, you may get into this inconsistency, blah, blah, blah. I don't really understand where it comes from. There's something goes wrong, obviously, with dynamic linking and maybe static linking as well, but the solution for that problem is to actually link against just lip LLVM. The kind of issue of LLVM config that there is not really an easy way to check whether you have the sleep LLVM installed or you don't have it. So yeah, as a solution, I do consider a CMake. If you don't know what, like, well, you probably know what CMake is, but if you don't know how it works and what it does, I encourage you to just read this documentation. It's good enough introduction. And from now on, I assume that you understand CMake. If not, please do yourself a favor, read about it, and then you can get back to slides, and yeah, it may make sense afterwards. So how do you use LLVM with CMake? Yeah, that's straightforward. So you just use the find package and you search for LLVM. When it's done, you get access to all the variables that LLVM distribution provides to you and also the libraries. So important part here, you may see in kind of red-ish color search pass, they are specified in this way. The reason, because as we tend to support many different versions of Clang and LLVM, different versions on different distributions of Linux and Mac, they have different locations of, like, different places where CMake looks for. So the recommendation is, yeah, to do it this way. Two other parts pass to LLVM that must be provided from the outside by your user, by a user of your tool. Because if you don't do it explicitly, then there is no guarantee, like, which actually version of LLVM is used, and it's getting a bit tricky. The good part that you can also do it the same with Clang, find package Clang, but it's broken on some Linux distributions, especially if you have, I think, Ubuntu, if you have 394 and 5 installed at the same time, then it doesn't work because, yeah, some prefixes, suffixes with versions, just in that case, yeah, recommendation is to use pre-compiled versions of Clang or LLVM from official website. Yeah, so once you've done that, you get access, yeah, as I said, to variables like LLVM includes yours, and you can teach your targets, like, where to look for headers, and you can link against LLVM support, for example, as you would link to against any other library in your project. Except that, yeah, again, you may face this problem, and it's really annoying. So the proper solution is to actually check if distribution of LLVM you use, if it supplies this LLVM target, LLVM library, then you link against just LLVM, otherwise you link against your libraries, like, yeah, any you want. Okay, I think we're done with CMake for now, and getting to, well, yeah, okay, we're not quite done with CMake. So how to support server versions? There are two approaches. This is the one of them. So basically, we have just a bunch of folders that provide a set of kind of glue code for different versions of LLVM. And then, yeah, the header files, they look like this. So basically, all the places that have some incompatibilities between versions, they were hidden behind some functions. And also, like, type depths used and so on, it's not much here, actually, I must admit. And you probably may face many more discrepancies between versions, but I think it's still manageable, and it's good enough approach. So in fact, in our case, we also used Orchjet, and it was changing a lot between different versions, like, a lot. And we first tried to come up with some interface to hide the changes behind. But yeah, at the end, we just realized that our use case, Orchjet is too general for us, and the maintaining this part is just harder than actually crafting our own JIT engine, which we did. And yeah, it just works so far. Yeah, I think the other approach is to use the define, like the macros, if, like, in your code base, like, sharp, if, LLVM version less than five or something. Yeah, it's still doable, but it's a bit messy, and yeah, I think just, as a personal preference, I like this approach more. Yeah, in the CMake, it looks like this then. Yeah. The real code is a bit more sophisticated, but it's just an example taken from our CMake file. Yeah, just check if there is a glue code for the version, then it picks it up. Otherwise, yeah, error. Yeah, so far, I was talking only about case where you compile your tool against precompiled version of LLVM. But it does make sense also to compile against sources for the following reasons. With precompiled version, you have, like, really fast compile time, but it's really hard to debug because you cannot dive, like, deep, deep into internals, and you only have interface, so it's hard to put print apps for sake of debugging. And there are no asserts, unless, again, unless you compile LLVM yourself. And so, for example, we didn't compile with asserts for maybe a year or something, and then we got several bug reports from a person who was adding 3BSD support. And this person used, yeah, built with assertions enabled, and we just learned that we violated many times, we violated some constraints of LLVM, and if you enable assertions, then, yeah, it crashes. But it worked in our case somehow, like, rather by accident, but still. Yeah. So, as a downside of building against source code, yeah, you have lots of time to actually build LLVM. So, the good part that you can also set up your CMake to support both things at the same time. So, either compile against precompiled version or against sources. So, the trick is simple. It's probably for GIL, but it works. And yeah, I'm glad about that. But yeah, so basically, you just check if the LLVM folder that a user provides, if it contains CMakeless file, then you just add LLVM itself as a subproject. So, now you own LLVM and know the other way around. The problem here that you don't have things like LLVM include years, and you don't know LLVM version because those files, they are not exposed, like, these variables are not exposed. Again, there are some solutions for that. Another for GIL solution. So, you can pick any random LLVM library and read its property, for example, include directories. And yeah, it works as well. So, as for the version, yeah, you can just parse the CMakeless file and use regular expressions to get the version out of it. And yeah, it just works. Yeah, and I think now we are done with CMake. So now, yeah, I want to talk about memory management. I'm not sure. Yeah. Is there any problem here in this code? Anyone knows? Okay. So, it may crash or may not. It depends on which version you use. I think three, eight, and four, it will crash, and on the recent versions, it will work, I think. So, the problem is that, yeah, or rather, let's start with the solution. You should just switch those two lines. The problem here that LLVM context actually owns the modules. So, and it stores just rough pointers, like vector or rough pointers to all the modules that were created. But in this example, yeah, the context deallocates, yeah, the LLVM context owns the modules, and in the destructor, it just calls the delete to the actual pointers. So, this code, when it executes, the LLVM context destroys, like, deallocates the modules, then happens the allocation of the vector with all the modules, and, yeah, double free. So, basically, you get a crash. For the same reason, this example, so this is, yeah, I think we had this problem before. So, assumption was that in the nested scope, you create a module, you do something with it, and it's just, like, disposed, like, deallocated after the scope ends. But we faced a problem on a machine, 32-bit machine with four gigabytes of RAM, because actually modules are not deallocated, and they are stored in the LLVM context. So, the solution here is to use some local context, then your memory is freed at the end of the scope. Another kind of annoying part was this approach. All the modules, they are loaded in this global context, and when the program terminates, you just see that, like, nothing is happening, and it's because the LLVM just frees the memory. So in that approach, this work is still done, it's still happening, but it's kind of distributed across the whole program lifecycle, so it's not, yeah, the execution time takes the same time, but it's just less annoying or frustrating. Okay, next topic is parallelization. So, the trick here to make things parallelizable is to not use LLVM passes. That's the bold statement, but I will elaborate just in a second. So I know, like, at least two classes in LLVM that are not thread safe, and that are heavily used. It's a LLVM context, and therefore, like, anything you do with modules, instructions, functions, it's also not thread safe, because it's, like, tightly coupled with a LLVM context. The same goes for the target machine, so if you use, like, or compiler, for example, or if you do any code generation, then it's not thread safe. So let's look at the example. Yeah, this tooling is just perfect for fair parallelization, because what it means is that you have, like, ten tasks, and you have ten threads, and you can just even display those tasks across threads, and you get, yeah, you gain some performance. So just as an example, let's say we have three bit code modules, and we want to load them into memory, parse, and, yeah, do some job. So if you want to do that across two threads, then each thread should have its own LLVM context. Otherwise, it's not thread safe. And what happens next? If you want to do some analysis, so in this example, we have six functions. It may make sense to split all those six functions across two threads, because you just do analysis. You do not do any modifications, and you can actually do that. Yeah, it works. So there is kind of bold line where a function escapes its thread boundaries, and it's fine. But when it comes to transformation, then you cannot do that anymore. So each function should be transformed in the context, or in the thread, or the same module of the same LLVM context. These are some numbers. So yeah, you see the steps are split, analysis, and transformation. So the transformation, at least in our case, is super fast, because, yeah, for some reason, it's super fast, because it does much less job. But analysis takes most of the time. And if we parallelize, like you can see the green color, it becomes just much faster. So that's why I'm saying that you maybe should avoid using passes, because they kind of incentivize you to put like analysis and transformation into the same thing. So yeah, if you split these steps, then you can parallelize better. But of course, you should measure everything all the time. So this is, yeah, in our example, in our case, just loading, like parsing bitcode files, loading them from disk takes much, much more time than analysis and transformation. It doesn't matter, like sequential execution parallel. So yeah, you still can use passes if that's the case for you. But yeah. Getting bitcode. Yes, I think, yeah, one of the trickiest topic, one of the kind of selling point of a LLVM that it has the bitcode that can be shared across different compilers and so on. That's still the case. But yeah, if you want to build some tooling, there is not really easy way to get this bitcode, depending on the compiler and so on. But there are some ways to do it for Clank, at least. Yeah, for other compilers languages. Yeah, please talk to developers of those languages. So normally, like compilation flow is like that. You have an object file. So you call a compiler with some flex. You have an object file. Then you call a linker and you get executable in the end. So with Clank, there is at least, yeah, the most famous flex. So if you call Clank, emit a LLVM. And instead of object file, you get a LLVM bitcode. And there is no executable. Another one, LTO. So you can call LTO, like dash FLTO, and you get, instead of the object files, well, you still get object files, but they do not contain machine code, but they rather contain a LLVM bitcode. And at the end, you have normal executable. So that works pretty well. If you have some well-defined build system, like CMake, for example, or even like Make files, if you have those intermediate files, but if you just call Clank and like 10 source files, then there is no output of bitcode or no output with object files, then it doesn't work. And the best approach that actually we use where we adopted recently the F embed bitcode. So if you call this code, then at any stage, you're either object files or executables, they do contain both machine code and LLVM bitcode. And you can actually extract it easily. So there is a tool called libEBC. I think it stands for embedded bitcode. So what it does, it just extracts the bitcode file from an executable. It works for some Linux and Mac and FreeBSD. And the original tool works, it just does extract the files and stores them locally on disk so that you can do some post-processing on them. And I made it work to make it work in memory. So everything happens in memory as I showed at the very beginning. So yeah, super straightforward. I will be up-streaming the patches soon, so the original version should get the support if we agree with maintainers. But otherwise, you can check the things out. Oh yeah, yeah, these are examples how it works. So you just call libcotill and then you get a bunch of bitcode files. Or yeah, you just call the tool. I think the last point, multi-wise support. So we are not that many people working on this project and it's really tough to maintain these things. So I can just recommend say two words, like background and ansible. Just like the way to go. So what background does is just the system that operates or that helps you manage virtual machines. It supports virtual box, VMware, Docker, I think, yeah, anything you can imagine it supports. So in that case, I have just like one file with that many lines that, yeah, then I run background up, Debian, for example, and I have virtual machine running ready with everything I need on it. So I can just start debugging right away. I can access it to it and then when I'm done, I just can destroy it to dispose the memory. And the good part of it that if you have some IDE that supports remote debugging, so as, for example, CLAN does have it, you can run this, set up this virtual machine, then connect with CLAN and you're like, this is at least, for example, how I work on Mac. I do have native experience as a Mac user, but I'm running and debugging my software on Linux. It's very useful. Yeah, the second part is ansible. So here you may notice like VM, Provision, Ansible, and then a playbook, Ubuntu YAML. So what playbooks are, it just like set of some tasks that needs to be done. This is an example of things. Yeah, it's just YAML file does something. And so for, in our case, most of the parts they are shared between all the other operating systems like FreeBSD and Linux, but it's slightly different on Mac. But the cool part about it that I can also run this, the Ansible thing locally. So for example, if I get a new Mac machine, I just use our Mac OS tasks to set up things locally. Yeah. What else? Yeah, the other benefit of this, the Ansible scripts or playbooks that you can also use it like on CI and everywhere you want. So at this point, yeah, I think that's pretty much it. So I try to come up with things that are general enough to be applicable for everyone. So this is not all the problems or issues we have solved so far. But I hope that it will help you to build better tooling around Telolvm. Thank you. Thank you. Thanks. Yeah. Thanks for your talk. I found it super useful. To kind of pile on the Telolvm config hate, I noticed you were linking against the Clang tooling. Were you, did you try to link it against any other parts of Clang, not Telolvm but Clang parts and find any issues with like linkage order? So the question is if I had any issues with the linkage order when using Telolvm config. And yes, I decided not to include it into the slides because that's a mess. But we had this problem, yeah, with different versions of Telolvm and Clang. There were like some problems with different order of things. So we ended up wrapping everything in like start group and group, linker flex. Or on Mac it was like something like dash W, like whole archive or something. So I mean it's just like super hard to support because you have if macOS then you have one bunch of scripts, else if Linux, other and so on so forth. That was CMake, no problems whatsoever, just works. Yeah, it's awesome to see how wide support you have across all these different OSes and versions as well. It's really cool stuff. Thank you. Yeah. As far as I know, org.jit was supposed to improve on MCJIT but in the end you still end up using your own JIT. So what was wrong with it? Yeah, so the question is that why don't we use org.jit or MCJIT and come up with our kind of own solution. So one of the problems that we had with MCJIT initially that it requires you to give up ownership of all the modules. So for all the modules it's like unique pointer and you need to give up it so you don't own them anymore. And in our case we wanted to own it early. So we cannot use MCJIT. Before we switched to org.jit because it didn't have this problem initially. But as the time goes, there was this problem as well. And the API changed drastically. So it was just too much overhead to actually support our own kind of interface to the org.jit. So and we don't need most of the features that it provides like on request completion and yeah, we just lowered program at once and that's it. So thank you. Enjoy the day and see you around.
 Okay, so hello everyone, thanks for coming and thanks for having me here.  Today I want to share some experience about building a LLVM-based tools.  Yeah, just a few words about myself.  So my name is Alex, I work at a company called PT Scientist JMBH.  You can find me on the internet if you want to, like my blog or Twitter.  So yeah, this is the agenda for today's talk.  So basically these are just like sets of some, somehow like random, but generally enough  tips and tricks that might be helpful if you want to build something based on LLVM.  So the tool I'm talking about is called MAL.  I build it with some friends and colleagues like for several years, like three years,  something.  The source code is available on GitHub.  If you want to know more about the tool itself, then I encourage you to watch my talk from  Fosdam two years ago.  Yeah, besides that, I'm not going to talk about the tool.  So everything will be kind of general.  So a few interesting things.  In my opinion that the tool works on Linux, for BSD and MacOS.  It may work on Windows, but just no one has access to Windows machine.  And the nice part also that it can be compiled against any LLVM version between, like, yeah,  in between of three, nine and seven.  Yeah, and another interesting part in my opinion is I think it has quite, it provides  quite nice user experience.  So here I have just normal program.  Yeah, it runs like some tests in this case, but then I can also do the following.  Oops, sorry.  Yeah, I just run my tool against that binary and it works.  And it operates on the bitcode level.  So I find it quite nice comparing to some other tools that I've seen.  But let's get back to the slides and begin.  So just to have some framework in mind, I defined LLVM-based tool as something that  operates on LLVM bitcode and it does the following things.  It parses or loads the bitcode files in memory, like creating LLVM modules.  Then it does some analysis and then some transformation potentially.  And in the end, it does some processing.  So the tool can be something like a compiler or a foscaretor or like in our case, tool  for mutation testing.  So I think this approach is general enough.  Of course, there might be examples where it doesn't fit, but I just want to have some  kind of structure.  So the first thing that you should consider, of course, is the build system.  And I think lots of articles and documentation, like suggest to use LLVM config, it may work  if you just start something fast or some prototype, but in the long run, it's not really good.  And there are a few quirks, so this is how it usually looks like.  So you just use a clank, then you call LLVM config, you ask for C++ flags and linker flags  and so on.  So the first problem, the C++ flags.  So these are likely not the flags you want, but these are the flags that LLVM was compiled  with.  So here, the first one, just some part of it, just an example, include yours valid, you  need it, obviously, but for example, this dash W error, I think it won't compile with  GCC because it's some clank-specific thing, and it may also not compile with older versions  of clank.  So yeah, another point that it's, unless you compile LLVM yourself, you will likely  get optimized versions.  So if you build your tool using this C++ flags, then it will be compiled in optimized way  all the time.  So yeah, you should just be aware of that.  Another thing that the LLVM config for libraries and linker flags, it works really good, really  well, but there are some other issue.  If let's say you have on your machine installed more than one version of LLVM and you have  flip LLVM dilip, then eventually you may get this problem.  So if you compile against, if you link against some library, specific library like LLVM support,  for example, and you run, you may get into this inconsistency, blah, blah, blah.  I don't really understand where it comes from.  There's something goes wrong, obviously, with dynamic linking and maybe static linking as  well, but the solution for that problem is to actually link against just lip LLVM.  The kind of issue of LLVM config that there is not really an easy way to check whether  you have the sleep LLVM installed or you don't have it.  So yeah, as a solution, I do consider a CMake.  If you don't know what, like, well, you probably know what CMake is, but if you don't know  how it works and what it does, I encourage you to just read this documentation.  It's good enough introduction.  And from now on, I assume that you understand CMake.  If not, please do yourself a favor, read about it, and then you can get back to slides,  and yeah, it may make sense afterwards.  So how do you use LLVM with CMake?  Yeah, that's straightforward.  So you just use the find package and you search for LLVM.  When it's done, you get access to all the variables that LLVM distribution provides  to you and also the libraries.  So important part here, you may see in kind of red-ish color search pass, they are specified  in this way.  The reason, because as we tend to support many different versions of Clang and LLVM,  different versions on different distributions of Linux and Mac, they have different locations  of, like, different places where CMake looks for.  So the recommendation is, yeah, to do it this way.  Two other parts pass to LLVM that must be provided from the outside by your user, by  a user of your tool.  Because if you don't do it explicitly, then there is no guarantee, like, which actually  version of LLVM is used, and it's getting a bit tricky.  The good part that you can also do it the same with Clang, find package Clang, but it's  broken on some Linux distributions, especially if you have, I think, Ubuntu, if you have  394 and 5 installed at the same time, then it doesn't work because, yeah, some prefixes,  suffixes with versions, just in that case, yeah, recommendation is to use pre-compiled  versions of Clang or LLVM from official website.  Yeah, so once you've done that, you get access, yeah, as I said, to variables like LLVM  includes yours, and you can teach your targets, like, where to look for headers, and you can  link against LLVM support, for example, as you would link to against any other library  in your project.  Except that, yeah, again, you may face this problem, and it's really annoying.  So the proper solution is to actually check if distribution of LLVM you use, if it supplies  this LLVM target, LLVM library, then you link against just LLVM, otherwise you link  against your libraries, like, yeah, any you want.  Okay, I think we're done with CMake for now, and getting to, well, yeah, okay, we're not  quite done with CMake.  So how to support server versions?  There are two approaches.  This is the one of them.  So basically, we have just a bunch of folders that provide a set of kind of glue code for  different versions of LLVM.  And then, yeah, the header files, they look like this.  So basically, all the places that have some incompatibilities between versions, they were  hidden behind some functions.  And also, like, type depths used and so on, it's not much here, actually, I must admit.  And you probably may face many more discrepancies between versions, but I think it's still manageable,  and it's good enough approach.  So in fact, in our case, we also used Orchjet, and it was changing a lot between different  versions, like, a lot.  And we first tried to come up with some interface to hide the changes behind.  But yeah, at the end, we just realized that our use case, Orchjet is too general for us,  and the maintaining this part is just harder than actually crafting our own JIT engine,  which we did.  And yeah, it just works so far.  Yeah, I think the other approach is to use the define, like the macros, if, like, in  your code base, like, sharp, if, LLVM version less than five or something.  Yeah, it's still doable, but it's a bit messy, and yeah, I think just, as a personal preference,  I like this approach more.  Yeah, in the CMake, it looks like this then.  Yeah.  The real code is a bit more sophisticated, but it's just an example taken from our CMake  file.  Yeah, just check if there is a glue code for the version, then it picks it up.  Otherwise, yeah, error.  Yeah, so far, I was talking only about case where you compile your tool against precompiled  version of LLVM.  But it does make sense also to compile against sources for the following reasons.  With precompiled version, you have, like, really fast compile time, but it's really  hard to debug because you cannot dive, like, deep, deep into internals, and you only have  interface, so it's hard to put print apps for sake of debugging.  And there are no asserts, unless, again, unless you compile LLVM yourself.  And so, for example, we didn't compile with asserts for maybe a year or something, and  then we got several bug reports from a person who was adding 3BSD support.  And this person used, yeah, built with assertions enabled, and we just learned that we violated  many times, we violated some constraints of LLVM, and if you enable assertions, then,  yeah, it crashes.  But it worked in our case somehow, like, rather by accident, but still.  Yeah.  So, as a downside of building against source code, yeah, you have lots of time to actually  build LLVM.  So, the good part that you can also set up your CMake to support both things at the same  time.  So, either compile against precompiled version or against sources.  So, the trick is simple.  It's probably for GIL, but it works.  And yeah, I'm glad about that.  But yeah, so basically, you just check if the LLVM folder that a user provides, if it  contains CMakeless file, then you just add LLVM itself as a subproject.  So, now you own LLVM and know the other way around.  The problem here that you don't have things like LLVM include years, and you don't know  LLVM version because those files, they are not exposed, like, these variables are not  exposed.  Again, there are some solutions for that.  Another for GIL solution.  So, you can pick any random LLVM library and read its property, for example, include directories.  And yeah, it works as well.  So, as for the version, yeah, you can just parse the CMakeless file and use regular expressions  to get the version out of it.  And yeah, it just works.  Yeah, and I think now we are done with CMake.  So now, yeah, I want to talk about memory management.  I'm not sure.  Yeah.  Is there any problem here in this code?  Anyone knows?  Okay.  So, it may crash or may not.  It depends on which version you use.  I think three, eight, and four, it will crash, and on the recent versions, it will work,  I think.  So, the problem is that, yeah, or rather, let's start with the solution.  You should just switch those two lines.  The problem here that LLVM context actually owns the modules.  So, and it stores just rough pointers, like vector or rough pointers to all the modules  that were created.  But in this example, yeah, the context deallocates, yeah, the LLVM context owns the modules,  and in the destructor, it just calls the delete to the actual pointers.  So, this code, when it executes, the LLVM context destroys, like, deallocates the modules,  then happens the allocation of the vector with all the modules, and, yeah, double free.  So, basically, you get a crash.  For the same reason, this example, so this is, yeah, I think we had this problem before.  So, assumption was that in the nested scope, you create a module, you do something with  it, and it's just, like, disposed, like, deallocated after the scope ends.  But we faced a problem on a machine, 32-bit machine with four gigabytes of RAM, because  actually modules are not deallocated, and they are stored in the LLVM context.  So, the solution here is to use some local context, then your memory is freed at the  end of the scope.  Another kind of annoying part was this approach.  All the modules, they are loaded in this global context, and when the program terminates,  you just see that, like, nothing is happening, and it's because the LLVM just frees the memory.  So in that approach, this work is still done, it's still happening, but it's kind of distributed  across the whole program lifecycle, so it's not, yeah, the execution time takes the same  time, but it's just less annoying or frustrating.  Okay, next topic is parallelization.  So, the trick here to make things parallelizable is to not use LLVM passes.  That's the bold statement, but I will elaborate just in a second.  So I know, like, at least two classes in LLVM that are not thread safe, and that are heavily  used.  It's a LLVM context, and therefore, like, anything you do with modules, instructions,  functions, it's also not thread safe, because it's, like, tightly coupled with a LLVM context.  The same goes for the target machine, so if you use, like, or compiler, for example,  or if you do any code generation, then it's not thread safe.  So let's look at the example.  Yeah, this tooling is just perfect for fair parallelization, because what it means is  that you have, like, ten tasks, and you have ten threads, and you can just even display  those tasks across threads, and you get, yeah, you gain some performance.  So just as an example, let's say we have three bit code modules, and we want to load them  into memory, parse, and, yeah, do some job.  So if you want to do that across two threads, then each thread should have its own LLVM  context.  Otherwise, it's not thread safe.  And what happens next?  If you want to do some analysis, so in this example, we have six functions.  It may make sense to split all those six functions across two threads, because you just do analysis.  You do not do any modifications, and you can actually do that.  Yeah, it works.  So there is kind of bold line where a function escapes its thread boundaries, and it's fine.  But when it comes to transformation, then you cannot do that anymore.  So each function should be transformed in the context, or in the thread, or the same  module of the same LLVM context.  These are some numbers.  So yeah, you see the steps are split, analysis, and transformation.  So the transformation, at least in our case, is super fast, because, yeah, for some reason,  it's super fast, because it does much less job.  But analysis takes most of the time.  And if we parallelize, like you can see the green color, it becomes just much faster.  So that's why I'm saying that you maybe should avoid using passes, because they kind of incentivize  you to put like analysis and transformation into the same thing.  So yeah, if you split these steps, then you can parallelize better.  But of course, you should measure everything all the time.  So this is, yeah, in our example, in our case, just loading, like parsing bitcode files,  loading them from disk takes much, much more time than analysis and transformation.  It doesn't matter, like sequential execution parallel.  So yeah, you still can use passes if that's the case for you.  But yeah.  Getting bitcode.  Yes, I think, yeah, one of the trickiest topic, one of the kind of selling point of a LLVM  that it has the bitcode that can be shared across different compilers and so on.  That's still the case.  But yeah, if you want to build some tooling, there is not really easy way to get this bitcode,  depending on the compiler and so on.  But there are some ways to do it for Clank, at least.  Yeah, for other compilers languages.  Yeah, please talk to developers of those languages.  So normally, like compilation flow is like that.  You have an object file.  So you call a compiler with some flex.  You have an object file.  Then you call a linker and you get executable in the end.  So with Clank, there is at least, yeah, the most famous flex.  So if you call Clank, emit a LLVM.  And instead of object file, you get a LLVM bitcode.  And there is no executable.  Another one, LTO.  So you can call LTO, like dash FLTO, and you get, instead of the object files, well, you  still get object files, but they do not contain machine code, but they rather contain a LLVM  bitcode.  And at the end, you have normal executable.  So that works pretty well.  If you have some well-defined build system, like CMake, for example, or even like Make  files, if you have those intermediate files, but if you just call Clank and like 10 source  files, then there is no output of bitcode or no output with object files, then it doesn't  work.  And the best approach that actually we use where we adopted recently the F embed bitcode.  So if you call this code, then at any stage, you're either object files or executables,  they do contain both machine code and LLVM bitcode.  And you can actually extract it easily.  So there is a tool called libEBC.  I think it stands for embedded bitcode.  So what it does, it just extracts the bitcode file from an executable.  It works for some Linux and Mac and FreeBSD.  And the original tool works, it just does extract the files and stores them locally on  disk so that you can do some post-processing on them.  And I made it work to make it work in memory.  So everything happens in memory as I showed at the very beginning.  So yeah, super straightforward.  I will be up-streaming the patches soon, so the original version should get the support  if we agree with maintainers.  But otherwise, you can check the things out.  Oh yeah, yeah, these are examples how it works.  So you just call libcotill and then you get a bunch of bitcode files.  Or yeah, you just call the tool.  I think the last point, multi-wise support.  So we are not that many people working on this project and it's really tough to maintain  these things.  So I can just recommend say two words, like background and ansible.  Just like the way to go.  So what background does is just the system that operates or that helps you manage virtual  machines.  It supports virtual box, VMware, Docker, I think, yeah, anything you can imagine it supports.  So in that case, I have just like one file with that many lines that, yeah, then I run  background up, Debian, for example, and I have virtual machine running ready with everything  I need on it.  So I can just start debugging right away.  I can access it to it and then when I'm done, I just can destroy it to dispose the memory.  And the good part of it that if you have some IDE that supports remote debugging, so as,  for example, CLAN does have it, you can run this, set up this virtual machine, then connect  with CLAN and you're like, this is at least, for example, how I work on Mac.  I do have native experience as a Mac user, but I'm running and debugging my software  on Linux.  It's very useful.  Yeah, the second part is ansible.  So here you may notice like VM, Provision, Ansible, and then a playbook, Ubuntu YAML.  So what playbooks are, it just like set of some tasks that needs to be done.  This is an example of things.  Yeah, it's just YAML file does something.  And so for, in our case, most of the parts they are shared between all the other operating  systems like FreeBSD and Linux, but it's slightly different on Mac.  But the cool part about it that I can also run this, the Ansible thing locally.  So for example, if I get a new Mac machine, I just use our Mac OS tasks to set up things  locally.  Yeah.  What else?  Yeah, the other benefit of this, the Ansible scripts or playbooks that you can also use  it like on CI and everywhere you want.  So at this point, yeah, I think that's pretty much it.  So I try to come up with things that are general enough to be applicable for everyone.  So this is not all the problems or issues we have solved so far.  But I hope that it will help you to build better tooling around Telolvm.  Thank you.  Thank you.  Thanks.  Yeah.  Thanks for your talk.  I found it super useful.  To kind of pile on the Telolvm config hate, I noticed you were linking against the Clang  tooling.  Were you, did you try to link it against any other parts of Clang, not Telolvm but Clang  parts and find any issues with like linkage order?  So the question is if I had any issues with the linkage order when using Telolvm config.  And yes, I decided not to include it into the slides because that's a mess.  But we had this problem, yeah, with different versions of Telolvm and Clang.  There were like some problems with different order of things.  So we ended up wrapping everything in like start group and group, linker flex.  Or on Mac it was like something like dash W, like whole archive or something.  So I mean it's just like super hard to support because you have if macOS then you have one  bunch of scripts, else if Linux, other and so on so forth.  That was CMake, no problems whatsoever, just works.  Yeah, it's awesome to see how wide support you have across all these different OSes  and versions as well.  It's really cool stuff.  Thank you.  Yeah.  As far as I know, org.jit was supposed to improve on MCJIT but in the end you still end up using  your own JIT.  So what was wrong with it?  Yeah, so the question is that why don't we use org.jit or MCJIT and come up with our  kind of own solution.  So one of the problems that we had with MCJIT initially that it requires you to give up  ownership of all the modules.  So for all the modules it's like unique pointer and you need to give up it so you don't own  them anymore.  And in our case we wanted to own it early.  So we cannot use MCJIT.  Before we switched to org.jit because it didn't have this problem initially.  But as the time goes, there was this problem as well.  And the API changed drastically.  So it was just too much overhead to actually support our own kind of interface to the org.jit.  So and we don't need most of the features that it provides like on request completion  and yeah, we just lowered program at once and that's it.  So thank you.  Enjoy the day and see you around.
Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_0_143.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_144_1043.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_1044_2399.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_2400_2855.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_2856_2951.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_2952_3071.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_3072_3407.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_3408_3503.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_3504_4487.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_4488_4607.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_4608_5267.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_5268_11039.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_11040_12791.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_12792_13439.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_13440_13571.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_13572_14591.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_14592_17291.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_17292_17459.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_17460_19271.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_19272_22799.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_22800_22991.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_22992_23051.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_23052_23423.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_23424_27263.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_27264_27779.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_27780_27911.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_27912_28295.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_28296_30431.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_30432_31751.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_31752_34955.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_34956_36095.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_36096_37127.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_37128_37631.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_37632_40835.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_40836_41087.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_41088_41327.png Building_an_LLVM-based_tool_Lessons_Learned_n87IPlFc1BI_41328_41441.png
Okay, so hello everyone, thanks for coming and thanks for having me here. Today I want to share some experience about building a LLVM-based tools. Yeah, just a few words about myself. So my name is Alex, I work at a company called PT Scientist JMBH. You can find me on the internet if you want to, like my blog or Twitter. So yeah, this is the agenda for today's talk. So basically these are just like sets of some, somehow like random, but generally enough tips and tricks that might be helpful if you want to build something based on LLVM. So the tool I'm talking about is called MAL. I build it with some friends and colleagues like for several years, like three years, something. The source code is available on GitHub. If you want to know more about the tool itself, then I encourage you to watch my talk from Fosdam two years ago. Yeah, besides that, I'm not going to talk about the tool. So everything will be kind of general. So a few interesting things. In my opinion that the tool works on Linux, for BSD and MacOS. It may work on Windows, but just no one has access to Windows machine. And the nice part also that it can be compiled against any LLVM version between, like, yeah, in between of three, nine and seven. Yeah, and another interesting part in my opinion is I think it has quite, it provides quite nice user experience. So here I have just normal program. Yeah, it runs like some tests in this case, but then I can also do the following. Oops, sorry. Yeah, I just run my tool against that binary and it works. And it operates on the bitcode level. So I find it quite nice comparing to some other tools that I've seen. But let's get back to the slides and begin. So just to have some framework in mind, I defined LLVM-based tool as something that operates on LLVM bitcode and it does the following things. It parses or loads the bitcode files in memory, like creating LLVM modules. Then it does some analysis and then some transformation potentially. And in the end, it does some processing. So the tool can be something like a compiler or a foscaretor or like in our case, tool for mutation testing. So I think this approach is general enough. Of course, there might be examples where it doesn't fit, but I just want to have some kind of structure. So the first thing that you should consider, of course, is the build system. And I think lots of articles and documentation, like suggest to use LLVM config, it may work if you just start something fast or some prototype, but in the long run, it's not really good. And there are a few quirks, so this is how it usually looks like. So you just use a clank, then you call LLVM config, you ask for C++ flags and linker flags and so on. So the first problem, the C++ flags. So these are likely not the flags you want, but these are the flags that LLVM was compiled with. So here, the first one, just some part of it, just an example, include yours valid, you need it, obviously, but for example, this dash W error, I think it won't compile with GCC because it's some clank-specific thing, and it may also not compile with older versions of clank. So yeah, another point that it's, unless you compile LLVM yourself, you will likely get optimized versions. So if you build your tool using this C++ flags, then it will be compiled in optimized way all the time. So yeah, you should just be aware of that. Another thing that the LLVM config for libraries and linker flags, it works really good, really well, but there are some other issue. If let's say you have on your machine installed more than one version of LLVM and you have flip LLVM dilip, then eventually you may get this problem. So if you compile against, if you link against some library, specific library like LLVM support, for example, and you run, you may get into this inconsistency, blah, blah, blah. I don't really understand where it comes from. There's something goes wrong, obviously, with dynamic linking and maybe static linking as well, but the solution for that problem is to actually link against just lip LLVM. The kind of issue of LLVM config that there is not really an easy way to check whether you have the sleep LLVM installed or you don't have it. So yeah, as a solution, I do consider a CMake. If you don't know what, like, well, you probably know what CMake is, but if you don't know how it works and what it does, I encourage you to just read this documentation. It's good enough introduction. And from now on, I assume that you understand CMake. If not, please do yourself a favor, read about it, and then you can get back to slides, and yeah, it may make sense afterwards. So how do you use LLVM with CMake? Yeah, that's straightforward. So you just use the find package and you search for LLVM. When it's done, you get access to all the variables that LLVM distribution provides to you and also the libraries. So important part here, you may see in kind of red-ish color search pass, they are specified in this way. The reason, because as we tend to support many different versions of Clang and LLVM, different versions on different distributions of Linux and Mac, they have different locations of, like, different places where CMake looks for. So the recommendation is, yeah, to do it this way. Two other parts pass to LLVM that must be provided from the outside by your user, by a user of your tool. Because if you don't do it explicitly, then there is no guarantee, like, which actually version of LLVM is used, and it's getting a bit tricky. The good part that you can also do it the same with Clang, find package Clang, but it's broken on some Linux distributions, especially if you have, I think, Ubuntu, if you have 394 and 5 installed at the same time, then it doesn't work because, yeah, some prefixes, suffixes with versions, just in that case, yeah, recommendation is to use pre-compiled versions of Clang or LLVM from official website. Yeah, so once you've done that, you get access, yeah, as I said, to variables like LLVM includes yours, and you can teach your targets, like, where to look for headers, and you can link against LLVM support, for example, as you would link to against any other library in your project. Except that, yeah, again, you may face this problem, and it's really annoying. So the proper solution is to actually check if distribution of LLVM you use, if it supplies this LLVM target, LLVM library, then you link against just LLVM, otherwise you link against your libraries, like, yeah, any you want. Okay, I think we're done with CMake for now, and getting to, well, yeah, okay, we're not quite done with CMake. So how to support server versions? There are two approaches. This is the one of them. So basically, we have just a bunch of folders that provide a set of kind of glue code for different versions of LLVM. And then, yeah, the header files, they look like this. So basically, all the places that have some incompatibilities between versions, they were hidden behind some functions. And also, like, type depths used and so on, it's not much here, actually, I must admit. And you probably may face many more discrepancies between versions, but I think it's still manageable, and it's good enough approach. So in fact, in our case, we also used Orchjet, and it was changing a lot between different versions, like, a lot. And we first tried to come up with some interface to hide the changes behind. But yeah, at the end, we just realized that our use case, Orchjet is too general for us, and the maintaining this part is just harder than actually crafting our own JIT engine, which we did. And yeah, it just works so far. Yeah, I think the other approach is to use the define, like the macros, if, like, in your code base, like, sharp, if, LLVM version less than five or something. Yeah, it's still doable, but it's a bit messy, and yeah, I think just, as a personal preference, I like this approach more. Yeah, in the CMake, it looks like this then. Yeah. The real code is a bit more sophisticated, but it's just an example taken from our CMake file. Yeah, just check if there is a glue code for the version, then it picks it up. Otherwise, yeah, error. Yeah, so far, I was talking only about case where you compile your tool against precompiled version of LLVM. But it does make sense also to compile against sources for the following reasons. With precompiled version, you have, like, really fast compile time, but it's really hard to debug because you cannot dive, like, deep, deep into internals, and you only have interface, so it's hard to put print apps for sake of debugging. And there are no asserts, unless, again, unless you compile LLVM yourself. And so, for example, we didn't compile with asserts for maybe a year or something, and then we got several bug reports from a person who was adding 3BSD support. And this person used, yeah, built with assertions enabled, and we just learned that we violated many times, we violated some constraints of LLVM, and if you enable assertions, then, yeah, it crashes. But it worked in our case somehow, like, rather by accident, but still. Yeah. So, as a downside of building against source code, yeah, you have lots of time to actually build LLVM. So, the good part that you can also set up your CMake to support both things at the same time. So, either compile against precompiled version or against sources. So, the trick is simple. It's probably for GIL, but it works. And yeah, I'm glad about that. But yeah, so basically, you just check if the LLVM folder that a user provides, if it contains CMakeless file, then you just add LLVM itself as a subproject. So, now you own LLVM and know the other way around. The problem here that you don't have things like LLVM include years, and you don't know LLVM version because those files, they are not exposed, like, these variables are not exposed. Again, there are some solutions for that. Another for GIL solution. So, you can pick any random LLVM library and read its property, for example, include directories. And yeah, it works as well. So, as for the version, yeah, you can just parse the CMakeless file and use regular expressions to get the version out of it. And yeah, it just works. Yeah, and I think now we are done with CMake. So now, yeah, I want to talk about memory management. I'm not sure. Yeah. Is there any problem here in this code? Anyone knows? Okay. So, it may crash or may not. It depends on which version you use. I think three, eight, and four, it will crash, and on the recent versions, it will work, I think. So, the problem is that, yeah, or rather, let's start with the solution. You should just switch those two lines. The problem here that LLVM context actually owns the modules. So, and it stores just rough pointers, like vector or rough pointers to all the modules that were created. But in this example, yeah, the context deallocates, yeah, the LLVM context owns the modules, and in the destructor, it just calls the delete to the actual pointers. So, this code, when it executes, the LLVM context destroys, like, deallocates the modules, then happens the allocation of the vector with all the modules, and, yeah, double free. So, basically, you get a crash. For the same reason, this example, so this is, yeah, I think we had this problem before. So, assumption was that in the nested scope, you create a module, you do something with it, and it's just, like, disposed, like, deallocated after the scope ends. But we faced a problem on a machine, 32-bit machine with four gigabytes of RAM, because actually modules are not deallocated, and they are stored in the LLVM context. So, the solution here is to use some local context, then your memory is freed at the end of the scope. Another kind of annoying part was this approach. All the modules, they are loaded in this global context, and when the program terminates, you just see that, like, nothing is happening, and it's because the LLVM just frees the memory. So in that approach, this work is still done, it's still happening, but it's kind of distributed across the whole program lifecycle, so it's not, yeah, the execution time takes the same time, but it's just less annoying or frustrating. Okay, next topic is parallelization. So, the trick here to make things parallelizable is to not use LLVM passes. That's the bold statement, but I will elaborate just in a second. So I know, like, at least two classes in LLVM that are not thread safe, and that are heavily used. It's a LLVM context, and therefore, like, anything you do with modules, instructions, functions, it's also not thread safe, because it's, like, tightly coupled with a LLVM context. The same goes for the target machine, so if you use, like, or compiler, for example, or if you do any code generation, then it's not thread safe. So let's look at the example. Yeah, this tooling is just perfect for fair parallelization, because what it means is that you have, like, ten tasks, and you have ten threads, and you can just even display those tasks across threads, and you get, yeah, you gain some performance. So just as an example, let's say we have three bit code modules, and we want to load them into memory, parse, and, yeah, do some job. So if you want to do that across two threads, then each thread should have its own LLVM context. Otherwise, it's not thread safe. And what happens next? If you want to do some analysis, so in this example, we have six functions. It may make sense to split all those six functions across two threads, because you just do analysis. You do not do any modifications, and you can actually do that. Yeah, it works. So there is kind of bold line where a function escapes its thread boundaries, and it's fine. But when it comes to transformation, then you cannot do that anymore. So each function should be transformed in the context, or in the thread, or the same module of the same LLVM context. These are some numbers. So yeah, you see the steps are split, analysis, and transformation. So the transformation, at least in our case, is super fast, because, yeah, for some reason, it's super fast, because it does much less job. But analysis takes most of the time. And if we parallelize, like you can see the green color, it becomes just much faster. So that's why I'm saying that you maybe should avoid using passes, because they kind of incentivize you to put like analysis and transformation into the same thing. So yeah, if you split these steps, then you can parallelize better. But of course, you should measure everything all the time. So this is, yeah, in our example, in our case, just loading, like parsing bitcode files, loading them from disk takes much, much more time than analysis and transformation. It doesn't matter, like sequential execution parallel. So yeah, you still can use passes if that's the case for you. But yeah. Getting bitcode. Yes, I think, yeah, one of the trickiest topic, one of the kind of selling point of a LLVM that it has the bitcode that can be shared across different compilers and so on. That's still the case. But yeah, if you want to build some tooling, there is not really easy way to get this bitcode, depending on the compiler and so on. But there are some ways to do it for Clank, at least. Yeah, for other compilers languages. Yeah, please talk to developers of those languages. So normally, like compilation flow is like that. You have an object file. So you call a compiler with some flex. You have an object file. Then you call a linker and you get executable in the end. So with Clank, there is at least, yeah, the most famous flex. So if you call Clank, emit a LLVM. And instead of object file, you get a LLVM bitcode. And there is no executable. Another one, LTO. So you can call LTO, like dash FLTO, and you get, instead of the object files, well, you still get object files, but they do not contain machine code, but they rather contain a LLVM bitcode. And at the end, you have normal executable. So that works pretty well. If you have some well-defined build system, like CMake, for example, or even like Make files, if you have those intermediate files, but if you just call Clank and like 10 source files, then there is no output of bitcode or no output with object files, then it doesn't work. And the best approach that actually we use where we adopted recently the F embed bitcode. So if you call this code, then at any stage, you're either object files or executables, they do contain both machine code and LLVM bitcode. And you can actually extract it easily. So there is a tool called libEBC. I think it stands for embedded bitcode. So what it does, it just extracts the bitcode file from an executable. It works for some Linux and Mac and FreeBSD. And the original tool works, it just does extract the files and stores them locally on disk so that you can do some post-processing on them. And I made it work to make it work in memory. So everything happens in memory as I showed at the very beginning. So yeah, super straightforward. I will be up-streaming the patches soon, so the original version should get the support if we agree with maintainers. But otherwise, you can check the things out. Oh yeah, yeah, these are examples how it works. So you just call libcotill and then you get a bunch of bitcode files. Or yeah, you just call the tool. I think the last point, multi-wise support. So we are not that many people working on this project and it's really tough to maintain these things. So I can just recommend say two words, like background and ansible. Just like the way to go. So what background does is just the system that operates or that helps you manage virtual machines. It supports virtual box, VMware, Docker, I think, yeah, anything you can imagine it supports. So in that case, I have just like one file with that many lines that, yeah, then I run background up, Debian, for example, and I have virtual machine running ready with everything I need on it. So I can just start debugging right away. I can access it to it and then when I'm done, I just can destroy it to dispose the memory. And the good part of it that if you have some IDE that supports remote debugging, so as, for example, CLAN does have it, you can run this, set up this virtual machine, then connect with CLAN and you're like, this is at least, for example, how I work on Mac. I do have native experience as a Mac user, but I'm running and debugging my software on Linux. It's very useful. Yeah, the second part is ansible. So here you may notice like VM, Provision, Ansible, and then a playbook, Ubuntu YAML. So what playbooks are, it just like set of some tasks that needs to be done. This is an example of things. Yeah, it's just YAML file does something. And so for, in our case, most of the parts they are shared between all the other operating systems like FreeBSD and Linux, but it's slightly different on Mac. But the cool part about it that I can also run this, the Ansible thing locally. So for example, if I get a new Mac machine, I just use our Mac OS tasks to set up things locally. Yeah. What else? Yeah, the other benefit of this, the Ansible scripts or playbooks that you can also use it like on CI and everywhere you want. So at this point, yeah, I think that's pretty much it. So I try to come up with things that are general enough to be applicable for everyone. So this is not all the problems or issues we have solved so far. But I hope that it will help you to build better tooling around Telolvm. Thank you. Thank you. Thanks. Yeah. Thanks for your talk. I found it super useful. To kind of pile on the Telolvm config hate, I noticed you were linking against the Clang tooling. Were you, did you try to link it against any other parts of Clang, not Telolvm but Clang parts and find any issues with like linkage order? So the question is if I had any issues with the linkage order when using Telolvm config. And yes, I decided not to include it into the slides because that's a mess. But we had this problem, yeah, with different versions of Telolvm and Clang. There were like some problems with different order of things. So we ended up wrapping everything in like start group and group, linker flex. Or on Mac it was like something like dash W, like whole archive or something. So I mean it's just like super hard to support because you have if macOS then you have one bunch of scripts, else if Linux, other and so on so forth. That was CMake, no problems whatsoever, just works. Yeah, it's awesome to see how wide support you have across all these different OSes and versions as well. It's really cool stuff. Thank you. Yeah. As far as I know, org.jit was supposed to improve on MCJIT but in the end you still end up using your own JIT. So what was wrong with it? Yeah, so the question is that why don't we use org.jit or MCJIT and come up with our kind of own solution. So one of the problems that we had with MCJIT initially that it requires you to give up ownership of all the modules. So for all the modules it's like unique pointer and you need to give up it so you don't own them anymore. And in our case we wanted to own it early. So we cannot use MCJIT. Before we switched to org.jit because it didn't have this problem initially. But as the time goes, there was this problem as well. And the API changed drastically. So it was just too much overhead to actually support our own kind of interface to the org.jit. So and we don't need most of the features that it provides like on request completion and yeah, we just lowered program at once and that's it. So thank you. Enjoy the day and see you around.
some experience about building a LLVM-based years, something. The source code is available on works on Linux, for BSD and MacOS. It may work on compiled against any LLVM version between, like, Yeah, it runs like some tests in this case, but in our case, tool for mutation testing. So I CMake. So how to support server versions? There things at the same time. So, either compile against precompiled version or against sources. done with CMake. So now, yeah, I want to talk free. So, basically, you get a crash. For the bitcode that can be shared across different systems like FreeBSD and Linux, but it's slightly will help you to build better tooling around