Using eBPF for Linux Performance Analyses
Zaitsev, Peter
eBPF is one of the most powerful instrumentation frameworks in modern Linux Kernel. In this presentation we will cover eBPF basics as well as usage of Interactive eBPF tools, showing practical examples when and how they can help to resolve performance problems. Finally we will look into Cloudflare’s ebpf_exporter and show how you can integrate eBPF with Prometheus for Long Term Monitoring
 So folks, how many of you are familiar with EBPF here? OK, wonderful, wonderful. About half of you. That's great. Well, before we get to that, let me say a few words about myself to manage your expectations. So I'm a performance geek who turned to be CEO. So now I had to work more, I would guess, on the company performance, then assistant performance. But you know what? I still keep my passion, and that's why I go and talk on the lot of technical conference still. In this presentation, we'll look into EBPF overview and how it works on Linux, as well as look at the practical examples of the EBPF-based tools, which I hope you find helpful and be able to apply to new environments. So EBPF are history and its support in Linux. EBPF, it stands for Extended Berkeley Pocket Filter, which is a weird name for a tool we want to use for monitoring, isn't it? Well, and reality is that Ease originated from the Berkeley pocket filter, which was exactly designed to build efficient virtual machine for pocket filter, and then extended E. That means it was pretty much extended to be more stuff. So Linux, kernel, has support for that virtual machine called EBPF. In general, EBPF is Event Processing Framework, which is used a lot for monitoring, but that also can be used for some other stuff indeed in the network stack. In modern versions, we have the JIT compiler, which compiles those programs for the higher efficiency. EBPF in Linux appeared first in 2014, so it has been out there for quite a while, but it has still been very actively improved. And in recent years, it's also been integrated in a pair of tooling, which is this wonderful framework for interactive performance analysis in Linux. If you look at the improvements in recent kernels, you can see the list of changes in large. And pretty much every major or minor kernel release does include some additional features in EMPF, which can range from the new instrumentation points to the various new types. What kind of data structures we can use, how we can process and aggregate those events. So as I mentioned, what EBPF is, is this kind of a special bytecode, like a small program, which you can go ahead and insert in the kernel, and which would be executed when different events are being triggered. What is great about EBPF is what is a checks done before those programs are included. So it's not impossible, but it is hard to screw yourself up. So it is hard to write an EBPF program, to basic mistake when you boom, try to insert it in the trace point, and your system completely crashes. For example, if you just put a tight loop in the EBPF program, it would not pass a validation and it wouldn't include. Right now, even for the EBPF itself, it's based on a bytecode where LLVM-C-Land can compile the programs into that EBPF bytecode, and that is what typically used as a part of the EBPF platform. Now what is interesting in this case is what that compilation tends to be kernel dependent, and that is why the installation of EBPF is a little bit more involved in a lot of cases. But for the good stuff with LLVM, that means few of us would actually need to run EBPF programs directly. Here is, if you're curious, EBPF code example, which actually comes from a TCP dump, which is another tool which uses EBPF for packet filtering, and it has the need tool where you can actually dump the EBPF programs in what looks like the assembly-like code for the bytecode that represents. OK. EBPF has support both on the kernel side and on the user side. And in this case, I have an image from a wonderful, wonderful resource of Brandon Gregg. If you are interested in EBPF on Linux and using that and tools, Brandon Gregg's website has absolutely a fantastic resource. But I think it's a very nice illustration out there. What we can see is one way or another, we will have a user program to generate bytecode, which will be verified by the kernel through BPF. It can connect to the different interfaces in kernels from KPro, ViewPro, DrapeSpo, and Perf events. And then what the map is, is what that data structure, which can be used to accumulate the performance statistics. That probe will run, accumulate data in this case, and then we can use, again, in the user space, some program to read that data in the map, in the display that, as you'll actually see in the tools I'll show you. This is now a wonderful image, which shows you all kind of different tools, which can be used to travel, shoot, or analyze different Linux kernel subsystem. And you can see that the EBPF ecosystem is large, and there are already many tools which use EBPF, so you don't have to write your own probes. Here is another wonderful image from the same source, which really shows us in a different illustration what Linux kernel versions support, what kind of EBPF features. And I'm not spending a lot of time on that, because I think these things are self-explanatory, and you can research them later on on the slides. Here is also from Brandon's Greg, the interesting idea, a feature which shows the different tools. It shows by how easy it is to use, to how mature it is. And you can see what there have been a lot of tools like raw BPF, for example, was pretty mature, but that's very brutal to use. And that's nice to use what we have in the tools in this kind of corner, which become both easy to use, powerful, and mature. So EBPF, I think, is now at a very great time in Linux kernel, but it's very powerful, but at the same time very mature to use. Now, one thing I see from looking at EBPF tools is not all of them are available in the single package. The first two packages which you may want to look at would be Perf Tools Unstable, in unstable version, there are more stuff for you to use. And then there's also this iUvisor project, which has tools like BCC and PLI, which can be used for a lot of other kind of simple shell scripts and otherwise to provide those. So these are both pretty mature tools. You can get them to installed relatively easily. In this case, that's just an example. You have a repository which has a package which makes it relatively not too painful. Now, I mentioned already what EBPF depends on the kernel. That is why you can see it has to pull in the kernel headers to understand exactly the data structures you're working in, what exactly different offsets for different data, and so on and so forth. OK. Let's now look at some tools in actions which are based on EBPF. The first one is Profiler. What Profile, and that's the name of the tool, allows us to trace the given program and look at where different threads are in the stack. This is different, for example, from Perf, because the tools like Perf, they typically show you CPU usage. In this case, you will see where threads are getting stuck, where that is doing some CPU compute, where they are stuck waiting on Vutix, waiting on disk.io, or whatever, which can be very helpful to understand what exactly is going on in the program, why it's not running so fast. Another cool tool is understanding your IOL latency, especially with modern storage in the cloud and so on and so forth. You often have to deal with not if you're average, but if you're outliers. And this tool pretty much allows us to understand the latency of your VLOG device. In this case, it's kind of quite natural. We're not having a lot of outliers, but this is actually interesting. We can see this VLOG device, we had one of the IOL requests which took, what is it? I think that's microseconds. So that took two seconds or something to complete, which is if you, for example, run in the database server and that happens to be something like your log flush request, then it's bad. It will have a very bad impact out there. And either, which is a very nice tool, which allows us to snoop of a VLOG device IOL. So if you ever wondered where your IOL is sequential, or if that's kind of random and what's not, you can rather easy to trace that with the tool, and then you can run some program easier to analyze, plot, visualize your IOL pattern in more details. This is the VLOG device. There are also tools to provide you the same information for file systems. And there are different tools for, this is for EXT4. We can see how much file system main operations like read, write, FC, and take. Because again, for all kind of different storage reasons, as well as internal kernel reasons, we may see file systems causing stalls, which can impact the operations of our programs, which may be hard to travel should IOLs, because you probably do not instrument every single IOL operation in your program. There is other tool, which is related to that. And if you do not want to trace everything or build a histogram, you can also use this tool to understand what kind of IOL operations do not meet your, let's call it, service level objective. So in this case, we can say, hey, what are your operations took more than 10 milliseconds? You can put, obviously, any number you want. And that tells, well, you know what? There have been those, these are, these are requests which have taken so much time. Now, another interesting statistics. For me, I was always wondering, well, Linux has this cache. But how efficient that is, really. How do I really see, do I have a lot of hits? If I have more memory in the cache, is it really getting more hits or less? Well, with EVPF, you can do that. There is a cache start tool, which shows me how many total requests have been reading from the cache, how many hits, misses, how many blocks has been dotted, and so on. And that really can help you to understand your cache performance. Another interesting thing. So if you look at your applications, which are a CPU bound, one of the critical items in terms of a way would be responsive and perform well, is when the application is ready to run, how long does it take Scandal to find the CPU to actually schedule it on? Because if all the CPUs are busy, then you will have to wait until the CPU becomes available, or preempted, and so on and so forth. This tool allows you to actually see that from a run queue delay profile, and you can see how long it takes before the program became ready to run to its scheduled, which is, I think, a very cool way to understand where you're having the CPU saturation. It's much better than just looking at your CPU usage averages. OK. The next one. If you ever wondered for your program, which you have maybe not in the shell, but in some more complicated languages, what are the tools and other stuff it executes? Well, there is this wonderful tool called execsnup, where you can look wherever for a whole system or for a specific program, what are different programs being started? And that can help you to understand what the program does better, or if there are some problems, something fails, you can often figure that out from this tool. The next one is open files. That, I think, is another also very helpful thing. I find that helpful when programs do not have diagnostic error messages set very well. It says, oh, there is an IOR error, or can't find a file. What file? I mean, some Linux programs are getting the error messages which are as bad as Windows errors. Right? So that allows you to deal with those programs, because you can understand what files the program's accessing. And from that, you can often infer what could possibly go wrong and travel through that. Right? Besides files, you can also trace what the programs connect to in your system. This allows us help to trace the TCP connections, where I would go in TCP connections in your system, which, again, can be very helpful to understand how programs work and travel through them as needed. This one I find a very cool one. Right? So in many cases, what we see with database travel shooting, right? For example, somebody complains, hey, the query is slow. But the query should not be slow, right? Because if you look at the logs at the, let's say, my school site, and it's all fast application, it's slow. Possibly that was communicated because of some network issues. In a lot of the modern networks, the latency you would see is being quite good. But if you have some packet being dropped and retrace meet happen, then that can introduce a huge delays before internet communication. The first two, you can actually see what exactly network retrace meet happen, not just in aggregates, which you can get from NetStuff, but for specific connections. And then for those specific connections, I can also understand what program write application if a user was involved out there. The next one, DNS lookup. That is a latency measurement also sometimes not easy to figure out. And I think it's often not taking account. But DNS lookup may be a huge performance impact for you connecting to the service. Here are over tools which are available in the version of BCCI used. And those tools are constantly being developed and extended. Moving on. For PLI, that is an other tool, which is wonderful to get some additional scripting if you want, besides running custom tools. For example, you can run this kind of simple probe which can say, hey, I'm connecting to this kernel C-scope and give me the histogram of very strong values. Relatively simple. Gives us a beautiful chart. That is example of a PLI language. It uses a custom language. But if you do the programming, that is not too hard language to learn and understand. OK. This is all the tools I showed you which are based on kind of real time troubleshooting analysis, which is wonderful. But what if you want to have some stats you want to analyze in the past? Yes, and there is a tool for that. The state of art, I believe, in this case is CloudFlares evpf exporter, which, as the name says, is exporter for Prometheus. So you can get pretty much any data from evpf in a Prometheus which you can do wherever you can plot it. You can use an alert manager and do all the other good stuff. Now, here is a nice graph in this case which I took from a CloudFlares layer stock which tells us that while all of those things, they have the same average value on both axes, these are completely different pictures. If you haven't seen that image, I think that's kind of quite cool. That is why we need to go beyond the averages and even the mean values and to really understand the histogram. So if you look evpf exporter, it uses BCC library to get the data. And pretty much you can take the data from BCC programs and output that in a Prometheus format. One thing you have to be careful running exporter in production is the overhead. So if you put, for example, a very complex probe and if you put that on a very common operations which you run in millions of times a second, you can have quite an overhead. As you can see, CloudFlares' measured stats can be way over 100%. So be careful what you run in production versus what you run in a test or you only enable when a system is not healthy. So for evpf exporter, there are boundaries available. You can get that and also provide the configuration information. You can plug it in in a standard Prometheus, of course. I also show you how to integrate that with our open source monitoring tool called PMM, where you can pretty much add it as an external exporter. And what you can get in this case here, as an example, is IELA latency histogram. Just the same as I showed you early on in the presentation, which we can then go ahead and plot as IELA histogram over time to visualize that in Grafana. And that can be used for us, for example, to help us to understand the outliers. Like, well, in this case, we had some requests which have been taken more than 33 milliseconds, which is kind of slow for this SSD storage in this day and age. So what's coming in evpf? What I'm very excited in evpf is bpf trace, which is kind of in this style is very featureful in detrace replacement. Right now, it is not quite mature yet in terms of having packages available. But it is really getting there. And I would say that probably would be one of the very powerful evpf tools in the next year or two. Here is, for reading least, as I prepared, I went through a lot of articles. I thought I'll just leave it on the slide. And that's it. I think I have exactly a minute and a half for questions. So please remain seated. Any questions? That was so unclear, no questions, right? Yes, so the new tools appear, and there are detrace, ftrace, LTNG, tons of that, 15. Which ones are likely to go away? Maybe you have compared them. Yeah. Well, that is a good question, right? I mean, somebody before us talks, hey, Peter, why don't you talk about detrace and Linux, right? And I say, well, detrace was wonderful, but it's kind of dead, right? I mean, detrace is available, for example, in Oracle Linux. But how many of you here are running Oracle Linux? To prove a point, right? So in this case, I believe what the evpf at this point is is the leading monitor, trace and instrumentation framework for Linux for this kind of advanced instrumentation use case, which is there to stay, right? And I think that is where a future is. Make sense? I have a question, which is a lot of these tools seem very, very useful to run in production systems, because that's where you find the really weird problems that you cannot debug and reproduce anywhere. But the question is, I always see these things running as root and being advertised as unstable. So is there a better way for me to not be running things that are marked as unstable under root on my production system? Oh, well, if you, well, there are many ways, right? But I mean, in the end, if you are inserting some custom code into a kernel, right, you are doing something related to dangerous. And as I mentioned, if the performance overhead, at least, right, while evpf is designed to avoid you crashing the kernel, you can make it run multiple times slower, right, in this case. So in this case, that is advanced monitoring tools, which have to be done. Now, what I would imagine in this case, right, if you want developers to do, then build the safe tools, right, and let them only to run those from sudo, right, or something for that particular use cases which you are authorized. That's what I would think. You have presented a couple of tools which are quite similar in the, well, in the targets that they can do with S-rays, for instance. So how do they compare to S-rays? And do you think S-rays should be rewritten in order to leverage evpf capabilities? The great question is how is that, is that compares to S-rays, right, the overhead? Evpf has, you know, orders of magnitude lower overhead than what the S-rays does. So should we write S-rays to leverage evpf? Well, you know, possibly, right? I mean, I'm not sure. Maybe there is actually work to do that already, right? But in general, for the tools, right, the overhead I think is right now is the main difference with what you can do with S-rays. Any more questions? Yes. Oh, you're getting your exercise here, right? Up and down, up and down. It's good for me. Yeah, you can. You next? I'm done. OK. So all the way up. How do you have the? We actually put histogram data of the audio coming off the microphone. Oh, that's very cool. Yeah, so my question is to the audience and to the speaker, is anybody using this in production now? Cloudflare. The exporter. No. Yeah, Cloudflare has an exporter, right? I mean, there is a lot of people writing about using that in production. I know that Brandon Gregg uses that very accurately at Netflix, right? But he's using the tools individually at runtime, not the exporter. Not exporter. Yeah, I'm actually running the exporter in production. Oh, you're just asking about the exporter. That is a good question. I would have raised my hand to my own question. But I was just wondering if anybody else. Oh, you use the exporter in production? Yes. And it's not 23% slower. I've benchmarked it as well. It's pretty close. Yeah, what I'm saying again is you have to be smart with probes you set up, right? You can set up a probe to fuck yourself up, right? As if many other stuff are in the thing is life. Any other questions? OK, well, thank you.
 So folks, how many of you are familiar with EBPF here?  OK, wonderful, wonderful.  About half of you.  That's great.  Well, before we get to that, let me say a few words about  myself to manage your expectations.  So I'm a performance geek who turned to be CEO.  So now I had to work more, I would guess, on the company  performance, then assistant performance.  But you know what?  I still keep my passion, and that's why I go and talk on  the lot of technical conference still.  In this presentation, we'll look into EBPF overview and how  it works on Linux, as well as look at the practical examples  of the EBPF-based tools, which I hope you find helpful and be  able to apply to new environments.  So EBPF are history and its support in Linux.  EBPF, it stands for Extended Berkeley Pocket Filter, which  is a weird name for a tool we want to use for monitoring,  isn't it?  Well, and reality is that Ease originated from the Berkeley  pocket filter, which was exactly designed to build  efficient virtual machine for pocket filter, and then  extended E. That means it was pretty much extended to be  more stuff.  So Linux, kernel, has support for that virtual machine  called EBPF.  In general, EBPF is Event Processing Framework, which is  used a lot for monitoring, but that also can be used for some  other stuff indeed in the network stack.  In modern versions, we have the JIT compiler, which compiles  those programs for the higher efficiency.  EBPF in Linux appeared first in 2014, so it has been out  there for quite a while, but it has still been very  actively improved.  And in recent years, it's also been integrated in a pair of  tooling, which is this wonderful framework for  interactive performance analysis in Linux.  If you look at the improvements in recent kernels, you can see  the list of changes in large.  And pretty much every major or minor kernel release does  include some additional features in EMPF, which can  range from the new instrumentation points to the  various new types.  What kind of data structures we can use, how we can process  and aggregate those events.  So as I mentioned, what EBPF is, is this kind of a special  bytecode, like a small program, which you can go ahead and  insert in the kernel, and which would be executed when  different events are being triggered.  What is great about EBPF is what is a checks done before  those programs are included.  So it's not impossible, but it is hard to  screw yourself up.  So it is hard to write an EBPF program, to basic mistake  when you boom, try to insert it in the trace point, and your  system completely crashes.  For example, if you just put a tight loop in the EBPF  program, it would not pass a validation and it wouldn't  include.  Right now, even for the EBPF itself, it's based on a  bytecode where LLVM-C-Land can compile the programs into  that EBPF bytecode, and that is what typically used as a  part of the EBPF platform.  Now what is interesting in this case is what that  compilation tends to be kernel dependent, and that is why  the installation of EBPF is a little bit more involved in  a lot of cases.  But for the good stuff with LLVM, that means few of us  would actually need to run EBPF programs directly.  Here is, if you're curious, EBPF code example, which  actually comes from a TCP dump, which is another tool which  uses EBPF for packet filtering, and it has the need tool  where you can actually dump the EBPF programs in what looks  like the assembly-like code for the bytecode that  represents.  OK.  EBPF has support both on the kernel side and on the user  side.  And in this case, I have an image from a wonderful,  wonderful resource of Brandon Gregg.  If you are interested in EBPF on Linux and using that and  tools, Brandon Gregg's website has absolutely a  fantastic resource.  But I think it's a very nice illustration out there.  What we can see is one way or another, we will have a user  program to generate bytecode, which will be verified by  the kernel through BPF.  It can connect to the different interfaces in kernels  from KPro, ViewPro, DrapeSpo, and Perf events.  And then what the map is, is what that data structure, which  can be used to accumulate the performance statistics.  That probe will run, accumulate data in this case, and then  we can use, again, in the user space, some program to read  that data in the map, in the display that, as you'll  actually see in the tools I'll show you.  This is now a wonderful image, which shows you all kind of  different tools, which can be used to travel, shoot, or  analyze different Linux kernel subsystem.  And you can see that the EBPF ecosystem is large, and there  are already many tools which use EBPF, so you don't have to  write your own probes.  Here is another wonderful image from the same source, which  really shows us in a different illustration what Linux  kernel versions support, what kind of EBPF features.  And I'm not spending a lot of time on that, because I think  these things are self-explanatory, and you can  research them later on on the slides.  Here is also from Brandon's Greg, the interesting  idea, a feature which shows the different tools.  It shows by how easy it is to use, to how  mature it is.  And you can see what there have been a lot of tools like  raw BPF, for example, was pretty mature, but that's very  brutal to use.  And that's nice to use what we have in the tools in this kind  of corner, which become both easy to use, powerful, and  mature.  So EBPF, I think, is now at a very great time in Linux  kernel, but it's very powerful, but at the same time very  mature to use.  Now, one thing I see from looking at EBPF tools is not  all of them are available in the single package.  The first two packages which you may want to look at would  be Perf Tools Unstable, in unstable version, there are  more stuff for you to use.  And then there's also this iUvisor project, which has  tools like BCC and PLI, which can be used for a lot of  other kind of simple shell scripts and  otherwise to provide those.  So these are both pretty mature tools.  You can get them to installed relatively easily.  In this case, that's just an example.  You have a repository which has a package which makes it  relatively not too painful.  Now, I mentioned already what EBPF depends on the kernel.  That is why you can see it has to pull in the kernel  headers to understand exactly the data structures you're  working in, what exactly different offsets for different  data, and so on and so forth.  OK.  Let's now look at some tools in actions which are based on  EBPF.  The first one is Profiler.  What Profile, and that's the name of the tool, allows us to  trace the given program and look at where different  threads are in the stack.  This is different, for example, from Perf, because the  tools like Perf, they typically show you CPU usage.  In this case, you will see where threads are getting  stuck, where that is doing some CPU compute, where they are  stuck waiting on Vutix, waiting on disk.io, or  whatever, which can be very helpful to understand what  exactly is going on in the program, why it's not  running so fast.  Another cool tool is understanding your IOL latency,  especially with modern storage in the cloud and so on and so  forth.  You often have to deal with not if you're average, but if you're  outliers.  And this tool pretty much allows us to understand the  latency of your VLOG device.  In this case, it's kind of quite natural.  We're not having a lot of outliers, but this is actually  interesting.  We can see this VLOG device, we had one of the IOL requests  which took, what is it?  I think that's microseconds.  So that took two seconds or something to complete, which  is if you, for example, run in the database server and that  happens to be something like your log flush request, then  it's bad.  It will have a very bad impact out there.  And either, which is a very nice tool, which allows us to  snoop of a VLOG device IOL.  So if you ever wondered where your IOL is sequential, or if  that's kind of random and what's not, you can rather easy  to trace that with the tool, and then you can run some  program easier to analyze, plot, visualize your IOL  pattern in more details.  This is the VLOG device.  There are also tools to provide you the same information for  file systems.  And there are different tools for, this is for EXT4.  We can see how much file system main operations like read,  write, FC, and take.  Because again, for all kind of different storage reasons, as  well as internal kernel reasons, we may see file systems  causing stalls, which can impact the operations of our  programs, which may be hard to travel should IOLs, because  you probably do not instrument every single IOL  operation in your program.  There is other tool, which is related to that.  And if you do not want to trace everything or build a  histogram, you can also use this tool to understand what kind  of IOL operations do not meet your, let's call it, service  level objective.  So in this case, we can say, hey, what are your operations  took more than 10 milliseconds?  You can put, obviously, any number you want.  And that tells, well, you know what?  There have been those, these are, these are requests which  have taken so much time.  Now, another interesting statistics.  For me, I was always wondering, well, Linux has this cache.  But how efficient that is, really.  How do I really see, do I have a lot of hits?  If I have more memory in the cache, is it really getting  more hits or less?  Well, with EVPF, you can do that.  There is a cache start tool, which shows me how many total  requests have been reading from the cache, how many hits,  misses, how many blocks has been dotted, and so on.  And that really can help you to understand your cache  performance.  Another interesting thing.  So if you look at your applications, which are a CPU  bound, one of the critical items in terms of a way  would be responsive and perform well, is when the application  is ready to run, how long does it take  Scandal to find the CPU to actually schedule it on?  Because if all the CPUs are busy, then you  will have to wait until the CPU becomes available,  or preempted, and so on and so forth.  This tool allows you to actually see that from a run queue  delay profile, and you can see how long it takes before the  program became ready to run to its scheduled, which is, I  think, a very cool way to understand where you're having  the CPU saturation.  It's much better than just looking at your CPU usage  averages.  OK.  The next one.  If you ever wondered for your program, which you have maybe  not in the shell, but in some more complicated languages,  what are the tools and other stuff it executes?  Well, there is this wonderful tool called execsnup, where you  can look wherever for a whole system or for a specific  program, what are different programs being started?  And that can help you to understand what the program  does better, or if there are some problems, something fails,  you can often figure that out from this tool.  The next one is open files.  That, I think, is another also very helpful thing.  I find that helpful when programs do not have diagnostic  error messages set very well.  It says, oh, there is an IOR error, or can't find a file.  What file?  I mean, some Linux programs are getting the error messages  which are as bad as Windows errors.  Right?  So that allows you to deal with those programs, because you  can understand what files the program's accessing.  And from that, you can often infer what could possibly go  wrong and travel through that.  Right? Besides files, you can also trace what the programs  connect to in your system.  This allows us help to trace the TCP connections, where I would  go in TCP connections in your system, which, again, can be  very helpful to understand how programs work and travel  through them as needed.  This one I find a very cool one.  Right?  So in many cases, what we see with database  travel shooting, right?  For example, somebody complains, hey, the query is slow.  But the query should not be slow, right?  Because if you look at the logs at the, let's say, my  school site, and it's all fast application, it's slow.  Possibly that was communicated because of some network issues.  In a lot of the modern networks, the latency you would see  is being quite good.  But if you have some packet being dropped and retrace  meet happen, then that can introduce a huge delays before  internet communication.  The first two, you can actually see what exactly network  retrace meet happen, not just in aggregates, which you can get  from NetStuff, but for specific connections.  And then for those specific connections, I can also  understand what program write application if a user was  involved out there.  The next one, DNS lookup.  That is a latency measurement also sometimes not easy to  figure out.  And I think it's often not taking account.  But DNS lookup may be a huge performance impact for you  connecting to the service.  Here are over tools which are available in the version of  BCCI used.  And those tools are constantly being developed and  extended.  Moving on.  For PLI, that is an other tool, which is wonderful to get  some additional scripting if you want, besides running  custom tools.  For example, you can run this kind of simple probe which  can say, hey, I'm connecting to this kernel C-scope and give  me the histogram of very strong values.  Relatively simple.  Gives us a beautiful chart.  That is example of a PLI language.  It uses a custom language.  But if you do the programming, that is not too hard language  to learn and understand.  OK.  This is all the tools I showed you which are based on  kind of real time troubleshooting analysis, which  is wonderful.  But what if you want to have some stats you want to  analyze in the past?  Yes, and there is a tool for that.  The state of art, I believe, in this case is CloudFlares  evpf exporter, which, as the name says, is exporter for  Prometheus.  So you can get pretty much any data from evpf in a  Prometheus which you can do wherever you can plot it.  You can use an alert manager and do all the other good stuff.  Now, here is a nice graph in this case which I took from a  CloudFlares layer stock which tells us that while all of  those things, they have the same average value on both  axes, these are completely different pictures.  If you haven't seen that image, I think that's  kind of quite cool.  That is why we need to go beyond the averages and even the  mean values and to really understand the histogram.  So if you look evpf exporter, it uses BCC library to get the  data.  And pretty much you can take the data from BCC programs and  output that in a Prometheus format.  One thing you have to be careful running exporter in  production is the overhead.  So if you put, for example, a very complex probe and if you  put that on a very common operations which you run in  millions of times a second, you can have  quite an overhead.  As you can see, CloudFlares' measured  stats can be way over 100%.  So be careful what you run in production versus what you run  in a test or you only enable when a system is not healthy.  So for evpf exporter, there are boundaries available.  You can get that and also provide the configuration  information.  You can plug it in in a standard Prometheus, of course.  I also show you how to integrate that with our open source  monitoring tool called PMM, where you can pretty much add it  as an external exporter.  And what you can get in this case here, as an example, is  IELA latency histogram.  Just the same as I showed you early on in the presentation,  which we can then go ahead and plot as IELA histogram over  time to visualize that in Grafana.  And that can be used for us, for example, to help us to  understand the outliers.  Like, well, in this case, we had some requests which have  been taken more than 33 milliseconds, which is kind of  slow for this SSD storage in this day and age.  So what's coming in evpf?  What I'm very excited in evpf is bpf trace, which is kind of  in this style is very featureful in detrace replacement.  Right now, it is not quite mature yet in terms of having  packages available.  But it is really getting there.  And I would say that probably would be one of the very  powerful evpf tools in the next year or two.  Here is, for reading least, as I prepared, I went through a  lot of articles.  I thought I'll just leave it on the slide.  And that's it.  I think I have exactly a minute and a half for questions.  So please remain seated.  Any questions?  That was so unclear, no questions, right?  Yes, so the new tools appear, and there are detrace,  ftrace, LTNG, tons of that, 15.  Which ones are likely to go away?  Maybe you have compared them.  Yeah.  Well, that is a good question, right?  I mean, somebody before us talks, hey, Peter, why don't you  talk about detrace and Linux, right?  And I say, well, detrace was wonderful, but it's kind of  dead, right?  I mean, detrace is available, for example, in Oracle Linux.  But how many of you here are running Oracle Linux?  To prove a point, right?  So in this case, I believe what the evpf at this point is  is the leading monitor, trace and instrumentation framework  for Linux for this kind of advanced instrumentation  use case, which is there to stay, right?  And I think that is where a future is.  Make sense?  I have a question, which is a lot of these tools seem very,  very useful to run in production systems, because that's  where you find the really weird problems that you cannot debug  and reproduce anywhere.  But the question is, I always see these things running as  root and being advertised as unstable.  So is there a better way for me to not be running things that  are marked as unstable under root on my production system?  Oh, well, if you, well, there are many ways, right?  But I mean, in the end, if you are inserting some custom  code into a kernel, right, you are doing something  related to dangerous.  And as I mentioned, if the performance overhead,  at least, right, while evpf is designed to avoid you  crashing the kernel, you can make it run multiple times  slower, right, in this case.  So in this case, that is advanced monitoring tools,  which have to be done.  Now, what I would imagine in this case, right, if you want  developers to do, then build the safe tools, right,  and let them only to run those from sudo, right, or something  for that particular use cases which you are authorized.  That's what I would think.  You have presented a couple of tools which are quite similar  in the, well, in the targets that they can do with S-rays,  for instance.  So how do they compare to S-rays?  And do you think S-rays should be rewritten in order to  leverage evpf capabilities?  The great question is how is that, is that  compares to S-rays, right, the overhead?  Evpf has, you know, orders of magnitude lower overhead than  what the S-rays does.  So should we write S-rays to leverage evpf?  Well, you know, possibly, right?  I mean, I'm not sure.  Maybe there is actually work to do that already, right?  But in general, for the tools, right, the overhead I think  is right now is the main difference with what you  can do with S-rays.  Any more questions?  Yes.  Oh, you're getting your exercise here, right?  Up and down, up and down.  It's good for me.  Yeah, you can.  You next?  I'm done.  OK.  So all the way up.  How do you have the?  We actually put histogram data of the audio coming off the  microphone.  Oh, that's very cool.  Yeah, so my question is to the audience and to the speaker,  is anybody using this in production now?  Cloudflare.  The exporter.  No.  Yeah, Cloudflare has an exporter, right?  I mean, there is a lot of people writing about using that  in production.  I know that Brandon Gregg uses that very accurately at  Netflix, right?  But he's using the tools individually at runtime, not  the exporter.  Not exporter.  Yeah, I'm actually running the exporter in production.  Oh, you're just asking about the exporter.  That is a good question.  I would have raised my hand to my own question.  But I was just wondering if anybody else.  Oh, you use the exporter in production?  Yes.  And it's not 23% slower.  I've benchmarked it as well.  It's pretty close.  Yeah, what I'm saying again is you have to be smart with  probes you set up, right?  You can set up a probe to fuck yourself up, right?  As if many other stuff are in the thing is life.  Any other questions?  OK, well, thank you.
Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_0_47.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_48_155.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_156_203.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_204_1463.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_1464_1619.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_1620_3287.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_3288_3827.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_3828_4715.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_4716_7211.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_7212_7895.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_7896_9455.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_9456_10043.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_10044_11819.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_11820_12803.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_12804_13655.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_13656_13847.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_13848_14807.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_14808_16991.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_16992_18131.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_18132_19115.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_19116_20411.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_20412_22103.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_22104_23051.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_23052_24107.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_24108_24815.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_24816_27023.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_27024_27275.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_27276_27611.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_27612_28055.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_28056_28391.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_28392_28739.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_28740_28835.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_28836_29363.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_29364_30359.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_30360_30719.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_30720_31547.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_31548_31931.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_31932_32147.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_32148_32687.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_32688_33023.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_33024_33467.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_33468_33563.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_33564_34415.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_34416_34655.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_34656_43523.png Using_eBPF_for_Linux_Performance_Analyses_pQRvxCpVykM_43524_43642.png
So folks, how many of you are familiar with EBPF here? OK, wonderful, wonderful. About half of you. That's great. Well, before we get to that, let me say a few words about myself to manage your expectations. So I'm a performance geek who turned to be CEO. So now I had to work more, I would guess, on the company performance, then assistant performance. But you know what? I still keep my passion, and that's why I go and talk on the lot of technical conference still. In this presentation, we'll look into EBPF overview and how it works on Linux, as well as look at the practical examples of the EBPF-based tools, which I hope you find helpful and be able to apply to new environments. So EBPF are history and its support in Linux. EBPF, it stands for Extended Berkeley Pocket Filter, which is a weird name for a tool we want to use for monitoring, isn't it? Well, and reality is that Ease originated from the Berkeley pocket filter, which was exactly designed to build efficient virtual machine for pocket filter, and then extended E. That means it was pretty much extended to be more stuff. So Linux, kernel, has support for that virtual machine called EBPF. In general, EBPF is Event Processing Framework, which is used a lot for monitoring, but that also can be used for some other stuff indeed in the network stack. In modern versions, we have the JIT compiler, which compiles those programs for the higher efficiency. EBPF in Linux appeared first in 2014, so it has been out there for quite a while, but it has still been very actively improved. And in recent years, it's also been integrated in a pair of tooling, which is this wonderful framework for interactive performance analysis in Linux. If you look at the improvements in recent kernels, you can see the list of changes in large. And pretty much every major or minor kernel release does include some additional features in EMPF, which can range from the new instrumentation points to the various new types. What kind of data structures we can use, how we can process and aggregate those events. So as I mentioned, what EBPF is, is this kind of a special bytecode, like a small program, which you can go ahead and insert in the kernel, and which would be executed when different events are being triggered. What is great about EBPF is what is a checks done before those programs are included. So it's not impossible, but it is hard to screw yourself up. So it is hard to write an EBPF program, to basic mistake when you boom, try to insert it in the trace point, and your system completely crashes. For example, if you just put a tight loop in the EBPF program, it would not pass a validation and it wouldn't include. Right now, even for the EBPF itself, it's based on a bytecode where LLVM-C-Land can compile the programs into that EBPF bytecode, and that is what typically used as a part of the EBPF platform. Now what is interesting in this case is what that compilation tends to be kernel dependent, and that is why the installation of EBPF is a little bit more involved in a lot of cases. But for the good stuff with LLVM, that means few of us would actually need to run EBPF programs directly. Here is, if you're curious, EBPF code example, which actually comes from a TCP dump, which is another tool which uses EBPF for packet filtering, and it has the need tool where you can actually dump the EBPF programs in what looks like the assembly-like code for the bytecode that represents. OK. EBPF has support both on the kernel side and on the user side. And in this case, I have an image from a wonderful, wonderful resource of Brandon Gregg. If you are interested in EBPF on Linux and using that and tools, Brandon Gregg's website has absolutely a fantastic resource. But I think it's a very nice illustration out there. What we can see is one way or another, we will have a user program to generate bytecode, which will be verified by the kernel through BPF. It can connect to the different interfaces in kernels from KPro, ViewPro, DrapeSpo, and Perf events. And then what the map is, is what that data structure, which can be used to accumulate the performance statistics. That probe will run, accumulate data in this case, and then we can use, again, in the user space, some program to read that data in the map, in the display that, as you'll actually see in the tools I'll show you. This is now a wonderful image, which shows you all kind of different tools, which can be used to travel, shoot, or analyze different Linux kernel subsystem. And you can see that the EBPF ecosystem is large, and there are already many tools which use EBPF, so you don't have to write your own probes. Here is another wonderful image from the same source, which really shows us in a different illustration what Linux kernel versions support, what kind of EBPF features. And I'm not spending a lot of time on that, because I think these things are self-explanatory, and you can research them later on on the slides. Here is also from Brandon's Greg, the interesting idea, a feature which shows the different tools. It shows by how easy it is to use, to how mature it is. And you can see what there have been a lot of tools like raw BPF, for example, was pretty mature, but that's very brutal to use. And that's nice to use what we have in the tools in this kind of corner, which become both easy to use, powerful, and mature. So EBPF, I think, is now at a very great time in Linux kernel, but it's very powerful, but at the same time very mature to use. Now, one thing I see from looking at EBPF tools is not all of them are available in the single package. The first two packages which you may want to look at would be Perf Tools Unstable, in unstable version, there are more stuff for you to use. And then there's also this iUvisor project, which has tools like BCC and PLI, which can be used for a lot of other kind of simple shell scripts and otherwise to provide those. So these are both pretty mature tools. You can get them to installed relatively easily. In this case, that's just an example. You have a repository which has a package which makes it relatively not too painful. Now, I mentioned already what EBPF depends on the kernel. That is why you can see it has to pull in the kernel headers to understand exactly the data structures you're working in, what exactly different offsets for different data, and so on and so forth. OK. Let's now look at some tools in actions which are based on EBPF. The first one is Profiler. What Profile, and that's the name of the tool, allows us to trace the given program and look at where different threads are in the stack. This is different, for example, from Perf, because the tools like Perf, they typically show you CPU usage. In this case, you will see where threads are getting stuck, where that is doing some CPU compute, where they are stuck waiting on Vutix, waiting on disk.io, or whatever, which can be very helpful to understand what exactly is going on in the program, why it's not running so fast. Another cool tool is understanding your IOL latency, especially with modern storage in the cloud and so on and so forth. You often have to deal with not if you're average, but if you're outliers. And this tool pretty much allows us to understand the latency of your VLOG device. In this case, it's kind of quite natural. We're not having a lot of outliers, but this is actually interesting. We can see this VLOG device, we had one of the IOL requests which took, what is it? I think that's microseconds. So that took two seconds or something to complete, which is if you, for example, run in the database server and that happens to be something like your log flush request, then it's bad. It will have a very bad impact out there. And either, which is a very nice tool, which allows us to snoop of a VLOG device IOL. So if you ever wondered where your IOL is sequential, or if that's kind of random and what's not, you can rather easy to trace that with the tool, and then you can run some program easier to analyze, plot, visualize your IOL pattern in more details. This is the VLOG device. There are also tools to provide you the same information for file systems. And there are different tools for, this is for EXT4. We can see how much file system main operations like read, write, FC, and take. Because again, for all kind of different storage reasons, as well as internal kernel reasons, we may see file systems causing stalls, which can impact the operations of our programs, which may be hard to travel should IOLs, because you probably do not instrument every single IOL operation in your program. There is other tool, which is related to that. And if you do not want to trace everything or build a histogram, you can also use this tool to understand what kind of IOL operations do not meet your, let's call it, service level objective. So in this case, we can say, hey, what are your operations took more than 10 milliseconds? You can put, obviously, any number you want. And that tells, well, you know what? There have been those, these are, these are requests which have taken so much time. Now, another interesting statistics. For me, I was always wondering, well, Linux has this cache. But how efficient that is, really. How do I really see, do I have a lot of hits? If I have more memory in the cache, is it really getting more hits or less? Well, with EVPF, you can do that. There is a cache start tool, which shows me how many total requests have been reading from the cache, how many hits, misses, how many blocks has been dotted, and so on. And that really can help you to understand your cache performance. Another interesting thing. So if you look at your applications, which are a CPU bound, one of the critical items in terms of a way would be responsive and perform well, is when the application is ready to run, how long does it take Scandal to find the CPU to actually schedule it on? Because if all the CPUs are busy, then you will have to wait until the CPU becomes available, or preempted, and so on and so forth. This tool allows you to actually see that from a run queue delay profile, and you can see how long it takes before the program became ready to run to its scheduled, which is, I think, a very cool way to understand where you're having the CPU saturation. It's much better than just looking at your CPU usage averages. OK. The next one. If you ever wondered for your program, which you have maybe not in the shell, but in some more complicated languages, what are the tools and other stuff it executes? Well, there is this wonderful tool called execsnup, where you can look wherever for a whole system or for a specific program, what are different programs being started? And that can help you to understand what the program does better, or if there are some problems, something fails, you can often figure that out from this tool. The next one is open files. That, I think, is another also very helpful thing. I find that helpful when programs do not have diagnostic error messages set very well. It says, oh, there is an IOR error, or can't find a file. What file? I mean, some Linux programs are getting the error messages which are as bad as Windows errors. Right? So that allows you to deal with those programs, because you can understand what files the program's accessing. And from that, you can often infer what could possibly go wrong and travel through that. Right? Besides files, you can also trace what the programs connect to in your system. This allows us help to trace the TCP connections, where I would go in TCP connections in your system, which, again, can be very helpful to understand how programs work and travel through them as needed. This one I find a very cool one. Right? So in many cases, what we see with database travel shooting, right? For example, somebody complains, hey, the query is slow. But the query should not be slow, right? Because if you look at the logs at the, let's say, my school site, and it's all fast application, it's slow. Possibly that was communicated because of some network issues. In a lot of the modern networks, the latency you would see is being quite good. But if you have some packet being dropped and retrace meet happen, then that can introduce a huge delays before internet communication. The first two, you can actually see what exactly network retrace meet happen, not just in aggregates, which you can get from NetStuff, but for specific connections. And then for those specific connections, I can also understand what program write application if a user was involved out there. The next one, DNS lookup. That is a latency measurement also sometimes not easy to figure out. And I think it's often not taking account. But DNS lookup may be a huge performance impact for you connecting to the service. Here are over tools which are available in the version of BCCI used. And those tools are constantly being developed and extended. Moving on. For PLI, that is an other tool, which is wonderful to get some additional scripting if you want, besides running custom tools. For example, you can run this kind of simple probe which can say, hey, I'm connecting to this kernel C-scope and give me the histogram of very strong values. Relatively simple. Gives us a beautiful chart. That is example of a PLI language. It uses a custom language. But if you do the programming, that is not too hard language to learn and understand. OK. This is all the tools I showed you which are based on kind of real time troubleshooting analysis, which is wonderful. But what if you want to have some stats you want to analyze in the past? Yes, and there is a tool for that. The state of art, I believe, in this case is CloudFlares evpf exporter, which, as the name says, is exporter for Prometheus. So you can get pretty much any data from evpf in a Prometheus which you can do wherever you can plot it. You can use an alert manager and do all the other good stuff. Now, here is a nice graph in this case which I took from a CloudFlares layer stock which tells us that while all of those things, they have the same average value on both axes, these are completely different pictures. If you haven't seen that image, I think that's kind of quite cool. That is why we need to go beyond the averages and even the mean values and to really understand the histogram. So if you look evpf exporter, it uses BCC library to get the data. And pretty much you can take the data from BCC programs and output that in a Prometheus format. One thing you have to be careful running exporter in production is the overhead. So if you put, for example, a very complex probe and if you put that on a very common operations which you run in millions of times a second, you can have quite an overhead. As you can see, CloudFlares' measured stats can be way over 100%. So be careful what you run in production versus what you run in a test or you only enable when a system is not healthy. So for evpf exporter, there are boundaries available. You can get that and also provide the configuration information. You can plug it in in a standard Prometheus, of course. I also show you how to integrate that with our open source monitoring tool called PMM, where you can pretty much add it as an external exporter. And what you can get in this case here, as an example, is IELA latency histogram. Just the same as I showed you early on in the presentation, which we can then go ahead and plot as IELA histogram over time to visualize that in Grafana. And that can be used for us, for example, to help us to understand the outliers. Like, well, in this case, we had some requests which have been taken more than 33 milliseconds, which is kind of slow for this SSD storage in this day and age. So what's coming in evpf? What I'm very excited in evpf is bpf trace, which is kind of in this style is very featureful in detrace replacement. Right now, it is not quite mature yet in terms of having packages available. But it is really getting there. And I would say that probably would be one of the very powerful evpf tools in the next year or two. Here is, for reading least, as I prepared, I went through a lot of articles. I thought I'll just leave it on the slide. And that's it. I think I have exactly a minute and a half for questions. So please remain seated. Any questions? That was so unclear, no questions, right? Yes, so the new tools appear, and there are detrace, ftrace, LTNG, tons of that, 15. Which ones are likely to go away? Maybe you have compared them. Yeah. Well, that is a good question, right? I mean, somebody before us talks, hey, Peter, why don't you talk about detrace and Linux, right? And I say, well, detrace was wonderful, but it's kind of dead, right? I mean, detrace is available, for example, in Oracle Linux. But how many of you here are running Oracle Linux? To prove a point, right? So in this case, I believe what the evpf at this point is is the leading monitor, trace and instrumentation framework for Linux for this kind of advanced instrumentation use case, which is there to stay, right? And I think that is where a future is. Make sense? I have a question, which is a lot of these tools seem very, very useful to run in production systems, because that's where you find the really weird problems that you cannot debug and reproduce anywhere. But the question is, I always see these things running as root and being advertised as unstable. So is there a better way for me to not be running things that are marked as unstable under root on my production system? Oh, well, if you, well, there are many ways, right? But I mean, in the end, if you are inserting some custom code into a kernel, right, you are doing something related to dangerous. And as I mentioned, if the performance overhead, at least, right, while evpf is designed to avoid you crashing the kernel, you can make it run multiple times slower, right, in this case. So in this case, that is advanced monitoring tools, which have to be done. Now, what I would imagine in this case, right, if you want developers to do, then build the safe tools, right, and let them only to run those from sudo, right, or something for that particular use cases which you are authorized. That's what I would think. You have presented a couple of tools which are quite similar in the, well, in the targets that they can do with S-rays, for instance. So how do they compare to S-rays? And do you think S-rays should be rewritten in order to leverage evpf capabilities? The great question is how is that, is that compares to S-rays, right, the overhead? Evpf has, you know, orders of magnitude lower overhead than what the S-rays does. So should we write S-rays to leverage evpf? Well, you know, possibly, right? I mean, I'm not sure. Maybe there is actually work to do that already, right? But in general, for the tools, right, the overhead I think is right now is the main difference with what you can do with S-rays. Any more questions? Yes. Oh, you're getting your exercise here, right? Up and down, up and down. It's good for me. Yeah, you can. You next? I'm done. OK. So all the way up. How do you have the? We actually put histogram data of the audio coming off the microphone. Oh, that's very cool. Yeah, so my question is to the audience and to the speaker, is anybody using this in production now? Cloudflare. The exporter. No. Yeah, Cloudflare has an exporter, right? I mean, there is a lot of people writing about using that in production. I know that Brandon Gregg uses that very accurately at Netflix, right? But he's using the tools individually at runtime, not the exporter. Not exporter. Yeah, I'm actually running the exporter in production. Oh, you're just asking about the exporter. That is a good question. I would have raised my hand to my own question. But I was just wondering if anybody else. Oh, you use the exporter in production? Yes. And it's not 23% slower. I've benchmarked it as well. It's pretty close. Yeah, what I'm saying again is you have to be smart with probes you set up, right? You can set up a probe to fuck yourself up, right? As if many other stuff are in the thing is life. Any other questions? OK, well, thank you.
presentation, we'll look into EBPF overview and how it works on Linux, as well as look at the practical examples of the EBPF-based tools, which illustration out there. What we can see is one usage. In this case, you will see where threads being started? And that can help you to data from evpf in a Prometheus which you can do would be one of the very powerful evpf tools in that they can do with S-rays, for instance. So