System Tap
Wielaard, Mark
SystemTap an application that observes your system on multiple levels, from kernel, libraries, applications, java to database transactions. SystemTap is a new player in the monitoring world! Ever wondered what your system is really doing? Even if your whole system consists of Free Software you might still be scratching your head about what is really going on. Reading all the sources or staring at a core dump after the fact only gets you that far. You have to observe your system while it is running. Luckily the last few years there has been lots of improvements in GNU/Linux around how to monitor, trace, profile and debug your system. Kernel ftrace, pref, tracepoints, better debuginfo, application and library probe points, tapsets and compatibility with dtrace. Systemtap can take advantage of it all and provides a powerful way to get full system observability. Want to dive deep into applications, java processes or the linux kernel without needing to stop or interrupt anything? SystemTap is the tool of choice for complex tasks that may require live analysis, programmable on-line response, and whole-system symbolic access. SystemTap can also handle simple tracing jobs. Learn how to setup SystemTap, what ready to run scripts there are, and how to tailor your probes to specific applications or kernel usage patterns. SystemTap can observe on multiple levels, from kernel, libraries, applications, java to database transactions. To help users to get a better view what an application is doing you can add tapsets and static markers. And such markers can be made compatible with dtrace (to make your solaris hacker friends happy). There are already lots of packages that provide such high level observability, Postgresql, Java hotspot, Xorg, Python, Firefox, etc. Learn how to rapid prototype observability of your application through tapsets with function and statement probes (on any existing binary) and how to add high-level (zero-overhead) markers to your package sources that tell users about everything important, passes, transactions, service starting/stopping, etc.
 Okay, shall we start? Hi, I'm Mark Willard. I currently work for Red Hat and they actually pay me to hack on system tab, which is fun. Some of you may know me from some of my Java work and if you were thinking, oh, Mark is going to talk about Java that's going to be a bit disappointing, there will be one example at the end using system tab with Java, but all the rest is boring old school languages. The talk itself is in two parts. One about system tab itself. Why do we need it? What can I do? The second part is also about system tab, but what we need besides system tab to make it really useful. What you could do with your programs to make system tab and observability on Knudlinix better. Then there will also be a little demonstration of indeed with Java how you make something like a Java runtime more observable with system tab. It's not really a circle, but kind of a circle. Why do we want to have something like system tab? If you are looking at what your system is doing, there are a lot of nice tools. You can do profiling with just the top to see what running on your system using O profile or the new perf tool. You can use some tracing tools as trace to trace your system calls, L trace for library tracing, the new F trace tool to trace kernel function calls. Of course if you then have figured out what is going on, you can drop in to the debugger and really inspect what is going on. The problem is that these are all separate tools. Profiling tools often work system wide while tracing tools work one application at a time. With debugging you often just stop your program to inspect what is going on. We wanted all the advantages of these programs without any of the disadvantages. Of course that also means with system tab you give up some things you can do with the other tools. We hope that with system tab you have at least one tool that keeps you going for a while without having to switch between these things. The tracing tools. What is nice about tracing tools is you can see what is happening while things are running. It is often faster and nicer than printf debugging because it is fully automatic printf debugging. You get a quick overview of your code flow. But the limitations are there. Of the specialized tools you can trace system calls or you can trace a function calls or you can trace the kernel. There is often limited filtering. It means that you quickly get too much information. When you try to grab it out and your system is really busy you might actually end up changing the system you are observing. With profiling the nice thing is that it also runs while your system is running. It samples and that means you get statistics about what is going on. You often can see system wide what is going on. Whether it is in a library in your kernel in your program. But it is often limited to time sampling. It would be nice if we could sample every interesting event. And often it is a large data dump you profile for a while and then you have all this data which you then analyze after the fact. Although there are really nice after-effect analyzers for the various profiling tools. And with debugging we have the full context of what we are observing. Often you can access the variables, the parameters of the functions. You can look in memory, record search, you can get a back trace. You even have conditional breakpoints. You could actually run a program and only stop it if some context is interesting. The limitation is again it stops the program under inspection. Which is nice if you want to sit down and look what is really going on. But it is not so nice if a problem only occurs on your production system. And people are actually using that system. You would rather just look at what is going on and not stop a server. And with debugging again it is one program at a time, not system wide. So with system tab we kind of made a mix. It is an obstructive, difficult word. Don't stop so you can look at your system if you don't hold your system or allow it to hold up your system too long. It should be system wide. You can monitor multiple events about synchronous. This happens now, a function call is exited and asynchronous based on time. And it is scriptable. So you can do in place filtering. So you don't have to filter after the fact. And you can collect statistics in place. It is not super powerful but powerful enough. And what we wanted was to make all this safe. So enforce that it is nonstop and obtrusive. Doesn't use too much memory. Just that will allow you to monitor things and then if it notices your monitoring takes too long, it let the system run. You can overwrite that but by default you are not allowed to do anything dangerous. Use too much memory. Use too much time. That kind of stuff. So how does it look? It is a pretty simple setup. Basically you have an event and you have a handler for that event. And that event can be something like a function entry for example in the kernel. Or it could be a specific statement in a process that get passed or it could be a timer that triggers the beginning and end of a script is an event. We did that so in the beginning event you can set something up and in the end event you can create a report from all the statistics you collected. And there is this idea of aliases. So you can alias multiple interesting events as if it is one. There will be an example of that. And a handler can do simple filtering on some conditionals in the context of the probe and then you can immediately say don't do that. I am done. And there are simple control structures for each is for the associate areas and physical variables and you can loop over those. Of course there are limits that see oh it did too many steps. It is limiting the whole system too much. And that is kind of the default. We always want that there is no noticeable impact on the system unless you explicitly specify yes I want to use megabytes of memory and I know what I am doing. And there are three kinds of variables. So you have primitives, numbers, strings, associative errors and statistical aggregates where you can store values and you can get the count, the sum, the average, the maximum, the minimum out of it. And of course there are some helper functions for actually logging something, printing, getting the time, what is the current process, that kind of stuff. Well I showed some examples. This is the simplest you invoke system tab as step. Either you give a script on the comment line or you have a script in a file and you can specify a target process if you want which actually means while this process is running this script should run. And to see what kinds of events there are you have the step minus L list and you can give it a pattern of events you want to see. And if you use minus capital L you actually get all the context variables also. So I'll show some examples. I wanted to do it without any tricks and just have Fedora 12 installed. And Fedora 12 is really nice because for a couple of contexts it really helps to have debug info available and the latest Fedora has a back port of the variable tracking assignment patches for the next GCC. So the debug info is really nice. It has system tab nicely integrated. It has the latest kernel. I had to do some tricks. So I did install all debug info beforehand because otherwise it will be a bit long. System tab will tell you if it needs specific debug info files. So I did that. I added myself to the stop dev group so I can actually that means I've elevated privileges so I can actually see everything. There are a couple of different groups you can put users in so they can only use some scripts and there is a imprivileged mode in which you can only inspect your own programs. And I actually installed the Python and Java packages from the raw height version because the version in Fedora 12 didn't have all the probes that I wanted to show. I believe that Java packages is actually going to be updated and next week it will be in Fedora 12. So let's see. Oh, help. Yeah. So to show that you can do some simple tracing, you can pro pro pro process, spin LS, show me all functions and just log the probe point where you place the probe. And then we actually run it over spin LS. Functions. Right. Yeah, sorry. The resolution is a bit. But what you can see is that it actually put probes on all the functions. This is actually a function from an included file. And nice thing is this is also how you can actually specify the probe point. So you can also specify them as they would be found in the source files. And it can do that because the debug info files have all the knowledge of how the source and the binary maps back to the source files. And to show you can also have some. Let's only print the probe function and show bits. Parameters. What did I do? I actually wrote this out. How do you? Okay, sorry. So here you see that again in a usually big thing. Sometimes it cannot find all the context. For example, here the print color indicator doesn't actually know what the parameters are. So for most functions it can actually see what arguments were given. To actually show a bit that you can also format it nicely because this is just too much output. Here we actually use a, we probe each function call. Then we print out, we indent. We add a bit of indent to the current thread. And on each return we decrease the indent and we show arrows if we're going into a function or comeback. If we probe and trace in this way you can actually see, now I can show you that it's also system wide because I forgot to actually give it a target process but I can actually run LS here. And it saw the LS running on my system. The thread indent function also prints the number of milliseconds, the process name, and the thread ID. And here you can see that, well, you can use system tap as a fancy L trace but you can actually nicely format and filter all the results. And no, let's skip that one for a minute because we're running out. This is all process based but you can also trace on probe. Let's go open, show all the files that are being opened on the system at this time. So here we're doing system-wiped probing we probe. You can see that Gdome settings is opening prog mounts a lot. And Sentmil opens prog load efforts every second. Interesting. What did I want to say? I just wanted to show that you can do system-wide tracing and get output. The problem here, of course, is you get quickly too much output who is really doing something. Screen is a bit small. The idea of this script is that you have some globals which are actually associate arrays for every virtual file system read and virtual file system write. For each executable you add the count of what they were reading and writing. And then you see a timer based probe which we actually don't use as a probe but now as outputting all the data we collected so we go through the names in the writes and then to the names in the reads. We create totals and then we print everything and then we clean up. And oh yeah, five seconds. So yeah, luckily the thing happened what I wanted to show is that even now but this is a very quiet system you already see that by outputting all this you can already have step I'll be on top of what you're tracing. To help a bit with that kind of anomalies you can also use typical variables where you every time put in the next number. What it will actually do is since we don't actually print out this global step will actually run this program and when we quit it I hope yes. It actually prints out for everything at monitors. The clock app let's it did two reads, the minimum read of a thousand bytes, the maximum is the same, the sum of course then two thousand on average it did a read of a thousand bytes. Okay. So what I showed is with system tab you can do fancy scripting of all these events at least. But we need something to observe and for that we need events that we want to observe and we need to have some context. And most of what I showed was very low level and I think we nicely covered the low level stuff. You can have timers, you can monitor processes, signals, function calls, you can monitor in the kernel and we're working on data watches that you can actually watch data being changed and we hope to have events for the performance counters in newer CPUs. And especially for C, C++ based programs the debug can force pretty good these days. So GCC provides us a lot of context. The trick is a bit to get all these techniques also used by other tools. We actually hope that all the tracing profiling and debugging tools also use some of what we're using because then you get more people hacking on more of these low level events. But what we really would like is more high level events. So we have two systems for that. So you can write tab sets. That's basically the aliases. So we have a tab set for the virtual file system and it defines probes and it aliases those to the correct kernel functions and it knows which arguments contain what and then sets up the context for you. And that's really nice but what we would really like is that kind of context not that we go through the kernel or program or something you want to observe and then somebody goes through all the source code. But we would like them in the source code where the developers write their code and they say this might be an interesting point to observe. This is an event and here's some context for an event that might be interesting if you want to know what my program is doing. And this is where you can help. You write programs. In the kernel we are making nice progress. There are actually two solutions. To be honest the kernel markers are being deprecated and the trace points are the new way to do it. A system tab can observe them both. And of course there are a lot of tab sets already for the kernel. For user space we are trying to do a trick. There's detrace which is kind of similar in spirit to a system tab. Unfortunately it's not really suitable for new Linux systems. It's basically just because the license is completely incompatible. But they had a great idea. They have user static dynamic probes. Which is macros which you put in your program to indicate here is something interesting. We had the same idea and we did it slightly different and then we thought well this is actually not nice. Because then we make people choose do you support detrace or do you support system tab. So we are actually discouraging people from using the step probe indicators and encourage people to put detrace probe macros in their source code. The implementation is completely different but if you compile your program on a slar system it will make those events visible to detrace. And if you compile on a new Linux system system tab we'll see them. Then we even provide a fake detrace script that kind of does the same things in your build. So there are actually a couple of programs like Postgres which already have those. If you configure with enable detrace those pro points get put in and you can see them through system tab. Where should you put your pro points? What are interesting events? And most of the time that is when you would add a verbose flag and say this is happening now. In those cases it's also nice to also have a static pro point there. Because with a verbose flag or if you use logging you are controlling what the user sees. Often they know better. They know they want to see something in one run or on the whole system and they only need some context. One interesting thing was when we implemented system tab itself. So of course we have minus v and it parses the scripts and compiles the scripts. So it has a couple of parses. I was actually adding the verbose output. Oh, I actually want to know how much memory this is using. Oh, I print out the memory. And the timings. Then suddenly it was, wait, but I have a better tool for this. Because if you have these pro points then we actually provide functions like the memory the process is using at that time or the time and task spent. And we actually provide the actual values. Not just the strings. So people don't have to parse out your minus verbose output. What was funny was that we added probes to each pass. So you can see stop, start phase one, the compile phase. And you can see it and that phase and you can see it start and stop the parsing phase. And what is interesting is that now we can actually run this verbose output system wide. We just run our test suite and because we have aggregates, we have statistics, you can actually run the test suite, have a little script, monitor all the stop passes and then say on average pass three takes the most memory. And the argument at that time was this particular script. Am I going to fast? No, I'm on time. Okay. I'm going a little too fast. That just means you get to ask more questions. So another way to use this is to make things that aren't C or C++ or the kernel or like scripting languages or a language runtime more visible. So for, yep. Actually now I have time for a little story. It was kind of interesting because we really wanted these detrace probes to be completely compatible so people could just put those probes in their programs and detrace would see it and system that would see it and the whole world would be happy. What was funny was that then we tried to see how Python was instrumented. And one of the sad things is that detrace is actually forked in a Apple macOS version of detrace and a Sun, now Oracle, I guess, or a Solaris detrace and neither of them actually submitted their pro points upstream. So we actually tried very hard to be compatible and now we might actually submit our version upstream and then they still look slightly different. There's not a big difference but the argument order and the precise pro point names are different on macOS and on Solaris which is kind of silly. Anyway, the nice thing in the Python case was that it was already kind of done. We just had to pick either the Solaris or the Apple version and indeed when you compile Python this way you get a couple of probes but the function entry and the function return are most interesting ones. Again we do some thread indenting to show the flow and the argument names are not that nice but I believe argument two is the function name, argument one is the file name and argument three is the source line. That must be it. Well, we can just run it. I forgot again but it doesn't really matter. Yep. I didn't even do anything in Python yet. But as you can see now you can actually see inside your Python program. You can actually see what functions are entered when, where they were defined in the source code. So one thing we don't have for Python but we do have for Java is back traces. That's sometimes a bit more work. But this means that you can now kind of instrument and look into your Python program as if it was a normal C program. At least according to system data. I probably should show that we can do the same with Java programs now. Java was also nice because at least in the case of hotspot which was the CEN VM which is now under the GPL. They actually spent a lot of time to instrument all the interesting parts. So you can now see when your Java program starts to garbage collect or when it loads a class or when it unloads a class loader. And the nice thing in that case is that you can also combine it with your other probe. So you can see when was some swapping out of memory going on in the kernel does that correspond to my garbage collection. And the nice thing is that maybe I just do one example. So in this case for example, what's nice is that since system depth can look through the whole stack you can get an interesting back trace where you actually see. So what I actually did was when the GNI Java interface gets argument length is called give me a full back trace of this point. And what you can see is that it can not only show Java functions method calls but it also knows whether it's jitted. Well, this was just an hello world program. So probably nothing is compiled but it knows whether it is interpreted. It can see some parts of the everything is interpreted. That's not fun. Otherwise it could see some of the compiler work, the actual hotspot methods like it at the start where you can see the setup till you actually hit your Java main program. So we take some questions. Wait. I must at least give you these two URLs. All my examples were a bit simple because I didn't have that much time to explain everything but there's a really nice examples page and there is a beginner's guide which explains a lot about the examples. And there's also Wiki where you can learn how to precisely put user space probes in your application. Yes. Can we use this? Yeah, can you tell us a bit more about the performance impact of doing a system tap on kernel calls as calls? So we try to keep the overhead really low. I'm not sure what our thresholds precisely are but every probe is executed in place. So there are no context switches and in principle all the locking is just dumped in the buffer which is red later. So the impact is pretty small for most things. So is there a possibility to define a priority in terms of repassation of resources between the monitoring activity and the production activity? Is there the ability, if you take the example of SNMP protocol on routers and things like this is always defined as the lowest priority. And is there the ability to define a priority in terms of the right execution of the system and the monitoring activity? I haven't really thought about it. SNMP is, that was what you were asking SNMP. It's a higher level thing. This is much more lower level. Any thoughts on using system taps to determine memory usage of kernel modules? So go, what's this kernel module using? Why is it chewing up that much memory? You can partly do that. There are actually trace points on K-Malek for example. So you could trace that and look at the color of K-Malek and map it to a module. I'm not sure there might be something like that in the examples. Probably not precisely for modules but there is something for how much K-Malek is called from where. How much of the debug info is system tap understanding? For example, how can it print a struct? Yes, you can system tap, see the debug info for the struct and you can actually dereference and get the fields of a struct. Maybe it's a little bit a silly question but would it be possible to implement this for Windows? I don't know. Most Windows has of course a lot of tracing possibilities and it's always nice to have similar interfaces. Yeah, let's say it has been 10 years since I used Windows so I don't know. Sorry. Thank you. Is there a library with a programmatic interface to it? So for example, could Python look at the profiling information of its own probes and expose that within Python? No. Is that being considered or is that interesting? Maybe something like that could be done. I believe Ruby has something like that. Has there been any discussion of adding anything like that? No, not yet. To be honest, that was one trick. The Python support was just done. Not necessarily Python but is there an in... Hi. Is there any specific support for monitoring multiple systems and looking at relationships between what different machines are doing and how they're causing thoughts on each other? I didn't completely understand. You mean monitoring other systems than the current running one? Yeah, or monitoring distributed systems running on multiple machines. We have a client server set up. So you could kind of spawn scripts on other machines. But it's not that much work out yet.
 Okay, shall we start?  Hi, I'm Mark Willard.  I currently work for Red Hat and they actually pay me to hack on system tab, which is fun.  Some of you may know me from some of my Java work and if you were thinking, oh, Mark is  going to talk about Java that's going to be a bit disappointing, there will be one example  at the end using system tab with Java, but all the rest is boring old school languages.  The talk itself is in two parts.  One about system tab itself.  Why do we need it?  What can I do?  The second part is also about system tab, but what we need besides system tab to make it  really useful.  What you could do with your programs to make system tab and observability on Knudlinix  better.  Then there will also be a little demonstration of indeed with Java how you make something  like a Java runtime more observable with system tab.  It's not really a circle, but kind of a circle.  Why do we want to have something like system tab?  If you are looking at what your system is doing, there are a lot of nice tools.  You can do profiling with just the top to see what running on your system using O profile  or the new perf tool.  You can use some tracing tools as trace to trace your system calls, L trace for library  tracing, the new F trace tool to trace kernel function calls.  Of course if you then have figured out what is going on, you can drop in to the debugger  and really inspect what is going on.  The problem is that these are all separate tools.  Profiling tools often work system wide while tracing tools work one application at a time.  With debugging you often just stop your program to inspect what is going on.  We wanted all the advantages of these programs without any of the disadvantages.  Of course that also means with system tab you give up some things you can do with the  other tools.  We hope that with system tab you have at least one tool that keeps you going for a while  without having to switch between these things.  The tracing tools.  What is nice about tracing tools is you can see what is happening while things are running.  It is often faster and nicer than printf debugging because it is fully automatic printf debugging.  You get a quick overview of your code flow.  But the limitations are there.  Of the specialized tools you can trace system calls or you can trace a function calls or  you can trace the kernel.  There is often limited filtering.  It means that you quickly get too much information.  When you try to grab it out and your system is really busy you might actually end up changing  the system you are observing.  With profiling the nice thing is that it also runs while your system is running.  It samples and that means you get statistics about what is going on.  You often can see system wide what is going on.  Whether it is in a library in your kernel in your program.  But it is often limited to time sampling.  It would be nice if we could sample every interesting event.  And often it is a large data dump you profile for a while and then you have all this data  which you then analyze after the fact.  Although there are really nice after-effect analyzers for the various profiling tools.  And with debugging we have the full context of what we are observing.  Often you can access the variables, the parameters of the functions.  You can look in memory, record search, you can get a back trace.  You even have conditional breakpoints.  You could actually run a program and only stop it if some context is interesting.  The limitation is again it stops the program under inspection.  Which is nice if you want to sit down and look what is really going on.  But it is not so nice if a problem only occurs on your production system.  And people are actually using that system.  You would rather just look at what is going on and not stop a server.  And with debugging again it is one program at a time, not system wide.  So with system tab we kind of made a mix.  It is an obstructive, difficult word.  Don't stop so you can look at your system if you don't hold your system or allow it  to hold up your system too long.  It should be system wide.  You can monitor multiple events about synchronous.  This happens now, a function call is exited and asynchronous based on time.  And it is scriptable.  So you can do in place filtering.  So you don't have to filter after the fact.  And you can collect statistics in place.  It is not super powerful but powerful enough.  And what we wanted was to make all this safe.  So enforce that it is nonstop and obtrusive.  Doesn't use too much memory.  Just that will allow you to monitor things and then if it notices your monitoring takes  too long, it let the system run.  You can overwrite that but by default you are not allowed to do anything dangerous.  Use too much memory.  Use too much time.  That kind of stuff.  So how does it look?  It is a pretty simple setup.  Basically you have an event and you have a handler for that event.  And that event can be something like a function entry for example in the kernel.  Or it could be a specific statement in a process that get passed or it could be a timer that  triggers the beginning and end of a script is an event.  We did that so in the beginning event you can set something up and in the end event  you can create a report from all the statistics you collected.  And there is this idea of aliases.  So you can alias multiple interesting events as if it is one.  There will be an example of that.  And a handler can do simple filtering on some conditionals in the context of the probe and  then you can immediately say don't do that.  I am done.  And there are simple control structures for each is for the associate areas and physical  variables and you can loop over those.  Of course there are limits that see oh it did too many steps.  It is limiting the whole system too much.  And that is kind of the default.  We always want that there is no noticeable impact on the system unless you explicitly  specify yes I want to use megabytes of memory and I know what I am doing.  And there are three kinds of variables.  So you have primitives, numbers, strings, associative errors and statistical aggregates  where you can store values and you can get the count, the sum, the average, the maximum,  the minimum out of it.  And of course there are some helper functions for actually logging something, printing,  getting the time, what is the current process, that kind of stuff.  Well I showed some examples.  This is the simplest you invoke system tab as step.  Either you give a script on the comment line or you have a script in a file and you can  specify a target process if you want which actually means while this process is running  this script should run.  And to see what kinds of events there are you have the step minus L list and you can  give it a pattern of events you want to see.  And if you use minus capital L you actually get all the context variables also.  So I'll show some examples.  I wanted to do it without any tricks and just have Fedora 12 installed.  And Fedora 12 is really nice because for a couple of contexts it really helps to have  debug info available and the latest Fedora has a back port of the variable tracking assignment  patches for the next GCC.  So the debug info is really nice.  It has system tab nicely integrated.  It has the latest kernel.  I had to do some tricks.  So I did install all debug info beforehand because otherwise it will be a bit long.  System tab will tell you if it needs specific debug info files.  So I did that.  I added myself to the stop dev group so I can actually that means I've elevated privileges  so I can actually see everything.  There are a couple of different groups you can put users in so they can only use some  scripts and there is a imprivileged mode in which you can only inspect your own programs.  And I actually installed the Python and Java packages from the raw height version because  the version in Fedora 12 didn't have all the probes that I wanted to show.  I believe that Java packages is actually going to be updated and next week it will be in  Fedora 12.  So let's see.  Oh, help.  Yeah.  So to show that you can do some simple tracing, you can pro pro pro process, spin LS, show  me all functions and just log the probe point where you place the probe.  And then we actually run it over spin LS.  Functions.  Right.  Yeah, sorry.  The resolution is a bit.  But what you can see is that it actually put probes on all the functions.  This is actually a function from an included file.  And nice thing is this is also how you can actually specify the probe point.  So you can also specify them as they would be found in the source files.  And it can do that because the debug info files have all the knowledge of how the source  and the binary maps back to the source files.  And to show you can also have some.  Let's only print the probe function and show bits.  Parameters.  What did I do?  I actually wrote this out.  How do you?  Okay, sorry.  So here you see that again in a usually big thing.  Sometimes it cannot find all the context.  For example, here the print color indicator doesn't actually know what the parameters  are.  So for most functions it can actually see what arguments were given.  To actually show a bit that you can also format it nicely because this is just too much output.  Here we actually use a, we probe each function call.  Then we print out, we indent.  We add a bit of indent to the current thread.  And on each return we decrease the indent and we show arrows if we're going into a function  or comeback.  If we probe and trace in this way you can actually see, now I can show you that it's  also system wide because I forgot to actually give it a target process but I can actually  run LS here.  And it saw the LS running on my system.  The thread indent function also prints the number of milliseconds, the process name,  and the thread ID.  And here you can see that, well, you can use system tap as a fancy L trace but you can  actually nicely format and filter all the results.  And no, let's skip that one for a minute because we're running out.  This is all process based but you can also trace on probe.  Let's go open, show all the files that are being opened on the system at this time.  So here we're doing system-wiped probing we probe.  You can see that Gdome settings is opening prog mounts a lot.  And Sentmil opens prog load efforts every second.  Interesting.  What did I want to say?  I just wanted to show that you can do system-wide tracing and get output.  The problem here, of course, is you get quickly too much output who is really doing something.  Screen is a bit small.  The idea of this script is that you have some globals which are actually associate arrays  for every virtual file system read and virtual file system write.  For each executable you add the count of what they were reading and writing.  And then you see a timer based probe which we actually don't use as a probe but now as  outputting all the data we collected so we go through the names in the writes and then  to the names in the reads.  We create totals and then we print everything and then we clean up.  And oh yeah, five seconds.  So yeah, luckily the thing happened what I wanted to show is that even now but this  is a very quiet system you already see that by outputting all this you can already have  step I'll be on top of what you're tracing.  To help a bit with that kind of anomalies you can also use  typical variables where you every time put in the next number.  What it will actually do is since we don't actually print out this global step will actually  run this program and when we quit it I hope yes.  It actually prints out for everything at monitors.  The clock app let's it did two reads, the minimum read of a thousand bytes, the maximum  is the same, the sum of course then two thousand on average it did a read of a thousand bytes.  Okay.  So what I showed is with system tab you can do fancy scripting of all these events at  least.  But we need something to observe and for that we need events that we want to observe and  we need to have some context.  And most of what I showed was very low level and I think we nicely covered the low level  stuff.  You can have timers, you can monitor processes, signals, function calls, you can monitor in  the kernel and we're working on data watches that you can actually watch data being changed  and we hope to have events for the performance counters in newer CPUs.  And especially for C, C++ based programs the debug can force pretty good these days.  So GCC provides us a lot of context.  The trick is a bit to get all these techniques also used by other tools.  We actually hope that all the tracing profiling and debugging tools also use some of what  we're using because then you get more people hacking on more of these low level events.  But what we really would like is more high level events.  So we have two systems for that.  So you can write tab sets.  That's basically the aliases.  So we have a tab set for the virtual file system and it defines probes and it aliases  those to the correct kernel functions and it knows which arguments contain what and  then sets up the context for you.  And that's really nice but what we would really like is that kind of context not that we  go through the kernel or program or something you want to observe and then somebody goes  through all the source code.  But we would like them in the source code where the developers write their code and  they say this might be an interesting point to observe.  This is an event and here's some context for an event that might be interesting if you  want to know what my program is doing.  And this is where you can help.  You write programs.  In the kernel we are making nice progress.  There are actually two solutions.  To be honest the kernel markers are being deprecated and the trace points are the new  way to do it.  A system tab can observe them both.  And of course there are a lot of tab sets already for the kernel.  For user space we are trying to do a trick.  There's detrace which is kind of similar in spirit to a system tab.  Unfortunately it's not really suitable for new Linux systems.  It's basically just because the license is completely incompatible.  But they had a great idea.  They have user static dynamic probes.  Which is macros which you put in your program to indicate here is something interesting.  We had the same idea and we did it slightly different and then we thought well this is  actually not nice.  Because then we make people choose do you support detrace or do you support system tab.  So we are actually discouraging people from using the step probe indicators and encourage  people to put detrace probe macros in their source code.  The implementation is completely different but if you compile your program on a slar  system it will make those events visible to detrace.  And if you compile on a new Linux system system tab we'll see them.  Then we even provide a fake detrace script that kind of does the same things in your  build.  So there are actually a couple of programs like Postgres which already have those.  If you configure with enable detrace those pro points get put in and you can see them  through system tab.  Where should you put your pro points?  What are interesting events?  And most of the time that is when you would add a verbose flag and say this is happening  now.  In those cases it's also nice to also have a static pro point there.  Because with a verbose flag or if you use logging you are controlling what the user  sees.  Often they know better.  They know they want to see something in one run or on the whole system and they only need  some context.  One interesting thing was when we implemented system tab itself.  So of course we have minus v and it parses the scripts and compiles the scripts.  So it has a couple of parses.  I was actually adding the verbose output.  Oh, I actually want to know how much memory this is using.  Oh, I print out the memory.  And the timings.  Then suddenly it was, wait, but I have a better tool for this.  Because if you have these pro points then we actually provide functions like the memory  the process is using at that time or the time and task spent.  And we actually provide the actual values.  Not just the strings.  So people don't have to parse out your minus verbose output.  What was funny was that we added probes to each pass.  So you can see stop, start phase one, the compile phase.  And you can see it and that phase and you can see it start and stop the parsing phase.  And what is interesting is that now we can actually run this verbose output system wide.  We just run our test suite and because we have aggregates, we have statistics, you can  actually run the test suite, have a little script, monitor all the stop passes and then  say on average pass three takes the most memory.  And the argument at that time was this particular script.  Am I going to fast?  No, I'm on time.  Okay.  I'm going a little too fast.  That just means you get to ask more questions.  So another way to use this is to make things that aren't C or C++ or the kernel or like  scripting languages or a language runtime more visible.  So for, yep.  Actually now I have time for a little story.  It was kind of interesting because we really wanted these detrace probes to be completely  compatible so people could just put those probes in their programs and detrace would  see it and system that would see it and the whole world would be happy.  What was funny was that then we tried to see how Python was instrumented.  And one of the sad things is that detrace is actually forked in a Apple macOS version  of detrace and a Sun, now Oracle, I guess, or a Solaris detrace and neither of them actually  submitted their pro points upstream.  So we actually tried very hard to be compatible and now we might actually submit our version  upstream and then they still look slightly different.  There's not a big difference but the argument order and the precise pro point names are  different on macOS and on Solaris which is kind of silly.  Anyway, the nice thing in the Python case was that it was already kind of done.  We just had to pick either the Solaris or the Apple version and indeed when you compile  Python this way you get a couple of probes but the function entry and the function return  are most interesting ones.  Again we do some thread indenting to show the flow and the argument names are not that  nice but I believe argument two is the function name, argument one is the file name and argument  three is the source line.  That must be it.  Well, we can just run it.  I forgot again but it doesn't really matter.  Yep.  I didn't even do anything in Python yet.  But as you can see now you can actually see inside your Python program.  You can actually see what functions are entered when, where they were defined in the source  code.  So one thing we don't have for Python but we do have for Java is back traces.  That's sometimes a bit more work.  But this means that you can now kind of instrument and look into your Python program as if it  was a normal C program.  At least according to system data.  I probably should show that we can do the same with Java programs now.  Java was also nice because at least in the case of hotspot which was the CEN VM which  is now under the GPL.  They actually spent a lot of time to instrument all the interesting parts.  So you can now see when your Java program starts to garbage collect or when it loads  a class or when it unloads a class loader.  And the nice thing in that case is that you can also combine it with your other probe.  So you can see when was some swapping out of memory going on in the kernel does that  correspond to my garbage collection.  And the nice thing is that maybe I just do one example.  So in this case for example, what's nice is that since system depth can look through the  whole stack you can get an interesting back trace where you actually see.  So what I actually did was when the GNI Java interface gets argument length is called give  me a full back trace of this point.  And what you can see is that it can not only show Java functions method calls but it also  knows whether it's jitted.  Well, this was just an hello world program.  So probably nothing is compiled but it knows whether it is interpreted.  It can see some parts of the everything is interpreted.  That's not fun.  Otherwise it could see some of the compiler work, the actual hotspot methods like it at  the start where you can see the setup till you actually hit your Java main program.  So we take some questions.  Wait.  I must at least give you these two URLs.  All my examples were a bit simple because I didn't have that much time to explain everything  but there's a really nice examples page and there is a beginner's guide which explains  a lot about the examples.  And there's also Wiki where you can learn how to precisely put user space probes in  your application.  Yes.  Can we use this?  Yeah, can you tell us a bit more about the performance impact of doing a system tap on  kernel calls as calls?  So we try to keep the overhead really low.  I'm not sure what our thresholds precisely are but every probe is executed in place.  So there are no context switches and in principle all the locking is just dumped in the buffer  which is red later.  So the impact is pretty small for most things.  So is there a possibility to define a priority in terms of repassation of resources between  the monitoring activity and the production activity?  Is there the ability, if you take the example of SNMP protocol on routers and things like  this is always defined as the lowest priority.  And is there the ability to define a priority in terms of the right execution of the system  and the monitoring activity?  I haven't really thought about it.  SNMP is, that was what you were asking SNMP.  It's a higher level thing.  This is much more lower level.  Any thoughts on using system taps to determine memory usage of kernel modules?  So go, what's this kernel module using?  Why is it chewing up that much memory?  You can partly do that.  There are actually trace points on K-Malek for example.  So you could trace that and look at the color of K-Malek and map it to a module.  I'm not sure there might be something like that in the examples.  Probably not precisely for modules but there is something for how much K-Malek is called  from where.  How much of the debug info is system tap understanding?  For example, how can it print a struct?  Yes, you can system tap, see the debug info for the struct and you can actually dereference  and get the fields of a struct.  Maybe it's a little bit a silly question but would it be possible to implement this for  Windows?  I don't know.  Most Windows has of course a lot of tracing possibilities and it's always nice to have  similar interfaces.  Yeah, let's say it has been 10 years since I used Windows so I don't know.  Sorry.  Thank you.  Is there a library with a programmatic interface to it?  So for example, could Python look at the profiling information of its own probes and expose that  within Python?  No.  Is that being considered or is that interesting?  Maybe something like that could be done.  I believe Ruby has something like that.  Has there been any discussion of adding anything like that?  No, not yet.  To be honest, that was one trick.  The Python support was just done.  Not necessarily Python but is there an in...  Hi.  Is there any specific support for monitoring multiple systems and looking at relationships  between what different machines are doing and how they're causing thoughts on each other?  I didn't completely understand.  You mean monitoring other systems than the current running one?  Yeah, or monitoring distributed systems running on multiple machines.  We have a client server set up.  So you could kind of spawn scripts on other machines.  But it's not that much work out yet.
WhatismysystemdoingFullSystemObservabilitywithSystemTap_0_131.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_132_179.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_180_8027.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_8028_23795.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_23796_23867.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_23868_23951.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_23952_23999.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_24000_25331.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_25332_25547.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_25548_27455.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_27456_27779.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_27780_28127.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_28128_28187.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_28188_28379.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_28380_29195.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_29196_29483.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_29484_30563.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_30564_32987.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_32988_33035.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_33036_33095.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_33096_33191.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_33192_33251.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_33252_33335.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_33336_33383.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_33384_33431.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_33432_33515.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_33516_33563.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_33564_33611.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_33612_33659.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_33660_33707.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_33708_33767.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_33768_34631.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_34632_34811.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_34812_35639.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_35640_36107.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_36108_37127.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_37128_37559.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_37560_39035.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_39036_39095.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_39096_45767.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_45768_47999.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_48000_54287.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_54288_54467.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_54468_58079.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_58080_62387.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_62388_62555.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_62556_62687.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_62688_62999.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_63000_63263.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_63264_63647.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_63648_64127.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_64128_64235.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_64236_64283.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_64284_64871.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_64872_64919.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_64920_65111.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_65112_66179.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_66180_68903.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_68904_69215.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_69216_69263.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_69264_70223.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_70224_72947.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_72948_73115.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_73116_74255.png WhatismysystemdoingFullSystemObservabilitywithSystemTap_74256_74372.png
Okay, shall we start? Hi, I'm Mark Willard. I currently work for Red Hat and they actually pay me to hack on system tab, which is fun. Some of you may know me from some of my Java work and if you were thinking, oh, Mark is going to talk about Java that's going to be a bit disappointing, there will be one example at the end using system tab with Java, but all the rest is boring old school languages. The talk itself is in two parts. One about system tab itself. Why do we need it? What can I do? The second part is also about system tab, but what we need besides system tab to make it really useful. What you could do with your programs to make system tab and observability on Knudlinix better. Then there will also be a little demonstration of indeed with Java how you make something like a Java runtime more observable with system tab. It's not really a circle, but kind of a circle. Why do we want to have something like system tab? If you are looking at what your system is doing, there are a lot of nice tools. You can do profiling with just the top to see what running on your system using O profile or the new perf tool. You can use some tracing tools as trace to trace your system calls, L trace for library tracing, the new F trace tool to trace kernel function calls. Of course if you then have figured out what is going on, you can drop in to the debugger and really inspect what is going on. The problem is that these are all separate tools. Profiling tools often work system wide while tracing tools work one application at a time. With debugging you often just stop your program to inspect what is going on. We wanted all the advantages of these programs without any of the disadvantages. Of course that also means with system tab you give up some things you can do with the other tools. We hope that with system tab you have at least one tool that keeps you going for a while without having to switch between these things. The tracing tools. What is nice about tracing tools is you can see what is happening while things are running. It is often faster and nicer than printf debugging because it is fully automatic printf debugging. You get a quick overview of your code flow. But the limitations are there. Of the specialized tools you can trace system calls or you can trace a function calls or you can trace the kernel. There is often limited filtering. It means that you quickly get too much information. When you try to grab it out and your system is really busy you might actually end up changing the system you are observing. With profiling the nice thing is that it also runs while your system is running. It samples and that means you get statistics about what is going on. You often can see system wide what is going on. Whether it is in a library in your kernel in your program. But it is often limited to time sampling. It would be nice if we could sample every interesting event. And often it is a large data dump you profile for a while and then you have all this data which you then analyze after the fact. Although there are really nice after-effect analyzers for the various profiling tools. And with debugging we have the full context of what we are observing. Often you can access the variables, the parameters of the functions. You can look in memory, record search, you can get a back trace. You even have conditional breakpoints. You could actually run a program and only stop it if some context is interesting. The limitation is again it stops the program under inspection. Which is nice if you want to sit down and look what is really going on. But it is not so nice if a problem only occurs on your production system. And people are actually using that system. You would rather just look at what is going on and not stop a server. And with debugging again it is one program at a time, not system wide. So with system tab we kind of made a mix. It is an obstructive, difficult word. Don't stop so you can look at your system if you don't hold your system or allow it to hold up your system too long. It should be system wide. You can monitor multiple events about synchronous. This happens now, a function call is exited and asynchronous based on time. And it is scriptable. So you can do in place filtering. So you don't have to filter after the fact. And you can collect statistics in place. It is not super powerful but powerful enough. And what we wanted was to make all this safe. So enforce that it is nonstop and obtrusive. Doesn't use too much memory. Just that will allow you to monitor things and then if it notices your monitoring takes too long, it let the system run. You can overwrite that but by default you are not allowed to do anything dangerous. Use too much memory. Use too much time. That kind of stuff. So how does it look? It is a pretty simple setup. Basically you have an event and you have a handler for that event. And that event can be something like a function entry for example in the kernel. Or it could be a specific statement in a process that get passed or it could be a timer that triggers the beginning and end of a script is an event. We did that so in the beginning event you can set something up and in the end event you can create a report from all the statistics you collected. And there is this idea of aliases. So you can alias multiple interesting events as if it is one. There will be an example of that. And a handler can do simple filtering on some conditionals in the context of the probe and then you can immediately say don't do that. I am done. And there are simple control structures for each is for the associate areas and physical variables and you can loop over those. Of course there are limits that see oh it did too many steps. It is limiting the whole system too much. And that is kind of the default. We always want that there is no noticeable impact on the system unless you explicitly specify yes I want to use megabytes of memory and I know what I am doing. And there are three kinds of variables. So you have primitives, numbers, strings, associative errors and statistical aggregates where you can store values and you can get the count, the sum, the average, the maximum, the minimum out of it. And of course there are some helper functions for actually logging something, printing, getting the time, what is the current process, that kind of stuff. Well I showed some examples. This is the simplest you invoke system tab as step. Either you give a script on the comment line or you have a script in a file and you can specify a target process if you want which actually means while this process is running this script should run. And to see what kinds of events there are you have the step minus L list and you can give it a pattern of events you want to see. And if you use minus capital L you actually get all the context variables also. So I'll show some examples. I wanted to do it without any tricks and just have Fedora 12 installed. And Fedora 12 is really nice because for a couple of contexts it really helps to have debug info available and the latest Fedora has a back port of the variable tracking assignment patches for the next GCC. So the debug info is really nice. It has system tab nicely integrated. It has the latest kernel. I had to do some tricks. So I did install all debug info beforehand because otherwise it will be a bit long. System tab will tell you if it needs specific debug info files. So I did that. I added myself to the stop dev group so I can actually that means I've elevated privileges so I can actually see everything. There are a couple of different groups you can put users in so they can only use some scripts and there is a imprivileged mode in which you can only inspect your own programs. And I actually installed the Python and Java packages from the raw height version because the version in Fedora 12 didn't have all the probes that I wanted to show. I believe that Java packages is actually going to be updated and next week it will be in Fedora 12. So let's see. Oh, help. Yeah. So to show that you can do some simple tracing, you can pro pro pro process, spin LS, show me all functions and just log the probe point where you place the probe. And then we actually run it over spin LS. Functions. Right. Yeah, sorry. The resolution is a bit. But what you can see is that it actually put probes on all the functions. This is actually a function from an included file. And nice thing is this is also how you can actually specify the probe point. So you can also specify them as they would be found in the source files. And it can do that because the debug info files have all the knowledge of how the source and the binary maps back to the source files. And to show you can also have some. Let's only print the probe function and show bits. Parameters. What did I do? I actually wrote this out. How do you? Okay, sorry. So here you see that again in a usually big thing. Sometimes it cannot find all the context. For example, here the print color indicator doesn't actually know what the parameters are. So for most functions it can actually see what arguments were given. To actually show a bit that you can also format it nicely because this is just too much output. Here we actually use a, we probe each function call. Then we print out, we indent. We add a bit of indent to the current thread. And on each return we decrease the indent and we show arrows if we're going into a function or comeback. If we probe and trace in this way you can actually see, now I can show you that it's also system wide because I forgot to actually give it a target process but I can actually run LS here. And it saw the LS running on my system. The thread indent function also prints the number of milliseconds, the process name, and the thread ID. And here you can see that, well, you can use system tap as a fancy L trace but you can actually nicely format and filter all the results. And no, let's skip that one for a minute because we're running out. This is all process based but you can also trace on probe. Let's go open, show all the files that are being opened on the system at this time. So here we're doing system-wiped probing we probe. You can see that Gdome settings is opening prog mounts a lot. And Sentmil opens prog load efforts every second. Interesting. What did I want to say? I just wanted to show that you can do system-wide tracing and get output. The problem here, of course, is you get quickly too much output who is really doing something. Screen is a bit small. The idea of this script is that you have some globals which are actually associate arrays for every virtual file system read and virtual file system write. For each executable you add the count of what they were reading and writing. And then you see a timer based probe which we actually don't use as a probe but now as outputting all the data we collected so we go through the names in the writes and then to the names in the reads. We create totals and then we print everything and then we clean up. And oh yeah, five seconds. So yeah, luckily the thing happened what I wanted to show is that even now but this is a very quiet system you already see that by outputting all this you can already have step I'll be on top of what you're tracing. To help a bit with that kind of anomalies you can also use typical variables where you every time put in the next number. What it will actually do is since we don't actually print out this global step will actually run this program and when we quit it I hope yes. It actually prints out for everything at monitors. The clock app let's it did two reads, the minimum read of a thousand bytes, the maximum is the same, the sum of course then two thousand on average it did a read of a thousand bytes. Okay. So what I showed is with system tab you can do fancy scripting of all these events at least. But we need something to observe and for that we need events that we want to observe and we need to have some context. And most of what I showed was very low level and I think we nicely covered the low level stuff. You can have timers, you can monitor processes, signals, function calls, you can monitor in the kernel and we're working on data watches that you can actually watch data being changed and we hope to have events for the performance counters in newer CPUs. And especially for C, C++ based programs the debug can force pretty good these days. So GCC provides us a lot of context. The trick is a bit to get all these techniques also used by other tools. We actually hope that all the tracing profiling and debugging tools also use some of what we're using because then you get more people hacking on more of these low level events. But what we really would like is more high level events. So we have two systems for that. So you can write tab sets. That's basically the aliases. So we have a tab set for the virtual file system and it defines probes and it aliases those to the correct kernel functions and it knows which arguments contain what and then sets up the context for you. And that's really nice but what we would really like is that kind of context not that we go through the kernel or program or something you want to observe and then somebody goes through all the source code. But we would like them in the source code where the developers write their code and they say this might be an interesting point to observe. This is an event and here's some context for an event that might be interesting if you want to know what my program is doing. And this is where you can help. You write programs. In the kernel we are making nice progress. There are actually two solutions. To be honest the kernel markers are being deprecated and the trace points are the new way to do it. A system tab can observe them both. And of course there are a lot of tab sets already for the kernel. For user space we are trying to do a trick. There's detrace which is kind of similar in spirit to a system tab. Unfortunately it's not really suitable for new Linux systems. It's basically just because the license is completely incompatible. But they had a great idea. They have user static dynamic probes. Which is macros which you put in your program to indicate here is something interesting. We had the same idea and we did it slightly different and then we thought well this is actually not nice. Because then we make people choose do you support detrace or do you support system tab. So we are actually discouraging people from using the step probe indicators and encourage people to put detrace probe macros in their source code. The implementation is completely different but if you compile your program on a slar system it will make those events visible to detrace. And if you compile on a new Linux system system tab we'll see them. Then we even provide a fake detrace script that kind of does the same things in your build. So there are actually a couple of programs like Postgres which already have those. If you configure with enable detrace those pro points get put in and you can see them through system tab. Where should you put your pro points? What are interesting events? And most of the time that is when you would add a verbose flag and say this is happening now. In those cases it's also nice to also have a static pro point there. Because with a verbose flag or if you use logging you are controlling what the user sees. Often they know better. They know they want to see something in one run or on the whole system and they only need some context. One interesting thing was when we implemented system tab itself. So of course we have minus v and it parses the scripts and compiles the scripts. So it has a couple of parses. I was actually adding the verbose output. Oh, I actually want to know how much memory this is using. Oh, I print out the memory. And the timings. Then suddenly it was, wait, but I have a better tool for this. Because if you have these pro points then we actually provide functions like the memory the process is using at that time or the time and task spent. And we actually provide the actual values. Not just the strings. So people don't have to parse out your minus verbose output. What was funny was that we added probes to each pass. So you can see stop, start phase one, the compile phase. And you can see it and that phase and you can see it start and stop the parsing phase. And what is interesting is that now we can actually run this verbose output system wide. We just run our test suite and because we have aggregates, we have statistics, you can actually run the test suite, have a little script, monitor all the stop passes and then say on average pass three takes the most memory. And the argument at that time was this particular script. Am I going to fast? No, I'm on time. Okay. I'm going a little too fast. That just means you get to ask more questions. So another way to use this is to make things that aren't C or C++ or the kernel or like scripting languages or a language runtime more visible. So for, yep. Actually now I have time for a little story. It was kind of interesting because we really wanted these detrace probes to be completely compatible so people could just put those probes in their programs and detrace would see it and system that would see it and the whole world would be happy. What was funny was that then we tried to see how Python was instrumented. And one of the sad things is that detrace is actually forked in a Apple macOS version of detrace and a Sun, now Oracle, I guess, or a Solaris detrace and neither of them actually submitted their pro points upstream. So we actually tried very hard to be compatible and now we might actually submit our version upstream and then they still look slightly different. There's not a big difference but the argument order and the precise pro point names are different on macOS and on Solaris which is kind of silly. Anyway, the nice thing in the Python case was that it was already kind of done. We just had to pick either the Solaris or the Apple version and indeed when you compile Python this way you get a couple of probes but the function entry and the function return are most interesting ones. Again we do some thread indenting to show the flow and the argument names are not that nice but I believe argument two is the function name, argument one is the file name and argument three is the source line. That must be it. Well, we can just run it. I forgot again but it doesn't really matter. Yep. I didn't even do anything in Python yet. But as you can see now you can actually see inside your Python program. You can actually see what functions are entered when, where they were defined in the source code. So one thing we don't have for Python but we do have for Java is back traces. That's sometimes a bit more work. But this means that you can now kind of instrument and look into your Python program as if it was a normal C program. At least according to system data. I probably should show that we can do the same with Java programs now. Java was also nice because at least in the case of hotspot which was the CEN VM which is now under the GPL. They actually spent a lot of time to instrument all the interesting parts. So you can now see when your Java program starts to garbage collect or when it loads a class or when it unloads a class loader. And the nice thing in that case is that you can also combine it with your other probe. So you can see when was some swapping out of memory going on in the kernel does that correspond to my garbage collection. And the nice thing is that maybe I just do one example. So in this case for example, what's nice is that since system depth can look through the whole stack you can get an interesting back trace where you actually see. So what I actually did was when the GNI Java interface gets argument length is called give me a full back trace of this point. And what you can see is that it can not only show Java functions method calls but it also knows whether it's jitted. Well, this was just an hello world program. So probably nothing is compiled but it knows whether it is interpreted. It can see some parts of the everything is interpreted. That's not fun. Otherwise it could see some of the compiler work, the actual hotspot methods like it at the start where you can see the setup till you actually hit your Java main program. So we take some questions. Wait. I must at least give you these two URLs. All my examples were a bit simple because I didn't have that much time to explain everything but there's a really nice examples page and there is a beginner's guide which explains a lot about the examples. And there's also Wiki where you can learn how to precisely put user space probes in your application. Yes. Can we use this? Yeah, can you tell us a bit more about the performance impact of doing a system tap on kernel calls as calls? So we try to keep the overhead really low. I'm not sure what our thresholds precisely are but every probe is executed in place. So there are no context switches and in principle all the locking is just dumped in the buffer which is red later. So the impact is pretty small for most things. So is there a possibility to define a priority in terms of repassation of resources between the monitoring activity and the production activity? Is there the ability, if you take the example of SNMP protocol on routers and things like this is always defined as the lowest priority. And is there the ability to define a priority in terms of the right execution of the system and the monitoring activity? I haven't really thought about it. SNMP is, that was what you were asking SNMP. It's a higher level thing. This is much more lower level. Any thoughts on using system taps to determine memory usage of kernel modules? So go, what's this kernel module using? Why is it chewing up that much memory? You can partly do that. There are actually trace points on K-Malek for example. So you could trace that and look at the color of K-Malek and map it to a module. I'm not sure there might be something like that in the examples. Probably not precisely for modules but there is something for how much K-Malek is called from where. How much of the debug info is system tap understanding? For example, how can it print a struct? Yes, you can system tap, see the debug info for the struct and you can actually dereference and get the fields of a struct. Maybe it's a little bit a silly question but would it be possible to implement this for Windows? I don't know. Most Windows has of course a lot of tracing possibilities and it's always nice to have similar interfaces. Yeah, let's say it has been 10 years since I used Windows so I don't know. Sorry. Thank you. Is there a library with a programmatic interface to it? So for example, could Python look at the profiling information of its own probes and expose that within Python? No. Is that being considered or is that interesting? Maybe something like that could be done. I believe Ruby has something like that. Has there been any discussion of adding anything like that? No, not yet. To be honest, that was one trick. The Python support was just done. Not necessarily Python but is there an in... Hi. Is there any specific support for monitoring multiple systems and looking at relationships between what different machines are doing and how they're causing thoughts on each other? I didn't completely understand. You mean monitoring other systems than the current running one? Yeah, or monitoring distributed systems running on multiple machines. We have a client server set up. So you could kind of spawn scripts on other machines. But it's not that much work out yet.
and observability on Knudlinix better. Then there with Java how you make something like a Java are looking at what your system is doing, there system using O profile or the new perf tool. You system is really busy you might actually end up while your system is running. It samples and that means you get statistics about what is going on. Whether it is in a library in your kernel in your data dump you profile for a while and then you access the variables, the parameters of the you can get a back trace. You even have you don't have to filter after the fact. And you does it look? It is a pretty simple setup. actually see everything. There are a couple of simple tracing, you can pro pro pro process, spin you can see is that it actually put probes on all the functions. This is actually a function from probe and trace in this way you can actually see, through the names in the writes and then to the can have timers, you can monitor processes, high level events. So we have two systems for might be an interesting point to observe. This is solutions. To be honest the kernel markers are And of course there are a lot of tab sets already to also have a static pro point there. Because one run or on the whole system and they only need compiles the scripts. So it has a couple of pro points then we actually provide functions little script, monitor all the stop passes and another way to use this is to make things that world would be happy. What was funny was that and look into your Python program as if it was a a full back trace of this point. And what you can put user space probes in your application. Yes. debug info for the struct and you can actually been any discussion of adding anything like that? specific support for monitoring multiple systems