Is my system fast?
Becker, Frank
Computer systems are complex. Most software applications are distributed and expected to scale. That does not make them any simpler. Further, there is the real world all of that is expected to work in. To analyze performance of such a system isn't less complex. Luckily, there is help. The Open Source universe is full of excellent tools that can help. The talk introduces a few of them. The talk walks down the path of comparing two Linux servers that look pretty much the same from the specs. Introducing test-tools like stress-ng, sysjitter, fio, packetdrill, and friends it is shown how to get a good overview of the systems CPU, memory, disk, and network performance. Since benchmark results are usually time variant a few basic statistical terms are refreshed and used for visualizing those results employing different chart types. The field of performance analysis is wide. There isn't more time in 45 minutes than just to scratch the surface. At the end of the talk the audience will have a few pointers of how to start evaluating their own Linux systems to provide the right performance for their applications.
 All right. So actually, what a conference to have Leonard as kind of an opening act to my talk. So this is going to be kind of a general talk about system performance with systems, I mean Linux systems, and to be more specific, it's basically about Linux service. Before I start, a little bit about myself. I actually like computers a lot. I like them fast, and I've been lucky enough to always find employers who kind of provide me those. So I used to work for AMD in Dresden, which is about two hours south of where we are right now. And I was there when we had the first one gigahertz processor or x86 processor, I should say. I was there when AMD introduced the x86 46 bit extensions. I was quite interested in time. And since five and a half years, I'm now with Amazon. And what we do in Dresden is basically all the kernel and hypervisors that Amazon runs on their servers. And I truly can tell you a lot. And the kernel and hypervisors is coming from a group that has multiple offices, but a big one is in Dresden. I promised to put this up so we are hiring. If you're interested in this, have a look for talk tests. Several people here. All right. But for the talk, talk. So I agree the title is a little silly because what does it really mean to be fast? That guy here, he was fast. So about two weeks ago in Berlin, there was a marathon and the guy run it in a little bit over two hours. So two hours, one minute or so. Cut a new world record. And the interesting thing is, you know, you have one number and you can compare them. And that's really it. You know, you say go and you measure a four o'clock time. And when the guy comes back in after the 42 kilometers, you have a number and you can compare those things. Computers are sometimes, unfortunately, why not, unfortunately, but they are way more complex than that. And there are way more things you can measure. So let's have a look how a desktop is presenting you basically an idea of how fast it is. So GNOME has this system monitor, they call it. And on top you see CPU usage. Over time, you see memory usage and network usage over time. I mean, not a big surprise, but I probably would also add your hard drive. That's also an interesting number to see how much this guy we have. I guess most of you with SSH into a server and run something like top or H-top, which gives you a little bit more information. One of the numbers you probably cannot really read here is load. I remember back at the AMD days, so I basically started in the IT infrastructure team there and we had a weekly ops meeting for database service and all those things. And one of the numbers we always looked at was load. And I think every other week or so there started a big discussion what this figure actually means and if it's useful to look at it or not. And we repeated this over and over again. And that is basically the slightest for my co-workers. So from the documentation in the file that is implemented, it basically is the number of running processes plus the number of uninterruptible processes in the run queue of the news kernel. And since this number goes up and down like crazy, you want to have an average over one five and 50 minutes and that was an exponential decay. It's totally interesting to look of the, to check out the history of, you know, where load actually came from. And here's a quote from the guy, Matthias Oeris, who actually added this uninterruptible processes, which at the time mainly meant the skyro. And you can read all of that in a blog post from Brandon Gregg, who was a hero when it comes to performance measurements. But I will quote him a couple of times in this talk. The thing to remember is load is a very good indicator for humans to look at the system and see what it's doing. But if you start to think about it for longer than 10 seconds, what this really means and if this is actually telling you anything, then look for something else. Before we dive into all of that a little bit deeper, I'd like to come to a common understanding of some statistics and chart types. So the first thing is the mean value versus the median. Mean basically means you have a bunch of numbers and the sum of this and divide by the count of those. Median is you line them all up, order them and then pick the one in the middle. Interestingly, the median is the one you should choose for performance analysis in many cases, or probably most cases, because it's way more robust. You will see that in a second. Another thing you should be familiar with because that is extremely helpful is things like mean, standard deviation, percentile. So that is the probability density function of the normal distribution. Percentile basically means again you order all your measurements in a long row, order them by size or by number and then you would have the 50% percentile right in the middle. For this normal distribution, the mean and the median are exactly at the same point, right in the middle and well, to sigma you have the 79. whatever percentile and that means that basically all your measurements or that 97% of your measurements are below this number. So why is it important? Just imagine you measure over time some APIs and some web API, you know, which you query and what you're interested in, how long does it take to get a response latency for that thing. The green thing is the median and you see that this number is pretty robust, right? So if you are actually, if you want to eliminate outliers from your matrix, that is really the thing to use. The light blue one is the mean value where you see this is indicating a little bit and if you serve this service to real users, you're probably interested in the percentile that I think is the 95% percentile here and you really see how this goes up and that still means that 5% of your users have a worse experience at this point, which is not good. All right, another thing to always keep in mind when you do performance measurements is cashers. So here's a guy who measured latency, I believe, for his hard drive and what you see is in this chart in the histogram here, it goes up then it goes down and it goes up again and well, by the title you probably will have guessed. Here you basically measure cash, so it might be you really measure just access to memory here, where that probably is where your system has to really scrape things off the hard drive. And so of course the first comment on this blog was to use DirectIO to basically bypass the cash. For latency, I was expecting that this is hard to read here. Things to keep in mind. So if you want to check that out, there are many more examples on this gist. If you say that one CPU cycle, which takes about 0.3 nanoseconds, would be one second, right? Then a level 3 cash access would be something like 43 seconds, which is quite some time compared to a one cash access, which is three seconds. However, if you go further and you talk to real memory, or you have to talk to real memory, that already would be six minutes. I mean, you can read all of that here, but another number, a TCP retransmit, which one, two, three seconds in this perspective would already be something like a hundred years. And a physical system reboot, let's say this takes five minutes, and that's why I put this up here. That means at this time, if you would take this time back in the world history, that was when the Sahara still was in the ocean. So that is something I think you should remember from time to time that really cashes are a very good idea. And well, if you have too much network I.O. that already adds a lot of latency. So I took this table out of this book to be honest with you. You should be reading this book and not listening to me. Brian Gregg is really one of the guys to listen to. I truly highly recommend this book. I would know a good place where to order it, but I leave that to you. We have seen a couple of chart types already. I had this line chart. I had the histogram bar charts. This has nothing to do with computers, but I wanted to introduce the concept of those scatter plots. So this is basically how much people pay in a restaurant for their bill, sorry, how much tip people give in a restaurant depending on the price of the bill and of the gender. So what do you see if you go out for lunch that's pretty much the same. So in this chart you have multiple dimensions. You have gender. You have the tip, of course, the total bill. And you see those clusters of information. And interestingly, when it comes to dinner, then male customers tend to give way more tip than female ones. I don't know why. Another chart type which is very, very informative for humans, those are heat maps. What you basically see is, I mean, down here is a heat map for network latency, where at some point a bandwidth limitation kicks in and then you see, you know, how the latency is distributed for your packets and that is not something a single number could kind of tell you. Or a distribution. Another thing that also Brent and Prussia quite a lot of flame graphs, they also are extremely informative of understanding what goes on in your system. Right. Now I'd like to come back to more categories we were at before. That's CPU, memory, network IO. And let's start with CPU performance. So basically the question is how long does a computational task take? I was asking one of our principle engineers how he would put this in three bullet points for a slide and his answer was it all depends. But it's true. It really, really, really depends on the architecture. So my first computer had a Z80 CPU in there. That was something you could easily understand. Modern computers or modern CPUs are quite complex systems. It also depends on caches as we saw before memory latency, IPC, so that is instructions per cycle, clock speed, and if you are after integer or a floating point performance. So on the good old days clock speed was the thing. I already mentioned we had the one gigahertz and then it was two and three and then people found out oh actually we cannot scale this up to infinity. Then they started to add more cores and stuff. But of course if you have an application which is bound to one core than the other 71 you saw before are kind of vital. So the answer really is it's a mix of different micro benchmarks you would need to run to answer this question and this really comes down to your workload or to your application to really say if you have the right CPU performance or not. There are standardized ways like spec CPU but this costs money to use. A free thing, a free benchmarking or stressing tool I find extremely valuable is StressNG. So by now they have something like 70 CPU specific. We call this Stressers. Basically it's micro benchmarks and the tool lets you say okay I want to have everything you consider to be a CPU benchmark or a CPU Stresser and a CPU cache Stresser. Run this for a minute and just dump the matrix and then you get a long long table like this. They call actually the result BogoOps which is not a bad name because you never know really what that number at the end of the day means but you know you can compare those numbers between different CPUs and different architectures. And actually the art at the end is to pick the right ones out of those and combine them in the right way. And yeah that is basically what the spec consortium does with spec CPU. Even though you have to be careful that you use the same config if you want to compare two spec CPU runs with each other. So I mentioned StressNG. Overall they have now about 200 of those Stressers. They hit various subsystems like block IO for network. You can do things like threshing caches or basically invalidating them. They also target certain kernel interfaces and yeah it's also a very good tool to check if your system is robust. Right let's get to block IO. The question is I mean we want to read and write files. They are split into blocks in different sizes and one of the applications would be to read a long long sequential block or blocks of you know a big file basically. Want to get this off your of your hard drive. Where another type of an application probably requires quick random access and then seek time is a big thing. So like you will see in the others the tradeoff always is high throughput versus low latency. A few more parameters one has to understand when we talk about block IO. So IOPS are input output operations per second or people call it IO capacity and sometimes you have manufacturers that come up with a big big IOPS number and they want to impress you with that. It's not always the truth to what your block devices are capable of. Another parameter to look at is Q-DAPS. That basically means how many IO operations can be in flight or can be queued at a certain time or at the same time I should say. And latency basically means how long does this IO operation take. So and when you have a big Q-DAPS and you basically get tons of IOPS but it doesn't mean that your latency is actually where it should be for your application. Let's assume that your application needs a very low latency. So and you have a little formula how to calculate your IOPS. That is a nice visualization of the Linux storage system I found. So basically up here is file systems and caches so that is the thing where you can bypass that. And you have the whole block layer and down here your devices and when you benchmark block IO that is also something to have in the back of your head. There are many many tools to measure all those parameters. The one I find extremely useful or the one which I use the most is FIO. It stands for flexible IO mapper and it's actually from the kernel maintainer that is in charge of this block IO layer in the Linux kernel. To get started with it I recommend to look at the examples that come with it. There are even examples how you can measure S3. So it's capable of many things as you will see over the next couple of slides. It also can do even HTTP requests. So this is a default command so name is basically the file on the disk we use to write. We say we want to have a random read write benchmark. We want to have direct IO so we don't want to measure memory performance in terms of caches. We use the IO engine lib aio. The block size is 1 megabyte. The mix for random read write is that we want to have 75% read and 25 write which is usually a pretty good mix. The size of this file here we create for writing is 1 gig. We want to have group reporting. I mean even though it doesn't fit on one slide the output and IO depth of 4 means we have 4 IO operations in flight but this is not what on the search controllers and we want to have two jobs in parallel. Then you get tons of output. There are block articles that discuss those. I also have to warn you that with newer versions of FIO that from time to time changes if you want to pass that don't FIO has FIO can output all of that in for instance JSON and other file formats you can easily pass later on. What you get here is basically the read IOPS are like this. I have 166 IO operations per second. That gives us a bandwidth of 166 megabytes. Another thing where I may be confused a little bit is he gives you this S-LED, C-LED and LED. Those are latency for submission and completion. Another thing you should be aware of. I know it's probably hard to read. Always look at the units here because sometimes you can get confused why those numbers don't add up and then you see it's all about the units. You also get percentiles in your output and yeah there's tons of more information. It even reports your page faults. Whenever you say major page faults then it's probably something else. Network. We basically have the same problem but this time we call it transmit and receive. Receive is reading and transmit writing to network. We have the same things. It might be we want to stream this talk to the internet then we are all about throughput. It might mean we want to do a vibe call over the internet. Then latency is a big issue. Basically round trip time. We have the problem that networks usually are unreliable. If you transport something over the internet there's no guarantee that your packets arrive. I showed you this kind of picture before. It's actually done with FIO. FIO also can target a client server and then it's the same thing. We measure IO. If you take the latency for each packet that gets transferred and use something like my purple. In that case you can get this very nice heat map charts where you can try to understand what's going on for many many hours. If you know what's going on here we again kick in bandwidth limitation and then at some point those two spread out here. So far I haven't found out what this means. I said one of the challenges is that you have packet loss and packet corruptions and retransmits and all those things. There's a nice module in the Linux kernel called net.am. You probably would put this in between your client and service where you measure your network performance and you can with three lines introduce those errors at this point. Basically you load it into kernel to your root Qtask you edit and then you do things like I want to have a delay introduced between 50 and 10 milliseconds. I want to have them normally distributed. I want to have 0.1% corruption and I want to have 0.1% packet loss and yeah that's what you get. So basically the latency and stuff is much more spread out. So for memory again it's the same thing read write so we are interested in throughput. We are interested in latency. We have to make sure to not measure cache performance. Other things to keep in mind are page faults or how you actually access your memory. So if you have numer nodes keep that in mind to really measure the thing you want to. One option of course is again stressNG. So this time we use the class memory and VM. Again that takes time to find the right stresses for your workload. There are other benchmarks like this one called Xmem. Mainly developed by Microsoft but it's open source which basically has all the features you might be interested in. And also supports basically many many different platforms. So in case you're interested in ARM Xmem serves the purpose. Right some other performance measurement tools I find useful and it came to my mind when I made the slides. First of all again I'd like to stress Brands Block. So there are tons of tools like I don't know where to start. Perf is a very important one. Strace, IOTop, Blocktrace, IOSstat. There's a great talk by Brand himself. You'll find a link on brand-director.com. The HML totally worths to to spend this hour. And he basically goes over many of those tools here and explains in which cases they are useful. Another one I found extremely helpful is sysjitter. So sometimes you have the problem that, well, yeah, maybe I explained first what sysjitter does. So it's basically running a thread on each CPU of your system. And it measures the time the thread gets stopped. And this gives you a very good measure of, you know, how much jitter is introduced in your system. Of course, you should run this on an idle system, because otherwise it doesn't mean much. But that is a very good tool to measure that and to see if something strange is going on in your system. And yeah, in that case, it run over four CPUs, and you have percentiles and everything. And here you see the total interruption time in percent. So this system, first of all, looks pretty even and also looks pretty much okay. And then you have those tools that more or less try to do everything. So those are more frameworks. One of them is suspense. So if you don't want to get into the details of finding out what are the best CPU micro benchmarks and stuff, then maybe suspense or something similar is good. So they have one number you get for CPU, one for memory, and other things. They also have a MySQL database benchmark. Yeah. And you cannot do your own. And I'm not sure if they provide charting there. But this gets you a job done pretty quickly. Another one is HPCC. I especially mentioned that for the stream memory benchmarks. They are pretty useful. I like the chart where they basically have the numbers of all the benchmarks they support. And then you get the circle, which is good. And you get this weird distribution where one benchmark is good and another one is not. PerfKit is something from Google where they basically wanted to provide a framework where you can benchmark different cloud providers. Even though you're not, even if you're not interested in cloud computing but have your own service, they have a long list of benchmarks that make sense. And the other good thing is they already come pre-configured. So you know which parameters are a good pick. And of course you can run those things individually. Another one, probably the largest one is the Pharaonix test suite. They really have a large collection of benchmarks and tests. In there, it's a framework. Yeah, you will see on the web page they sometimes try to attract readers in kind of a clickbait way. A big warning, make sure you do not upload your test results. The Pharaonix servers, it is said that some companies did this accidentally with products they didn't even have shipped. That's probably not really what you want. Right. And I'd like to close my talk with LK study. I found a couple of days ago, so the headline is running a database server on EC2, your clock could be slowing it down. So they obviously also had the performance problem and they did long analysis and they figured that they were using a clock source, which required SS call and that totally slowed them down. So sure, use TSC where you don't have to execute SS call to get the time. The thing I want to say with that is it might be that you really try to chase down performance bottlenecks and then it's something completely different. So last slide. Understand your systems and bottlenecks. Monitor continuously and also benchmark from time to time. All right. Thank you.
 All right. So actually, what a conference to have Leonard as kind of an opening act to  my talk. So this is going to be kind of a general talk about system performance with  systems, I mean Linux systems, and to be more specific, it's basically about Linux  service. Before I start, a little bit about myself. I actually like computers a  lot. I like them fast, and I've been lucky enough to always find employers who  kind of provide me those. So I used to work for AMD in Dresden, which is about two  hours south of where we are right now. And I was there when we had the first  one gigahertz processor or x86 processor, I should say. I was there when AMD  introduced the x86 46 bit extensions. I was quite interested in time. And since  five and a half years, I'm now with Amazon. And what we do in Dresden is  basically all the kernel and hypervisors that Amazon runs on their servers. And I  truly can tell you a lot. And the kernel and hypervisors is coming from a group  that has multiple offices, but a big one is in Dresden. I promised to put this up  so we are hiring. If you're interested in this, have a look for talk tests. Several  people here. All right. But for the talk, talk. So I agree the title is a little  silly because what does it really mean to be fast? That guy here, he was fast. So  about two weeks ago in Berlin, there was a marathon and the guy run it in a  little bit over two hours. So two hours, one minute or so. Cut a new world record.  And the interesting thing is, you know, you have one number and you can compare  them. And that's really it. You know, you say go and you measure a  four o'clock time. And when the guy comes back in after the 42 kilometers, you  have a number and you can compare those things. Computers are sometimes,  unfortunately, why not, unfortunately, but they are way more complex than that.  And there are way more things you can measure. So let's have a look how a  desktop is presenting you basically an idea of how fast it is. So GNOME has  this system monitor, they call it. And on top you see CPU usage. Over time, you  see memory usage and network usage over time. I mean, not a big surprise, but I  probably would also add your hard drive. That's also an interesting number to see  how much this guy we have. I guess most of you with SSH into a server and run  something like top or H-top, which gives you a little bit more information. One of  the numbers you probably cannot really read here is load. I remember back at  the AMD days, so I basically started in the IT infrastructure team there and we  had a weekly ops meeting for database service and all those things. And one of  the numbers we always looked at was load. And I think every other week or so  there started a big discussion what this figure actually means and if it's useful  to look at it or not. And we repeated this over and over again. And that is  basically the slightest for my co-workers. So from the documentation in the  file that is implemented, it basically is the number of running processes plus  the number of uninterruptible processes in the run queue of the news kernel. And  since this number goes up and down like crazy, you want to have an average over  one five and 50 minutes and that was an exponential decay. It's totally  interesting to look of the, to check out the history of, you know, where load  actually came from. And here's a quote from the guy, Matthias Oeris, who actually  added this uninterruptible processes, which at the time mainly meant the skyro.  And you can read all of that in a blog post from Brandon Gregg, who was a hero  when it comes to performance measurements. But I will quote him a couple of  times in this talk. The thing to remember is load is a very good indicator for  humans to look at the system and see what it's doing. But if you start to think  about it for longer than 10 seconds, what this really means and if this is  actually telling you anything, then look for something else. Before we dive into  all of that a little bit deeper, I'd like to come to a common understanding of  some statistics and chart types. So the first thing is the mean value versus the  median. Mean basically means you have a bunch of numbers and the sum of this and  divide by the count of those. Median is you line them all up, order them and then  pick the one in the middle. Interestingly, the median is the one you should choose for  performance analysis in many cases, or probably most cases, because it's way more  robust. You will see that in a second. Another thing you should be familiar with  because that is extremely helpful is things like mean, standard deviation,  percentile. So that is the probability density function of the normal  distribution. Percentile basically means again you order all your measurements  in a long row, order them by size or by number and then you would have the 50%  percentile right in the middle. For this normal distribution, the mean and  the median are exactly at the same point, right in the middle and well, to sigma you  have the 79. whatever percentile and that means that basically all your  measurements or that 97% of your measurements are below this number.  So why is it important? Just imagine you measure over time some APIs and some  web API, you know, which you query and what you're interested in, how long does it  take to get a response latency for that thing. The green thing is the median and  you see that this number is pretty robust, right? So if you are actually, if you  want to eliminate outliers from your matrix, that is really the thing to use.  The light blue one is the mean value where you see this is indicating a little  bit and if you serve this service to real users, you're probably interested in  the percentile that I think is the 95% percentile here and you really see how  this goes up and that still means that 5% of your users have a  worse experience at this point, which is not good.  All right, another thing to always keep in mind when you do performance  measurements is cashers. So here's a guy who measured latency, I believe, for his  hard drive and what you see is in this chart in the histogram here, it goes up  then it goes down and it goes up again and well, by the title you probably will  have guessed. Here you basically measure cash, so it might be you really measure  just access to memory here, where that probably is where your system has to  really scrape things off the hard drive. And so of course the first comment on  this blog was to use DirectIO to basically bypass the cash.  For latency, I was expecting that this is hard to read here. Things to keep in  mind. So if you want to check that out, there are many more examples on this  gist. If you say that one CPU cycle, which takes about 0.3 nanoseconds, would  be one second, right? Then a level 3 cash access would be something like 43  seconds, which is quite some time compared to a one cash access, which is  three seconds. However, if you go further and you talk to real memory, or you  have to talk to real memory, that already would be six minutes. I mean, you can  read all of that here, but another number, a TCP retransmit, which one, two,  three seconds in this perspective would already be something like a hundred  years. And a physical system reboot, let's say this takes five minutes, and  that's why I put this up here. That means at this time, if you would take this  time back in the world history, that was when the Sahara still was in the ocean.  So that is something I think you should remember from time to time that really  cashes are a very good idea. And well, if you have too much network I.O. that  already adds a lot of latency. So I took this table out of this book to be  honest with you. You should be reading this book and not listening to me.  Brian Gregg is really one of the guys to listen to. I truly highly recommend this  book. I would know a good place where to order it, but I leave that to you. We have  seen a couple of chart types already. I had this line chart. I had the histogram  bar charts. This has nothing to do with computers, but I wanted to introduce the  concept of those scatter plots. So this is basically how much people pay in a  restaurant for their bill, sorry, how much tip people give in a restaurant  depending on the price of the bill and of the gender. So what do you see if you  go out for lunch that's pretty much the same. So in this chart you have multiple  dimensions. You have gender. You have the tip, of course, the total bill. And you  see those clusters of information. And interestingly, when it comes to dinner,  then male customers tend to give way more tip than female ones. I don't know why.  Another chart type which is very, very informative for humans, those are heat  maps. What you basically see is, I mean, down here is a heat map for network  latency, where at some point a bandwidth limitation kicks in and then you see, you  know, how the latency is distributed for your packets and that is not something a  single number could kind of tell you. Or a distribution. Another thing that also  Brent and Prussia quite a lot of flame graphs, they also are extremely  informative of understanding what goes on in your system. Right. Now I'd like to  come back to more categories we were at before. That's CPU, memory,  network IO. And let's start with CPU performance. So basically the question is  how long does a computational task take? I was asking one of our principle  engineers how he would put this in three bullet points for a slide and his answer  was it all depends. But it's true. It really, really, really depends on the  architecture. So my first computer had a Z80 CPU in there. That was something you  could easily understand. Modern computers or modern CPUs are quite complex systems.  It also depends on caches as we saw before memory latency, IPC, so that is  instructions per cycle, clock speed, and if you are after integer or a floating  point performance. So on the good old days clock speed was the thing. I already  mentioned we had the one gigahertz and then it was two and three and then people  found out oh actually we cannot scale this up to infinity. Then they started to  add more cores and stuff. But of course if you have an application which is  bound to one core than the other 71 you saw before are kind of vital. So the  answer really is it's a mix of different micro benchmarks you would need to run  to answer this question and this really comes down to your workload or to your  application to really say if you have the right CPU performance or not. There are  standardized ways like spec CPU but this costs money to use. A free thing, a free  benchmarking or stressing tool I find extremely valuable is StressNG. So by  now they have something like 70 CPU specific. We call this Stressers. Basically  it's micro benchmarks and the tool lets you say okay I want to have everything  you consider to be a CPU benchmark or a CPU Stresser and a CPU cache Stresser.  Run this for a minute and just dump the matrix and then you get a long long  table like this. They call actually the result BogoOps which is not a bad name  because you never know really what that number at the end of the day means but  you know you can compare those numbers between different CPUs and different  architectures. And actually the art at the end is to pick the right ones out of  those and combine them in the right way. And yeah that is basically what the  spec consortium does with spec CPU. Even though you have to be careful that you  use the same config if you want to compare two spec CPU runs with each other.  So I mentioned StressNG. Overall they have now about 200 of those Stressers.  They hit various subsystems like block IO for network. You can do things like  threshing caches or basically invalidating them. They also target certain kernel  interfaces and yeah it's also a very good tool to check if your system is  robust. Right let's get to block IO. The question is I mean we want to read and  write files. They are split into blocks in different sizes and one of the  applications would be to read a long long sequential block or blocks of you  know a big file basically. Want to get this off your of your hard drive.  Where another type of an application probably requires quick random access and  then seek time is a big thing. So like you will see in the others the tradeoff  always is high throughput versus low latency. A few more parameters one has  to understand when we talk about block IO. So IOPS are input output operations  per second or people call it IO capacity and sometimes you have  manufacturers that come up with a big big IOPS number and they want to impress  you with that. It's not always the truth to what your block devices  are capable of. Another parameter to look at is Q-DAPS. That basically means  how many IO operations can be in flight or can be queued at a certain time or  at the same time I should say. And latency basically means how long does  this IO operation take. So and when you have a big Q-DAPS and you basically get  tons of IOPS but it doesn't mean that your latency is actually where it  should be for your application. Let's assume that your application needs a  very low latency. So and you have a little formula how to calculate your IOPS.  That is a nice visualization of the Linux storage system I found. So  basically up here is file systems and caches so that is the thing where you  can bypass that. And you have the whole block layer and down here your  devices and when you benchmark block IO that is also something to have in the  back of your head. There are many many tools to measure all those parameters.  The one I find extremely useful or the one which I use the most is FIO. It  stands for flexible IO mapper and it's actually from the kernel maintainer that  is in charge of this block IO layer in the Linux kernel. To get started with it  I recommend to look at the examples that come with it. There are even examples how  you can measure S3. So it's capable of many things as you will see over the  next couple of slides. It also can do even HTTP requests. So this is a default  command so name is basically the file on the disk we use to write. We say we want  to have a random read write benchmark. We want to have direct IO so we don't want  to measure memory performance in terms of caches. We use the IO engine lib aio.  The block size is 1 megabyte. The mix for random read write is that we want to  have 75% read and 25 write which is usually a pretty good mix. The size of  this file here we create for writing is 1 gig. We want to have group reporting. I  mean even though it doesn't fit on one slide the output and IO depth of 4 means  we have 4 IO operations in flight but this is not what on the search  controllers and we want to have two jobs in parallel. Then you get tons of  output. There are block articles that discuss those. I also have to warn you  that with newer versions of FIO that from time to time changes if you want to  pass that don't FIO has FIO can output all of that in for instance JSON and  other file formats you can easily pass later on. What you get here is basically  the read IOPS are like this. I have 166 IO operations per second. That gives us  a bandwidth of 166 megabytes. Another thing where I may be confused a little bit  is he gives you this S-LED, C-LED and LED. Those are latency for submission and  completion. Another thing you should be aware of. I know it's probably hard to  read. Always look at the units here because sometimes you can get confused  why those numbers don't add up and then you see it's all about the units. You  also get percentiles in your output and yeah there's tons of more information. It  even reports your page faults. Whenever you say major page faults then it's  probably something else.  Network. We basically have the same problem but this time we call it  transmit and receive. Receive is reading and transmit writing to network. We have  the same things. It might be we want to stream this talk to the internet then  we are all about throughput. It might mean we want to do a vibe call over the  internet. Then latency is a big issue. Basically round trip time. We have  the problem that networks usually are unreliable. If you transport something  over the internet there's no guarantee that your packets arrive.  I showed you this kind of picture before. It's actually done with FIO. FIO  also can target a client server and then it's the same thing. We  measure IO. If you take the latency for each packet that gets transferred  and use something like my purple. In that case you can get this very nice  heat map charts where you can try to understand what's going on for many  many hours. If you know what's going on here we again kick in bandwidth  limitation and then at some point those two spread out here. So far I haven't  found out what this means. I said one of the challenges is that you have  packet loss and packet corruptions and retransmits and all those things. There's  a nice module in the Linux kernel called net.am. You probably would put this in  between your client and service where you measure your network performance and  you can with three lines introduce those errors at this point. Basically you  load it into kernel to your root Qtask you edit and then you do things like I  want to have a delay introduced between 50 and 10 milliseconds. I want to have  them normally distributed. I want to have 0.1% corruption and I want to have  0.1% packet loss and yeah that's what you get. So basically the  latency and stuff is much more spread out. So for memory again it's the same  thing read write so we are interested in throughput. We are interested in  latency. We have to make sure to not measure cache performance. Other things  to keep in mind are page faults or how you actually access your memory. So if  you have numer nodes keep that in mind to really measure the thing you want to.  One option of course is again stressNG. So this time we use the class memory and  VM. Again that takes time to find the right stresses for your workload. There  are other benchmarks like this one called Xmem. Mainly developed by Microsoft but  it's open source which basically has all the features you might be interested in.  And also supports basically many many different platforms. So in case you're  interested in ARM Xmem serves the purpose. Right some other performance  measurement tools I find useful and it came to my mind when I made the slides.  First of all again I'd like to stress Brands Block. So there are tons of  tools like I don't know where to start. Perf is a very important one. Strace,  IOTop, Blocktrace, IOSstat. There's a great talk by Brand himself.  You'll find a link on brand-director.com. The HML totally worths to  to spend this hour. And he basically goes over many of those tools here and  explains in which cases they are useful. Another one I found extremely  helpful is sysjitter. So sometimes you have the problem that, well, yeah, maybe I explained  first what sysjitter does. So it's basically running a thread on each CPU of your system.  And it measures the time the thread gets stopped. And this gives you a very good measure of,  you know, how much jitter is introduced in your system. Of course, you should run this  on an idle system, because otherwise it doesn't mean much. But that is a very good tool to  measure that and to see if something strange is going on in your system. And yeah, in that  case, it run over four CPUs, and you have percentiles and everything. And here you see the total  interruption time in percent. So this system, first of all, looks pretty even and also looks  pretty much okay. And then you have those tools that more or less try to do everything.  So those are more frameworks. One of them is suspense. So if you don't want to get into  the details of finding out what are the best CPU micro benchmarks and stuff, then maybe  suspense or something similar is good. So they have one number you get for CPU, one for memory,  and other things. They also have a MySQL database benchmark. Yeah. And you cannot do  your own. And I'm not sure if they provide charting there. But this gets you a job done  pretty quickly. Another one is HPCC. I especially mentioned that for the stream memory benchmarks.  They are pretty useful. I like the chart where they basically have the numbers of all the  benchmarks they support. And then you get the circle, which is good. And you get this  weird distribution where one benchmark is good and another one is not. PerfKit is something  from Google where they basically wanted to provide a framework where you can benchmark  different cloud providers. Even though you're not, even if you're not interested in cloud  computing but have your own service, they have a long list of benchmarks that make sense.  And the other good thing is they already come pre-configured. So you know which parameters  are a good pick. And of course you can run those things individually. Another one, probably  the largest one is the Pharaonix test suite. They really have a large collection of benchmarks  and tests. In there, it's a framework. Yeah, you will see on the web page they sometimes  try to attract readers in kind of a clickbait way. A big warning, make sure you do not upload  your test results. The Pharaonix servers, it is said that some companies did this accidentally  with products they didn't even have shipped. That's probably not really what you want.  Right. And I'd like to close my talk with LK study. I found a couple of days ago, so  the headline is running a database server on EC2, your clock could be slowing it down.  So they obviously also had the performance problem and they did long analysis and they  figured that they were using a clock source, which required SS call and that totally slowed  them down. So sure, use TSC where you don't have to execute SS call to get the time. The  thing I want to say with that is it might be that you really try to chase down performance  bottlenecks and then it's something completely different. So last slide. Understand your systems  and bottlenecks. Monitor continuously and also benchmark from time to time. All right.  Thank you.
Is_my_system_fast__pOOpYUnKCD0_0_155.png Is_my_system_fast__pOOpYUnKCD0_156_335.png Is_my_system_fast__pOOpYUnKCD0_336_431.png Is_my_system_fast__pOOpYUnKCD0_432_683.png Is_my_system_fast__pOOpYUnKCD0_684_839.png Is_my_system_fast__pOOpYUnKCD0_840_959.png Is_my_system_fast__pOOpYUnKCD0_960_2531.png Is_my_system_fast__pOOpYUnKCD0_2532_2855.png Is_my_system_fast__pOOpYUnKCD0_2856_4439.png Is_my_system_fast__pOOpYUnKCD0_4440_5279.png Is_my_system_fast__pOOpYUnKCD0_5280_6455.png Is_my_system_fast__pOOpYUnKCD0_6456_8759.png Is_my_system_fast__pOOpYUnKCD0_8760_9191.png Is_my_system_fast__pOOpYUnKCD0_9192_10175.png Is_my_system_fast__pOOpYUnKCD0_10176_11915.png Is_my_system_fast__pOOpYUnKCD0_11916_13511.png Is_my_system_fast__pOOpYUnKCD0_13512_13559.png Is_my_system_fast__pOOpYUnKCD0_13560_14279.png Is_my_system_fast__pOOpYUnKCD0_14280_14375.png Is_my_system_fast__pOOpYUnKCD0_14376_14423.png Is_my_system_fast__pOOpYUnKCD0_14424_14747.png Is_my_system_fast__pOOpYUnKCD0_14748_15035.png Is_my_system_fast__pOOpYUnKCD0_15036_15203.png Is_my_system_fast__pOOpYUnKCD0_15204_15371.png Is_my_system_fast__pOOpYUnKCD0_15372_15503.png Is_my_system_fast__pOOpYUnKCD0_15504_17183.png Is_my_system_fast__pOOpYUnKCD0_17184_17351.png Is_my_system_fast__pOOpYUnKCD0_17352_17939.png Is_my_system_fast__pOOpYUnKCD0_17940_18611.png Is_my_system_fast__pOOpYUnKCD0_18612_20147.png Is_my_system_fast__pOOpYUnKCD0_20148_21143.png Is_my_system_fast__pOOpYUnKCD0_21144_21467.png Is_my_system_fast__pOOpYUnKCD0_21468_23399.png Is_my_system_fast__pOOpYUnKCD0_23400_25715.png Is_my_system_fast__pOOpYUnKCD0_25716_26531.png Is_my_system_fast__pOOpYUnKCD0_26532_26843.png Is_my_system_fast__pOOpYUnKCD0_26844_26951.png Is_my_system_fast__pOOpYUnKCD0_26952_28079.png Is_my_system_fast__pOOpYUnKCD0_28080_29207.png Is_my_system_fast__pOOpYUnKCD0_29208_31391.png Is_my_system_fast__pOOpYUnKCD0_31392_32315.png Is_my_system_fast__pOOpYUnKCD0_32316_32723.png Is_my_system_fast__pOOpYUnKCD0_32724_32975.png Is_my_system_fast__pOOpYUnKCD0_32976_35231.png Is_my_system_fast__pOOpYUnKCD0_35232_37955.png Is_my_system_fast__pOOpYUnKCD0_37956_39107.png Is_my_system_fast__pOOpYUnKCD0_39108_40199.png Is_my_system_fast__pOOpYUnKCD0_40200_40655.png Is_my_system_fast__pOOpYUnKCD0_40656_42215.png Is_my_system_fast__pOOpYUnKCD0_42216_42503.png Is_my_system_fast__pOOpYUnKCD0_42504_43211.png Is_my_system_fast__pOOpYUnKCD0_43212_43751.png Is_my_system_fast__pOOpYUnKCD0_43752_44531.png Is_my_system_fast__pOOpYUnKCD0_44532_44579.png Is_my_system_fast__pOOpYUnKCD0_44580_44939.png Is_my_system_fast__pOOpYUnKCD0_44940_46067.png Is_my_system_fast__pOOpYUnKCD0_46068_47699.png Is_my_system_fast__pOOpYUnKCD0_47700_48911.png Is_my_system_fast__pOOpYUnKCD0_48912_49739.png Is_my_system_fast__pOOpYUnKCD0_49740_50807.png Is_my_system_fast__pOOpYUnKCD0_50808_51947.png Is_my_system_fast__pOOpYUnKCD0_51948_53351.png Is_my_system_fast__pOOpYUnKCD0_53352_53951.png Is_my_system_fast__pOOpYUnKCD0_53952_54092.png
All right. So actually, what a conference to have Leonard as kind of an opening act to my talk. So this is going to be kind of a general talk about system performance with systems, I mean Linux systems, and to be more specific, it's basically about Linux service. Before I start, a little bit about myself. I actually like computers a lot. I like them fast, and I've been lucky enough to always find employers who kind of provide me those. So I used to work for AMD in Dresden, which is about two hours south of where we are right now. And I was there when we had the first one gigahertz processor or x86 processor, I should say. I was there when AMD introduced the x86 46 bit extensions. I was quite interested in time. And since five and a half years, I'm now with Amazon. And what we do in Dresden is basically all the kernel and hypervisors that Amazon runs on their servers. And I truly can tell you a lot. And the kernel and hypervisors is coming from a group that has multiple offices, but a big one is in Dresden. I promised to put this up so we are hiring. If you're interested in this, have a look for talk tests. Several people here. All right. But for the talk, talk. So I agree the title is a little silly because what does it really mean to be fast? That guy here, he was fast. So about two weeks ago in Berlin, there was a marathon and the guy run it in a little bit over two hours. So two hours, one minute or so. Cut a new world record. And the interesting thing is, you know, you have one number and you can compare them. And that's really it. You know, you say go and you measure a four o'clock time. And when the guy comes back in after the 42 kilometers, you have a number and you can compare those things. Computers are sometimes, unfortunately, why not, unfortunately, but they are way more complex than that. And there are way more things you can measure. So let's have a look how a desktop is presenting you basically an idea of how fast it is. So GNOME has this system monitor, they call it. And on top you see CPU usage. Over time, you see memory usage and network usage over time. I mean, not a big surprise, but I probably would also add your hard drive. That's also an interesting number to see how much this guy we have. I guess most of you with SSH into a server and run something like top or H-top, which gives you a little bit more information. One of the numbers you probably cannot really read here is load. I remember back at the AMD days, so I basically started in the IT infrastructure team there and we had a weekly ops meeting for database service and all those things. And one of the numbers we always looked at was load. And I think every other week or so there started a big discussion what this figure actually means and if it's useful to look at it or not. And we repeated this over and over again. And that is basically the slightest for my co-workers. So from the documentation in the file that is implemented, it basically is the number of running processes plus the number of uninterruptible processes in the run queue of the news kernel. And since this number goes up and down like crazy, you want to have an average over one five and 50 minutes and that was an exponential decay. It's totally interesting to look of the, to check out the history of, you know, where load actually came from. And here's a quote from the guy, Matthias Oeris, who actually added this uninterruptible processes, which at the time mainly meant the skyro. And you can read all of that in a blog post from Brandon Gregg, who was a hero when it comes to performance measurements. But I will quote him a couple of times in this talk. The thing to remember is load is a very good indicator for humans to look at the system and see what it's doing. But if you start to think about it for longer than 10 seconds, what this really means and if this is actually telling you anything, then look for something else. Before we dive into all of that a little bit deeper, I'd like to come to a common understanding of some statistics and chart types. So the first thing is the mean value versus the median. Mean basically means you have a bunch of numbers and the sum of this and divide by the count of those. Median is you line them all up, order them and then pick the one in the middle. Interestingly, the median is the one you should choose for performance analysis in many cases, or probably most cases, because it's way more robust. You will see that in a second. Another thing you should be familiar with because that is extremely helpful is things like mean, standard deviation, percentile. So that is the probability density function of the normal distribution. Percentile basically means again you order all your measurements in a long row, order them by size or by number and then you would have the 50% percentile right in the middle. For this normal distribution, the mean and the median are exactly at the same point, right in the middle and well, to sigma you have the 79. whatever percentile and that means that basically all your measurements or that 97% of your measurements are below this number. So why is it important? Just imagine you measure over time some APIs and some web API, you know, which you query and what you're interested in, how long does it take to get a response latency for that thing. The green thing is the median and you see that this number is pretty robust, right? So if you are actually, if you want to eliminate outliers from your matrix, that is really the thing to use. The light blue one is the mean value where you see this is indicating a little bit and if you serve this service to real users, you're probably interested in the percentile that I think is the 95% percentile here and you really see how this goes up and that still means that 5% of your users have a worse experience at this point, which is not good. All right, another thing to always keep in mind when you do performance measurements is cashers. So here's a guy who measured latency, I believe, for his hard drive and what you see is in this chart in the histogram here, it goes up then it goes down and it goes up again and well, by the title you probably will have guessed. Here you basically measure cash, so it might be you really measure just access to memory here, where that probably is where your system has to really scrape things off the hard drive. And so of course the first comment on this blog was to use DirectIO to basically bypass the cash. For latency, I was expecting that this is hard to read here. Things to keep in mind. So if you want to check that out, there are many more examples on this gist. If you say that one CPU cycle, which takes about 0.3 nanoseconds, would be one second, right? Then a level 3 cash access would be something like 43 seconds, which is quite some time compared to a one cash access, which is three seconds. However, if you go further and you talk to real memory, or you have to talk to real memory, that already would be six minutes. I mean, you can read all of that here, but another number, a TCP retransmit, which one, two, three seconds in this perspective would already be something like a hundred years. And a physical system reboot, let's say this takes five minutes, and that's why I put this up here. That means at this time, if you would take this time back in the world history, that was when the Sahara still was in the ocean. So that is something I think you should remember from time to time that really cashes are a very good idea. And well, if you have too much network I.O. that already adds a lot of latency. So I took this table out of this book to be honest with you. You should be reading this book and not listening to me. Brian Gregg is really one of the guys to listen to. I truly highly recommend this book. I would know a good place where to order it, but I leave that to you. We have seen a couple of chart types already. I had this line chart. I had the histogram bar charts. This has nothing to do with computers, but I wanted to introduce the concept of those scatter plots. So this is basically how much people pay in a restaurant for their bill, sorry, how much tip people give in a restaurant depending on the price of the bill and of the gender. So what do you see if you go out for lunch that's pretty much the same. So in this chart you have multiple dimensions. You have gender. You have the tip, of course, the total bill. And you see those clusters of information. And interestingly, when it comes to dinner, then male customers tend to give way more tip than female ones. I don't know why. Another chart type which is very, very informative for humans, those are heat maps. What you basically see is, I mean, down here is a heat map for network latency, where at some point a bandwidth limitation kicks in and then you see, you know, how the latency is distributed for your packets and that is not something a single number could kind of tell you. Or a distribution. Another thing that also Brent and Prussia quite a lot of flame graphs, they also are extremely informative of understanding what goes on in your system. Right. Now I'd like to come back to more categories we were at before. That's CPU, memory, network IO. And let's start with CPU performance. So basically the question is how long does a computational task take? I was asking one of our principle engineers how he would put this in three bullet points for a slide and his answer was it all depends. But it's true. It really, really, really depends on the architecture. So my first computer had a Z80 CPU in there. That was something you could easily understand. Modern computers or modern CPUs are quite complex systems. It also depends on caches as we saw before memory latency, IPC, so that is instructions per cycle, clock speed, and if you are after integer or a floating point performance. So on the good old days clock speed was the thing. I already mentioned we had the one gigahertz and then it was two and three and then people found out oh actually we cannot scale this up to infinity. Then they started to add more cores and stuff. But of course if you have an application which is bound to one core than the other 71 you saw before are kind of vital. So the answer really is it's a mix of different micro benchmarks you would need to run to answer this question and this really comes down to your workload or to your application to really say if you have the right CPU performance or not. There are standardized ways like spec CPU but this costs money to use. A free thing, a free benchmarking or stressing tool I find extremely valuable is StressNG. So by now they have something like 70 CPU specific. We call this Stressers. Basically it's micro benchmarks and the tool lets you say okay I want to have everything you consider to be a CPU benchmark or a CPU Stresser and a CPU cache Stresser. Run this for a minute and just dump the matrix and then you get a long long table like this. They call actually the result BogoOps which is not a bad name because you never know really what that number at the end of the day means but you know you can compare those numbers between different CPUs and different architectures. And actually the art at the end is to pick the right ones out of those and combine them in the right way. And yeah that is basically what the spec consortium does with spec CPU. Even though you have to be careful that you use the same config if you want to compare two spec CPU runs with each other. So I mentioned StressNG. Overall they have now about 200 of those Stressers. They hit various subsystems like block IO for network. You can do things like threshing caches or basically invalidating them. They also target certain kernel interfaces and yeah it's also a very good tool to check if your system is robust. Right let's get to block IO. The question is I mean we want to read and write files. They are split into blocks in different sizes and one of the applications would be to read a long long sequential block or blocks of you know a big file basically. Want to get this off your of your hard drive. Where another type of an application probably requires quick random access and then seek time is a big thing. So like you will see in the others the tradeoff always is high throughput versus low latency. A few more parameters one has to understand when we talk about block IO. So IOPS are input output operations per second or people call it IO capacity and sometimes you have manufacturers that come up with a big big IOPS number and they want to impress you with that. It's not always the truth to what your block devices are capable of. Another parameter to look at is Q-DAPS. That basically means how many IO operations can be in flight or can be queued at a certain time or at the same time I should say. And latency basically means how long does this IO operation take. So and when you have a big Q-DAPS and you basically get tons of IOPS but it doesn't mean that your latency is actually where it should be for your application. Let's assume that your application needs a very low latency. So and you have a little formula how to calculate your IOPS. That is a nice visualization of the Linux storage system I found. So basically up here is file systems and caches so that is the thing where you can bypass that. And you have the whole block layer and down here your devices and when you benchmark block IO that is also something to have in the back of your head. There are many many tools to measure all those parameters. The one I find extremely useful or the one which I use the most is FIO. It stands for flexible IO mapper and it's actually from the kernel maintainer that is in charge of this block IO layer in the Linux kernel. To get started with it I recommend to look at the examples that come with it. There are even examples how you can measure S3. So it's capable of many things as you will see over the next couple of slides. It also can do even HTTP requests. So this is a default command so name is basically the file on the disk we use to write. We say we want to have a random read write benchmark. We want to have direct IO so we don't want to measure memory performance in terms of caches. We use the IO engine lib aio. The block size is 1 megabyte. The mix for random read write is that we want to have 75% read and 25 write which is usually a pretty good mix. The size of this file here we create for writing is 1 gig. We want to have group reporting. I mean even though it doesn't fit on one slide the output and IO depth of 4 means we have 4 IO operations in flight but this is not what on the search controllers and we want to have two jobs in parallel. Then you get tons of output. There are block articles that discuss those. I also have to warn you that with newer versions of FIO that from time to time changes if you want to pass that don't FIO has FIO can output all of that in for instance JSON and other file formats you can easily pass later on. What you get here is basically the read IOPS are like this. I have 166 IO operations per second. That gives us a bandwidth of 166 megabytes. Another thing where I may be confused a little bit is he gives you this S-LED, C-LED and LED. Those are latency for submission and completion. Another thing you should be aware of. I know it's probably hard to read. Always look at the units here because sometimes you can get confused why those numbers don't add up and then you see it's all about the units. You also get percentiles in your output and yeah there's tons of more information. It even reports your page faults. Whenever you say major page faults then it's probably something else. Network. We basically have the same problem but this time we call it transmit and receive. Receive is reading and transmit writing to network. We have the same things. It might be we want to stream this talk to the internet then we are all about throughput. It might mean we want to do a vibe call over the internet. Then latency is a big issue. Basically round trip time. We have the problem that networks usually are unreliable. If you transport something over the internet there's no guarantee that your packets arrive. I showed you this kind of picture before. It's actually done with FIO. FIO also can target a client server and then it's the same thing. We measure IO. If you take the latency for each packet that gets transferred and use something like my purple. In that case you can get this very nice heat map charts where you can try to understand what's going on for many many hours. If you know what's going on here we again kick in bandwidth limitation and then at some point those two spread out here. So far I haven't found out what this means. I said one of the challenges is that you have packet loss and packet corruptions and retransmits and all those things. There's a nice module in the Linux kernel called net.am. You probably would put this in between your client and service where you measure your network performance and you can with three lines introduce those errors at this point. Basically you load it into kernel to your root Qtask you edit and then you do things like I want to have a delay introduced between 50 and 10 milliseconds. I want to have them normally distributed. I want to have 0.1% corruption and I want to have 0.1% packet loss and yeah that's what you get. So basically the latency and stuff is much more spread out. So for memory again it's the same thing read write so we are interested in throughput. We are interested in latency. We have to make sure to not measure cache performance. Other things to keep in mind are page faults or how you actually access your memory. So if you have numer nodes keep that in mind to really measure the thing you want to. One option of course is again stressNG. So this time we use the class memory and VM. Again that takes time to find the right stresses for your workload. There are other benchmarks like this one called Xmem. Mainly developed by Microsoft but it's open source which basically has all the features you might be interested in. And also supports basically many many different platforms. So in case you're interested in ARM Xmem serves the purpose. Right some other performance measurement tools I find useful and it came to my mind when I made the slides. First of all again I'd like to stress Brands Block. So there are tons of tools like I don't know where to start. Perf is a very important one. Strace, IOTop, Blocktrace, IOSstat. There's a great talk by Brand himself. You'll find a link on brand-director.com. The HML totally worths to to spend this hour. And he basically goes over many of those tools here and explains in which cases they are useful. Another one I found extremely helpful is sysjitter. So sometimes you have the problem that, well, yeah, maybe I explained first what sysjitter does. So it's basically running a thread on each CPU of your system. And it measures the time the thread gets stopped. And this gives you a very good measure of, you know, how much jitter is introduced in your system. Of course, you should run this on an idle system, because otherwise it doesn't mean much. But that is a very good tool to measure that and to see if something strange is going on in your system. And yeah, in that case, it run over four CPUs, and you have percentiles and everything. And here you see the total interruption time in percent. So this system, first of all, looks pretty even and also looks pretty much okay. And then you have those tools that more or less try to do everything. So those are more frameworks. One of them is suspense. So if you don't want to get into the details of finding out what are the best CPU micro benchmarks and stuff, then maybe suspense or something similar is good. So they have one number you get for CPU, one for memory, and other things. They also have a MySQL database benchmark. Yeah. And you cannot do your own. And I'm not sure if they provide charting there. But this gets you a job done pretty quickly. Another one is HPCC. I especially mentioned that for the stream memory benchmarks. They are pretty useful. I like the chart where they basically have the numbers of all the benchmarks they support. And then you get the circle, which is good. And you get this weird distribution where one benchmark is good and another one is not. PerfKit is something from Google where they basically wanted to provide a framework where you can benchmark different cloud providers. Even though you're not, even if you're not interested in cloud computing but have your own service, they have a long list of benchmarks that make sense. And the other good thing is they already come pre-configured. So you know which parameters are a good pick. And of course you can run those things individually. Another one, probably the largest one is the Pharaonix test suite. They really have a large collection of benchmarks and tests. In there, it's a framework. Yeah, you will see on the web page they sometimes try to attract readers in kind of a clickbait way. A big warning, make sure you do not upload your test results. The Pharaonix servers, it is said that some companies did this accidentally with products they didn't even have shipped. That's probably not really what you want. Right. And I'd like to close my talk with LK study. I found a couple of days ago, so the headline is running a database server on EC2, your clock could be slowing it down. So they obviously also had the performance problem and they did long analysis and they figured that they were using a clock source, which required SS call and that totally slowed them down. So sure, use TSC where you don't have to execute SS call to get the time. The thing I want to say with that is it might be that you really try to chase down performance bottlenecks and then it's something completely different. So last slide. Understand your systems and bottlenecks. Monitor continuously and also benchmark from time to time. All right. Thank you.
system performance with systems, I mean Linux those. So I used to work for AMD in Dresden, are way more complex than that. And there are way that is implemented, it basically is the number all of that in a blog post from Brandon Gregg, is a very good indicator for humans to look at understanding of some statistics and chart types. choose for performance analysis in many cases, or in, how long does it take to get a response you probably will have guessed. Here you talk to real memory, or you have to talk to real lunch that's pretty much the same. So in this down here is a heat map for network latency, distributed for your packets and that is not you have the right CPU performance or not. There number at the end of the day means but you know basically many many different platforms. So in like I don't know where to start. Perf is a very total interruption time in percent. So this tools that more or less try to do everything. So those are more frameworks. One of them is to provide a framework where you can benchmark computing but have your own service, they have a results. The Pharaonix servers, it is said that