gstat_exporter: Graphing FreeBSD disk utilization with Prometheus
Rasmussen, Thomas Steen
All in a days work: How to write a Prometheus gstat_exporter and integrate it in a Grafana Dashboard
 Thank you. Hello, everybody. We have plenty of time. I submitted this as a lightning talk and was allocated an hour. So if you have questions or comments or anything, please feel free to interrupt me. Does anybody use Prometheus already? Or is this new for everybody? A few people? You heard of it? Great. We got a Prometheus exporter in previous year 12. Somebody in the previous year world at least knows and uses Prometheus apart from me. A few months ago, I wrote an exporter to make disk utilization metrics available to Prometheus from FreeBSD. And I was encouraged to come here and talk about it. I did a blog post about it and was encouraged to come here and tell you. Since many people probably don't know Prometheus, I'll start out by talking a little bit about what it is and what it does and why it's awesome. And then the concept of exporters. And then we'll get to GSTAD and the actual exporter that I wrote for that. I also have a Grafana dashboard to actually visualize the data connected. And if everything works and my internet connection keeps working, then I have a live demo of some servers at my workplace using this. So it's always nice to see some actual real data. But I'm not making any promises because my connection has been flaggy. So let's see when we get to it. Live demos are always, as you know, a bit tricky sometimes. So my name is Thomas Rasmussen. I'm co-tuggling on the internet. I'm born and live in Copenhagen in Denmark. And I do system architecture, build and administer FreeBSD servers. I do some programming as well and was recently laid off with three months with salary. So I'm just going to fast them and relaxing and enjoying myself at the moment and looking for something else to do. I run a DNS service called Uncensored DNS for 10 years now, 11 years, which is before Google DNS and before all the other public recursive DNS services. I started this as an alternative to the ISP sensor, often censored ISP DNS servers. And it's been running since I also help organize a bond hack, which is an annual Danish hacker camp. I highly recommend going. It is a great time. It's a week with tents and laptops and hacking around with all kinds of fun things. And I have a bunch of minor open source projects on GitHub. I use FreeBSD exclusively on servers. I've used this since 5.2.1, I think was my first installed server. And on my laptops, I run something called CubeSOS, which is an awesome operating system. I wish it was freeBSD based, but it's Fedora and send based. The concept is you have different virtual machines for different contexts. So I have one for work and one for playing with the bond hack stuff and they can really disposable browsers for surfing. So the VM just disappears. So if you happen to click something bad, then it's just nothing really happens. So awesome concept. So I'm here to talk about this GSTAT exporter. This is actually the first time I've been in the BSD room. There's never been room, but one of the perks of being a speaker is that you have to let me in. So I wrote this tweet and someone picked up on it and recommended I come to foster and talk about it. And here we are. So Prometheus in their own words, it's an open source monitoring system, dimensional data model, flexible query language, efficient time series database. And that is pretty much it. It is very, the whole ecosystem around it is very unique see in that, that they do one thing and they do it well. Prometheus in itself is a time series database. It really doesn't do anything except collect metrics and stick them in a time series database and then have it has an API to make them available to Grafana, for example. So you can pull the data back out. It doesn't do alerting. It has something there's something called alert manager, which from the same team, but it's a separate separate pieces of software, separate configuration file and stuff, which handles alerting and alert dependencies and which teams should get what alert and page to do the support and all that stuff. And monitoring has often I think been many years been. It's been a bit of a, like, there was a void left when Nagios kind of stopped being modern and I've been, I've been using subjects and a few other things, but nothing has really been. It's like they tried to do too much. Somehow it's a big job both to gather the data and visualize it and do alerting and everything and I haven't really been happy with anything. You know, like the first time you tried set of us and thought this is what a file system should have been like all along. Where have you been all my life? This is how I feel about managing with Prometheus. It is an excellent system. So it is based on they call it dimensional time series data. It means that your metrics can have labels. They call it. So if you have a metric called HTTP requests total, for example, for a web server, then and you get a number when you when you scrape the data. Say you had a hundred requests. Then that data can be can have multiple dimensions and that it can have labels to say what the path of the request was or what the HTTP status code was. So you can ask to get only the HTTP for requests, for example, and then make a graph of those on your in your graph. This is incredibly powerful. It's actually a relatively simple concept, but it enriches the data greatly. Prometheus and alert manager and all the accompanying software is all written and go. It was. It was started at the sound cloud some years ago. It's been open source since 50 2015 and there's no. Company that owns it now. It's a regular open source project with many contributors and a great community and great IC channel and stuff. So if you have questions, they're very happy to help. So this is the diagram of the ecosystem from their website. Central to the whole thing is the Prometheus server. It's a time series database. And it stalls the metrics, of course, on some sort of disk storage. It's a pole model. Prometheus. That's one of the things that some people has to get used to, which means that Prometheus connects to the stuff. It's monitoring and pose the metrics over HTTP. There's various ways around that if you absolutely cannot do it like that. But that is the idea. For example, the free BST, just a little exporter that came and arrived in free BST 12 is supposed to be run in I not the under in at the and Prometheus connects to it and post out the metrics. So and that that's what the exporters make available. They open an HTTP endpoint and it's just a plain text key value list of lines. I'll have examples later. There's something called push gateway that if you have a job that runs for an hour and then has some metrics to submit at the end of that, you can use push gateway for that. It has a very extensive service discovery system. So if you are AWS or Kubernetes user or you use one of the many, many cloud services available, there's probably a discovery service discovery thing available for it. At work, I work at work and I'm not speaking until I got laid off. We use the file service discovery thing and just exported a list of customers and IP addresses and that was automatically ingested by Prometheus. So when we add a new customer in our central provisioning system, it automatically shows up and starts pulling the data from it. And to talk to Prometheus and query the data and get it out, they invented something called prompt cool. It is what doesn't really have anything to do with SQL. It is a custom query language specifically for this and there will be examples of this later and it is a very powerful and easy to use language. And like I said, alert manager is a separate component. So when you add some threshold or something you want to be alerted on in Prometheus, it pushes an alert to alert manager and alert manager handles all the needy gritty with who is on vacation now and who has the duty and who does what. So all that complexity is separate from Prometheus. Okay, so we generally speak of black box and white box monitoring. Black box monitoring is when you ping or whatever or just poke something from the outside check if it's listening on port 80 or whatever you might do. So, for example, the previous stage is a good example or an exporter for GSTAT data, for example, means you're kind of inside the system, exposing the metrics to the outside. And that's a much better way to do it than black box monitoring. It has support for that as well. If you need to like we do at work ping 10,000 customers every five seconds, then you do that with something called black box exporter. But mostly and primarily it is a white box monitoring. It is supposed to be used with the exporters built into whatever you are monitoring. So if you have a system like our provisioning system at work, it's easy to, for example, export metrics for the queue length for provisioning their customers, the router needs to be switched, the queue needs to be configured and there's sometimes a queue if there's many changes at once. And it's very easy when you're in the code to export this metric and just do a slash permit use or slash metrics endpoint in your system and export it. It's over HTTP, like I said, and it is designed to handle everything you can throw at it. You shouldn't be piggy. You should instrument all parts of your system. Anything that might someday make sense. And even if you think it doesn't, because sometimes these subtle little changes and whatever can turn out to be important. And then it's very nice to have the history. It is very efficient. It is designed to handle hundreds of thousands of metrics and on a single server commodity hardware. It doesn't do, it has a little bit of, you can, when you're exploring the data in the Prometheus web interface, it can draw a simple crude graph, but it's not meant for graphing. It's meant to be used with Grafana or something else, but I highly recommend it is excellent integrated into Grafana. There's client libraries and many languages. The GStatics code I wrote is written in Python and you start out with these 10 lines or something from the Prometheus example client and you just add name, define your metrics saying we have a, in GStat we have a disk IO, busy percent message metric or whatever and an inner feed data into them and it takes care of listening on HTTP and serving the data up in the right order. And it is easy to use and they have of course in Go and many other languages, client libraries. So it is easy to get started writing your own exporter. So it is like I said very efficient. They say about three and a half bytes per data point and it is remarkably efficient when pulling in data. We went to, we replaced our Starbucks installation, which was thrashing the disks using Postgres as a backend. I love Postgres. I don't love Starbucks, but it was a decent setup, but it was, I mean, we could barely handle polling them once per minute, the 5, 8,000 customers or however much we had back then. And when we switched to Prometheus, we're polling them once every five seconds. We're doing additional metrics besides what we did with Starbucks and the server is bored now. It's not doing nothing. So we went from absolutely thrashing the drives to the server being almost idle. It is amazingly efficient. If you go beyond what one server can handle, it supports sharding and federation. So you can have many Prometheus servers working together and it does that very well. And alert manager as well supports high availability stuff. So you can send an alert to any number of alert manager instances and only one of them will actually alert you. At work, the Prometheus installation I was managing there had about 750,000 time series and about 13 and a half thousand per second metrics put into the database. And that's small in Prometheus terms. That's not an issue at all. You can do that on a laptop or on whatever. It's in a jail, all my stuff runs in jails and you can easily have like one for each team. It's not like managing an elk instance or something like that where you have to, you can just wave up an EO if you want one for the network team and one for the operations team or whatever. You can easily do that. So the query language from QL. I bought a few, few examples and it's easy to look up. It has a page or two and then if you understand basic, basic statistics and math, then it should be easy to use. First things first, the HTTP requests total is the example metric we're using here. So that's a that's that was the internet connection that stopped working. So that just returns the metric HTTP request total. If, for example, I want the metric, but only if the job label is API server and the handler is slave slash API slash comments, then I put it in the curly brackets and the only returns the number of HTTP requests that had those, those labels. It also return a range vector. So you get, you get all the values for given for, for example, for a five minute interval. If you need that, you can use regular expressions for the labels and and you can do for matching and stuff for the for the label values as well. I use those labels a lot in the G set exporter to save stuff like the disks serial numbers and stuff. We'll get to that later. A bit more advanced. It has, it has some keywords to do. For example, it can calculate the weight over five minutes for, for this metric or even do a quantize 95th percentile is used a lot when when charging for bandwidth and hosting operations and stuff like that or whatever you need to to use it. And it is, it is very flexible. It can also show the, you can query the, the alert, the number of alerts firing in a system and when, when the installation grows, that's the sort of stuff you want on your front page of your graphana instance, you can have zero or five alerts or whatever. Okay. So, Prometheus connects to an exporter and gets fed metrics and ingests them. And these metrics are exported by, by exporters and machine metrics like, like the normal metrics like Ram and an IO and network traffic and stuff like that are exported by something called node exporter, which we have in ports and works well. But on free BSE, it doesn't have this utilization metrics that has does on Linux, of course, but, but not on free BSE, which is why I started out because when you're troubleshooting stuff on a database server or something, it is very nice to have metrics for, for this utilization. And it's very nice to have a list of exporters for every type of software you can almost imagine it is there's hundreds of them. Each is allocated a port on this list, you just go to the weekend allocate a port and for the G start exporter, I got port 92 something, something. And you can search for free BSE on it and find a few others that there's a jail exporter, which is very nice that somebody made that uses our CTO to get IOPS and memory use and stuff for for Jail's and then exports it in a, in a way so Prometheus can ingest it so you can get graphs to say how much Ram each jail is using and how many IOPS and stuff that is very nice if you're using Jails for yeah, if you're using Jails at all. Okay, so, you know, I guess you all know G start. It looks like this. It is a top style thing that shows the the gym devices and read and write operations at per default. It can show a delete and other stuff as well. Actually have no wait. I don't because I'm not online. Okay, so this is G start and these are the numbers I was interested in getting into a graph and that's where we're here today. There's a dash C flag for for G start that makes it output in in CSV mode is called in the man page instead of the top style display and unfortunately a bit of a complaint would be that it ends it. When you enable dash C, it also enables endless mode so it keeps running. You can just make it output values once and exit. So that's that should probably be added because usually in in exporters when Prometheus connects you scrape the data and then you return it. But I've had to keep it running endlessly. The G start and just streaming the output from it into the collector. Also, if G start could somehow export the counters, it bases the top display on. So these are, you know, the difference between counters engages and and we lose a bit of precision because I'm reading the G start values as counters. It shows me how much there's an update frequency in G start. So every five seconds, every one second or something and and stuff that happens in between, let's say every five seconds is kind of lost. And if you had the counters instead Prometheus would be able to. It doesn't matter if you read every five or 10 seconds, then you would get all the information. And I'm not a C person, but looking at the code, it looks like it is counters when it comes out of the kernel and and somewhere it is converted to the to the human friendly display that we see in G start. Okay, so with the with G start, we really have everything we need with the dash C mode. We have everything we need to write on a write an exporter. Like I said, the client library, Prometheus client libraries is easy to get started with the G start exporter is 240 lines. It sounds like more than this because it's formatted with these code, beautifying tools and they are very generous with the new lines. So it's on GitHub and yeah port 9248 is a it's a G static port. It's not that they don't add them to services files everywhere because they're so dynamic and there's so many of them, but they do have the page on their website or the wiki on GitHub to keep track of which ports are used by what. Okay, so running the exporter locally and and grabbing for a DA zero and grabbing for right. This is the output for for the G start exporter. You see it's just plain text and this is just fetch outputting to to STD out. So it's a it's a plain text response and it's just lines of metric name and then labels and then finally a space and then a number. Which is the actual value for that metric with that combination of labels for that specific time. As you can see, I've stopped all the info I can get from the home. It's called the home identifier or something. There's a home command that can return up here and all these need metrics so I can get. Yeah, for example, sex or size and RPM and the serial number for the discs and the size. Which is very nice because as we will see later in Grafana, it makes it easy to to filter, for example, and show only the NVMe drives only the spinning drives or only the 12 terabyte drives or whatever you need. Okay, so as I said, it doesn't visualize stuff by itself, but the integration to Grafana is really good. It can once you add the previous instance as a data source in Grafana, when you start typing a metric name, it auto completes it and it really works very, very well. And you can filter a dashboard by label values and I'll show you what I mean by that later. But that, for example, means that I can easily in Grafana just select one disc if I'm suspecting that one disc is foggy or select only the 5400 RPM drives across all servers if I should, for some reason, have the need for that. I published the dashboard at the Grafana dashboard published in place and if you don't know Grafana, I highly recommend it. It is very easy to make. It makes pretty graphs for you. Like you don't have to decide on colors and stuff. It uses colors that look well together and generally just works extremely well. Okay, I'm going to need to fix my internet connection. So talk about yourself for 30 seconds while I get this. Okay. Just work for the love of, sorry. Yes, I would if I could. It's a long and boring explanation why it doesn't work. And usually the phone just works, but of course, right now, it doesn't. Yeah, thanks guys, but it's not going to work. Really? This is great advertisement for Linux. Well, cubes is special. It is a bit. But it really isn't. I'm glad I want that this might happen ahead of time. Yes. Okay. So maybe we skipped a live demo since it's not working. I'm sorry about that. I'll talk a bit more and then try again in a few minutes when it's time to recover. I do have some screenshots in the beginning that I can go back to. But oh, yeah, thanks. It was a pretty quick project. I did it in a day. Started in the morning investigating how I would do it. And then I wrote the code during the day and in the evening I did the dashboard and then I wrote a blog post and posted it. So it was in the scope of a day you can write an export of something including doing a nice dashboard and making everything work. If GSTAT gets support for outputting just align and then exiting, I'll probably change the way it calls GSTAT. So it doesn't keep it running perpetually, but just calls it when it is scraped. And maybe it could add flags to support if there's a need for it. GSTAT can show consumers as well as producers in the GEM stack. And some people might need this for reasons I can't really. The Grafana dashboard needs some fine tuning. And if anybody wants to... Oh, fuck off. If anybody feels like helping with that, then you're very welcome. I'm going to get this one quick final attempt. Well, I have it open fortunately so we can show you some of the things I was talking about. The labels I add to the... The labels I add to the metrics can be used at Grafana to make this nice little table in the top that shows this is... In the top we have the filters that Grafana makes it possible to filter by label values. So this shows server names and this shows the different GEMs and the different disk sizes we have. And sector sizes and RPM and stuff. This is where I would, for example, I didn't know we still had a 5400 RPM drive, but after I installed this, then I got a bit better overview of the entire... And it's a nice way to see, for example, serial number. If you know what disk is performing poorly, you can easily see the serial number and ask your on-site hands to fix it. And other than that, there's sections for each of the metrics GeoMix ports. So there's a latency graph for read, write and delete latency. And there's a bandwidth section for... Of course it can't. Trust me, there's bandwidth graph behind if I was online. And there's an IAPS section and the Q-depth. This is showing 30 days. I've had this running for a few months since I wrote it. And one of the fun things is that if you ran a very large set of S-pools, you probably know that the scrubbing can take almost forever to complete. Which means that many people with large pools end up doing a nightly scrub job that just runs it through the night and poses it in the working hours. And that's the fun. This is the busy present. And the scrubs are these... This is 30 days. So you can see we scrub all weekend and then every night. And then all weekend and every night. So it's a fun pattern to observe. And we found a bad disk in one of the servers using this. I can't find the data now, of course, but it works really well to get an overview of the disk utilization. And that's actually... That's all I wanted. So that's very nice. Any questions? We have plenty of time. Yes? My OCD screams at me that the media size should be just a number without the string. Yeah, but it's exported directly from the string that the geom info exports. So I could cut it off and then sell Grafana that it's a byte value and it would humanize it for me. You're right. That would be easy. Grafana knows... It knows units. So this graph, I've told it that this is a number of seconds and then it shows all the scales, the y-axis to show that this is milliseconds. Of course, not seconds, fortunately. And it understands megabytes per second and stuff like that. So it is very lovely to work with. Yes? Have you had any trouble with Prometheus or Grafana because you're on FreeBSD instead of Linux? Apart from it not exporting disk utilization? No. Oh, right. The question was, have I had any problems running a Grafana or Prometheus on FreeBSD because it's FreeBSD and not Linux? No. Apart from the node exporter, that's the thing that exposes system metrics like memory and disk IO and stuff like that. Apart from it not exporting disk IO and not exporting ZFS data, then no, everything works very well. The ports work perfectly and they are updated timely and run very well. The latest version or the upcoming version of node exporter does have FreeBSD ZFS support and I'm very much looking forward to that. And combined with this, then you can get a great impression of how your storage is doing. Yeah? Yes? I have questions in a bit disjoint topics. The first one is, you were touching keywords like alerting and federation and this kind of stuff. So my question is about these servers what you were scraping. They were only scraping endpoints or they were scraped and processed at the same site. Or you had one central Prometheus server and then scraping all the endpoints. Yes, one central server that connects out to, you configure it, tell it which targets to scrape and then it just sucks up all the metrics that are available at that endpoint. Okay. Have it meant some problem with delay or the latency? No, I mean, you would think, but no, it just really works very well. It's a bit of a radical, both the pull model and the idea that it does the scrape as it connects. So if FreeBSD doesn't connect for a while, then there's no data collected. But it really works very well and turns out that it doesn't matter too much as long as you use counters, it would matter for GSTAT exporter because you would lose the precision if you reboot your Prometheus server, then you would lose the five minutes or whatever it takes. But if you have counters available, then you don't because it doesn't matter if Prometheus has gone for a while. It'll figure out how to, what's the word I'm looking for, put. Yeah, exactly. It'll smooth it out. So it works very well and people are doing some like the big hosting operators. I have some like 500 Prometheus servers doing millions and millions of data points per second. Some crazy big instances. So for my users with 100 servers or something, I know it's always nice to have to know that it can handle way more than you need. This also means that you have one central server, so you have to maintain your alerts on one server, right? So the wire tools or the alert conditions have to be written on one server. Well, the alert manager can, is a separate jail. I mean, it could be a separate server if you want it. Yeah, I mean, okay. Yeah, a different question in the intro of your talk. You mentioned that you take care of this BornHack camp. Yes. What's the topic that attracts? Yeah, it is stuff like this. It is InfoSec. It is also hardware, open source and really brought a few, if you've been to the German CCC camp or the Dutch Big Hacker camps, it is very much inspired by those. So it is kind of reminds a little bit of first time except that people live in tents and there's not as many people, of course. About 400 people I think this year and it's slowly growing. It's the fifth year this time. So, and it's an annual event and it's getting traction. It's a lovely little event. So, any more questions? Yes. Yeah, I was trying to find a monitoring solution and using to have a good experience from you first. Absolutely. Did you try some of this competing text that you know telegraph and infox. Have you seen about that? No, not really. I haven't used it enough to, to have not set it up. I've only briefly been exposed to it when somebody else had configured it. So I don't know enough. But you know how you can sometimes sense that something has momentum like a project like every, I think everywhere I look, Prometheus exporters are popping up like natively inside whatever engine X or whatever. Suddenly has a way to explore and and that tells me that, you know, I'm not the only one who feels that this is the right approach. And that's very nice because I think we have been, I've been looking for something nice for months or in four years and years. I've never been really, really happy even back when Nagius was the thing to run. I've never really been very happy with it. I think everybody really kind of loved to hate it. So I am so, so happy that finally something and it also, I mean, the aesthetics are also very important because even though technically it's the same numbers. If it's an acrylic graph or a pretty graph, but it just still makes a difference. It's nice to work with. I have one response to that. So just looking at influx while it's nice to work with in at least some aspects. Prometheus has much, much more advanced math available. Like for example, in influx, it's not possible in a single query to get. For example, you get the return codes of HTTP requests. In influx, it's not possible to get what proportion of requests were errors. It's possible in graphite. It's possible in Prometheus. It's not possible in influx. In jade graph works with Pymus. It's one. Thank you. Any more questions? I have a quick question because I'm rarely surrounded by a lot of previous people. So I just. Oh, damn, I went offline. Oh, and I can't even show it. Okay. Well, the question, though, the problem is, you know, the jail exporter I was talking about, which makes it with users. I'll see to export jail metrics and then then you can do graphs for, for example, memory use inside a jail. So I use that, of course, because I use jails for all my stuff. And it's very nice to see if a disk is being trashed. You can easily see which jail is causing this postgres doing an auto vacuum or what's going on. RCTL counts shared memory twice in jails or as many times I have what I have in this tab that I can switch over to because I'm offline. I have two jails on this server. It has a hundred hundred and twenty eight gigabytes of RAM. I have two postgres jails and RCTL tells me that one jail is using about 130 gigs and the other jail is using 80 gigs, which is about 200 gigs. And there's 40 more jails and it's so clearly something is wrong. And I've been able to track down as much as it is. I don't know what the fix is, but it is absolutely misreporting memory use in jails. And since if I have a hundred postgres workers in each of them is using two gigabytes of RAM, then most of it is shared. What you want RCTL to export is how much memory would be freed if I shut the jail down. I don't care if a hundred workers is using two gigabytes. You shouldn't report 200 gigabytes memory use. Does anybody know the is this just a bug or is there is this not possible? What I'm asking for. I mean, not between jails between between a hundred postgres workers in the same jail, for example, and then each of those they have access to the same shared memory space. And it's been counted a hundred times because there's a hundred workers. So it tells me that one jail is using 140 gigs of RAM on 128 gig server, which clearly is not true. I wonder how it would look like if it was running outside of jail if it would be reported the same way? It would. But we can talk about it later after I just it's been bugging me because. I finally got some nice jail graphs and the memory if use it is just way, way off. I was like, what the hell is wrong? How can it be using 300 gigabytes of RAM when it has 128 and RCTL exports the V memory use as well? And that's of course a high number and higher than the as 1.2 terabytes of memory of the memory. I think. But the memory use should be. We agree that it should be however much would be freed if the jail was shut down. Right. Otherwise, what's the point of reporting them? I mean, you see my you see what I mean? Okay. I think I'll open up PR and write to a mailing list and see if I can. I think I can catch somebody's attention with that. But other than that, the Jail is really awesome and stuff like IOPS and that works very well. So you can see which which Jail is trashing at this goal or using a lot of CPU. I think that's it for me. We're done with a quarter or quarter of an hour to spare. So we're going to use that time to for self reflection and the self improvement. And thanks for having me and I hope you enjoyed first them and my talk and. Thank you.
 Thank you.  Hello, everybody.  We have plenty of time.  I submitted this as a lightning talk and was allocated an hour.  So if you have questions or comments or anything, please feel free to interrupt me.  Does anybody use Prometheus already?  Or is this new for everybody?  A few people?  You heard of it?  Great.  We got a Prometheus exporter in previous year 12.  Somebody in the previous year world at least knows and uses Prometheus apart from me.  A few months ago, I wrote an exporter to make disk utilization metrics available to Prometheus from FreeBSD.  And I was encouraged to come here and talk about it.  I did a blog post about it and was encouraged to come here and tell you.  Since many people probably don't know Prometheus, I'll start out by talking a little bit about what it is and what it does and why it's awesome.  And then the concept of exporters.  And then we'll get to GSTAD and the actual exporter that I wrote for that.  I also have a Grafana dashboard to actually visualize the data connected.  And if everything works and my internet connection keeps working, then I have a live demo of some servers at my workplace using this.  So it's always nice to see some actual real data.  But I'm not making any promises because my connection has been flaggy.  So let's see when we get to it. Live demos are always, as you know, a bit tricky sometimes.  So my name is Thomas Rasmussen. I'm co-tuggling on the internet.  I'm born and live in Copenhagen in Denmark.  And I do system architecture, build and administer FreeBSD servers.  I do some programming as well and was recently laid off with three months with salary.  So I'm just going to fast them and relaxing and enjoying myself at the moment and looking for something else to do.  I run a DNS service called Uncensored DNS for 10 years now, 11 years, which is before Google DNS and before all the other public recursive DNS services.  I started this as an alternative to the ISP sensor, often censored ISP DNS servers.  And it's been running since I also help organize a bond hack, which is an annual Danish hacker camp.  I highly recommend going. It is a great time.  It's a week with tents and laptops and hacking around with all kinds of fun things.  And I have a bunch of minor open source projects on GitHub.  I use FreeBSD exclusively on servers. I've used this since 5.2.1, I think was my first installed server.  And on my laptops, I run something called CubeSOS, which is an awesome operating system.  I wish it was freeBSD based, but it's Fedora and send based.  The concept is you have different virtual machines for different contexts.  So I have one for work and one for playing with the bond hack stuff and they can really disposable browsers for surfing.  So the VM just disappears. So if you happen to click something bad, then it's just nothing really happens.  So awesome concept.  So I'm here to talk about this GSTAT exporter.  This is actually the first time I've been in the BSD room. There's never been room, but one of the perks of being a speaker is that you have to let me in.  So I wrote this tweet and someone picked up on it and recommended I come to foster and talk about it.  And here we are.  So Prometheus in their own words, it's an open source monitoring system, dimensional data model, flexible query language, efficient time series database.  And that is pretty much it.  It is very, the whole ecosystem around it is very unique see in that, that they do one thing and they do it well.  Prometheus in itself is a time series database.  It really doesn't do anything except collect metrics and stick them in a time series database and then have it has an API to make them available to Grafana, for example.  So you can pull the data back out.  It doesn't do alerting. It has something there's something called alert manager, which from the same team, but it's a separate separate pieces of software, separate configuration file and stuff, which handles alerting and alert dependencies and which teams should get what  alert and page to do the support and all that stuff.  And monitoring has often I think been many years been.  It's been a bit of a, like, there was a void left when Nagios kind of stopped being modern and I've been, I've been using subjects and a few other things, but nothing has really been.  It's like they tried to do too much.  Somehow it's a big job both to gather the data and visualize it and do alerting and everything and I haven't really been happy with anything.  You know, like the first time you tried set of us and thought this is what a file system should have been like all along. Where have you been all my life?  This is how I feel about managing with Prometheus. It is an excellent system.  So it is based on they call it dimensional time series data.  It means that your metrics can have labels.  They call it.  So if you have a metric called HTTP requests total, for example, for a web server, then and you get a number when you when you scrape the data.  Say you had a hundred requests.  Then that data can be can have multiple dimensions and that it can have labels to say what the path of the request was or what the HTTP status code was.  So you can ask to get only the HTTP for requests, for example, and then make a graph of those on your in your graph.  This is incredibly powerful.  It's actually a relatively simple concept, but it enriches the data greatly.  Prometheus and alert manager and all the accompanying software is all written and go.  It was.  It was started at the sound cloud some years ago. It's been open source since 50 2015 and there's no.  Company that owns it now. It's a regular open source project with many contributors and a great community and great IC channel and stuff.  So if you have questions, they're very happy to help.  So this is the diagram of the ecosystem from their website.  Central to the whole thing is the Prometheus server. It's a time series database.  And it stalls the metrics, of course, on some sort of disk storage.  It's a pole model. Prometheus. That's one of the things that some people has to get used to, which means that Prometheus connects to the stuff.  It's monitoring and pose the metrics over HTTP.  There's various ways around that if you absolutely cannot do it like that.  But that is the idea. For example, the free BST, just a little exporter that came and arrived in free BST 12 is supposed to be run in I not the under in at the and Prometheus connects to it and post out the metrics.  So and that that's what the exporters make available.  They open an HTTP endpoint and it's just a plain text key value list of lines.  I'll have examples later. There's something called push gateway that if you have a job that runs for an hour and then has some metrics to submit at the end of that, you can use push gateway for that.  It has a very extensive service discovery system.  So if you are AWS or Kubernetes user or you use one of the many, many cloud services available, there's probably a discovery service discovery thing available for it.  At work, I work at work and I'm not speaking until I got laid off.  We use the file service discovery thing and just exported a list of customers and IP addresses and that was automatically ingested by Prometheus.  So when we add a new customer in our central provisioning system, it automatically shows up and starts pulling the data from it.  And to talk to Prometheus and query the data and get it out, they invented something called prompt cool.  It is what doesn't really have anything to do with SQL.  It is a custom query language specifically for this and there will be examples of this later and it is a very powerful and easy to use language.  And like I said, alert manager is a separate component.  So when you add some threshold or something you want to be alerted on in Prometheus, it pushes an alert to alert manager and alert manager handles all the needy gritty with who is on vacation now and who has the duty and who does what.  So all that complexity is separate from Prometheus.  Okay, so we generally speak of black box and white box monitoring.  Black box monitoring is when you ping or whatever or just poke something from the outside check if it's listening on port 80 or whatever you might do.  So, for example, the previous stage is a good example or an exporter for GSTAT data, for example, means you're kind of inside the system, exposing the metrics to the outside.  And that's a much better way to do it than black box monitoring.  It has support for that as well. If you need to like we do at work ping 10,000 customers every five seconds, then you do that with something called black box exporter.  But mostly and primarily it is a white box monitoring.  It is supposed to be used with the exporters built into whatever you are monitoring.  So if you have a system like our provisioning system at work, it's easy to, for example, export metrics for the queue length for provisioning their customers, the router needs to be switched,  the queue needs to be configured and there's sometimes a queue if there's many changes at once. And it's very easy when you're in the code to export this metric and just do a slash permit use or slash metrics endpoint in your system and export it.  It's over HTTP, like I said, and it is designed to handle everything you can throw at it. You shouldn't be piggy. You should instrument all parts of your system.  Anything that might someday make sense. And even if you think it doesn't, because sometimes these subtle little changes and whatever can turn out to be important.  And then it's very nice to have the history. It is very efficient. It is designed to handle hundreds of thousands of metrics and on a single server commodity hardware.  It doesn't do, it has a little bit of, you can, when you're exploring the data in the Prometheus web interface, it can draw a simple crude graph, but it's not meant for graphing.  It's meant to be used with Grafana or something else, but I highly recommend it is excellent integrated into Grafana.  There's client libraries and many languages. The GStatics code I wrote is written in Python and you start out with these 10 lines or something from the Prometheus example client and you just add name, define your metrics saying we have a, in GStat we have a  disk IO, busy percent message metric or whatever and an inner feed data into them and it takes care of listening on HTTP and serving the data up in the right order.  And it is easy to use and they have of course in Go and many other languages, client libraries. So it is easy to get started writing your own exporter.  So it is like I said very efficient. They say about three and a half bytes per data point and it is remarkably efficient when pulling in data.  We went to, we replaced our Starbucks installation, which was thrashing the disks using Postgres as a backend.  I love Postgres. I don't love Starbucks, but it was a decent setup, but it was, I mean, we could barely handle polling them once per minute, the 5, 8,000 customers or however much we had back then.  And when we switched to Prometheus, we're polling them once every five seconds. We're doing additional metrics besides what we did with Starbucks and the server is bored now.  It's not doing nothing. So we went from absolutely thrashing the drives to the server being almost idle. It is amazingly efficient.  If you go beyond what one server can handle, it supports sharding and federation. So you can have many Prometheus servers working together and it does that very well.  And alert manager as well supports high availability stuff. So you can send an alert to any number of alert manager instances and only one of them will actually alert you.  At work, the Prometheus installation I was managing there had about 750,000 time series and about 13 and a half thousand per second metrics put into the database.  And that's small in Prometheus terms. That's not an issue at all. You can do that on a laptop or on whatever.  It's in a jail, all my stuff runs in jails and you can easily have like one for each team. It's not like managing an elk instance or something like that where you have to, you can just wave up an EO if you want one for the network team and one for the operations team or whatever.  You can easily do that.  So the query language from QL. I bought a few, few examples and it's easy to look up. It has a page or two and then if you understand basic, basic statistics and math, then it should be easy to use.  First things first, the HTTP requests total is the example metric we're using here. So that's a that's that was the internet connection that stopped working.  So that just returns the metric HTTP request total. If, for example, I want the metric, but only if the job label is API server and the handler is slave slash API slash comments, then I put it in the curly brackets and the only returns the number of HTTP requests that had those, those labels.  It also return a range vector. So you get, you get all the values for given for, for example, for a five minute interval. If you need that, you can use regular expressions for the labels and and you can do for matching and stuff for the for the label values as well.  I use those labels a lot in the G set exporter to save stuff like the disks serial numbers and stuff. We'll get to that later.  A bit more advanced. It has, it has some keywords to do. For example, it can calculate the weight over five minutes for, for this metric or even do a quantize 95th percentile is used a lot when when charging for bandwidth and hosting operations and stuff like that or whatever you need to to use it.  And it is, it is very flexible. It can also show the, you can query the, the alert, the number of alerts firing in a system and when, when the installation grows, that's the sort of stuff you want on your front page of your graphana instance, you can have zero or five alerts or whatever.  Okay. So, Prometheus connects to an exporter and gets fed metrics and ingests them. And these metrics are exported by, by exporters and machine metrics like, like the normal metrics like Ram and an IO and network traffic and stuff like that are exported by something called node exporter, which we have in ports and works well.  But on free BSE, it doesn't have this utilization metrics that has does on Linux, of course, but, but not on free BSE, which is why I started out because when you're troubleshooting stuff on a database server or something, it is very nice to have metrics for, for this utilization.  And it's very nice to have a list of exporters for every type of software you can almost imagine it is there's hundreds of them.  Each is allocated a port on this list, you just go to the weekend allocate a port and for the G start exporter, I got port 92 something, something.  And you can search for free BSE on it and find a few others that there's a jail exporter, which is very nice that somebody made that uses our CTO to get IOPS and memory use and stuff for for Jail's and then exports it in a, in a way so Prometheus can ingest it so you can get graphs to say how much Ram each jail is using and how many IOPS and stuff that is very nice if you're using Jails for yeah, if you're using Jails at all.  Okay, so, you know, I guess you all know G start.  It looks like this. It is a top style thing that shows the the gym devices and read and write operations at per default. It can show a delete and other stuff as well.  Actually have no wait. I don't because I'm not online.  Okay, so this is G start and these are the numbers I was interested in getting into a graph and that's where we're here today.  There's a dash C flag for for G start that makes it output in in CSV mode is called in the man page instead of the top style display and unfortunately a bit of a complaint would be that it ends it.  When you enable dash C, it also enables endless mode so it keeps running. You can just make it output values once and exit. So that's that should probably be added because usually in in exporters when Prometheus connects you scrape the data and then you return it.  But I've had to keep it running endlessly. The G start and just streaming the output from it into the collector.  Also, if G start could somehow export the counters, it bases the top display on.  So these are, you know, the difference between counters engages and and we lose a bit of precision because I'm reading the G start values as counters. It shows me how much there's an update frequency in G start.  So every five seconds, every one second or something and and stuff that happens in between, let's say every five seconds is kind of lost.  And if you had the counters instead Prometheus would be able to. It doesn't matter if you read every five or 10 seconds, then you would get all the information.  And I'm not a C person, but looking at the code, it looks like it is counters when it comes out of the kernel and and somewhere it is converted to the to the human friendly display that we see in G start.  Okay, so with the with G start, we really have everything we need with the dash C mode. We have everything we need to write on a write an exporter.  Like I said, the client library, Prometheus client libraries is easy to get started with the G start exporter is 240 lines.  It sounds like more than this because it's formatted with these code, beautifying tools and they are very generous with the new lines.  So it's on GitHub and yeah port 9248 is a it's a G static port.  It's not that they don't add them to services files everywhere because they're so dynamic and there's so many of them, but they do have the page on their website or the wiki on GitHub to keep track of which ports are used by what.  Okay, so running the exporter locally and and grabbing for a DA zero and grabbing for right. This is the output for for the G start exporter.  You see it's just plain text and this is just fetch outputting to to STD out. So it's a it's a plain text response and it's just lines of metric name and then labels and then finally a space and then a number.  Which is the actual value for that metric with that combination of labels for that specific time.  As you can see, I've stopped all the info I can get from the home.  It's called the home identifier or something. There's a home command that can return up here and all these need metrics so I can get.  Yeah, for example, sex or size and RPM and the serial number for the discs and the size.  Which is very nice because as we will see later in Grafana, it makes it easy to to filter, for example, and show only the NVMe drives only the spinning drives or only the 12 terabyte drives or whatever you need.  Okay, so as I said, it doesn't visualize stuff by itself, but the integration to Grafana is really good.  It can once you add the previous instance as a data source in Grafana, when you start typing a metric name, it auto completes it and it really works very, very well.  And you can filter a dashboard by label values and I'll show you what I mean by that later.  But that, for example, means that I can easily in Grafana just select one disc if I'm suspecting that one disc is foggy or select only the 5400 RPM drives across all servers if I should, for some reason, have the need for that.  I published the dashboard at the Grafana dashboard published in place and if you don't know Grafana, I highly recommend it.  It is very easy to make. It makes pretty graphs for you.  Like you don't have to decide on colors and stuff. It uses colors that look well together and generally just works extremely well.  Okay, I'm going to need to fix my internet connection. So talk about yourself for 30 seconds while I get this.  Okay.  Just work for the love of, sorry.  Yes, I would if I could. It's a long and boring explanation why it doesn't work.  And usually the phone just works, but of course, right now, it doesn't.  Yeah, thanks guys, but it's not going to work.  Really? This is great advertisement for Linux.  Well, cubes is special.  It is a bit.  But it really isn't.  I'm glad I want that this might happen ahead of time.  Yes.  Okay.  So maybe we skipped a live demo since it's not working. I'm sorry about that.  I'll talk a bit more and then try again in a few minutes when it's time to recover.  I do have some screenshots in the beginning that I can go back to. But oh, yeah, thanks.  It was a pretty quick project. I did it in a day.  Started in the morning investigating how I would do it.  And then I wrote the code during the day and in the evening I did the dashboard and then I wrote a blog post and posted it.  So it was in the scope of a day you can write an export of something including doing a nice dashboard and making everything work.  If GSTAT gets support for outputting just align and then exiting, I'll probably change the way it calls GSTAT.  So it doesn't keep it running perpetually, but just calls it when it is scraped.  And maybe it could add flags to support if there's a need for it.  GSTAT can show consumers as well as producers in the GEM stack.  And some people might need this for reasons I can't really.  The Grafana dashboard needs some fine tuning.  And if anybody wants to...  Oh, fuck off.  If anybody feels like helping with that, then you're very welcome.  I'm going to get this one quick final attempt.  Well, I have it open fortunately so we can show you some of the things I was talking about.  The labels I add to the...  The labels I add to the metrics can be used at Grafana to make this nice little table in the top that shows this is...  In the top we have the filters that Grafana makes it possible to filter by label values.  So this shows server names and this shows the different GEMs and the different disk sizes we have.  And sector sizes and RPM and stuff.  This is where I would, for example, I didn't know we still had a 5400 RPM drive, but after I installed this, then I got a bit better overview of the entire...  And it's a nice way to see, for example, serial number.  If you know what disk is performing poorly, you can easily see the serial number and ask your on-site hands to fix it.  And other than that, there's sections for each of the metrics GeoMix ports.  So there's a latency graph for read, write and delete latency.  And there's a bandwidth section for...  Of course it can't.  Trust me, there's bandwidth graph behind if I was online.  And there's an IAPS section and the Q-depth.  This is showing 30 days.  I've had this running for a few months since I wrote it.  And one of the fun things is that if you ran a very large set of S-pools, you probably know that the scrubbing can take almost forever to complete.  Which means that many people with large pools end up doing a nightly scrub job that just runs it through the night and poses it in the working hours.  And that's the fun. This is the busy present.  And the scrubs are these... This is 30 days.  So you can see we scrub all weekend and then every night.  And then all weekend and every night.  So it's a fun pattern to observe.  And we found a bad disk in one of the servers using this.  I can't find the data now, of course, but it works really well to get an overview of the disk utilization.  And that's actually... That's all I wanted.  So that's very nice.  Any questions?  We have plenty of time. Yes?  My OCD screams at me that the media size should be just a number without the string.  Yeah, but it's exported directly from the string that the geom info exports.  So I could cut it off and then sell Grafana that it's a byte value and it would humanize it for me.  You're right. That would be easy.  Grafana knows... It knows units.  So this graph, I've told it that this is a number of seconds and then it shows all the scales, the y-axis to show that this is milliseconds.  Of course, not seconds, fortunately.  And it understands megabytes per second and stuff like that.  So it is very lovely to work with.  Yes?  Have you had any trouble with Prometheus or Grafana because you're on FreeBSD instead of Linux?  Apart from it not exporting disk utilization?  No.  Oh, right. The question was, have I had any problems running a Grafana or Prometheus on FreeBSD because it's FreeBSD and not Linux?  No. Apart from the node exporter, that's the thing that exposes system metrics like memory and disk IO and stuff like that.  Apart from it not exporting disk IO and not exporting ZFS data, then no, everything works very well.  The ports work perfectly and they are updated timely and run very well.  The latest version or the upcoming version of node exporter does have FreeBSD ZFS support and I'm very much looking forward to that.  And combined with this, then you can get a great impression of how your storage is doing.  Yeah?  Yes?  I have questions in a bit disjoint topics.  The first one is, you were touching keywords like alerting and federation and this kind of stuff.  So my question is about these servers what you were scraping.  They were only scraping endpoints or they were scraped and processed at the same site.  Or you had one central Prometheus server and then scraping all the endpoints.  Yes, one central server that connects out to, you configure it, tell it which targets to scrape and then it just sucks up all the metrics that are available at that endpoint.  Okay.  Have it meant some problem with delay or the latency?  No, I mean, you would think, but no, it just really works very well.  It's a bit of a radical, both the pull model and the idea that it does the scrape as it connects.  So if FreeBSD doesn't connect for a while, then there's no data collected.  But it really works very well and turns out that it doesn't matter too much as long as you use counters, it would matter for GSTAT exporter because you would lose the precision if you reboot your Prometheus server, then you would lose the five minutes or whatever it takes.  But if you have counters available, then you don't because it doesn't matter if Prometheus has gone for a while.  It'll figure out how to, what's the word I'm looking for, put.  Yeah, exactly.  It'll smooth it out.  So it works very well and people are doing some like the big hosting operators.  I have some like 500 Prometheus servers doing millions and millions of data points per second.  Some crazy big instances.  So for my users with 100 servers or something, I know it's always nice to have to know that it can handle way more than you need.  This also means that you have one central server, so you have to maintain your alerts on one server, right?  So the wire tools or the alert conditions have to be written on one server.  Well, the alert manager can, is a separate jail.  I mean, it could be a separate server if you want it.  Yeah, I mean, okay.  Yeah, a different question in the intro of your talk.  You mentioned that you take care of this BornHack camp.  Yes.  What's the topic that attracts?  Yeah, it is stuff like this.  It is InfoSec.  It is also hardware, open source and really brought a few, if you've been to the German CCC camp or the Dutch Big Hacker camps, it is very much inspired by those.  So it is kind of reminds a little bit of first time except that people live in tents and there's not as many people, of course.  About 400 people I think this year and it's slowly growing.  It's the fifth year this time.  So, and it's an annual event and it's getting traction.  It's a lovely little event.  So, any more questions?  Yes.  Yeah, I was trying to find a monitoring solution and using to have a good experience from you first.  Absolutely.  Did you try some of this competing text that you know telegraph and infox.  Have you seen about that?  No, not really.  I haven't used it enough to, to have not set it up.  I've only briefly been exposed to it when somebody else had configured it.  So I don't know enough.  But you know how you can sometimes sense that something has momentum like a project like every, I think everywhere I look, Prometheus exporters are popping up like natively inside whatever engine X or whatever.  Suddenly has a way to explore and and that tells me that, you know, I'm not the only one who feels that this is the right approach.  And that's very nice because I think we have been, I've been looking for something nice for months or in four years and years.  I've never been really, really happy even back when Nagius was the thing to run.  I've never really been very happy with it.  I think everybody really kind of loved to hate it.  So I am so, so happy that finally something and it also, I mean, the aesthetics are also very important because even though technically it's the same numbers.  If it's an acrylic graph or a pretty graph, but it just still makes a difference.  It's nice to work with.  I have one response to that.  So just looking at influx while it's nice to work with in at least some aspects.  Prometheus has much, much more advanced math available.  Like for example, in influx, it's not possible in a single query to get.  For example, you get the return codes of HTTP requests.  In influx, it's not possible to get what proportion of requests were errors.  It's possible in graphite.  It's possible in Prometheus.  It's not possible in influx.  In jade graph works with Pymus.  It's one.  Thank you.  Any more questions?  I have a quick question because I'm rarely surrounded by a lot of previous people.  So I just.  Oh, damn, I went offline.  Oh, and I can't even show it.  Okay.  Well, the question, though, the problem is, you know, the jail exporter I was talking about, which makes it with users.  I'll see to export jail metrics and then then you can do graphs for, for example, memory use inside a jail.  So I use that, of course, because I use jails for all my stuff.  And it's very nice to see if a disk is being trashed.  You can easily see which jail is causing this postgres doing an auto vacuum or what's going on.  RCTL counts shared memory twice in jails or as many times I have what I have in this tab that I can switch over to because I'm offline.  I have two jails on this server.  It has a hundred hundred and twenty eight gigabytes of RAM.  I have two postgres jails and RCTL tells me that one jail is using about 130 gigs and the other jail is using 80 gigs, which is about 200 gigs.  And there's 40 more jails and it's so clearly something is wrong.  And I've been able to track down as much as it is.  I don't know what the fix is, but it is absolutely misreporting memory use in jails.  And since if I have a hundred postgres workers in each of them is using two gigabytes of RAM, then most of it is shared.  What you want RCTL to export is how much memory would be freed if I shut the jail down.  I don't care if a hundred workers is using two gigabytes.  You shouldn't report 200 gigabytes memory use.  Does anybody know the is this just a bug or is there is this not possible?  What I'm asking for.  I mean, not between jails between between a hundred postgres workers in the same jail, for example, and then each of those they have access to the same shared memory space.  And it's been counted a hundred times because there's a hundred workers.  So it tells me that one jail is using 140 gigs of RAM on 128 gig server, which clearly is not true.  I wonder how it would look like if it was running outside of jail if it would be reported the same way?  It would.  But we can talk about it later after I just it's been bugging me because.  I finally got some nice jail graphs and the memory if use it is just way, way off.  I was like, what the hell is wrong?  How can it be using 300 gigabytes of RAM when it has 128 and RCTL exports the V memory use as well?  And that's of course a high number and higher than the as 1.2 terabytes of memory of the memory.  I think.  But the memory use should be.  We agree that it should be however much would be freed if the jail was shut down.  Right.  Otherwise, what's the point of reporting them?  I mean, you see my you see what I mean?  Okay.  I think I'll open up PR and write to a mailing list and see if I can.  I think I can catch somebody's attention with that.  But other than that, the Jail is really awesome and stuff like IOPS and that works very well.  So you can see which which Jail is trashing at this goal or using a lot of CPU.  I think that's it for me.  We're done with a quarter or quarter of an hour to spare.  So we're going to use that time to for self reflection and the self improvement.  And thanks for having me and I hope you enjoyed first them and my talk and.  Thank you.
freebsd_prometheus_0_155.png freebsd_prometheus_156_287.png freebsd_prometheus_288_1115.png freebsd_prometheus_1116_10211.png freebsd_prometheus_10212_10691.png freebsd_prometheus_10692_10763.png freebsd_prometheus_10764_14939.png freebsd_prometheus_14940_15647.png freebsd_prometheus_15648_15695.png freebsd_prometheus_15696_16127.png freebsd_prometheus_16128_19583.png freebsd_prometheus_19584_19847.png freebsd_prometheus_19848_28763.png freebsd_prometheus_28764_29147.png freebsd_prometheus_29148_35819.png freebsd_prometheus_35820_36035.png freebsd_prometheus_36036_37007.png freebsd_prometheus_37008_37943.png freebsd_prometheus_37944_38015.png freebsd_prometheus_38016_38111.png freebsd_prometheus_38112_38159.png freebsd_prometheus_38160_38363.png freebsd_prometheus_38364_38423.png freebsd_prometheus_38424_38783.png freebsd_prometheus_38784_39035.png freebsd_prometheus_39036_39131.png freebsd_prometheus_39132_39215.png freebsd_prometheus_39216_39803.png freebsd_prometheus_39804_39851.png freebsd_prometheus_39852_40019.png freebsd_prometheus_40020_40247.png freebsd_prometheus_40248_40895.png freebsd_prometheus_40896_40943.png freebsd_prometheus_40944_41123.png freebsd_prometheus_41124_41243.png freebsd_prometheus_41244_41555.png freebsd_prometheus_41556_41627.png freebsd_prometheus_41628_42215.png freebsd_prometheus_42216_42275.png freebsd_prometheus_42276_42551.png freebsd_prometheus_42552_42779.png freebsd_prometheus_42780_45071.png freebsd_prometheus_45072_45179.png freebsd_prometheus_45180_45503.png freebsd_prometheus_45504_45551.png freebsd_prometheus_45552_47231.png freebsd_prometheus_47232_47867.png freebsd_prometheus_47868_47927.png freebsd_prometheus_47928_48203.png freebsd_prometheus_48204_50447.png freebsd_prometheus_50448_52511.png freebsd_prometheus_52512_61799.png freebsd_prometheus_61800_63035.png freebsd_prometheus_63036_66095.png freebsd_prometheus_66096_66323.png freebsd_prometheus_66324_66371.png freebsd_prometheus_66372_66467.png freebsd_prometheus_66468_67907.png freebsd_prometheus_67908_67955.png freebsd_prometheus_67956_68027.png freebsd_prometheus_68028_68147.png
Thank you. Hello, everybody. We have plenty of time. I submitted this as a lightning talk and was allocated an hour. So if you have questions or comments or anything, please feel free to interrupt me. Does anybody use Prometheus already? Or is this new for everybody? A few people? You heard of it? Great. We got a Prometheus exporter in previous year 12. Somebody in the previous year world at least knows and uses Prometheus apart from me. A few months ago, I wrote an exporter to make disk utilization metrics available to Prometheus from FreeBSD. And I was encouraged to come here and talk about it. I did a blog post about it and was encouraged to come here and tell you. Since many people probably don't know Prometheus, I'll start out by talking a little bit about what it is and what it does and why it's awesome. And then the concept of exporters. And then we'll get to GSTAD and the actual exporter that I wrote for that. I also have a Grafana dashboard to actually visualize the data connected. And if everything works and my internet connection keeps working, then I have a live demo of some servers at my workplace using this. So it's always nice to see some actual real data. But I'm not making any promises because my connection has been flaggy. So let's see when we get to it. Live demos are always, as you know, a bit tricky sometimes. So my name is Thomas Rasmussen. I'm co-tuggling on the internet. I'm born and live in Copenhagen in Denmark. And I do system architecture, build and administer FreeBSD servers. I do some programming as well and was recently laid off with three months with salary. So I'm just going to fast them and relaxing and enjoying myself at the moment and looking for something else to do. I run a DNS service called Uncensored DNS for 10 years now, 11 years, which is before Google DNS and before all the other public recursive DNS services. I started this as an alternative to the ISP sensor, often censored ISP DNS servers. And it's been running since I also help organize a bond hack, which is an annual Danish hacker camp. I highly recommend going. It is a great time. It's a week with tents and laptops and hacking around with all kinds of fun things. And I have a bunch of minor open source projects on GitHub. I use FreeBSD exclusively on servers. I've used this since 5.2.1, I think was my first installed server. And on my laptops, I run something called CubeSOS, which is an awesome operating system. I wish it was freeBSD based, but it's Fedora and send based. The concept is you have different virtual machines for different contexts. So I have one for work and one for playing with the bond hack stuff and they can really disposable browsers for surfing. So the VM just disappears. So if you happen to click something bad, then it's just nothing really happens. So awesome concept. So I'm here to talk about this GSTAT exporter. This is actually the first time I've been in the BSD room. There's never been room, but one of the perks of being a speaker is that you have to let me in. So I wrote this tweet and someone picked up on it and recommended I come to foster and talk about it. And here we are. So Prometheus in their own words, it's an open source monitoring system, dimensional data model, flexible query language, efficient time series database. And that is pretty much it. It is very, the whole ecosystem around it is very unique see in that, that they do one thing and they do it well. Prometheus in itself is a time series database. It really doesn't do anything except collect metrics and stick them in a time series database and then have it has an API to make them available to Grafana, for example. So you can pull the data back out. It doesn't do alerting. It has something there's something called alert manager, which from the same team, but it's a separate separate pieces of software, separate configuration file and stuff, which handles alerting and alert dependencies and which teams should get what alert and page to do the support and all that stuff. And monitoring has often I think been many years been. It's been a bit of a, like, there was a void left when Nagios kind of stopped being modern and I've been, I've been using subjects and a few other things, but nothing has really been. It's like they tried to do too much. Somehow it's a big job both to gather the data and visualize it and do alerting and everything and I haven't really been happy with anything. You know, like the first time you tried set of us and thought this is what a file system should have been like all along. Where have you been all my life? This is how I feel about managing with Prometheus. It is an excellent system. So it is based on they call it dimensional time series data. It means that your metrics can have labels. They call it. So if you have a metric called HTTP requests total, for example, for a web server, then and you get a number when you when you scrape the data. Say you had a hundred requests. Then that data can be can have multiple dimensions and that it can have labels to say what the path of the request was or what the HTTP status code was. So you can ask to get only the HTTP for requests, for example, and then make a graph of those on your in your graph. This is incredibly powerful. It's actually a relatively simple concept, but it enriches the data greatly. Prometheus and alert manager and all the accompanying software is all written and go. It was. It was started at the sound cloud some years ago. It's been open source since 50 2015 and there's no. Company that owns it now. It's a regular open source project with many contributors and a great community and great IC channel and stuff. So if you have questions, they're very happy to help. So this is the diagram of the ecosystem from their website. Central to the whole thing is the Prometheus server. It's a time series database. And it stalls the metrics, of course, on some sort of disk storage. It's a pole model. Prometheus. That's one of the things that some people has to get used to, which means that Prometheus connects to the stuff. It's monitoring and pose the metrics over HTTP. There's various ways around that if you absolutely cannot do it like that. But that is the idea. For example, the free BST, just a little exporter that came and arrived in free BST 12 is supposed to be run in I not the under in at the and Prometheus connects to it and post out the metrics. So and that that's what the exporters make available. They open an HTTP endpoint and it's just a plain text key value list of lines. I'll have examples later. There's something called push gateway that if you have a job that runs for an hour and then has some metrics to submit at the end of that, you can use push gateway for that. It has a very extensive service discovery system. So if you are AWS or Kubernetes user or you use one of the many, many cloud services available, there's probably a discovery service discovery thing available for it. At work, I work at work and I'm not speaking until I got laid off. We use the file service discovery thing and just exported a list of customers and IP addresses and that was automatically ingested by Prometheus. So when we add a new customer in our central provisioning system, it automatically shows up and starts pulling the data from it. And to talk to Prometheus and query the data and get it out, they invented something called prompt cool. It is what doesn't really have anything to do with SQL. It is a custom query language specifically for this and there will be examples of this later and it is a very powerful and easy to use language. And like I said, alert manager is a separate component. So when you add some threshold or something you want to be alerted on in Prometheus, it pushes an alert to alert manager and alert manager handles all the needy gritty with who is on vacation now and who has the duty and who does what. So all that complexity is separate from Prometheus. Okay, so we generally speak of black box and white box monitoring. Black box monitoring is when you ping or whatever or just poke something from the outside check if it's listening on port 80 or whatever you might do. So, for example, the previous stage is a good example or an exporter for GSTAT data, for example, means you're kind of inside the system, exposing the metrics to the outside. And that's a much better way to do it than black box monitoring. It has support for that as well. If you need to like we do at work ping 10,000 customers every five seconds, then you do that with something called black box exporter. But mostly and primarily it is a white box monitoring. It is supposed to be used with the exporters built into whatever you are monitoring. So if you have a system like our provisioning system at work, it's easy to, for example, export metrics for the queue length for provisioning their customers, the router needs to be switched, the queue needs to be configured and there's sometimes a queue if there's many changes at once. And it's very easy when you're in the code to export this metric and just do a slash permit use or slash metrics endpoint in your system and export it. It's over HTTP, like I said, and it is designed to handle everything you can throw at it. You shouldn't be piggy. You should instrument all parts of your system. Anything that might someday make sense. And even if you think it doesn't, because sometimes these subtle little changes and whatever can turn out to be important. And then it's very nice to have the history. It is very efficient. It is designed to handle hundreds of thousands of metrics and on a single server commodity hardware. It doesn't do, it has a little bit of, you can, when you're exploring the data in the Prometheus web interface, it can draw a simple crude graph, but it's not meant for graphing. It's meant to be used with Grafana or something else, but I highly recommend it is excellent integrated into Grafana. There's client libraries and many languages. The GStatics code I wrote is written in Python and you start out with these 10 lines or something from the Prometheus example client and you just add name, define your metrics saying we have a, in GStat we have a disk IO, busy percent message metric or whatever and an inner feed data into them and it takes care of listening on HTTP and serving the data up in the right order. And it is easy to use and they have of course in Go and many other languages, client libraries. So it is easy to get started writing your own exporter. So it is like I said very efficient. They say about three and a half bytes per data point and it is remarkably efficient when pulling in data. We went to, we replaced our Starbucks installation, which was thrashing the disks using Postgres as a backend. I love Postgres. I don't love Starbucks, but it was a decent setup, but it was, I mean, we could barely handle polling them once per minute, the 5, 8,000 customers or however much we had back then. And when we switched to Prometheus, we're polling them once every five seconds. We're doing additional metrics besides what we did with Starbucks and the server is bored now. It's not doing nothing. So we went from absolutely thrashing the drives to the server being almost idle. It is amazingly efficient. If you go beyond what one server can handle, it supports sharding and federation. So you can have many Prometheus servers working together and it does that very well. And alert manager as well supports high availability stuff. So you can send an alert to any number of alert manager instances and only one of them will actually alert you. At work, the Prometheus installation I was managing there had about 750,000 time series and about 13 and a half thousand per second metrics put into the database. And that's small in Prometheus terms. That's not an issue at all. You can do that on a laptop or on whatever. It's in a jail, all my stuff runs in jails and you can easily have like one for each team. It's not like managing an elk instance or something like that where you have to, you can just wave up an EO if you want one for the network team and one for the operations team or whatever. You can easily do that. So the query language from QL. I bought a few, few examples and it's easy to look up. It has a page or two and then if you understand basic, basic statistics and math, then it should be easy to use. First things first, the HTTP requests total is the example metric we're using here. So that's a that's that was the internet connection that stopped working. So that just returns the metric HTTP request total. If, for example, I want the metric, but only if the job label is API server and the handler is slave slash API slash comments, then I put it in the curly brackets and the only returns the number of HTTP requests that had those, those labels. It also return a range vector. So you get, you get all the values for given for, for example, for a five minute interval. If you need that, you can use regular expressions for the labels and and you can do for matching and stuff for the for the label values as well. I use those labels a lot in the G set exporter to save stuff like the disks serial numbers and stuff. We'll get to that later. A bit more advanced. It has, it has some keywords to do. For example, it can calculate the weight over five minutes for, for this metric or even do a quantize 95th percentile is used a lot when when charging for bandwidth and hosting operations and stuff like that or whatever you need to to use it. And it is, it is very flexible. It can also show the, you can query the, the alert, the number of alerts firing in a system and when, when the installation grows, that's the sort of stuff you want on your front page of your graphana instance, you can have zero or five alerts or whatever. Okay. So, Prometheus connects to an exporter and gets fed metrics and ingests them. And these metrics are exported by, by exporters and machine metrics like, like the normal metrics like Ram and an IO and network traffic and stuff like that are exported by something called node exporter, which we have in ports and works well. But on free BSE, it doesn't have this utilization metrics that has does on Linux, of course, but, but not on free BSE, which is why I started out because when you're troubleshooting stuff on a database server or something, it is very nice to have metrics for, for this utilization. And it's very nice to have a list of exporters for every type of software you can almost imagine it is there's hundreds of them. Each is allocated a port on this list, you just go to the weekend allocate a port and for the G start exporter, I got port 92 something, something. And you can search for free BSE on it and find a few others that there's a jail exporter, which is very nice that somebody made that uses our CTO to get IOPS and memory use and stuff for for Jail's and then exports it in a, in a way so Prometheus can ingest it so you can get graphs to say how much Ram each jail is using and how many IOPS and stuff that is very nice if you're using Jails for yeah, if you're using Jails at all. Okay, so, you know, I guess you all know G start. It looks like this. It is a top style thing that shows the the gym devices and read and write operations at per default. It can show a delete and other stuff as well. Actually have no wait. I don't because I'm not online. Okay, so this is G start and these are the numbers I was interested in getting into a graph and that's where we're here today. There's a dash C flag for for G start that makes it output in in CSV mode is called in the man page instead of the top style display and unfortunately a bit of a complaint would be that it ends it. When you enable dash C, it also enables endless mode so it keeps running. You can just make it output values once and exit. So that's that should probably be added because usually in in exporters when Prometheus connects you scrape the data and then you return it. But I've had to keep it running endlessly. The G start and just streaming the output from it into the collector. Also, if G start could somehow export the counters, it bases the top display on. So these are, you know, the difference between counters engages and and we lose a bit of precision because I'm reading the G start values as counters. It shows me how much there's an update frequency in G start. So every five seconds, every one second or something and and stuff that happens in between, let's say every five seconds is kind of lost. And if you had the counters instead Prometheus would be able to. It doesn't matter if you read every five or 10 seconds, then you would get all the information. And I'm not a C person, but looking at the code, it looks like it is counters when it comes out of the kernel and and somewhere it is converted to the to the human friendly display that we see in G start. Okay, so with the with G start, we really have everything we need with the dash C mode. We have everything we need to write on a write an exporter. Like I said, the client library, Prometheus client libraries is easy to get started with the G start exporter is 240 lines. It sounds like more than this because it's formatted with these code, beautifying tools and they are very generous with the new lines. So it's on GitHub and yeah port 9248 is a it's a G static port. It's not that they don't add them to services files everywhere because they're so dynamic and there's so many of them, but they do have the page on their website or the wiki on GitHub to keep track of which ports are used by what. Okay, so running the exporter locally and and grabbing for a DA zero and grabbing for right. This is the output for for the G start exporter. You see it's just plain text and this is just fetch outputting to to STD out. So it's a it's a plain text response and it's just lines of metric name and then labels and then finally a space and then a number. Which is the actual value for that metric with that combination of labels for that specific time. As you can see, I've stopped all the info I can get from the home. It's called the home identifier or something. There's a home command that can return up here and all these need metrics so I can get. Yeah, for example, sex or size and RPM and the serial number for the discs and the size. Which is very nice because as we will see later in Grafana, it makes it easy to to filter, for example, and show only the NVMe drives only the spinning drives or only the 12 terabyte drives or whatever you need. Okay, so as I said, it doesn't visualize stuff by itself, but the integration to Grafana is really good. It can once you add the previous instance as a data source in Grafana, when you start typing a metric name, it auto completes it and it really works very, very well. And you can filter a dashboard by label values and I'll show you what I mean by that later. But that, for example, means that I can easily in Grafana just select one disc if I'm suspecting that one disc is foggy or select only the 5400 RPM drives across all servers if I should, for some reason, have the need for that. I published the dashboard at the Grafana dashboard published in place and if you don't know Grafana, I highly recommend it. It is very easy to make. It makes pretty graphs for you. Like you don't have to decide on colors and stuff. It uses colors that look well together and generally just works extremely well. Okay, I'm going to need to fix my internet connection. So talk about yourself for 30 seconds while I get this. Okay. Just work for the love of, sorry. Yes, I would if I could. It's a long and boring explanation why it doesn't work. And usually the phone just works, but of course, right now, it doesn't. Yeah, thanks guys, but it's not going to work. Really? This is great advertisement for Linux. Well, cubes is special. It is a bit. But it really isn't. I'm glad I want that this might happen ahead of time. Yes. Okay. So maybe we skipped a live demo since it's not working. I'm sorry about that. I'll talk a bit more and then try again in a few minutes when it's time to recover. I do have some screenshots in the beginning that I can go back to. But oh, yeah, thanks. It was a pretty quick project. I did it in a day. Started in the morning investigating how I would do it. And then I wrote the code during the day and in the evening I did the dashboard and then I wrote a blog post and posted it. So it was in the scope of a day you can write an export of something including doing a nice dashboard and making everything work. If GSTAT gets support for outputting just align and then exiting, I'll probably change the way it calls GSTAT. So it doesn't keep it running perpetually, but just calls it when it is scraped. And maybe it could add flags to support if there's a need for it. GSTAT can show consumers as well as producers in the GEM stack. And some people might need this for reasons I can't really. The Grafana dashboard needs some fine tuning. And if anybody wants to... Oh, fuck off. If anybody feels like helping with that, then you're very welcome. I'm going to get this one quick final attempt. Well, I have it open fortunately so we can show you some of the things I was talking about. The labels I add to the... The labels I add to the metrics can be used at Grafana to make this nice little table in the top that shows this is... In the top we have the filters that Grafana makes it possible to filter by label values. So this shows server names and this shows the different GEMs and the different disk sizes we have. And sector sizes and RPM and stuff. This is where I would, for example, I didn't know we still had a 5400 RPM drive, but after I installed this, then I got a bit better overview of the entire... And it's a nice way to see, for example, serial number. If you know what disk is performing poorly, you can easily see the serial number and ask your on-site hands to fix it. And other than that, there's sections for each of the metrics GeoMix ports. So there's a latency graph for read, write and delete latency. And there's a bandwidth section for... Of course it can't. Trust me, there's bandwidth graph behind if I was online. And there's an IAPS section and the Q-depth. This is showing 30 days. I've had this running for a few months since I wrote it. And one of the fun things is that if you ran a very large set of S-pools, you probably know that the scrubbing can take almost forever to complete. Which means that many people with large pools end up doing a nightly scrub job that just runs it through the night and poses it in the working hours. And that's the fun. This is the busy present. And the scrubs are these... This is 30 days. So you can see we scrub all weekend and then every night. And then all weekend and every night. So it's a fun pattern to observe. And we found a bad disk in one of the servers using this. I can't find the data now, of course, but it works really well to get an overview of the disk utilization. And that's actually... That's all I wanted. So that's very nice. Any questions? We have plenty of time. Yes? My OCD screams at me that the media size should be just a number without the string. Yeah, but it's exported directly from the string that the geom info exports. So I could cut it off and then sell Grafana that it's a byte value and it would humanize it for me. You're right. That would be easy. Grafana knows... It knows units. So this graph, I've told it that this is a number of seconds and then it shows all the scales, the y-axis to show that this is milliseconds. Of course, not seconds, fortunately. And it understands megabytes per second and stuff like that. So it is very lovely to work with. Yes? Have you had any trouble with Prometheus or Grafana because you're on FreeBSD instead of Linux? Apart from it not exporting disk utilization? No. Oh, right. The question was, have I had any problems running a Grafana or Prometheus on FreeBSD because it's FreeBSD and not Linux? No. Apart from the node exporter, that's the thing that exposes system metrics like memory and disk IO and stuff like that. Apart from it not exporting disk IO and not exporting ZFS data, then no, everything works very well. The ports work perfectly and they are updated timely and run very well. The latest version or the upcoming version of node exporter does have FreeBSD ZFS support and I'm very much looking forward to that. And combined with this, then you can get a great impression of how your storage is doing. Yeah? Yes? I have questions in a bit disjoint topics. The first one is, you were touching keywords like alerting and federation and this kind of stuff. So my question is about these servers what you were scraping. They were only scraping endpoints or they were scraped and processed at the same site. Or you had one central Prometheus server and then scraping all the endpoints. Yes, one central server that connects out to, you configure it, tell it which targets to scrape and then it just sucks up all the metrics that are available at that endpoint. Okay. Have it meant some problem with delay or the latency? No, I mean, you would think, but no, it just really works very well. It's a bit of a radical, both the pull model and the idea that it does the scrape as it connects. So if FreeBSD doesn't connect for a while, then there's no data collected. But it really works very well and turns out that it doesn't matter too much as long as you use counters, it would matter for GSTAT exporter because you would lose the precision if you reboot your Prometheus server, then you would lose the five minutes or whatever it takes. But if you have counters available, then you don't because it doesn't matter if Prometheus has gone for a while. It'll figure out how to, what's the word I'm looking for, put. Yeah, exactly. It'll smooth it out. So it works very well and people are doing some like the big hosting operators. I have some like 500 Prometheus servers doing millions and millions of data points per second. Some crazy big instances. So for my users with 100 servers or something, I know it's always nice to have to know that it can handle way more than you need. This also means that you have one central server, so you have to maintain your alerts on one server, right? So the wire tools or the alert conditions have to be written on one server. Well, the alert manager can, is a separate jail. I mean, it could be a separate server if you want it. Yeah, I mean, okay. Yeah, a different question in the intro of your talk. You mentioned that you take care of this BornHack camp. Yes. What's the topic that attracts? Yeah, it is stuff like this. It is InfoSec. It is also hardware, open source and really brought a few, if you've been to the German CCC camp or the Dutch Big Hacker camps, it is very much inspired by those. So it is kind of reminds a little bit of first time except that people live in tents and there's not as many people, of course. About 400 people I think this year and it's slowly growing. It's the fifth year this time. So, and it's an annual event and it's getting traction. It's a lovely little event. So, any more questions? Yes. Yeah, I was trying to find a monitoring solution and using to have a good experience from you first. Absolutely. Did you try some of this competing text that you know telegraph and infox. Have you seen about that? No, not really. I haven't used it enough to, to have not set it up. I've only briefly been exposed to it when somebody else had configured it. So I don't know enough. But you know how you can sometimes sense that something has momentum like a project like every, I think everywhere I look, Prometheus exporters are popping up like natively inside whatever engine X or whatever. Suddenly has a way to explore and and that tells me that, you know, I'm not the only one who feels that this is the right approach. And that's very nice because I think we have been, I've been looking for something nice for months or in four years and years. I've never been really, really happy even back when Nagius was the thing to run. I've never really been very happy with it. I think everybody really kind of loved to hate it. So I am so, so happy that finally something and it also, I mean, the aesthetics are also very important because even though technically it's the same numbers. If it's an acrylic graph or a pretty graph, but it just still makes a difference. It's nice to work with. I have one response to that. So just looking at influx while it's nice to work with in at least some aspects. Prometheus has much, much more advanced math available. Like for example, in influx, it's not possible in a single query to get. For example, you get the return codes of HTTP requests. In influx, it's not possible to get what proportion of requests were errors. It's possible in graphite. It's possible in Prometheus. It's not possible in influx. In jade graph works with Pymus. It's one. Thank you. Any more questions? I have a quick question because I'm rarely surrounded by a lot of previous people. So I just. Oh, damn, I went offline. Oh, and I can't even show it. Okay. Well, the question, though, the problem is, you know, the jail exporter I was talking about, which makes it with users. I'll see to export jail metrics and then then you can do graphs for, for example, memory use inside a jail. So I use that, of course, because I use jails for all my stuff. And it's very nice to see if a disk is being trashed. You can easily see which jail is causing this postgres doing an auto vacuum or what's going on. RCTL counts shared memory twice in jails or as many times I have what I have in this tab that I can switch over to because I'm offline. I have two jails on this server. It has a hundred hundred and twenty eight gigabytes of RAM. I have two postgres jails and RCTL tells me that one jail is using about 130 gigs and the other jail is using 80 gigs, which is about 200 gigs. And there's 40 more jails and it's so clearly something is wrong. And I've been able to track down as much as it is. I don't know what the fix is, but it is absolutely misreporting memory use in jails. And since if I have a hundred postgres workers in each of them is using two gigabytes of RAM, then most of it is shared. What you want RCTL to export is how much memory would be freed if I shut the jail down. I don't care if a hundred workers is using two gigabytes. You shouldn't report 200 gigabytes memory use. Does anybody know the is this just a bug or is there is this not possible? What I'm asking for. I mean, not between jails between between a hundred postgres workers in the same jail, for example, and then each of those they have access to the same shared memory space. And it's been counted a hundred times because there's a hundred workers. So it tells me that one jail is using 140 gigs of RAM on 128 gig server, which clearly is not true. I wonder how it would look like if it was running outside of jail if it would be reported the same way? It would. But we can talk about it later after I just it's been bugging me because. I finally got some nice jail graphs and the memory if use it is just way, way off. I was like, what the hell is wrong? How can it be using 300 gigabytes of RAM when it has 128 and RCTL exports the V memory use as well? And that's of course a high number and higher than the as 1.2 terabytes of memory of the memory. I think. But the memory use should be. We agree that it should be however much would be freed if the jail was shut down. Right. Otherwise, what's the point of reporting them? I mean, you see my you see what I mean? Okay. I think I'll open up PR and write to a mailing list and see if I can. I think I can catch somebody's attention with that. But other than that, the Jail is really awesome and stuff like IOPS and that works very well. So you can see which which Jail is trashing at this goal or using a lot of CPU. I think that's it for me. We're done with a quarter or quarter of an hour to spare. So we're going to use that time to for self reflection and the self improvement. And thanks for having me and I hope you enjoyed first them and my talk and. Thank you.
have a Grafana dashboard to actually visualize stuff for for Jail's and then exports it in a, in