Network Monitoring with InfluxDB 2 and Telegraf
McKay, David
Telegraf is an agent for collecting, processing, aggregating, and writing metrics. With over 200 plugins, Telegraf can fetch metrics from a variety of sources, allowing you to build aggregations and write those metrics to InfluxDB, Prometheus, Kafka, and many more targets. In this talk, we'll take a look at the different plugins Telegraf provides for collecting metrics about our networks, as well as the latest features of InfluxDB 2 that make sharing our configurations with others a piece of cake.
 you Hi there. Today I'm going to talk about network monitoring with one of my favorite tools, telegraph. My name is David McKay, I'm from Glasgow in Scotland, and I'm a developer advocate for a company called Equinix Metal. You may be familiar with their former name of Packet, the Bear Metal Cloud. I'm a cloud native steamer, which just means that I get to spend a large chunk of my time on YouTube playing with cloud native technologies and hopefully providing educational materials for us all to learn together. I'm a bit of a monitoring enthusiast. I spend a lot of my time monitoring systems and even things that aren't systems that probably shouldn't be monitored. I've always been a numbers driven person and I love to get data out of anything I can to make sure that I'm on the right path. It's also worth mentioning as a disclaimer that I was formerly a developer advocate for a company called Influx Data. You may be familiar with them as they are the people behind InfluxDB and telegraph itself. I think this is a good thing that I'm still here talking about telegraph after my time at Influx Data. I genuinely believe it's a great product that solves many, many problems and we should all be using it for something. What about telegraph? Well, it is a plugin driven collection agent written in Go. It's a single binary, very easy to deploy in a container or as a static binary and do things with it. What I really love about telegraph is that it is database agnostic. Even though it's by the team or the company that produces InfluxDB, telegraph supports over 40 output plugins. It can write to Kafka. It can write to Postgres. It can write to Prometheus. It can write to Cassandra. It doesn't matter where you want to store your time series metrics. Telegraph is still the right tool for you, or at least a tool for you that could be the right tool for you. It supports some pretty cool architectures based on some of the plugins that has available. We'll talk a little bit about that, but not too much as I'm going to refer to another talk that I've given before. Some of the highlights of telegraph that I think are worth mentioning now, so you get a better understanding of why I'm evangelizing it so much, is that, one, it can write to multiple databases at the same time. As you write your telegraph configurations to Go and collect your metrics, you can write them to InfluxDB, Prometheus, Cassandra, Kafka, all of these places. It gives you a level of resiliency and redundancy that we don't normally have if I'm monitoring an observability pipeline. Another great feature of telegraph is that it supports remote configuration. You can just ribut your telegraph configurations over HTTP using GitHub raw links, using Amazon S3 buckets, anything that exposes that file over HTTP. And telegraph can consume them. It means you can push your configurations centrally and distribute them to multiple edge locations, and you can really find a setup that works for you. What's also great is that by using telegraph, we get access to its ability to batch and buffer the metrics. If any of your output databases are down when it tries to do a flush to that database, it can store those metrics in memory to a configured size set by you that understands how much redundancy or resiliency you need for those metrics. Another really great thing is live health checks. Telegraph has all of these metrics in its memory. What if we could do aggregations on them to understand if our application is healthy? We don't need to add a slash health endpoint to our applications. If the code path gets to it and it returns a 200, we assume our application is healthy. That's not always the case. Our health checks should be more sophisticated. You can have telegraph, web these metrics and memory, perform those aggregations, look at the values and determine are my applications returning 200 responses to the 99% tail of our customers within an accepted SLO. If it's yes, it'll return healthy. If not, it can return not healthy. And that allows us to do things much more dynamically and sophisticated. Also, telegraph can be run as a serverless. It has a flag called run once which will boot up, collect the metrics, wait till it's flushed them out and then exit. Really, really great if you want to schedule it in that fashion. And finally, telegraph is composable. It has plugins that allow us to work over UNIX domain sockets, work over HTTP and be able to take information out, spit it back out, something else, whether it be HTTP or something else. All these different flexibility and things that allow us to just say, hey, I want to chain these telegraphs together to this other component and then chain telegraph on the other side of it. Now, I know that sounds very abstract. Doesn't make a lot of sense. But I will refer you to another talk that I gave previously. The telegraph tool belt is available on YouTube and it really breaks down how to use telegraph for these cloud native and distributed architectures. I definitely recommend you check that out. Alright, what are we going to talk about today? Well, I want to talk about network monitoring and telegraph ships with plugins that allow us to really determine how much network monitoring we want, depending on our use case. The first two are the most basic ones that are net and net stat. Telegraph allows us to collect those metrics and write them somewhere. We'll take a quick look at that before moving on to looking at SNMP and SNMP traps and how we can pick a telegraph to work with those agents. Then we'll very quickly take a look at ESTO and how that can be collected by telegraph. And then I'm going to finish with something a little bit special and we'll get to that at the end. So, I don't know if this is a warning, but I'm going to try to move relative to quickly through the configuration so that we can get to the demo and see in action. So don't worry, after a few plugins you're like, I can't fully grok it. I'm not seeing how this works. Well, there will be a demo and the configs. So just bear with me. Plugin money. Net. This is, of course, very easy to configure and enable. It has no configuration at all. We use Toml to configure telegraph. So we're just using two square brackets to say, hey, we want to enable the net input plugin. That's it. Job done. Now, the way this plugin works is it has primarily support for Linux and Unix based operating systems. It does support Max, Windows support or LimitEat support, but you will get a few things. It will monitor all of the up interfaces on the host, except for look back, although you can enable that if you want. And it's going to pull out basic networking statistics. It's going to understand how many bytes and packets were sent and received on those interfaces. It'll also be able to tell us how many drops we had and how many of the packets had an error state. It does have one configuration option if you want to use it to enable some SNMP metrics from the proc file system. So you can start there without that one option and you will get an understanding of the traffic leaving and coming into your host. We can then lay it on or add our net stat plugin. No surprises. It doesn't take any configuration. Telegraph is great this way. It doesn't want you to write configuration unless you really need to. So you'll notice a lot that you don't really need to provide anything. The net stat plugin also reads from slash proc. It's going to look using LSOF to understand which files are open because every file in Linux has a file. So it'll understand how many TCP connections we have opened, the state they're in, whether they're established, whether they're listening, whether they're closing. And we have some counts for UDPs because obviously it's a stateless protocol. Now you can get really far with this. You can just understand via standard deviations whether the machine has more traffic than normal, less traffic than normal, more errors than normal, etc. You're going to get a really good understanding of the host and what's happening at a network level. But we can do better. So let's talk a little bit about SNMP. SNMP for those that are not familiar stands for Simple Network Management Protocol. And that always makes me laugh because at first glance or even second glance or maybe many glances, it isn't actually that simple. There is a lot going on and a lot that you need to be able to translate and understand to know what is happening when you're using and consuming SNMP. Now there are a few reasons for the ways that it was designed. And I think it's important to really just have that as something that we understand when we're talking about SNMP. First, this protocol was invented in the 80s. If we look at the hardware in the early 80s and late 80s, we were shifting from 8-bit word size to 32. It wasn't quite the standard yet. The clock speeds on our CPUs ranged from 1 to 25 with the average being roughly 10 MHz. And you could expect the machine back then to probably have around 64K of RAM. From a networking point of view, the National Science Foundation, the initial proposers of SNMP, had a backbone circuit of 50 kilobits. Roughly the end user was getting 300 bits per second based on the stats I could find through Wikipedia and some other sources. So when SNMP was created, these really tight constraints with regards to CPU, memory and network bandwidth, which means that we have a lot of liberties today where we can transfer vast amounts of data over HTTP and text that is human readable and get away with it. It wasn't really the case in the 80s. SNMP was created in the early 80s, the first public RFC was 88. So things have changed, but the protocol itself hasn't changed too much, although there are versions 1, 2 and 3. So because of those constraints, we need to understand that SNMP as a protocol, tries to really only transfer OIDs or numbers. You can't read it and know what the field is, you need to be able to translate it. So as a prerequisite for the Telegraph plugin, you will need to use your standard SNMP tools available on a host. That includes SNMP Walk and SNMP Translator. It actually shell execs out to SNMP Translate to translate the OIDs to something more understandable. In order for that translation to work, you will need hosts that have the MIBS installed. Those can be fetched by most operating systems with a simple command, probably download dash MIBS. And of course you're going to need a lot of persistence, and I mean an awful lot of persistence. When you write SNMP configurations mapping the fields that you need into your TSDB, you just have to be patient and you have to really understand what is going on. So you should be comfortable using SNMP Walk and also an example of that on the demo. Now you can configure this plugin with nothing. Not going to get you very far, but you can. And what happens when you configure it with nothing is that Telegraph is going to inject a whole bunch of defaults. Those defaults are. This agent is available on localhost. We assume you're going to speak to the agent there. It will use version 1 by default. You can go on to provide authentication by using version 3 if you wish. It will assume the community is public and there are some intervals, timeouts and retries that can be configured. Although you're probably going to get away, at least in your experimental stage, using the default values. And nothing's going to happen. You're going to run Telegraph with this configuration and you're not going to see any metrics. And that's because SNMP has this whole table that we can walk to fill metrics out. And Telegraph is not going to consume them all by default because one thing databases don't like is large amounts of arbitrary data. So we need to provide the configuration to tell it what fields we care about from the SNMP table. So, pick the fields, get them into Telegraph configuration, be comfortable debugging SNMP using SNMP Walk. It is an invaluable tool for putting these configurations together. And whenever possible, steal prior art. If you're going to GitHub, you will find many configurations, many people that are already monitoring the devices that you want to monitor with SNMP. And that's not going to change very often. So you can really get really far with this. I also want to give a shout out to a former colleague of mine, Rob Kovac. We worked together at Influx a couple of years ago. He was very into network monitoring and has kept this repository online all this time where SNMP, Influx underscore SNMP on his username. It has configurations for all of the major routers that you can hopefully you're using within your network and your infrastructure from Juniper to ubiquity, etc. So these directories actually contain Telegraph configurations that map all of the OIDs to field names to store in the set of Influx. It's an invaluable resource if this is what you're trying to do. Now, we'll come back to SNMP in the demo, but we may also want to have support for SNMP traps. This is relatively new in Telegraph. It was added within just the last year, but Telegraph can run as a listener for those traps coming from your SNMP agents. What I would suggest is, well, sorry, this is the Wikipedia definition. The SNMP traps to kind of invert the control a little bit. Instead of you as the polar reaching out to your SNMP agent and pulling your metrics that you want on your regular interval, you can actually have that SNMP agent send messages back down to you and you can listen for them and consume them. There are a couple of use cases for that. We'll touch on that a little bit, but just know that Telegraph can receive those and you can work with them relatively easily. SNP traps are configured by just enabling the plug-in and providing the port number that we wish to listen on. Then the SNP agent can send those traps to the listener, nice and simple. Of course, we've also got the Sflow plug-in for Telegraph. If you're not sure about Sflow, there's this wonderful animation from the sflow.org website. Basically, what it is doing is it configures VIP tables, a sampling rate of all of your traffic, which will be written to the Sflow agent. That Sflow agent can then pass it on to one or more collectors. In our case, we're going to configure Telegraph to be that collector. In order to be the collector, we enable the Sflow plug-in and provide a port number. Again, Telegraph tries to keep this as simple as possible. Thank you, Telegraph. There's one other thing that I want to talk about outside of Telegraph, and that is InfluxDB2, and that it provides a very convenient manifest-style system for applying configurations and allowing us to consume and reuse other people's configurations. For today's demo, I have used two of these configurations. Very, very easy. Anything that exists in InfluxDB2 can be configured through this manifest format. That includes buckets, Telegraph configurations for the remote configuration that we talked about earlier. We can also provide dashboard users and organizations. This can all feel very familiar if you've used kube-control in the past. We kube-control apply-f. Influx is influx apply-f. Let's go straight into the demo now. Let's see if we can see how all of these pieces of the puzzle connect together to give us an understanding of what is happening within our infrastructure. All right. What I have here is a Linux machine running on EchoNix Metal, and I have Telegraph running in the background, collecting more network data through all the plugins that we just discussed. The first thing I want to do is pop into our Telegraph directory and we'll list the files. What we have is S-look configured as an input, as an MP, as an MP traps. We're writing to InfluxDB and to standard out. It just means that if I wanted to, I could debug that output from a standard file on a disk or through journal D. These configurations are very much exactly like I showed on the slides. Our S-look plugin is configured only with a service address. Our S and MP trap. Delete, delete, delete. It's configured with just a service address. We have our inputs system, which has our network plugin listed here. I've also got all the other basic system monitoring stuff that I would want. CPU, disk, disk.io, mem, processing swap, and another generic system plugin. The big, scary one, the S and MP conf. Enabling it is simple. We do have the fields mapping here. Let's have a quick look at this and then I'll show you how we get some of this information. If we scroll up, I have S and MPD configured on this machine to expose system metrics. You can see this is exact configuration from my slides. I'm just pointing the agent to local host with a community public version of one, so I'm not dealing with any authentication and keeping this all relatively simple. Then you can see we have a lot of field definitions. We're just using the S and MP.field and put syntax. We give it a name that we want to store inside of the database. Then we represent or tell it which OID maps to that field. If we need to do any conversions on data types, we can provide that as well. This telegraph configuration I got from a telegraph community template. If I drop out of here and pop open a new tab and type community templates, learn to spell templates, then we'll see there's this big massive YAML file here and our telegraph configuration is in there towards the bottom. If we go back to the community templates repository, you'll see there's a whole bunch here. There's not just the S and MP template. There's GitHub, there's Linux system, there's Manio, MongoDB. It's a template that the software that you're using has a template that you can apply. However, I also recommended that you get comfortable using your debugging tools, S and MP walk. This tool is invaluable. We specify the version, our community, and our agent. That will walk all the data that we have. Now, because I have the MIPS installed on the system, we're not going to see the OIDs on the left. We're going to see something that is more human readable. If we scroll up, you can just see fields that maybe I'm interested in and monitoring. Really, really good tool. It helps to be comfortable with it. These same IDs on the left can be used instead of the OIDs within the telegraph configuration. Now, what does that look like inside of nFoxDB? Let's just refresh this. It makes sure we've got all that up-to-date information. We can filter on the measurement that we're interested in. Firstly, let's take a look at S and MP. You can see the load averages because we're using S and MP for Linux system metrics. I didn't have access to any returns to be able to get anything a bit more interesting, except my printer. I didn't think you'd want to see me track the number of printed pages over time because it doesn't flop too much. You can see here my load averages come up and down over the last one hour. I've actually left this running, I think, for the last day. You can see something weird happening here. But really great way to fetch metrics and understand it. We can also take a look at our traps. We'll click on this. I just have to deselect them. I hit go. Let's just refresh that. I'm not sure what happened there. Let's try again. S and MP. Trap. I need to look at the last 12 hours. There we go. We'll see that we have a trap that has got some sort of field that is modifying the Sys Uptime instance. Now, traps are events. You cannot graph events. They need to be aggregated. However, nfluxDB allows us to escape that for a moment, and we can just view the raw event data. What we'll see here is that at 12.50, a little over an hour and 10 minutes ago, I restarted SNMD. In fact, if I do that again, we should see that the SNMP agent sends more traps to our listener. Now, telegraph is configured with a batch and a buffer and a flush interval. I'm really hoping I have it set to 10 seconds, but I cannot be entirely sure. Let's just hit go again, and there we go. Now, you can see here where's the time. There we go. 14.01 and 14.01, slightly different order, but you can see we have a notify shutdown and a cold start. Those traps are being consumed by nfluxDB. The configuration for that is very simple as well. There we go. SNMPD. Wait, because it's SNMPD.com. We just add a trap sync to localhost for the public community, and telegraph is listening on the standard port number. That just works. Nice and simple. Let's keep going. Let's take a look at sflow. You'll see we have lots of things going on here. Let's just quickly pull out a couple. We'll hit go. And the sflow stuff is also event-based data, not ideal for viewing on a graph. However, using nfluxDB templates, we can apply those to our cluster, get dashboards, but other people have built the aggregations to visualize that. But you can see here we have all of these events of traffic that has been sampled by IP tables. I've done some DNS stress testing on this machine, so you're going to see a lot of lookups to 1111, the cloud-flare DNS. And in fact, we can just quickly trigger some more. So we're going to go into GitHub. Let's do DNS stress. And this time I'm going to do 8888. And we'll just let that run for about 10 seconds to get some new sflow data into there. Now, the reason I'm using the DNS stress testing tool is just that I need enough packets heading the network to be sampled and show up in a database. Running your traditional ping or DNS lookup may not be enough for the sampling to kick in. Okay, so we're going to leave that like that. Now, next, I want to apply the nfluxDB templates. So I have this directly. I could use the HTTP URLs, however, I did make a couple of modifications to these templates. And just to show you, I'm going to open one up. You know, these are very Kubernetes-like. You can see I've got labels being applied here. I've got a variable here. I've got some telegraph configurations here, which I'm not actually using. And there's dashboards further up that I won't bore you with right now. However, I can do an nfluxapply-f sflow. And there's a lot of complete, I need to get, I need to ask somebody to fix that. But first, dashboards, there's nothing. We apply this template. We get this nice big list that tells us everything that it's going to create within our nfluxDB server. And we hit go. And I'm going to do the same for s and mp. Put yaml. Oh, that's great. I've got one yml and one yaml. And we set go. And then we go back over here. We have dashboards. I didn't write normal how to write available to me. And I think that's fantastic. So let's pop open our sflow source and destination. We can see traffic blurbs here. This one here was obviously testing with the Cloudflare DNS. And I think if we just drop this down to five minutes, we're going to see those 8888s. Now, I know that's painful to read. But you can see the destination IP is 8888. So we can see that network flow coming through. And if we go to our s and mp system monitoring again, this dashboard knows that I have s and mpd running on the Linux host. And it knows it can read the system monitoring coming from that. So I can see my system uptime, which is zero days. I can see CPU realisation. I'm barely touching this box. Memory allocation again, not really touching must. We've got some CPU load. This will be that DNS stress that I just ran. We can see the memory being freed when it's finished. We've got some use memory climbing, you know, all the basic stuff here and a little bit of IO work. But in fact, DB templates are a fantastic way to really visualise that data, especially from a networking standpoint, so that we don't need to understand those queries and write them ourselves. Because sometimes this will actually be quite challenging. All right, we showed s and mp, s and mp trap and s flow. Let's jump back to the slides. So telegraph and FoxDB really do provide a great way to monitor a network in traffic. But I want to do a quick bonus round before we finish. The problem with plugins is they are opinionated by the person that originally wrote the plugin. To try and be as little configuration as possible will give you as much value, but in the eyes of that single person. And your use case may not be the same as other people. You may want to expand on that or provide new metrics that aren't actually monitored. And this is a networking track. So I also want to talk about BPF because I think we all like working with BPF to one extent or another. So how can we use BPF with telegraph to get metrics into inFlexDB or any other time series database? Well, there's this plugin, relatively new in telegraph land, that came out last year called execd. Execd will allow telegraph to manage the running of any arbitrary application on the machine and handle fetching the metrics from its output and storing them in its normal metric collection pipeline. That means any language, any tool, bash, Ruby, Rust, Go, you want to write as long as you can spit out CSV or JSON, line protocol or any other supportive format, you can write your own plugins that telegraph will manage for you. And it is really, really easy. So what I have here is the Iovizer BCC repository cloned on this machine. I call this repository because it has a really great examples directory. And I'm proud that if you're going through the networking directory underneath examples, you'll see a whole bunch of things that already has BPF scripts available. I thought it would be cool using the DNS matching script here. If we could count how many times domains are looked up using this Keranoff trace. So if I can't my telegraph configuration, you'll see I've just got the flush interval set to five seconds so we can see some data hopefully in close to real time. I provide the command that I want to run, which in this case is just a Python script with this DNS matching code. And I'm telling it, I want to count all the lookups for Google.com. And then can specify a signal that I want to be able to handle. So if we want to be able to reload or restart that process, we can, you know, provide the means to do that. I've used JSON as most languages can already output JSON. So, you know, Python is very easy. I just tell telegraph, hey, this application will print out to standard out JSON documents that we want to be able to parse. We tell how to parse those documents so that we'll have a time key that is a Unix times some value. And there are two other keys on that JSON document that we want to store as tags so that they are indexed. That is the name of the domain that we're trying to resolve and the type of DNS lookup that we're doing. I then use an aggregator plugin, which allows us to in real time and in memory take event based data like a DNS lookup, something that cannot be graphed and build an aggregation on it. So in this case, it is going to every two seconds count how many requests were made to that domain and then give me an aggregated count for that window. And instead of writing to nfluxDB, I'm only going to write this out to standard out. Just means I can run telegraph like so. And then in a few seconds, hopefully we will start to see some data come to the terminal. Right on cue. Now, the reason we're seeing data and I'm not doing anything on this machine is I also have DNS stress running in the background. Looking up Google.com and Microsoft.com. And we can see here we get a lookups on Google.com quite a lot of times every two seconds. I think that's 15,000 up to 30,000 down here. And we're not seeing that monitor Microsoft.com at all because it was not part of our allow list. However, we can modify our telegraph conf and say, hey, you know what, we do want to see Microsoft.com counts. Like so. And we rerun telegraph. And again, and hopefully just a few seconds. Boom, boom, boom. Oh, no. Easy. And in that very simple demo, we have written a BPF trace, had telegraph managed the process. And parsed the JSON output from our exactly plug in and store it in flux DB or in this case, print to the terminal. Now, you may wonder what changes had to be made to this repository for that to work. Well, the only thing I really had to do was add the time module, remove any arbitrary print statements that aren't spitting out the JSON that we want to be able to parse. And then this application would just print all the questions and answers to the terminal. I modified it to do a loop and then print out using an F-formatted string, a JSON document with the counts that I want. The aggregator plug in tallies them up every two seconds and we get our metrics. Couldn't be easier. That's all I have for you today. I hope based on what you've seen, you now know that telegraph is a really interesting, powerful and fantastic application to be part of everyone's monitoring and metric collection pipeline. It has support for over 200 plugins, some really great networking support for SNMP, Traps, Eslo and even BPF if you're willing to write a little bit of code. I'm happy to take any questions now. Thank you for joining me and have a great day. Okay, David, thank you for your talk. Hello, guys. Let's start with Q&A. Martin from the Checking OK team is asking why not version 2 standard. Yes, so actually telegraph uses version 2 as the default value. So if you don't provide a version, you do get version 2. However, for my demo and to keep the config as simple as possible, as simple as it could as SNMP, I didn't want to handle authentication. So I used version 1 just to kind of keep that simple. Yeah, okay. Another question is from the same question. You can speak about figuring out SNMP in the middle of an end point. That wasn't a question. I think I'll touch on that anyway. I think it's a really valid statement from Martin and that configuring SNMP and I can't stress it's enough. It is not trivial or easy by any means. It requires a lot of patience. It requires a lot of debugging just an SNMP walk. Whenever possible, try and find other people that have already done it. That repository that I mentioned by a former colleague of mine is just invaluable for all those major providers of routers. You don't need to build this stuff yourself. However, if you are using more esoteric hardware, then you need to get very comfortable understanding and debugging and build those OIDs and MIBS and contacts. So yeah, not an easy task, unfortunately. Just as I said in your simple network, maybe it's not protocol, I think. Yeah, I think we've got to remember it's an old protocol. It's served us well over the years, but simple is not an extra. No, not sure. Yeah, we don't got any questions at this point. Yeah, I think it was a pretty cool talk. Thank you. Do you want to mention anything else? I think telegraph is an infinitely flexible tool. The architectures on which you can compose and join telegraphs together is awesome. Please do check out the talk that I mentioned earlier on. There's a link in the chat as well. But you can do a lot and using and leveraging the exact the plugin to be able to build your own plugins using native boundaries or scripts, anything on that Linux host really provides. It's almost a super baritur. Shutting point been able to hook BPF and get those values metrics aggregations into your time series database is really, really easy. So, you know, take advantage of exactly and use it to your advantage. Cool. Robin is writing I didn't get so much of the inflex DB one to two changes. What's the early question mark? So, yeah, and flux DB two, the biggest changes are the way that you configure the system primarily. So, you know, inflex DB two exposes a very Kubernetes like manifest format that allows you to apply YAML to your inflex DB server to configure the buckets, the users, the labeling, dashboards, everything in that system is configurable through that YAML interface. That is one of the biggest changes. And that's not mentioned in the talk, which is fairly very, very new. But there is a new storage engine currently being worked on and they open for inflex DB two called I ox, which is worth checking out is written in Rust. So added cool points there. And, but yeah, it may and the UI of course sorry and flux DB two exposes its own UI, which is a major change from the previous chronograph UI that was related with one point X. A lot of differences there built from the ground up really, really cool. So yeah, there are lots of changes, some of them are subtle, but the biggest one for me is a manifest format and declarative that declarative manifest format. Yeah, I think it's pretty cool. I guess people have to try it out and put it in the environment and just to use it so it's a try it out. I guess. So, yeah, I think pretty much that's it. Thank you for your talk. It was pretty nice talk. Oh, wait, anyone's talking again. How many time do we have? I think we've got six minutes. Yeah, sure. So looking at the live screen, my video feed looks like it's filmed on a potato where so yours looks really good. I'm not sure what's happened there. Do you see a MacBook in build? No, I'm going through my nice camera. Okay. And the video set the high. I don't know. So Robin is writing, I've been holding back writing my text in the one to two because it seems kind of odd. Yeah, let me touch on that a little bit then, Robin. See, I don't think that you need to hold back. I think during the influx DB two alphas and even the betas, there was a new storage format that was single sharded that was being experimented with, which was reverted before the GA of influx DB two. That just means that migrating from influx DB one to two is now a lot, lot simpler because you can still point it to the same data directory that you had with influx DB one. And you only have to migrate your dashboards users and buckets. And so it's actually a lot simpler now than it was. So, you know, start looking into that there is helper scripts available. There's also the influx DB community slack. I'm on there as well as all the influx teams so people can help you with that migration. But it's not as daunting as it was six months ago. That's for sure. A lot easier now. Sure. Okay, I think. No, that's it. Thank you for having me. Thank you for the people that asked questions and I hope people got something of value at this talk. Please feel free to reach out to me on Twitter or matrix or anywhere else. I'm always happy to help. Sure. Thank you for your talk. It was pretty nice. Yeah, I wish you a nice day. Thank you. Bye. Bye. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you.
 you  Hi there. Today I'm going to talk about network monitoring with one of my favorite tools, telegraph.  My name is David McKay, I'm from Glasgow in Scotland, and I'm a developer advocate for a company called Equinix Metal. You may be familiar with their former name of Packet, the Bear Metal Cloud.  I'm a cloud native steamer, which just means that I get to spend a large chunk of my time on YouTube playing with cloud native technologies and hopefully providing educational materials for us all to learn together.  I'm a bit of a monitoring enthusiast. I spend a lot of my time monitoring systems and even things that aren't systems that probably shouldn't be monitored.  I've always been a numbers driven person and I love to get data out of anything I can to make sure that I'm on the right path.  It's also worth mentioning as a disclaimer that I was formerly a developer advocate for a company called Influx Data. You may be familiar with them as they are the people behind InfluxDB and telegraph itself.  I think this is a good thing that I'm still here talking about telegraph after my time at Influx Data. I genuinely believe it's a great product that solves many, many problems and we should all be using it for something.  What about telegraph? Well, it is a plugin driven collection agent written in Go. It's a single binary, very easy to deploy in a container or as a static binary and do things with it.  What I really love about telegraph is that it is database agnostic. Even though it's by the team or the company that produces InfluxDB, telegraph supports over 40 output plugins.  It can write to Kafka. It can write to Postgres. It can write to Prometheus. It can write to Cassandra. It doesn't matter where you want to store your time series metrics.  Telegraph is still the right tool for you, or at least a tool for you that could be the right tool for you. It supports some pretty cool architectures based on some of the plugins that has available.  We'll talk a little bit about that, but not too much as I'm going to refer to another talk that I've given before.  Some of the highlights of telegraph that I think are worth mentioning now, so you get a better understanding of why I'm evangelizing it so much, is that, one, it can write to multiple databases at the same time.  As you write your telegraph configurations to Go and collect your metrics, you can write them to InfluxDB, Prometheus, Cassandra, Kafka, all of these places.  It gives you a level of resiliency and redundancy that we don't normally have if I'm monitoring an observability pipeline.  Another great feature of telegraph is that it supports remote configuration. You can just ribut your telegraph configurations over HTTP using GitHub raw links, using Amazon S3 buckets, anything that exposes that file over HTTP.  And telegraph can consume them. It means you can push your configurations centrally and distribute them to multiple edge locations, and you can really find a setup that works for you.  What's also great is that by using telegraph, we get access to its ability to batch and buffer the metrics.  If any of your output databases are down when it tries to do a flush to that database, it can store those metrics in memory to a configured size set by you that understands how much redundancy or resiliency you need for those metrics.  Another really great thing is live health checks.  Telegraph has all of these metrics in its memory. What if we could do aggregations on them to understand if our application is healthy? We don't need to add a slash health endpoint to our applications.  If the code path gets to it and it returns a 200, we assume our application is healthy. That's not always the case. Our health checks should be more sophisticated.  You can have telegraph, web these metrics and memory, perform those aggregations, look at the values and determine are my applications returning 200 responses to the 99% tail of our customers within an accepted SLO.  If it's yes, it'll return healthy. If not, it can return not healthy. And that allows us to do things much more dynamically and sophisticated.  Also, telegraph can be run as a serverless. It has a flag called run once which will boot up, collect the metrics, wait till it's flushed them out and then exit. Really, really great if you want to schedule it in that fashion.  And finally, telegraph is composable. It has plugins that allow us to work over UNIX domain sockets, work over HTTP and be able to take information out, spit it back out, something else, whether it be HTTP or something else.  All these different flexibility and things that allow us to just say, hey, I want to chain these telegraphs together to this other component and then chain telegraph on the other side of it.  Now, I know that sounds very abstract. Doesn't make a lot of sense. But I will refer you to another talk that I gave previously. The telegraph tool belt is available on YouTube and it really breaks down how to use telegraph for these cloud native and distributed architectures.  I definitely recommend you check that out. Alright, what are we going to talk about today? Well, I want to talk about network monitoring and telegraph ships with plugins that allow us to really determine how much network monitoring we want, depending on our use case.  The first two are the most basic ones that are net and net stat. Telegraph allows us to collect those metrics and write them somewhere. We'll take a quick look at that before moving on to looking at SNMP and SNMP traps and how we can pick a telegraph to work with those agents.  Then we'll very quickly take a look at ESTO and how that can be collected by telegraph. And then I'm going to finish with something a little bit special and we'll get to that at the end.  So, I don't know if this is a warning, but I'm going to try to move relative to quickly through the configuration so that we can get to the demo and see in action.  So don't worry, after a few plugins you're like, I can't fully grok it. I'm not seeing how this works. Well, there will be a demo and the configs. So just bear with me.  Plugin money. Net. This is, of course, very easy to configure and enable. It has no configuration at all. We use Toml to configure telegraph. So we're just using two square brackets to say, hey, we want to enable the net input plugin.  That's it. Job done. Now, the way this plugin works is it has primarily support for Linux and Unix based operating systems. It does support Max, Windows support or LimitEat support, but you will get a few things.  It will monitor all of the up interfaces on the host, except for look back, although you can enable that if you want. And it's going to pull out basic networking statistics. It's going to understand how many bytes and packets were sent and received on those interfaces.  It'll also be able to tell us how many drops we had and how many of the packets had an error state. It does have one configuration option if you want to use it to enable some SNMP metrics from the proc file system.  So you can start there without that one option and you will get an understanding of the traffic leaving and coming into your host.  We can then lay it on or add our net stat plugin. No surprises. It doesn't take any configuration. Telegraph is great this way. It doesn't want you to write configuration unless you really need to.  So you'll notice a lot that you don't really need to provide anything. The net stat plugin also reads from slash proc.  It's going to look using LSOF to understand which files are open because every file in Linux has a file. So it'll understand how many TCP connections we have opened, the state they're in, whether they're established, whether they're listening, whether they're closing.  And we have some counts for UDPs because obviously it's a stateless protocol.  Now you can get really far with this. You can just understand via standard deviations whether the machine has more traffic than normal, less traffic than normal, more errors than normal, etc.  You're going to get a really good understanding of the host and what's happening at a network level. But we can do better.  So let's talk a little bit about SNMP. SNMP for those that are not familiar stands for Simple Network Management Protocol.  And that always makes me laugh because at first glance or even second glance or maybe many glances, it isn't actually that simple.  There is a lot going on and a lot that you need to be able to translate and understand to know what is happening when you're using and consuming SNMP.  Now there are a few reasons for the ways that it was designed. And I think it's important to really just have that as something that we understand when we're talking about SNMP.  First, this protocol was invented in the 80s. If we look at the hardware in the early 80s and late 80s, we were shifting from 8-bit word size to 32.  It wasn't quite the standard yet. The clock speeds on our CPUs ranged from 1 to 25 with the average being roughly 10 MHz.  And you could expect the machine back then to probably have around 64K of RAM.  From a networking point of view, the National Science Foundation, the initial proposers of SNMP, had a backbone circuit of 50 kilobits.  Roughly the end user was getting 300 bits per second based on the stats I could find through Wikipedia and some other sources.  So when SNMP was created, these really tight constraints with regards to CPU, memory and network bandwidth,  which means that we have a lot of liberties today where we can transfer vast amounts of data over HTTP and text that is human readable and get away with it.  It wasn't really the case in the 80s. SNMP was created in the early 80s, the first public RFC was 88.  So things have changed, but the protocol itself hasn't changed too much, although there are versions 1, 2 and 3.  So because of those constraints, we need to understand that SNMP as a protocol, tries to really only transfer OIDs or numbers.  You can't read it and know what the field is, you need to be able to translate it.  So as a prerequisite for the Telegraph plugin, you will need to use your standard SNMP tools available on a host.  That includes SNMP Walk and SNMP Translator.  It actually shell execs out to SNMP Translate to translate the OIDs to something more understandable.  In order for that translation to work, you will need hosts that have the MIBS installed.  Those can be fetched by most operating systems with a simple command, probably download dash MIBS.  And of course you're going to need a lot of persistence, and I mean an awful lot of persistence.  When you write SNMP configurations mapping the fields that you need into your TSDB, you just have to be patient and you have to really understand what is going on.  So you should be comfortable using SNMP Walk and also an example of that on the demo.  Now you can configure this plugin with nothing.  Not going to get you very far, but you can.  And what happens when you configure it with nothing is that Telegraph is going to inject a whole bunch of defaults.  Those defaults are. This agent is available on localhost.  We assume you're going to speak to the agent there.  It will use version 1 by default.  You can go on to provide authentication by using version 3 if you wish.  It will assume the community is public and there are some intervals, timeouts and retries that can be configured.  Although you're probably going to get away, at least in your experimental stage, using the default values.  And nothing's going to happen.  You're going to run Telegraph with this configuration and you're not going to see any metrics.  And that's because SNMP has this whole table that we can walk to fill metrics out.  And Telegraph is not going to consume them all by default because one thing databases don't like is large amounts of arbitrary data.  So we need to provide the configuration to tell it what fields we care about from the SNMP table.  So, pick the fields, get them into Telegraph configuration, be comfortable debugging SNMP using SNMP Walk.  It is an invaluable tool for putting these configurations together.  And whenever possible, steal prior art.  If you're going to GitHub, you will find many configurations, many people that are already monitoring the devices that you want to monitor with SNMP.  And that's not going to change very often.  So you can really get really far with this.  I also want to give a shout out to a former colleague of mine, Rob Kovac.  We worked together at Influx a couple of years ago.  He was very into network monitoring and has kept this repository online all this time where SNMP, Influx underscore SNMP on his username.  It has configurations for all of the major routers that you can hopefully you're using within your network and your infrastructure from Juniper to ubiquity, etc.  So these directories actually contain Telegraph configurations that map all of the OIDs to field names to store in the set of Influx.  It's an invaluable resource if this is what you're trying to do.  Now, we'll come back to SNMP in the demo, but we may also want to have support for SNMP traps.  This is relatively new in Telegraph. It was added within just the last year, but Telegraph can run as a listener for those traps coming from your SNMP agents.  What I would suggest is, well, sorry, this is the Wikipedia definition.  The SNMP traps to kind of invert the control a little bit.  Instead of you as the polar reaching out to your SNMP agent and pulling your metrics that you want on your regular interval,  you can actually have that SNMP agent send messages back down to you and you can listen for them and consume them.  There are a couple of use cases for that. We'll touch on that a little bit, but just know that Telegraph can receive those and you can work with them relatively easily.  SNP traps are configured by just enabling the plug-in and providing the port number that we wish to listen on.  Then the SNP agent can send those traps to the listener, nice and simple.  Of course, we've also got the Sflow plug-in for Telegraph.  If you're not sure about Sflow, there's this wonderful animation from the sflow.org website.  Basically, what it is doing is it configures VIP tables, a sampling rate of all of your traffic, which will be written to the Sflow agent.  That Sflow agent can then pass it on to one or more collectors.  In our case, we're going to configure Telegraph to be that collector.  In order to be the collector, we enable the Sflow plug-in and provide a port number.  Again, Telegraph tries to keep this as simple as possible. Thank you, Telegraph.  There's one other thing that I want to talk about outside of Telegraph, and that is InfluxDB2,  and that it provides a very convenient manifest-style system for applying configurations  and allowing us to consume and reuse other people's configurations.  For today's demo, I have used two of these configurations. Very, very easy.  Anything that exists in InfluxDB2 can be configured through this manifest format.  That includes buckets, Telegraph configurations for the remote configuration that we talked about earlier.  We can also provide dashboard users and organizations.  This can all feel very familiar if you've used kube-control in the past.  We kube-control apply-f. Influx is influx apply-f.  Let's go straight into the demo now.  Let's see if we can see how all of these pieces of the puzzle connect together  to give us an understanding of what is happening within our infrastructure.  All right. What I have here is a Linux machine running on EchoNix Metal,  and I have Telegraph running in the background, collecting more network data  through all the plugins that we just discussed.  The first thing I want to do is pop into our Telegraph directory and we'll list the files.  What we have is S-look configured as an input, as an MP, as an MP traps.  We're writing to InfluxDB and to standard out.  It just means that if I wanted to, I could debug that output from a standard file on a disk or through journal D.  These configurations are very much exactly like I showed on the slides.  Our S-look plugin is configured only with a service address.  Our S and MP trap.  Delete, delete, delete.  It's configured with just a service address.  We have our inputs system, which has our network plugin listed here.  I've also got all the other basic system monitoring stuff that I would want.  CPU, disk, disk.io, mem, processing swap, and another generic system plugin.  The big, scary one, the S and MP conf.  Enabling it is simple. We do have the fields mapping here.  Let's have a quick look at this and then I'll show you how we get some of this information.  If we scroll up, I have S and MPD configured on this machine to expose system metrics.  You can see this is exact configuration from my slides.  I'm just pointing the agent to local host with a community public version of one, so I'm not dealing with any authentication and keeping this all relatively simple.  Then you can see we have a lot of field definitions.  We're just using the S and MP.field and put syntax.  We give it a name that we want to store inside of the database.  Then we represent or tell it which OID maps to that field.  If we need to do any conversions on data types, we can provide that as well.  This telegraph configuration I got from a telegraph community template.  If I drop out of here and pop open a new tab and type community templates, learn to spell templates,  then we'll see there's this big massive YAML file here and our telegraph configuration is in there towards the bottom.  If we go back to the community templates repository, you'll see there's a whole bunch here.  There's not just the S and MP template.  There's GitHub, there's Linux system, there's Manio, MongoDB.  It's a template that the software that you're using has a template that you can apply.  However, I also recommended that you get comfortable using your debugging tools, S and MP walk.  This tool is invaluable.  We specify the version, our community, and our agent.  That will walk all the data that we have.  Now, because I have the MIPS installed on the system, we're not going to see the OIDs on the left.  We're going to see something that is more human readable.  If we scroll up, you can just see fields that maybe I'm interested in and monitoring.  Really, really good tool.  It helps to be comfortable with it.  These same IDs on the left can be used instead of the OIDs within the telegraph configuration.  Now, what does that look like inside of nFoxDB?  Let's just refresh this.  It makes sure we've got all that up-to-date information.  We can filter on the measurement that we're interested in.  Firstly, let's take a look at S and MP.  You can see the load averages because we're using S and MP for Linux system metrics.  I didn't have access to any returns to be able to get anything a bit more interesting,  except my printer.  I didn't think you'd want to see me track the number of printed pages over time because it doesn't flop too much.  You can see here my load averages come up and down over the last one hour.  I've actually left this running, I think, for the last day.  You can see something weird happening here.  But really great way to fetch metrics and understand it.  We can also take a look at our traps.  We'll click on this.  I just have to deselect them.  I hit go.  Let's just refresh that.  I'm not sure what happened there.  Let's try again.  S and MP.  Trap.  I need to look at the last 12 hours.  There we go.  We'll see that we have a trap that has got some sort of field that is modifying the Sys Uptime instance.  Now, traps are events.  You cannot graph events.  They need to be aggregated.  However, nfluxDB allows us to escape that for a moment, and we can just view the raw event data.  What we'll see here is that at 12.50, a little over an hour and 10 minutes ago, I restarted SNMD.  In fact, if I do that again, we should see that the SNMP agent sends more traps to our listener.  Now, telegraph is configured with a batch and a buffer and a flush interval.  I'm really hoping I have it set to 10 seconds, but I cannot be entirely sure.  Let's just hit go again, and there we go.  Now, you can see here where's the time.  There we go.  14.01 and 14.01, slightly different order, but you can see we have a notify shutdown and a cold start.  Those traps are being consumed by nfluxDB.  The configuration for that is very simple as well.  There we go.  SNMPD.  Wait, because it's SNMPD.com.  We just add a trap sync to localhost for the public community, and telegraph is listening on the standard port number.  That just works.  Nice and simple.  Let's keep going.  Let's take a look at sflow.  You'll see we have lots of things going on here.  Let's just quickly pull out a couple.  We'll hit go.  And the sflow stuff is also event-based data, not ideal for viewing on a graph.  However, using nfluxDB templates, we can apply those to our cluster, get dashboards, but other people have built the aggregations to visualize that.  But you can see here we have all of these events of traffic that has been sampled by IP tables.  I've done some DNS stress testing on this machine, so you're going to see a lot of lookups to 1111, the cloud-flare DNS.  And in fact, we can just quickly trigger some more.  So we're going to go into GitHub.  Let's do DNS stress.  And this time I'm going to do 8888.  And we'll just let that run for about 10 seconds to get some new sflow data into there.  Now, the reason I'm using the DNS stress testing tool is just that I need enough packets heading the network to be sampled and show up in a database.  Running your traditional ping or DNS lookup may not be enough for the sampling to kick in.  Okay, so we're going to leave that like that.  Now, next, I want to apply the nfluxDB templates.  So I have this directly.  I could use the HTTP URLs, however, I did make a couple of modifications to these templates.  And just to show you, I'm going to open one up.  You know, these are very Kubernetes-like.  You can see I've got labels being applied here.  I've got a variable here.  I've got some telegraph configurations here, which I'm not actually using.  And there's dashboards further up that I won't bore you with right now.  However, I can do an nfluxapply-f sflow.  And there's a lot of complete, I need to get, I need to ask somebody to fix that.  But first, dashboards, there's nothing.  We apply this template.  We get this nice big list that tells us everything that it's going to create within our nfluxDB server.  And we hit go.  And I'm going to do the same for s and mp.  Put yaml.  Oh, that's great.  I've got one yml and one yaml.  And we set go.  And then we go back over here.  We have dashboards.  I didn't write normal how to write available to me.  And I think that's fantastic.  So let's pop open our sflow source and destination.  We can see traffic blurbs here.  This one here was obviously testing with the Cloudflare DNS.  And I think if we just drop this down to five minutes, we're going to see those 8888s.  Now, I know that's painful to read.  But you can see the destination IP is 8888.  So we can see that network flow coming through.  And if we go to our s and mp system monitoring again, this dashboard knows that I have s and mpd running on the Linux host.  And it knows it can read the system monitoring coming from that.  So I can see my system uptime, which is zero days.  I can see CPU realisation.  I'm barely touching this box.  Memory allocation again, not really touching must.  We've got some CPU load.  This will be that DNS stress that I just ran.  We can see the memory being freed when it's finished.  We've got some use memory climbing, you know, all the basic stuff here and a little bit of IO work.  But in fact, DB templates are a fantastic way to really visualise that data,  especially from a networking standpoint,  so that we don't need to understand those queries and write them ourselves.  Because sometimes this will actually be quite challenging.  All right, we showed s and mp, s and mp trap and s flow.  Let's jump back to the slides.  So telegraph and FoxDB really do provide a great way to monitor a network in traffic.  But I want to do a quick bonus round before we finish.  The problem with plugins is they are opinionated by the person that originally wrote the plugin.  To try and be as little configuration as possible will give you as much value,  but in the eyes of that single person.  And your use case may not be the same as other people.  You may want to expand on that or provide new metrics that aren't actually monitored.  And this is a networking track.  So I also want to talk about BPF because I think we all like working with BPF to one extent or another.  So how can we use BPF with telegraph to get metrics into inFlexDB or any other time series database?  Well, there's this plugin, relatively new in telegraph land, that came out last year called execd.  Execd will allow telegraph to manage the running of any arbitrary application on the machine  and handle fetching the metrics from its output and storing them in its normal metric collection pipeline.  That means any language, any tool, bash, Ruby, Rust, Go, you want to write  as long as you can spit out CSV or JSON, line protocol or any other supportive format,  you can write your own plugins that telegraph will manage for you.  And it is really, really easy.  So what I have here is the Iovizer BCC repository cloned on this machine.  I call this repository because it has a really great examples directory.  And I'm proud that if you're going through the networking directory underneath examples,  you'll see a whole bunch of things that already has BPF scripts available.  I thought it would be cool using the DNS matching script here.  If we could count how many times domains are looked up using this Keranoff trace.  So if I can't my telegraph configuration, you'll see I've just got the flush interval set to five seconds  so we can see some data hopefully in close to real time.  I provide the command that I want to run, which in this case is just a Python script with this DNS matching code.  And I'm telling it, I want to count all the lookups for Google.com.  And then can specify a signal that I want to be able to handle.  So if we want to be able to reload or restart that process, we can, you know, provide the means to do that.  I've used JSON as most languages can already output JSON.  So, you know, Python is very easy.  I just tell telegraph, hey, this application will print out to standard out JSON documents that we want to be able to parse.  We tell how to parse those documents so that we'll have a time key that is a Unix times some value.  And there are two other keys on that JSON document that we want to store as tags so that they are indexed.  That is the name of the domain that we're trying to resolve and the type of DNS lookup that we're doing.  I then use an aggregator plugin, which allows us to in real time and in memory take event based data like a DNS lookup,  something that cannot be graphed and build an aggregation on it.  So in this case, it is going to every two seconds count how many requests were made to that domain and then give me an aggregated count for that window.  And instead of writing to nfluxDB, I'm only going to write this out to standard out.  Just means I can run telegraph like so.  And then in a few seconds, hopefully we will start to see some data come to the terminal.  Right on cue.  Now, the reason we're seeing data and I'm not doing anything on this machine is I also have DNS stress running in the background.  Looking up Google.com and Microsoft.com.  And we can see here we get a lookups on Google.com quite a lot of times every two seconds.  I think that's 15,000 up to 30,000 down here.  And we're not seeing that monitor Microsoft.com at all because it was not part of our allow list.  However, we can modify our telegraph conf and say, hey, you know what, we do want to see Microsoft.com counts.  Like so. And we rerun telegraph.  And again, and hopefully just a few seconds.  Boom, boom, boom.  Oh, no.  Easy.  And in that very simple demo, we have written a BPF trace, had telegraph managed the process.  And parsed the JSON output from our exactly plug in and store it in flux DB or in this case, print to the terminal.  Now, you may wonder what changes had to be made to this repository for that to work.  Well, the only thing I really had to do was add the time module, remove any arbitrary print statements that aren't spitting out the JSON that we want to be able to parse.  And then this application would just print all the questions and answers to the terminal.  I modified it to do a loop and then print out using an F-formatted string, a JSON document with the counts that I want.  The aggregator plug in tallies them up every two seconds and we get our metrics.  Couldn't be easier.  That's all I have for you today.  I hope based on what you've seen, you now know that telegraph is a really interesting, powerful and fantastic application to be part of everyone's monitoring and metric collection pipeline.  It has support for over 200 plugins, some really great networking support for SNMP, Traps, Eslo and even BPF if you're willing to write a little bit of code.  I'm happy to take any questions now.  Thank you for joining me and have a great day.  Okay, David, thank you for your talk.  Hello, guys.  Let's start with Q&A.  Martin from the Checking OK team is asking why not version 2 standard.  Yes, so actually telegraph uses version 2 as the default value.  So if you don't provide a version, you do get version 2.  However, for my demo and to keep the config as simple as possible, as simple as it could as SNMP, I didn't want to handle authentication.  So I used version 1 just to kind of keep that simple.  Yeah, okay.  Another question is from the same question.  You can speak about figuring out SNMP in the middle of an end point.  That wasn't a question.  I think I'll touch on that anyway.  I think it's a really valid statement from Martin and that configuring SNMP and I can't stress it's enough.  It is not trivial or easy by any means.  It requires a lot of patience.  It requires a lot of debugging just an SNMP walk.  Whenever possible, try and find other people that have already done it.  That repository that I mentioned by a former colleague of mine is just invaluable for all those major providers of routers.  You don't need to build this stuff yourself.  However, if you are using more esoteric hardware, then you need to get very comfortable understanding and debugging and build those OIDs and MIBS and contacts.  So yeah, not an easy task, unfortunately.  Just as I said in your simple network, maybe it's not protocol, I think.  Yeah, I think we've got to remember it's an old protocol.  It's served us well over the years, but simple is not an extra.  No, not sure.  Yeah, we don't got any questions at this point.  Yeah, I think it was a pretty cool talk.  Thank you.  Do you want to mention anything else?  I think telegraph is an infinitely flexible tool.  The architectures on which you can compose and join telegraphs together is awesome.  Please do check out the talk that I mentioned earlier on.  There's a link in the chat as well.  But you can do a lot and using and leveraging the exact the plugin to be able to build your own plugins using native boundaries or scripts, anything on that Linux host really provides.  It's almost a super baritur.  Shutting point been able to hook BPF and get those values metrics aggregations into your time series database is really, really easy.  So, you know, take advantage of exactly and use it to your advantage.  Cool.  Robin is writing I didn't get so much of the inflex DB one to two changes.  What's the early question mark?  So, yeah, and flux DB two, the biggest changes are the way that you configure the system primarily.  So, you know, inflex DB two exposes a very Kubernetes like manifest format that allows you to apply YAML to your inflex DB server to configure the buckets, the users, the labeling, dashboards, everything in that system is configurable through that YAML interface.  That is one of the biggest changes.  And that's not mentioned in the talk, which is fairly very, very new. But there is a new storage engine currently being worked on and they open for inflex DB two called I ox, which is worth checking out is written in Rust.  So added cool points there.  And, but yeah, it may and the UI of course sorry and flux DB two exposes its own UI, which is a major change from the previous chronograph UI that was related with one point X.  A lot of differences there built from the ground up really, really cool.  So yeah, there are lots of changes, some of them are subtle, but the biggest one for me is a manifest format and declarative that declarative manifest format.  Yeah, I think it's pretty cool.  I guess people have to try it out and put it in the environment and just to use it so it's a try it out.  I guess.  So, yeah, I think pretty much that's it.  Thank you for your talk.  It was pretty nice talk.  Oh, wait, anyone's talking again.  How many time do we have?  I think we've got six minutes.  Yeah, sure.  So looking at the live screen, my video feed looks like it's filmed on a potato where so yours looks really good.  I'm not sure what's happened there.  Do you see a MacBook in build?  No, I'm going through my nice camera.  Okay.  And the video set the high.  I don't know.  So Robin is writing, I've been holding back writing my text in the one to two because it seems kind of odd.  Yeah, let me touch on that a little bit then, Robin.  See, I don't think that you need to hold back.  I think during the influx DB two alphas and even the betas, there was a new storage format that was single sharded that was being experimented with, which was reverted before the GA of influx DB two.  That just means that migrating from influx DB one to two is now a lot, lot simpler because you can still point it to the same data directory that you had with influx DB one.  And you only have to migrate your dashboards users and buckets.  And so it's actually a lot simpler now than it was.  So, you know, start looking into that there is helper scripts available.  There's also the influx DB community slack.  I'm on there as well as all the influx teams so people can help you with that migration. But it's not as daunting as it was six months ago. That's for sure.  A lot easier now.  Sure.  Okay, I think.  No, that's it.  Thank you for having me. Thank you for the people that asked questions and I hope people got something of value at this talk. Please feel free to reach out to me on Twitter or matrix or anywhere else.  I'm always happy to help.  Sure. Thank you for your talk. It was pretty nice.  Yeah, I wish you a nice day.  Thank you.  Bye. Bye.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.  Thank you.
nemoinflux_0_4175.png nemoinflux_4176_4331.png nemoinflux_4332_4895.png nemoinflux_4896_5339.png nemoinflux_5340_5615.png nemoinflux_5616_8459.png nemoinflux_8460_10547.png nemoinflux_10548_12527.png nemoinflux_12528_12659.png nemoinflux_12660_14447.png nemoinflux_14448_20015.png nemoinflux_20016_20387.png nemoinflux_20388_22463.png nemoinflux_22464_25187.png nemoinflux_25188_26915.png nemoinflux_26916_29363.png nemoinflux_29364_30527.png nemoinflux_30528_30815.png nemoinflux_30816_30875.png nemoinflux_30876_31367.png nemoinflux_31368_32183.png nemoinflux_32184_32447.png nemoinflux_32448_32687.png nemoinflux_32688_39407.png nemoinflux_39408_39719.png nemoinflux_39720_39995.png nemoinflux_39996_40079.png nemoinflux_40080_40175.png nemoinflux_40176_42395.png nemoinflux_42396_42611.png nemoinflux_42612_42755.png nemoinflux_42756_42899.png nemoinflux_42900_43751.png nemoinflux_43752_44735.png nemoinflux_44736_49427.png nemoinflux_49428_52187.png nemoinflux_52188_53003.png nemoinflux_53004_54251.png nemoinflux_54252_54335.png nemoinflux_54336_55211.png nemoinflux_55212_56579.png nemoinflux_56580_58319.png nemoinflux_58320_58931.png nemoinflux_58932_59195.png nemoinflux_59196_59315.png nemoinflux_59316_59603.png nemoinflux_59604_60131.png nemoinflux_60132_61475.png nemoinflux_61476_61727.png nemoinflux_61728_62039.png nemoinflux_62040_62207.png nemoinflux_62208_67760.png
you Hi there. Today I'm going to talk about network monitoring with one of my favorite tools, telegraph. My name is David McKay, I'm from Glasgow in Scotland, and I'm a developer advocate for a company called Equinix Metal. You may be familiar with their former name of Packet, the Bear Metal Cloud. I'm a cloud native steamer, which just means that I get to spend a large chunk of my time on YouTube playing with cloud native technologies and hopefully providing educational materials for us all to learn together. I'm a bit of a monitoring enthusiast. I spend a lot of my time monitoring systems and even things that aren't systems that probably shouldn't be monitored. I've always been a numbers driven person and I love to get data out of anything I can to make sure that I'm on the right path. It's also worth mentioning as a disclaimer that I was formerly a developer advocate for a company called Influx Data. You may be familiar with them as they are the people behind InfluxDB and telegraph itself. I think this is a good thing that I'm still here talking about telegraph after my time at Influx Data. I genuinely believe it's a great product that solves many, many problems and we should all be using it for something. What about telegraph? Well, it is a plugin driven collection agent written in Go. It's a single binary, very easy to deploy in a container or as a static binary and do things with it. What I really love about telegraph is that it is database agnostic. Even though it's by the team or the company that produces InfluxDB, telegraph supports over 40 output plugins. It can write to Kafka. It can write to Postgres. It can write to Prometheus. It can write to Cassandra. It doesn't matter where you want to store your time series metrics. Telegraph is still the right tool for you, or at least a tool for you that could be the right tool for you. It supports some pretty cool architectures based on some of the plugins that has available. We'll talk a little bit about that, but not too much as I'm going to refer to another talk that I've given before. Some of the highlights of telegraph that I think are worth mentioning now, so you get a better understanding of why I'm evangelizing it so much, is that, one, it can write to multiple databases at the same time. As you write your telegraph configurations to Go and collect your metrics, you can write them to InfluxDB, Prometheus, Cassandra, Kafka, all of these places. It gives you a level of resiliency and redundancy that we don't normally have if I'm monitoring an observability pipeline. Another great feature of telegraph is that it supports remote configuration. You can just ribut your telegraph configurations over HTTP using GitHub raw links, using Amazon S3 buckets, anything that exposes that file over HTTP. And telegraph can consume them. It means you can push your configurations centrally and distribute them to multiple edge locations, and you can really find a setup that works for you. What's also great is that by using telegraph, we get access to its ability to batch and buffer the metrics. If any of your output databases are down when it tries to do a flush to that database, it can store those metrics in memory to a configured size set by you that understands how much redundancy or resiliency you need for those metrics. Another really great thing is live health checks. Telegraph has all of these metrics in its memory. What if we could do aggregations on them to understand if our application is healthy? We don't need to add a slash health endpoint to our applications. If the code path gets to it and it returns a 200, we assume our application is healthy. That's not always the case. Our health checks should be more sophisticated. You can have telegraph, web these metrics and memory, perform those aggregations, look at the values and determine are my applications returning 200 responses to the 99% tail of our customers within an accepted SLO. If it's yes, it'll return healthy. If not, it can return not healthy. And that allows us to do things much more dynamically and sophisticated. Also, telegraph can be run as a serverless. It has a flag called run once which will boot up, collect the metrics, wait till it's flushed them out and then exit. Really, really great if you want to schedule it in that fashion. And finally, telegraph is composable. It has plugins that allow us to work over UNIX domain sockets, work over HTTP and be able to take information out, spit it back out, something else, whether it be HTTP or something else. All these different flexibility and things that allow us to just say, hey, I want to chain these telegraphs together to this other component and then chain telegraph on the other side of it. Now, I know that sounds very abstract. Doesn't make a lot of sense. But I will refer you to another talk that I gave previously. The telegraph tool belt is available on YouTube and it really breaks down how to use telegraph for these cloud native and distributed architectures. I definitely recommend you check that out. Alright, what are we going to talk about today? Well, I want to talk about network monitoring and telegraph ships with plugins that allow us to really determine how much network monitoring we want, depending on our use case. The first two are the most basic ones that are net and net stat. Telegraph allows us to collect those metrics and write them somewhere. We'll take a quick look at that before moving on to looking at SNMP and SNMP traps and how we can pick a telegraph to work with those agents. Then we'll very quickly take a look at ESTO and how that can be collected by telegraph. And then I'm going to finish with something a little bit special and we'll get to that at the end. So, I don't know if this is a warning, but I'm going to try to move relative to quickly through the configuration so that we can get to the demo and see in action. So don't worry, after a few plugins you're like, I can't fully grok it. I'm not seeing how this works. Well, there will be a demo and the configs. So just bear with me. Plugin money. Net. This is, of course, very easy to configure and enable. It has no configuration at all. We use Toml to configure telegraph. So we're just using two square brackets to say, hey, we want to enable the net input plugin. That's it. Job done. Now, the way this plugin works is it has primarily support for Linux and Unix based operating systems. It does support Max, Windows support or LimitEat support, but you will get a few things. It will monitor all of the up interfaces on the host, except for look back, although you can enable that if you want. And it's going to pull out basic networking statistics. It's going to understand how many bytes and packets were sent and received on those interfaces. It'll also be able to tell us how many drops we had and how many of the packets had an error state. It does have one configuration option if you want to use it to enable some SNMP metrics from the proc file system. So you can start there without that one option and you will get an understanding of the traffic leaving and coming into your host. We can then lay it on or add our net stat plugin. No surprises. It doesn't take any configuration. Telegraph is great this way. It doesn't want you to write configuration unless you really need to. So you'll notice a lot that you don't really need to provide anything. The net stat plugin also reads from slash proc. It's going to look using LSOF to understand which files are open because every file in Linux has a file. So it'll understand how many TCP connections we have opened, the state they're in, whether they're established, whether they're listening, whether they're closing. And we have some counts for UDPs because obviously it's a stateless protocol. Now you can get really far with this. You can just understand via standard deviations whether the machine has more traffic than normal, less traffic than normal, more errors than normal, etc. You're going to get a really good understanding of the host and what's happening at a network level. But we can do better. So let's talk a little bit about SNMP. SNMP for those that are not familiar stands for Simple Network Management Protocol. And that always makes me laugh because at first glance or even second glance or maybe many glances, it isn't actually that simple. There is a lot going on and a lot that you need to be able to translate and understand to know what is happening when you're using and consuming SNMP. Now there are a few reasons for the ways that it was designed. And I think it's important to really just have that as something that we understand when we're talking about SNMP. First, this protocol was invented in the 80s. If we look at the hardware in the early 80s and late 80s, we were shifting from 8-bit word size to 32. It wasn't quite the standard yet. The clock speeds on our CPUs ranged from 1 to 25 with the average being roughly 10 MHz. And you could expect the machine back then to probably have around 64K of RAM. From a networking point of view, the National Science Foundation, the initial proposers of SNMP, had a backbone circuit of 50 kilobits. Roughly the end user was getting 300 bits per second based on the stats I could find through Wikipedia and some other sources. So when SNMP was created, these really tight constraints with regards to CPU, memory and network bandwidth, which means that we have a lot of liberties today where we can transfer vast amounts of data over HTTP and text that is human readable and get away with it. It wasn't really the case in the 80s. SNMP was created in the early 80s, the first public RFC was 88. So things have changed, but the protocol itself hasn't changed too much, although there are versions 1, 2 and 3. So because of those constraints, we need to understand that SNMP as a protocol, tries to really only transfer OIDs or numbers. You can't read it and know what the field is, you need to be able to translate it. So as a prerequisite for the Telegraph plugin, you will need to use your standard SNMP tools available on a host. That includes SNMP Walk and SNMP Translator. It actually shell execs out to SNMP Translate to translate the OIDs to something more understandable. In order for that translation to work, you will need hosts that have the MIBS installed. Those can be fetched by most operating systems with a simple command, probably download dash MIBS. And of course you're going to need a lot of persistence, and I mean an awful lot of persistence. When you write SNMP configurations mapping the fields that you need into your TSDB, you just have to be patient and you have to really understand what is going on. So you should be comfortable using SNMP Walk and also an example of that on the demo. Now you can configure this plugin with nothing. Not going to get you very far, but you can. And what happens when you configure it with nothing is that Telegraph is going to inject a whole bunch of defaults. Those defaults are. This agent is available on localhost. We assume you're going to speak to the agent there. It will use version 1 by default. You can go on to provide authentication by using version 3 if you wish. It will assume the community is public and there are some intervals, timeouts and retries that can be configured. Although you're probably going to get away, at least in your experimental stage, using the default values. And nothing's going to happen. You're going to run Telegraph with this configuration and you're not going to see any metrics. And that's because SNMP has this whole table that we can walk to fill metrics out. And Telegraph is not going to consume them all by default because one thing databases don't like is large amounts of arbitrary data. So we need to provide the configuration to tell it what fields we care about from the SNMP table. So, pick the fields, get them into Telegraph configuration, be comfortable debugging SNMP using SNMP Walk. It is an invaluable tool for putting these configurations together. And whenever possible, steal prior art. If you're going to GitHub, you will find many configurations, many people that are already monitoring the devices that you want to monitor with SNMP. And that's not going to change very often. So you can really get really far with this. I also want to give a shout out to a former colleague of mine, Rob Kovac. We worked together at Influx a couple of years ago. He was very into network monitoring and has kept this repository online all this time where SNMP, Influx underscore SNMP on his username. It has configurations for all of the major routers that you can hopefully you're using within your network and your infrastructure from Juniper to ubiquity, etc. So these directories actually contain Telegraph configurations that map all of the OIDs to field names to store in the set of Influx. It's an invaluable resource if this is what you're trying to do. Now, we'll come back to SNMP in the demo, but we may also want to have support for SNMP traps. This is relatively new in Telegraph. It was added within just the last year, but Telegraph can run as a listener for those traps coming from your SNMP agents. What I would suggest is, well, sorry, this is the Wikipedia definition. The SNMP traps to kind of invert the control a little bit. Instead of you as the polar reaching out to your SNMP agent and pulling your metrics that you want on your regular interval, you can actually have that SNMP agent send messages back down to you and you can listen for them and consume them. There are a couple of use cases for that. We'll touch on that a little bit, but just know that Telegraph can receive those and you can work with them relatively easily. SNP traps are configured by just enabling the plug-in and providing the port number that we wish to listen on. Then the SNP agent can send those traps to the listener, nice and simple. Of course, we've also got the Sflow plug-in for Telegraph. If you're not sure about Sflow, there's this wonderful animation from the sflow.org website. Basically, what it is doing is it configures VIP tables, a sampling rate of all of your traffic, which will be written to the Sflow agent. That Sflow agent can then pass it on to one or more collectors. In our case, we're going to configure Telegraph to be that collector. In order to be the collector, we enable the Sflow plug-in and provide a port number. Again, Telegraph tries to keep this as simple as possible. Thank you, Telegraph. There's one other thing that I want to talk about outside of Telegraph, and that is InfluxDB2, and that it provides a very convenient manifest-style system for applying configurations and allowing us to consume and reuse other people's configurations. For today's demo, I have used two of these configurations. Very, very easy. Anything that exists in InfluxDB2 can be configured through this manifest format. That includes buckets, Telegraph configurations for the remote configuration that we talked about earlier. We can also provide dashboard users and organizations. This can all feel very familiar if you've used kube-control in the past. We kube-control apply-f. Influx is influx apply-f. Let's go straight into the demo now. Let's see if we can see how all of these pieces of the puzzle connect together to give us an understanding of what is happening within our infrastructure. All right. What I have here is a Linux machine running on EchoNix Metal, and I have Telegraph running in the background, collecting more network data through all the plugins that we just discussed. The first thing I want to do is pop into our Telegraph directory and we'll list the files. What we have is S-look configured as an input, as an MP, as an MP traps. We're writing to InfluxDB and to standard out. It just means that if I wanted to, I could debug that output from a standard file on a disk or through journal D. These configurations are very much exactly like I showed on the slides. Our S-look plugin is configured only with a service address. Our S and MP trap. Delete, delete, delete. It's configured with just a service address. We have our inputs system, which has our network plugin listed here. I've also got all the other basic system monitoring stuff that I would want. CPU, disk, disk.io, mem, processing swap, and another generic system plugin. The big, scary one, the S and MP conf. Enabling it is simple. We do have the fields mapping here. Let's have a quick look at this and then I'll show you how we get some of this information. If we scroll up, I have S and MPD configured on this machine to expose system metrics. You can see this is exact configuration from my slides. I'm just pointing the agent to local host with a community public version of one, so I'm not dealing with any authentication and keeping this all relatively simple. Then you can see we have a lot of field definitions. We're just using the S and MP.field and put syntax. We give it a name that we want to store inside of the database. Then we represent or tell it which OID maps to that field. If we need to do any conversions on data types, we can provide that as well. This telegraph configuration I got from a telegraph community template. If I drop out of here and pop open a new tab and type community templates, learn to spell templates, then we'll see there's this big massive YAML file here and our telegraph configuration is in there towards the bottom. If we go back to the community templates repository, you'll see there's a whole bunch here. There's not just the S and MP template. There's GitHub, there's Linux system, there's Manio, MongoDB. It's a template that the software that you're using has a template that you can apply. However, I also recommended that you get comfortable using your debugging tools, S and MP walk. This tool is invaluable. We specify the version, our community, and our agent. That will walk all the data that we have. Now, because I have the MIPS installed on the system, we're not going to see the OIDs on the left. We're going to see something that is more human readable. If we scroll up, you can just see fields that maybe I'm interested in and monitoring. Really, really good tool. It helps to be comfortable with it. These same IDs on the left can be used instead of the OIDs within the telegraph configuration. Now, what does that look like inside of nFoxDB? Let's just refresh this. It makes sure we've got all that up-to-date information. We can filter on the measurement that we're interested in. Firstly, let's take a look at S and MP. You can see the load averages because we're using S and MP for Linux system metrics. I didn't have access to any returns to be able to get anything a bit more interesting, except my printer. I didn't think you'd want to see me track the number of printed pages over time because it doesn't flop too much. You can see here my load averages come up and down over the last one hour. I've actually left this running, I think, for the last day. You can see something weird happening here. But really great way to fetch metrics and understand it. We can also take a look at our traps. We'll click on this. I just have to deselect them. I hit go. Let's just refresh that. I'm not sure what happened there. Let's try again. S and MP. Trap. I need to look at the last 12 hours. There we go. We'll see that we have a trap that has got some sort of field that is modifying the Sys Uptime instance. Now, traps are events. You cannot graph events. They need to be aggregated. However, nfluxDB allows us to escape that for a moment, and we can just view the raw event data. What we'll see here is that at 12.50, a little over an hour and 10 minutes ago, I restarted SNMD. In fact, if I do that again, we should see that the SNMP agent sends more traps to our listener. Now, telegraph is configured with a batch and a buffer and a flush interval. I'm really hoping I have it set to 10 seconds, but I cannot be entirely sure. Let's just hit go again, and there we go. Now, you can see here where's the time. There we go. 14.01 and 14.01, slightly different order, but you can see we have a notify shutdown and a cold start. Those traps are being consumed by nfluxDB. The configuration for that is very simple as well. There we go. SNMPD. Wait, because it's SNMPD.com. We just add a trap sync to localhost for the public community, and telegraph is listening on the standard port number. That just works. Nice and simple. Let's keep going. Let's take a look at sflow. You'll see we have lots of things going on here. Let's just quickly pull out a couple. We'll hit go. And the sflow stuff is also event-based data, not ideal for viewing on a graph. However, using nfluxDB templates, we can apply those to our cluster, get dashboards, but other people have built the aggregations to visualize that. But you can see here we have all of these events of traffic that has been sampled by IP tables. I've done some DNS stress testing on this machine, so you're going to see a lot of lookups to 1111, the cloud-flare DNS. And in fact, we can just quickly trigger some more. So we're going to go into GitHub. Let's do DNS stress. And this time I'm going to do 8888. And we'll just let that run for about 10 seconds to get some new sflow data into there. Now, the reason I'm using the DNS stress testing tool is just that I need enough packets heading the network to be sampled and show up in a database. Running your traditional ping or DNS lookup may not be enough for the sampling to kick in. Okay, so we're going to leave that like that. Now, next, I want to apply the nfluxDB templates. So I have this directly. I could use the HTTP URLs, however, I did make a couple of modifications to these templates. And just to show you, I'm going to open one up. You know, these are very Kubernetes-like. You can see I've got labels being applied here. I've got a variable here. I've got some telegraph configurations here, which I'm not actually using. And there's dashboards further up that I won't bore you with right now. However, I can do an nfluxapply-f sflow. And there's a lot of complete, I need to get, I need to ask somebody to fix that. But first, dashboards, there's nothing. We apply this template. We get this nice big list that tells us everything that it's going to create within our nfluxDB server. And we hit go. And I'm going to do the same for s and mp. Put yaml. Oh, that's great. I've got one yml and one yaml. And we set go. And then we go back over here. We have dashboards. I didn't write normal how to write available to me. And I think that's fantastic. So let's pop open our sflow source and destination. We can see traffic blurbs here. This one here was obviously testing with the Cloudflare DNS. And I think if we just drop this down to five minutes, we're going to see those 8888s. Now, I know that's painful to read. But you can see the destination IP is 8888. So we can see that network flow coming through. And if we go to our s and mp system monitoring again, this dashboard knows that I have s and mpd running on the Linux host. And it knows it can read the system monitoring coming from that. So I can see my system uptime, which is zero days. I can see CPU realisation. I'm barely touching this box. Memory allocation again, not really touching must. We've got some CPU load. This will be that DNS stress that I just ran. We can see the memory being freed when it's finished. We've got some use memory climbing, you know, all the basic stuff here and a little bit of IO work. But in fact, DB templates are a fantastic way to really visualise that data, especially from a networking standpoint, so that we don't need to understand those queries and write them ourselves. Because sometimes this will actually be quite challenging. All right, we showed s and mp, s and mp trap and s flow. Let's jump back to the slides. So telegraph and FoxDB really do provide a great way to monitor a network in traffic. But I want to do a quick bonus round before we finish. The problem with plugins is they are opinionated by the person that originally wrote the plugin. To try and be as little configuration as possible will give you as much value, but in the eyes of that single person. And your use case may not be the same as other people. You may want to expand on that or provide new metrics that aren't actually monitored. And this is a networking track. So I also want to talk about BPF because I think we all like working with BPF to one extent or another. So how can we use BPF with telegraph to get metrics into inFlexDB or any other time series database? Well, there's this plugin, relatively new in telegraph land, that came out last year called execd. Execd will allow telegraph to manage the running of any arbitrary application on the machine and handle fetching the metrics from its output and storing them in its normal metric collection pipeline. That means any language, any tool, bash, Ruby, Rust, Go, you want to write as long as you can spit out CSV or JSON, line protocol or any other supportive format, you can write your own plugins that telegraph will manage for you. And it is really, really easy. So what I have here is the Iovizer BCC repository cloned on this machine. I call this repository because it has a really great examples directory. And I'm proud that if you're going through the networking directory underneath examples, you'll see a whole bunch of things that already has BPF scripts available. I thought it would be cool using the DNS matching script here. If we could count how many times domains are looked up using this Keranoff trace. So if I can't my telegraph configuration, you'll see I've just got the flush interval set to five seconds so we can see some data hopefully in close to real time. I provide the command that I want to run, which in this case is just a Python script with this DNS matching code. And I'm telling it, I want to count all the lookups for Google.com. And then can specify a signal that I want to be able to handle. So if we want to be able to reload or restart that process, we can, you know, provide the means to do that. I've used JSON as most languages can already output JSON. So, you know, Python is very easy. I just tell telegraph, hey, this application will print out to standard out JSON documents that we want to be able to parse. We tell how to parse those documents so that we'll have a time key that is a Unix times some value. And there are two other keys on that JSON document that we want to store as tags so that they are indexed. That is the name of the domain that we're trying to resolve and the type of DNS lookup that we're doing. I then use an aggregator plugin, which allows us to in real time and in memory take event based data like a DNS lookup, something that cannot be graphed and build an aggregation on it. So in this case, it is going to every two seconds count how many requests were made to that domain and then give me an aggregated count for that window. And instead of writing to nfluxDB, I'm only going to write this out to standard out. Just means I can run telegraph like so. And then in a few seconds, hopefully we will start to see some data come to the terminal. Right on cue. Now, the reason we're seeing data and I'm not doing anything on this machine is I also have DNS stress running in the background. Looking up Google.com and Microsoft.com. And we can see here we get a lookups on Google.com quite a lot of times every two seconds. I think that's 15,000 up to 30,000 down here. And we're not seeing that monitor Microsoft.com at all because it was not part of our allow list. However, we can modify our telegraph conf and say, hey, you know what, we do want to see Microsoft.com counts. Like so. And we rerun telegraph. And again, and hopefully just a few seconds. Boom, boom, boom. Oh, no. Easy. And in that very simple demo, we have written a BPF trace, had telegraph managed the process. And parsed the JSON output from our exactly plug in and store it in flux DB or in this case, print to the terminal. Now, you may wonder what changes had to be made to this repository for that to work. Well, the only thing I really had to do was add the time module, remove any arbitrary print statements that aren't spitting out the JSON that we want to be able to parse. And then this application would just print all the questions and answers to the terminal. I modified it to do a loop and then print out using an F-formatted string, a JSON document with the counts that I want. The aggregator plug in tallies them up every two seconds and we get our metrics. Couldn't be easier. That's all I have for you today. I hope based on what you've seen, you now know that telegraph is a really interesting, powerful and fantastic application to be part of everyone's monitoring and metric collection pipeline. It has support for over 200 plugins, some really great networking support for SNMP, Traps, Eslo and even BPF if you're willing to write a little bit of code. I'm happy to take any questions now. Thank you for joining me and have a great day. Okay, David, thank you for your talk. Hello, guys. Let's start with Q&A. Martin from the Checking OK team is asking why not version 2 standard. Yes, so actually telegraph uses version 2 as the default value. So if you don't provide a version, you do get version 2. However, for my demo and to keep the config as simple as possible, as simple as it could as SNMP, I didn't want to handle authentication. So I used version 1 just to kind of keep that simple. Yeah, okay. Another question is from the same question. You can speak about figuring out SNMP in the middle of an end point. That wasn't a question. I think I'll touch on that anyway. I think it's a really valid statement from Martin and that configuring SNMP and I can't stress it's enough. It is not trivial or easy by any means. It requires a lot of patience. It requires a lot of debugging just an SNMP walk. Whenever possible, try and find other people that have already done it. That repository that I mentioned by a former colleague of mine is just invaluable for all those major providers of routers. You don't need to build this stuff yourself. However, if you are using more esoteric hardware, then you need to get very comfortable understanding and debugging and build those OIDs and MIBS and contacts. So yeah, not an easy task, unfortunately. Just as I said in your simple network, maybe it's not protocol, I think. Yeah, I think we've got to remember it's an old protocol. It's served us well over the years, but simple is not an extra. No, not sure. Yeah, we don't got any questions at this point. Yeah, I think it was a pretty cool talk. Thank you. Do you want to mention anything else? I think telegraph is an infinitely flexible tool. The architectures on which you can compose and join telegraphs together is awesome. Please do check out the talk that I mentioned earlier on. There's a link in the chat as well. But you can do a lot and using and leveraging the exact the plugin to be able to build your own plugins using native boundaries or scripts, anything on that Linux host really provides. It's almost a super baritur. Shutting point been able to hook BPF and get those values metrics aggregations into your time series database is really, really easy. So, you know, take advantage of exactly and use it to your advantage. Cool. Robin is writing I didn't get so much of the inflex DB one to two changes. What's the early question mark? So, yeah, and flux DB two, the biggest changes are the way that you configure the system primarily. So, you know, inflex DB two exposes a very Kubernetes like manifest format that allows you to apply YAML to your inflex DB server to configure the buckets, the users, the labeling, dashboards, everything in that system is configurable through that YAML interface. That is one of the biggest changes. And that's not mentioned in the talk, which is fairly very, very new. But there is a new storage engine currently being worked on and they open for inflex DB two called I ox, which is worth checking out is written in Rust. So added cool points there. And, but yeah, it may and the UI of course sorry and flux DB two exposes its own UI, which is a major change from the previous chronograph UI that was related with one point X. A lot of differences there built from the ground up really, really cool. So yeah, there are lots of changes, some of them are subtle, but the biggest one for me is a manifest format and declarative that declarative manifest format. Yeah, I think it's pretty cool. I guess people have to try it out and put it in the environment and just to use it so it's a try it out. I guess. So, yeah, I think pretty much that's it. Thank you for your talk. It was pretty nice talk. Oh, wait, anyone's talking again. How many time do we have? I think we've got six minutes. Yeah, sure. So looking at the live screen, my video feed looks like it's filmed on a potato where so yours looks really good. I'm not sure what's happened there. Do you see a MacBook in build? No, I'm going through my nice camera. Okay. And the video set the high. I don't know. So Robin is writing, I've been holding back writing my text in the one to two because it seems kind of odd. Yeah, let me touch on that a little bit then, Robin. See, I don't think that you need to hold back. I think during the influx DB two alphas and even the betas, there was a new storage format that was single sharded that was being experimented with, which was reverted before the GA of influx DB two. That just means that migrating from influx DB one to two is now a lot, lot simpler because you can still point it to the same data directory that you had with influx DB one. And you only have to migrate your dashboards users and buckets. And so it's actually a lot simpler now than it was. So, you know, start looking into that there is helper scripts available. There's also the influx DB community slack. I'm on there as well as all the influx teams so people can help you with that migration. But it's not as daunting as it was six months ago. That's for sure. A lot easier now. Sure. Okay, I think. No, that's it. Thank you for having me. Thank you for the people that asked questions and I hope people got something of value at this talk. Please feel free to reach out to me on Twitter or matrix or anywhere else. I'm always happy to help. Sure. Thank you for your talk. It was pretty nice. Yeah, I wish you a nice day. Thank you. Bye. Bye. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you.
them to InfluxDB, Prometheus, Cassandra, Kafka, monitoring and telegraph ships with plugins that those metrics and write them somewhere. We'll metrics from the proc file system. So you can will find many configurations, many people that running in the background, collecting more also take a look at our traps. We'll click on collection pipeline. It has support for over 200 telegraph is an infinitely flexible tool. The as well as all the influx teams so people can got something of value at this talk. Please feel