3 Rules to Rule Them All: Large-scale Network Monitoring
Hirschvogel, Martin
Wilms, Alex
So you want to monitor a large-scale network- where do you start? This talk will give you some practical tips in strategizing your network monitoring to avoid future problems, detect those you didn’t know are causing performance issues and save your time in configuration. You’ll learn practical tips, summarized into 3 simple rules, coming from the speaker’s 20+ years of experience as a network specialist. Whether you're starting your monitoring from scratch or improving an existing setup, these tips will be useful for you to have a holistic network monitoring. Along with the practical tips, the speaker will show some demos on how you can apply them using Checkmk, an open source IT monitoring software. We will discuss best practices to take advantage of rule-based monitoring in discovering all your network interfaces. After learning the fundamentals, we'll discuss how you can take it further with network flow monitoring using ntopng. This will help you troubleshoot network issues through in-depth network performance monitoring and network flow analyses. By the end of the talk, you’ll get to learn a holistic approach to network monitoring; saving your organization lots of time, as well as budget for hiring external consultants.
 Welcome everyone to our talk, the three rules to rule them all, the magic key to large scale network monitoring. We will start with a short introduction about ourselves and about the tool which we'll be using and then my colleague Alex will go into the three rules to rule them all, which will be a practical session where you can see a lot of things and learn a lot of things and then in the end be available for a Q&A session. So who are we? I'm Martin, I'm a project manager at Tribe 29, Tribe 29 is the company or the team developing CheckMK and Alex with us here is a CheckMK consultant and one of our assist admins and he started with Nagios in 1999 and fell in love with CheckMK in 2011, so almost 10 years now. What is CheckMK? CheckMK is an IT monitoring tool, you can use it for infrastructure and application monitoring and basically you can monitor almost anything, starts with servers, applications, databases, but also containers, container management platforms, cloud services and as with every monitoring tool you digest the data, then the monitoring tool basically suggests okay, this is good, this is not good and then basically the alerting starts afterwards with notifications in a lot of different ways via mail, via SMS, via incident management tools like PagetDuty, then you can also visualize the data, if you don't want to visualize it within CheckMK, you can also visualize it with Grafana for example and connect CheckMK with CMDVs like I do it. What makes CheckMK special is that we have more than 2000 plugins for what we call plug-and-play monitoring, so basically you can just monitor anything and then you do it in a couple of clicks, you don't need to configure a lot of things, it just goes quite smoothly. You can get CheckMK obviously in our open source edition, the CheckMK raw edition, obviously free of charge, it has all these plugins, you can get support by a fantastic community and we follow the open core model, so we do have an enterprise edition with a couple of more features with a better performance and that's where you get support from us. That's already everything about CheckMK, about us and I think enough of an introduction, I will hand over to Alex now who will give the actual talk. Yeah, thank you very much Martin, also one welcome from my side to our talk. So as Martin said, we will talk about the three rules to rule them all, which basically means how to set up CheckMK to do a very efficient network monitoring, so the goal here is we wanted the full visibility of all our network interfaces. It doesn't matter if it is an access port or an uplink port to another switch or a server interface of your ESX server or whatever and of course you want to do it as efficient as possible because every port really means every port, so we are quickly talking about thousands, tens of thousands or even more interfaces that we want to monitor. So we wanted to be able to do that also in large scale networks. So why do we want to do that? Every port can have errors and if it has, yeah, that definitely needs to be addressed and this is also important for end user ports, access ports because if there are errors, also their working experience with clients of applications will for sure be affected, so everything should be almost error free. The challenges end user ports are allowed to change state and speed, so if one goes home, takes a laptop with them, for sure the port will go down, dependent on the firmware implementation of the switch, it may also change speed or not. Some switches keep the last speed in the SNMP output, some others go down to the bare minimum like 10 megabit half duplex and they are allowed to do that. Very important ports, as we call them here for now, like uplinks are definitely not allowed to do that. So if you are uplink to your other switch goes down, most probably there is an unwanted issue, same happens when a server connection goes down from gigabit to 100 megabit, also you want to detect that. This means that we need to address these ports differently because they behave differently or are allowed to behave differently. So one solution for this can be we will use port names to determine between access and non-access ports. So to keep it simple, we could simply name just our important ports. So for example, if you have an uplink to switch number three, call it uplink switch three within your switch and check in case able to detect that alias. The next challenge is some windows, even if for example on Cisco the command is called description, they are not writing it into the SNMP description table but into the alias table. So we need a way to switch the tables that we are using for the port names. We need to be able to switch between those tables. So the solution is we will create two rules to switch between them and also we will create a so-called host tech which is the host property where we can say use this or use the other one. So it's demo time, so let me quickly switch to our demo. Here I'm going to install a check in case the raw edition. This is the upcoming 2.0 version here in beta stadium. And after installing, when it's done, I will create a site. So here I'm going to create a site called foster and I will set the admin password directly to foster so that we need to know how to log in. After creating, we can already start it. And we're done. So one quick introduction to a site. What is a site? It's a self-contained monitoring environment that can be independent from other sites. As example, you could create one site called production, another site called test and you can run tests in your test site without affecting the production. So it's a completely independent monitoring environment on users, on plugins, on setups, on rules, everything separate. Or it can be so-called distributed monitoring. So if you have really large scale setup, you could scale out and share the monitoring load over various instances. There is virtually no limit on how many instances you can run check and K, but managed from one central site. Another reason for distributed monitoring can be geographical reasons. So if your main headquarters is in Brussels and another location is in China, it doesn't make sense to monitor the location Shanghai from Brussels because of the latency. So it makes sense to create a site in China, a site in Russia, or wherever you need and manage them from the central site. So this is also possible. Now it's time to switch to our browser and to log in into our newly created site. So here it is and I will look in with the built-in administrator, which is check and K admin, CMK admin, and the password that I created, which is for STEM in this case. So what we see here now is not much. We see an empty site. We have zero services, but we are going to change that. Because of the screen resolution, I will for now get rid of this sidebar and turn it off for now. And now we are going to create our first host. So for that, we are going to our setup module and we're going to host. And we see here that we have the possibility to create a host or a folder. I want to create a folder here for now. And I will call it which and on a switch, we will have SNMP. So I'm turning on SNMP V2 here in this case with the credential public. Actually you don't need to do this because if you don't give anything here, we will use the community public by default. But let's do it like this. On the other hand, we don't have the data source normal check and K agent. Check and K agent is a small lightweight agent, which is intended for the normal operating systems like Linux, Windows, FreeBSD, whatever. Obviously we can't install it on a switch. So I'm turning it off here and I'm saying no on this. We don't have an agent. So all these settings here will be inherited starting from this folder downwards to subfolders or hosts within this folder. So we will save that. And when we create here now a host. All these settings are inherited here. So SNMP V2 with the community public. And after this we can already go and say, save and go to service configuration. So this will basically do an SNMP scan of this device and it will already detect some services. But it will do it in a way that we don't want that way, we will optimize it a little bit. Why do we want to do that? Because here we see interface but enlisted by interface index numbers. And we also see that something is missing here. So we have number one, two, three and four is missing. Five is here. Plus we see here the descriptions that have been set within the switch for the support. And yeah, set in the introduction, we want this as the service name. So I'm not going to save this here yet. I will do first the switch, the host attack to be able to switch between the alias and or the description table. So we will add a tech group here. We will call it internally if alias description maybe. Maybe without typos. We will add two tech choices which will result later in the host configuration property in another drop down as you've seen for the check and K or the SNMP settings. And the tech ID which is on top will be the default. So we should set here the most common setting and the most common setting is that manufacturers nowadays use the alias table. So let's give it the internal tech ID if alias. And for the second option, we want the description table. So use description. I'm saying safe. Now this tech exists and we have here a preview and this is the drop down that we created now. This tech by itself just exists. It does nothing. So we will need to connect it with the rule so that the switching the tech influences which rule is used. For that we're going to our setup module again. And now, yeah, because we have around 2000 plugins already, we have a lot of rules. So the best thing to do is to search here. So we will search for interface. And here we find the rule network interface and switch port discovery. This influences how to discover these interfaces. So we will create a rule. And this rule, the first setting that we will do is the appearance of the network interface. We will change that we don't want this index number. We want to use alias in case that the host tech that we created. So let's search here for interface and access as a condition to this rule. So if this tech is set to use alias, we will use alias. We basically don't need to set anything else here by this implicitly, we will turn on other things. We will turn on a match all interfaces, so we will also find the missing interfaces that we didn't find so far yet, because they were down. This could be specified much more in much more detail. But as we want to go as deep and as wide as possible with the monitoring and we want all interfaces, we basically leave it to all interfaces, because all interfaces matter. If they're up and causing errors, that's a problem. If they're up and causing a no error, also not problem. If they're down, they're not causing errors. But if they go up one day and start to cause errors, you will detect that. If you're not searching for every interface, you will miss that error and you don't want that. So all these settings are applied to conditions now. Here I will put the condition to the main directory, because what I'm setting up here will scale for every network interface. It doesn't matter if it's a network interface in a Windows server, Linux server, a switch, a firewall, a router, whatever. This will scale over your whole monitoring. So we're going to save that. And so far we're just using alias, but we need to create a second rule for the description. So we will simply clone that one. We will switch that to description. And we'll toggle here to use description, and then we have our second rule. We can go back to our host. And we can do another service discovery. And you will see a big difference here now. So all the interface numbers are gone, but we're on the ports where we configured names, like example on these access points or this firewall, we see the names. And this is what we can utilize later on to determine, are we talking about access port, which is a lot to go down to change speed, or are we talking about a port to my firewall, for example, the firewall, it shouldn't change speed, obviously. So now we can say, monitor undecided services. Then we can go and apply our changes and activate them. Now I will turn on the sidebar again, because now you will see we have one host, 35 services. Let's go to the services. At the moment they are still pending. While they are still pending, I will create the next rule. I will steal this service description because this is an unimported. And then my naming concept, un-name port is an access port. And I want to create now the rule to ignore state and speed changes. So I'm going to our setup module again. I will search again for interface. But this time not for network interface and switch port discovery, because we did this already. Now we want to influence how to deal with the result with the discovered services. So we will go to the common network interface and switch port rule. Also here we will create a rule again in the main directory. We will change the condition that it only applies to specific ports. So it will match all ports that start with gigabit ethernet, for example. In other cases, you also need for un-configured ports or un-named ports, you will need to add 10 gigabit ethernet, 40 gigabit ethernet on other switches like this, for example, for Cisco and Huawei switches. For HP switches, you will need to address it by number. So if they don't have a name, they will simply call it interface 001, 002, and so on. So let's create a regular expression here. When this interface description starts with several digits, then we want to match this rule. What do we want to do in this rule? We want to ignore the speed, because this is our access port. We want to ignore the state. And that's basically it. Here we can give a little comment so that also our colleagues know why we are setting up this rule. So it's not important to explain here what you're doing, because that's obviously visible in the rule. Here, it should be documented why you're doing it. You could give a description here like according to our naming scheme, unnamed interfaces like interface 001, or gigabit ethernet, or 40 and so on, our access ports are allowed to change state and speed, so basically everything why we are talking about this. That's being done. We can apply this. And nothing much will change. The change will only be visible during monitoring. If this will change state and speed, it will go red. If this one will change state and speed, it will stay green. So even if it's now, for example, up, if it will change speed to down, it definitely will go red and will give you a, it will not, sorry, the access port. This one, of course, will stay green. But all the name ports will get red if they change state or speed. Yeah, that's being said. We basically set everything up that we need now. So what we can do now, we can go again to our setup module. We can go to host. We will go again to switch and we will add another host. Again, because our rule and full structure, because we are not template based, all rules and settings are inherited from top, so we don't need to change anything here. I'm just giving it the DNS name. I'm saying save and go to service configuration. This is actually a quite big switch. So, and it's also in a remote location. So I'm expecting a longer runtime here for not making you bored waiting for this. We will do something else while it's running in the background. We background, we will go to host and we will create a new folder here. Oops. Sorry, this click was too fast. Add subfolder and we will create a folder called server here. On a server, we can install the check-in-k agent. We don't need to use SNMP and most probably shouldn't because you will get much more information and much more performance, with much more performance from the check-in-k agent. So we don't need to change anything here. We save the setting, this folder. We go here and we create a host. So let's add my approximate virtualization server here. Save and go to services. And here we go. Here are the services. And also when you scroll down here, you also see here that you get here the Linux interface names. So this is also an effect of applying these three rules that we created now also to everything because we put it into the main directory just being bound to this host tech. That's exactly what we want. And also here we can say monitor undecided services. Also I will show you again the properties. Here in the custom attributes, you will find this use allias because I made it the default in case that you get an unwanted result because here you put in an older HP switch or some firewalls that require that that you use the description table. Simply switch it here to use description and you will switch the discovery process to use the other as an MP table. That's being said. We can activate our changes again. Please look again at the sidebar. Three host 94 services. What still is missing here is a big switch. So we will go here again and go to the switch folder and go on our core switch. Now it will load what it has discovered before. So you see all the interfaces are there. There are many unnamed interfaces as you see here. But here also named interfaces. For example, we see VLAN interfaces here. We see named firewall ports now where the firewall is connected to and these things. And here we still see that it's on module A port 8. So here we can say monitor undecided. You can apply our changes. Again, I ask you to take a look at the service numbers. So still three host, but suddenly more than 200 services more because we detected them very efficiently. This is basically everything that you need to set up to have a common approach for all network interfaces in your network. It doesn't matter if you're talking about 50 host or 5000 or 50,000 host, you can apply that. So another thing that I quickly want to show you because this track that we're talking here is the talk to network monitoring and discovery or inventory. So I want to show you an inventory function. We're going to our setup module and we are searching for inventory. So we have a hardware and software inventory module and we will say do hardware and software inventory. Here are already some predefined rules, but they are just for checking case servers and Docker objects, but we want a common rule for everything. And we simply add here that we want to do also start a starter inventory. So that's enough. We don't need to apply any other conditions then do it in the main directory, do it on the global level, do it for everything. So we will save it. We will activate our changes. And we can go to host again here. For example, let's go to the smaller switch. I will do a rescue here to be a little bit quicker. And we see that we have a lot of inventory here already. Now somebody could ask why are you doing hardware and software inventory on a switch? Is the software changing? No, of course not. Hardware changing. Yeah, no module switch. It can happen. You can swap in modules or do something else. So simply let's go to show hardware and software inventory for the source. And here we can look at the hardware, for example, what kind of device do we have? Which physical components does it have? Does it have modules? Yeah, not really because that's a small switch, but that's actually not what I want to show you. I wanted to show you this because when I switch the sidebar, when I open the interface index, we can see here all the interfaces that we have. And we can see that here are ports which are right now down, but they are in the state used because a couple of days ago, this port was up. So we have a configurable threshold that we can set here before we declare a port as free. So maybe you know the industry standard of patching a cable. You want to patch a cable, you go down to your network rack, you open the network rack and you get shocked because of all the cable mess that you will find there. Also, you will find out that the whole switch is occupied already by patch cables and you think, oh man, this can't be. So it seems that colleagues didn't clean up when decommissioning a PC. So there the industry standard is, you will look for an port which is down. You will pray, you will plug out the cable, you will plug in your cable and hope that nobody will complain. And of course, what happens the next day is the manager is coming saying, I can't reach Google. So this can't happen with this status data that we're already getting from the switch. So that's a really nice feature about this hardware inventory on switches. Let's come back to the switch again, of course, I want to show you something. So when we go to the switch, you also see that we already get some metrics here and we can look deeper into the metrics. Of course, we're monitoring since the switch now since 10 minutes, so we don't have a lot of data here. But I wanted to give you a quick overview of what we're doing. So we are calculating the bandwidth in and out. This is a minimum maximum average. We see that we are recording this or that we have predefined views here for the last four hours, last 25 hours, so one day plus one hour or one week plus one day, a little bit more than a month, 400 days, a little bit more than a year. By default, we are recording here 720 days, which can be adjusted. So also here, when you switch here, of course, we don't have values here. One could think, what are we storing here? Because here we are sending out the data because we are not saving the data every minute, but we are taking the most significant bits, let's say, that we are storing. But for what? For minimum maximum average. Actually, we're doing for all. So you can switch here between minimum maximum average. Of course, again, because we're doing it since five minutes, you don't see much here. For a small feature, you can set a so-called needle here, which will mark exactly where this needle is. So you see the date and the exact values at that time. And this needle is saved. So even if you compare it with the filling rate of your Drive C on your Windows PC, you will find this needle there as well so that you can compare values between services and even hosts. We are not recording only the bandwidth, but also the error rate. As already said before, this error rate should be always almost zero. We have very low thresholds for that by default. 0.01% for warning and 0.1% of the total traffic for critical. My advice is never, ever change it except for wireless networks. Wireless networks are by definition shared networks. So they are shared with the neighboring access point, with your own access point, but also with atmospheric influences and lightning. Their error rates are pretty normal, but on everything that is connected to a cable, there shouldn't be errors. So don't change it. Also, we see broadcast and multicast rates for input and output. So you could detect broadcast storms and things like that here. We see this also for non-unicast, so multicast and broadcast together, versus unicast packages. And we can also monitor the length of the output queue of the switch if the switch is securing it to many packages instead of transferring them over the cable. So now we are coming to distributed monitoring. I want to show you a connection or how it looks when I'm doing a connection to one of our test sites. So again, I said mainly two reasons why to do distributed monitoring, scale out, or geographical reasons as I said in the introduction. And you can configure your connection to push the configuration so that you manage your remote sites from the central site, or you can create some breeder only. In this case, I will create some breed only. Let's call it FOSSTEM. Give it an alias FOSSTEM. We will connect to a host. In this case, it's the test site. We don't use the TLS encryption there, so I will do it in this case unencrypted, but of course in production you should do it encrypted. And I will choose here no replication with the site. It will be read only. So actually let me correct that one. And I need to give two more settings here. You need an URL prefix, how to reach that. Normally also, of course, you should do it via HTTP S, TLS encrypted. But as we are here in a local network, I can do it without. Okay, of course, this was a bad idea because our main site is already called FOSSTEM. So let's do it like this. That's being said, we can apply the changes. And I'm turning on this sidebar again so that you see what's happening here. And suddenly you see we have one on our line hosts and 4,055 services. So within one second you connect to an already existing site to get a central overview. And for example, if we look at here, we directly see some errors that are on the other side. For example, here are some services that need to be restarted because they have been patched recently or some updates have to be installed and things like that. Last but not least, because we are talking about network monitoring and today there will be two other talks by our colleagues from NTOB NG, Luca. We partnered up with NTOB to integrate NTOB into CheckMK to be able to see information that is gathered by NTOB in CheckMK to enrich CheckMK host information with information that is recorded by NTOB. For that I will switch to one of our test sites because the error is connected to NTOB. The test site is called Production. Let me switch back to the full screen. And here we could look for the NTOB information. I am struggling a little bit because of the video resolution here. Normally this doesn't look so big. The screen resolution here unfortunately has to be very small. According to FOSTA requirements. But here we will click on NTOB host and we see that there are hosts which have some NTOB information included. So let's go here to the host information and here let's get rid of the sidebar again. Again the screen resolution is actually too low. We see that we have some NTOB information. We have no engaged alerts for the source. There is no active alert flow. We see the traffic distribution sent to where those received and so on. We see the ports that NTOB detected in the network flows. We see the protocol breakdown TCP versus other protocols and with which peers communicating over which ports and protocols. So we can go to the traffic overview. So we see which traffic is creating the TCP Flack distribution. Port distribution which ports are unused with which peers it is communicating. So for example here it is communicating with one of our build servers. It is communicating with our firewall which application categories. So basically here a lot of check and K traffic is recorded. That's because this one is in a network which is a distance from our monitoring system and we are recording on the firewall and that's why we are just recording check and K for this particular host. We see the flows so maybe not so interesting. So maybe let's look quickly for another host. Here for example we have a firewall. So let's look here at the traffic and when we look at the flows here we should see different things. So we see some unknown protocols SNMP, other engaged alerts, no other past alerts. So this is the quick overview about our upcoming NTOB integration. Please take also look at the talks from NTOB from Luca and his colleague. That's it from my side so far. If you want to contact us you can do that via Twitter and of course you can write to us and also we have a stand at the first step. Of course we will join now the Q&A session. So thank you very much for your time and hope to talk to you soon.
 Welcome everyone to our talk, the three rules to rule them all, the magic key to large scale  network monitoring. We will start with a short introduction about ourselves and about the  tool which we'll be using and then my colleague Alex will go into the three rules to rule them  all, which will be a practical session where you can see a lot of things and learn a lot of  things and then in the end be available for a Q&A session. So who are we? I'm Martin,  I'm a project manager at Tribe 29, Tribe 29 is the company or the team developing CheckMK and  Alex with us here is a CheckMK consultant and one of our assist admins and he started with  Nagios in 1999 and fell in love with CheckMK in 2011, so almost 10 years now. What is CheckMK?  CheckMK is an IT monitoring tool, you can use it for infrastructure and application monitoring  and basically you can monitor almost anything, starts with servers, applications, databases,  but also containers, container management platforms, cloud services and as with every  monitoring tool you digest the data, then the monitoring tool basically suggests okay,  this is good, this is not good and then basically the alerting starts afterwards with notifications  in a lot of different ways via mail, via SMS, via incident management tools like PagetDuty,  then you can also visualize the data, if you don't want to visualize it within CheckMK,  you can also visualize it with Grafana for example and connect CheckMK with CMDVs like I do it.  What makes CheckMK special is that we have more than 2000 plugins for what we call  plug-and-play monitoring, so basically you can just monitor anything and then you do it in a  couple of clicks, you don't need to configure a lot of things, it just goes quite smoothly.  You can get CheckMK obviously in our open source edition, the CheckMK raw edition,  obviously free of charge, it has all these plugins, you can get support by a fantastic  community and we follow the open core model, so we do have an enterprise edition with a  couple of more features with a better performance and that's where you get support from us.  That's already everything about CheckMK, about us and I think enough of an introduction,  I will hand over to Alex now who will give the actual talk.  Yeah, thank you very much Martin, also one welcome from my side to our talk.  So as Martin said, we will talk about the three rules to rule them all,  which basically means how to set up CheckMK to do a very efficient network monitoring,  so the goal here is we wanted the full visibility of all our network interfaces.  It doesn't matter if it is an access port or an uplink port to another switch or a server  interface of your ESX server or whatever and of course you want to do it as efficient as  possible because every port really means every port, so we are quickly talking about thousands,  tens of thousands or even more interfaces that we want to monitor.  So we wanted to be able to do that also in large scale networks.  So why do we want to do that?  Every port can have errors and if it has, yeah, that definitely needs to be addressed  and this is also important for end user ports, access ports because if there are errors,  also their working experience with clients of applications will for sure be affected,  so everything should be almost error free.  The challenges end user ports are allowed to change state and speed, so if one goes home,  takes a laptop with them, for sure the port will go down, dependent on the firmware implementation  of the switch, it may also change speed or not.  Some switches keep the last speed in the SNMP output, some others go down to the bare minimum  like 10 megabit half duplex and they are allowed to do that.  Very important ports, as we call them here for now, like uplinks are definitely not  allowed to do that.  So if you are uplink to your other switch goes down, most probably there is an unwanted  issue, same happens when a server connection goes down from gigabit to 100 megabit, also  you want to detect that.  This means that we need to address these ports differently because they behave differently  or are allowed to behave differently.  So one solution for this can be we will use port names to determine between access and  non-access ports.  So to keep it simple, we could simply name just our important ports.  So for example, if you have an uplink to switch number three, call it uplink switch three  within your switch and check in case able to detect that alias.  The next challenge is some windows, even if for example on Cisco the command is called  description, they are not writing it into the SNMP description table but into the alias  table.  So we need a way to switch the tables that we are using for the port names.  We need to be able to switch between those tables.  So the solution is we will create two rules to switch between them and also we will create  a so-called host tech which is the host property where we can say use this or use the other  one.  So it's demo time, so let me quickly switch to our demo.  Here I'm going to install a check in case the raw edition.  This is the upcoming 2.0 version here in beta stadium.  And after installing, when it's done, I will create a site.  So here I'm going to create a site called foster and I will set the admin password directly  to foster so that we need to know how to log in.  After creating, we can already start it.  And we're done.  So one quick introduction to a site.  What is a site?  It's a self-contained monitoring environment that can be independent from other sites.  As example, you could create one site called production, another site called test and you  can run tests in your test site without affecting the production.  So it's a completely independent monitoring environment on users, on plugins, on setups,  on rules, everything separate.  Or it can be so-called distributed monitoring.  So if you have really large scale setup, you could scale out and share the monitoring load  over various instances.  There is virtually no limit on how many instances you can run check and K, but managed from  one central site.  Another reason for distributed monitoring can be geographical reasons.  So if your main headquarters is in Brussels and another location is in China, it doesn't  make sense to monitor the location Shanghai from Brussels because of the latency.  So it makes sense to create a site in China, a site in Russia, or wherever you need and  manage them from the central site.  So this is also possible.  Now it's time to switch to our browser and to log in into our newly created site.  So here it is and I will look in with the built-in administrator, which is check and  K admin, CMK admin, and the password that I created, which is for STEM in this case.  So what we see here now is not much.  We see an empty site.  We have zero services, but we are going to change that.  Because of the screen resolution, I will for now get rid of this sidebar and turn it off  for now.  And now we are going to create our first host.  So for that, we are going to our setup module and we're going to host.  And we see here that we have the possibility to create a host or a folder.  I want to create a folder here for now.  And I will call it which and on a switch, we will have SNMP.  So I'm turning on SNMP V2 here in this case with the credential public.  Actually you don't need to do this because if you don't give anything here, we will use  the community public by default.  But let's do it like this.  On the other hand, we don't have the data source normal check and K agent.  Check and K agent is a small lightweight agent, which is intended for the normal operating  systems like Linux, Windows, FreeBSD, whatever.  Obviously we can't install it on a switch.  So I'm turning it off here and I'm saying no on this.  We don't have an agent.  So all these settings here will be inherited starting from this folder downwards to subfolders  or hosts within this folder.  So we will save that.  And when we create here now a host.  All these settings are inherited here.  So SNMP V2 with the community public.  And after this we can already go and say, save and go to service configuration.  So this will basically do an SNMP scan of this device and it will already detect some  services.  But it will do it in a way that we don't want that way, we will optimize it a little bit.  Why do we want to do that?  Because here we see interface but enlisted by interface index numbers.  And we also see that something is missing here.  So we have number one, two, three and four is missing.  Five is here.  Plus we see here the descriptions that have been set within the switch for the support.  And yeah, set in the introduction, we want this as the service name.  So I'm not going to save this here yet.  I will do first the switch, the host attack to be able to switch between the alias and  or the description table.  So we will add a tech group here.  We will call it internally if alias description maybe.  Maybe without typos.  We will add two tech choices which will result later in the host configuration property in  another drop down as you've seen for the check and K or the SNMP settings.  And the tech ID which is on top will be the default.  So we should set here the most common setting and the most common setting is that manufacturers  nowadays use the alias table.  So let's give it the internal tech ID if alias.  And for the second option, we want the description table.  So use description.  I'm saying safe.  Now this tech exists and we have here a preview and this is the drop down that we created  now.  This tech by itself just exists.  It does nothing.  So we will need to connect it with the rule so that the switching the tech influences  which rule is used.  For that we're going to our setup module again.  And now, yeah, because we have around 2000 plugins already, we have a lot of rules.  So the best thing to do is to search here.  So we will search for interface.  And here we find the rule network interface and switch port discovery.  This influences how to discover these interfaces.  So we will create a rule.  And this rule, the first setting that we will do is the appearance of the network interface.  We will change that we don't want this index number.  We want to use alias in case that the host tech that we created.  So let's search here for interface and access as a condition to this rule.  So if this tech is set to use alias, we will use alias.  We basically don't need to set anything else here by this implicitly, we will turn on other  things.  We will turn on a match all interfaces, so we will also find the missing interfaces that  we didn't find so far yet, because they were down.  This could be specified much more in much more detail.  But as we want to go as deep and as wide as possible with the monitoring and we want  all interfaces, we basically leave it to all interfaces, because all interfaces matter.  If they're up and causing errors, that's a problem.  If they're up and causing a no error, also not problem.  If they're down, they're not causing errors.  But if they go up one day and start to cause errors, you will detect that.  If you're not searching for every interface, you will miss that error and you don't want  that.  So all these settings are applied to conditions now.  Here I will put the condition to the main directory, because what I'm setting up here  will scale for every network interface.  It doesn't matter if it's a network interface in a Windows server, Linux server, a switch,  a firewall, a router, whatever.  This will scale over your whole monitoring.  So we're going to save that.  And so far we're just using alias, but we need to create a second rule for the description.  So we will simply clone that one.  We will switch that to description.  And we'll toggle here to use description, and then we have our second rule.  We can go back to our host.  And we can do another service discovery.  And you will see a big difference here now.  So all the interface numbers are gone, but we're on the ports where we configured names,  like example on these access points or this firewall, we see the names.  And this is what we can utilize later on to determine, are we talking about access port,  which is a lot to go down to change speed, or are we talking about a port to my firewall,  for example, the firewall, it shouldn't change speed, obviously.  So now we can say, monitor undecided services.  Then we can go and apply our changes and activate them.  Now I will turn on the sidebar again, because now you will see we have one host, 35 services.  Let's go to the services.  At the moment they are still pending.  While they are still pending, I will create the next rule.  I will steal this service description because this is an unimported.  And then my naming concept, un-name port is an access port.  And I want to create now the rule to ignore state and speed changes.  So I'm going to our setup module again.  I will search again for interface.  But this time not for network interface and switch port discovery, because we did this  already.  Now we want to influence how to deal with the result with the discovered services.  So we will go to the common network interface and switch port rule.  Also here we will create a rule again in the main directory.  We will change the condition that it only applies to specific ports.  So it will match all ports that start with gigabit ethernet, for example.  In other cases, you also need for un-configured ports or un-named ports, you will need to  add 10 gigabit ethernet, 40 gigabit ethernet on other switches like this, for example,  for Cisco and Huawei switches.  For HP switches, you will need to address it by number.  So if they don't have a name, they will simply call it interface 001, 002, and so on.  So let's create a regular expression here.  When this interface description starts with several digits, then we want to match this  rule.  What do we want to do in this rule?  We want to ignore the speed, because this is our access port.  We want to ignore the state.  And that's basically it.  Here we can give a little comment so that also our colleagues know why we are setting  up this rule.  So it's not important to explain here what you're doing, because that's obviously visible  in the rule.  Here, it should be documented why you're doing it.  You could give a description here like according to our naming scheme, unnamed interfaces like  interface 001, or gigabit ethernet, or 40 and so on, our access ports are allowed to  change state and speed, so basically everything why we are talking about this.  That's being done.  We can apply this.  And nothing much will change.  The change will only be visible during monitoring.  If this will change state and speed, it will go red.  If this one will change state and speed, it will stay green.  So even if it's now, for example, up, if it will change speed to down, it definitely  will go red and will give you a, it will not, sorry, the access port.  This one, of course, will stay green.  But all the name ports will get red if they change state or speed.  Yeah, that's being said.  We basically set everything up that we need now.  So what we can do now, we can go again to our setup module.  We can go to host.  We will go again to switch and we will add another host.  Again, because our rule and full structure, because we are not template based, all rules  and settings are inherited from top, so we don't need to change anything here.  I'm just giving it the DNS name.  I'm saying save and go to service configuration.  This is actually a quite big switch.  So, and it's also in a remote location.  So I'm expecting a longer runtime here for not making you bored waiting for this.  We will do something else while it's running in the background.  We background, we will go to host and we will create a new folder here.  Oops.  Sorry, this click was too fast.  Add subfolder and we will create a folder called server here.  On a server, we can install the check-in-k agent.  We don't need to use SNMP and most probably shouldn't because you will get much more information  and much more performance, with much more performance from the check-in-k agent.  So we don't need to change anything here.  We save the setting, this folder.  We go here and we create a host.  So let's add my approximate virtualization server here.  Save and go to services.  And here we go.  Here are the services.  And also when you scroll down here, you also see here that you get here the Linux interface  names.  So this is also an effect of applying these three rules that we created now also to everything  because we put it into the main directory just being bound to this host tech.  That's exactly what we want.  And also here we can say monitor undecided services.  Also I will show you again the properties.  Here in the custom attributes, you will find this use allias because I made it the default  in case that you get an unwanted result because here you put in an older HP switch or some  firewalls that require that that you use the description table.  Simply switch it here to use description and you will switch the discovery process to use  the other as an MP table.  That's being said.  We can activate our changes again.  Please look again at the sidebar.  Three host 94 services.  What still is missing here is a big switch.  So we will go here again and go to the switch folder and go on our core switch.  Now it will load what it has discovered before.  So you see all the interfaces are there.  There are many unnamed interfaces as you see here.  But here also named interfaces.  For example, we see VLAN interfaces here.  We see named firewall ports now where the firewall is connected to and these things.  And here we still see that it's on module A port 8.  So here we can say monitor undecided.  You can apply our changes.  Again, I ask you to take a look at the service numbers.  So still three host, but suddenly more than 200 services more because we detected them  very efficiently.  This is basically everything that you need to set up to have a common approach for all  network interfaces in your network.  It doesn't matter if you're talking about 50 host or 5000 or 50,000 host, you can apply  that.  So another thing that I quickly want to show you because this track that we're talking  here is the talk to network monitoring and discovery or inventory.  So I want to show you an inventory function.  We're going to our setup module and we are searching for inventory.  So we have a hardware and software inventory module and we will say do hardware and software  inventory.  Here are already some predefined rules, but they are just for checking case servers and  Docker objects, but we want a common rule for everything.  And we simply add here that we want to do also start a starter inventory.  So that's enough.  We don't need to apply any other conditions then do it in the main directory, do it on  the global level, do it for everything.  So we will save it.  We will activate our changes.  And we can go to host again here.  For example, let's go to the smaller switch.  I will do a rescue here to be a little bit quicker.  And we see that we have a lot of inventory here already.  Now somebody could ask why are you doing hardware and software inventory on a switch?  Is the software changing?  No, of course not.  Hardware changing.  Yeah, no module switch.  It can happen.  You can swap in modules or do something else.  So simply let's go to show hardware and software inventory for the source.  And here we can look at the hardware, for example, what kind of device do we have?  Which physical components does it have?  Does it have modules?  Yeah, not really because that's a small switch, but that's actually not what I want  to show you.  I wanted to show you this because when I switch the sidebar, when I open the interface index,  we can see here all the interfaces that we have.  And we can see that here are ports which are right now down, but they are in the state  used because a couple of days ago, this port was up.  So we have a configurable threshold that we can set here before we declare a port as  free.  So maybe you know the industry standard of patching a cable.  You want to patch a cable, you go down to your network rack, you open the network rack  and you get shocked because of all the cable mess that you will find there.  Also, you will find out that the whole switch is occupied already by patch cables and you  think, oh man, this can't be.  So it seems that colleagues didn't clean up when decommissioning a PC.  So there the industry standard is, you will look for an port which is down.  You will pray, you will plug out the cable, you will plug in your cable and hope that  nobody will complain.  And of course, what happens the next day is the manager is coming saying, I can't reach  Google.  So this can't happen with this status data that we're already getting from the switch.  So that's a really nice feature about this hardware inventory on switches.  Let's come back to the switch again, of course, I want to show you something.  So when we go to the switch, you also see that we already get some metrics here and  we can look deeper into the metrics.  Of course, we're monitoring since the switch now since 10 minutes, so we don't have a lot  of data here.  But I wanted to give you a quick overview of what we're doing.  So we are calculating the bandwidth in and out.  This is a minimum maximum average.  We see that we are recording this or that we have predefined views here for the last  four hours, last 25 hours, so one day plus one hour or one week plus one day, a little  bit more than a month, 400 days, a little bit more than a year.  By default, we are recording here 720 days, which can be adjusted.  So also here, when you switch here, of course, we don't have values here.  One could think, what are we storing here?  Because here we are sending out the data because we are not saving the data every minute, but  we are taking the most significant bits, let's say, that we are storing.  But for what?  For minimum maximum average.  Actually, we're doing for all.  So you can switch here between minimum maximum average.  Of course, again, because we're doing it since five minutes, you don't see much here.  For a small feature, you can set a so-called needle here, which will mark exactly where  this needle is.  So you see the date and the exact values at that time.  And this needle is saved.  So even if you compare it with the filling rate of your Drive C on your Windows PC, you  will find this needle there as well so that you can compare values between services and  even hosts.  We are not recording only the bandwidth, but also the error rate.  As already said before, this error rate should be always almost zero.  We have very low thresholds for that by default.  0.01% for warning and 0.1% of the total traffic for critical.  My advice is never, ever change it except for wireless networks.  Wireless networks are by definition shared networks.  So they are shared with the neighboring access point, with your own access point, but also  with atmospheric influences and lightning.  Their error rates are pretty normal, but on everything that is connected to a cable, there  shouldn't be errors.  So don't change it.  Also, we see broadcast and multicast rates for input and output.  So you could detect broadcast storms and things like that here.  We see this also for non-unicast, so multicast and broadcast together, versus unicast packages.  And we can also monitor the length of the output queue of the switch if the switch is  securing it to many packages instead of transferring them over the cable.  So now we are coming to distributed monitoring.  I want to show you a connection or how it looks when I'm doing a connection to one of  our test sites.  So again, I said mainly two reasons why to do distributed monitoring, scale out, or geographical  reasons as I said in the introduction.  And you can configure your connection to push the configuration so that you manage your  remote sites from the central site, or you can create some breeder only.  In this case, I will create some breed only.  Let's call it FOSSTEM.  Give it an alias FOSSTEM.  We will connect to a host.  In this case, it's the test site.  We don't use the TLS encryption there, so I will do it in this case unencrypted, but  of course in production you should do it encrypted.  And I will choose here no replication with the site.  It will be read only.  So actually let me correct that one.  And I need to give two more settings here.  You need an URL prefix, how to reach that.  Normally also, of course, you should do it via HTTP S, TLS encrypted.  But as we are here in a local network, I can do it without.  Okay, of course, this was a bad idea because our main site is already called FOSSTEM.  So let's do it like this.  That's being said, we can apply the changes.  And I'm turning on this sidebar again so that you see what's happening here.  And suddenly you see we have one on our line hosts and 4,055 services.  So within one second you connect to an already existing site to get a central overview.  And for example, if we look at here, we directly see some errors that are on the other side.  For example, here are some services that need to be restarted because they have been patched  recently or some updates have to be installed and things like that.  Last but not least, because we are talking about network monitoring and today there will  be two other talks by our colleagues from NTOB NG, Luca.  We partnered up with NTOB to integrate NTOB into CheckMK to be able to see information  that is gathered by NTOB in CheckMK to enrich CheckMK host information with information that  is recorded by NTOB.  For that I will switch to one of our test sites because the error is connected to NTOB.  The test site is called Production.  Let me switch back to the full screen.  And here we could look for the NTOB information.  I am struggling a little bit because of the video resolution here.  Normally this doesn't look so big.  The screen resolution here unfortunately has to be very small.  According to FOSTA requirements.  But here we will click on NTOB host and we see that there are hosts which have some NTOB  information included.  So let's go here to the host information and here let's get rid of the sidebar again.  Again the screen resolution is actually too low.  We see that we have some NTOB information.  We have no engaged alerts for the source.  There is no active alert flow.  We see the traffic distribution sent to where those received and so on.  We see the ports that NTOB detected in the network flows.  We see the protocol breakdown TCP versus other protocols and with which peers communicating  over which ports and protocols.  So we can go to the traffic overview.  So we see which traffic is creating the TCP Flack distribution.  Port distribution which ports are unused with which peers it is communicating.  So for example here it is communicating with one of our build servers.  It is communicating with our firewall which application categories.  So basically here a lot of check and K traffic is recorded.  That's because this one is in a network which is a distance from our monitoring system and  we are recording on the firewall and that's why we are just recording check and K for  this particular host.  We see the flows so maybe not so interesting.  So maybe let's look quickly for another host.  Here for example we have a firewall.  So let's look here at the traffic and when we look at the flows here we should see different  things.  So we see some unknown protocols SNMP, other engaged alerts, no other past alerts.  So this is the quick overview about our upcoming NTOB integration.  Please take also look at the talks from NTOB from Luca and his colleague.  That's it from my side so far.  If you want to contact us you can do that via Twitter and of course you can write to  us and also we have a stand at the first step.  Of course we will join now the Q&A session.  So thank you very much for your time and hope to talk to you soon.
nemolargescalemon_0_4187.png nemolargescalemon_4188_9203.png nemolargescalemon_9204_13043.png nemolargescalemon_13044_14483.png nemolargescalemon_14484_14711.png nemolargescalemon_14712_14951.png nemolargescalemon_14952_15263.png nemolargescalemon_15264_15383.png nemolargescalemon_15384_15791.png nemolargescalemon_15792_15887.png nemolargescalemon_15888_16151.png nemolargescalemon_16152_20471.png nemolargescalemon_20472_21695.png nemolargescalemon_21696_21743.png nemolargescalemon_21744_23399.png nemolargescalemon_23400_24539.png nemolargescalemon_24540_24923.png nemolargescalemon_24924_25319.png nemolargescalemon_25320_30335.png nemolargescalemon_30336_30431.png nemolargescalemon_30432_31511.png nemolargescalemon_31512_31967.png nemolargescalemon_31968_32075.png nemolargescalemon_32076_34571.png nemolargescalemon_34572_34715.png nemolargescalemon_34716_35603.png nemolargescalemon_35604_36575.png nemolargescalemon_36576_37775.png nemolargescalemon_37776_37991.png nemolargescalemon_37992_38915.png nemolargescalemon_38916_39539.png nemolargescalemon_39540_44123.png nemolargescalemon_44124_45851.png nemolargescalemon_45852_46295.png nemolargescalemon_46296_46535.png nemolargescalemon_46536_47243.png nemolargescalemon_47244_47603.png nemolargescalemon_47604_47879.png nemolargescalemon_47880_48443.png nemolargescalemon_48444_48503.png nemolargescalemon_48504_48575.png nemolargescalemon_48576_49175.png nemolargescalemon_49176_49691.png nemolargescalemon_49692_50579.png nemolargescalemon_50580_50747.png nemolargescalemon_50748_51443.png nemolargescalemon_51444_52343.png nemolargescalemon_52344_52859.png nemolargescalemon_52860_54215.png nemolargescalemon_54216_54935.png nemolargescalemon_54936_55816.png
Welcome everyone to our talk, the three rules to rule them all, the magic key to large scale network monitoring. We will start with a short introduction about ourselves and about the tool which we'll be using and then my colleague Alex will go into the three rules to rule them all, which will be a practical session where you can see a lot of things and learn a lot of things and then in the end be available for a Q&A session. So who are we? I'm Martin, I'm a project manager at Tribe 29, Tribe 29 is the company or the team developing CheckMK and Alex with us here is a CheckMK consultant and one of our assist admins and he started with Nagios in 1999 and fell in love with CheckMK in 2011, so almost 10 years now. What is CheckMK? CheckMK is an IT monitoring tool, you can use it for infrastructure and application monitoring and basically you can monitor almost anything, starts with servers, applications, databases, but also containers, container management platforms, cloud services and as with every monitoring tool you digest the data, then the monitoring tool basically suggests okay, this is good, this is not good and then basically the alerting starts afterwards with notifications in a lot of different ways via mail, via SMS, via incident management tools like PagetDuty, then you can also visualize the data, if you don't want to visualize it within CheckMK, you can also visualize it with Grafana for example and connect CheckMK with CMDVs like I do it. What makes CheckMK special is that we have more than 2000 plugins for what we call plug-and-play monitoring, so basically you can just monitor anything and then you do it in a couple of clicks, you don't need to configure a lot of things, it just goes quite smoothly. You can get CheckMK obviously in our open source edition, the CheckMK raw edition, obviously free of charge, it has all these plugins, you can get support by a fantastic community and we follow the open core model, so we do have an enterprise edition with a couple of more features with a better performance and that's where you get support from us. That's already everything about CheckMK, about us and I think enough of an introduction, I will hand over to Alex now who will give the actual talk. Yeah, thank you very much Martin, also one welcome from my side to our talk. So as Martin said, we will talk about the three rules to rule them all, which basically means how to set up CheckMK to do a very efficient network monitoring, so the goal here is we wanted the full visibility of all our network interfaces. It doesn't matter if it is an access port or an uplink port to another switch or a server interface of your ESX server or whatever and of course you want to do it as efficient as possible because every port really means every port, so we are quickly talking about thousands, tens of thousands or even more interfaces that we want to monitor. So we wanted to be able to do that also in large scale networks. So why do we want to do that? Every port can have errors and if it has, yeah, that definitely needs to be addressed and this is also important for end user ports, access ports because if there are errors, also their working experience with clients of applications will for sure be affected, so everything should be almost error free. The challenges end user ports are allowed to change state and speed, so if one goes home, takes a laptop with them, for sure the port will go down, dependent on the firmware implementation of the switch, it may also change speed or not. Some switches keep the last speed in the SNMP output, some others go down to the bare minimum like 10 megabit half duplex and they are allowed to do that. Very important ports, as we call them here for now, like uplinks are definitely not allowed to do that. So if you are uplink to your other switch goes down, most probably there is an unwanted issue, same happens when a server connection goes down from gigabit to 100 megabit, also you want to detect that. This means that we need to address these ports differently because they behave differently or are allowed to behave differently. So one solution for this can be we will use port names to determine between access and non-access ports. So to keep it simple, we could simply name just our important ports. So for example, if you have an uplink to switch number three, call it uplink switch three within your switch and check in case able to detect that alias. The next challenge is some windows, even if for example on Cisco the command is called description, they are not writing it into the SNMP description table but into the alias table. So we need a way to switch the tables that we are using for the port names. We need to be able to switch between those tables. So the solution is we will create two rules to switch between them and also we will create a so-called host tech which is the host property where we can say use this or use the other one. So it's demo time, so let me quickly switch to our demo. Here I'm going to install a check in case the raw edition. This is the upcoming 2.0 version here in beta stadium. And after installing, when it's done, I will create a site. So here I'm going to create a site called foster and I will set the admin password directly to foster so that we need to know how to log in. After creating, we can already start it. And we're done. So one quick introduction to a site. What is a site? It's a self-contained monitoring environment that can be independent from other sites. As example, you could create one site called production, another site called test and you can run tests in your test site without affecting the production. So it's a completely independent monitoring environment on users, on plugins, on setups, on rules, everything separate. Or it can be so-called distributed monitoring. So if you have really large scale setup, you could scale out and share the monitoring load over various instances. There is virtually no limit on how many instances you can run check and K, but managed from one central site. Another reason for distributed monitoring can be geographical reasons. So if your main headquarters is in Brussels and another location is in China, it doesn't make sense to monitor the location Shanghai from Brussels because of the latency. So it makes sense to create a site in China, a site in Russia, or wherever you need and manage them from the central site. So this is also possible. Now it's time to switch to our browser and to log in into our newly created site. So here it is and I will look in with the built-in administrator, which is check and K admin, CMK admin, and the password that I created, which is for STEM in this case. So what we see here now is not much. We see an empty site. We have zero services, but we are going to change that. Because of the screen resolution, I will for now get rid of this sidebar and turn it off for now. And now we are going to create our first host. So for that, we are going to our setup module and we're going to host. And we see here that we have the possibility to create a host or a folder. I want to create a folder here for now. And I will call it which and on a switch, we will have SNMP. So I'm turning on SNMP V2 here in this case with the credential public. Actually you don't need to do this because if you don't give anything here, we will use the community public by default. But let's do it like this. On the other hand, we don't have the data source normal check and K agent. Check and K agent is a small lightweight agent, which is intended for the normal operating systems like Linux, Windows, FreeBSD, whatever. Obviously we can't install it on a switch. So I'm turning it off here and I'm saying no on this. We don't have an agent. So all these settings here will be inherited starting from this folder downwards to subfolders or hosts within this folder. So we will save that. And when we create here now a host. All these settings are inherited here. So SNMP V2 with the community public. And after this we can already go and say, save and go to service configuration. So this will basically do an SNMP scan of this device and it will already detect some services. But it will do it in a way that we don't want that way, we will optimize it a little bit. Why do we want to do that? Because here we see interface but enlisted by interface index numbers. And we also see that something is missing here. So we have number one, two, three and four is missing. Five is here. Plus we see here the descriptions that have been set within the switch for the support. And yeah, set in the introduction, we want this as the service name. So I'm not going to save this here yet. I will do first the switch, the host attack to be able to switch between the alias and or the description table. So we will add a tech group here. We will call it internally if alias description maybe. Maybe without typos. We will add two tech choices which will result later in the host configuration property in another drop down as you've seen for the check and K or the SNMP settings. And the tech ID which is on top will be the default. So we should set here the most common setting and the most common setting is that manufacturers nowadays use the alias table. So let's give it the internal tech ID if alias. And for the second option, we want the description table. So use description. I'm saying safe. Now this tech exists and we have here a preview and this is the drop down that we created now. This tech by itself just exists. It does nothing. So we will need to connect it with the rule so that the switching the tech influences which rule is used. For that we're going to our setup module again. And now, yeah, because we have around 2000 plugins already, we have a lot of rules. So the best thing to do is to search here. So we will search for interface. And here we find the rule network interface and switch port discovery. This influences how to discover these interfaces. So we will create a rule. And this rule, the first setting that we will do is the appearance of the network interface. We will change that we don't want this index number. We want to use alias in case that the host tech that we created. So let's search here for interface and access as a condition to this rule. So if this tech is set to use alias, we will use alias. We basically don't need to set anything else here by this implicitly, we will turn on other things. We will turn on a match all interfaces, so we will also find the missing interfaces that we didn't find so far yet, because they were down. This could be specified much more in much more detail. But as we want to go as deep and as wide as possible with the monitoring and we want all interfaces, we basically leave it to all interfaces, because all interfaces matter. If they're up and causing errors, that's a problem. If they're up and causing a no error, also not problem. If they're down, they're not causing errors. But if they go up one day and start to cause errors, you will detect that. If you're not searching for every interface, you will miss that error and you don't want that. So all these settings are applied to conditions now. Here I will put the condition to the main directory, because what I'm setting up here will scale for every network interface. It doesn't matter if it's a network interface in a Windows server, Linux server, a switch, a firewall, a router, whatever. This will scale over your whole monitoring. So we're going to save that. And so far we're just using alias, but we need to create a second rule for the description. So we will simply clone that one. We will switch that to description. And we'll toggle here to use description, and then we have our second rule. We can go back to our host. And we can do another service discovery. And you will see a big difference here now. So all the interface numbers are gone, but we're on the ports where we configured names, like example on these access points or this firewall, we see the names. And this is what we can utilize later on to determine, are we talking about access port, which is a lot to go down to change speed, or are we talking about a port to my firewall, for example, the firewall, it shouldn't change speed, obviously. So now we can say, monitor undecided services. Then we can go and apply our changes and activate them. Now I will turn on the sidebar again, because now you will see we have one host, 35 services. Let's go to the services. At the moment they are still pending. While they are still pending, I will create the next rule. I will steal this service description because this is an unimported. And then my naming concept, un-name port is an access port. And I want to create now the rule to ignore state and speed changes. So I'm going to our setup module again. I will search again for interface. But this time not for network interface and switch port discovery, because we did this already. Now we want to influence how to deal with the result with the discovered services. So we will go to the common network interface and switch port rule. Also here we will create a rule again in the main directory. We will change the condition that it only applies to specific ports. So it will match all ports that start with gigabit ethernet, for example. In other cases, you also need for un-configured ports or un-named ports, you will need to add 10 gigabit ethernet, 40 gigabit ethernet on other switches like this, for example, for Cisco and Huawei switches. For HP switches, you will need to address it by number. So if they don't have a name, they will simply call it interface 001, 002, and so on. So let's create a regular expression here. When this interface description starts with several digits, then we want to match this rule. What do we want to do in this rule? We want to ignore the speed, because this is our access port. We want to ignore the state. And that's basically it. Here we can give a little comment so that also our colleagues know why we are setting up this rule. So it's not important to explain here what you're doing, because that's obviously visible in the rule. Here, it should be documented why you're doing it. You could give a description here like according to our naming scheme, unnamed interfaces like interface 001, or gigabit ethernet, or 40 and so on, our access ports are allowed to change state and speed, so basically everything why we are talking about this. That's being done. We can apply this. And nothing much will change. The change will only be visible during monitoring. If this will change state and speed, it will go red. If this one will change state and speed, it will stay green. So even if it's now, for example, up, if it will change speed to down, it definitely will go red and will give you a, it will not, sorry, the access port. This one, of course, will stay green. But all the name ports will get red if they change state or speed. Yeah, that's being said. We basically set everything up that we need now. So what we can do now, we can go again to our setup module. We can go to host. We will go again to switch and we will add another host. Again, because our rule and full structure, because we are not template based, all rules and settings are inherited from top, so we don't need to change anything here. I'm just giving it the DNS name. I'm saying save and go to service configuration. This is actually a quite big switch. So, and it's also in a remote location. So I'm expecting a longer runtime here for not making you bored waiting for this. We will do something else while it's running in the background. We background, we will go to host and we will create a new folder here. Oops. Sorry, this click was too fast. Add subfolder and we will create a folder called server here. On a server, we can install the check-in-k agent. We don't need to use SNMP and most probably shouldn't because you will get much more information and much more performance, with much more performance from the check-in-k agent. So we don't need to change anything here. We save the setting, this folder. We go here and we create a host. So let's add my approximate virtualization server here. Save and go to services. And here we go. Here are the services. And also when you scroll down here, you also see here that you get here the Linux interface names. So this is also an effect of applying these three rules that we created now also to everything because we put it into the main directory just being bound to this host tech. That's exactly what we want. And also here we can say monitor undecided services. Also I will show you again the properties. Here in the custom attributes, you will find this use allias because I made it the default in case that you get an unwanted result because here you put in an older HP switch or some firewalls that require that that you use the description table. Simply switch it here to use description and you will switch the discovery process to use the other as an MP table. That's being said. We can activate our changes again. Please look again at the sidebar. Three host 94 services. What still is missing here is a big switch. So we will go here again and go to the switch folder and go on our core switch. Now it will load what it has discovered before. So you see all the interfaces are there. There are many unnamed interfaces as you see here. But here also named interfaces. For example, we see VLAN interfaces here. We see named firewall ports now where the firewall is connected to and these things. And here we still see that it's on module A port 8. So here we can say monitor undecided. You can apply our changes. Again, I ask you to take a look at the service numbers. So still three host, but suddenly more than 200 services more because we detected them very efficiently. This is basically everything that you need to set up to have a common approach for all network interfaces in your network. It doesn't matter if you're talking about 50 host or 5000 or 50,000 host, you can apply that. So another thing that I quickly want to show you because this track that we're talking here is the talk to network monitoring and discovery or inventory. So I want to show you an inventory function. We're going to our setup module and we are searching for inventory. So we have a hardware and software inventory module and we will say do hardware and software inventory. Here are already some predefined rules, but they are just for checking case servers and Docker objects, but we want a common rule for everything. And we simply add here that we want to do also start a starter inventory. So that's enough. We don't need to apply any other conditions then do it in the main directory, do it on the global level, do it for everything. So we will save it. We will activate our changes. And we can go to host again here. For example, let's go to the smaller switch. I will do a rescue here to be a little bit quicker. And we see that we have a lot of inventory here already. Now somebody could ask why are you doing hardware and software inventory on a switch? Is the software changing? No, of course not. Hardware changing. Yeah, no module switch. It can happen. You can swap in modules or do something else. So simply let's go to show hardware and software inventory for the source. And here we can look at the hardware, for example, what kind of device do we have? Which physical components does it have? Does it have modules? Yeah, not really because that's a small switch, but that's actually not what I want to show you. I wanted to show you this because when I switch the sidebar, when I open the interface index, we can see here all the interfaces that we have. And we can see that here are ports which are right now down, but they are in the state used because a couple of days ago, this port was up. So we have a configurable threshold that we can set here before we declare a port as free. So maybe you know the industry standard of patching a cable. You want to patch a cable, you go down to your network rack, you open the network rack and you get shocked because of all the cable mess that you will find there. Also, you will find out that the whole switch is occupied already by patch cables and you think, oh man, this can't be. So it seems that colleagues didn't clean up when decommissioning a PC. So there the industry standard is, you will look for an port which is down. You will pray, you will plug out the cable, you will plug in your cable and hope that nobody will complain. And of course, what happens the next day is the manager is coming saying, I can't reach Google. So this can't happen with this status data that we're already getting from the switch. So that's a really nice feature about this hardware inventory on switches. Let's come back to the switch again, of course, I want to show you something. So when we go to the switch, you also see that we already get some metrics here and we can look deeper into the metrics. Of course, we're monitoring since the switch now since 10 minutes, so we don't have a lot of data here. But I wanted to give you a quick overview of what we're doing. So we are calculating the bandwidth in and out. This is a minimum maximum average. We see that we are recording this or that we have predefined views here for the last four hours, last 25 hours, so one day plus one hour or one week plus one day, a little bit more than a month, 400 days, a little bit more than a year. By default, we are recording here 720 days, which can be adjusted. So also here, when you switch here, of course, we don't have values here. One could think, what are we storing here? Because here we are sending out the data because we are not saving the data every minute, but we are taking the most significant bits, let's say, that we are storing. But for what? For minimum maximum average. Actually, we're doing for all. So you can switch here between minimum maximum average. Of course, again, because we're doing it since five minutes, you don't see much here. For a small feature, you can set a so-called needle here, which will mark exactly where this needle is. So you see the date and the exact values at that time. And this needle is saved. So even if you compare it with the filling rate of your Drive C on your Windows PC, you will find this needle there as well so that you can compare values between services and even hosts. We are not recording only the bandwidth, but also the error rate. As already said before, this error rate should be always almost zero. We have very low thresholds for that by default. 0.01% for warning and 0.1% of the total traffic for critical. My advice is never, ever change it except for wireless networks. Wireless networks are by definition shared networks. So they are shared with the neighboring access point, with your own access point, but also with atmospheric influences and lightning. Their error rates are pretty normal, but on everything that is connected to a cable, there shouldn't be errors. So don't change it. Also, we see broadcast and multicast rates for input and output. So you could detect broadcast storms and things like that here. We see this also for non-unicast, so multicast and broadcast together, versus unicast packages. And we can also monitor the length of the output queue of the switch if the switch is securing it to many packages instead of transferring them over the cable. So now we are coming to distributed monitoring. I want to show you a connection or how it looks when I'm doing a connection to one of our test sites. So again, I said mainly two reasons why to do distributed monitoring, scale out, or geographical reasons as I said in the introduction. And you can configure your connection to push the configuration so that you manage your remote sites from the central site, or you can create some breeder only. In this case, I will create some breed only. Let's call it FOSSTEM. Give it an alias FOSSTEM. We will connect to a host. In this case, it's the test site. We don't use the TLS encryption there, so I will do it in this case unencrypted, but of course in production you should do it encrypted. And I will choose here no replication with the site. It will be read only. So actually let me correct that one. And I need to give two more settings here. You need an URL prefix, how to reach that. Normally also, of course, you should do it via HTTP S, TLS encrypted. But as we are here in a local network, I can do it without. Okay, of course, this was a bad idea because our main site is already called FOSSTEM. So let's do it like this. That's being said, we can apply the changes. And I'm turning on this sidebar again so that you see what's happening here. And suddenly you see we have one on our line hosts and 4,055 services. So within one second you connect to an already existing site to get a central overview. And for example, if we look at here, we directly see some errors that are on the other side. For example, here are some services that need to be restarted because they have been patched recently or some updates have to be installed and things like that. Last but not least, because we are talking about network monitoring and today there will be two other talks by our colleagues from NTOB NG, Luca. We partnered up with NTOB to integrate NTOB into CheckMK to be able to see information that is gathered by NTOB in CheckMK to enrich CheckMK host information with information that is recorded by NTOB. For that I will switch to one of our test sites because the error is connected to NTOB. The test site is called Production. Let me switch back to the full screen. And here we could look for the NTOB information. I am struggling a little bit because of the video resolution here. Normally this doesn't look so big. The screen resolution here unfortunately has to be very small. According to FOSTA requirements. But here we will click on NTOB host and we see that there are hosts which have some NTOB information included. So let's go here to the host information and here let's get rid of the sidebar again. Again the screen resolution is actually too low. We see that we have some NTOB information. We have no engaged alerts for the source. There is no active alert flow. We see the traffic distribution sent to where those received and so on. We see the ports that NTOB detected in the network flows. We see the protocol breakdown TCP versus other protocols and with which peers communicating over which ports and protocols. So we can go to the traffic overview. So we see which traffic is creating the TCP Flack distribution. Port distribution which ports are unused with which peers it is communicating. So for example here it is communicating with one of our build servers. It is communicating with our firewall which application categories. So basically here a lot of check and K traffic is recorded. That's because this one is in a network which is a distance from our monitoring system and we are recording on the firewall and that's why we are just recording check and K for this particular host. We see the flows so maybe not so interesting. So maybe let's look quickly for another host. Here for example we have a firewall. So let's look here at the traffic and when we look at the flows here we should see different things. So we see some unknown protocols SNMP, other engaged alerts, no other past alerts. So this is the quick overview about our upcoming NTOB integration. Please take also look at the talks from NTOB from Luca and his colleague. That's it from my side so far. If you want to contact us you can do that via Twitter and of course you can write to us and also we have a stand at the first step. Of course we will join now the Q&A session. So thank you very much for your time and hope to talk to you soon.
network monitoring. We will start with a short which we'll be using and then my colleague Alex will go into the three rules to rule them all, which will be a practical session where you can see a lot of things and learn a lot of things and then in the end be available for a Q&A session. now. What is CheckMK? CheckMK is an IT monitoring can get CheckMK obviously in our open source and of course you want to do it as efficient as influences how to discover these interfaces. So with the monitoring and we want all interfaces, all interfaces matter. If they're up and causing detect that. If you're not searching for every give you a, it will not, sorry, the access port. and go to service configuration. This is actually more performance from the check-in-k agent. So we say monitor undecided. You can apply our changes. Again, I ask you to take a look at the service that you need to set up to have a common approach for all network interfaces in your network. It talk to network monitoring and discovery or are searching for inventory. So we have a we will save it. We will activate our changes. and software inventory on a switch? Is the coming to distributed monitoring. I want to show different things. So we see some unknown for your time and hope to talk to you soon.