Bringing monitoring into the 21st century
Hansen, Soren
We stumbled upon ancient (circa 1960) scrolls of wisdom in the field of statistics and applied it to modern day monitoring systems. The result is a monitoring system that detects anomalies instead of relying on statically defined thresholds as well as predicts failures long before they become problems. There are many, many things wrong with the typical, free monitoring solutions available today. - They rely on probing rather than look at actual data generated by real users of the monitored system/service. - Their definition of a failure is simply some value exceeding a statically defined threshold. - They don't tell you something is wrong until it's too late. In this talk, I'll present an alternative. * It prefers passive monitoring to gather data about the experience real users have with the system/service, * It detects any anomaly. Unexpected low values can be indicative of problems, too! * It uses various statistical methods to predict problems before they occur. It's easy to imagine a file system that fills up another 5% every month and it's easy to predict how full it'll be in 5 months. It's more complicated (but certainly doable!) to make an automatically computed, educated guess as to whether you'll be able to handle the annual spike in demand on your webshop this holiday season based on the load over the preceding months.
 My name is Sohn and I'm here today to complain to you about monitoring. First, this is the agenda for today. First I'm going to introduce the subject, introduce myself. Then I'm going to go through some of my favorite pet peeves that I have with the current state of free software monitoring. And then I'm going to talk a little bit about what I'm going to do about it and finally questions and answers. Okay, so for the last 10 years I've run various things in production like everything from mail servers to web servers and all sorts of different things. And being a lazy person I don't really want to spend time looking at these things, looking after them when I can have a computer look after my computers for me. So I've always been a fan of monitoring systems and having them do all my older, really tedious work for me. For me personally that means that's typically meant that I use Nagios and Munim. So Nagios for status monitoring and alerting and sending text messages and whatnot. And then Munim to create graphs and stuff so I can use that for, so I can look at that if I'm really bored. I realize that there are many, many other tools out there. I mean there are way more than these. There are tons of free software solutions for monitoring different things. But because I'm used to Nagios and Munim, so I'm probably going to be mentioning those quite a bit. But as far as I know all the problems that I'm going to explain today apply equally well to all of these. So when I say Nagios and Munim I'm not particularly trying to pick on them because everything else sucks as well. Okay, so the first thing I want to talk about, well it's the third thing by the way, the first problem that I want to talk about is active versus passive monitoring. There's a tendency in the current software selection that we have for doing monitoring to favor active monitoring over passive monitoring. So just to get us all on the same page, basically active monitoring is when you're probing the system that you're testing. So you're basically emulating a real client to verify that it's doing what you're expecting it to do. Now I'm going to be, as I said, I'm going to be mentioning Nagios and Munim a lot. I'm also going to be assuming that what we're monitoring is a web application. So there's a web from then, there's a database server. Maybe there's a, let's pretend it's a web shop, so there might be a payment gateway as well. Just so that we have an example that is reasonably simple to understand for most people. But again, the problems that I'm talking about, they apply equally well to any other thing that you might be monitoring. So when you're doing active monitoring of a system like that, what you typically do is you connect to it and you verify that you get a response in a time of fashion. Not a request obviously. So you pretend that you're a real client and your time, how long it takes to get a response. And based on that, you make a decision as to whether you think that the service is in a great shape or in a not so great shape or in great trouble. Now on the other hand, if you're doing passive monitoring, you're just watching things that are happening anyway. So you have requests coming in from real users. So instead of making your own emulated, sterile sort of requests, you're just looking at whatever your users are doing already and making your value judgment based on that. So you're deciding whether they are seeing appropriate response times or appropriate response codes and what have you. So that's the major difference between the two. And I'm going to try to convince you that passive monitoring beats the crowd of active monitoring any day. So the first immediately obvious reason why this is the case is that it puts your users or your customers experience with your application at the center of what you care about. Because you're looking at the request that they're making, you're looking at the experience that they have because you're timing that instead of making some half-assed sterile request to your application and timing that. So you're acknowledging that there's a difference between what they might be seeing quantitatively compared to what you're seeing because it is a different, I mean, it is a much more complex system that they're exercising. They have their own bandwidth limitations and they're doing slightly different things as well. But there's also a massive qualitative difference between these things because you're not actually making sure that things work. It's much like when you're doing, if you have unit tests for your software application, that doesn't guarantee that the whole really works. You're making sure that the individual components, they're doing what they're supposed to, but that doesn't mean that the whole is in perfect working order. And anyone who has done this in the real world will know that just because your monitoring system can connect to your web server and fetch some stupid static page very quickly and it can connect to your database server as well, that doesn't necessarily mean that your web application is in perfect working order. Because just because that is true, that doesn't mean that your web server can actually reach your database server. There might be a network problem, there might be all sorts of different problems. If it can actually connect, sorry, it may not be able to authenticate, maybe it's using different credentials, maybe it forgot its credentials, I mean, any number of things could go wrong. If it can actually authenticate, it might have typos in the queries that it's trying to run. If it has perfectly typed queries, it might be really slow because you're missing an index on a table in a database somewhere and compared to when you were developing it, you might have like a massively larger data set, so everything just slows down. But so the point is, this is extremely difficult to replicate with active monitoring. Even if you try to take care of all these things, maybe you're creating a, instead of fetching a static web page from your web server, from your monitoring system, you might have a much more complicated page that tries to exercise all these things. It might actually try to connect to the database, issue a set of queries that you know from your application that it might be running and so on and so on. So you might be able to get it close, but inevitably there will be something that you're stubbing out, simple as that. There's always going to be something you're stubbing out. Again, assuming that this is a, if this is a web shop, your active monitoring system might go and click on different things to make sure that you can actually, you know, see the items that you're selling. But maybe you don't put anything in your shopping cart. If you do put it in your shopping cart, you might not go through the entire checkout process. If you do, you probably don't send it off to your payment gateway because that's going to cost you money. And if you do that, you probably don't charge the, you either probably don't charge the full amount because then it's going to cost you a whole lot of money, or maybe you're buying some kind of fake item that doesn't exist, and then that might work, but all the other ones might not. If you do make the credit cards transaction and you do buy something real, you probably don't send a packaging slip to your shipping and handling people for them to pack it up and send it somewhere. And if you do, you probably don't have a machine built that will make sure that the postman eventually brings it to you and unpacks it and makes sure it's the right stuff in there. So inevitably, there's going to be some part of the process that you're trying to emulate that you're going to be stopping out or changing. So no matter how hard you try this, you're never going to be exercising the same things as your real users are going to be exercising when they're actually using your request. Now, to add insult to injury, not only are your tests not as comprehensive as it would be if you looked at the real data, if this is just a moderately busy website, consider that last time I checked, the default time between a check in Nagios was three minutes. In those three minutes, again, if this is a moderately busy website, you will have had dozens, maybe even hundreds of thousands of requests. So you have tons of data that you're just throwing away. And this is good data. It's better than what you would have been getting with your active monitoring, but you're just throwing away. And that is a damn shame. So the next thing I have is something I like to call temporal tunnel vision. So the typical anatomy of a Nagios plug-in is so that it tries to exercise the system that you're testing. So it tries to connect to it, and then it records how long time it took, and eventually it times out. It doesn't wait forever, but it tries to connect, and then it records how long time it took. And then if it's within some threshold, like, say, two seconds, then it's in the green area, and everything is fine. If it's between two and four seconds, then it's not quite so good, so it reports back that it's in a warning state. And if it's above four seconds, then it just exposes, and says, oh, no, it starts sending text messages, and what have you. But as soon as this information leaves the plug-in, the actual data point, which in this case is maybe 1.9 seconds or whatever, thereabouts, as soon as it leaves the plug-in, that is just really debug information. Nagios doesn't record it. It only looks at what is going on at this very instant. It only knows, you know, what is the state right now. This data point doesn't feed into the data model of Nagios at all. It doesn't care. It's just a, it's only for presentation when you're looking at it right now. It doesn't remember it, it doesn't record it, it doesn't do anything with this number. And the reason that this is so incredibly tragic is that if we didn't look through, if we didn't have this tunnel vision of only looking at the slice of history, that is the current, we might be able to see a trend that would have told us where this was going. Now, if this is the history for the last two months, but all we're seeing is the current, we think that everything is fine. But if you look at the history, it doesn't take a PhD in statistics to realize, you know, halfway through that this is going to be a problem now, about a month from now. But we don't even have the data. So even if we managed to apply some of this logic, I mean, we don't even have the data available to us, because the tools just throw it away. Now, Nagios throws it away, but Muenin doesn't. So a lot of these systems, either they build into one or maybe have separate systems like Nagios and Muenin. So you might be generating graphs, so you might be recording this, you might actually have this graph available to you. But if you have just a moderate amount of systems and metrics that you care about, you might have like thousands of graphs like these, and you're not going to sit around looking at them and to see that, oh yeah, that's going to be a problem in about a month. Because, I mean, who does that? Very boring people, right? I mean, no one. No one does that. It's just not the way it works, because everyone is busy dealing with things when they blow up a minute from now. So they don't have time to look at all of this. And the reason this is so incredibly tragic is that it's, we have had algorithms for like since the 60s that could look at this data and very accurately forecast when this is going to be a problem. And this is not complicated stuff. I mean, it's stuff that you can do on the back of an envelope. I could do it with pen and paper. It is so simple. It is constant space complexity. It is dead simple stuff. And we've known about this since the 60s, and we're not doing it. It is absolutely tragic. I mean, when we were here, the system should have added a reminder to our calendar saying, well, you know, on March 1st, this is going to be, this will have blown up. So you should probably, you know, look at it now instead of waiting until it actually blows up. And I think this is, I think this is absolutely tragic, because there's really no excuse because it's so simple. And the same, not the same algorithms, but there are similar algorithms that are equally old that can do the same thing for things that look much, much better. Things that look much more complicated than this. I mean, if it's going up and down, if it's seasonal, if it's, I mean, it can, you can very accurately say something about what's going to happen next with this, with these things, but we don't. And I think that's the shame. Okay. So the reason that we monitor things is that we like to detect when things are not normal. This is perfectly sound because this is what we care about. We want to know when something is normal so we can, you know, investigate or handle it in some way. The problem is that the way that typical monitoring systems design whether something is normal has absolutely nothing to do with normality. Nothing. So a typical example is, is, this might be the number of requests that our database server is seeing per second. This, let's just pretend that this is 3000 requests a second. And we know that, that is, that is a fine value. It's not going to be a problem until it reaches about 4000. So the way that we, we tell and tell our monitoring system about this, the way that we codify this is that we define a static upper limit. It's, it's the exact same thing we saw here. I mean, as soon as it does an upper limit for the time this may take, and as soon as you're above that, then it's a problem. So we always focus on these upper limits of various things of, you know, how this, how full the discs are, how long times things take, how many requests per second you're seeing, how much bandwidth you're consuming, and so on and so on. We always focus on these upper limits. And, and that's fine when it's going like this, because we know that if it goes up to 4000, that's going to turn into an operational problem, because our database server is going to be under too much load and we're going to have queues that are going to explode. We're going to have all sorts of problems. But the thing is that the real world is just more complicated than that. Now what if this happens? Now this looks like something I would want to know about. This is definitely not normal. I mean, this is a problem. Probably. The thing is I don't know. But my monitoring system is never going to tell me that, oh, you had something that was too low, because we're always focusing on setting a static upper limit that is, don't ever go about, about this. Now you could suggest that, okay, so what do we do? We set like a maximum of 4000 and a minimum of what, 2000? Because, I mean, that would have given us an alert. And I guess that's a start. But then you might have things that look like this. These are not real graphs for monitoring systems. I made this up, by the way. Maybe you can tell. So this might be over the course of a day. So we have something that goes up and it goes down and then goes up again. It's very, I mean, maybe it depends on when people are awake or something, right? So in order to do this, you would have to, in order to specify an upper and lower normal bound of values for this thing, you would have, you know, the upper bound going right up here somewhere and the lower bound right down here. Otherwise you'd be getting alerts every single night or every single, you know, day at high noon or whatever. But what if this happens? So if there's a dip here, this is well above the minimum that we defined. And this one is way, way below, I mean, the maximum we defined. But this is definitely not normal. And this is something I would want to know about. And with the current systems available to me, I mean, they're not going to tell me about this. The only way I would have known about this is if, you know, if there was like a, some other symptom that I could detect or if customers called and said that there was a, something was wrong. Or if I, you know, spent, you know, all my time looking at my thousands of graphs to notice, you know, these things that look, you know, kind of strange. And even if you did do that, I mean, this is very, this is a very pretty graph. I mean, this is a very uniform graph. It's very easy to see when something is, is, is not normal. But in many cases, these, these go up and down a lot. It is, it can be very complicated or difficult to spot even with the human eye to when there's something that doesn't look normal. Now, this is a, this is a CPU graph from a backup server. So if we said that this is midnight, so it's rather day, it doesn't do anything at all. It just sits around idling. At midnight, all the other servers start throwing data at it. This takes a bit of CPU time in, you know, doing SSH, decryption, you know, whatever. So this takes a bit of CPU time. And eventually they finish. And then from 2am to, I don't know, to 3.30am. It starts compressing the data. And this is a massively CPU intensive operation. And there's a lot of data. So this takes a long time. And during this time, it is, the CPUs are pegged at 100%. So if you want to monitor that, if you want to, I mean, it's very common to monitor your CPU load of servers. So you say that, well, on average, on a sliding, on a five minute sliding window, you should never have more than 80% utilization because that's going to, assuming that the assumption is that that's, that's a problem because if that's, if that goes on, then you're going to be in trouble. But on this server, I mean, this is completely expected, right? We expect it for several hours a day to be at the max. So we can't just monitor on, on, on the maximum value that over in the five minute sliding window because we're going to be getting text messages every single night. And I don't know about you, but that doesn't sound very, very pleasant. So what we can do instead is that we could just say, well, between two and four AM, we just completed disabled monitoring on the server. But then what if one day this spike just doesn't show up? If the job fails to run, you'd never know because you just disabled monitor, you just disabled, you just disabled notifications altogether. So you just, you have no idea. Now, what can we do about this instead? Now, if we go back to this one, as I said, there are algorithms available to us that can tell us based on the history of the service, what to expect from it in the future. What this basically means is that before we even collect a metric, we have a pretty good idea about it, what it's supposed to be. So before we even, before we receive another metric from the passive monitor, we know what it's supposed to be. And that means that when we get it, and it's very different from what we were expecting, we can know that this is not normal. We have detected an anomaly. And this works perfectly for all these things. So if you have something that's always pretty much level and suddenly it drops, this is not normal. This is not what it used to do. And then you can raise an alert. In this case, you can, you know, from history because you've looked at this service for like days, weeks, months, months, years, you know, whatever. You know that it always goes up and down, except the one day where it doesn't. It does something wildly different. And you will know because when you come to this point, the monitoring system has an expectation that it keeps going up. Because that's what it did yesterday and the day before and last week and last year. But it didn't. And then it can send you an alert. And we have the exact same thing here. Not only will it detect if it, if it's completely absent, or if it takes too long, or if there's another extra spike at some point, at some time of the day. So, yeah, one would might be tempted to configure. So the way you would do this is that you would configure the monitoring system to have like different expectations for different times of the day. So that could absolutely explode the configuration file if you had to specify, you know, for each particular metric, this is what to expect for this time of day, or this is what to expect for this time of day, and this is what to expect for this time of day. So it would absolutely explode. But what you would be doing, what you would be using to feed into this would be, you know, the graphs that you've already generated. So you'd be taking data from your monitoring system and feed it back into your monitoring system's configuration file. That is the working sort of an insane person. So of course what you do instead is that you use, I mean, the forecasts that your system is able to give you already. And then trigger when that is abnormal. Okay. So what am I going to do about it? Not a whole lot, because I'm kind of busy doing a lot of other things. But what I've been working on for a while now is a project called Surveyor. Surveyor has an extremely simple data model. It has users. Each user can have a number of services configured. The server service has metrics that gets fed to it. And a service can also have any number of plugins configured. Now the way it's supposed to work is that these plugins are configured as URLs. So the way it works is that you have some agents that might be doing active monitoring if you must, but typically, hopefully, they'll be doing passive monitoring. So they collect some data from somewhere and they send it to Surveyor and they get a response back saying, all right, I'll take care of it. What Surveyor does is it adds some extra information to the data points that it got from you. It adds a timestamp, a service ID, and various things. And then it adds some saved state. And it sends it to all the different plugins that you've configured. So this is a completely JSON-based interface. You don't have to have all these plugins running on your server. They can run on someone on your own server or someone else's server or whatever. It doesn't really matter. So it basically sends this JSON object to a plugin through this URL, or at this URL. And then it gets back a response. This response can include some saved state. So many of these algorithms that I mentioned that can do this stuff, that can do this prediction stuff, they work in constant space. So they don't really need to store anything on the server that's running the plugin. They just need like a tiny, I mean, five values. It's all they need to store to remember where we're going and where we're headed. So what they can do is... I need to point over here, don't I? Right. So when they've looked at this data, they've analyzed it, they've come up with a... they've decided whether it's a good or it's a bad... And then they include some state that they send back to Surveyor. Now the next time that Surveyor receives a data point from this particular service, it includes it in the request that it sends to the plugin. So this means that you can have these algorithms, these forecasting algorithms. I mean, they shouldn't take more than... I can't imagine that it'd be more than 10 lines of PHP or Python or whatever. It's really simple stuff. And you don't even have to worry about saving the state because Surveyor will do that for you. Now when it responds, it can say, yep, that looked perfectly fine. That was exactly what I expected. Everything's good. Or it can say that an anomaly was detected. It can include different kinds of information and whatever. And if it sends back... if it tells Surveyor back that this was anomalous, something that this is not what I expected, then Surveyor can send you a text message and you can act upon that and you can do various things. So you can have plugins. I mean, this is all there is to Surveyor right now. It doesn't do anything other than this. But it's very simple to add a plugin that feeds this data into our D-Tool to create graphs. It's very simple to make one of these forecasting algorithms. It's very simple to really add more simple plugins to this. So the idea is that this will... I mean, you can run this on your own servers and whatever. Or you can use an instance of it that I'm going to set up when I get around to it. And then anyone who wants to write plugins for it, they can just publish the URLs for it and you can configure those for your services. And the world will be a better place, pretty much. You can also have a plugin that instead of worrying about alerting on anomalies, it can maybe give you a list of top 10 next... I mean, the 10 next problems that you're most likely to run into in your data center because it knows... I mean, it can detect things that are heading towards failure. And then it can tell you that, well, the next thing that's going to fail is probably this disk over here that's going to run full. It might only be at 30% now, but it's rising at a very critical pace. So this is going to be a problem next week, but it's only at 30% now, so your typical monitoring system wouldn't have known. But we can know that now. Right. So, Sovela lives in a number of different places. The primary place is on Launchpad. This is confusing to look at. So it lives on Launchpad. That's where you can report bugs and whatever. There's a mirror on GitHub. There's a homepage for it. There's all sorts of different things. And it would be great if people would help me write it, make it awesome, or just write plugins for it, or do any... Do just help me work on it because we're going to make the world a better place. This was way faster than I expected it to. Questions? There are microphones there, there, and up there. How we detect the difference between the evolution of normality and abnormality. For instance, I'm having a web shop in Belgium and the customers are shopping mostly between 8am and 10pm. Then I'm opening a shop in Australia or something. And all of a sudden there are a lot more data requests coming in and other times. How would you make the difference between the evolution or the change in normality and abnormality? The question is how would I tell the difference between a change in normality, so to speak? If you're used to having your user base mostly in one part of the world and with the corresponding peak in load and dips in our values in load, then if you suddenly have a user group in a very different part of the world, that graph is going to look very different. The data is going to look very different. Yes, the first time that happens, as soon as that happens, the first time you would... What are you? Oh, there you are. The first time it happens you're going to get an alert. When you acknowledge that this is going on, this is what... If I have received the alert, then this doesn't work right now. But the way it could work is that you could tell the system that normality has changed. You can either change it into just a completely... Don't send any alerts right now. Just look at what's happening now and learn from that. Or maybe you can favor... Depending on how stable the trends and the patterns are, you can tell it to which extent you want to favor very fresh data or very old data. And if you know that things are changing very quickly right now, you could tell it to give more preference to current data because it's... to kind of catch up with the change in normality. But it's a good question. Thanks. Hi. I missed the first part of the talk for which I apologize. I was at the key signing. But at any rate, if I missed this, apologies. Have you given consideration to doing this stuff at very, very large scale? Tens of thousands, hundreds of thousands of nodes, and the implications of using TCP and JSON instead of something that's field proven like SNMP? Not a whole lot. No. I haven't really... I intentionally didn't try to optimize those sorts of things yet because I don't really know where it's going to fall apart. I have chosen a data store for it that I believe will scale well. I'm using React for it right now, which is a distributed hash table to kind of do its distribution, which I trust to be able to handle even very, very, very big volumes of data. But in terms of the configuration communication between the different components, no, I haven't really... I haven't thought about changing that to any degree. Thank you. The message passing diagram that you showed suggests that you either need a fair amount of processing power for working out... Sorry, I can't hear it all. The message passing algorithm diagram that you showed suggests you either need a fair amount of processing power for running some algorithms or a reasonable sized database for historical data. Does that rely or reside on the system on the right-hand side, the remote central server, or on the agents on the left-hand side? Where do you see that information storage and processing being located? The way I'm imagining it right now is that a surveillance, I mean, the central component will keep a record of everything. And then if one of these plugins need historical data to do something, it can query for it. And then depending on when you configure it, you should be able to tell it that... Basically, the plugin I have that needs the most historical data only needs, like, last year. And then you can start discarding data that is older than one year. And that keeps it sort of under control. Hopefully, at least. So from the earlier question that you had from up here, I would respond 99.9% of us aren't Google and Facebook. We don't need to worry about those for most of us, and they've probably built their own systems for your answer to JSON being too heavy. But my question was actually things like monthly events and yearly events. It's easy even for one week over a little wild collect data, but being woken up at midnight every month or every year is still going to happen with a monitoring system like that, because you basically need forever's history to see those things. How would you cope with that? I'm sorry, I couldn't quite hear you. I didn't catch you completely. Sorry. Okay, you have a nice little graph for each week or each day, the patterns that happen over there. There's fantastic stuff for that recently, a site that was doing this really well. But the problem is monthly events, yearly events, the big roll-ups that happen at the end of each month or each year are going to be a special case and are still going to wake you if you're setting alerts for it's gone outside normal. There's always... Not necessarily. I mean, some of these algorithms you can apply, you can tell them not just that you can have multiple seasons. So you can tell that there is a strong daily season that you should look at, but there might also be a monthly and a yearly or something else that makes sense for your particular use case. Cool. And then it can... Even though it knows that there is one pattern to follow for the day, it can know that it has another every month, there's something that only happens once a month or once a year. So you can have multiple seasons that can feed into this algorithm. Great. Thank you. Are you involved in the monitoring sucks, hackathon? Sorry? Are you involved in the monitoring sucks, hackathon? No, I'm not. I have to go home, unfortunately. I didn't hear about it until after I put... So, no, I'm afraid not. Hi. I understand the idea of the Surveyor, but I also understand so in development, it's not finished. And it has a plugin architecture. So if I would use it today, what kind of anomaly detection does it have out of the box? Or do I need to first write my own anomaly detection plugins to get it working? It doesn't actually... I haven't actually written any plugins yet because I've been busy trying to get the stuff that you just saw get that finished. But as I said, some of the easier ones, the easier algorithms, it is literally ten lines of code. And I have that code written already. I just haven't turned it into a plugin for this. Yeah, because I understand because you said it's very easy, like ten lines of code, but probably not for everyone. Very easy to write such a thing. So I mean, and usually to get started, you need to have some examples. Absolutely. So that's definitely the idea that the system will ship with a number of different plugins that will do all the things that I've already mentioned. And this is not far off. Probably next week. It's simple stuff. I just haven't done the roundtrip yet. And it will be included and available and all this stuff. And I'll probably also be writing kind of skeleton plugins in PHP and in Python and whatever. So if someone wants to write plugins, it's really easy to just take that code and start adding simple stuff. So if I would say, check it out next week and download the example plugin, what kind of anomaly would that, would it be able to detect? Well, the first plugin that I'm going to add is one that will be very good at detecting, you know, the, this stuff. So stuff that doesn't actually have any seasonality to it, but only has a, like a trend that isn't affected by, you know, with the time of day or so on and so on. A linear trend? Not necessarily. It doesn't have to be linear. It just, it doesn't actually have to be linear. It will very quickly notice if it's going exponential or something. So that, that is no more than a week away. It is, it is very simple. The other stuff I need to, for the next one that actually can do seasonality, I'll, I have to let, I mean, add a way for it to extract data from Surveyor again, so that it can, it can work with that. But it's not far off. It's simple and it will be included in this. It's not going to be something you have to do yourself. Okay. Thank you very much. Thank you. Looking forward. Yeah. I would be very happy to write some of these anomaly detections for myself and I can, of course, go to contribute them. But you, you're mentioning, okay, these are very simple things and these exist from the sixties. Would you have any pointers like how to start, like, where, which commonly known algorithms? Sure. Mathematical models or whatever. The one I'm referring to from the sixties is, is one called Hold Winter's Double and Triple exponential smoothing. If you can apply that and get extremely good results for stuff like this. But I mean, there has been tons of research done since then and there are much more sophisticated algorithms, but this stuff is just really, really low hanging fruit. Yeah. But did you do some research on, because I did search as well for some things and I had a very hard time to find actually usable things. Okay. I'm not a mathematician, mathematicians. And so it's not so easy for me to find my way around. There's actually, there's a scientific journal dedicated to forecasting. If you, if you Google for, I think it's just called Journal of Forecasting. So every, I think it's every year. I mean, they publish a major journal. They have conventions every year about, you know, research into forecasting and stuff. And there's lots of really excellent and really quite readable articles in there that you can use to, to, to catch up on things. It is very valuable. There's a guy called Rob Hindman, which is, I think he's the editor in chief of that journal, but he's also one of the, the foremost researchers into the subject. And he has a webpage where he explains all sorts of things and he, he writes all his example code in GNU, R. I mean, the statistical tool from GNU. So it's very, very approachable if you have just a basic mathematical understanding of if you know about the tools already. So that's, so those are probably things to look at. Okay. Thank you. Thank you. So we're talking about monitoring. So we're talking about a large amount of data. How does Surveyor store this data? The data is stored in, in, in RIOC, which is one of the, one of the new fangled, no SQL data stores. It is, I mostly chose it because I, I, I very much liked the data model in it. It scales extremely well horizontally. And I don't really need a lot of fancy things from the data store. I just need a way to be able to store data and pull it out again really quickly. Anything else is just, really just fluff. And for those sorts of things, RIOC is absolutely perfect because it doesn't, that the operational model of it just like makes it so incredibly trustworthy. Because I don't want to get up at night to fix my monitoring service. Can you please repeat this name of no SQL? It's RIOC. R-I-A-K. RIOC. Thanks. So, working. So in a couple of cases, I might have, you know, thank you. Very distigrate. Many gigabytes of old logs that I could extract a whole load of running data from your how long query. He's talking about that kind of thing. Does your plugin model allow me to go back through those and historically pick, pick out when a particular prediction model would have predicted a, an issue. So I'm going to compare that to my maintenance logs and things so that I can tell would this have been a bad thing to have run? Would this have woke me up every midnight? You know, when would this have woke me up historically? I don't think, I'm sorry. I don't think I'm, something's beeping. What's going on? Oh, that's me. I'm leaning on something. Sorry. Hope it doesn't blow up. Sorry, I didn't completely catch the question. So it's about what I'm doing. So if I've got a lot of historical data, you know, logs I can process into the kind of, kind of events you're looking at, how long we request talk or something. Is, does your model for your new tool allow me to pump that and find out when it would have alerted me? Not right now. Conceptually though, is that, does that break your model or not? Well, that's a good question. I mean, the current model, only really, I mean, you're just feeding it raw values. Depending on the type of, depending on the source, you might want to change, you might want to do some pre-processing in the agent that you have that's collecting the data to kind of make it, turn it into something that is more of a, it's kind of hard to, hmm, I haven't really thought about how to put this into words. Sorry. So if we're using like terminology from RRD tool, it has like gauges, which is something like a temperature would be like a gauge. Yeah. And then it has, you know, bytes per time unit, which is a different thing because the way that you graph it is separate. And I would expect that kind of pre-processing to live in the agent. So, so they shouldn't really have to know about it. And hopefully the plugins wouldn't either. But maybe I haven't thought it through enough. Maybe that's not going to work in the real world. But that's what I'm thinking right now. Okay. Thanks. So I have all this data in Surveyor and is it intended to also do graphs of this? It will, I haven't really figured out if I'm, I try to do graphs directly in Surveyor. But because of React, because I'm storing every data point as a single key in React, generating a graph that spans a year, if you have metrics per minute, I mean, that is a lot of, you know, individual data points that have to be pulled out and it takes incredible amounts of time. And I would have to completely change the data model to make that work. And I haven't really figured out with myself if I decided not to do graphs in Surveyor because I'm, I keep insisting on wanting to use React. Or if I actually think that the best thing to do is to put it into a plugin. So there could be a plugin that just takes the data and feeds it into RLD tool and then has that generate data graphs because RLD tool is really good at it. But I don't know. I mean, using React is not set in stone. It's just what I tend to prefer right now. But it's making it very difficult to do graphs directly in Surveyor because of the time it takes to get individual data points out. Have you thought about the problem with React that it doesn't have data center or a rack awareness? It doesn't have what? Data center or rack awareness that, for example, if you have three data centers, one node in each data center, and then multiple nodes in each data center and one data center completely goes down. Right. So it may happen that you can't read or write your data anymore depending on how your formulator writes. I know that React, there's an enterprise version of React that is supposedly really good at that. So if you have multiple sites, it will actually, the distribution model is apparently sensible. But I'm not having really, I mean, I don't know of any other system that does that really well. So if I would have to attack that problem myself, I would have to basically build some kind of distribution system or reliability model on top of it. And that is just, I mean, that's too soon. Yeah. Okay. Thank you. Sure. All right. Cool. Thank you. Thank you.
 My name is Sohn and I'm here today to complain to you about monitoring.  First, this is the agenda for today. First I'm going to introduce the subject, introduce myself.  Then I'm going to go through some of my favorite pet peeves that I have with the current state of free software monitoring.  And then I'm going to talk a little bit about what I'm going to do about it and finally questions and answers.  Okay, so for the last 10 years I've run various things in production like everything from  mail servers to web servers and all sorts of different things.  And being a lazy person I don't really want to spend time looking at these things,  looking after them when I can have a computer look after my computers for me.  So I've always been a fan of monitoring systems and having them do all my older, really tedious work for me.  For me personally that means that's typically meant that I use Nagios and Munim.  So Nagios for status monitoring and alerting and sending text messages and whatnot.  And then Munim to create graphs and stuff so I can use that for, so I can look at that if I'm really bored.  I realize that there are many, many other tools out there.  I mean there are way more than these.  There are tons of free software solutions for monitoring different things.  But because I'm used to Nagios and Munim, so I'm probably going to be mentioning those quite a bit.  But as far as I know all the problems that I'm going to explain today apply equally well to all of these.  So when I say Nagios and Munim I'm not particularly trying to pick on them because everything else sucks as well.  Okay, so the first thing I want to talk about, well it's the third thing by the way,  the first problem that I want to talk about is active versus passive monitoring.  There's a tendency in the current software selection that we have for doing monitoring to favor active monitoring over passive monitoring.  So just to get us all on the same page, basically active monitoring is when you're probing the system that you're testing.  So you're basically emulating a real client to verify that it's doing what you're expecting it to do.  Now I'm going to be, as I said, I'm going to be mentioning Nagios and Munim a lot.  I'm also going to be assuming that what we're monitoring is a web application.  So there's a web from then, there's a database server.  Maybe there's a, let's pretend it's a web shop, so there might be a payment gateway as well.  Just so that we have an example that is reasonably simple to understand for most people.  But again, the problems that I'm talking about, they apply equally well to any other thing that you might be monitoring.  So when you're doing active monitoring of a system like that, what you typically do is you connect to it and you verify that you get a response in a time of fashion.  Not a request obviously.  So you pretend that you're a real client and your time, how long it takes to get a response.  And based on that, you make a decision as to whether you think that the service is in a great shape or in a not so great shape or in great trouble.  Now on the other hand, if you're doing passive monitoring, you're just watching things that are happening anyway.  So you have requests coming in from real users.  So instead of making your own emulated, sterile sort of requests, you're just looking at whatever your users are doing already and making your value judgment based on that.  So you're deciding whether they are seeing appropriate response times or appropriate response codes and what have you.  So that's the major difference between the two.  And I'm going to try to convince you that passive monitoring beats the crowd of active monitoring any day.  So the first immediately obvious reason why this is the case is that it puts your users or your customers experience with your application at the center of what you care about.  Because you're looking at the request that they're making, you're looking at the experience that they have because you're timing that instead of making some half-assed sterile request to your application and timing that.  So you're acknowledging that there's a difference between what they might be seeing quantitatively compared to what you're seeing because it is a different, I mean,  it is a much more complex system that they're exercising.  They have their own bandwidth limitations and they're doing slightly different things as well.  But there's also a massive qualitative difference between these things because you're not actually making sure that things work.  It's much like when you're doing, if you have unit tests for your software application, that doesn't guarantee that the whole really works.  You're making sure that the individual components, they're doing what they're supposed to, but that doesn't mean that the whole is in perfect working order.  And anyone who has done this in the real world will know that just because your monitoring system can connect to your web server and fetch some stupid static page very quickly and it can connect to your database server as well,  that doesn't necessarily mean that your web application is in perfect working order.  Because just because that is true, that doesn't mean that your web server can actually reach your database server.  There might be a network problem, there might be all sorts of different problems.  If it can actually connect, sorry, it may not be able to authenticate, maybe it's using different credentials, maybe it forgot its credentials, I mean, any number of things could go wrong.  If it can actually authenticate, it might have typos in the queries that it's trying to run.  If it has perfectly typed queries, it might be really slow because you're missing an index on a table in a database somewhere and compared to when you were developing it,  you might have like a massively larger data set, so everything just slows down.  But so the point is, this is extremely difficult to replicate with active monitoring.  Even if you try to take care of all these things, maybe you're creating a, instead of fetching a static web page from your web server, from your monitoring system,  you might have a much more complicated page that tries to exercise all these things.  It might actually try to connect to the database, issue a set of queries that you know from your application that it might be running and so on and so on.  So you might be able to get it close, but inevitably there will be something that you're stubbing out, simple as that.  There's always going to be something you're stubbing out.  Again, assuming that this is a, if this is a web shop, your active monitoring system might go and click on different things to make sure that you can actually,  you know, see the items that you're selling.  But maybe you don't put anything in your shopping cart.  If you do put it in your shopping cart, you might not go through the entire checkout process.  If you do, you probably don't send it off to your payment gateway because that's going to cost you money.  And if you do that, you probably don't charge the, you either probably don't charge the full amount because then it's going to cost you a whole lot of money,  or maybe you're buying some kind of fake item that doesn't exist, and then that might work, but all the other ones might not.  If you do make the credit cards transaction and you do buy something real, you probably don't send a packaging slip to your shipping and handling people for them to pack it up and send it somewhere.  And if you do, you probably don't have a machine built that will make sure that the postman eventually brings it to you and unpacks it and makes sure it's the right stuff in there.  So inevitably, there's going to be some part of the process that you're trying to emulate that you're going to be stopping out or changing.  So no matter how hard you try this, you're never going to be exercising the same things as your real users are going to be exercising when they're actually using your request.  Now, to add insult to injury, not only are your tests not as comprehensive as it would be if you looked at the real data, if this is just a moderately busy website,  consider that last time I checked, the default time between a check in Nagios was three minutes.  In those three minutes, again, if this is a moderately busy website, you will have had dozens, maybe even hundreds of thousands of requests.  So you have tons of data that you're just throwing away.  And this is good data. It's better than what you would have been getting with your active monitoring, but you're just throwing away.  And that is a damn shame.  So the next thing I have is something I like to call temporal tunnel vision.  So the typical anatomy of a Nagios plug-in is so that it tries to exercise the system that you're testing.  So it tries to connect to it, and then it records how long time it took, and eventually it times out.  It doesn't wait forever, but it tries to connect, and then it records how long time it took.  And then if it's within some threshold, like, say, two seconds, then it's in the green area, and everything is fine.  If it's between two and four seconds, then it's not quite so good, so it reports back that it's in a warning state.  And if it's above four seconds, then it just exposes, and says, oh, no, it starts sending text messages, and what have you.  But as soon as this information leaves the plug-in, the actual data point, which in this case is maybe 1.9 seconds or whatever, thereabouts,  as soon as it leaves the plug-in, that is just really debug information.  Nagios doesn't record it. It only looks at what is going on at this very instant.  It only knows, you know, what is the state right now.  This data point doesn't feed into the data model of Nagios at all. It doesn't care.  It's just a, it's only for presentation when you're looking at it right now.  It doesn't remember it, it doesn't record it, it doesn't do anything with this number.  And the reason that this is so incredibly tragic is that if we didn't look through, if we didn't have this tunnel vision of only looking at the slice of history,  that is the current, we might be able to see a trend that would have told us where this was going.  Now, if this is the history for the last two months, but all we're seeing is the current, we think that everything is fine.  But if you look at the history, it doesn't take a PhD in statistics to realize, you know, halfway through that this is going to be a problem now, about a month from now.  But we don't even have the data. So even if we managed to apply some of this logic, I mean, we don't even have the data available to us,  because the tools just throw it away.  Now, Nagios throws it away, but Muenin doesn't.  So a lot of these systems, either they build into one or maybe have separate systems like Nagios and Muenin.  So you might be generating graphs, so you might be recording this, you might actually have this graph available to you.  But if you have just a moderate amount of systems and metrics that you care about, you might have like thousands of graphs like these,  and you're not going to sit around looking at them and to see that, oh yeah, that's going to be a problem in about a month.  Because, I mean, who does that? Very boring people, right?  I mean, no one. No one does that. It's just not the way it works, because everyone is busy dealing with things when they blow up a minute from now.  So they don't have time to look at all of this.  And the reason this is so incredibly tragic is that it's, we have had algorithms for like since the 60s that could look at this data  and very accurately forecast when this is going to be a problem.  And this is not complicated stuff. I mean, it's stuff that you can do on the back of an envelope.  I could do it with pen and paper. It is so simple. It is constant space complexity.  It is dead simple stuff. And we've known about this since the 60s, and we're not doing it.  It is absolutely tragic. I mean, when we were here, the system should have added a reminder to our calendar saying,  well, you know, on March 1st, this is going to be, this will have blown up.  So you should probably, you know, look at it now instead of waiting until it actually blows up.  And I think this is, I think this is absolutely tragic, because there's really no excuse because it's so simple.  And the same, not the same algorithms, but there are similar algorithms that are equally old that can do the same thing for things that look much, much better.  Things that look much more complicated than this. I mean, if it's going up and down, if it's seasonal, if it's, I mean, it can,  you can very accurately say something about what's going to happen next with this, with these things, but we don't.  And I think that's the shame.  Okay. So the reason that we monitor things is that we like to detect when things are not normal.  This is perfectly sound because this is what we care about. We want to know when something is normal so we can, you know,  investigate or handle it in some way. The problem is that the way that typical monitoring systems design whether something is normal has absolutely nothing to do with normality.  Nothing.  So a typical example is, is, this might be the number of requests that our database server is seeing per second.  This, let's just pretend that this is 3000 requests a second. And we know that, that is, that is a fine value.  It's not going to be a problem until it reaches about 4000.  So the way that we, we tell and tell our monitoring system about this, the way that we codify this is that we define a static upper limit.  It's, it's the exact same thing we saw here. I mean, as soon as it does an upper limit for the time this may take, and as soon as you're above that, then it's a problem.  So we always focus on these upper limits of various things of, you know, how this, how full the discs are, how long times things take, how many requests per second you're seeing,  how much bandwidth you're consuming, and so on and so on. We always focus on these upper limits.  And, and that's fine when it's going like this, because we know that if it goes up to 4000, that's going to turn into an operational problem,  because our database server is going to be under too much load and we're going to have queues that are going to explode.  We're going to have all sorts of problems.  But the thing is that the real world is just more complicated than that.  Now what if this happens?  Now this looks like something I would want to know about. This is definitely not normal. I mean, this is a problem.  Probably. The thing is I don't know. But my monitoring system is never going to tell me that, oh, you had something that was too low,  because we're always focusing on setting a static upper limit that is, don't ever go about, about this.  Now you could suggest that, okay, so what do we do? We set like a maximum of 4000 and a minimum of what, 2000?  Because, I mean, that would have given us an alert. And I guess that's a start.  But then you might have things that look like this. These are not real graphs for monitoring systems.  I made this up, by the way. Maybe you can tell.  So this might be over the course of a day. So we have something that goes up and it goes down and then goes up again.  It's very, I mean, maybe it depends on when people are awake or something, right?  So in order to do this, you would have to, in order to specify an upper and lower normal bound of values for this thing,  you would have, you know, the upper bound going right up here somewhere and the lower bound right down here.  Otherwise you'd be getting alerts every single night or every single, you know, day at high noon or whatever.  But what if this happens? So if there's a dip here, this is well above the minimum that we defined.  And this one is way, way below, I mean, the maximum we defined. But this is definitely not normal.  And this is something I would want to know about. And with the current systems available to me,  I mean, they're not going to tell me about this. The only way I would have known about this is if, you know,  if there was like a, some other symptom that I could detect or if customers called and said that there was a, something was wrong.  Or if I, you know, spent, you know, all my time looking at my thousands of graphs to notice, you know, these things that look, you know, kind of strange.  And even if you did do that, I mean, this is very, this is a very pretty graph. I mean, this is a very uniform graph.  It's very easy to see when something is, is, is not normal. But in many cases, these, these go up and down a lot.  It is, it can be very complicated or difficult to spot even with the human eye to when there's something that doesn't look normal.  Now, this is a, this is a CPU graph from a backup server. So if we said that this is midnight, so it's rather day, it doesn't do anything at all.  It just sits around idling. At midnight, all the other servers start throwing data at it. This takes a bit of CPU time in, you know,  doing SSH, decryption, you know, whatever. So this takes a bit of CPU time. And eventually they finish.  And then from 2am to, I don't know, to 3.30am. It starts compressing the data. And this is a massively CPU intensive operation.  And there's a lot of data. So this takes a long time. And during this time, it is, the CPUs are pegged at 100%.  So if you want to monitor that, if you want to, I mean, it's very common to monitor your CPU load of servers.  So you say that, well, on average, on a sliding, on a five minute sliding window, you should never have more than 80% utilization because that's going to,  assuming that the assumption is that that's, that's a problem because if that's, if that goes on, then you're going to be in trouble.  But on this server, I mean, this is completely expected, right? We expect it for several hours a day to be at the max.  So we can't just monitor on, on, on the maximum value that over in the five minute sliding window because we're going to be getting text messages every single night.  And I don't know about you, but that doesn't sound very, very pleasant.  So what we can do instead is that we could just say, well, between two and four AM, we just completed disabled monitoring on the server.  But then what if one day this spike just doesn't show up? If the job fails to run, you'd never know because you just disabled monitor, you just disabled,  you just disabled notifications altogether. So you just, you have no idea.  Now, what can we do about this instead?  Now, if we go back to this one, as I said, there are algorithms available to us that can tell us based on the history of the service,  what to expect from it in the future.  What this basically means is that before we even collect a metric, we have a pretty good idea about it, what it's supposed to be.  So before we even, before we receive another metric from the passive monitor, we know what it's supposed to be.  And that means that when we get it, and it's very different from what we were expecting, we can know that this is not normal.  We have detected an anomaly. And this works perfectly for all these things.  So if you have something that's always pretty much level and suddenly it drops, this is not normal.  This is not what it used to do. And then you can raise an alert.  In this case, you can, you know, from history because you've looked at this service for like days, weeks, months, months, years, you know, whatever.  You know that it always goes up and down, except the one day where it doesn't. It does something wildly different.  And you will know because when you come to this point, the monitoring system has an expectation that it keeps going up.  Because that's what it did yesterday and the day before and last week and last year.  But it didn't. And then it can send you an alert.  And we have the exact same thing here. Not only will it detect if it, if it's completely absent,  or if it takes too long, or if there's another extra spike at some point, at some time of the day.  So, yeah, one would might be tempted to configure.  So the way you would do this is that you would configure the monitoring system to have like different expectations for different times of the day.  So that could absolutely explode the configuration file if you had to specify, you know, for each particular metric,  this is what to expect for this time of day, or this is what to expect for this time of day, and this is what to expect for this time of day.  So it would absolutely explode. But what you would be doing, what you would be using to feed into this would be, you know, the graphs that you've already generated.  So you'd be taking data from your monitoring system and feed it back into your monitoring system's configuration file.  That is the working sort of an insane person.  So of course what you do instead is that you use, I mean, the forecasts that your system is able to give you already.  And then trigger when that is abnormal.  Okay. So what am I going to do about it?  Not a whole lot, because I'm kind of busy doing a lot of other things.  But what I've been working on for a while now is a project called Surveyor.  Surveyor has an extremely simple data model. It has users.  Each user can have a number of services configured.  The server service has metrics that gets fed to it. And a service can also have any number of plugins configured.  Now the way it's supposed to work is that these plugins are configured as URLs.  So the way it works is that you have some agents that might be doing active monitoring if you must, but typically, hopefully, they'll be doing passive monitoring.  So they collect some data from somewhere and they send it to Surveyor and they get a response back saying, all right, I'll take care of it.  What Surveyor does is it adds some extra information to the data points that it got from you.  It adds a timestamp, a service ID, and various things.  And then it adds some saved state. And it sends it to all the different plugins that you've configured.  So this is a completely JSON-based interface. You don't have to have all these plugins running on your server.  They can run on someone on your own server or someone else's server or whatever. It doesn't really matter.  So it basically sends this JSON object to a plugin through this URL, or at this URL.  And then it gets back a response. This response can include some saved state.  So many of these algorithms that I mentioned that can do this stuff, that can do this prediction stuff, they work in constant space.  So they don't really need to store anything on the server that's running the plugin. They just need like a tiny, I mean, five values.  It's all they need to store to remember where we're going and where we're headed.  So what they can do is... I need to point over here, don't I?  Right. So when they've looked at this data, they've analyzed it, they've come up with a... they've decided whether it's a good or it's a bad...  And then they include some state that they send back to Surveyor.  Now the next time that Surveyor receives a data point from this particular service, it includes it in the request that it sends to the plugin.  So this means that you can have these algorithms, these forecasting algorithms.  I mean, they shouldn't take more than... I can't imagine that it'd be more than 10 lines of PHP or Python or whatever.  It's really simple stuff. And you don't even have to worry about saving the state because Surveyor will do that for you.  Now when it responds, it can say, yep, that looked perfectly fine. That was exactly what I expected. Everything's good.  Or it can say that an anomaly was detected. It can include different kinds of information and whatever.  And if it sends back... if it tells Surveyor back that this was anomalous, something that this is not what I expected,  then Surveyor can send you a text message and you can act upon that and you can do various things.  So you can have plugins. I mean, this is all there is to Surveyor right now. It doesn't do anything other than this.  But it's very simple to add a plugin that feeds this data into our D-Tool to create graphs.  It's very simple to make one of these forecasting algorithms. It's very simple to really add more simple plugins to this.  So the idea is that this will... I mean, you can run this on your own servers and whatever.  Or you can use an instance of it that I'm going to set up when I get around to it.  And then anyone who wants to write plugins for it, they can just publish the URLs for it and you can configure those for your services.  And the world will be a better place, pretty much.  You can also have a plugin that instead of worrying about alerting on anomalies, it can maybe give you a list of top 10 next...  I mean, the 10 next problems that you're most likely to run into in your data center because it knows...  I mean, it can detect things that are heading towards failure.  And then it can tell you that, well, the next thing that's going to fail is probably this disk over here that's going to run full.  It might only be at 30% now, but it's rising at a very critical pace.  So this is going to be a problem next week, but it's only at 30% now, so your typical monitoring system wouldn't have known.  But we can know that now.  Right.  So, Sovela lives in a number of different places.  The primary place is on Launchpad. This is confusing to look at.  So it lives on Launchpad. That's where you can report bugs and whatever.  There's a mirror on GitHub. There's a homepage for it. There's all sorts of different things.  And it would be great if people would help me write it, make it awesome, or just write plugins for it, or do any...  Do just help me work on it because we're going to make the world a better place.  This was way faster than I expected it to.  Questions?  There are microphones there, there, and up there.  How we detect the difference between the evolution of normality and abnormality.  For instance, I'm having a web shop in Belgium and the customers are shopping mostly between 8am and 10pm.  Then I'm opening a shop in Australia or something.  And all of a sudden there are a lot more data requests coming in and other times.  How would you make the difference between the evolution or the change in normality and abnormality?  The question is how would I tell the difference between a change in normality, so to speak?  If you're used to having your user base mostly in one part of the world and with the corresponding peak in load and dips in our values in load,  then if you suddenly have a user group in a very different part of the world, that graph is going to look very different.  The data is going to look very different.  Yes, the first time that happens, as soon as that happens, the first time you would...  What are you? Oh, there you are.  The first time it happens you're going to get an alert.  When you acknowledge that this is going on, this is what...  If I have received the alert, then this doesn't work right now.  But the way it could work is that you could tell the system that normality has changed.  You can either change it into just a completely...  Don't send any alerts right now. Just look at what's happening now and learn from that.  Or maybe you can favor...  Depending on how stable the trends and the patterns are, you can tell it to which extent you want to favor very fresh data or very old data.  And if you know that things are changing very quickly right now, you could tell it to give more preference to current data  because it's... to kind of catch up with the change in normality.  But it's a good question. Thanks.  Hi. I missed the first part of the talk for which I apologize. I was at the key signing.  But at any rate, if I missed this, apologies.  Have you given consideration to doing this stuff at very, very large scale?  Tens of thousands, hundreds of thousands of nodes, and the implications of using TCP and JSON instead of something that's field proven like SNMP?  Not a whole lot. No.  I haven't really... I intentionally didn't try to optimize those sorts of things yet because I don't really know where it's going to fall apart.  I have chosen a data store for it that I believe will scale well.  I'm using React for it right now, which is a distributed hash table to kind of do its distribution,  which I trust to be able to handle even very, very, very big volumes of data.  But in terms of the configuration communication between the different components,  no, I haven't really... I haven't thought about changing that to any degree.  Thank you.  The message passing diagram that you showed suggests that you either need a fair amount of processing power for working out...  Sorry, I can't hear it all.  The message passing algorithm diagram that you showed suggests you either need a fair amount of processing power for running some algorithms  or a reasonable sized database for historical data. Does that rely or reside on the system on the right-hand side,  the remote central server, or on the agents on the left-hand side?  Where do you see that information storage and processing being located?  The way I'm imagining it right now is that a surveillance, I mean, the central component will keep a record of everything.  And then if one of these plugins need historical data to do something, it can query for it.  And then depending on when you configure it, you should be able to tell it that...  Basically, the plugin I have that needs the most historical data only needs, like, last year.  And then you can start discarding data that is older than one year. And that keeps it sort of under control.  Hopefully, at least.  So from the earlier question that you had from up here, I would respond 99.9% of us aren't Google and Facebook.  We don't need to worry about those for most of us, and they've probably built their own systems for your answer to JSON being too heavy.  But my question was actually things like monthly events and yearly events.  It's easy even for one week over a little wild collect data, but being woken up at midnight every month or every year is still going to happen with a monitoring system like that,  because you basically need forever's history to see those things. How would you cope with that?  I'm sorry, I couldn't quite hear you. I didn't catch you completely. Sorry.  Okay, you have a nice little graph for each week or each day, the patterns that happen over there.  There's fantastic stuff for that recently, a site that was doing this really well.  But the problem is monthly events, yearly events, the big roll-ups that happen at the end of each month or each year are going to be a special case and are still going to wake you if you're setting alerts for it's gone outside normal.  There's always...  Not necessarily. I mean, some of these algorithms you can apply, you can tell them not just that you can have multiple seasons.  So you can tell that there is a strong daily season that you should look at, but there might also be a monthly and a yearly or something else that makes sense for your particular use case.  Cool.  And then it can... Even though it knows that there is one pattern to follow for the day, it can know that it has another every month, there's something that only happens once a month or once a year.  So you can have multiple seasons that can feed into this algorithm.  Great. Thank you.  Are you involved in the monitoring sucks, hackathon?  Sorry? Are you involved in the monitoring sucks, hackathon?  No, I'm not. I have to go home, unfortunately. I didn't hear about it until after I put... So, no, I'm afraid not.  Hi.  I understand the idea of the Surveyor, but I also understand so in development, it's not finished.  And it has a plugin architecture. So if I would use it today, what kind of anomaly detection does it have out of the box?  Or do I need to first write my own anomaly detection plugins to get it working?  It doesn't actually... I haven't actually written any plugins yet because I've been busy trying to get the stuff that you just saw get that finished.  But as I said, some of the easier ones, the easier algorithms, it is literally ten lines of code.  And I have that code written already. I just haven't turned it into a plugin for this.  Yeah, because I understand because you said it's very easy, like ten lines of code, but probably not for everyone.  Very easy to write such a thing. So I mean, and usually to get started, you need to have some examples.  Absolutely. So that's definitely the idea that the system will ship with a number of different plugins that will do all the things that I've already mentioned.  And this is not far off. Probably next week. It's simple stuff. I just haven't done the roundtrip yet.  And it will be included and available and all this stuff.  And I'll probably also be writing kind of skeleton plugins in PHP and in Python and whatever.  So if someone wants to write plugins, it's really easy to just take that code and start adding simple stuff.  So if I would say, check it out next week and download the example plugin, what kind of anomaly would that, would it be able to detect?  Well, the first plugin that I'm going to add is one that will be very good at detecting, you know, the, this stuff.  So stuff that doesn't actually have any seasonality to it, but only has a, like a trend that isn't affected by, you know, with the time of day or so on and so on.  A linear trend?  Not necessarily. It doesn't have to be linear. It just, it doesn't actually have to be linear.  It will very quickly notice if it's going exponential or something.  So that, that is no more than a week away. It is, it is very simple.  The other stuff I need to, for the next one that actually can do seasonality, I'll, I have to let, I mean, add a way for it to extract data from Surveyor again, so that it can, it can work with that.  But it's not far off. It's simple and it will be included in this. It's not going to be something you have to do yourself.  Okay. Thank you very much.  Thank you.  Looking forward.  Yeah.  I would be very happy to write some of these anomaly detections for myself and I can, of course, go to contribute them.  But you, you're mentioning, okay, these are very simple things and these exist from the sixties.  Would you have any pointers like how to start, like, where, which commonly known algorithms?  Sure. Mathematical models or whatever.  The one I'm referring to from the sixties is, is one called Hold Winter's Double and Triple exponential smoothing.  If you can apply that and get extremely good results for stuff like this.  But I mean, there has been tons of research done since then and there are much more sophisticated algorithms, but this stuff is just really, really low hanging fruit.  Yeah.  But did you do some research on, because I did search as well for some things and I had a very hard time to find actually usable things.  Okay. I'm not a mathematician, mathematicians.  And so it's not so easy for me to find my way around.  There's actually, there's a scientific journal dedicated to forecasting.  If you, if you Google for, I think it's just called Journal of Forecasting.  So every, I think it's every year.  I mean, they publish a major journal.  They have conventions every year about, you know, research into forecasting and stuff.  And there's lots of really excellent and really quite readable articles in there that you can use to, to, to catch up on things.  It is very valuable.  There's a guy called Rob Hindman, which is, I think he's the editor in chief of that journal, but he's also one of the, the foremost researchers into the subject.  And he has a webpage where he explains all sorts of things and he, he writes all his example code in GNU, R.  I mean, the statistical tool from GNU.  So it's very, very approachable if you have just a basic mathematical understanding of if you know about the tools already.  So that's, so those are probably things to look at.  Okay. Thank you.  Thank you.  So we're talking about monitoring.  So we're talking about a large amount of data.  How does Surveyor store this data?  The data is stored in, in, in RIOC, which is one of the, one of the new fangled, no SQL data stores.  It is, I mostly chose it because I, I, I very much liked the data model in it.  It scales extremely well horizontally.  And I don't really need a lot of fancy things from the data store.  I just need a way to be able to store data and pull it out again really quickly.  Anything else is just, really just fluff.  And for those sorts of things, RIOC is absolutely perfect because it doesn't, that the operational model of it just like makes it so incredibly trustworthy.  Because I don't want to get up at night to fix my monitoring service.  Can you please repeat this name of no SQL?  It's RIOC. R-I-A-K.  RIOC. Thanks.  So, working.  So in a couple of cases, I might have, you know, thank you.  Very distigrate.  Many gigabytes of old logs that I could extract a whole load of running data from your how long query.  He's talking about that kind of thing.  Does your plugin model allow me to go back through those and historically pick, pick out when a particular prediction model would have predicted a,  an issue.  So I'm going to compare that to my maintenance logs and things so that I can tell would this have been a bad thing to have run?  Would this have woke me up every midnight?  You know, when would this have woke me up historically?  I don't think, I'm sorry.  I don't think I'm, something's beeping.  What's going on?  Oh, that's me.  I'm leaning on something.  Sorry.  Hope it doesn't blow up.  Sorry, I didn't completely catch the question.  So it's about what I'm doing.  So if I've got a lot of historical data, you know, logs I can process into the kind of, kind of events you're looking at, how long we request talk or something.  Is, does your model for your new tool allow me to pump that and find out when it would have alerted me?  Not right now.  Conceptually though, is that, does that break your model or not?  Well, that's a good question.  I mean, the current model, only really, I mean, you're just feeding it raw values.  Depending on the type of, depending on the source, you might want to change, you might want to do some pre-processing in the agent that you have that's collecting the data to kind of make it, turn it into something that is more of a, it's kind of hard to, hmm, I haven't really thought about how to put this into words.  Sorry.  So if we're using like terminology from RRD tool, it has like gauges, which is something like a temperature would be like a gauge.  Yeah.  And then it has, you know, bytes per time unit, which is a different thing because the way that you graph it is separate.  And I would expect that kind of pre-processing to live in the agent.  So, so they shouldn't really have to know about it.  And hopefully the plugins wouldn't either.  But maybe I haven't thought it through enough.  Maybe that's not going to work in the real world.  But that's what I'm thinking right now.  Okay.  Thanks.  So I have all this data in Surveyor and is it intended to also do graphs of this?  It will, I haven't really figured out if I'm, I try to do graphs directly in Surveyor.  But because of React, because I'm storing every data point as a single key in React, generating a graph that spans a year, if you have metrics per minute,  I mean, that is a lot of, you know, individual data points that have to be pulled out and it takes incredible amounts of time.  And I would have to completely change the data model to make that work.  And I haven't really figured out with myself if I decided not to do graphs in Surveyor because I'm, I keep insisting on wanting to use React.  Or if I actually think that the best thing to do is to put it into a plugin.  So there could be a plugin that just takes the data and feeds it into RLD tool and then has that generate data graphs because RLD tool is really good at it.  But I don't know.  I mean, using React is not set in stone.  It's just what I tend to prefer right now.  But it's making it very difficult to do graphs directly in Surveyor because of the time it takes to get individual data points out.  Have you thought about the problem with React that it doesn't have data center or a rack awareness?  It doesn't have what?  Data center or rack awareness that, for example, if you have three data centers, one node in each data center,  and then multiple nodes in each data center and one data center completely goes down.  Right.  So it may happen that you can't read or write your data anymore depending on how your formulator writes.  I know that React, there's an enterprise version of React that is supposedly really good at that.  So if you have multiple sites, it will actually, the distribution model is apparently sensible.  But I'm not having really, I mean, I don't know of any other system that does that really well.  So if I would have to attack that problem myself, I would have to basically build some kind of distribution system or reliability model on top of it.  And that is just, I mean, that's too soon.  Yeah.  Okay.  Thank you.  Sure.  All right.  Cool.  Thank you.  Thank you.
Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_0_1211.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_1212_1391.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_1392_2075.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_2076_5687.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_5688_6287.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_6288_6371.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_6372_6575.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_6576_6887.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_6888_7223.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_7224_7523.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_7524_8831.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_8832_14483.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_14484_14567.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_14568_16607.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_16608_16691.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_16692_16811.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_16812_16931.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_16932_17183.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_17184_17963.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_17964_18983.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_18984_19067.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_19068_19487.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_19488_20111.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_20112_20639.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_20640_21047.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_21048_21119.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_21120_21299.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_21300_34199.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_34200_34511.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_34512_35603.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_35604_36227.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_36228_36995.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_36996_46979.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_46980_48923.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_48924_49487.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_49488_52415.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_52416_59999.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_60000_60275.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_60276_69383.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_69384_72011.png Bringing_monitoring_into_the_21st_century_hs6Hjlpf5ck_2_72012_73024.png
My name is Sohn and I'm here today to complain to you about monitoring. First, this is the agenda for today. First I'm going to introduce the subject, introduce myself. Then I'm going to go through some of my favorite pet peeves that I have with the current state of free software monitoring. And then I'm going to talk a little bit about what I'm going to do about it and finally questions and answers. Okay, so for the last 10 years I've run various things in production like everything from mail servers to web servers and all sorts of different things. And being a lazy person I don't really want to spend time looking at these things, looking after them when I can have a computer look after my computers for me. So I've always been a fan of monitoring systems and having them do all my older, really tedious work for me. For me personally that means that's typically meant that I use Nagios and Munim. So Nagios for status monitoring and alerting and sending text messages and whatnot. And then Munim to create graphs and stuff so I can use that for, so I can look at that if I'm really bored. I realize that there are many, many other tools out there. I mean there are way more than these. There are tons of free software solutions for monitoring different things. But because I'm used to Nagios and Munim, so I'm probably going to be mentioning those quite a bit. But as far as I know all the problems that I'm going to explain today apply equally well to all of these. So when I say Nagios and Munim I'm not particularly trying to pick on them because everything else sucks as well. Okay, so the first thing I want to talk about, well it's the third thing by the way, the first problem that I want to talk about is active versus passive monitoring. There's a tendency in the current software selection that we have for doing monitoring to favor active monitoring over passive monitoring. So just to get us all on the same page, basically active monitoring is when you're probing the system that you're testing. So you're basically emulating a real client to verify that it's doing what you're expecting it to do. Now I'm going to be, as I said, I'm going to be mentioning Nagios and Munim a lot. I'm also going to be assuming that what we're monitoring is a web application. So there's a web from then, there's a database server. Maybe there's a, let's pretend it's a web shop, so there might be a payment gateway as well. Just so that we have an example that is reasonably simple to understand for most people. But again, the problems that I'm talking about, they apply equally well to any other thing that you might be monitoring. So when you're doing active monitoring of a system like that, what you typically do is you connect to it and you verify that you get a response in a time of fashion. Not a request obviously. So you pretend that you're a real client and your time, how long it takes to get a response. And based on that, you make a decision as to whether you think that the service is in a great shape or in a not so great shape or in great trouble. Now on the other hand, if you're doing passive monitoring, you're just watching things that are happening anyway. So you have requests coming in from real users. So instead of making your own emulated, sterile sort of requests, you're just looking at whatever your users are doing already and making your value judgment based on that. So you're deciding whether they are seeing appropriate response times or appropriate response codes and what have you. So that's the major difference between the two. And I'm going to try to convince you that passive monitoring beats the crowd of active monitoring any day. So the first immediately obvious reason why this is the case is that it puts your users or your customers experience with your application at the center of what you care about. Because you're looking at the request that they're making, you're looking at the experience that they have because you're timing that instead of making some half-assed sterile request to your application and timing that. So you're acknowledging that there's a difference between what they might be seeing quantitatively compared to what you're seeing because it is a different, I mean, it is a much more complex system that they're exercising. They have their own bandwidth limitations and they're doing slightly different things as well. But there's also a massive qualitative difference between these things because you're not actually making sure that things work. It's much like when you're doing, if you have unit tests for your software application, that doesn't guarantee that the whole really works. You're making sure that the individual components, they're doing what they're supposed to, but that doesn't mean that the whole is in perfect working order. And anyone who has done this in the real world will know that just because your monitoring system can connect to your web server and fetch some stupid static page very quickly and it can connect to your database server as well, that doesn't necessarily mean that your web application is in perfect working order. Because just because that is true, that doesn't mean that your web server can actually reach your database server. There might be a network problem, there might be all sorts of different problems. If it can actually connect, sorry, it may not be able to authenticate, maybe it's using different credentials, maybe it forgot its credentials, I mean, any number of things could go wrong. If it can actually authenticate, it might have typos in the queries that it's trying to run. If it has perfectly typed queries, it might be really slow because you're missing an index on a table in a database somewhere and compared to when you were developing it, you might have like a massively larger data set, so everything just slows down. But so the point is, this is extremely difficult to replicate with active monitoring. Even if you try to take care of all these things, maybe you're creating a, instead of fetching a static web page from your web server, from your monitoring system, you might have a much more complicated page that tries to exercise all these things. It might actually try to connect to the database, issue a set of queries that you know from your application that it might be running and so on and so on. So you might be able to get it close, but inevitably there will be something that you're stubbing out, simple as that. There's always going to be something you're stubbing out. Again, assuming that this is a, if this is a web shop, your active monitoring system might go and click on different things to make sure that you can actually, you know, see the items that you're selling. But maybe you don't put anything in your shopping cart. If you do put it in your shopping cart, you might not go through the entire checkout process. If you do, you probably don't send it off to your payment gateway because that's going to cost you money. And if you do that, you probably don't charge the, you either probably don't charge the full amount because then it's going to cost you a whole lot of money, or maybe you're buying some kind of fake item that doesn't exist, and then that might work, but all the other ones might not. If you do make the credit cards transaction and you do buy something real, you probably don't send a packaging slip to your shipping and handling people for them to pack it up and send it somewhere. And if you do, you probably don't have a machine built that will make sure that the postman eventually brings it to you and unpacks it and makes sure it's the right stuff in there. So inevitably, there's going to be some part of the process that you're trying to emulate that you're going to be stopping out or changing. So no matter how hard you try this, you're never going to be exercising the same things as your real users are going to be exercising when they're actually using your request. Now, to add insult to injury, not only are your tests not as comprehensive as it would be if you looked at the real data, if this is just a moderately busy website, consider that last time I checked, the default time between a check in Nagios was three minutes. In those three minutes, again, if this is a moderately busy website, you will have had dozens, maybe even hundreds of thousands of requests. So you have tons of data that you're just throwing away. And this is good data. It's better than what you would have been getting with your active monitoring, but you're just throwing away. And that is a damn shame. So the next thing I have is something I like to call temporal tunnel vision. So the typical anatomy of a Nagios plug-in is so that it tries to exercise the system that you're testing. So it tries to connect to it, and then it records how long time it took, and eventually it times out. It doesn't wait forever, but it tries to connect, and then it records how long time it took. And then if it's within some threshold, like, say, two seconds, then it's in the green area, and everything is fine. If it's between two and four seconds, then it's not quite so good, so it reports back that it's in a warning state. And if it's above four seconds, then it just exposes, and says, oh, no, it starts sending text messages, and what have you. But as soon as this information leaves the plug-in, the actual data point, which in this case is maybe 1.9 seconds or whatever, thereabouts, as soon as it leaves the plug-in, that is just really debug information. Nagios doesn't record it. It only looks at what is going on at this very instant. It only knows, you know, what is the state right now. This data point doesn't feed into the data model of Nagios at all. It doesn't care. It's just a, it's only for presentation when you're looking at it right now. It doesn't remember it, it doesn't record it, it doesn't do anything with this number. And the reason that this is so incredibly tragic is that if we didn't look through, if we didn't have this tunnel vision of only looking at the slice of history, that is the current, we might be able to see a trend that would have told us where this was going. Now, if this is the history for the last two months, but all we're seeing is the current, we think that everything is fine. But if you look at the history, it doesn't take a PhD in statistics to realize, you know, halfway through that this is going to be a problem now, about a month from now. But we don't even have the data. So even if we managed to apply some of this logic, I mean, we don't even have the data available to us, because the tools just throw it away. Now, Nagios throws it away, but Muenin doesn't. So a lot of these systems, either they build into one or maybe have separate systems like Nagios and Muenin. So you might be generating graphs, so you might be recording this, you might actually have this graph available to you. But if you have just a moderate amount of systems and metrics that you care about, you might have like thousands of graphs like these, and you're not going to sit around looking at them and to see that, oh yeah, that's going to be a problem in about a month. Because, I mean, who does that? Very boring people, right? I mean, no one. No one does that. It's just not the way it works, because everyone is busy dealing with things when they blow up a minute from now. So they don't have time to look at all of this. And the reason this is so incredibly tragic is that it's, we have had algorithms for like since the 60s that could look at this data and very accurately forecast when this is going to be a problem. And this is not complicated stuff. I mean, it's stuff that you can do on the back of an envelope. I could do it with pen and paper. It is so simple. It is constant space complexity. It is dead simple stuff. And we've known about this since the 60s, and we're not doing it. It is absolutely tragic. I mean, when we were here, the system should have added a reminder to our calendar saying, well, you know, on March 1st, this is going to be, this will have blown up. So you should probably, you know, look at it now instead of waiting until it actually blows up. And I think this is, I think this is absolutely tragic, because there's really no excuse because it's so simple. And the same, not the same algorithms, but there are similar algorithms that are equally old that can do the same thing for things that look much, much better. Things that look much more complicated than this. I mean, if it's going up and down, if it's seasonal, if it's, I mean, it can, you can very accurately say something about what's going to happen next with this, with these things, but we don't. And I think that's the shame. Okay. So the reason that we monitor things is that we like to detect when things are not normal. This is perfectly sound because this is what we care about. We want to know when something is normal so we can, you know, investigate or handle it in some way. The problem is that the way that typical monitoring systems design whether something is normal has absolutely nothing to do with normality. Nothing. So a typical example is, is, this might be the number of requests that our database server is seeing per second. This, let's just pretend that this is 3000 requests a second. And we know that, that is, that is a fine value. It's not going to be a problem until it reaches about 4000. So the way that we, we tell and tell our monitoring system about this, the way that we codify this is that we define a static upper limit. It's, it's the exact same thing we saw here. I mean, as soon as it does an upper limit for the time this may take, and as soon as you're above that, then it's a problem. So we always focus on these upper limits of various things of, you know, how this, how full the discs are, how long times things take, how many requests per second you're seeing, how much bandwidth you're consuming, and so on and so on. We always focus on these upper limits. And, and that's fine when it's going like this, because we know that if it goes up to 4000, that's going to turn into an operational problem, because our database server is going to be under too much load and we're going to have queues that are going to explode. We're going to have all sorts of problems. But the thing is that the real world is just more complicated than that. Now what if this happens? Now this looks like something I would want to know about. This is definitely not normal. I mean, this is a problem. Probably. The thing is I don't know. But my monitoring system is never going to tell me that, oh, you had something that was too low, because we're always focusing on setting a static upper limit that is, don't ever go about, about this. Now you could suggest that, okay, so what do we do? We set like a maximum of 4000 and a minimum of what, 2000? Because, I mean, that would have given us an alert. And I guess that's a start. But then you might have things that look like this. These are not real graphs for monitoring systems. I made this up, by the way. Maybe you can tell. So this might be over the course of a day. So we have something that goes up and it goes down and then goes up again. It's very, I mean, maybe it depends on when people are awake or something, right? So in order to do this, you would have to, in order to specify an upper and lower normal bound of values for this thing, you would have, you know, the upper bound going right up here somewhere and the lower bound right down here. Otherwise you'd be getting alerts every single night or every single, you know, day at high noon or whatever. But what if this happens? So if there's a dip here, this is well above the minimum that we defined. And this one is way, way below, I mean, the maximum we defined. But this is definitely not normal. And this is something I would want to know about. And with the current systems available to me, I mean, they're not going to tell me about this. The only way I would have known about this is if, you know, if there was like a, some other symptom that I could detect or if customers called and said that there was a, something was wrong. Or if I, you know, spent, you know, all my time looking at my thousands of graphs to notice, you know, these things that look, you know, kind of strange. And even if you did do that, I mean, this is very, this is a very pretty graph. I mean, this is a very uniform graph. It's very easy to see when something is, is, is not normal. But in many cases, these, these go up and down a lot. It is, it can be very complicated or difficult to spot even with the human eye to when there's something that doesn't look normal. Now, this is a, this is a CPU graph from a backup server. So if we said that this is midnight, so it's rather day, it doesn't do anything at all. It just sits around idling. At midnight, all the other servers start throwing data at it. This takes a bit of CPU time in, you know, doing SSH, decryption, you know, whatever. So this takes a bit of CPU time. And eventually they finish. And then from 2am to, I don't know, to 3.30am. It starts compressing the data. And this is a massively CPU intensive operation. And there's a lot of data. So this takes a long time. And during this time, it is, the CPUs are pegged at 100%. So if you want to monitor that, if you want to, I mean, it's very common to monitor your CPU load of servers. So you say that, well, on average, on a sliding, on a five minute sliding window, you should never have more than 80% utilization because that's going to, assuming that the assumption is that that's, that's a problem because if that's, if that goes on, then you're going to be in trouble. But on this server, I mean, this is completely expected, right? We expect it for several hours a day to be at the max. So we can't just monitor on, on, on the maximum value that over in the five minute sliding window because we're going to be getting text messages every single night. And I don't know about you, but that doesn't sound very, very pleasant. So what we can do instead is that we could just say, well, between two and four AM, we just completed disabled monitoring on the server. But then what if one day this spike just doesn't show up? If the job fails to run, you'd never know because you just disabled monitor, you just disabled, you just disabled notifications altogether. So you just, you have no idea. Now, what can we do about this instead? Now, if we go back to this one, as I said, there are algorithms available to us that can tell us based on the history of the service, what to expect from it in the future. What this basically means is that before we even collect a metric, we have a pretty good idea about it, what it's supposed to be. So before we even, before we receive another metric from the passive monitor, we know what it's supposed to be. And that means that when we get it, and it's very different from what we were expecting, we can know that this is not normal. We have detected an anomaly. And this works perfectly for all these things. So if you have something that's always pretty much level and suddenly it drops, this is not normal. This is not what it used to do. And then you can raise an alert. In this case, you can, you know, from history because you've looked at this service for like days, weeks, months, months, years, you know, whatever. You know that it always goes up and down, except the one day where it doesn't. It does something wildly different. And you will know because when you come to this point, the monitoring system has an expectation that it keeps going up. Because that's what it did yesterday and the day before and last week and last year. But it didn't. And then it can send you an alert. And we have the exact same thing here. Not only will it detect if it, if it's completely absent, or if it takes too long, or if there's another extra spike at some point, at some time of the day. So, yeah, one would might be tempted to configure. So the way you would do this is that you would configure the monitoring system to have like different expectations for different times of the day. So that could absolutely explode the configuration file if you had to specify, you know, for each particular metric, this is what to expect for this time of day, or this is what to expect for this time of day, and this is what to expect for this time of day. So it would absolutely explode. But what you would be doing, what you would be using to feed into this would be, you know, the graphs that you've already generated. So you'd be taking data from your monitoring system and feed it back into your monitoring system's configuration file. That is the working sort of an insane person. So of course what you do instead is that you use, I mean, the forecasts that your system is able to give you already. And then trigger when that is abnormal. Okay. So what am I going to do about it? Not a whole lot, because I'm kind of busy doing a lot of other things. But what I've been working on for a while now is a project called Surveyor. Surveyor has an extremely simple data model. It has users. Each user can have a number of services configured. The server service has metrics that gets fed to it. And a service can also have any number of plugins configured. Now the way it's supposed to work is that these plugins are configured as URLs. So the way it works is that you have some agents that might be doing active monitoring if you must, but typically, hopefully, they'll be doing passive monitoring. So they collect some data from somewhere and they send it to Surveyor and they get a response back saying, all right, I'll take care of it. What Surveyor does is it adds some extra information to the data points that it got from you. It adds a timestamp, a service ID, and various things. And then it adds some saved state. And it sends it to all the different plugins that you've configured. So this is a completely JSON-based interface. You don't have to have all these plugins running on your server. They can run on someone on your own server or someone else's server or whatever. It doesn't really matter. So it basically sends this JSON object to a plugin through this URL, or at this URL. And then it gets back a response. This response can include some saved state. So many of these algorithms that I mentioned that can do this stuff, that can do this prediction stuff, they work in constant space. So they don't really need to store anything on the server that's running the plugin. They just need like a tiny, I mean, five values. It's all they need to store to remember where we're going and where we're headed. So what they can do is... I need to point over here, don't I? Right. So when they've looked at this data, they've analyzed it, they've come up with a... they've decided whether it's a good or it's a bad... And then they include some state that they send back to Surveyor. Now the next time that Surveyor receives a data point from this particular service, it includes it in the request that it sends to the plugin. So this means that you can have these algorithms, these forecasting algorithms. I mean, they shouldn't take more than... I can't imagine that it'd be more than 10 lines of PHP or Python or whatever. It's really simple stuff. And you don't even have to worry about saving the state because Surveyor will do that for you. Now when it responds, it can say, yep, that looked perfectly fine. That was exactly what I expected. Everything's good. Or it can say that an anomaly was detected. It can include different kinds of information and whatever. And if it sends back... if it tells Surveyor back that this was anomalous, something that this is not what I expected, then Surveyor can send you a text message and you can act upon that and you can do various things. So you can have plugins. I mean, this is all there is to Surveyor right now. It doesn't do anything other than this. But it's very simple to add a plugin that feeds this data into our D-Tool to create graphs. It's very simple to make one of these forecasting algorithms. It's very simple to really add more simple plugins to this. So the idea is that this will... I mean, you can run this on your own servers and whatever. Or you can use an instance of it that I'm going to set up when I get around to it. And then anyone who wants to write plugins for it, they can just publish the URLs for it and you can configure those for your services. And the world will be a better place, pretty much. You can also have a plugin that instead of worrying about alerting on anomalies, it can maybe give you a list of top 10 next... I mean, the 10 next problems that you're most likely to run into in your data center because it knows... I mean, it can detect things that are heading towards failure. And then it can tell you that, well, the next thing that's going to fail is probably this disk over here that's going to run full. It might only be at 30% now, but it's rising at a very critical pace. So this is going to be a problem next week, but it's only at 30% now, so your typical monitoring system wouldn't have known. But we can know that now. Right. So, Sovela lives in a number of different places. The primary place is on Launchpad. This is confusing to look at. So it lives on Launchpad. That's where you can report bugs and whatever. There's a mirror on GitHub. There's a homepage for it. There's all sorts of different things. And it would be great if people would help me write it, make it awesome, or just write plugins for it, or do any... Do just help me work on it because we're going to make the world a better place. This was way faster than I expected it to. Questions? There are microphones there, there, and up there. How we detect the difference between the evolution of normality and abnormality. For instance, I'm having a web shop in Belgium and the customers are shopping mostly between 8am and 10pm. Then I'm opening a shop in Australia or something. And all of a sudden there are a lot more data requests coming in and other times. How would you make the difference between the evolution or the change in normality and abnormality? The question is how would I tell the difference between a change in normality, so to speak? If you're used to having your user base mostly in one part of the world and with the corresponding peak in load and dips in our values in load, then if you suddenly have a user group in a very different part of the world, that graph is going to look very different. The data is going to look very different. Yes, the first time that happens, as soon as that happens, the first time you would... What are you? Oh, there you are. The first time it happens you're going to get an alert. When you acknowledge that this is going on, this is what... If I have received the alert, then this doesn't work right now. But the way it could work is that you could tell the system that normality has changed. You can either change it into just a completely... Don't send any alerts right now. Just look at what's happening now and learn from that. Or maybe you can favor... Depending on how stable the trends and the patterns are, you can tell it to which extent you want to favor very fresh data or very old data. And if you know that things are changing very quickly right now, you could tell it to give more preference to current data because it's... to kind of catch up with the change in normality. But it's a good question. Thanks. Hi. I missed the first part of the talk for which I apologize. I was at the key signing. But at any rate, if I missed this, apologies. Have you given consideration to doing this stuff at very, very large scale? Tens of thousands, hundreds of thousands of nodes, and the implications of using TCP and JSON instead of something that's field proven like SNMP? Not a whole lot. No. I haven't really... I intentionally didn't try to optimize those sorts of things yet because I don't really know where it's going to fall apart. I have chosen a data store for it that I believe will scale well. I'm using React for it right now, which is a distributed hash table to kind of do its distribution, which I trust to be able to handle even very, very, very big volumes of data. But in terms of the configuration communication between the different components, no, I haven't really... I haven't thought about changing that to any degree. Thank you. The message passing diagram that you showed suggests that you either need a fair amount of processing power for working out... Sorry, I can't hear it all. The message passing algorithm diagram that you showed suggests you either need a fair amount of processing power for running some algorithms or a reasonable sized database for historical data. Does that rely or reside on the system on the right-hand side, the remote central server, or on the agents on the left-hand side? Where do you see that information storage and processing being located? The way I'm imagining it right now is that a surveillance, I mean, the central component will keep a record of everything. And then if one of these plugins need historical data to do something, it can query for it. And then depending on when you configure it, you should be able to tell it that... Basically, the plugin I have that needs the most historical data only needs, like, last year. And then you can start discarding data that is older than one year. And that keeps it sort of under control. Hopefully, at least. So from the earlier question that you had from up here, I would respond 99.9% of us aren't Google and Facebook. We don't need to worry about those for most of us, and they've probably built their own systems for your answer to JSON being too heavy. But my question was actually things like monthly events and yearly events. It's easy even for one week over a little wild collect data, but being woken up at midnight every month or every year is still going to happen with a monitoring system like that, because you basically need forever's history to see those things. How would you cope with that? I'm sorry, I couldn't quite hear you. I didn't catch you completely. Sorry. Okay, you have a nice little graph for each week or each day, the patterns that happen over there. There's fantastic stuff for that recently, a site that was doing this really well. But the problem is monthly events, yearly events, the big roll-ups that happen at the end of each month or each year are going to be a special case and are still going to wake you if you're setting alerts for it's gone outside normal. There's always... Not necessarily. I mean, some of these algorithms you can apply, you can tell them not just that you can have multiple seasons. So you can tell that there is a strong daily season that you should look at, but there might also be a monthly and a yearly or something else that makes sense for your particular use case. Cool. And then it can... Even though it knows that there is one pattern to follow for the day, it can know that it has another every month, there's something that only happens once a month or once a year. So you can have multiple seasons that can feed into this algorithm. Great. Thank you. Are you involved in the monitoring sucks, hackathon? Sorry? Are you involved in the monitoring sucks, hackathon? No, I'm not. I have to go home, unfortunately. I didn't hear about it until after I put... So, no, I'm afraid not. Hi. I understand the idea of the Surveyor, but I also understand so in development, it's not finished. And it has a plugin architecture. So if I would use it today, what kind of anomaly detection does it have out of the box? Or do I need to first write my own anomaly detection plugins to get it working? It doesn't actually... I haven't actually written any plugins yet because I've been busy trying to get the stuff that you just saw get that finished. But as I said, some of the easier ones, the easier algorithms, it is literally ten lines of code. And I have that code written already. I just haven't turned it into a plugin for this. Yeah, because I understand because you said it's very easy, like ten lines of code, but probably not for everyone. Very easy to write such a thing. So I mean, and usually to get started, you need to have some examples. Absolutely. So that's definitely the idea that the system will ship with a number of different plugins that will do all the things that I've already mentioned. And this is not far off. Probably next week. It's simple stuff. I just haven't done the roundtrip yet. And it will be included and available and all this stuff. And I'll probably also be writing kind of skeleton plugins in PHP and in Python and whatever. So if someone wants to write plugins, it's really easy to just take that code and start adding simple stuff. So if I would say, check it out next week and download the example plugin, what kind of anomaly would that, would it be able to detect? Well, the first plugin that I'm going to add is one that will be very good at detecting, you know, the, this stuff. So stuff that doesn't actually have any seasonality to it, but only has a, like a trend that isn't affected by, you know, with the time of day or so on and so on. A linear trend? Not necessarily. It doesn't have to be linear. It just, it doesn't actually have to be linear. It will very quickly notice if it's going exponential or something. So that, that is no more than a week away. It is, it is very simple. The other stuff I need to, for the next one that actually can do seasonality, I'll, I have to let, I mean, add a way for it to extract data from Surveyor again, so that it can, it can work with that. But it's not far off. It's simple and it will be included in this. It's not going to be something you have to do yourself. Okay. Thank you very much. Thank you. Looking forward. Yeah. I would be very happy to write some of these anomaly detections for myself and I can, of course, go to contribute them. But you, you're mentioning, okay, these are very simple things and these exist from the sixties. Would you have any pointers like how to start, like, where, which commonly known algorithms? Sure. Mathematical models or whatever. The one I'm referring to from the sixties is, is one called Hold Winter's Double and Triple exponential smoothing. If you can apply that and get extremely good results for stuff like this. But I mean, there has been tons of research done since then and there are much more sophisticated algorithms, but this stuff is just really, really low hanging fruit. Yeah. But did you do some research on, because I did search as well for some things and I had a very hard time to find actually usable things. Okay. I'm not a mathematician, mathematicians. And so it's not so easy for me to find my way around. There's actually, there's a scientific journal dedicated to forecasting. If you, if you Google for, I think it's just called Journal of Forecasting. So every, I think it's every year. I mean, they publish a major journal. They have conventions every year about, you know, research into forecasting and stuff. And there's lots of really excellent and really quite readable articles in there that you can use to, to, to catch up on things. It is very valuable. There's a guy called Rob Hindman, which is, I think he's the editor in chief of that journal, but he's also one of the, the foremost researchers into the subject. And he has a webpage where he explains all sorts of things and he, he writes all his example code in GNU, R. I mean, the statistical tool from GNU. So it's very, very approachable if you have just a basic mathematical understanding of if you know about the tools already. So that's, so those are probably things to look at. Okay. Thank you. Thank you. So we're talking about monitoring. So we're talking about a large amount of data. How does Surveyor store this data? The data is stored in, in, in RIOC, which is one of the, one of the new fangled, no SQL data stores. It is, I mostly chose it because I, I, I very much liked the data model in it. It scales extremely well horizontally. And I don't really need a lot of fancy things from the data store. I just need a way to be able to store data and pull it out again really quickly. Anything else is just, really just fluff. And for those sorts of things, RIOC is absolutely perfect because it doesn't, that the operational model of it just like makes it so incredibly trustworthy. Because I don't want to get up at night to fix my monitoring service. Can you please repeat this name of no SQL? It's RIOC. R-I-A-K. RIOC. Thanks. So, working. So in a couple of cases, I might have, you know, thank you. Very distigrate. Many gigabytes of old logs that I could extract a whole load of running data from your how long query. He's talking about that kind of thing. Does your plugin model allow me to go back through those and historically pick, pick out when a particular prediction model would have predicted a, an issue. So I'm going to compare that to my maintenance logs and things so that I can tell would this have been a bad thing to have run? Would this have woke me up every midnight? You know, when would this have woke me up historically? I don't think, I'm sorry. I don't think I'm, something's beeping. What's going on? Oh, that's me. I'm leaning on something. Sorry. Hope it doesn't blow up. Sorry, I didn't completely catch the question. So it's about what I'm doing. So if I've got a lot of historical data, you know, logs I can process into the kind of, kind of events you're looking at, how long we request talk or something. Is, does your model for your new tool allow me to pump that and find out when it would have alerted me? Not right now. Conceptually though, is that, does that break your model or not? Well, that's a good question. I mean, the current model, only really, I mean, you're just feeding it raw values. Depending on the type of, depending on the source, you might want to change, you might want to do some pre-processing in the agent that you have that's collecting the data to kind of make it, turn it into something that is more of a, it's kind of hard to, hmm, I haven't really thought about how to put this into words. Sorry. So if we're using like terminology from RRD tool, it has like gauges, which is something like a temperature would be like a gauge. Yeah. And then it has, you know, bytes per time unit, which is a different thing because the way that you graph it is separate. And I would expect that kind of pre-processing to live in the agent. So, so they shouldn't really have to know about it. And hopefully the plugins wouldn't either. But maybe I haven't thought it through enough. Maybe that's not going to work in the real world. But that's what I'm thinking right now. Okay. Thanks. So I have all this data in Surveyor and is it intended to also do graphs of this? It will, I haven't really figured out if I'm, I try to do graphs directly in Surveyor. But because of React, because I'm storing every data point as a single key in React, generating a graph that spans a year, if you have metrics per minute, I mean, that is a lot of, you know, individual data points that have to be pulled out and it takes incredible amounts of time. And I would have to completely change the data model to make that work. And I haven't really figured out with myself if I decided not to do graphs in Surveyor because I'm, I keep insisting on wanting to use React. Or if I actually think that the best thing to do is to put it into a plugin. So there could be a plugin that just takes the data and feeds it into RLD tool and then has that generate data graphs because RLD tool is really good at it. But I don't know. I mean, using React is not set in stone. It's just what I tend to prefer right now. But it's making it very difficult to do graphs directly in Surveyor because of the time it takes to get individual data points out. Have you thought about the problem with React that it doesn't have data center or a rack awareness? It doesn't have what? Data center or rack awareness that, for example, if you have three data centers, one node in each data center, and then multiple nodes in each data center and one data center completely goes down. Right. So it may happen that you can't read or write your data anymore depending on how your formulator writes. I know that React, there's an enterprise version of React that is supposedly really good at that. So if you have multiple sites, it will actually, the distribution model is apparently sensible. But I'm not having really, I mean, I don't know of any other system that does that really well. So if I would have to attack that problem myself, I would have to basically build some kind of distribution system or reliability model on top of it. And that is just, I mean, that's too soon. Yeah. Okay. Thank you. Sure. All right. Cool. Thank you. Thank you.
have with the current state of free software are many, many other tools out there. I mean there are way more than these. There are tons of problems that I'm going to explain today apply about, well it's the third thing by the way, the you're probing the system that you're testing. So that, you make a decision as to whether you think you that passive monitoring beats the crowd of at the experience that they have because you're the same things as your real users are going to actual data point, which in this case is maybe we don't even have the data available to us, know, look at it now instead of waiting until it better. Things that look much more complicated some way. The problem is that the way that typical monitoring systems design whether And we know that, that is, that is a fine value. you know, how this, how full the discs are, how is going to be under too much load and we're over the course of a day. So we have something graph. It's very easy to see when something is, these go up and down a lot. It is, it can be very available to us that can tell us based on the We have detected an anomaly. And this works alert. In this case, you can, you know, from distribution, which I trust to be able to handle happen with a monitoring system like that, it has another every month, there's something actually have any seasonality to it, but only has