Antipatterns
Leitner, Felix von
Felix von Leitner will give a talk on Admin Antipatterns. Admin Antipatterns are Patterns of System-Administrators that you can find in almost any IT Environment that should be avoided.
 Welcome to the second day of the OnCloud conference 2018. It is my great pleasure and honor to introduce you to Felix Leitner from Code Blau. He will give a talk on admin anti-patterns and maybe a little bit to the background of Felix von Leitner. He does the FIFA's blog. It's one of the top 50 blogs in Germany and has a lot of insights, especially encoding best practices and of course on admin stuff. It's really a great pleasure to have him here. I wish you a good talk, a good day. Like yesterday, my name is John. If you need anything, don't hesitate. Just contact me. I'll be around here and help you with anything you need. Thank you very much and have a nice day. Good morning. First problem, this talk I was asked to do here, I originally did somewhere else at the German users group. It's in German. At least the slides are. I'll try to translate. However, you will lose the famous German humor. I hope you can cope. Anyway, I was a little surprised when I first heard about OnCloud and I thought I should have a slide about this. The hacking community has their own little slang words. One of them is about taking over somebody else's machine if you hack somebody or something. It's called to own it. Connotation is not good. Connotation is, oh, you hacked him so bad that you have more ownership rights than him now. I was impressed by your naming of OnCloud. Just like Oracle calling something unbreakable. Let's see. Again, I did this talk already. I did another talk similar at the 34C3, which is the annual CCC conference. It was also called anti-patterns, but it was about programming software development. Then I was asked to do another one, but with a little spin on admins for the German users group. I found out in preparing that talk that there's actually more to say about admin anti-patterns than about software development anti-patterns. Both talks are available on the Internet, so if you want to see them, maybe you already saw them. There will be some overlap if you know them already. One of my problems is that I have no idea who's in the audience here. I was like, hmm. I asked John and he says, oh, it's all admins. They will know. Let's do a little test. If you know what this does, please raise your hand. I don't need the details. I just want to know. All right. What about this? Who knows how to do this by hand if Elzov is not available? It's getting more complicated. Okay. Thinning out a bit. All right. Okay. Good. I think we'll manage. Anti-patterns. The dictionary definition anti-pattern is something you do in response to a recurring problem that you hope will solve the problem, but usually it explodes in your hands and is usually ineffective or even counterproductive. I gave the talk a little motto and I stole it from Mike Twain. I think it's one of the most profound insights in the world. If you can read and don't, you are no better than someone who can't read. Don't be happy with Wikipedia being there and you can look something up. Learn it. There's a little structure to the talk. It's like a running gag, really. It starts with some kind of problem. Some problem and then someone does something. This is how they do it. Then the effect, which is usually bad because it's about anti-patterns. This joke is really old and stale and I wouldn't have done it, but it turns out there actually is a problem factory in Java, so I couldn't walk away from that. About hacking and admins in general. I've been playing video games for a while, so my worldview was influenced by video games. This is an old one. I'll show you some character pictures. As you level up, you notice the figure gets older. It's even better with the wizards. Look at the wizards. The beard is growing. I figured, okay, that's how you know that someone is a good hacker. I mean, cut this. Who knows who that is on the left? It's Alan Cox. He used to write the IP stack in Linux 2.0. That is actually a hacker. First time I saw him, I thought he was a homeless guy who went into the wrong building. No, he's actually a hacker and he's a good hacker. My first admin job, I expected something like this. Glorious beards. I walk in the room and it turns out there's nobody with beard anywhere. All these young people and shaved old people. The oldest guy was just changing tapes. Nobody tested me. Nobody asked anything. They just gave me the root password. I was like, oh, I was a bit confused. That's when I learned that being an admin is not about knowing all these skills that you need to be a good admin. Admin is about being young and being happy with small payment. You can deduct something about admins you see in the wild. There's two kinds of them. First one wants to solve problems and the other one wants to administrate. There are some admins who don't actually want to see problems go away. They're perfectly happy with their being problems around because otherwise they wouldn't be needed anymore. That's not good. Two kinds of admin. The first one is the new guy and he wants to see things work smoothly and he wants to learn new stuff. The other one is basically staying in the way and saying don't touch anything. You can see both admins causing problems every now and then. I've been trying to distribute the anti-prediction between them. A usual problem in administration is something should be automated. Something you do all the time or you don't want to do all the time. There's two things that can go wrong. First one is you automate too late to the game. This is easy enough to do manually. We don't really need to automate this. You do it manually all these years with all the typos and wasted time. That's a very common thing you see. The other one is just as bad. If you go let's do this now and then it turns out nobody actually needs that. So time is wasted. I mean most of the wisdom of the world you can find on XKCD so I brought three of them. This is the first one. Pass me the salt and he's like oh I'm working on a system to pass arbitrary condiments. This is a common problem. Don't fall for this. What about the common case of automation? You go okay hold my beer I'm automating this and you go. Everything goes away for six weeks. It's not heard of again. And comes back not having solved the problem. But on the way found 23 other problems and solved those. So I mean most of these are a little I mean they look like oh who's that stupid but you see the I mean this is from experience I've seen all of these in actual companies. Some administration is usually done by underpaid young people. So the management doesn't really enforce or expect quality as much as they should. And I think it's a co-dependency both sides are wrong. You should hold yourself to a high standard. So if the admin comes back and solves unrelated problems usually the solution is based on some prehistoric stuff that's only available on this old Vax that has not been patched. It does more harm than good. And the second next case right you expect it like the upper curve you go oh I'm writing the solution that's a short short time and then the solution takes all the problem away and I have all this free time but that's not how it works. And we know it doesn't work like that before we do it. Nobody should be surprised by this. So if you do it like that usually it pays off eventually but everyone will be long dead by then. So I'm trying to illustrate this here with a well known thing from the future. So another effect is that while you're trying to solve the problem nobody is touching it because it's being worked on so it stays there and sometimes it lays eggs. So yeah I mean this isn't good. Even if you automate something you still need to solve the problem. But wait for the automation because it may never succeed. Usually you have this problem that because nobody works on the other problem because it's being solved by someone somewhere that business processes get stuck. Right I mean this is all you probably have seen this before in administration. So next problem after automation is that you have software and it sucks. Usually software you run has known problems. You have an old version or there is a patch available. So you know the software is bad. So the obvious idea is you go to the vendor who sold you the software get some patches. So typically you don't send your top personnel to the vendor you send like the intern and the intern goes to the vendor and doesn't find anything. This is on purpose by the way because the vendor doesn't want new customers to see how bad things are. So they hide the security stuff. Some vendors go so far as to hide it behind a paywall or you have to sign in in the A database vendors have started to put in their terms of service that you cannot publish benchmarks about the database which I find really important. So I mean this is all things are done right and you send not your best guy but the intern right so the intern doesn't find anything and if the intern finds something usually the description of the patch is useless. That's on purpose too. The official excuse is that you don't want to give away to the attacker how to attack the system. But that's I don't think that's the truth. Vendors just you know they have all the spoiler plates so nobody actually sees how bad things are. So what's the intern supposed to do right there are some security mailing lists you can go to. These are old school mailing lists there are other ones but these mailing lists are really high volume. If you subscribe to them that's basically all you do right the intern goes away and can't do anything else because there's this flood of security announcements. And it turns out usually because it's the intern doesn't actually know what software you have. So he sees all these announcements and you know gets a few of them that he recognizes and the rest is just wasted. So all this stuff happens in the wild I mean it makes you wonder how things work at all. So but that's assumed the intern is very bright and actually finds a patch that is applicable. Finds a patch wants to put it on the system obviously. And then the other admin comes and blocks him. The old guy and the old guy says oh we need to wait a few weeks to see if the patch is good. Who here has seen that happen? Oh that's surprising if you I think you're lying to me. So this is very common. Oh no we can't put out this patch it might crash. And the effect is obviously that all systems in the company have all the known security problems of the last months. No surprise here. And this is all self inflicted. It's like this huge shotgun that you point to your foot you know. There is no need for any of this pain. So the next problem is we have all this malware in the company. What do we do? Well the obvious solution is to get some snake oil products like antiviruses you know network intrusion detection whatever Zeme. I have a nice, whoops I have a nice illustration here. Usually you can easily detect snake oil by how bullshit the marketing is. So I collected a field that I found very nice. The predictive security cloud. How about some cognitive SOC whatever that is. Oh and blockchain of course. So in practice I've actually seen that at the gig. They gave us a two core system and the first core was busy running semantic snake oil. And the second core was busy running McAfee and they killed each other all the time. So the second core was logging all the attacks it prevented. And if you wanted to open a tab in IE which is the only browser they gave me it took like two minutes. System was just busy. Not just CPU was busy. Storage was busy too. Storage was busy because they put each other in quarantine all the time. And then reinstalled. So it was really bad. And you would think that someone notices that before they give you the PC for work. But this happens. Another thing that happens is giant monitoring systems. Sometimes they actually really monitor you. Like what keystrokes you do and the web pages you visit depending on your jurisdiction. But in general the common thing you can see is that all these monitoring usually ends up somewhere. Nobody is reading that. So usually when I do a pen test for a company I ask them if they saw me. And I'm not trying to hide. That's not the point. I'm not being paid to hide. I'm being paid to try out problems. And I've been seen once. Everyone has all this huge monitoring stuff but nobody ever reads the logs. So I call those heights lifter which is like a heating appliance. They cause all this heat and it's nice and warm. But that's about it. So we talked about patches. Sometimes there's a service pack. This is a little older problem. The older admins among you will know this. And there used to be this, this of course you want to install it, but there used to be this like mystical idea. No, you can only take the even numbers or only the odd numbers and the other ones are bad which is the same thing as with the patches obviously. So same thing as before. You get all this malware and you can't get rid of it. So this patch thing is really important to me because it's so easy to see how not to do it. So this is a common thing you hear. We only install patches that we really need. Who has heard that before? Most big companies have their own department to find out which patches are needed and which ones aren't. And this sounds like a good idea but it's not. So first of all, you don't know. You have no clue. How would you know which patches are needed and which aren't? The systems are so complex these days. Even if you don't use a component, it might still be used by some other component. So you don't know, right? Secondly I've seen this firsthand. If I go to a vendor and to a source code audit and find 100 bugs, they will usually fix a few of them at least. But if you look at the next patch, none of them will be mentioned. Why would they mention bugs that nobody outside knows except me and I signed in NDA? So if you get a patch from a vendor, you can be sure that there's at least a dozen really horrible things it fixes that they don't want to tell you about because it would make them look bad. Install all patches all the time, right? The third one, this I think is the important argument for big companies. The patches are tested by the vendor but only under the assumption that you put in all the other patches. So if you skip patches, then you're on your own. How would the vendor support you if you have like this fractally different landscape because you skip this patch and this patch? You don't know. And neither does the vendor. They don't even have a machine with your exact patch level. So always install all the patches. If you forget everything else about this keynote, remember this. Always install all patches. So sometimes a new software version is coming out. It's a related issue to the patches, right? So you want to install it and then the old guy goes, no, no, it's a.0. We don't trust.0. We are waiting for.1. And you know, GCC is an important open source package. They just changed the versioning scheme so there is no.0. They start with.1 because nobody installs.0 anyway. So I mean, you can predict how this is going to continue, right? This is also a common problem in practice. You see all these really old installations and nobody wants to touch them. So let's update it. Right? How do you do it? Well, you put in all the patches. Well, some of these patches aren't deterministic, right? So the effect is that you have three old systems where you installed all the patches, but they're all subtly different, right? So this problem is common enough that these days it's one of the main arguments for containers like Docker. You don't install the patch, you install the container and then you know your installation is the same as it should be. And this is also, I mean, there's no reason why it should have been like that. And I'm assuming that Docker will have a similar problem at some point and then we'll have the next great idea. But all of these are self-inflicted. Okay, so let's update something. Different problem. And then you sometimes see that the admin doesn't want to update, right? The admin says, no, no, we can't patch because we will lose support if we patch. And at first I thought they're making fun of me, but there actually is environments where you have like an old Solaris and you have some kind of Oracle database on it or whatever, you know, substitute. And if you patch the Solaris, the Oracle support will tell you you're not supported anymore. This actually happens. I've seen this happen with telco equipment, like the SMS gateway at your telco. And what they do is they install three firewalls before it, as if that, you know, does anything. But you know, this is what happens, right? So we can't patch it, we lose support. And then you often hear, well, it's not old, it's stable. Yes, it's old. Stable is bullshit. It's not stable. It's known bad. We know it's bad. There is an update, you know? It's not, this continues to amaze me. You see all these really ancient machines and you go, who's using that? They go, oh, nobody's using that, you know, it's too old, all the commands don't work. Well, yeah, you get the strangest excuses like, well, we can't turn off SSL version 2 because we're talking to this, you know, mythical machine. We don't even know if it's there anymore. A customer of ours told us we can't turn off SSL 2. Yes, you can. You have to. It's broken. So it's not stable. It's known bad. I would like to change the terminology here. It's not stable. It's known bad. You have to understand that new releases don't fall from the sky, you know? Somebody is releasing them. They have a reason. You don't release something because you have free time. You release something because, you know, the old version is bad and you have a better version. It's easy. Yeah, this one is starting to gain ground. They go like, oh, I can't work here. The admin is too strict, you know? All this stuff is configured too close. I can't install software. I need my Solitaire game. And then they go, oh, I want to bring my own device. And you know, this is less of a problem now than it used to be. It used to be this, you know? Your tablet isn't part of our device management. You can't bring it. You know, so what happens is that the CEO is forwarding his mail to Yahoo and reads it with the tablet and the guest Wi-Fi. You know, I've seen this dozens of times. This is what happens if the admin tries to lock things down. It doesn't work. You have to work with your people, not against them. So this is like a meta problem. You know, oh, it doesn't work. It's horrible and you have to diagnose it and the first idea is, ah, the product is shit. And then, ah, the vendor is shit. Notice how the exclamation marks grow. And then, oh, it's a compiler bug. You know, the excuses get more and more absurd. No, no, it's a kernel bug. This is like the, the, the, the hunters have to usually tell these stories about these mythical beasts they, they slew in the wild. Same thing with admins. And usually when, you know, the, the, the someone looks over his shoulder and sees, well, you, you missed the semicolon here. Don't go in that rabbit hole and go, oh, it's the compiler. No, it's probably not. I mean, there are compiler bugs, but your problem is probably not a compiler bug. Most of these are really trivial if you look at them, so it's easy to make fun of them. But they will happen to you at some point. So this is a typical problem for admins. We want to know which of our hosts are online. We have this data center with all these machines. We want to know which ones are working, right? So let's do some monitoring, obviously. So, hmm, you need to find out which, which hosts are there. So you need to talk to them. Where do you get the list of hosts? And I've seen this a couple of times. They get the list from pinging everyone and seeing who answers. So you ping all the machines you know are there and are answering. And surprisingly, everyone is there, right? So always, always look closely where you get your data because stuff like that happens all the time. So what do we do when something breaks? Typical admin problem, well, we have a backup, right? Who has a backup? Okay, okay. Yeah, I've seen this a couple of times. Oh, you know, storage is expensive. Let's put it in the cloud. Or you know, maybe you have this deal with, with whoever and they give you free cloud storage and you say, oh, we have a few terabytes of cloud storage. Let's put the backup there. So you put the tar ball in the cloud and it contains all the config files with the credentials and the personal data from users, the TLS keys, all of this has happened, you know. The private keys, obviously. The descriptor recovery keys, all of that shit is in the cloud. And it usually gets found because, you know, there are search engines for that. One of them is called Shodan. People find stuff on Shodan daily. It's so much that nobody actually realizes usually if something gets found because it just gets drowned out in the flood. There's so much stuff in the cloud. So don't put your stuff in the cloud. This is a classic in the banking environment. Oh, telephone calls are so expensive. Let's use voice over IP. And then they go, well, we have the switch and there's a few free ports. And then it turns out that the Cisco voice over IP phones want a trunk port. So if you pull out the phone and pull in your network, your laptop, you can go into any VLAN. And it's a different department. See, the people who bought the phones are a different department than the people who set up the network. So nobody realizes. This used to work in all of the banks. So what if we need some integration? We have all these components and we want them to talk to each other. I know scripting languages. Let's do scripting. And usually some admin hacks something, you know, the idea you use like this, another XKCD, like I know regular expressions. And usually it works like that too. You have like amazed people on the ground and the script is like, ooh. But you just want to solve the problem. There's no quality assurance, no quality testing. Usually there's not even a version control system, there's no ticketing. It's just a small integration script. So in practice, the quality of integration is much lower than the quality of the products, the components you're trying to integrate. I don't know why this keeps happening because it makes everyone look bad. Usually you hear all these complaints and companies about how bad the components are and it's not actually the components, it's integration that your own people did. But you know, the admin is like, well, you know, I logged in and it worked. So that's it. Usually admins have enough on their plate that they lose interest immediately and start doing other stuff. Right? So I've seen a few companies actually, I have an illustration for that. I've seen a few companies where the admin did the integration in some language they wanted to learn. Not something they knew, something they wanted to learn. Right? Okay, this is another classic. Oh, it's very slow. We need to find out why it's slow. So let's deploy some agents. And this is my favorite picture in the world. If you look closely, you see it's completely useless. Things like connecting, loading, please wait, searching. It's like zero content and usually monitoring systems are like that. They give you like, woo, Starship Enterprise and there's no content. But all the agents that deploy to take the measurements really slow things down more. So yeah, desperation, swapping from monitoring is a thing. Actually everything is even slower than. This one really makes me sad because I heard it and I thought it's like a satire or something. So it was a company and they outsourced their data center to some other company and they told them you have to keep all the machines. But we want you to guarantee us an SLA and it has to be cheaper than what we're doing now. And you have to keep the people. So how is the company supposed to do that? They skimmed on support, obviously. No maintenance. Everything broke down all the time and this happens all the time. If you outsource something with unrealistic goals, then they're not going to get met. So the next step is, well, we go on the cloud and I was like, why would you go on the cloud? The previous deal already fell apart. Why would the cloud be better? And then the obvious counter argument is if you put your data in the cloud, it's not your data anymore, obviously. But I got this excuse. They said, oh, it can't get any worse than it is now. And AWS at least reacts to tickets we open. And I thought, how is, what? Yeah, they don't help me, but at least I get a form mail, form letter. We take your problems very seriously. Yeah, okay, this is another one. Okay, I'm just a little bit self deprecating. Usually you see companies at some point go, okay, we have so many security issues. We need a security team. Right? And they go, let's get the best of the best. 24 seven operations, we call it our own cert. And you get people like this. And then everyone else in the company goes, well, it's not my problem anymore. And they keep on causing more problems because security, someone else is dealing with that now. Right? Okay, we're back to updates. This is the perennial problem. So I'm trying to illustrate something that, you know, sometimes I'm at a customer and I get to see what someone is typing. But I can't say anything. Right? And you see them do something and go, right? So again, we're monitoring, right? And the guy goes, okay, easy. We need to deploy something. And then, hmm, where's the host list coming from? Right? Oh, it's probably in the wiki. At this point, I usually go, and they go, oh, let's go to the wiki. And the wiki is up again. And then the host list is from 1997, historical artifact. So okay, let's ping everything. Let's ping them all. And they go, okay. Pinging all of them. And then, you know, of course, command on phone because it's a BSD, whatever. Okay, let's do it with a shell. And this is taking forever because every ping runs into a timeout for all the systems that are not answering, right? So it's like, oh, snore. Get a coffee. At some point, I was so annoyed that I wrote a little ping that can do this in parallel. That's how bad this problem is. And it happens all the time everywhere. It's open source. You can have it. So I showed this to a customer and he was like, oh, that's awesome. Let's extract the IP numbers. And I kid you not. You know, you could have waited for the other ping in that time. It's done. It's done. Okay, so I showed them regular expressions, right? And they go, oh, I can do this. Right? So, you know, have you heard of AWK? Oh, yeah. That's awesome. I heard this once and I couldn't contain my laughter. I was like, oh, of course you could use HP OpenView. Let's boot the console. Sure. Anyway, so this really gets on my nerves because I think you owe it to yourself to know your tools. It's like if you get somebody to fix your kitchen and the guy goes, oh, let's open the manual, find out what to do, I mean, it's like what? No. Right? So I like to call this the monster truck mode because usually you have a problem but it's not here. It's there. And on the way there, you see all these other things that don't work. And you go, okay, I don't really want to deal with this. I want to deal with that. And you fix none of them properly, if at all. Right? So it's like you have this monster truck and you drive over everything on the way. And the result is obviously that all the broken stuff stays broken. Right? This is a real problem in big installations. Everything is broken. Everybody knows everything is broken but nobody is fixing anything. They only fix the stuff that is halting production now and the rest gets worse. So everything stays broken. Right? The result is something like this. You're like, not my job, you know? Let someone else fix that. Yeah. This is also very common. It's like never touch a running system. Yeah, you do touch a running system. That's your job. That's what you're being paid for to fix the running system. So it continues running. And you see all these temporary hacks and fixes. And nobody ever removes something that is not needed anymore but you just, oh, let's add some more. Also, there's always time pressure. There's never a moment where you go, oh, I have time now. I can fix this properly. No, that doesn't happen. If you wait for that, it's not going to happen. Right? Problems, problems just stay there. And sometimes they get worse. They usually get worse in a moment when you don't really need it. This is one of my favorite pictures. And I go, oh, I think we missed a button there. Yeah, I already mentioned this. I mean, I'm not here to teach you how to use the born shell, but you really should learn that. It's like the least you can do. Like literally, it's literally the least you can do as an admin is to learn your tools. Okay, I'm not going through all of these. The last one has probably bitten everyone in this room at some point. Oh, there's a space somewhere in one of the five names. It doesn't work. Yeah, learn your tools. Really, I mean, there's no excuse. It's not like you're a physician. You're the admin. That's also really annoying to me. If you go somewhere and you see, well, you could have fixed it in five minutes, or you can manage it in half an hour, and they go, let's manage it. And then it still crashes, but it restarts. And for some people, that's more attractive than fixing the problem. I really don't understand that. I mean, you see that all around you, even on Google, sometimes the query gets stuck. And that's because some backend crashed somewhere, and you know, it's being restarted, but still, it's not working. Okay, the next one is quite common, too, when they go, well, I could do this, but it's not my job. We have a department for that. Usually, it's like the firewall department. It's not my job. So we call the firewall department, and it takes about six months. And then the rule is in, and you can't get it out again. So this is high comedy. If you ever have spare time and you know someone working in the firewall department, let them show you the list of rules. Half of them will be useless, not needed anymore. That machine doesn't even exist anymore. Same thing for logging. Like, you have three teams. They have three different log destinations, and they don't talk to each other. And sometimes they even cite data protection laws to not talk to each other, which I find really, really awesome. It's like, that guy is the enemy. I'm not giving him anything. Yeah, this is more development thing, but I'm still listening to it here because sometimes people who move in the cloud have the same kind of issue. So if you develop software these days, you upload your sources to GitHub, and then the rest is done by some service somewhere else in some other cloud, and you don't even know who's really doing it. And then people deploy the result into some app store without having seen it. Like we have no reason to trust any of this. And when you talk to people about this, they go, well, you know, I trust this more than our own people because they are fucking stupid. Well, hire good people then. Okay, this is a bit esoteric. I'm going to skip this. I'm showing you the picture because it's nice. Sometimes in companies, they close the USB ports and you can't log in remotely. You have to go through three jump posts. And to log in in the jump posts, you need to go through some common password broker tool. And then at the end system, everyone is always logging in as root. So all that stuff is just useless. It's just, I don't know why people do it. One of the highlights of my career was when I was being asked to build a monitoring system for a ZIM. So the M in ZIM stands for monitoring. The guy bought a monitoring system and was, I don't trust that thing. We need a monitoring system for the monitoring system. Yeah, I've actually seen this happen too. So the groovy thing is not theoretically, right? So the admin needs to do something and it takes a language he wants to learn, not a language he knows. Maybe it is a language he knows, but usually it's something they think would look good on the resume. Right? And then they go, okay, I'll fix this in small talk. And then obviously instantly that fix becomes part of the critical infrastructure and the admin at some point leaves. And then you have a small talk script that nobody understands. Actually I've seen in the window source tree binaries that someone built at some time and they don't have the sources for them. But it's part of the build process. So this happens to everyone. It's not, I'm not talking down on small incompetent companies. This happens all over the place. Usually I like to think of this, you know, if you have all these old people who understand stuff that if they die, it's gone. These are the Navajo code talkers from the second world war. Yeah, also that it's like you break something, admit it, own it, own it. Don't go Han Solo. You need something called blameless post mortem. It needs to be possible to discuss what caused the problem without people leaving the room because they think they're being singled out. Right? You have to establish that it's a cultural thing. This one really, really pisses me off too when you see all no, no, we're not rewriting that it's still good. Right? Everything, the whole system is really bad. You can't even boot it without, you know, error messages. And they go, oh, let's continue. It still works. See, I can still move. It's not that bad. So usually when I have this slide, people think of windows and there's a point, right? There's a point about looking down on windows, all this crap that the internet is trying to give you. But it's not a windows issue. It's a generic issue. People install stuff that I don't really understand. Notice the part when he says, well, every time I remove it after 24 to 48 hours, it's back. Yeah, it's the invisible hand of the market. Bullshit. All right. I'm done with most of the content. On the end of my talks, I usually have a few slides with free advice. Take it or leave it. So if you're management, you need to be happy if someone finds a bug. Don't punish anyone. Even the guy who wrote the code that's buggy. Don't punish them. Give them an opportunity to learn. Show them the bug. Don't have anyone else fix it. Have that guy fix it. And make sure you have a feedback system in your company. I've seen places where the security team fixes bugs and the developer doesn't even realize you made a mistake. Obviously, that's the same mistake again. Nobody told them. You need feedback. You need a culture of being able to own your bugs and problems. So if you're management and you're planning a system, don't tell them how to do it. You don't know. If it's obvious enough that you can tell, then you're no help. If it's not obvious, you're probably making it worse. Trust your people. If you're an admin, this is the goal. Never have emergencies. Automate everything until you have free time all day and then learn new stuff. That's your goal as admin. If you're doing firefighting all day, there will be never any end to that. Some people still do it because they think they're not needed anymore if they fix everything. That's not how it works. So this is for anyone. Don't look to others. Look to yourself. If you orient your own self-worth on others, it's going to fail. There's no good outcome. So if the other guy is about the same, then why would you put in more work? It's good, right? No motivation. So if the other guy is worse than you, then you're ahead. No need to put in more work. If the other guy is better than you, you go, oh, I have no chance. So you give up, right? Never look to others, look to yourself in the past. Be better than yesterday. That's it. If you have any questions, I think I've already exhausted my time, but you can email me if you want to. And I'll be here for a few more, like half an hour or something at least. Thank you. Thank you.
 Welcome to the second day of the OnCloud conference 2018.  It is my great pleasure and honor to introduce you to Felix Leitner from Code Blau.  He will give a talk on admin anti-patterns and maybe a little bit to the background of  Felix von Leitner.  He does the FIFA's blog.  It's one of the top 50 blogs in Germany and has a lot of insights, especially encoding  best practices and of course on admin stuff.  It's really a great pleasure to have him here.  I wish you a good talk, a good day.  Like yesterday, my name is John.  If you need anything, don't hesitate.  Just contact me.  I'll be around here and help you with anything you need.  Thank you very much and have a nice day.  Good morning.  First problem, this talk I was asked to do here, I originally did somewhere else at the  German users group.  It's in German.  At least the slides are.  I'll try to translate.  However, you will lose the famous German humor.  I hope you can cope.  Anyway, I was a little surprised when I first heard about OnCloud and I thought I should  have a slide about this.  The hacking community has their own little slang words.  One of them is about taking over somebody else's machine if you hack somebody or something.  It's called to own it.  Connotation is not good.  Connotation is, oh, you hacked him so bad that you have more ownership rights than him  now.  I was impressed by your naming of OnCloud.  Just like Oracle calling something unbreakable.  Let's see.  Again, I did this talk already.  I did another talk similar at the 34C3, which is the annual CCC conference.  It was also called anti-patterns, but it was about programming software development.  Then I was asked to do another one, but with a little spin on admins for the German users  group.  I found out in preparing that talk that there's actually more to say about admin anti-patterns  than about software development anti-patterns.  Both talks are available on the Internet, so if you want to see them, maybe you already  saw them.  There will be some overlap if you know them already.  One of my problems is that I have no idea who's in the audience here.  I was like, hmm.  I asked John and he says, oh, it's all admins.  They will know.  Let's do a little test.  If you know what this does, please raise your hand.  I don't need the details.  I just want to know.  All right.  What about this?  Who knows how to do this by hand if Elzov is not available?  It's getting more complicated.  Okay.  Thinning out a bit.  All right.  Okay.  Good.  I think we'll manage.  Anti-patterns.  The dictionary definition anti-pattern is something you do in response to a recurring problem  that you hope will solve the problem, but usually it explodes in your hands and is usually  ineffective or even counterproductive.  I gave the talk a little motto and I stole it from Mike Twain.  I think it's one of the most profound insights in the world.  If you can read and don't, you are no better than someone who can't read.  Don't be happy with Wikipedia being there and you can look something up.  Learn it.  There's a little structure to the talk.  It's like a running gag, really.  It starts with some kind of problem.  Some problem and then someone does something.  This is how they do it.  Then the effect, which is usually bad because it's about anti-patterns.  This joke is really old and stale and I wouldn't have done it, but it turns out there actually  is a problem factory in Java, so I couldn't walk away from that.  About hacking and admins in general.  I've been playing video games for a while, so my worldview was influenced by video games.  This is an old one.  I'll show you some character pictures.  As you level up, you notice the figure gets older.  It's even better with the wizards.  Look at the wizards.  The beard is growing.  I figured, okay, that's how you know that someone is a good hacker.  I mean, cut this.  Who knows who that is on the left?  It's Alan Cox.  He used to write the IP stack in Linux 2.0.  That is actually a hacker.  First time I saw him, I thought he was a homeless guy who went into the wrong building.  No, he's actually a hacker and he's a good hacker.  My first admin job, I expected something like this.  Glorious beards.  I walk in the room and it turns out there's nobody with beard anywhere.  All these young people and shaved old people.  The oldest guy was just changing tapes.  Nobody tested me.  Nobody asked anything.  They just gave me the root password.  I was like, oh, I was a bit confused.  That's when I learned that being an admin is not about knowing all these skills that  you need to be a good admin.  Admin is about being young and being happy with small payment.  You can deduct something about admins you see in the wild.  There's two kinds of them.  First one wants to solve problems and the other one wants to administrate.  There are some admins who don't actually want to see problems go away.  They're perfectly happy with their being problems around because otherwise they wouldn't be  needed anymore.  That's not good.  Two kinds of admin.  The first one is the new guy and he wants to see things work smoothly and he wants to  learn new stuff.  The other one is basically staying in the way and saying don't touch anything.  You can see both admins causing problems every now and then.  I've been trying to distribute the anti-prediction between them.  A usual problem in administration is something should be automated.  Something you do all the time or you don't want to do all the time.  There's two things that can go wrong.  First one is you automate too late to the game.  This is easy enough to do manually.  We don't really need to automate this.  You do it manually all these years with all the typos and wasted time.  That's a very common thing you see.  The other one is just as bad.  If you go let's do this now and then it turns out nobody actually needs that.  So time is wasted.  I mean most of the wisdom of the world you can find on XKCD so I brought three of them.  This is the first one.  Pass me the salt and he's like oh I'm working on a system to pass arbitrary condiments.  This is a common problem.  Don't fall for this.  What about the common case of automation?  You go okay hold my beer I'm automating this and you go.  Everything goes away for six weeks.  It's not heard of again.  And comes back not having solved the problem.  But on the way found 23 other problems and solved those.  So I mean most of these are a little I mean they look like oh who's that stupid but you  see the I mean this is from experience I've seen all of these in actual companies.  Some administration is usually done by underpaid young people.  So the management doesn't really enforce or expect quality as much as they should.  And I think it's a co-dependency both sides are wrong.  You should hold yourself to a high standard.  So if the admin comes back and solves unrelated problems usually the solution is based on  some prehistoric stuff that's only available on this old Vax that has not been patched.  It does more harm than good.  And the second next case right you expect it like the upper curve you go oh I'm writing  the solution that's a short short time and then the solution takes all the problem away  and I have all this free time but that's not how it works.  And we know it doesn't work like that before we do it.  Nobody should be surprised by this.  So if you do it like that usually it pays off eventually but everyone will be long dead  by then.  So I'm trying to illustrate this here with a well known thing from the future.  So another effect is that while you're trying to solve the problem nobody is touching it  because it's being worked on so it stays there and sometimes it lays eggs.  So yeah I mean this isn't good.  Even if you automate something you still need to solve the problem.  But wait for the automation because it may never succeed.  Usually you have this problem that because nobody works on the other problem because  it's being solved by someone somewhere that business processes get stuck.  Right I mean this is all you probably have seen this before in administration.  So next problem after automation is that you have software and it sucks.  Usually software you run has known problems.  You have an old version or there is a patch available.  So you know the software is bad.  So the obvious idea is you go to the vendor who sold you the software get some patches.  So typically you don't send your top personnel to the vendor you send like the intern and  the intern goes to the vendor and doesn't find anything.  This is on purpose by the way because the vendor doesn't want new customers to see how  bad things are.  So they hide the security stuff.  Some vendors go so far as to hide it behind a paywall or you have to sign in in the A  database vendors have started to put in their terms of service that you cannot publish benchmarks  about the database which I find really important.  So I mean this is all things are done right and you send not your best guy but the intern  right so the intern doesn't find anything and if the intern finds something usually  the description of the patch is useless.  That's on purpose too.  The official excuse is that you don't want to give away to the attacker how to attack  the system.  But that's I don't think that's the truth.  Vendors just you know they have all the spoiler plates so nobody actually sees how bad things  are.  So what's the intern supposed to do right there are some security mailing lists you can  go to.  These are old school mailing lists there are other ones but these mailing lists are really  high volume.  If you subscribe to them that's basically all you do right the intern goes away and can't  do anything else because there's this flood of security announcements.  And it turns out usually because it's the intern doesn't actually know what software  you have.  So he sees all these announcements and you know gets a few of them that he recognizes  and the rest is just wasted.  So all this stuff happens in the wild I mean it makes you wonder how things work at all.  So but that's assumed the intern is very bright and actually finds a patch that is applicable.  Finds a patch wants to put it on the system obviously.  And then the other admin comes and blocks him.  The old guy and the old guy says oh we need to wait a few weeks to see if the patch is  good.  Who here has seen that happen?  Oh that's surprising if you I think you're lying to me.  So this is very common.  Oh no we can't put out this patch it might crash.  And the effect is obviously that all systems in the company have all the known security  problems of the last months.  No surprise here.  And this is all self inflicted.  It's like this huge shotgun that you point to your foot you know.  There is no need for any of this pain.  So the next problem is we have all this malware in the company.  What do we do?  Well the obvious solution is to get some snake oil products like antiviruses you know network  intrusion detection whatever Zeme.  I have a nice, whoops I have a nice illustration here.  Usually you can easily detect snake oil by how bullshit the marketing is.  So I collected a field that I found very nice.  The predictive security cloud.  How about some cognitive SOC whatever that is.  Oh and blockchain of course.  So in practice I've actually seen that at the gig.  They gave us a two core system and the first core was busy running semantic snake oil.  And the second core was busy running McAfee and they killed each other all the time.  So the second core was logging all the attacks it prevented.  And if you wanted to open a tab in IE which is the only browser they gave me it took like  two minutes.  System was just busy.  Not just CPU was busy.  Storage was busy too.  Storage was busy because they put each other in quarantine all the time.  And then reinstalled.  So it was really bad.  And you would think that someone notices that before they give you the PC for work.  But this happens.  Another thing that happens is giant monitoring systems.  Sometimes they actually really monitor you.  Like what keystrokes you do and the web pages you visit depending on your jurisdiction.  But in general the common thing you can see is that all these monitoring usually ends up  somewhere.  Nobody is reading that.  So usually when I do a pen test for a company I ask them if they saw me.  And I'm not trying to hide.  That's not the point.  I'm not being paid to hide.  I'm being paid to try out problems.  And I've been seen once.  Everyone has all this huge monitoring stuff but nobody ever reads the logs.  So I call those heights lifter which is like a heating appliance.  They cause all this heat and it's nice and warm.  But that's about it.  So we talked about patches.  Sometimes there's a service pack.  This is a little older problem.  The older admins among you will know this.  And there used to be this, this of course you want to install it, but there used to be  this like mystical idea.  No, you can only take the even numbers or only the odd numbers and the other ones are  bad which is the same thing as with the patches obviously.  So same thing as before.  You get all this malware and you can't get rid of it.  So this patch thing is really important to me because it's so easy to see how not to  do it.  So this is a common thing you hear.  We only install patches that we really need.  Who has heard that before?  Most big companies have their own department to find out which patches are needed and which  ones aren't.  And this sounds like a good idea but it's not.  So first of all, you don't know.  You have no clue.  How would you know which patches are needed and which aren't?  The systems are so complex these days.  Even if you don't use a component, it might still be used by some other component.  So you don't know, right?  Secondly I've seen this firsthand.  If I go to a vendor and to a source code audit and find 100 bugs, they will usually fix a  few of them at least.  But if you look at the next patch, none of them will be mentioned.  Why would they mention bugs that nobody outside knows except me and I signed in NDA?  So if you get a patch from a vendor, you can be sure that there's at least a dozen really  horrible things it fixes that they don't want to tell you about because it would make them  look bad.  Install all patches all the time, right?  The third one, this I think is the important argument for big companies.  The patches are tested by the vendor but only under the assumption that you put in all the  other patches.  So if you skip patches, then you're on your own.  How would the vendor support you if you have like this fractally different landscape because  you skip this patch and this patch?  You don't know.  And neither does the vendor.  They don't even have a machine with your exact patch level.  So always install all the patches.  If you forget everything else about this keynote, remember this.  Always install all patches.  So sometimes a new software version is coming out.  It's a related issue to the patches, right?  So you want to install it and then the old guy goes, no, no, it's a.0.  We don't trust.0.  We are waiting for.1.  And you know, GCC is an important open source package.  They just changed the versioning scheme so there is no.0.  They start with.1 because nobody installs.0 anyway.  So I mean, you can predict how this is going to continue, right?  This is also a common problem in practice.  You see all these really old installations and nobody wants to touch them.  So let's update it.  Right?  How do you do it?  Well, you put in all the patches.  Well, some of these patches aren't deterministic, right?  So the effect is that you have three old systems where you installed all the patches, but they're  all subtly different, right?  So this problem is common enough that these days it's one of the main arguments for containers  like Docker.  You don't install the patch, you install the container and then you know your installation  is the same as it should be.  And this is also, I mean, there's no reason why it should have been like that.  And I'm assuming that Docker will have a similar problem at some point and then we'll have  the next great idea.  But all of these are self-inflicted.  Okay, so let's update something.  Different problem.  And then you sometimes see that the admin doesn't want to update, right?  The admin says, no, no, we can't patch because we will lose support if we patch.  And at first I thought they're making fun of me, but there actually is environments where  you have like an old Solaris and you have some kind of Oracle database on it or whatever,  you know, substitute.  And if you patch the Solaris, the Oracle support will tell you you're not supported anymore.  This actually happens.  I've seen this happen with telco equipment, like the SMS gateway at your telco.  And what they do is they install three firewalls before it, as if that, you know, does anything.  But you know, this is what happens, right?  So we can't patch it, we lose support.  And then you often hear, well, it's not old, it's stable.  Yes, it's old.  Stable is bullshit.  It's not stable.  It's known bad.  We know it's bad.  There is an update, you know?  It's not, this continues to amaze me.  You see all these really ancient machines and you go, who's using that?  They go, oh, nobody's using that, you know, it's too old, all the commands don't work.  Well, yeah, you get the strangest excuses like, well, we can't turn off SSL version 2 because  we're talking to this, you know, mythical machine.  We don't even know if it's there anymore.  A customer of ours told us we can't turn off SSL 2.  Yes, you can.  You have to.  It's broken.  So it's not stable.  It's known bad.  I would like to change the terminology here.  It's not stable.  It's known bad.  You have to understand that new releases don't fall from the sky, you know?  Somebody is releasing them.  They have a reason.  You don't release something because you have free time.  You release something because, you know, the old version is bad and you have a better version.  It's easy.  Yeah, this one is starting to gain ground.  They go like, oh, I can't work here.  The admin is too strict, you know?  All this stuff is configured too close.  I can't install software.  I need my Solitaire game.  And then they go, oh, I want to bring my own device.  And you know, this is less of a problem now than it used to be.  It used to be this, you know?  Your tablet isn't part of our device management.  You can't bring it.  You know, so what happens is that the CEO is forwarding his mail to Yahoo and reads it  with the tablet and the guest Wi-Fi.  You know, I've seen this dozens of times.  This is what happens if the admin tries to lock things down.  It doesn't work.  You have to work with your people, not against them.  So this is like a meta problem.  You know, oh, it doesn't work.  It's horrible and you have to diagnose it and the first idea is, ah, the product is  shit.  And then, ah, the vendor is shit.  Notice how the exclamation marks grow.  And then, oh, it's a compiler bug.  You know, the excuses get more and more absurd.  No, no, it's a kernel bug.  This is like the, the, the, the hunters have to usually tell these stories about these  mythical beasts they, they slew in the wild.  Same thing with admins.  And usually when, you know, the, the, the someone looks over his shoulder and sees, well, you,  you missed the semicolon here.  Don't go in that rabbit hole and go, oh, it's the compiler.  No, it's probably not.  I mean, there are compiler bugs, but your problem is probably not a compiler bug.  Most of these are really trivial if you look at them, so it's easy to make fun of them.  But they will happen to you at some point.  So this is a typical problem for admins.  We want to know which of our hosts are online.  We have this data center with all these machines.  We want to know which ones are working, right?  So let's do some monitoring, obviously.  So, hmm, you need to find out which, which hosts are there.  So you need to talk to them.  Where do you get the list of hosts?  And I've seen this a couple of times.  They get the list from pinging everyone and seeing who answers.  So you ping all the machines you know are there and are answering.  And surprisingly, everyone is there, right?  So always, always look closely where you get your data because stuff like that happens  all the time.  So what do we do when something breaks?  Typical admin problem, well, we have a backup, right?  Who has a backup?  Okay, okay.  Yeah, I've seen this a couple of times.  Oh, you know, storage is expensive.  Let's put it in the cloud.  Or you know, maybe you have this deal with, with whoever and they give you free cloud  storage and you say, oh, we have a few terabytes of cloud storage.  Let's put the backup there.  So you put the tar ball in the cloud and it contains all the config files with the credentials  and the personal data from users, the TLS keys, all of this has happened, you know.  The private keys, obviously.  The descriptor recovery keys, all of that shit is in the cloud.  And it usually gets found because, you know, there are search engines for that.  One of them is called Shodan.  People find stuff on Shodan daily.  It's so much that nobody actually realizes usually if something gets found because it  just gets drowned out in the flood.  There's so much stuff in the cloud.  So don't put your stuff in the cloud.  This is a classic in the banking environment.  Oh, telephone calls are so expensive.  Let's use voice over IP.  And then they go, well, we have the switch and there's a few free ports.  And then it turns out that the Cisco voice over IP phones want a trunk port.  So if you pull out the phone and pull in your network, your laptop, you can go into any  VLAN.  And it's a different department.  See, the people who bought the phones are a different department than the people who  set up the network.  So nobody realizes.  This used to work in all of the banks.  So what if we need some integration?  We have all these components and we want them to talk to each other.  I know scripting languages.  Let's do scripting.  And usually some admin hacks something, you know, the idea you use like this, another  XKCD, like I know regular expressions.  And usually it works like that too.  You have like amazed people on the ground and the script is like, ooh.  But you just want to solve the problem.  There's no quality assurance, no quality testing.  Usually there's not even a version control system, there's no ticketing.  It's just a small integration script.  So in practice, the quality of integration is much lower than the quality of the products,  the components you're trying to integrate.  I don't know why this keeps happening because it makes everyone look bad.  Usually you hear all these complaints and companies about how bad the components are  and it's not actually the components, it's integration that your own people did.  But you know, the admin is like, well, you know, I logged in and it worked.  So that's it.  Usually admins have enough on their plate that they lose interest immediately and start  doing other stuff.  Right?  So I've seen a few companies actually, I have an illustration for that.  I've seen a few companies where the admin did the integration in some language they wanted  to learn.  Not something they knew, something they wanted to learn.  Right?  Okay, this is another classic.  Oh, it's very slow.  We need to find out why it's slow.  So let's deploy some agents.  And this is my favorite picture in the world.  If you look closely, you see it's completely useless.  Things like connecting, loading, please wait, searching.  It's like zero content and usually monitoring systems are like that.  They give you like, woo, Starship Enterprise and there's no content.  But all the agents that deploy to take the measurements really slow things down more.  So yeah, desperation, swapping from monitoring is a thing.  Actually everything is even slower than.  This one really makes me sad because I heard it and I thought it's like a satire or something.  So it was a company and they outsourced their data center to some other company and they  told them you have to keep all the machines.  But we want you to guarantee us an SLA and it has to be cheaper than what we're doing  now.  And you have to keep the people.  So how is the company supposed to do that?  They skimmed on support, obviously.  No maintenance.  Everything broke down all the time and this happens all the time.  If you outsource something with unrealistic goals, then they're not going to get met.  So the next step is, well, we go on the cloud and I was like, why would you go on the cloud?  The previous deal already fell apart.  Why would the cloud be better?  And then the obvious counter argument is if you put your data in the cloud, it's not  your data anymore, obviously.  But I got this excuse.  They said, oh, it can't get any worse than it is now.  And AWS at least reacts to tickets we open.  And I thought, how is, what?  Yeah, they don't help me, but at least I get a form mail, form letter.  We take your problems very seriously.  Yeah, okay, this is another one.  Okay, I'm just a little bit self deprecating.  Usually you see companies at some point go, okay, we have so many security issues.  We need a security team.  Right?  And they go, let's get the best of the best.  24 seven operations, we call it our own cert.  And you get people like this.  And then everyone else in the company goes, well, it's not my problem anymore.  And they keep on causing more problems because security, someone else is dealing with that  now.  Right?  Okay, we're back to updates.  This is the perennial problem.  So I'm trying to illustrate something that, you know, sometimes I'm at a customer and  I get to see what someone is typing.  But I can't say anything.  Right?  And you see them do something and go, right?  So again, we're monitoring, right?  And the guy goes, okay, easy.  We need to deploy something.  And then, hmm, where's the host list coming from?  Right?  Oh, it's probably in the wiki.  At this point, I usually go, and they go, oh, let's go to the wiki.  And the wiki is up again.  And then the host list is from 1997, historical artifact.  So okay, let's ping everything.  Let's ping them all.  And they go, okay.  Pinging all of them.  And then, you know, of course, command on phone because it's a BSD, whatever.  Okay, let's do it with a shell.  And this is taking forever because every ping runs into a timeout for all the systems that  are not answering, right?  So it's like, oh, snore.  Get a coffee.  At some point, I was so annoyed that I wrote a little ping that can do this in parallel.  That's how bad this problem is.  And it happens all the time everywhere.  It's open source.  You can have it.  So I showed this to a customer and he was like, oh, that's awesome.  Let's extract the IP numbers.  And I kid you not.  You know, you could have waited for the other ping in that time.  It's done.  It's done.  Okay, so I showed them regular expressions, right?  And they go, oh, I can do this.  Right?  So, you know, have you heard of AWK?  Oh, yeah.  That's awesome.  I heard this once and I couldn't contain my laughter.  I was like, oh, of course you could use HP OpenView.  Let's boot the console.  Sure.  Anyway, so this really gets on my nerves because I think you owe it to yourself to know your  tools.  It's like if you get somebody to fix your kitchen and the guy goes, oh, let's open the  manual, find out what to do, I mean, it's like what?  No.  Right?  So I like to call this the monster truck mode because usually you have a problem but  it's not here.  It's there.  And on the way there, you see all these other things that don't work.  And you go, okay, I don't really want to deal with this.  I want to deal with that.  And you fix none of them properly, if at all.  Right?  So it's like you have this monster truck and you drive over everything on the way.  And the result is obviously that all the broken stuff stays broken.  Right?  This is a real problem in big installations.  Everything is broken.  Everybody knows everything is broken but nobody is fixing anything.  They only fix the stuff that is halting production now and the rest gets worse.  So everything stays broken.  Right?  The result is something like this.  You're like, not my job, you know?  Let someone else fix that.  Yeah.  This is also very common.  It's like never touch a running system.  Yeah, you do touch a running system.  That's your job.  That's what you're being paid for to fix the running system.  So it continues running.  And you see all these temporary hacks and fixes.  And nobody ever removes something that is not needed anymore but you just, oh, let's  add some more.  Also, there's always time pressure.  There's never a moment where you go, oh, I have time now.  I can fix this properly.  No, that doesn't happen.  If you wait for that, it's not going to happen.  Right?  Problems, problems just stay there.  And sometimes they get worse.  They usually get worse in a moment when you don't really need it.  This is one of my favorite pictures.  And I go, oh, I think we missed a button there.  Yeah, I already mentioned this.  I mean, I'm not here to teach you how to use the born shell, but you really should learn  that.  It's like the least you can do.  Like literally, it's literally the least you can do as an admin is to learn your tools.  Okay, I'm not going through all of these.  The last one has probably bitten everyone in this room at some point.  Oh, there's a space somewhere in one of the five names.  It doesn't work.  Yeah, learn your tools.  Really, I mean, there's no excuse.  It's not like you're a physician.  You're the admin.  That's also really annoying to me.  If you go somewhere and you see, well, you could have fixed it in five minutes, or you  can manage it in half an hour, and they go, let's manage it.  And then it still crashes, but it restarts.  And for some people, that's more attractive than fixing the problem.  I really don't understand that.  I mean, you see that all around you, even on Google, sometimes the query gets stuck.  And that's because some backend crashed somewhere, and you know, it's being restarted, but still,  it's not working.  Okay, the next one is quite common, too, when they go, well, I could do this, but it's not  my job.  We have a department for that.  Usually, it's like the firewall department.  It's not my job.  So we call the firewall department, and it takes about six months.  And then the rule is in, and you can't get it out again.  So this is high comedy.  If you ever have spare time and you know someone working in the firewall department, let them  show you the list of rules.  Half of them will be useless, not needed anymore.  That machine doesn't even exist anymore.  Same thing for logging.  Like, you have three teams.  They have three different log destinations, and they don't talk to each other.  And sometimes they even cite data protection laws to not talk to each other, which I find  really, really awesome.  It's like, that guy is the enemy.  I'm not giving him anything.  Yeah, this is more development thing, but I'm still listening to it here because sometimes  people who move in the cloud have the same kind of issue.  So if you develop software these days, you upload your sources to GitHub, and then the  rest is done by some service somewhere else in some other cloud, and you don't even know  who's really doing it.  And then people deploy the result into some app store without having seen it.  Like we have no reason to trust any of this.  And when you talk to people about this, they go, well, you know, I trust this more than  our own people because they are fucking stupid.  Well, hire good people then.  Okay, this is a bit esoteric.  I'm going to skip this.  I'm showing you the picture because it's nice.  Sometimes in companies, they close the USB ports and you can't log in remotely.  You have to go through three jump posts.  And to log in in the jump posts, you need to go through some common password broker  tool.  And then at the end system, everyone is always logging in as root.  So all that stuff is just useless.  It's just, I don't know why people do it.  One of the highlights of my career was when I was being asked to build a monitoring system  for a ZIM.  So the M in ZIM stands for monitoring.  The guy bought a monitoring system and was, I don't trust that thing.  We need a monitoring system for the monitoring system.  Yeah, I've actually seen this happen too.  So the groovy thing is not theoretically, right?  So the admin needs to do something and it takes a language he wants to learn, not a  language he knows.  Maybe it is a language he knows, but usually it's something they think would look good  on the resume.  Right?  And then they go, okay, I'll fix this in small talk.  And then obviously instantly that fix becomes part of the critical infrastructure and the  admin at some point leaves.  And then you have a small talk script that nobody understands.  Actually I've seen in the window source tree binaries that someone built at some time and  they don't have the sources for them.  But it's part of the build process.  So this happens to everyone.  It's not, I'm not talking down on small incompetent companies.  This happens all over the place.  Usually I like to think of this, you know, if you have all these old people who understand  stuff that if they die, it's gone.  These are the Navajo code talkers from the second world war.  Yeah, also that it's like you break something, admit it, own it, own it.  Don't go Han Solo.  You need something called blameless post mortem.  It needs to be possible to discuss what caused the problem without people leaving the room  because they think they're being singled out.  Right?  You have to establish that it's a cultural thing.  This one really, really pisses me off too when you see all no, no, we're not rewriting  that it's still good.  Right?  Everything, the whole system is really bad.  You can't even boot it without, you know, error messages.  And they go, oh, let's continue.  It still works.  See, I can still move.  It's not that bad.  So usually when I have this slide, people think of windows and there's a point, right?  There's a point about looking down on windows, all this crap that the internet is trying  to give you.  But it's not a windows issue.  It's a generic issue.  People install stuff that I don't really understand.  Notice the part when he says, well, every time I remove it after 24 to 48 hours, it's  back.  Yeah, it's the invisible hand of the market.  Bullshit.  All right.  I'm done with most of the content.  On the end of my talks, I usually have a few slides with free advice.  Take it or leave it.  So if you're management, you need to be happy if someone finds a bug.  Don't punish anyone.  Even the guy who wrote the code that's buggy.  Don't punish them.  Give them an opportunity to learn.  Show them the bug.  Don't have anyone else fix it.  Have that guy fix it.  And make sure you have a feedback system in your company.  I've seen places where the security team fixes bugs and the developer doesn't even realize  you made a mistake.  Obviously, that's the same mistake again.  Nobody told them.  You need feedback.  You need a culture of being able to own your bugs and problems.  So if you're management and you're planning a system, don't tell them how to do it.  You don't know.  If it's obvious enough that you can tell, then you're no help.  If it's not obvious, you're probably making it worse.  Trust your people.  If you're an admin, this is the goal.  Never have emergencies.  Automate everything until you have free time all day and then learn new stuff.  That's your goal as admin.  If you're doing firefighting all day, there will be never any end to that.  Some people still do it because they think they're not needed anymore if they fix everything.  That's not how it works.  So this is for anyone.  Don't look to others.  Look to yourself.  If you orient your own self-worth on others, it's going to fail.  There's no good outcome.  So if the other guy is about the same, then why would you put in more work?  It's good, right?  No motivation.  So if the other guy is worse than you, then you're ahead.  No need to put in more work.  If the other guy is better than you, you go, oh, I have no chance.  So you give up, right?  Never look to others, look to yourself in the past.  Be better than yesterday.  That's it.  If you have any questions, I think I've already exhausted my time, but you can email me if  you want to.  And I'll be here for a few more, like half an hour or something at least.  Thank you.  Thank you.
ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_0_71.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_72_203.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_204_251.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_252_299.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_300_467.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_468_539.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_540_671.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_672_731.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_732_779.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_780_1007.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_1008_1055.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_1056_1103.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_1104_1211.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_1212_1679.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_1680_1739.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_1740_7415.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_7416_7595.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_7596_7727.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_7728_7895.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_7896_8339.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_8340_8399.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_8400_8675.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_8676_8819.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_8820_9095.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_9096_10319.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_10320_12071.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_12072_12875.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_12876_13571.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_13572_15983.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_15984_16787.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_16788_16955.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_16956_23411.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_23412_23615.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_23616_23915.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_23916_29147.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_29148_35975.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_35976_47075.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_47076_47195.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_47196_47435.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_47436_51143.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_51144_52331.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_52332_52895.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_52896_54611.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_54612_56291.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_56292_59075.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_59076_60299.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_60300_65267.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_65268_65363.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_65364_69755.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_69756_73427.png ownCloud_conference_2018_Felix_von_Leitner___Fefes_Blog__Admin_Antipatterns_4Hvj1I8mQCo_73428_74011.png
Welcome to the second day of the OnCloud conference 2018. It is my great pleasure and honor to introduce you to Felix Leitner from Code Blau. He will give a talk on admin anti-patterns and maybe a little bit to the background of Felix von Leitner. He does the FIFA's blog. It's one of the top 50 blogs in Germany and has a lot of insights, especially encoding best practices and of course on admin stuff. It's really a great pleasure to have him here. I wish you a good talk, a good day. Like yesterday, my name is John. If you need anything, don't hesitate. Just contact me. I'll be around here and help you with anything you need. Thank you very much and have a nice day. Good morning. First problem, this talk I was asked to do here, I originally did somewhere else at the German users group. It's in German. At least the slides are. I'll try to translate. However, you will lose the famous German humor. I hope you can cope. Anyway, I was a little surprised when I first heard about OnCloud and I thought I should have a slide about this. The hacking community has their own little slang words. One of them is about taking over somebody else's machine if you hack somebody or something. It's called to own it. Connotation is not good. Connotation is, oh, you hacked him so bad that you have more ownership rights than him now. I was impressed by your naming of OnCloud. Just like Oracle calling something unbreakable. Let's see. Again, I did this talk already. I did another talk similar at the 34C3, which is the annual CCC conference. It was also called anti-patterns, but it was about programming software development. Then I was asked to do another one, but with a little spin on admins for the German users group. I found out in preparing that talk that there's actually more to say about admin anti-patterns than about software development anti-patterns. Both talks are available on the Internet, so if you want to see them, maybe you already saw them. There will be some overlap if you know them already. One of my problems is that I have no idea who's in the audience here. I was like, hmm. I asked John and he says, oh, it's all admins. They will know. Let's do a little test. If you know what this does, please raise your hand. I don't need the details. I just want to know. All right. What about this? Who knows how to do this by hand if Elzov is not available? It's getting more complicated. Okay. Thinning out a bit. All right. Okay. Good. I think we'll manage. Anti-patterns. The dictionary definition anti-pattern is something you do in response to a recurring problem that you hope will solve the problem, but usually it explodes in your hands and is usually ineffective or even counterproductive. I gave the talk a little motto and I stole it from Mike Twain. I think it's one of the most profound insights in the world. If you can read and don't, you are no better than someone who can't read. Don't be happy with Wikipedia being there and you can look something up. Learn it. There's a little structure to the talk. It's like a running gag, really. It starts with some kind of problem. Some problem and then someone does something. This is how they do it. Then the effect, which is usually bad because it's about anti-patterns. This joke is really old and stale and I wouldn't have done it, but it turns out there actually is a problem factory in Java, so I couldn't walk away from that. About hacking and admins in general. I've been playing video games for a while, so my worldview was influenced by video games. This is an old one. I'll show you some character pictures. As you level up, you notice the figure gets older. It's even better with the wizards. Look at the wizards. The beard is growing. I figured, okay, that's how you know that someone is a good hacker. I mean, cut this. Who knows who that is on the left? It's Alan Cox. He used to write the IP stack in Linux 2.0. That is actually a hacker. First time I saw him, I thought he was a homeless guy who went into the wrong building. No, he's actually a hacker and he's a good hacker. My first admin job, I expected something like this. Glorious beards. I walk in the room and it turns out there's nobody with beard anywhere. All these young people and shaved old people. The oldest guy was just changing tapes. Nobody tested me. Nobody asked anything. They just gave me the root password. I was like, oh, I was a bit confused. That's when I learned that being an admin is not about knowing all these skills that you need to be a good admin. Admin is about being young and being happy with small payment. You can deduct something about admins you see in the wild. There's two kinds of them. First one wants to solve problems and the other one wants to administrate. There are some admins who don't actually want to see problems go away. They're perfectly happy with their being problems around because otherwise they wouldn't be needed anymore. That's not good. Two kinds of admin. The first one is the new guy and he wants to see things work smoothly and he wants to learn new stuff. The other one is basically staying in the way and saying don't touch anything. You can see both admins causing problems every now and then. I've been trying to distribute the anti-prediction between them. A usual problem in administration is something should be automated. Something you do all the time or you don't want to do all the time. There's two things that can go wrong. First one is you automate too late to the game. This is easy enough to do manually. We don't really need to automate this. You do it manually all these years with all the typos and wasted time. That's a very common thing you see. The other one is just as bad. If you go let's do this now and then it turns out nobody actually needs that. So time is wasted. I mean most of the wisdom of the world you can find on XKCD so I brought three of them. This is the first one. Pass me the salt and he's like oh I'm working on a system to pass arbitrary condiments. This is a common problem. Don't fall for this. What about the common case of automation? You go okay hold my beer I'm automating this and you go. Everything goes away for six weeks. It's not heard of again. And comes back not having solved the problem. But on the way found 23 other problems and solved those. So I mean most of these are a little I mean they look like oh who's that stupid but you see the I mean this is from experience I've seen all of these in actual companies. Some administration is usually done by underpaid young people. So the management doesn't really enforce or expect quality as much as they should. And I think it's a co-dependency both sides are wrong. You should hold yourself to a high standard. So if the admin comes back and solves unrelated problems usually the solution is based on some prehistoric stuff that's only available on this old Vax that has not been patched. It does more harm than good. And the second next case right you expect it like the upper curve you go oh I'm writing the solution that's a short short time and then the solution takes all the problem away and I have all this free time but that's not how it works. And we know it doesn't work like that before we do it. Nobody should be surprised by this. So if you do it like that usually it pays off eventually but everyone will be long dead by then. So I'm trying to illustrate this here with a well known thing from the future. So another effect is that while you're trying to solve the problem nobody is touching it because it's being worked on so it stays there and sometimes it lays eggs. So yeah I mean this isn't good. Even if you automate something you still need to solve the problem. But wait for the automation because it may never succeed. Usually you have this problem that because nobody works on the other problem because it's being solved by someone somewhere that business processes get stuck. Right I mean this is all you probably have seen this before in administration. So next problem after automation is that you have software and it sucks. Usually software you run has known problems. You have an old version or there is a patch available. So you know the software is bad. So the obvious idea is you go to the vendor who sold you the software get some patches. So typically you don't send your top personnel to the vendor you send like the intern and the intern goes to the vendor and doesn't find anything. This is on purpose by the way because the vendor doesn't want new customers to see how bad things are. So they hide the security stuff. Some vendors go so far as to hide it behind a paywall or you have to sign in in the A database vendors have started to put in their terms of service that you cannot publish benchmarks about the database which I find really important. So I mean this is all things are done right and you send not your best guy but the intern right so the intern doesn't find anything and if the intern finds something usually the description of the patch is useless. That's on purpose too. The official excuse is that you don't want to give away to the attacker how to attack the system. But that's I don't think that's the truth. Vendors just you know they have all the spoiler plates so nobody actually sees how bad things are. So what's the intern supposed to do right there are some security mailing lists you can go to. These are old school mailing lists there are other ones but these mailing lists are really high volume. If you subscribe to them that's basically all you do right the intern goes away and can't do anything else because there's this flood of security announcements. And it turns out usually because it's the intern doesn't actually know what software you have. So he sees all these announcements and you know gets a few of them that he recognizes and the rest is just wasted. So all this stuff happens in the wild I mean it makes you wonder how things work at all. So but that's assumed the intern is very bright and actually finds a patch that is applicable. Finds a patch wants to put it on the system obviously. And then the other admin comes and blocks him. The old guy and the old guy says oh we need to wait a few weeks to see if the patch is good. Who here has seen that happen? Oh that's surprising if you I think you're lying to me. So this is very common. Oh no we can't put out this patch it might crash. And the effect is obviously that all systems in the company have all the known security problems of the last months. No surprise here. And this is all self inflicted. It's like this huge shotgun that you point to your foot you know. There is no need for any of this pain. So the next problem is we have all this malware in the company. What do we do? Well the obvious solution is to get some snake oil products like antiviruses you know network intrusion detection whatever Zeme. I have a nice, whoops I have a nice illustration here. Usually you can easily detect snake oil by how bullshit the marketing is. So I collected a field that I found very nice. The predictive security cloud. How about some cognitive SOC whatever that is. Oh and blockchain of course. So in practice I've actually seen that at the gig. They gave us a two core system and the first core was busy running semantic snake oil. And the second core was busy running McAfee and they killed each other all the time. So the second core was logging all the attacks it prevented. And if you wanted to open a tab in IE which is the only browser they gave me it took like two minutes. System was just busy. Not just CPU was busy. Storage was busy too. Storage was busy because they put each other in quarantine all the time. And then reinstalled. So it was really bad. And you would think that someone notices that before they give you the PC for work. But this happens. Another thing that happens is giant monitoring systems. Sometimes they actually really monitor you. Like what keystrokes you do and the web pages you visit depending on your jurisdiction. But in general the common thing you can see is that all these monitoring usually ends up somewhere. Nobody is reading that. So usually when I do a pen test for a company I ask them if they saw me. And I'm not trying to hide. That's not the point. I'm not being paid to hide. I'm being paid to try out problems. And I've been seen once. Everyone has all this huge monitoring stuff but nobody ever reads the logs. So I call those heights lifter which is like a heating appliance. They cause all this heat and it's nice and warm. But that's about it. So we talked about patches. Sometimes there's a service pack. This is a little older problem. The older admins among you will know this. And there used to be this, this of course you want to install it, but there used to be this like mystical idea. No, you can only take the even numbers or only the odd numbers and the other ones are bad which is the same thing as with the patches obviously. So same thing as before. You get all this malware and you can't get rid of it. So this patch thing is really important to me because it's so easy to see how not to do it. So this is a common thing you hear. We only install patches that we really need. Who has heard that before? Most big companies have their own department to find out which patches are needed and which ones aren't. And this sounds like a good idea but it's not. So first of all, you don't know. You have no clue. How would you know which patches are needed and which aren't? The systems are so complex these days. Even if you don't use a component, it might still be used by some other component. So you don't know, right? Secondly I've seen this firsthand. If I go to a vendor and to a source code audit and find 100 bugs, they will usually fix a few of them at least. But if you look at the next patch, none of them will be mentioned. Why would they mention bugs that nobody outside knows except me and I signed in NDA? So if you get a patch from a vendor, you can be sure that there's at least a dozen really horrible things it fixes that they don't want to tell you about because it would make them look bad. Install all patches all the time, right? The third one, this I think is the important argument for big companies. The patches are tested by the vendor but only under the assumption that you put in all the other patches. So if you skip patches, then you're on your own. How would the vendor support you if you have like this fractally different landscape because you skip this patch and this patch? You don't know. And neither does the vendor. They don't even have a machine with your exact patch level. So always install all the patches. If you forget everything else about this keynote, remember this. Always install all patches. So sometimes a new software version is coming out. It's a related issue to the patches, right? So you want to install it and then the old guy goes, no, no, it's a.0. We don't trust.0. We are waiting for.1. And you know, GCC is an important open source package. They just changed the versioning scheme so there is no.0. They start with.1 because nobody installs.0 anyway. So I mean, you can predict how this is going to continue, right? This is also a common problem in practice. You see all these really old installations and nobody wants to touch them. So let's update it. Right? How do you do it? Well, you put in all the patches. Well, some of these patches aren't deterministic, right? So the effect is that you have three old systems where you installed all the patches, but they're all subtly different, right? So this problem is common enough that these days it's one of the main arguments for containers like Docker. You don't install the patch, you install the container and then you know your installation is the same as it should be. And this is also, I mean, there's no reason why it should have been like that. And I'm assuming that Docker will have a similar problem at some point and then we'll have the next great idea. But all of these are self-inflicted. Okay, so let's update something. Different problem. And then you sometimes see that the admin doesn't want to update, right? The admin says, no, no, we can't patch because we will lose support if we patch. And at first I thought they're making fun of me, but there actually is environments where you have like an old Solaris and you have some kind of Oracle database on it or whatever, you know, substitute. And if you patch the Solaris, the Oracle support will tell you you're not supported anymore. This actually happens. I've seen this happen with telco equipment, like the SMS gateway at your telco. And what they do is they install three firewalls before it, as if that, you know, does anything. But you know, this is what happens, right? So we can't patch it, we lose support. And then you often hear, well, it's not old, it's stable. Yes, it's old. Stable is bullshit. It's not stable. It's known bad. We know it's bad. There is an update, you know? It's not, this continues to amaze me. You see all these really ancient machines and you go, who's using that? They go, oh, nobody's using that, you know, it's too old, all the commands don't work. Well, yeah, you get the strangest excuses like, well, we can't turn off SSL version 2 because we're talking to this, you know, mythical machine. We don't even know if it's there anymore. A customer of ours told us we can't turn off SSL 2. Yes, you can. You have to. It's broken. So it's not stable. It's known bad. I would like to change the terminology here. It's not stable. It's known bad. You have to understand that new releases don't fall from the sky, you know? Somebody is releasing them. They have a reason. You don't release something because you have free time. You release something because, you know, the old version is bad and you have a better version. It's easy. Yeah, this one is starting to gain ground. They go like, oh, I can't work here. The admin is too strict, you know? All this stuff is configured too close. I can't install software. I need my Solitaire game. And then they go, oh, I want to bring my own device. And you know, this is less of a problem now than it used to be. It used to be this, you know? Your tablet isn't part of our device management. You can't bring it. You know, so what happens is that the CEO is forwarding his mail to Yahoo and reads it with the tablet and the guest Wi-Fi. You know, I've seen this dozens of times. This is what happens if the admin tries to lock things down. It doesn't work. You have to work with your people, not against them. So this is like a meta problem. You know, oh, it doesn't work. It's horrible and you have to diagnose it and the first idea is, ah, the product is shit. And then, ah, the vendor is shit. Notice how the exclamation marks grow. And then, oh, it's a compiler bug. You know, the excuses get more and more absurd. No, no, it's a kernel bug. This is like the, the, the, the hunters have to usually tell these stories about these mythical beasts they, they slew in the wild. Same thing with admins. And usually when, you know, the, the, the someone looks over his shoulder and sees, well, you, you missed the semicolon here. Don't go in that rabbit hole and go, oh, it's the compiler. No, it's probably not. I mean, there are compiler bugs, but your problem is probably not a compiler bug. Most of these are really trivial if you look at them, so it's easy to make fun of them. But they will happen to you at some point. So this is a typical problem for admins. We want to know which of our hosts are online. We have this data center with all these machines. We want to know which ones are working, right? So let's do some monitoring, obviously. So, hmm, you need to find out which, which hosts are there. So you need to talk to them. Where do you get the list of hosts? And I've seen this a couple of times. They get the list from pinging everyone and seeing who answers. So you ping all the machines you know are there and are answering. And surprisingly, everyone is there, right? So always, always look closely where you get your data because stuff like that happens all the time. So what do we do when something breaks? Typical admin problem, well, we have a backup, right? Who has a backup? Okay, okay. Yeah, I've seen this a couple of times. Oh, you know, storage is expensive. Let's put it in the cloud. Or you know, maybe you have this deal with, with whoever and they give you free cloud storage and you say, oh, we have a few terabytes of cloud storage. Let's put the backup there. So you put the tar ball in the cloud and it contains all the config files with the credentials and the personal data from users, the TLS keys, all of this has happened, you know. The private keys, obviously. The descriptor recovery keys, all of that shit is in the cloud. And it usually gets found because, you know, there are search engines for that. One of them is called Shodan. People find stuff on Shodan daily. It's so much that nobody actually realizes usually if something gets found because it just gets drowned out in the flood. There's so much stuff in the cloud. So don't put your stuff in the cloud. This is a classic in the banking environment. Oh, telephone calls are so expensive. Let's use voice over IP. And then they go, well, we have the switch and there's a few free ports. And then it turns out that the Cisco voice over IP phones want a trunk port. So if you pull out the phone and pull in your network, your laptop, you can go into any VLAN. And it's a different department. See, the people who bought the phones are a different department than the people who set up the network. So nobody realizes. This used to work in all of the banks. So what if we need some integration? We have all these components and we want them to talk to each other. I know scripting languages. Let's do scripting. And usually some admin hacks something, you know, the idea you use like this, another XKCD, like I know regular expressions. And usually it works like that too. You have like amazed people on the ground and the script is like, ooh. But you just want to solve the problem. There's no quality assurance, no quality testing. Usually there's not even a version control system, there's no ticketing. It's just a small integration script. So in practice, the quality of integration is much lower than the quality of the products, the components you're trying to integrate. I don't know why this keeps happening because it makes everyone look bad. Usually you hear all these complaints and companies about how bad the components are and it's not actually the components, it's integration that your own people did. But you know, the admin is like, well, you know, I logged in and it worked. So that's it. Usually admins have enough on their plate that they lose interest immediately and start doing other stuff. Right? So I've seen a few companies actually, I have an illustration for that. I've seen a few companies where the admin did the integration in some language they wanted to learn. Not something they knew, something they wanted to learn. Right? Okay, this is another classic. Oh, it's very slow. We need to find out why it's slow. So let's deploy some agents. And this is my favorite picture in the world. If you look closely, you see it's completely useless. Things like connecting, loading, please wait, searching. It's like zero content and usually monitoring systems are like that. They give you like, woo, Starship Enterprise and there's no content. But all the agents that deploy to take the measurements really slow things down more. So yeah, desperation, swapping from monitoring is a thing. Actually everything is even slower than. This one really makes me sad because I heard it and I thought it's like a satire or something. So it was a company and they outsourced their data center to some other company and they told them you have to keep all the machines. But we want you to guarantee us an SLA and it has to be cheaper than what we're doing now. And you have to keep the people. So how is the company supposed to do that? They skimmed on support, obviously. No maintenance. Everything broke down all the time and this happens all the time. If you outsource something with unrealistic goals, then they're not going to get met. So the next step is, well, we go on the cloud and I was like, why would you go on the cloud? The previous deal already fell apart. Why would the cloud be better? And then the obvious counter argument is if you put your data in the cloud, it's not your data anymore, obviously. But I got this excuse. They said, oh, it can't get any worse than it is now. And AWS at least reacts to tickets we open. And I thought, how is, what? Yeah, they don't help me, but at least I get a form mail, form letter. We take your problems very seriously. Yeah, okay, this is another one. Okay, I'm just a little bit self deprecating. Usually you see companies at some point go, okay, we have so many security issues. We need a security team. Right? And they go, let's get the best of the best. 24 seven operations, we call it our own cert. And you get people like this. And then everyone else in the company goes, well, it's not my problem anymore. And they keep on causing more problems because security, someone else is dealing with that now. Right? Okay, we're back to updates. This is the perennial problem. So I'm trying to illustrate something that, you know, sometimes I'm at a customer and I get to see what someone is typing. But I can't say anything. Right? And you see them do something and go, right? So again, we're monitoring, right? And the guy goes, okay, easy. We need to deploy something. And then, hmm, where's the host list coming from? Right? Oh, it's probably in the wiki. At this point, I usually go, and they go, oh, let's go to the wiki. And the wiki is up again. And then the host list is from 1997, historical artifact. So okay, let's ping everything. Let's ping them all. And they go, okay. Pinging all of them. And then, you know, of course, command on phone because it's a BSD, whatever. Okay, let's do it with a shell. And this is taking forever because every ping runs into a timeout for all the systems that are not answering, right? So it's like, oh, snore. Get a coffee. At some point, I was so annoyed that I wrote a little ping that can do this in parallel. That's how bad this problem is. And it happens all the time everywhere. It's open source. You can have it. So I showed this to a customer and he was like, oh, that's awesome. Let's extract the IP numbers. And I kid you not. You know, you could have waited for the other ping in that time. It's done. It's done. Okay, so I showed them regular expressions, right? And they go, oh, I can do this. Right? So, you know, have you heard of AWK? Oh, yeah. That's awesome. I heard this once and I couldn't contain my laughter. I was like, oh, of course you could use HP OpenView. Let's boot the console. Sure. Anyway, so this really gets on my nerves because I think you owe it to yourself to know your tools. It's like if you get somebody to fix your kitchen and the guy goes, oh, let's open the manual, find out what to do, I mean, it's like what? No. Right? So I like to call this the monster truck mode because usually you have a problem but it's not here. It's there. And on the way there, you see all these other things that don't work. And you go, okay, I don't really want to deal with this. I want to deal with that. And you fix none of them properly, if at all. Right? So it's like you have this monster truck and you drive over everything on the way. And the result is obviously that all the broken stuff stays broken. Right? This is a real problem in big installations. Everything is broken. Everybody knows everything is broken but nobody is fixing anything. They only fix the stuff that is halting production now and the rest gets worse. So everything stays broken. Right? The result is something like this. You're like, not my job, you know? Let someone else fix that. Yeah. This is also very common. It's like never touch a running system. Yeah, you do touch a running system. That's your job. That's what you're being paid for to fix the running system. So it continues running. And you see all these temporary hacks and fixes. And nobody ever removes something that is not needed anymore but you just, oh, let's add some more. Also, there's always time pressure. There's never a moment where you go, oh, I have time now. I can fix this properly. No, that doesn't happen. If you wait for that, it's not going to happen. Right? Problems, problems just stay there. And sometimes they get worse. They usually get worse in a moment when you don't really need it. This is one of my favorite pictures. And I go, oh, I think we missed a button there. Yeah, I already mentioned this. I mean, I'm not here to teach you how to use the born shell, but you really should learn that. It's like the least you can do. Like literally, it's literally the least you can do as an admin is to learn your tools. Okay, I'm not going through all of these. The last one has probably bitten everyone in this room at some point. Oh, there's a space somewhere in one of the five names. It doesn't work. Yeah, learn your tools. Really, I mean, there's no excuse. It's not like you're a physician. You're the admin. That's also really annoying to me. If you go somewhere and you see, well, you could have fixed it in five minutes, or you can manage it in half an hour, and they go, let's manage it. And then it still crashes, but it restarts. And for some people, that's more attractive than fixing the problem. I really don't understand that. I mean, you see that all around you, even on Google, sometimes the query gets stuck. And that's because some backend crashed somewhere, and you know, it's being restarted, but still, it's not working. Okay, the next one is quite common, too, when they go, well, I could do this, but it's not my job. We have a department for that. Usually, it's like the firewall department. It's not my job. So we call the firewall department, and it takes about six months. And then the rule is in, and you can't get it out again. So this is high comedy. If you ever have spare time and you know someone working in the firewall department, let them show you the list of rules. Half of them will be useless, not needed anymore. That machine doesn't even exist anymore. Same thing for logging. Like, you have three teams. They have three different log destinations, and they don't talk to each other. And sometimes they even cite data protection laws to not talk to each other, which I find really, really awesome. It's like, that guy is the enemy. I'm not giving him anything. Yeah, this is more development thing, but I'm still listening to it here because sometimes people who move in the cloud have the same kind of issue. So if you develop software these days, you upload your sources to GitHub, and then the rest is done by some service somewhere else in some other cloud, and you don't even know who's really doing it. And then people deploy the result into some app store without having seen it. Like we have no reason to trust any of this. And when you talk to people about this, they go, well, you know, I trust this more than our own people because they are fucking stupid. Well, hire good people then. Okay, this is a bit esoteric. I'm going to skip this. I'm showing you the picture because it's nice. Sometimes in companies, they close the USB ports and you can't log in remotely. You have to go through three jump posts. And to log in in the jump posts, you need to go through some common password broker tool. And then at the end system, everyone is always logging in as root. So all that stuff is just useless. It's just, I don't know why people do it. One of the highlights of my career was when I was being asked to build a monitoring system for a ZIM. So the M in ZIM stands for monitoring. The guy bought a monitoring system and was, I don't trust that thing. We need a monitoring system for the monitoring system. Yeah, I've actually seen this happen too. So the groovy thing is not theoretically, right? So the admin needs to do something and it takes a language he wants to learn, not a language he knows. Maybe it is a language he knows, but usually it's something they think would look good on the resume. Right? And then they go, okay, I'll fix this in small talk. And then obviously instantly that fix becomes part of the critical infrastructure and the admin at some point leaves. And then you have a small talk script that nobody understands. Actually I've seen in the window source tree binaries that someone built at some time and they don't have the sources for them. But it's part of the build process. So this happens to everyone. It's not, I'm not talking down on small incompetent companies. This happens all over the place. Usually I like to think of this, you know, if you have all these old people who understand stuff that if they die, it's gone. These are the Navajo code talkers from the second world war. Yeah, also that it's like you break something, admit it, own it, own it. Don't go Han Solo. You need something called blameless post mortem. It needs to be possible to discuss what caused the problem without people leaving the room because they think they're being singled out. Right? You have to establish that it's a cultural thing. This one really, really pisses me off too when you see all no, no, we're not rewriting that it's still good. Right? Everything, the whole system is really bad. You can't even boot it without, you know, error messages. And they go, oh, let's continue. It still works. See, I can still move. It's not that bad. So usually when I have this slide, people think of windows and there's a point, right? There's a point about looking down on windows, all this crap that the internet is trying to give you. But it's not a windows issue. It's a generic issue. People install stuff that I don't really understand. Notice the part when he says, well, every time I remove it after 24 to 48 hours, it's back. Yeah, it's the invisible hand of the market. Bullshit. All right. I'm done with most of the content. On the end of my talks, I usually have a few slides with free advice. Take it or leave it. So if you're management, you need to be happy if someone finds a bug. Don't punish anyone. Even the guy who wrote the code that's buggy. Don't punish them. Give them an opportunity to learn. Show them the bug. Don't have anyone else fix it. Have that guy fix it. And make sure you have a feedback system in your company. I've seen places where the security team fixes bugs and the developer doesn't even realize you made a mistake. Obviously, that's the same mistake again. Nobody told them. You need feedback. You need a culture of being able to own your bugs and problems. So if you're management and you're planning a system, don't tell them how to do it. You don't know. If it's obvious enough that you can tell, then you're no help. If it's not obvious, you're probably making it worse. Trust your people. If you're an admin, this is the goal. Never have emergencies. Automate everything until you have free time all day and then learn new stuff. That's your goal as admin. If you're doing firefighting all day, there will be never any end to that. Some people still do it because they think they're not needed anymore if they fix everything. That's not how it works. So this is for anyone. Don't look to others. Look to yourself. If you orient your own self-worth on others, it's going to fail. There's no good outcome. So if the other guy is about the same, then why would you put in more work? It's good, right? No motivation. So if the other guy is worse than you, then you're ahead. No need to put in more work. If the other guy is better than you, you go, oh, I have no chance. So you give up, right? Never look to others, look to yourself in the past. Be better than yesterday. That's it. If you have any questions, I think I've already exhausted my time, but you can email me if you want to. And I'll be here for a few more, like half an hour or something at least. Thank you. Thank you.
Blau. He will give a talk on admin anti-patterns administration is something should be automated. wisdom of the world you can find on XKCD so I illustrate something that, you know, sometimes