SD-WAN a New Hop
Gordeychik, Sergey
The software defined wide-area network is technology based on SDN approach applied to branch office connections in Enterprises. According to Gartner's predictions, more than 50% of routers will be replaced with SD-WAN Solutions by 2020. The SD-WAN can have firewalls and other perimeter security features on board which makes them attractive targets for attackers. Vendors promise "on-the-fly agility, security" and many other benefits. But what does "security" really mean from a hand-on perspective? Most of SD-WAN solutions are distributed as Linux-based Virtual Appliances or a Cloud-centric service which can make them low-hanging fruit even for script kiddie. Complexity of SDN creates additional security issues and cybersecurity pro should address it before an attack occurs. This presentation will introduce practical analysis of different SD-WAN solutions from the attacker perspective. Attack surface, threat model and real-world vulnerabilities in SD-WAN solutions will be presented.
 Our next speaker is Sergei Godecic. Sergei has been doing security research, products and services for the past 15 years, since 2011 he is director and scriptwriter at Positive Hack Days Forum, the largest cybersecurity event in Eastern Europe. Sergei has for instance been working at Capersky Lab and Positive Technologies. He is also a visiting professor at Harbour Space University in Barcelona and leader of the SCADA Strange Lab, Industrial Cybersecurity Research Team. Today Sergei will talk about how to hack software to find networks and keeping your sanity while doing it. Let's give a warm round of applause for Sergei Godecic. Hello, hello, good night. Let's start to refresh our memories. This is a big honor for me to speak on the 35C3 because my first talk here was on the 29C3 with SCADA Strange Lab team and I think I can skip this introduction. Thanks for our host because everything is here and what I want to say about me, still I am very Russian, live in Abu Dhabi and do all this stuff because I saw this album in the airplane when I flight here. So accept Bitcoin only. So let's start to talk about software-defined networks. What is software-defined networks in general and is divine in particular case? It's a magic. So according to Gartner, it will kill impilates, it will replace all your Cisco and Juniper devices or Huawei if you prefer Chinese, but it's bad, you know, according to last news. And it will solve all your network problems because it has AI inside and it will magically optimize network operations and do everything including security. So because it's perfectly safe to implement inquiry networks, efficiently and security. So, okay, sounds good. What is actually software-defined networks? It's so simple. If you're familiar with the routers and switches, like it's just accept packets and send it somewhere according to the routing tables or the mock tables, but the software-defined networks is quite different. And when we tried with our team to understand how it works, our first impression was like this. We are hackers, we don't want to deal with this shit, but the only challenge we met before you hack something, you need to actually activate it and make it work. That is why we start to understand how software-defined networks and SD1 works. So what's the main difference between traditional LAN and SD1? In traditional LAN, you have different devices which solve very specific purpose, for instance, switches or routing or firewalling or network load balancing. In case of SD1, you have just a server which runs operation systems, in most cases it's like Linux. And on the top of this operation system, you have very specific models like CPE, which do specific network functions. It can be firewalling, it can be routing, it can be switching, it can be network load balancing. So you replace specific devices with one big server, which magically do everything with AI and in the cloud. Sure. So in the SD1, we have several layers. So low-level is a data plan when actually you process packets and decide how to go it, in which way how to firewall or drop it. We have control plan which manage different routers, different devices. We have management plan which can help to apply policies and orchestration plan because it's serious things that should have something which is called orchestration. But on the technical plan, again, we have hardware with operation system and a layer which called network function virtualization. What's network function virtualization? It's the way to apply different network functions to a specific device. It's very useful, for instance, to the network operators who will provide you with a specific box. And if you want to activate any functions, it can be web application, firewall or the sandboxes. We just upload very specific virtual machine. It can be Docker, it can be QVM image to your hardware and you can start to use it. Because inside of this box, you already have all the system infrastructure which process packets and pass it from one virtual network function to the other. This helps to organize things like the service chain-in. So you can distribute different network functions on the branch level, in the cloud level or on the HQ level. For instance, things like content filtering, which can be very heavy from performance perspective, can be distributed. As example, on the branch level, you can use simple things like the antivirus to process content. On the HQ level, you can use more heavy things like sandboxing. And if antivirus or specific rules see that this content is suspicious, it will forward it to HQ through the MPDS or other VPN and next processes in the HQ. Or you can also analyze it in the cloud through the simple things like the cloud threat intelligence where actually your SD1 box will send MD5 hash to the cloud and check is it good, contact, or bad. Or send all files to the cloud to double check it. Not bad. And I think that is why SD1 becomes more popular. And you can see that even military guys in US, they decide to switch to SD1 because of security, cost saving and all these benefits. Okay, security sounds very familiar for us and we decide to obviously hack it. I think most of you have experience in hacking of different network appliances and you know that sometimes you need to have complex things like an R&S soldering kit or debugger, jitagulator, etc. But not in this case. Because SD1 actually is not a appliance, it's a virtual appliance. And to start hacking, all you need is to go to the AWS or Azure and just activate this virtual appliance for, I don't know, 10 bucks per month. And next step is get root on it. It's a very good talk presented on different conferences including Zero Nights, how to hack virtual appliance and we use this like a checklist for our research. Because if you hack in virtual appliance, you already have access to this system. You can mount file system to the other virtual appliance. You can grab ETC shadow. You can find a lot of different backdoors just through the static analysis. But all the good things start from the Google. For instance, to find admin password for one of the SD1 appliance, we just Google for GitHub and found what most of the scripts which use it to automate with the appliance use a username administrator and password. Versa one to three. And actually, we found that this username and password is hard coded there because there is no way to change it. Next step to root it is just to Google for all vulnerabilities. For instance, in Silver Peak, we found that guys had reported vulnerability in September 2015 and it's still working in March 2018. So Google works because Google is full strong with this one. Next thing is, grab is always work as a string, you know. And using grab and code or code word password, you can find a lot of interesting things like hard coded password in different location in the configuration files in the database connection string. And then you can find a lot of interesting things like hard coded password in the system locks because, again, it's virtual appliance and someone had deployed before you start to use it. So in the locks, there are a lot of life interesting information. In the shadow file, like in one of Cisco appliance, ETC shadow file which use this encryption in 2018. You can do some forensic because, again, if you get virtual appliance, someone had deployed. And sometimes we're trying to hide this and you can see that cutting the best history. You can find that someone RAM scrub AWS shell script which actually set up different password, etc. So if you somehow can recover the script, you can find a lot of interesting information in this kind of password of admin users. And you can see that from this password you can find the hash and next try to brute force. It's just was my guess that maybe because Versa have password Versa123, maybe other network appliance like Silverpick. Have similar password Silverpick123 and this guess was successful. And it's good because you cannot stop the progress. If you have experience with red teaming in the enterprise network, you know that Cisco Cisco or Huawei with the administrator and Huawei123, it's quite common password. In this case, it's more complex thing sometimes where you lead like this tattery network stuff, you know. But still if you did not get the root with these simple steps, again, it's virtual appliance and you can always patch it. So you can change for instance, hash in the ETC shadow, you can change boot script, you can change remote management configuration, the password. And next boot in this configuration and get root password to do next step, security assessment. So security assessment, in the beginning we did in very, you know, not scientific way, we just hack all the things. But after all, we did some, let's say, scientific research and we have an article, I will give you a link like SD1 Threat Landscape with the step by step assessment what you should hack to get maximum results. But let's like mix this thing with funny hacks and the scientific approach. So from the system engineer point of view, SD1 have hardware, which of the shell hardware, operation system, in most cases again, of the shell Linux and different virtual services. Let's start from the operation system because again, everything you saw in the recent talk about BMC and remote management interfaces related to hardware, it still works here unless it is disabled by the vendor, but it's highly unlikely. On operation system, we did very simple research, we just check the patch level of the all components installed on this box. And you can see that patch level ridiculously old. For instance, all these things we found, it was an open SSL library, which was released in May 2006. It's for network devices with security functions. But our guess was that we choose this library because this library is too old to be vulnerable to heartbreak attack. And SD1 wins because the oldest open SSL library we found in commercial product wasn't before, wasn't Semantic WinCC, which was released in 2007. So SD1 is like old school really. Next thing related to operation system configuration is sudo. And it's actually everywhere and actually everywhere for management interfaces, including web services, shell, etc. And it's implemented in terrible way. As you can see, TRIP-W data have all ability to execute all command and some scripts just execute any command through the sudo. That is why if you have any small vulnerabilities in the web interfaces, like in this case, it's command injection, you can execute command with sudo. So it's again 1990s. Next point is software, not system, but software design point of view. It's from software design where a lot of open source component which implement IP stack, routing. But from management perspective, they use mostly HTTP and things around it. So let's analyze HTTP and web management interfaces. So in this case, it's not so old school like a system site. Everything based on Node.js and JavaScript, it's like very cool. But under the hood, you can find hard code mix from the Perl, Java, PHP, whatever, which looks like guys developed it last 10 years. With all this modern Node.js stuff, developers confuse the client and the server because you know JavaScript is on both sides and it's hard to understand where is server with client side. I will show you examples. And there are a lot of simple things like slow HTTP, those attacks, which should be fixed for a long time ago, but still you can stop web interface with few HTTP requests. So few examples about client side. JSON-C-SRF is everywhere, so almost no web interface implement protection from cross-site request forgery in proper way. XSS is everywhere and this is not a problem. So it's a response from the product manager of one vendor. They told me that XSS from cross-site scripting for web application is not the issue because Chrome blocks it. It's just an example of using XSS on such appliances. In this appliance, we can use combination of the XSS and cross-site request forgery to download and upload certificate which use it to authenticate with the server which like control plan with management server. It's just one HTTP request. And obviously there is no response from the vendor. We just silently fix it, so we decide to publish it through the full disclosure. One example of the perfect authentication. So in this case, you can see that it's client-side JavaScript which just send request to the server to the login status function. And if user is gogid, it's go to the request page. In our case, go to the username and password page. So this 100% client-side. No other checks on the server side just if you can change it. Second example is just perfect. So I think he tried to port the authentication from the server side to client side and understand what is JavaScript which still JavaScript but it doesn't work in browser. And he just like commented and say if username is with and password is with, then go home. So authentication is passed. So this thing. Highlighted box. So this authentication on this box. Next funny things which related to SD1 is about different privilege escalation. If you're already able to get access to any of virtual appliance inside, you can try to establish connection from these appliance to other appliance through the local host function. So why this can be interesting? Because there are a lot of open source component for the remote management. For instance, like challenge box which provide you like shell through the web interface or Moonin or Solar which like system management boxes. We just bind it to local host. So you cannot establish connection from outside because it's listening to the local host. But if you're already on these books and you can connect from these books to local host, then this establish this connection works. And this works because on each virtual, on which appliance you have a lot of virtual appliance which still listen to one IP addresses. If you have experience with the Docker for instance. So all Docker containers have all IP address. But from the all computer is actually connection from local host. So if you own any of the virtual appliance, you can own next all virtual machines installed here. This give us different interesting ways to escalate privilege inside the box. For instance, if you able to, let me switch to the laser pointer. You can see it, okay. If you can get access to traffic processor for instance for some TCP stack vulnerabilities. Next you can get access to management application and this management application in most cases have no any traffic filtering and trust to management application of all virtual network functions which run on these appliance. So you can do horizontal privilege escalation or next jump down to operation system level and next go to the management plan upstairs to the management appliances. But it's really boring to find the application vulnerabilities in such big amount of code. That is why we just download this code from the different network applies and drop it into the interactive code analysis system. In this case we use positive technologies application inspector and this help us to find a lot of vulnerabilities including such funny things like for instance, fully patched vulnerability in the Citrix SD1 which was patched in 2017. But it still works if you use not get HTTP request but post HTTP request. So we patched it once again. In this case better. Also it's a Udall lesson style vulnerability so it's obviously patched traversal but it's just reminded that attachments lead to the jealousy and the shadow of grid that is. So if you send attachment shadow you can get shadow file. So this is a full list of sorry full list of vulnerabilities we found during just source code analysis without you know brain interaction. Next step is crypto because security appliances shouldn't implement cryptography and there are two things is SSL TLS and type is that in most cases SSL TLS is used to protect management interfaces between the different appliances. And because it's automatic it's used different kind of automatic setup we found that there are a lot of things related to the unsecured configuration for instance we can use SSL TLS without forward secrecy. So if you have access to the certificate you can sniff off traffic even without full man in the middle different things related to the old Cypress use like a triple desk or RC4. For IPsec we found that in most cases they used very strange way to select the certificate or the precarious keys which in most cases just hard coded. So one example from real life our example we will publish soon from again Citrix Netscaller. These appliances use master control node protocol to communicate between the orchestration plan control plan and data plan. So let's run from the TCP 2156 and use TLS without forward secrecy and what interesting certificate located in the home TALERY user certificate and account WWW data have full access to this certificate. For some reason I don't know. Okay it should maybe read it but why to write. And what interesting all SD1 appliances we able to find during our security assessment we use same key pair which located in appliance edge. All SD1 appliances in the world used to protect communication between management and chain same key pair. So if you know this key pair and it's obviously you can just cut it from the file system you can passively or actively sniff traffic do manage the middle spoof management appliance. And if this device have any web application vulnerability you can overwrite it. I don't know why but maybe if in next turn we will change this certificate you can download old and do manage the middle again. Interesting stuff we found we run some tests from the DDoS attacks and found that Surikatan which use it in SD1 appliance are the ideas is vulnerable to regux DDoS. It's old story some regular expression can spend a lot of CPU time if you send specific queries. It's fixed in the default Surikatan kit but still working some modern SD1 solution. And for sure if you do some fuzzing it's always work and give you some fun. Unfortunately we cannot present reverse engineering part because most of such SD1 solution we have restriction in the licensing agreement to the reverse engineering. But just for fun I think that some of engineers way also love Star Wars and have Marvel. That is why in civilization functions cause Marvel sucks. So just an overview of detected vulnerabilities. So green is good or bad so good for vendor but for us we are unable to detect it. But you can see that most of classes of vulnerabilities like hard codes, broken access control, old products or Linux components or third party components were in most of such system you can find in most of such system. So just select any SD1 and make a shot. The interesting thing in SD1 is zero touch deployment. So it's very cool feature. For instance let's imagine that you have a branch office. You need to deploy branch office. With SD1 it's absolutely not necessary to go where or establish telnet or SSH connection and try to upload configuration. All you need is just to ship this device, note with unique ID, set up it through the cloud console and ship this device to remote location. This device will automatically connect to any internet it's around, connect to the cloud server, the node configuration and start operation. So for example how it works in the Citrix system. We have visa plans which ship it to the office. It's first try to establish connection with the surrounding the plans. If no, it's try to go to the zero touch deployment service, present own ID and next this service will provide all configuration which you upload through the SD1. So from security perspective this scheme looks terrible. Why? Because this SD1 and cloud deployment server should be friendly. But any attacker, not. If you know ID, if you can brute force this ID, you can pretend to be this device. If you have any weaknesses in implementation of this management server, you can own all devices which deployed through this service. And as you can see even Cisco which is like the best device from security perspective we found on this product line, let's say, have enough vulnerabilities. And this you can see that zero touch provision in command injection vulnerability. So it's cloud server which to which all network device should connect sometimes. But also we found very funny things related to the distribution of this device. Because as I told you in the beginning, most of such device can be activated as a cloud appliance through the AWS or the other cloud services. And we found that most of default images use old version with non vulnerabilities. So you go to AWS, you trust the vendor, you activate the system and you receive attack because there is no vulnerability. It's remind me in those story, I'm really old man, sorry. You know code red, this kind of key door, this kind of warms. And this real disaster when you just install fresh Windows 2000 which have Internet information server by default. And just connect it into Internet to download patches you receive new infection and need to reinstall it from the beginning. So these things look similar for me but is much worse because in this case is a security network device installed on the perimeter of your network. This overview of up to date statistics, you can see that very few vendors, actually no one of vendors have up to date version in the deployed on AWS. So I think it's abuse of the force. But it's also very interesting part of story as a security researchers, we always work in the responsible disclosure way. And as you can see, according to this article, some vendors, we also understand that responsible disclosure is very important to communicate with the user. It's important to communicate with community to fix issues and we even have product security incident response team in place. Great. But when we try to submit vulnerability report to this vendor, we unable to find the email of this product security incident response team. So there is no pool. We tried to Google for it a different way, no luck, but we found that guys who did similar research before, they found a great way, send email to CEO of this company. Unfortunately, my Google foo is not good enough, but I unable to find the mail, but I found this guy on the link it in. He answered it in few minutes actually and put me in contact. So if you try to deal with as the one vulnerability reporting just do this way. So we prepared this table about different vendors, how we communicate with researchers. And you can see that actually Cisco, Citrix and the account which actually VMware, not bad, but all the rest is just the beginning. This is my favorite mail from the one vendor. When we send notification, we start to ask them why we send this email from the Gmail. Do we have official ID, what they mean I need to present my passport or whatever to submit vulnerabilities. The funnest thing here, this vendor wrote that where device is not generic web service which have full Internet explosion. So after reading this email, I go sleep and during the night someone told me. So to understand the threat escape of SD1, we built a bunch of script which works on the top of the different search engines like census, showdown, Google. And use also Nmap, low scripting platform to fingerprint different SD1 solution. We have an article published as SD1 Internet sensors and also some tools which can generate beautiful maps if you want to present on the house communication congress because in other cases it's useless. And what we found that there are not so many SD1 devices yet in the Internet, about 3,000 management interface which contains no vulnerabilities and you can know it in a few minutes. And also we built some kind of vulnerability assessment tool which help you to find no vulnerabilities in these SD1 devices. This example for open SSH patch level, as you can see with like some CVs from 2010, 2014, etc. This open source, you can find it on the GitHub. We have two versions, one SD1 harvester which use Google showdown and census to collect information of all the all-verse Internet. And also SD1 infiltrator which is bunch of the network Nmap script engine scripts and you can use it during penetration testing. So it's not necessary to be connected to the Internet, you can just use it inside the network. When we did this research, we also found interesting article from Silence about dark web market when the bad guys sell user names in password to different network appliances, let's say enterprise level network appliances. And we found that there are some IP addresses which we found during our assessment Internet harvesting in this list. And in our experience, there is no such thing as a luck. We tried to find why such appliances can be so easily hacked and obviously the default password which hardcoded, sometimes never changed, was used on these appliances. We tried to reach vendors and say, guys, maybe it's a bad idea to use, for instance, hardcoded SNMP, not hardcoded, but default community like public and again public for read write. But they told us that SNMP is off by default, but still simple showdown search show that more than 200 users of this SD1, we enable this SNMP service and still use default password. So we have a lot of tools which publish it in our GitHub and please contribute. There are a lot of things to do with the new fingerprints for SD1 harvester and infiltrator with SD1 threat landscape description with new vulnerabilities and also it's like special CCC release. We start to publish metasploit models for the SD1. So we have public vulnerability description. You can create own models for it. And conclusions. So from my perspective how SD1 development like cycle works, someone come to us with brilliant idea, okay, let's build SD1 because Garner told us that it's like brilliant. They have AI in the cloud. So what we have to do, we can download bunch of open sources, put together, set up default routing things and after all use it as SD1. So SD1 is a bunch of open source which not bad, but still you need care about it and you need install patches, configure in proper way and maintain. These complex products have problem with patch management, have a lot of management interface like machine to machine and also human to machine interfaces. Have a lot of weak defaults like password, hard coded certificate, PSK keys for the IPsec and many vendors unfortunately have issues with the patching responsible disclosure. And this in the cloud. So if you decide to switch your network to the SD1 hack it before buy, all you will fail. So thank you for your attention. I want to ask you to give big hand to the SD1 new hope team to Dennis, Maxim, Nikolai, Nikita, Alec and Antoni who did most of the things here. I just like a frontman of this group. Thank you so much guys. Applause Any questions? Yep. Aside from generic things that any random Linux server can have on web interface on SSH, have you looked at any specific SD1 security problems like with encapsulation of tunnels or some stuff like that? So SD1, you know, there is no technology like SD1. So for SDN there is kind of protocol which more or less reused in different vendor, in different solution and SD1 every vendor implement things in own way. As an example, it's like Citric management protocol, etc. One thing we did here, but we didn't publish it, is for the virtualization. Because again this VNF story, very interesting because if you have vulnerability in any virtual function, next you can get access to our virtual machine. But again, problem here, there is no standard and different vendors call VNF, so implement VNF in different way. It can be QVM, it can be just a script which we upload to our appliance. So you're saying because everyone is writing their own code, there are a lot of bugs to find for people who put the work in? Yeah. Okay. And you can just, like, it's not necessary to try to buy things through the A-buy to hack it, you can just go to AWS and activate it for free. Okay, Mike, one please. Mike will be simple, there was a lot of vendors that you were looking in. What about Juniper SD1? Um, we're not all yet. Looking forward for controlled investigation. Thank you. Good try. And yes, please. Okay, say, thanks for the talk. You mentioned a lot of these virtual machines were running in hard Linux or something. How do you, what are they running? Is that good stuff? What hash levels are they usually on? These kernels? Is it like always 2.6 or something like that? Not always 2.6, it was like the worst example. So, some of us like, you know, newer. So, it's not necessary. Again, for network function virtualization, some vendors, they call VNF, just a bunch of script with download and change configuration of defaults. And this case we can use very old kernel. Other way, use more recent version. Is, is, sorry. Yeah, that's okay. And the hardening, how is it done? Is it hardening and pulling experiments? In most cases, no way. So, no armor or things like this. No Ger security or Linux. Nothing. Okay, thanks. All right, your question. Have you looked into Cisco, Muraki? No, we, for Cisco, we, sorry, it's, it's night. Let me find the list. For Cisco, we did our exercise with the VipTeller. All right, last question or there please. Just advice, maybe you drop a slide in with the 90s code, they want the password back. Sorry. Because there, there were so many hard coded passwords everywhere. Maybe you just should just drop a slide in the 90s code, they want the password or lock in the back. I don't know. So it's public things. So was that a question? It's just recommendation. All right, we have a few minutes left, but if there are no questions left, then I would call it a day. So another warm round of applause for Saturday. Thank you.
 Our next speaker is Sergei Godecic.  Sergei has been doing security research, products and services for the past 15 years,  since 2011 he is director and scriptwriter at Positive Hack Days Forum,  the largest cybersecurity event in Eastern Europe.  Sergei has for instance been working at Capersky Lab and Positive Technologies.  He is also a visiting professor at Harbour Space University in Barcelona  and leader of the SCADA Strange Lab, Industrial Cybersecurity Research Team.  Today Sergei will talk about how to hack software to find networks  and keeping your sanity while doing it.  Let's give a warm round of applause for Sergei Godecic.  Hello, hello, good night. Let's start to refresh our memories.  This is a big honor for me to speak on the 35C3 because my first talk here was on the 29C3  with SCADA Strange Lab team and I think I can skip this introduction.  Thanks for our host because everything is here and what I want to say about me,  still I am very Russian, live in Abu Dhabi and do all this stuff because I saw this album in the airplane when I flight here.  So accept Bitcoin only.  So let's start to talk about software-defined networks.  What is software-defined networks in general and is divine in particular case?  It's a magic.  So according to Gartner, it will kill impilates, it will replace all your Cisco and Juniper devices  or Huawei if you prefer Chinese, but it's bad, you know, according to last news.  And it will solve all your network problems because it has AI inside  and it will magically optimize network operations and do everything including security.  So because it's perfectly safe to implement inquiry networks, efficiently and security.  So, okay, sounds good.  What is actually software-defined networks?  It's so simple.  If you're familiar with the routers and switches, like it's just accept packets and send it  somewhere according to the routing tables or the mock tables, but the software-defined networks is quite different.  And when we tried with our team to understand how it works, our first impression was like this.  We are hackers, we don't want to deal with this shit, but the only challenge we met before you hack something,  you need to actually activate it and make it work.  That is why we start to understand how software-defined networks and SD1 works.  So what's the main difference between traditional LAN and SD1?  In traditional LAN, you have different devices which solve very specific purpose, for instance,  switches or routing or firewalling or network load balancing.  In case of SD1, you have just a server which runs operation systems, in most cases it's like Linux.  And on the top of this operation system, you have very specific models like CPE,  which do specific network functions.  It can be firewalling, it can be routing, it can be switching, it can be network load balancing.  So you replace specific devices with one big server, which magically do everything with AI and in the cloud.  Sure.  So in the SD1, we have several layers.  So low-level is a data plan when actually you process packets and decide how to go it,  in which way how to firewall or drop it.  We have control plan which manage different routers, different devices.  We have management plan which can help to apply policies and orchestration plan  because it's serious things that should have something which is called orchestration.  But on the technical plan, again, we have hardware with operation system  and a layer which called network function virtualization.  What's network function virtualization?  It's the way to apply different network functions to a specific device.  It's very useful, for instance, to the network operators who will provide you with a specific box.  And if you want to activate any functions, it can be web application, firewall or the sandboxes.  We just upload very specific virtual machine.  It can be Docker, it can be QVM image to your hardware and you can start to use it.  Because inside of this box, you already have all the system infrastructure  which process packets and pass it from one virtual network function to the other.  This helps to organize things like the service chain-in.  So you can distribute different network functions on the branch level, in the cloud level or on the HQ level.  For instance, things like content filtering, which can be very heavy from performance perspective, can be distributed.  As example, on the branch level, you can use simple things like the antivirus to process content.  On the HQ level, you can use more heavy things like sandboxing.  And if antivirus or specific rules see that this content is suspicious,  it will forward it to HQ through the MPDS or other VPN and next processes in the HQ.  Or you can also analyze it in the cloud through the simple things like the cloud threat intelligence  where actually your SD1 box will send MD5 hash to the cloud and check is it good, contact, or bad.  Or send all files to the cloud to double check it.  Not bad.  And I think that is why SD1 becomes more popular.  And you can see that even military guys in US, they decide to switch to SD1 because of security, cost saving and all these benefits.  Okay, security sounds very familiar for us and we decide to obviously hack it.  I think most of you have experience in hacking of different network appliances and you know that sometimes you need to have complex things like an R&S soldering kit or debugger, jitagulator, etc.  But not in this case.  Because SD1 actually is not a appliance, it's a virtual appliance.  And to start hacking, all you need is to go to the AWS or Azure and just activate this virtual appliance for, I don't know, 10 bucks per month.  And next step is get root on it.  It's a very good talk presented on different conferences including Zero Nights, how to hack virtual appliance and we use this like a checklist for our research.  Because if you hack in virtual appliance, you already have access to this system.  You can mount file system to the other virtual appliance.  You can grab ETC shadow.  You can find a lot of different backdoors just through the static analysis.  But all the good things start from the Google.  For instance, to find admin password for one of the SD1 appliance, we just Google for GitHub and found what most of the scripts which use it to automate with the appliance use a username administrator and password.  Versa one to three.  And actually, we found that this username and password is hard coded there because there is no way to change it.  Next step to root it is just to Google for all vulnerabilities.  For instance, in Silver Peak, we found that guys had reported vulnerability in September 2015 and it's still working in March 2018.  So Google works because Google is full strong with this one.  Next thing is, grab is always work as a string, you know.  And using grab and code or code word password, you can find a lot of interesting things like hard coded password in different location in the configuration files in the database connection string.  And then you can find a lot of interesting things like hard coded password in the system locks because, again, it's virtual appliance and someone had deployed before you start to use it.  So in the locks, there are a lot of life interesting information.  In the shadow file, like in one of Cisco appliance, ETC shadow file which use this encryption in 2018.  You can do some forensic because, again, if you get virtual appliance, someone had deployed.  And sometimes we're trying to hide this and you can see that cutting the best history.  You can find that someone RAM scrub AWS shell script which actually set up different password, etc.  So if you somehow can recover the script, you can find a lot of interesting information in this kind of password of admin users.  And you can see that from this password you can find the hash and next try to brute force.  It's just was my guess that maybe because Versa have password Versa123, maybe other network appliance like Silverpick.  Have similar password Silverpick123 and this guess was successful.  And it's good because you cannot stop the progress.  If you have experience with red teaming in the enterprise network, you know that Cisco Cisco or Huawei with the administrator and Huawei123, it's quite common password.  In this case, it's more complex thing sometimes where you lead like this tattery network stuff, you know.  But still if you did not get the root with these simple steps, again, it's virtual appliance and you can always patch it.  So you can change for instance, hash in the ETC shadow, you can change boot script, you can change remote management configuration, the password.  And next boot in this configuration and get root password to do next step, security assessment.  So security assessment, in the beginning we did in very, you know, not scientific way, we just hack all the things.  But after all, we did some, let's say, scientific research and we have an article, I will give you a link like SD1 Threat Landscape with the step by step assessment what you should hack to get maximum results.  But let's like mix this thing with funny hacks and the scientific approach.  So from the system engineer point of view, SD1 have hardware, which of the shell hardware, operation system, in most cases again, of the shell Linux and different virtual services.  Let's start from the operation system because again, everything you saw in the recent talk about BMC and remote management interfaces related to hardware, it still works here unless it is disabled by the vendor, but it's highly unlikely.  On operation system, we did very simple research, we just check the patch level of the all components installed on this box.  And you can see that patch level ridiculously old.  For instance, all these things we found, it was an open SSL library, which was released in May 2006. It's for network devices with security functions.  But our guess was that we choose this library because this library is too old to be vulnerable to heartbreak attack.  And SD1 wins because the oldest open SSL library we found in commercial product wasn't before, wasn't Semantic WinCC, which was released in 2007.  So SD1 is like old school really.  Next thing related to operation system configuration is sudo. And it's actually everywhere and actually everywhere for management interfaces, including web services, shell, etc.  And it's implemented in terrible way. As you can see, TRIP-W data have all ability to execute all command and some scripts just execute any command through the sudo.  That is why if you have any small vulnerabilities in the web interfaces, like in this case, it's command injection, you can execute command with sudo.  So it's again 1990s.  Next point is software, not system, but software design point of view. It's from software design where a lot of open source component which implement IP stack, routing.  But from management perspective, they use mostly HTTP and things around it.  So let's analyze HTTP and web management interfaces.  So in this case, it's not so old school like a system site.  Everything based on Node.js and JavaScript, it's like very cool.  But under the hood, you can find hard code mix from the Perl, Java, PHP, whatever, which looks like guys developed it last 10 years.  With all this modern Node.js stuff, developers confuse the client and the server because you know JavaScript is on both sides and it's hard to understand where is server with client side.  I will show you examples.  And there are a lot of simple things like slow HTTP, those attacks, which should be fixed for a long time ago, but still you can stop web interface with few HTTP requests.  So few examples about client side. JSON-C-SRF is everywhere, so almost no web interface implement protection from cross-site request forgery in proper way.  XSS is everywhere and this is not a problem.  So it's a response from the product manager of one vendor. They told me that XSS from cross-site scripting for web application is not the issue because Chrome blocks it.  It's just an example of using XSS on such appliances. In this appliance, we can use combination of the XSS and cross-site request forgery to download and upload certificate which use it to authenticate with the server which like control plan with management server.  It's just one HTTP request.  And obviously there is no response from the vendor. We just silently fix it, so we decide to publish it through the full disclosure.  One example of the perfect authentication. So in this case, you can see that it's client-side JavaScript which just send request to the server to the login status function.  And if user is gogid, it's go to the request page. In our case, go to the username and password page.  So this 100% client-side. No other checks on the server side just if you can change it.  Second example is just perfect. So I think he tried to port the authentication from the server side to client side and understand what is JavaScript which still JavaScript but it doesn't work in browser.  And he just like commented and say if username is with and password is with, then go home. So authentication is passed.  So this thing. Highlighted box. So this authentication on this box.  Next funny things which related to SD1 is about different privilege escalation.  If you're already able to get access to any of virtual appliance inside, you can try to establish connection from these appliance to other appliance through the local host function.  So why this can be interesting? Because there are a lot of open source component for the remote management.  For instance, like challenge box which provide you like shell through the web interface or Moonin or Solar which like system management boxes.  We just bind it to local host. So you cannot establish connection from outside because it's listening to the local host.  But if you're already on these books and you can connect from these books to local host, then this establish this connection works.  And this works because on each virtual, on which appliance you have a lot of virtual appliance which still listen to one IP addresses.  If you have experience with the Docker for instance. So all Docker containers have all IP address.  But from the all computer is actually connection from local host.  So if you own any of the virtual appliance, you can own next all virtual machines installed here.  This give us different interesting ways to escalate privilege inside the box.  For instance, if you able to, let me switch to the laser pointer.  You can see it, okay. If you can get access to traffic processor for instance for some TCP stack vulnerabilities.  Next you can get access to management application and this management application in most cases have no any traffic filtering and trust to management application of all virtual network functions which run on these appliance.  So you can do horizontal privilege escalation or next jump down to operation system level and next go to the management plan upstairs to the management appliances.  But it's really boring to find the application vulnerabilities in such big amount of code. That is why we just download this code from the different network applies and drop it into the interactive code analysis system.  In this case we use positive technologies application inspector and this help us to find a lot of vulnerabilities including such funny things like for instance, fully patched vulnerability in the Citrix SD1 which was patched in 2017.  But it still works if you use not get HTTP request but post HTTP request. So we patched it once again. In this case better.  Also it's a Udall lesson style vulnerability so it's obviously patched traversal but it's just reminded that attachments lead to the jealousy and the shadow of grid that is.  So if you send attachment shadow you can get shadow file. So this is a full list of sorry full list of vulnerabilities we found during just source code analysis without you know brain interaction.  Next step is crypto because security appliances shouldn't implement cryptography and there are two things is SSL TLS and type is that in most cases SSL TLS is used to protect management interfaces between the different appliances.  And because it's automatic it's used different kind of automatic setup we found that there are a lot of things related to the unsecured configuration for instance we can use SSL TLS without forward secrecy.  So if you have access to the certificate you can sniff off traffic even without full man in the middle different things related to the old Cypress use like a triple desk or RC4.  For IPsec we found that in most cases they used very strange way to select the certificate or the precarious keys which in most cases just hard coded.  So one example from real life our example we will publish soon from again Citrix Netscaller. These appliances use master control node protocol to communicate between the orchestration plan control plan and data plan.  So let's run from the TCP 2156 and use TLS without forward secrecy and what interesting certificate located in the home TALERY user certificate and account WWW data have full access to this certificate.  For some reason I don't know. Okay it should maybe read it but why to write.  And what interesting all SD1 appliances we able to find during our security assessment we use same key pair which located in appliance edge.  All SD1 appliances in the world used to protect communication between management and chain same key pair.  So if you know this key pair and it's obviously you can just cut it from the file system you can passively or actively sniff traffic do manage the middle spoof management appliance.  And if this device have any web application vulnerability you can overwrite it.  I don't know why but maybe if in next turn we will change this certificate you can download old and do manage the middle again.  Interesting stuff we found we run some tests from the DDoS attacks and found that Surikatan which use it in SD1 appliance are the ideas is vulnerable to regux DDoS.  It's old story some regular expression can spend a lot of CPU time if you send specific queries.  It's fixed in the default Surikatan kit but still working some modern SD1 solution.  And for sure if you do some fuzzing it's always work and give you some fun.  Unfortunately we cannot present reverse engineering part because most of such SD1 solution we have restriction in the licensing agreement to the reverse engineering.  But just for fun I think that some of engineers way also love Star Wars and have Marvel.  That is why in civilization functions cause Marvel sucks.  So just an overview of detected vulnerabilities.  So green is good or bad so good for vendor but for us we are unable to detect it.  But you can see that most of classes of vulnerabilities like hard codes, broken access control, old products or Linux components or third party components were in most of such system you can find in most of such system.  So just select any SD1 and make a shot.  The interesting thing in SD1 is zero touch deployment.  So it's very cool feature.  For instance let's imagine that you have a branch office.  You need to deploy branch office.  With SD1 it's absolutely not necessary to go where or establish telnet or SSH connection and try to upload configuration.  All you need is just to ship this device, note with unique ID, set up it through the cloud console and ship this device to remote location.  This device will automatically connect to any internet it's around, connect to the cloud server, the node configuration and start operation.  So for example how it works in the Citrix system.  We have visa plans which ship it to the office.  It's first try to establish connection with the surrounding the plans.  If no, it's try to go to the zero touch deployment service, present own ID and next this service will provide all configuration which you upload through the SD1.  So from security perspective this scheme looks terrible.  Why? Because this SD1 and cloud deployment server should be friendly.  But any attacker, not.  If you know ID, if you can brute force this ID, you can pretend to be this device.  If you have any weaknesses in implementation of this management server, you can own all devices which deployed through this service.  And as you can see even Cisco which is like the best device from security perspective we found on this product line, let's say, have enough vulnerabilities.  And this you can see that zero touch provision in command injection vulnerability.  So it's cloud server which to which all network device should connect sometimes.  But also we found very funny things related to the distribution of this device.  Because as I told you in the beginning, most of such device can be activated as a cloud appliance through the AWS or the other cloud services.  And we found that most of default images use old version with non vulnerabilities.  So you go to AWS, you trust the vendor, you activate the system and you receive attack because there is no vulnerability.  It's remind me in those story, I'm really old man, sorry.  You know code red, this kind of key door, this kind of warms.  And this real disaster when you just install fresh Windows 2000 which have Internet information server by default.  And just connect it into Internet to download patches you receive new infection and need to reinstall it from the beginning.  So these things look similar for me but is much worse because in this case is a security network device installed on the perimeter of your network.  This overview of up to date statistics, you can see that very few vendors, actually no one of vendors have up to date version in the deployed on AWS.  So I think it's abuse of the force.  But it's also very interesting part of story as a security researchers, we always work in the responsible disclosure way.  And as you can see, according to this article, some vendors, we also understand that responsible disclosure is very important to communicate with the user.  It's important to communicate with community to fix issues and we even have product security incident response team in place.  Great.  But when we try to submit vulnerability report to this vendor, we unable to find the email of this product security incident response team.  So there is no pool.  We tried to Google for it a different way, no luck, but we found that guys who did similar research before, they found a great way, send email to CEO of this company.  Unfortunately, my Google foo is not good enough, but I unable to find the mail, but I found this guy on the link it in.  He answered it in few minutes actually and put me in contact.  So if you try to deal with as the one vulnerability reporting just do this way.  So we prepared this table about different vendors, how we communicate with researchers.  And you can see that actually Cisco, Citrix and the account which actually VMware, not bad, but all the rest is just the beginning.  This is my favorite mail from the one vendor.  When we send notification, we start to ask them why we send this email from the Gmail.  Do we have official ID, what they mean I need to present my passport or whatever to submit vulnerabilities.  The funnest thing here, this vendor wrote that where device is not generic web service which have full Internet explosion.  So after reading this email, I go sleep and during the night someone told me.  So to understand the threat escape of SD1, we built a bunch of script which works on the top of the different search engines like census, showdown, Google.  And use also Nmap, low scripting platform to fingerprint different SD1 solution.  We have an article published as SD1 Internet sensors and also some tools which can generate beautiful maps if you want to present on the house communication congress because in other cases it's useless.  And what we found that there are not so many SD1 devices yet in the Internet, about 3,000 management interface which contains no vulnerabilities and you can know it in a few minutes.  And also we built some kind of vulnerability assessment tool which help you to find no vulnerabilities in these SD1 devices.  This example for open SSH patch level, as you can see with like some CVs from 2010, 2014, etc.  This open source, you can find it on the GitHub. We have two versions, one SD1 harvester which use Google showdown and census to collect information of all the all-verse Internet.  And also SD1 infiltrator which is bunch of the network Nmap script engine scripts and you can use it during penetration testing.  So it's not necessary to be connected to the Internet, you can just use it inside the network.  When we did this research, we also found interesting article from Silence about dark web market when the bad guys sell user names in password to different network appliances, let's say enterprise level network appliances.  And we found that there are some IP addresses which we found during our assessment Internet harvesting in this list.  And in our experience, there is no such thing as a luck.  We tried to find why such appliances can be so easily hacked and obviously the default password which hardcoded, sometimes never changed, was used on these appliances.  We tried to reach vendors and say, guys, maybe it's a bad idea to use, for instance, hardcoded SNMP, not hardcoded, but default community like public and again public for read write.  But they told us that SNMP is off by default, but still simple showdown search show that more than 200 users of this SD1, we enable this SNMP service and still use default password.  So we have a lot of tools which publish it in our GitHub and please contribute.  There are a lot of things to do with the new fingerprints for SD1 harvester and infiltrator with SD1 threat landscape description with new vulnerabilities and also it's like special CCC release.  We start to publish metasploit models for the SD1.  So we have public vulnerability description. You can create own models for it.  And conclusions.  So from my perspective how SD1 development like cycle works, someone come to us with brilliant idea, okay, let's build SD1 because Garner told us that it's like brilliant.  They have AI in the cloud.  So what we have to do, we can download bunch of open sources, put together, set up default routing things and after all use it as SD1.  So SD1 is a bunch of open source which not bad, but still you need care about it and you need install patches, configure in proper way and maintain.  These complex products have problem with patch management, have a lot of management interface like machine to machine and also human to machine interfaces.  Have a lot of weak defaults like password, hard coded certificate, PSK keys for the IPsec and many vendors unfortunately have issues with the patching responsible disclosure.  And this in the cloud.  So if you decide to switch your network to the SD1 hack it before buy, all you will fail.  So thank you for your attention. I want to ask you to give big hand to the SD1 new hope team to Dennis, Maxim, Nikolai, Nikita, Alec and Antoni who did most of the things here.  I just like a frontman of this group. Thank you so much guys.  Applause  Any questions?  Yep.  Aside from generic things that any random Linux server can have on web interface on SSH, have you looked at any specific SD1 security problems like with encapsulation of tunnels or some stuff like that?  So SD1, you know, there is no technology like SD1.  So for SDN there is kind of protocol which more or less reused in different vendor, in different solution and SD1 every vendor implement things in own way.  As an example, it's like Citric management protocol, etc.  One thing we did here, but we didn't publish it, is for the virtualization.  Because again this VNF story, very interesting because if you have vulnerability in any virtual function, next you can get access to our virtual machine.  But again, problem here, there is no standard and different vendors call VNF, so implement VNF in different way.  It can be QVM, it can be just a script which we upload to our appliance.  So you're saying because everyone is writing their own code, there are a lot of bugs to find for people who put the work in?  Yeah.  Okay.  And you can just, like, it's not necessary to try to buy things through the A-buy to hack it, you can just go to AWS and activate it for free.  Okay, Mike, one please.  Mike will be simple, there was a lot of vendors that you were looking in.  What about Juniper SD1?  Um, we're not all yet.  Looking forward for controlled investigation.  Thank you.  Good try.  And yes, please.  Okay, say, thanks for the talk.  You mentioned a lot of these virtual machines were running in hard Linux or something.  How do you, what are they running?  Is that good stuff?  What hash levels are they usually on?  These kernels?  Is it like always 2.6 or something like that?  Not always 2.6, it was like the worst example.  So, some of us like, you know, newer.  So, it's not necessary.  Again, for network function virtualization, some vendors, they call VNF, just a bunch of script with download and change configuration of defaults.  And this case we can use very old kernel.  Other way, use more recent version.  Is, is, sorry.  Yeah, that's okay.  And the hardening, how is it done?  Is it hardening and pulling experiments?  In most cases, no way.  So, no armor or things like this.  No Ger security or Linux.  Nothing.  Okay, thanks.  All right, your question.  Have you looked into Cisco, Muraki?  No, we, for Cisco, we, sorry, it's, it's night.  Let me find the list.  For Cisco, we did our exercise with the VipTeller.  All right, last question or there please.  Just advice, maybe you drop a slide in with the 90s code, they want the password back.  Sorry.  Because there, there were so many hard coded passwords everywhere.  Maybe you just should just drop a slide in the 90s code, they want the password or lock in the back.  I don't know. So it's public things.  So was that a question?  It's just recommendation.  All right, we have a few minutes left, but if there are no questions left, then I would call it a day.  So another warm round of applause for Saturday.  Thank you.
35c3-9446-sd-wan_a_new_hop_0_59.png 35c3-9446-sd-wan_a_new_hop_60_107.png 35c3-9446-sd-wan_a_new_hop_108_515.png 35c3-9446-sd-wan_a_new_hop_516_743.png 35c3-9446-sd-wan_a_new_hop_744_1031.png 35c3-9446-sd-wan_a_new_hop_1032_1151.png 35c3-9446-sd-wan_a_new_hop_1152_1607.png 35c3-9446-sd-wan_a_new_hop_1608_1691.png 35c3-9446-sd-wan_a_new_hop_1692_1739.png 35c3-9446-sd-wan_a_new_hop_1740_2315.png 35c3-9446-sd-wan_a_new_hop_2316_2531.png 35c3-9446-sd-wan_a_new_hop_2532_4391.png 35c3-9446-sd-wan_a_new_hop_4392_4487.png 35c3-9446-sd-wan_a_new_hop_4488_4967.png 35c3-9446-sd-wan_a_new_hop_4968_6563.png 35c3-9446-sd-wan_a_new_hop_6564_12167.png 35c3-9446-sd-wan_a_new_hop_12168_13151.png 35c3-9446-sd-wan_a_new_hop_13152_14243.png 35c3-9446-sd-wan_a_new_hop_14244_18251.png 35c3-9446-sd-wan_a_new_hop_18252_20603.png 35c3-9446-sd-wan_a_new_hop_20604_22763.png 35c3-9446-sd-wan_a_new_hop_22764_25187.png 35c3-9446-sd-wan_a_new_hop_25188_25235.png 35c3-9446-sd-wan_a_new_hop_25236_33167.png 35c3-9446-sd-wan_a_new_hop_33168_33359.png 35c3-9446-sd-wan_a_new_hop_33360_34715.png 35c3-9446-sd-wan_a_new_hop_34716_36671.png 35c3-9446-sd-wan_a_new_hop_36672_36731.png 35c3-9446-sd-wan_a_new_hop_36732_36827.png 35c3-9446-sd-wan_a_new_hop_36828_36923.png 35c3-9446-sd-wan_a_new_hop_36924_43655.png 35c3-9446-sd-wan_a_new_hop_43656_44543.png 35c3-9446-sd-wan_a_new_hop_44544_47063.png 35c3-9446-sd-wan_a_new_hop_47064_50855.png 35c3-9446-sd-wan_a_new_hop_50856_50939.png 35c3-9446-sd-wan_a_new_hop_50940_56399.png 35c3-9446-sd-wan_a_new_hop_56400_57275.png 35c3-9446-sd-wan_a_new_hop_57276_59219.png 35c3-9446-sd-wan_a_new_hop_59220_59483.png 35c3-9446-sd-wan_a_new_hop_59484_61415.png 35c3-9446-sd-wan_a_new_hop_61416_61775.png 35c3-9446-sd-wan_a_new_hop_61776_62675.png 35c3-9446-sd-wan_a_new_hop_62676_64655.png 35c3-9446-sd-wan_a_new_hop_64656_64703.png 35c3-9446-sd-wan_a_new_hop_64704_64763.png 35c3-9446-sd-wan_a_new_hop_64764_64823.png 35c3-9446-sd-wan_a_new_hop_64824_64967.png 35c3-9446-sd-wan_a_new_hop_64968_65507.png 35c3-9446-sd-wan_a_new_hop_65508_65915.png 35c3-9446-sd-wan_a_new_hop_65916_67091.png 35c3-9446-sd-wan_a_new_hop_67092_68039.png 35c3-9446-sd-wan_a_new_hop_68040_68195.png 35c3-9446-sd-wan_a_new_hop_68196_68243.png 35c3-9446-sd-wan_a_new_hop_68244_68999.png 35c3-9446-sd-wan_a_new_hop_69000_70811.png 35c3-9446-sd-wan_a_new_hop_70812_70859.png 35c3-9446-sd-wan_a_new_hop_70860_71435.png 35c3-9446-sd-wan_a_new_hop_71436_71579.png 35c3-9446-sd-wan_a_new_hop_71580_71663.png 35c3-9446-sd-wan_a_new_hop_71664_72767.png 35c3-9446-sd-wan_a_new_hop_72768_72875.png 35c3-9446-sd-wan_a_new_hop_72876_72923.png 35c3-9446-sd-wan_a_new_hop_72924_72983.png 35c3-9446-sd-wan_a_new_hop_72984_73043.png 35c3-9446-sd-wan_a_new_hop_73044_73139.png 35c3-9446-sd-wan_a_new_hop_73140_73271.png 35c3-9446-sd-wan_a_new_hop_73272_73439.png 35c3-9446-sd-wan_a_new_hop_73440_73591.png
Our next speaker is Sergei Godecic. Sergei has been doing security research, products and services for the past 15 years, since 2011 he is director and scriptwriter at Positive Hack Days Forum, the largest cybersecurity event in Eastern Europe. Sergei has for instance been working at Capersky Lab and Positive Technologies. He is also a visiting professor at Harbour Space University in Barcelona and leader of the SCADA Strange Lab, Industrial Cybersecurity Research Team. Today Sergei will talk about how to hack software to find networks and keeping your sanity while doing it. Let's give a warm round of applause for Sergei Godecic. Hello, hello, good night. Let's start to refresh our memories. This is a big honor for me to speak on the 35C3 because my first talk here was on the 29C3 with SCADA Strange Lab team and I think I can skip this introduction. Thanks for our host because everything is here and what I want to say about me, still I am very Russian, live in Abu Dhabi and do all this stuff because I saw this album in the airplane when I flight here. So accept Bitcoin only. So let's start to talk about software-defined networks. What is software-defined networks in general and is divine in particular case? It's a magic. So according to Gartner, it will kill impilates, it will replace all your Cisco and Juniper devices or Huawei if you prefer Chinese, but it's bad, you know, according to last news. And it will solve all your network problems because it has AI inside and it will magically optimize network operations and do everything including security. So because it's perfectly safe to implement inquiry networks, efficiently and security. So, okay, sounds good. What is actually software-defined networks? It's so simple. If you're familiar with the routers and switches, like it's just accept packets and send it somewhere according to the routing tables or the mock tables, but the software-defined networks is quite different. And when we tried with our team to understand how it works, our first impression was like this. We are hackers, we don't want to deal with this shit, but the only challenge we met before you hack something, you need to actually activate it and make it work. That is why we start to understand how software-defined networks and SD1 works. So what's the main difference between traditional LAN and SD1? In traditional LAN, you have different devices which solve very specific purpose, for instance, switches or routing or firewalling or network load balancing. In case of SD1, you have just a server which runs operation systems, in most cases it's like Linux. And on the top of this operation system, you have very specific models like CPE, which do specific network functions. It can be firewalling, it can be routing, it can be switching, it can be network load balancing. So you replace specific devices with one big server, which magically do everything with AI and in the cloud. Sure. So in the SD1, we have several layers. So low-level is a data plan when actually you process packets and decide how to go it, in which way how to firewall or drop it. We have control plan which manage different routers, different devices. We have management plan which can help to apply policies and orchestration plan because it's serious things that should have something which is called orchestration. But on the technical plan, again, we have hardware with operation system and a layer which called network function virtualization. What's network function virtualization? It's the way to apply different network functions to a specific device. It's very useful, for instance, to the network operators who will provide you with a specific box. And if you want to activate any functions, it can be web application, firewall or the sandboxes. We just upload very specific virtual machine. It can be Docker, it can be QVM image to your hardware and you can start to use it. Because inside of this box, you already have all the system infrastructure which process packets and pass it from one virtual network function to the other. This helps to organize things like the service chain-in. So you can distribute different network functions on the branch level, in the cloud level or on the HQ level. For instance, things like content filtering, which can be very heavy from performance perspective, can be distributed. As example, on the branch level, you can use simple things like the antivirus to process content. On the HQ level, you can use more heavy things like sandboxing. And if antivirus or specific rules see that this content is suspicious, it will forward it to HQ through the MPDS or other VPN and next processes in the HQ. Or you can also analyze it in the cloud through the simple things like the cloud threat intelligence where actually your SD1 box will send MD5 hash to the cloud and check is it good, contact, or bad. Or send all files to the cloud to double check it. Not bad. And I think that is why SD1 becomes more popular. And you can see that even military guys in US, they decide to switch to SD1 because of security, cost saving and all these benefits. Okay, security sounds very familiar for us and we decide to obviously hack it. I think most of you have experience in hacking of different network appliances and you know that sometimes you need to have complex things like an R&S soldering kit or debugger, jitagulator, etc. But not in this case. Because SD1 actually is not a appliance, it's a virtual appliance. And to start hacking, all you need is to go to the AWS or Azure and just activate this virtual appliance for, I don't know, 10 bucks per month. And next step is get root on it. It's a very good talk presented on different conferences including Zero Nights, how to hack virtual appliance and we use this like a checklist for our research. Because if you hack in virtual appliance, you already have access to this system. You can mount file system to the other virtual appliance. You can grab ETC shadow. You can find a lot of different backdoors just through the static analysis. But all the good things start from the Google. For instance, to find admin password for one of the SD1 appliance, we just Google for GitHub and found what most of the scripts which use it to automate with the appliance use a username administrator and password. Versa one to three. And actually, we found that this username and password is hard coded there because there is no way to change it. Next step to root it is just to Google for all vulnerabilities. For instance, in Silver Peak, we found that guys had reported vulnerability in September 2015 and it's still working in March 2018. So Google works because Google is full strong with this one. Next thing is, grab is always work as a string, you know. And using grab and code or code word password, you can find a lot of interesting things like hard coded password in different location in the configuration files in the database connection string. And then you can find a lot of interesting things like hard coded password in the system locks because, again, it's virtual appliance and someone had deployed before you start to use it. So in the locks, there are a lot of life interesting information. In the shadow file, like in one of Cisco appliance, ETC shadow file which use this encryption in 2018. You can do some forensic because, again, if you get virtual appliance, someone had deployed. And sometimes we're trying to hide this and you can see that cutting the best history. You can find that someone RAM scrub AWS shell script which actually set up different password, etc. So if you somehow can recover the script, you can find a lot of interesting information in this kind of password of admin users. And you can see that from this password you can find the hash and next try to brute force. It's just was my guess that maybe because Versa have password Versa123, maybe other network appliance like Silverpick. Have similar password Silverpick123 and this guess was successful. And it's good because you cannot stop the progress. If you have experience with red teaming in the enterprise network, you know that Cisco Cisco or Huawei with the administrator and Huawei123, it's quite common password. In this case, it's more complex thing sometimes where you lead like this tattery network stuff, you know. But still if you did not get the root with these simple steps, again, it's virtual appliance and you can always patch it. So you can change for instance, hash in the ETC shadow, you can change boot script, you can change remote management configuration, the password. And next boot in this configuration and get root password to do next step, security assessment. So security assessment, in the beginning we did in very, you know, not scientific way, we just hack all the things. But after all, we did some, let's say, scientific research and we have an article, I will give you a link like SD1 Threat Landscape with the step by step assessment what you should hack to get maximum results. But let's like mix this thing with funny hacks and the scientific approach. So from the system engineer point of view, SD1 have hardware, which of the shell hardware, operation system, in most cases again, of the shell Linux and different virtual services. Let's start from the operation system because again, everything you saw in the recent talk about BMC and remote management interfaces related to hardware, it still works here unless it is disabled by the vendor, but it's highly unlikely. On operation system, we did very simple research, we just check the patch level of the all components installed on this box. And you can see that patch level ridiculously old. For instance, all these things we found, it was an open SSL library, which was released in May 2006. It's for network devices with security functions. But our guess was that we choose this library because this library is too old to be vulnerable to heartbreak attack. And SD1 wins because the oldest open SSL library we found in commercial product wasn't before, wasn't Semantic WinCC, which was released in 2007. So SD1 is like old school really. Next thing related to operation system configuration is sudo. And it's actually everywhere and actually everywhere for management interfaces, including web services, shell, etc. And it's implemented in terrible way. As you can see, TRIP-W data have all ability to execute all command and some scripts just execute any command through the sudo. That is why if you have any small vulnerabilities in the web interfaces, like in this case, it's command injection, you can execute command with sudo. So it's again 1990s. Next point is software, not system, but software design point of view. It's from software design where a lot of open source component which implement IP stack, routing. But from management perspective, they use mostly HTTP and things around it. So let's analyze HTTP and web management interfaces. So in this case, it's not so old school like a system site. Everything based on Node.js and JavaScript, it's like very cool. But under the hood, you can find hard code mix from the Perl, Java, PHP, whatever, which looks like guys developed it last 10 years. With all this modern Node.js stuff, developers confuse the client and the server because you know JavaScript is on both sides and it's hard to understand where is server with client side. I will show you examples. And there are a lot of simple things like slow HTTP, those attacks, which should be fixed for a long time ago, but still you can stop web interface with few HTTP requests. So few examples about client side. JSON-C-SRF is everywhere, so almost no web interface implement protection from cross-site request forgery in proper way. XSS is everywhere and this is not a problem. So it's a response from the product manager of one vendor. They told me that XSS from cross-site scripting for web application is not the issue because Chrome blocks it. It's just an example of using XSS on such appliances. In this appliance, we can use combination of the XSS and cross-site request forgery to download and upload certificate which use it to authenticate with the server which like control plan with management server. It's just one HTTP request. And obviously there is no response from the vendor. We just silently fix it, so we decide to publish it through the full disclosure. One example of the perfect authentication. So in this case, you can see that it's client-side JavaScript which just send request to the server to the login status function. And if user is gogid, it's go to the request page. In our case, go to the username and password page. So this 100% client-side. No other checks on the server side just if you can change it. Second example is just perfect. So I think he tried to port the authentication from the server side to client side and understand what is JavaScript which still JavaScript but it doesn't work in browser. And he just like commented and say if username is with and password is with, then go home. So authentication is passed. So this thing. Highlighted box. So this authentication on this box. Next funny things which related to SD1 is about different privilege escalation. If you're already able to get access to any of virtual appliance inside, you can try to establish connection from these appliance to other appliance through the local host function. So why this can be interesting? Because there are a lot of open source component for the remote management. For instance, like challenge box which provide you like shell through the web interface or Moonin or Solar which like system management boxes. We just bind it to local host. So you cannot establish connection from outside because it's listening to the local host. But if you're already on these books and you can connect from these books to local host, then this establish this connection works. And this works because on each virtual, on which appliance you have a lot of virtual appliance which still listen to one IP addresses. If you have experience with the Docker for instance. So all Docker containers have all IP address. But from the all computer is actually connection from local host. So if you own any of the virtual appliance, you can own next all virtual machines installed here. This give us different interesting ways to escalate privilege inside the box. For instance, if you able to, let me switch to the laser pointer. You can see it, okay. If you can get access to traffic processor for instance for some TCP stack vulnerabilities. Next you can get access to management application and this management application in most cases have no any traffic filtering and trust to management application of all virtual network functions which run on these appliance. So you can do horizontal privilege escalation or next jump down to operation system level and next go to the management plan upstairs to the management appliances. But it's really boring to find the application vulnerabilities in such big amount of code. That is why we just download this code from the different network applies and drop it into the interactive code analysis system. In this case we use positive technologies application inspector and this help us to find a lot of vulnerabilities including such funny things like for instance, fully patched vulnerability in the Citrix SD1 which was patched in 2017. But it still works if you use not get HTTP request but post HTTP request. So we patched it once again. In this case better. Also it's a Udall lesson style vulnerability so it's obviously patched traversal but it's just reminded that attachments lead to the jealousy and the shadow of grid that is. So if you send attachment shadow you can get shadow file. So this is a full list of sorry full list of vulnerabilities we found during just source code analysis without you know brain interaction. Next step is crypto because security appliances shouldn't implement cryptography and there are two things is SSL TLS and type is that in most cases SSL TLS is used to protect management interfaces between the different appliances. And because it's automatic it's used different kind of automatic setup we found that there are a lot of things related to the unsecured configuration for instance we can use SSL TLS without forward secrecy. So if you have access to the certificate you can sniff off traffic even without full man in the middle different things related to the old Cypress use like a triple desk or RC4. For IPsec we found that in most cases they used very strange way to select the certificate or the precarious keys which in most cases just hard coded. So one example from real life our example we will publish soon from again Citrix Netscaller. These appliances use master control node protocol to communicate between the orchestration plan control plan and data plan. So let's run from the TCP 2156 and use TLS without forward secrecy and what interesting certificate located in the home TALERY user certificate and account WWW data have full access to this certificate. For some reason I don't know. Okay it should maybe read it but why to write. And what interesting all SD1 appliances we able to find during our security assessment we use same key pair which located in appliance edge. All SD1 appliances in the world used to protect communication between management and chain same key pair. So if you know this key pair and it's obviously you can just cut it from the file system you can passively or actively sniff traffic do manage the middle spoof management appliance. And if this device have any web application vulnerability you can overwrite it. I don't know why but maybe if in next turn we will change this certificate you can download old and do manage the middle again. Interesting stuff we found we run some tests from the DDoS attacks and found that Surikatan which use it in SD1 appliance are the ideas is vulnerable to regux DDoS. It's old story some regular expression can spend a lot of CPU time if you send specific queries. It's fixed in the default Surikatan kit but still working some modern SD1 solution. And for sure if you do some fuzzing it's always work and give you some fun. Unfortunately we cannot present reverse engineering part because most of such SD1 solution we have restriction in the licensing agreement to the reverse engineering. But just for fun I think that some of engineers way also love Star Wars and have Marvel. That is why in civilization functions cause Marvel sucks. So just an overview of detected vulnerabilities. So green is good or bad so good for vendor but for us we are unable to detect it. But you can see that most of classes of vulnerabilities like hard codes, broken access control, old products or Linux components or third party components were in most of such system you can find in most of such system. So just select any SD1 and make a shot. The interesting thing in SD1 is zero touch deployment. So it's very cool feature. For instance let's imagine that you have a branch office. You need to deploy branch office. With SD1 it's absolutely not necessary to go where or establish telnet or SSH connection and try to upload configuration. All you need is just to ship this device, note with unique ID, set up it through the cloud console and ship this device to remote location. This device will automatically connect to any internet it's around, connect to the cloud server, the node configuration and start operation. So for example how it works in the Citrix system. We have visa plans which ship it to the office. It's first try to establish connection with the surrounding the plans. If no, it's try to go to the zero touch deployment service, present own ID and next this service will provide all configuration which you upload through the SD1. So from security perspective this scheme looks terrible. Why? Because this SD1 and cloud deployment server should be friendly. But any attacker, not. If you know ID, if you can brute force this ID, you can pretend to be this device. If you have any weaknesses in implementation of this management server, you can own all devices which deployed through this service. And as you can see even Cisco which is like the best device from security perspective we found on this product line, let's say, have enough vulnerabilities. And this you can see that zero touch provision in command injection vulnerability. So it's cloud server which to which all network device should connect sometimes. But also we found very funny things related to the distribution of this device. Because as I told you in the beginning, most of such device can be activated as a cloud appliance through the AWS or the other cloud services. And we found that most of default images use old version with non vulnerabilities. So you go to AWS, you trust the vendor, you activate the system and you receive attack because there is no vulnerability. It's remind me in those story, I'm really old man, sorry. You know code red, this kind of key door, this kind of warms. And this real disaster when you just install fresh Windows 2000 which have Internet information server by default. And just connect it into Internet to download patches you receive new infection and need to reinstall it from the beginning. So these things look similar for me but is much worse because in this case is a security network device installed on the perimeter of your network. This overview of up to date statistics, you can see that very few vendors, actually no one of vendors have up to date version in the deployed on AWS. So I think it's abuse of the force. But it's also very interesting part of story as a security researchers, we always work in the responsible disclosure way. And as you can see, according to this article, some vendors, we also understand that responsible disclosure is very important to communicate with the user. It's important to communicate with community to fix issues and we even have product security incident response team in place. Great. But when we try to submit vulnerability report to this vendor, we unable to find the email of this product security incident response team. So there is no pool. We tried to Google for it a different way, no luck, but we found that guys who did similar research before, they found a great way, send email to CEO of this company. Unfortunately, my Google foo is not good enough, but I unable to find the mail, but I found this guy on the link it in. He answered it in few minutes actually and put me in contact. So if you try to deal with as the one vulnerability reporting just do this way. So we prepared this table about different vendors, how we communicate with researchers. And you can see that actually Cisco, Citrix and the account which actually VMware, not bad, but all the rest is just the beginning. This is my favorite mail from the one vendor. When we send notification, we start to ask them why we send this email from the Gmail. Do we have official ID, what they mean I need to present my passport or whatever to submit vulnerabilities. The funnest thing here, this vendor wrote that where device is not generic web service which have full Internet explosion. So after reading this email, I go sleep and during the night someone told me. So to understand the threat escape of SD1, we built a bunch of script which works on the top of the different search engines like census, showdown, Google. And use also Nmap, low scripting platform to fingerprint different SD1 solution. We have an article published as SD1 Internet sensors and also some tools which can generate beautiful maps if you want to present on the house communication congress because in other cases it's useless. And what we found that there are not so many SD1 devices yet in the Internet, about 3,000 management interface which contains no vulnerabilities and you can know it in a few minutes. And also we built some kind of vulnerability assessment tool which help you to find no vulnerabilities in these SD1 devices. This example for open SSH patch level, as you can see with like some CVs from 2010, 2014, etc. This open source, you can find it on the GitHub. We have two versions, one SD1 harvester which use Google showdown and census to collect information of all the all-verse Internet. And also SD1 infiltrator which is bunch of the network Nmap script engine scripts and you can use it during penetration testing. So it's not necessary to be connected to the Internet, you can just use it inside the network. When we did this research, we also found interesting article from Silence about dark web market when the bad guys sell user names in password to different network appliances, let's say enterprise level network appliances. And we found that there are some IP addresses which we found during our assessment Internet harvesting in this list. And in our experience, there is no such thing as a luck. We tried to find why such appliances can be so easily hacked and obviously the default password which hardcoded, sometimes never changed, was used on these appliances. We tried to reach vendors and say, guys, maybe it's a bad idea to use, for instance, hardcoded SNMP, not hardcoded, but default community like public and again public for read write. But they told us that SNMP is off by default, but still simple showdown search show that more than 200 users of this SD1, we enable this SNMP service and still use default password. So we have a lot of tools which publish it in our GitHub and please contribute. There are a lot of things to do with the new fingerprints for SD1 harvester and infiltrator with SD1 threat landscape description with new vulnerabilities and also it's like special CCC release. We start to publish metasploit models for the SD1. So we have public vulnerability description. You can create own models for it. And conclusions. So from my perspective how SD1 development like cycle works, someone come to us with brilliant idea, okay, let's build SD1 because Garner told us that it's like brilliant. They have AI in the cloud. So what we have to do, we can download bunch of open sources, put together, set up default routing things and after all use it as SD1. So SD1 is a bunch of open source which not bad, but still you need care about it and you need install patches, configure in proper way and maintain. These complex products have problem with patch management, have a lot of management interface like machine to machine and also human to machine interfaces. Have a lot of weak defaults like password, hard coded certificate, PSK keys for the IPsec and many vendors unfortunately have issues with the patching responsible disclosure. And this in the cloud. So if you decide to switch your network to the SD1 hack it before buy, all you will fail. So thank you for your attention. I want to ask you to give big hand to the SD1 new hope team to Dennis, Maxim, Nikolai, Nikita, Alec and Antoni who did most of the things here. I just like a frontman of this group. Thank you so much guys. Applause Any questions? Yep. Aside from generic things that any random Linux server can have on web interface on SSH, have you looked at any specific SD1 security problems like with encapsulation of tunnels or some stuff like that? So SD1, you know, there is no technology like SD1. So for SDN there is kind of protocol which more or less reused in different vendor, in different solution and SD1 every vendor implement things in own way. As an example, it's like Citric management protocol, etc. One thing we did here, but we didn't publish it, is for the virtualization. Because again this VNF story, very interesting because if you have vulnerability in any virtual function, next you can get access to our virtual machine. But again, problem here, there is no standard and different vendors call VNF, so implement VNF in different way. It can be QVM, it can be just a script which we upload to our appliance. So you're saying because everyone is writing their own code, there are a lot of bugs to find for people who put the work in? Yeah. Okay. And you can just, like, it's not necessary to try to buy things through the A-buy to hack it, you can just go to AWS and activate it for free. Okay, Mike, one please. Mike will be simple, there was a lot of vendors that you were looking in. What about Juniper SD1? Um, we're not all yet. Looking forward for controlled investigation. Thank you. Good try. And yes, please. Okay, say, thanks for the talk. You mentioned a lot of these virtual machines were running in hard Linux or something. How do you, what are they running? Is that good stuff? What hash levels are they usually on? These kernels? Is it like always 2.6 or something like that? Not always 2.6, it was like the worst example. So, some of us like, you know, newer. So, it's not necessary. Again, for network function virtualization, some vendors, they call VNF, just a bunch of script with download and change configuration of defaults. And this case we can use very old kernel. Other way, use more recent version. Is, is, sorry. Yeah, that's okay. And the hardening, how is it done? Is it hardening and pulling experiments? In most cases, no way. So, no armor or things like this. No Ger security or Linux. Nothing. Okay, thanks. All right, your question. Have you looked into Cisco, Muraki? No, we, for Cisco, we, sorry, it's, it's night. Let me find the list. For Cisco, we did our exercise with the VipTeller. All right, last question or there please. Just advice, maybe you drop a slide in with the 90s code, they want the password back. Sorry. Because there, there were so many hard coded passwords everywhere. Maybe you just should just drop a slide in the 90s code, they want the password or lock in the back. I don't know. So it's public things. So was that a question? It's just recommendation. All right, we have a few minutes left, but if there are no questions left, then I would call it a day. So another warm round of applause for Saturday. Thank you.
performance perspective, can be distributed. As have experience in hacking of different network through the static analysis. But all the good with the step by step assessment what you should approach. So from the system engineer point of based on Node.js and JavaScript, it's like very appliances. But it's really boring to find the for us we are unable to detect it. But you can office. You need to deploy branch office. With through the SD1. So from security perspective service. And as you can see even Cisco which is through the AWS or the other cloud services. And system and you receive attack because there is no installed on the perimeter of your network. This communicate with community to fix issues and we threat escape of SD1, we built a bunch of script find no vulnerabilities in these SD1 devices. that more than 200 users of this SD1, we enable PSK keys for the IPsec and many vendors hack it before buy, all you will fail. So thank know, there is no technology like SD1. So for SDN