UPnProxyPot: Fake the Funk, Become a Blackhat Proxy, MITM their TLS...
Seaman, Chad
UPnP sucks, everybody knows it, especially blackhat proxy operators. UPnProxyPot was developed to MITM these operators to see what they're doing with their IoT proxy networks and campaigns. We'll cover SSDP, UPnP, UPnProxy research/campaigns as well as cover a new Golang based honeypot, so we can all snoop on them together! REFERENCES:-hacks.org (OG disclosure)?v=FU6qX0-GHRU (DEF CON 19 talk I attended)-paper/upnproxy-blackhat-proxies-via-nat-injections-white-paper.pdf (my initial UPnProxy research)-eternalsilence.html
 Welcome to my talk. This is UP in Proxy Pot, fake the funk, become a black hat proxy, men in the middle of their TLS and scrape the wire. For begin, I'm Chad Seaman, but around here at DefCon you can just call me Dirt. I am part of the ACMA CERT team. I'm actually a team lead and senior engineer on that team. For those of you unfamiliar with the CERT team, which is probably all of you, you may have heard of some of our research before. We focus on DDoS and emerging threats research, but that typically leads us down the path of malware and botnets and proxies and other good stuff. So before we begin, you're going to see me talk about IoT. A lot of you are going to think IoT means Internet of Things. That's not true for me. It's the Internet of Trash. Whenever you see me say IoT, that's what I mean. So what's this talk about? Well, SSTP and UPnP have been widely vulnerable on IoT devices for nearly 20 years. It's not only possible, but also very easy to turn these devices into proxy servers. When attackers find vulnerable IoT devices susceptible to this kind of attack, they turn these devices into short-lived proxy server and delete their tracks when they're done. If they don't delete their tracks, the tracks will delete themselves. We're going to cover SSTP and UPnP, previous UPnP proxy research campaigns conducted by me, and finally UPnP proxy pot, how it works and findings from a year of geographically distributed deployments. So first things first, SSTP and UPnP. SSTP stands for Simple Service Discovery Protocol. It's a technology that's built for the land, uses broadcast addressing with HTTP over HTTP. It essentially allows machines on a land to announce themselves and or hear announcements from their neighbors or network peers and then expose them to UPnP, which will in turn expose services such as printing and media sharing and network configuration, all that kind of stuff. UPnP is universal plug and play. It's also built for the land. It's good old HTTP and SOAP, SOAP as in XML. It lets machines on a land inquire about the services configuration options offered by that device. It also allows them to access those services and or potentially modify those configurations. So a good example of this is an Xbox, right? An Xbox or a PC game may need you to forward certain ports or certain traffic around the firewall rather than deal with the state of managing that in the net. So that's what UPnP basically enables. It allows your Xbox to go forward and say, hey, poke a hole in the firewall, send everything on UDP, you know, one, two, three, four, over to me. So what is wrong with these technologies? Well, for SSTP, IoT devices are notoriously bad at deploying this correctly. The same is true for UPnP. It's built for the land, but they stick it on the WAN just because, I don't know, just because it was a reflected DDoS vector up and coming most popular MVP, King of the Hill, whatever you want to call it in 2014 and 2015. It's still fairly popular for that, but it's not as popular, mostly due to other vectors becoming more popular, not because it's gotten that much less abused. We're still finding this bullshit everywhere. Older products are still on the internet. Amazingly, newer products and new is there in quotes, but some newer products and by newer I mean within the past few years still have these problems. So 20 years later at a minimum, 14 years, and here we are still having the same old problems. So UPnP, Universal Plug and Play, once again, built for the land, but they seem to just love to stick this stuff on the WAN too. It treats the land as a safe space, which is fine, but the WAN is not a safe space. So when you're listening on the WAN and thinking it's a land, it's a little bit of a problem. It just does whatever it's trusted network peers to help to do, does not require off. There's not really a whole lot protecting it. Information disclosure, it will tell you everything, model numbers, makes, serial numbers. On top of that, it will tell you how to talk to it, what services it exposes, what configurations, what data you can push in and what data you can expect to get out. It facilitates configuration changes on some devices. It makes it very easy to do those changes. In some cases, there are known RCE injections in the UPnP daemons in the soap handling. So your basically soap can get remote command execution on the underlying device. So let's take a quick look at the history here and why I talked about 20 years. The first instance that I could find of somebody exposing something here is 2003, Bjorn Stickler. He came public with a net gear UPnP information disclosure. A couple years later, I'm going to slaughter this guy's name, Armin Hemel, I think, in 2006 gave a talk at the SANE conference and he launched a website called UPnPhacks.org. There's a ton of great info here. His talk really kind of blew the lid off of all the problems that UPnP actually had and all of the potential vectors it could expose people to. And then in 2011, Daniel Garcia gave a talk at DEF CON 19 called UPnP Mapping. It was a great talk. It kind of touched on this proxy and capability and some of the problems with UPnP. I was in the crowd. It freaked me out enough that I think my TP link crowd at the time was actually impacted by this and they went ahead and I went ahead and remoted into my home network and disabled UPnP from the talk while I was in the talk from my phone. So a brief history of UPnP proxy. So UPnP proxy in 2014, like I said, SSDP is the new up and coming DDoS vector starting to see abuse pretty widely. We were at that time we were known as the PLX cert. We were asked to write about it. Start digging into it and put on advisory and all that good stuff. So in 2015, this was happening at the end of 2014. In early 2015, the SSDP research leads me to discover UPnP and it kind of turns on that 2011 talk in my head. I'm like, oh man, I remember this being like a shit show. So in 2016, I decided that since it's been about a decade after the same conference talk and UPnP hacks, it might be fun to revisit this and see how bad this landscape is. Are we talking hundreds of thousands? Are we talking millions? And just talk about and kind of try and bring the fact that this is 10 years later, these things are still a problem and these threats still exist in the real world and everybody just kind of seemed to have forgotten about it. So I start writing that paper. The reason it's relevant is because I had to write a tool chain to test some of these theories and concepts. This is a tool chain here that was for testing the NAT injection capabilities on exposed UPnP devices. So in the top there, you see the SSDP banner that we get back. We take the 192.168.01. We change that to the public facing IP address that we found UPnP responding on. In our SOAP payload, we set that port 5555 is going to point into 192.168.01, which we know is the router at this point on port 80. We then issue that SOAP request via curl. And what we see here is before the injection and after the injection in the scan results there. TCP 80s filtered, you couldn't get to it. But once I opened TCP 55555 and then I hit it in a browser, I am greeted with the admin login page. So that's a little bit of a problem being able to get around the firewall that easy. As I'm doing this research in September of 2016, we get hit with a 620 gigabit per second sustained DDoS attack from a botnet. At that point, the botnet was unknown. It ultimately got named Marai. So as I'm digging into that, I'm inspecting attack sources. I'm seeing lots of IoT. There's a decent overlap with the existing identified UPnP dataset that I had from my decade after disclosure research. I decided that the UPnP info leaks could maybe help. And I start scraping those and poking these devices in general trying to figure out what the heck they are. So it turns out correlation is not causation. The fact that these devices were present in the Marai botnet has nothing to do with Marai. It's just that shitty devices are shitty. And if it's compromisable one way, it's probably compromisable about two or three in the Internet of Trash space that is. So having already written the script to dump the NAT tables as part of the NAT injection testing, I started doing that just to see, you know, maybe there's something weird going on in there that we could figure out. Like I said, it was not related at all. But what I did notice when I did that was there were some really weird entries in some of these devices out in the wild. The entries pointing to DNS servers. They pointed to Akamai CDN servers. They had been pointing at HTTP and HTTPS web servers, which is really interesting. But I have other shit to do. Got a really big botnet. I got to figure out what the hell is going on. So I kind of just stick that in the mental back burner and move on. So on the timeline here, we're down here at the Marai botnet in huge DDoS. So while I'm investigating that, I accidentally uncover the UPM proxy stuff, but I'm too busy dealing with this botnet. 2017 things start to calm down. Marai, at least I have tooling to be able to better track it and handle it. So I start looking back at some of my other research and I decide I'm going to look at what some of those really weird NAT entries were on some of those devices. And I began scanning the entire Internet and dumping all of the NAT tables of all of these exposed UPMP daemons. This is when we uncover the UPM proxy campaigns. So UPM proxy uncovered by the numbers. There were 4.8 million SSDP responders in that data set. 765,000 had exposed UPMP. It's roughly 16%. So of those, 65,000 were actively injected with UPMP entries. That's 9% of the total vulnerable population and 1.3% of the total responders. Of those, 17,599 unique endpoints were identified as being injected in these devices. Typically, if a device had one injection, it had multiples. The most injected destination had 18.8 million instances across 23,236 devices. The second most injected destination had 11 million instances across 59,943 devices. I point this out because it shows two kind of campaigns running simultaneously here. The most injected destination obviously had a lot more instances of injections across a much smaller pool of devices. And then the second most injected destination had a lot less injections but a much larger swath of devices that they were injected on. All in all, there were 15.9 million injections to DNS servers, 9.5 million injections to web servers, and 155,000 injections to HTTPS servers. While I'm doing this research, I'm talking to some fellow researchers and friends and one of the guys goes, hey, I think my friend's working on something very similar. Would you be interested in talking to them? Absolutely I was. I'm very sorry to the researcher I talked to. I don't remember your name and I can't find the email. And Symantec did not give you a shout out on their blog. So thank you for your hard work and I'm sorry. What they ultimately found was that there was an APT group and they were running this inception framework where the attackers were basically using these UP and proxy instances and they were chaining them together. So they would log into their VPS, they would inject a proxy route that pointed to another UP and proxy vulnerable device. They would then use that injection to inject another route, then use that injection to inject another route that ultimately pointed out to their target destination, which was a cloud storage provider for uploading their malware. And then they would use that to upload their malware to the cloud platform. And this is partly to get around detection, right? A lot of times you'll have these lists of known proxies, known endpoints, etc. And when you've got a pool of tens of thousands of home devices that aren't on any of those lists, you're much less likely to set off some alarm bells when you log in and upload a nasty file. So really interesting research. I gave him my tools, he was able to confirm it was what we thought it was. And then I was able to confirm in my data that I could see some of the similar clustering. So what you're seeing here in the graph on the right is two different bubble graphs. The size of the circle is respective to the number of outbound routes found on that device. And then every blue line is pointing in the direction of a relationship. There's an arrow, so where you see the thick blue on one side, that is the tips of the arrows running into one another. And there's clearly two different strategies. The top cluster, you have a larger pool with a handful of routes that go out, and they all point into a smaller pool of devices that may only route to one or two or three things. And in the bottom cluster, you see a centralized, high route out collection, and then they all point out to different endpoints. So it's just a different structure, a different strategy of building that chaining, but the chaining exists and I thought that was pretty cool. So we find all this stuff, but it's not super widespread. I'm just kidding, it's everywhere. So there's 73 brands and over 400 models that we could identify. And it's important there that I say we could identify, because we were only able to successfully fingerprint with confidence about 24% of these based on information leaks. And those information leaks weren't just what came from the UPMP Damage. Sure, it helped a ton because it's a super chatty, it exposes quite a bit of information about the device, like I said. But we would also go so far as to attempt to see what other reports were there, SSH banners, anything like that that we could potentially fingerprint on, we tried. And still, we can only get about 24%. So that is quite a considerable amount. 73 brands and 400 models is a nice chunk, but you got to remember that there's 76% that we couldn't even identify. So who knows what they are. This publication goes live. And you know, the crowd goes mild. Nobody really cared. It didn't really get a lot of attention. I was pretty disappointed. I thought it was a very cool finding. But and I'm downplaying that a little bit. So a couple people cared. The people that needed to care care. So I'll take that as a win. The research did get some industry attention through some some trust groups and work groups and stuff, did get elevated and passed along. And it was ultimately used to help some ISPs support the case internally for cleanup and, you know, sanitation efforts. So progress is made behind the scenes. Some networks start filtering SSDP. That's all good. You'll see that the result of that in the next couple slides. Ultimately, I get an email from a journalist. She's doing some work. And she was recording this new show. Her name is Justine Underhill. And she wanted to talk to me about UP and proxy because the episode she was recording was on IoT and security and everything else. So while we're recording this video, I decided that, you know, she's asking questions about how hard is this? How long does it take? How much does it cost? And I'm like, well, I'll just I'll just show you. So I pull out the laptop, jump on the internet, run Z map and I hit the first 1000 things that respond to my SSDP probe. And then I start dumping their NAT tables. And while I'm sitting there showing her this, I'm like, man, I think we just found something new. Like this, this wasn't in the previous scans. And this is how I accidentally discovered the eternal silence stuff, which it was cool because we, we didn't really have a solid case. I had proposed that attackers could use this to route around the firewall, but we didn't really have a solid proof from our existing scans that that was occurring yet. So these are what the injections look like after they go through our logging process and are converted into JSON. You can see that they are targeting IPs inside the land 192.168.10 space, probably from the information leak from the SSDP banner. And then so like there on 166, you can see that they tried to open a port forward to 139 and port 445. So they're injecting routes into the land space. And they are targeting Samba or SMB. We named it eternal silence because Samba and SMB are clearly being targeted by eternal blue pretty heavily at that point. And the Spanish there, there's in the new port mapping description, there's some Spanish, and I am terrible at Spanish. So my gringo Spanish is galete silenciosa, which go ahead and laugh at me. But it roughly translates to silent cookie. So you PMP eternal silence is discovered and published. Ultimately, 3.5 million SSDP responders. So some of that cleanup effort worked. We found almost a million less devices than we did in our previous research. 227,000 instances of exposed UPNP 45,000 had active eternal silence injections. There's no way to really know what they were up to. But based on what they were targeting, the eternal blue link is an educated guess. And that educated guess is based on if I were evil. That's what I would do, right? I've got this surefire SMB exploit. But everything that's running it on the internet has already been popped. But you know what? If I can find a way around some of these firewalls, I can probably find some devices that are still listening on that that haven't been patched. And now I've got a new place to drop my ransomware. So that's cool. All this research is cool. But we still have problems. The research up to this point, it has been via passive identification. This requires scanning the entire internet regularly to find stuff. It's time consuming. We get lots of hate mail and threats for scanning stuff. People don't like when you scan the internet. Guys, relax. Relax. You're allowed to scan the internet. It's not a crime. Okay. It's still time consuming. It results in a ton of logs because we're dumping all of these net tables. So it ends up with gigs and gigs and gigs of logs per scan. On top of that, it's very time sensitive, right? We know that the attackers can delete their entries. We know that the entries time out. So the odds that we're finding anything at all is pretty surprising, especially when you consider that that that reality. So the real problem here is that we can tell where they're doing stuff and where they're pointing stuff, but we don't actually have visibility into what they're doing with it. So if we see them injecting, you know, port 25, we assume it's spam, but we have no idea. They could be dropping O days against, you know, SNTP servers, no clue. So we need to fix that. And that's where UP and proxy pot kind of enters the fight, if you will. So what is UP and proxy pot, the 50,000 foot view? It listens for SSDP probes and it directs attackers into a fake UP and P instance. The UP and P and emulation is good enough to get to the injection phase. It's not a full implementation. It could be improved, but it's good enough. From there, we offer on the fly proxy capabilities with man in the middle content inspection and logging TLS stripping is also supported. All of this is easy to modify in the sense that if you want to pretend to be a different device, all you have to do is change some XML files and some text files on disk. It doesn't require code changes to change your device profile per se. So it offers session based P cap capabilities. So you can come back later and inspect the traffic that went over the sockets and it's written in go, Lange and bash. So SSDP emulation, the SSDP response that is currently in the project was lifted directly from the most abused device that we discovered during the UP and proxy research. It's stored in a flat file and disk. You can change it without modifying any code. The one gotcha is if you update the SSDP banner and it changes the port on which the UP and P Damon is listening on, this is what the attackers pivot on. So you will need to change the listening port in code that the UP and P Damon listens on. I didn't have a configuration file set up when I wrote this in my initial thing. It's an improvement that could be made. UP and P emulation. So the UP and P responses are lifted also from those same most abused devices. All the HTML and XML stored in flat files updating them requires no code changes. UP and P emulation serves basic files, handles NAT interactions. The attacker supplied SOAP is parsed and handled via Regex. It will respond with proper error payloads if criteria are not met or XML is malformed. Responses must contain attacker supplied data so that these responses use standard printf formatting. So if you need to change your thing and the attacker supplied port needs to be in this chunk of XML, you can just put the %d and it will be there. So on the fly proxy, this is kind of unique because the attacker gets to control their proxy configuration themselves. So we had to support that. So attackers submit their proxy config via SOAP just like they are talking to UP and P. We parse them and then create a session of sorts and then we scrape and log plain text across the proxy session in both directions. If they are proxying to TCP 443, it's a special use case and we assume that connection is a TLS connection and we do some special man in the middle of there. So stripping TLS and this is a hard slide to read. It's very upsetting. So attackers actually do some verification when they are using the TLS connections. The initial deployment saw connections but they would bail before actually pushing data across that connection. So we have to do some verification. Attackers are fingerprinting certs. Initially they were doing this via the subject line. There is an automated cloning process where we began by pulling the domain out of the client hello. We then go forward to the injected endpoint and we get the cert with the respective SNI that was provided in the client hello. We copy the remote cert and we mirror it into a self-signed clone cert and this all happens in real time when they first establish their connection. This allows us to, it was allowing us to bypass their fingerprinting and actually get plain text out of the TLS flows. Literally yesterday as I'm recording this, literally yesterday it stopped working and I don't know why. I don't know if they've changed their fingerprinting. I don't know what is really going on but I have a year and a half, well I have months worth of logs at this point that have this functional and now that it's about to go open source it breaks. So I'm sorry. I hope we can figure it out. So the other feature is the automated p-capping. The project uses GoPacket. It allows us to create p-caps on the fly using Berkeley packet filters that are scoped to the individual sessions, the individual proxy sessions. As attackers interact with the proxy injection the p-caps are automatically collected. If you find something interesting in the logs you can find the associated p-cap and see the entire session easily in whatever your favorite, you know, p-cap packet muncher is, wire shard, TCP dump, whatever. If you run out of disk space on your deployed honeypot this is probably why. That is from a single machine down there at the bottom. You can see that we had 81,100 p-caps that we ultimately collected for different sessions and of those p-caps it resulted in almost five and a half gigs of disk consumed. So this part hurts as well. The initial deployment was for one year and it was four nodes deployed across a single VPS provider. There were geos from Dallas to London to Tokyo. 300 gigs of p-caps and logs were ultimately collected. Hundreds of millions of captured proxy sessions and billions of log lines. And you know I downloaded all that and I destroyed the cluster and then I accidentally lost the backup. So I figured this out literally like a day after I submitted the CFP to DEF CON. Luckily I had a couple months before everything was accepted and approved. So I was able to deploy a smaller four node cluster for the two months between CFP and what you're seeing now. So ultimately four nodes deployed US, UK, India, Japan. 39 gigs of p-caps and logs collected 230,000 captured proxy sessions and 22 million lines of logs. The good news is I did have some notes so not everything was lost from the previous deployment and the trends that I saw in the new data are spot on for the trends I saw in the old data. There's not a whole lot's changed just a lot less data to back up the claims but I promise you it's pretty much identical. So observations. The first thing is that they don't blindly inject their their proxies. They actually come and do some testing. So injections they first come and they insert a test proxy instance. A test proxy instance. Once they confirm that it works then they inject a real proxy. They utilize it and then they attempt to delete it. I say attempt well we'll cover that. So this is the process of an injection. We see them show up with their msearch banner. We respond with our ssdp response here and you see that we point them to 192.168.01 on port 2048 and then etsylinuxigdgatedesk.xml. We see them come back within the same second and they then request that. You can see ssdpn and upnpn there. They request that and from there they think that we are the device they're looking for. Once they have confirmed that then they come back and they attempt to add their port mapping. So in this case they're adding an entry that will force us to listen to on port 22280. It's a tcp socket and any traffic received on that is going to be redirected to port 80 on the host at 74.6231.21. You can see down there the new port mapping description kind of mirrors the external port that they use that's sync and then a number and then the new lease duration is 600 seconds. So this will time out after 600 seconds. Then they come back and they utilize that newly injected proxy. So here you can see they sync 2280. Everything up there between the curly braces is the proxy configuration. We can see the source. We can see where ultimately they're going to point to. So 93191 3976 on port 57388 is going to send traffic to 74 6231 21 on port 80. Ultimately we intercept a get request to YAHU with no headers so it's super easy to spot and YAHU because they move to HTTPS ultimately issues a 301 permanently moved and this is all they really need for their fingerprint. Once that's done we see the attacker come back and they attempt to delete the port mapping but they send us malformed XML. What's interesting here is that the malformed XML apparently has I don't know if they forgot a null at the end of the envelope but it continues as a buffer over read and what we ultimately see here is XML that is not related to this request but it just happened to neighbor it in memory. What's interesting here in this case it's the same injection they just sent us. What isn't that interesting? What's more interesting and there are other instances where there is XML information leakage from the buffer over read from other devices that they may have been talking to recently. In this case at some point they were talking to a D-Link DSL 2730U. If you check that out you can see that it is a popular item on a popular e-commerce website. It's actually a choice item on that e-commerce website and it has 3100 readings. For 1,329 rupees you can buy this device and that's about $19 US I believe and you can inadvertently be a Black Hat Proxy too. These are some of the top talkers or sorry some of the top injected test endpoints. You can see there's Akamai Yahoo a few others in there. That top one is clearly the standout winner the 893910512 and they're going to ip.shtml. That's a special page. It returns your public-facing IP address which here I've clearly modified it and then this UBCIEG plug that they use for some kind of identification I'm assuming. There's also a very large campaign being run against Google. This is predominantly all the TLS traffic. It's very weird. I don't know what it is. Out of the 59,924 intercepted requests going across the TLS sockets all of them 100% targeted Google. This is click fraud SEO. I don't know what it is. This is an example of a caught request. They're searching for a Cisco spark board factory reset. We can see their accept language. We can see their cookies. The user agent they used all that good stuff. We can see even they seem to be coming from Dallas based on the information linkage in the URL but I can't really confirm that. I can't really confirm that. Also here we see the response. We get a 200 okay for their search. We see the cookies. It's not here but we would actually have the full page content and everything. It's disabled in this case because that was a lot of log lines. I mean gigs and gigs and gigs at Google pages. In total they sent 57,237 search terms. There are no really clear patterns. They're from all different geos that they target. They use a ton of different user agents. And each request gets basically one search per session. So you can see that top result there has only shown up 55 times out of 57,000 requests. It's just a search for the word Samsung in quotes which is weird. Some of the funnier searches that were captured 72-hour deodorant, antivirus download now, Marlboro summer camp, leather trousers outfit, fa fa fa fa slot hack. I don't even know what that is. I should probably Google some of these to see but I haven't yet. And like I said they're very geographically distributing their stuff across the Google platform. There are domains in here that I don't even know what country they affiliated with. So did you know there's a dot bj.as.jm.md.ee. They're clearly targeting Google.com the most followed by co.uk but still there's so many that they're hitting it's crazy. This is the user agent profiles. So they sent 293 different user agents. And then you can see there's almost normalized clusters of user agent distribution across the abuse. And these are some of the top talkers. So it's not what you'd expect, right? You would, I guess, I mean, I guess kind of it is what you'd expect if it's a single abuser. But the the nature of the queries almost make it seem organic. It doesn't seem like it has an abuse pattern. It almost seems like real end users. But then it's not real end users showing up and popping holes in this stuff. They're all being ferried through a handful of top talkers. And then you have your outliers. If we look at the top 10, we see that Worldstream, Worldstream, Worldstream, Worldstream, OVH, Worldstream, OVH, Worldstream, OVH and Avast. So let me just put it this way. If you work at Worldstream, find me at the bar. If you work at OVH, find me at the bar. If you work at Avast, find me at the bar and I'll buy your drink if you tell me what the hell is going on there. Because I don't know why Avast would be showing up in this data set, but there they are. So some theories on this. The queries to me seem too oddly human. They're in a bunch of different languages. They're stuff like, you know, the best car insurance in Dallas, Fort Worth area. Okay. So it's too organic to be just purely automated abuse, in my opinion. And I'm not sure that the people that are using these proxies are aware that they're using them. I have this theory that it may be some kind of residential proxy reseller or some kind of, you know, ultimate anonymous VPN service provider or something. And these people think that they're getting like these super secret, you know, high privacy stuff. And I'm just sitting here in the middle reading their traffic, which is a problem. So those are my theories. I'd love to hear more theories. You'll also intercept some other stuff outside of TLS. Here, for example, are some spam messages that were being routed. These guys were injecting outlook servers. And then, you know, you can watch the entire interaction, you can watch as they send their, their hello, you can see as they confirm different addresses are deliverable, and then they build their message and shoot it across. You get to see all of that. The good news here is that spam haas is doing God's work and stopping a lot of that abuse from actually succeeding. This was just a fun finding from the older data set. So while this project was going on, Belarus had a very tumultuous political event where a bunch of people went out and protested the recent election. And as a result, Belarus shut down their internet to news and political websites. And while that's going on, suddenly I started seeing these guys popping up in UP and proxy. So the top site was SB.by, which is a news website. And it looks like they were trying to get to the registration and then solve some captures. The other was photo belted up by which is a stock imaging host, I think, or something. But they were actually doing command, not command injections, SQL injections, which I found pretty funny. And then the third one here is mail.rec.gov.by. According to Google translate, this is the central commission of the Republic of Belarus elections or something along those lines. And they were just trying to check their mail. It looks like, and then on the bottom one here, we've got a news outlet, ont.by, and they're trying to get to what appears to be their exchange server. So I just found that kind of interesting. All right. So that's a lot of history, a lot of observations, but now comes the cool part. I'm open sourcing all of this. So anybody that wants to take this, stick it on the internet, play with it, modify it, whatever you want to do, have at it, it's yours. And this way we can all kind of share the fun and see what's going on with these campaigns. And if you find really cool stuff, I'd love to hear about it. So with that open source announcement, let's get some stuff out of the way. First things first, this project was for fun, it's for research. And it was for me to practice my Golang during COVID. Second, I apologize for my shitty code. I know it's shitty. I've learned more Golang since then and learned more design patterns in Golang since then. And I really understand how shitty my code is. I'm sorry. Like I said, it was a research project. It's not commercial grade software. It served its purpose and it did well enough to serve that purpose. And that's all I really needed out of it. Yes, there are bugs. Thank you for noticing. If you open an issue, there's a great chance I'm not going to address it. Maybe someone else in the community will, but this is not my top priority anymore. So I would encourage you to learn some Golang and maybe submit a pull request instead if you'd like to fix that bug. Yes, it's hacky. I know. I am a hacky developer. I'm not your enterprise leading, scrum running, everything else developer. So it is what it is. If you have ideas to fix or improve stuff, it's open source. Have at it, boss. Fork away, send pull requests, whatever. I'll likely accept a pull request. I will likely ignore your issue that you submit to the GitHub. So some ideas for improvements. If you want to hit the ground running, logging could be improved. Content injection could be a thing in a world where people are abusing this stuff. I imagine that you could stick JavaScript in pages or you could tamper with cookies or you could inject plugs of text that might be indexable that you could maybe turn up on a search engine later. I don't know. These are just some ideas I've had. There is a memory leak. I know. The run script actually restarts the binary every hour to get around this because I haven't had time to actually troubleshoot it. Yes, it runs on screen. I regret nothing. Feel free to properly demonize it if you'd like. This I think would be the biggest benefit. If you randomize the SSDP banners and listen on multiple popular exposed UPNP ports, I have a feeling you're going to see a much more diverse set of attackers show up. My findings may be myopic because I'm pretending to be a single device and that single device is what's being targeted by these people. If you were to diversify the target that you paint for your attacker, it's possible you will also diversify your findings. Some additional ideas for improvement. The cert caching. When I wrote this, the cert caching was not taking in S&I differences. It works on Google because all Google servers are just Google. If you were to say have an injection that pointed to some place that had multiple domains associated with it, your one cloned cert is only going to be the one that is aligned with that initial request. You could improve that by in the cert cache, actually using the S&I value, the domain name that was used when that cert was cloned, improved TLS handling, like I said, it was working. It has stopped working. Improving that would probably fix the problem. Improved cert cloning. Cloned more fields to better emulate the remote, the endpoint cert that you're trying to clone. Improved error handling because I didn't really handle any errors. Anything is an improvement. Improved basically everything else. You can find most of the information you'll need in the readme file. If there is anything I missed, please feel free to submit a pull request with the updates you found when deploying the stuff. It's written in Golang, but it does have Linux dependencies. It will run on any operating system so long as it's Linux. You can deploy a node in VPS very easily. You could also run it on Raspberry Pis or Odroid or anything else. Stick it on the DMZ and it will be good to go. Typically, you start to see abuse within the first 24 to 48 hours of deployment. It may be even lower than that. The last thing, if you find something cool, please hit me up on LinkedIn and let me know about it. I'd love to hear if you find new and interesting trends in some of your deployments. That wraps it up. Go grab the project, pull it down, hack it up, compile it, deploy it. Let me know what you find. Go have fun.
 Welcome to my talk. This is UP in Proxy Pot, fake the funk, become a black hat proxy,  men in the middle of their TLS and scrape the wire.  For begin, I'm Chad Seaman, but around here at DefCon you can just call me Dirt. I am  part of the ACMA CERT team. I'm actually a team lead and senior engineer on that team.  For those of you unfamiliar with the CERT team, which is probably all of you, you may  have heard of some of our research before. We focus on DDoS and emerging threats research,  but that typically leads us down the path of malware and botnets and proxies and other  good stuff.  So before we begin, you're going to see me talk about IoT. A lot of you are going to  think IoT means Internet of Things. That's not true for me. It's the Internet of Trash.  Whenever you see me say IoT, that's what I mean.  So what's this talk about? Well, SSTP and UPnP have been widely vulnerable on IoT devices  for nearly 20 years. It's not only possible, but also very easy to turn these devices into  proxy servers. When attackers find vulnerable IoT devices susceptible to this kind of attack,  they turn these devices into short-lived proxy server and delete their tracks when they're  done. If they don't delete their tracks, the tracks will delete themselves.  We're going to cover SSTP and UPnP, previous UPnP proxy research campaigns conducted by  me, and finally UPnP proxy pot, how it works and findings from a year of geographically  distributed deployments.  So first things first, SSTP and UPnP. SSTP stands for Simple Service Discovery Protocol.  It's a technology that's built for the land, uses broadcast addressing with HTTP over  HTTP. It essentially allows machines on a land to announce themselves and or hear announcements  from their neighbors or network peers and then expose them to UPnP, which will in turn  expose services such as printing and media sharing and network configuration, all that  kind of stuff. UPnP is universal plug and play. It's also built for the land. It's good old  HTTP and SOAP, SOAP as in XML. It lets machines on a land inquire about the services configuration  options offered by that device. It also allows them to access those services and or potentially  modify those configurations.  So a good example of this is an Xbox, right? An Xbox or a PC game may need you to forward  certain ports or certain traffic around the firewall rather than deal with the state of  managing that in the net. So that's what UPnP basically enables. It allows your Xbox to go  forward and say, hey, poke a hole in the firewall, send everything on UDP, you know, one, two,  three, four, over to me.  So what is wrong with these technologies? Well, for SSTP, IoT devices are notoriously  bad at deploying this correctly. The same is true for UPnP. It's built for the land,  but they stick it on the WAN just because, I don't know, just because it was a reflected  DDoS vector up and coming most popular MVP, King of the Hill, whatever you want to call  it in 2014 and 2015. It's still fairly popular for that, but it's not as popular, mostly  due to other vectors becoming more popular, not because it's gotten that much less abused.  We're still finding this bullshit everywhere. Older products are still on the internet.  Amazingly, newer products and new is there in quotes, but some newer products and by  newer I mean within the past few years still have these problems. So 20 years later at  a minimum, 14 years, and here we are still having the same old problems. So UPnP, Universal  Plug and Play, once again, built for the land, but they seem to just love to stick this stuff  on the WAN too. It treats the land as a safe space, which is fine, but the WAN is not a  safe space. So when you're listening on the WAN and thinking it's a land, it's a little  bit of a problem. It just does whatever it's trusted network peers to help to do, does  not require off. There's not really a whole lot protecting it. Information disclosure,  it will tell you everything, model numbers, makes, serial numbers. On top of that, it  will tell you how to talk to it, what services it exposes, what configurations, what data  you can push in and what data you can expect to get out. It facilitates configuration changes  on some devices. It makes it very easy to do those changes. In some cases, there are  known RCE injections in the UPnP daemons in the soap handling. So your basically soap  can get remote command execution on the underlying device. So let's take a quick look at the  history here and why I talked about 20 years. The first instance that I could find of somebody  exposing something here is 2003, Bjorn Stickler. He came public with a net gear UPnP information  disclosure. A couple years later, I'm going to slaughter this guy's name, Armin Hemel,  I think, in 2006 gave a talk at the SANE conference and he launched a website called  UPnPhacks.org. There's a ton of great info here. His talk really kind of blew the lid  off of all the problems that UPnP actually had and all of the potential vectors it could  expose people to. And then in 2011, Daniel Garcia gave a talk at DEF CON 19 called UPnP  Mapping. It was a great talk. It kind of touched on this proxy and capability and some of the  problems with UPnP. I was in the crowd. It freaked me out enough that I think my TP link  crowd at the time was actually impacted by this and they went ahead and I went ahead  and remoted into my home network and disabled UPnP from the talk while I was in the talk  from my phone. So a brief history of UPnP proxy. So UPnP proxy in 2014, like I said,  SSDP is the new up and coming DDoS vector starting to see abuse pretty widely. We  were at that time we were known as the PLX cert. We were asked to write about it. Start  digging into it and put on advisory and all that good stuff. So in 2015, this was happening  at the end of 2014. In early 2015, the SSDP research leads me to discover UPnP and it  kind of turns on that 2011 talk in my head. I'm like, oh man, I remember this being like  a shit show. So in 2016, I decided that since it's been about a decade after the same conference  talk and UPnP hacks, it might be fun to revisit this and see how bad this landscape is. Are  we talking hundreds of thousands? Are we talking millions? And just talk about and  kind of try and bring the fact that this is 10 years later, these things are still a  problem and these threats still exist in the real world and everybody just kind of seemed  to have forgotten about it. So I start writing that paper. The reason it's relevant is because  I had to write a tool chain to test some of these theories and concepts. This is a tool  chain here that was for testing the NAT injection capabilities on exposed UPnP devices. So in  the top there, you see the SSDP banner that we get back. We take the 192.168.01. We change  that to the public facing IP address that we found UPnP responding on. In our SOAP payload,  we set that port 5555 is going to point into 192.168.01, which we know is the router at  this point on port 80. We then issue that SOAP request via curl. And what we see here  is before the injection and after the injection in the scan results there. TCP 80s filtered,  you couldn't get to it. But once I opened TCP 55555 and then I hit it in a browser, I  am greeted with the admin login page. So that's a little bit of a problem being able to get  around the firewall that easy. As I'm doing this research in September of 2016, we get  hit with a 620 gigabit per second sustained DDoS attack from a botnet. At that point,  the botnet was unknown. It ultimately got named Marai. So as I'm digging into that,  I'm inspecting attack sources. I'm seeing lots of IoT. There's a decent overlap with  the existing identified UPnP dataset that I had from my decade after disclosure research.  I decided that the UPnP info leaks could maybe help. And I start scraping those and poking  these devices in general trying to figure out what the heck they are. So it turns out  correlation is not causation. The fact that these devices were present in the Marai botnet  has nothing to do with Marai. It's just that shitty devices are shitty. And if it's  compromisable one way, it's probably compromisable about two or three in the Internet of Trash  space that is. So having already written the script to dump the NAT tables as part of the  NAT injection testing, I started doing that just to see, you know, maybe there's something  weird going on in there that we could figure out. Like I said, it was not related at all.  But what I did notice when I did that was there were some really weird entries in some of  these devices out in the wild. The entries pointing to DNS servers. They pointed to Akamai  CDN servers. They had been pointing at HTTP and HTTPS web servers, which is really interesting.  But I have other shit to do. Got a really big botnet. I got to figure out what the hell  is going on. So I kind of just stick that in the mental back burner and move on. So  on the timeline here, we're down here at the Marai botnet in huge DDoS. So while I'm  investigating that, I accidentally uncover the UPM proxy stuff, but I'm too busy dealing  with this botnet. 2017 things start to calm down. Marai, at least I have tooling to be  able to better track it and handle it. So I start looking back at some of my other research  and I decide I'm going to look at what some of those really weird NAT entries were on  some of those devices. And I began scanning the entire Internet and dumping all of the  NAT tables of all of these exposed UPMP daemons. This is when we uncover the UPM proxy campaigns.  So UPM proxy uncovered by the numbers. There were 4.8 million SSDP responders in that data  set. 765,000 had exposed UPMP. It's roughly 16%. So of those, 65,000 were actively injected  with UPMP entries. That's 9% of the total vulnerable population and 1.3% of the total  responders. Of those, 17,599 unique endpoints were identified as being injected in these  devices. Typically, if a device had one injection, it had multiples. The most injected destination  had 18.8 million instances across 23,236 devices. The second most injected destination had 11  million instances across 59,943 devices. I point this out because it shows two kind  of campaigns running simultaneously here. The most injected destination obviously had  a lot more instances of injections across a much smaller pool of devices. And then the  second most injected destination had a lot less injections but a much larger swath of  devices that they were injected on. All in all, there were 15.9 million injections to  DNS servers, 9.5 million injections to web servers, and 155,000 injections to HTTPS  servers. While I'm doing this research, I'm talking to some fellow researchers and friends  and one of the guys goes, hey, I think my friend's working on something very similar.  Would you be interested in talking to them? Absolutely I was. I'm very sorry to the researcher  I talked to. I don't remember your name and I can't find the email. And Symantec did not  give you a shout out on their blog. So thank you for your hard work and I'm sorry. What  they ultimately found was that there was an APT group and they were running this inception  framework where the attackers were basically using these UP and proxy instances and they  were chaining them together. So they would log into their VPS, they would inject a proxy  route that pointed to another UP and proxy vulnerable device. They would then use that  injection to inject another route, then use that injection to inject another route that  ultimately pointed out to their target destination, which was a cloud storage provider for uploading  their malware. And then they would use that to upload their malware to the cloud platform.  And this is partly to get around detection, right? A lot of times you'll have these lists  of known proxies, known endpoints, etc. And when you've got a pool of tens of thousands  of home devices that aren't on any of those lists, you're much less likely to set off  some alarm bells when you log in and upload a nasty file. So really interesting research.  I gave him my tools, he was able to confirm it was what we thought it was. And then I  was able to confirm in my data that I could see some of the similar clustering. So what  you're seeing here in the graph on the right is two different bubble graphs. The size of  the circle is respective to the number of outbound routes found on that device. And  then every blue line is pointing in the direction of a relationship. There's an arrow, so where  you see the thick blue on one side, that is the tips of the arrows running into one another.  And there's clearly two different strategies. The top cluster, you have a larger pool with  a handful of routes that go out, and they all point into a smaller pool of devices that  may only route to one or two or three things. And in the bottom cluster, you see a centralized,  high route out collection, and then they all point out to different endpoints. So it's  just a different structure, a different strategy of building that chaining, but the chaining  exists and I thought that was pretty cool. So we find all this stuff, but it's not super  widespread. I'm just kidding, it's everywhere. So there's 73 brands and over 400 models that  we could identify. And it's important there that I say we could identify, because we were  only able to successfully fingerprint with confidence about 24% of these based on information  leaks. And those information leaks weren't just what came from the UPMP Damage. Sure,  it helped a ton because it's a super chatty, it exposes quite a bit of information about  the device, like I said. But we would also go so far as to attempt to see what other  reports were there, SSH banners, anything like that that we could potentially fingerprint  on, we tried. And still, we can only get about 24%. So that is quite a considerable amount.  73 brands and 400 models is a nice chunk, but you got to remember that there's 76% that  we couldn't even identify. So who knows what they are. This publication goes live. And  you know, the crowd goes mild. Nobody really cared. It didn't really get a lot of attention.  I was pretty disappointed. I thought it was a very cool finding. But and I'm downplaying  that a little bit. So a couple people cared. The people that needed to care care. So I'll  take that as a win. The research did get some industry attention through some some trust  groups and work groups and stuff, did get elevated and passed along. And it was ultimately  used to help some ISPs support the case internally for cleanup and, you know, sanitation efforts.  So progress is made behind the scenes. Some networks start filtering SSDP. That's all  good. You'll see that the result of that in the next couple slides. Ultimately, I get  an email from a journalist. She's doing some work. And she was recording this new show.  Her name is Justine Underhill. And she wanted to talk to me about UP and proxy because the  episode she was recording was on IoT and security and everything else. So while we're recording  this video, I decided that, you know, she's asking questions about how hard is this? How  long does it take? How much does it cost? And I'm like, well, I'll just I'll just show  you. So I pull out the laptop, jump on the internet, run Z map and I hit the first 1000  things that respond to my SSDP probe. And then I start dumping their NAT tables. And while  I'm sitting there showing her this, I'm like, man, I think we just found something new.  Like this, this wasn't in the previous scans. And this is how I accidentally discovered the  eternal silence stuff, which it was cool because we, we didn't really have a solid case. I  had proposed that attackers could use this to route around the firewall, but we didn't  really have a solid proof from our existing scans that that was occurring yet.  So these are what the injections look like after they go through our logging process  and are converted into JSON. You can see that they are targeting IPs inside the land 192.168.10  space, probably from the information leak from the SSDP banner. And then so like there  on 166, you can see that they tried to open a port forward to 139 and port 445. So they're  injecting routes into the land space. And they are targeting Samba or SMB. We named it eternal  silence because Samba and SMB are clearly being targeted by eternal blue pretty heavily at that  point. And the Spanish there, there's in the new port mapping description, there's some Spanish,  and I am terrible at Spanish. So my gringo Spanish is galete silenciosa, which go ahead and laugh  at me. But it roughly translates to silent cookie. So you PMP eternal silence is discovered and  published. Ultimately, 3.5 million SSDP responders. So some of that cleanup effort worked. We found  almost a million less devices than we did in our previous research. 227,000 instances of exposed  UPNP 45,000 had active eternal silence injections. There's no way to really know what they were up  to. But based on what they were targeting, the eternal blue link is an educated guess. And that  educated guess is based on if I were evil. That's what I would do, right? I've got this surefire  SMB exploit. But everything that's running it on the internet has already been popped. But you  know what? If I can find a way around some of these firewalls, I can probably find some devices  that are still listening on that that haven't been patched. And now I've got a new place to drop  my ransomware. So that's cool. All this research is cool. But we still have problems. The research  up to this point, it has been via passive identification. This requires scanning the entire  internet regularly to find stuff. It's time consuming. We get lots of hate mail and threats  for scanning stuff. People don't like when you scan the internet. Guys, relax. Relax. You're  allowed to scan the internet. It's not a crime. Okay. It's still time consuming. It results in a  ton of logs because we're dumping all of these net tables. So it ends up with gigs and gigs and  gigs of logs per scan. On top of that, it's very time sensitive, right? We know that the attackers  can delete their entries. We know that the entries time out. So the odds that we're finding anything  at all is pretty surprising, especially when you consider that that that reality. So the real  problem here is that we can tell where they're doing stuff and where they're pointing stuff,  but we don't actually have visibility into what they're doing with it. So if we see them injecting,  you know, port 25, we assume it's spam, but we have no idea. They could be dropping O days  against, you know, SNTP servers, no clue. So we need to fix that. And that's where UP and proxy  pot kind of enters the fight, if you will. So what is UP and proxy pot, the 50,000 foot view? It  listens for SSDP probes and it directs attackers into a fake UP and P instance. The UP and P  and emulation is good enough to get to the injection phase. It's not a full implementation.  It could be improved, but it's good enough. From there, we offer on the fly proxy capabilities  with man in the middle content inspection and logging TLS stripping is also supported. All of  this is easy to modify in the sense that if you want to pretend to be a different device,  all you have to do is change some XML files and some text files on disk. It doesn't require  code changes to change your device profile per se. So it offers session based P cap capabilities.  So you can come back later and inspect the traffic that went over the sockets and it's written in  go, Lange and bash. So SSDP emulation, the SSDP response that is currently in the project was  lifted directly from the most abused device that we discovered during the UP and proxy research.  It's stored in a flat file and disk. You can change it without modifying any code.  The one gotcha is if you update the SSDP banner and it changes the port on which the UP and P  Damon is listening on, this is what the attackers pivot on. So you will need to change the listening  port in code that the UP and P Damon listens on. I didn't have a configuration file set up when I  wrote this in my initial thing. It's an improvement that could be made. UP and P emulation. So the  UP and P responses are lifted also from those same most abused devices. All the HTML and XML  stored in flat files updating them requires no code changes. UP and P emulation serves basic files,  handles NAT interactions. The attacker supplied SOAP is parsed and handled via Regex. It will  respond with proper error payloads if criteria are not met or XML is malformed. Responses must  contain attacker supplied data so that these responses use standard printf formatting. So if  you need to change your thing and the attacker supplied port needs to be in this chunk of XML,  you can just put the %d and it will be there. So on the fly proxy, this is kind of unique  because the attacker gets to control their proxy configuration themselves. So we had to support  that. So attackers submit their proxy config via SOAP just like they are talking to UP and P.  We parse them and then create a session of sorts and then we scrape and log plain text  across the proxy session in both directions. If they are proxying to TCP 443, it's a special  use case and we assume that connection is a TLS connection and we do some special man in the  middle of there. So stripping TLS and this is a hard slide to read. It's very upsetting.  So attackers actually do some verification when they are using the TLS connections. The initial  deployment saw connections but they would bail before actually pushing data across that connection.  So we have to do some verification. Attackers are fingerprinting certs. Initially they were  doing this via the subject line. There is an automated cloning process where we began by  pulling the domain out of the client hello. We then go forward to the injected endpoint  and we get the cert with the respective SNI that was provided in the client hello. We copy the  remote cert and we mirror it into a self-signed clone cert and this all happens in real time when  they first establish their connection. This allows us to, it was allowing us to bypass their fingerprinting  and actually get plain text out of the TLS flows. Literally yesterday as I'm recording this, literally  yesterday it stopped working and I don't know why. I don't know if they've changed their fingerprinting.  I don't know what is really going on but I have a year and a half, well I have months worth of logs  at this point that have this functional and now that it's about to go open source it breaks. So  I'm sorry. I hope we can figure it out. So the other feature is the automated p-capping. The  project uses GoPacket. It allows us to create p-caps on the fly using Berkeley packet filters that  are scoped to the individual sessions, the individual proxy sessions. As attackers interact with the  proxy injection the p-caps are automatically collected. If you find something interesting  in the logs you can find the associated p-cap and see the entire session easily in whatever your  favorite, you know, p-cap packet muncher is, wire shard, TCP dump, whatever. If you run out of  disk space on your deployed honeypot this is probably why. That is from a single machine down  there at the bottom. You can see that we had 81,100 p-caps that we ultimately collected for  different sessions and of those p-caps it resulted in almost five and a half gigs of disk consumed.  So this part hurts as well. The initial deployment was for one year and it was four nodes  deployed across a single VPS provider. There were geos from Dallas to London to Tokyo. 300 gigs of  p-caps and logs were ultimately collected. Hundreds of millions of captured proxy sessions  and billions of log lines. And you know I downloaded all that and I destroyed the cluster  and then I accidentally lost the backup. So I figured this out literally like a day after  I submitted the CFP to DEF CON. Luckily I had a couple months before everything was accepted  and approved. So I was able to deploy a smaller four node cluster for the two months between CFP  and what you're seeing now. So ultimately four nodes deployed US, UK, India, Japan. 39 gigs of  p-caps and logs collected 230,000 captured proxy sessions and 22 million lines of logs.  The good news is I did have some notes so not everything was lost from the previous deployment  and the trends that I saw in the new data are spot on for the trends I saw in the old data.  There's not a whole lot's changed just a lot less data to back up the claims but I promise you  it's pretty much identical. So observations. The first thing is that they don't blindly inject  their their proxies. They actually come and do some testing. So injections they first come and  they insert a test proxy instance. A test proxy instance. Once they confirm that it works then  they inject a real proxy. They utilize it and then they attempt to delete it. I say attempt well  we'll cover that. So this is the process of an injection. We see them show up with their  msearch banner. We respond with our ssdp response here and you see that we point them to 192.168.01  on port 2048 and then etsylinuxigdgatedesk.xml. We see them come back within the same second  and they then request that. You can see ssdpn and upnpn there. They request that and from there  they think that we are the device they're looking for. Once they have confirmed that then they come  back and they attempt to add their port mapping. So in this case they're adding an entry that will  force us to listen to on port 22280. It's a tcp socket and any traffic received on that is going  to be redirected to port 80 on the host at 74.6231.21. You can see down there the new port mapping  description kind of mirrors the external port that they use that's sync and then a number  and then the new lease duration is 600 seconds. So this will time out after 600 seconds.  Then they come back and they utilize that newly injected proxy. So here you can see they sync  2280. Everything up there between the curly braces is the proxy configuration. We can see  the source. We can see where ultimately they're going to point to. So 93191 3976 on port 57388  is going to send traffic to 74 6231 21 on port 80. Ultimately we intercept a get request to YAHU  with no headers so it's super easy to spot and YAHU because they move to HTTPS ultimately  issues a 301 permanently moved and this is all they really need for their fingerprint. Once  that's done we see the attacker come back and they attempt to delete the port mapping but they  send us malformed XML. What's interesting here is that the malformed XML apparently has I don't  know if they forgot a null at the end of the envelope but it continues as a buffer over read  and what we ultimately see here is XML that is not related to this request but it just happened  to neighbor it in memory. What's interesting here in this case it's the same injection they just  sent us. What isn't that interesting? What's more interesting and there are other instances where  there is XML information leakage from the buffer over read from other devices that they may have  been talking to recently. In this case at some point they were talking to a D-Link DSL 2730U.  If you check that out you can see that it is a popular item on a popular e-commerce website.  It's actually a choice item on that e-commerce website and it has 3100 readings. For 1,329  rupees you can buy this device and that's about $19 US I believe and you can inadvertently be a  Black Hat Proxy too. These are some of the top talkers or sorry some of the top injected test  endpoints. You can see there's Akamai Yahoo a few others in there. That top one is clearly the  standout winner the 893910512 and they're going to ip.shtml. That's a special page. It returns your  public-facing IP address which here I've clearly modified it and then this UBCIEG plug that they  use for some kind of identification I'm assuming. There's also a very large campaign being run against  Google. This is predominantly all the TLS traffic. It's very weird. I don't know what it is. Out of  the 59,924 intercepted requests going across the TLS sockets all of them 100% targeted Google.  This is click fraud SEO. I don't know what it is. This is an example of a caught request. They're  searching for a Cisco spark board factory reset. We can see their accept language. We can see their  cookies. The user agent they used all that good stuff. We can see even they seem to be coming  from Dallas based on the information linkage in the URL but I can't really confirm that.  I can't really confirm that. Also here we see the response. We get a 200 okay for their search.  We see the cookies. It's not here but we would actually have the full page content and everything.  It's disabled in this case because that was a lot of log lines. I mean gigs and gigs and gigs at  Google pages. In total they sent 57,237 search terms. There are no really clear patterns.  They're from all different geos that they target. They use a ton of different user agents.  And each request gets basically one search per session. So you can see that top result there  has only shown up 55 times out of 57,000 requests. It's just a search for the word Samsung in quotes  which is weird. Some of the funnier searches that were captured 72-hour deodorant,  antivirus download now, Marlboro summer camp, leather trousers outfit,  fa fa fa fa slot hack. I don't even know what that is. I should probably Google some of these to see  but I haven't yet. And like I said they're very geographically distributing their stuff  across the Google platform. There are domains in here that I don't even know what country they  affiliated with. So did you know there's a dot bj.as.jm.md.ee. They're clearly targeting  Google.com the most followed by co.uk but still there's so many that they're hitting it's crazy.  This is the user agent profiles. So they sent 293 different user agents.  And then you can see there's almost normalized clusters of user agent distribution across the  abuse. And these are some of the top talkers. So it's not what you'd expect, right? You would,  I guess, I mean, I guess kind of it is what you'd expect if it's a single abuser. But the  the nature of the queries almost make it seem organic. It doesn't seem like it has an abuse  pattern. It almost seems like real end users. But then it's not real end users showing up and  popping holes in this stuff. They're all being ferried through a handful of top talkers. And then  you have your outliers. If we look at the top 10, we see that Worldstream, Worldstream, Worldstream,  Worldstream, OVH, Worldstream, OVH, Worldstream, OVH and Avast. So let me just put it this way.  If you work at Worldstream, find me at the bar. If you work at OVH, find me at the bar. If you  work at Avast, find me at the bar and I'll buy your drink if you tell me what the hell is going on  there. Because I don't know why Avast would be showing up in this data set, but there they are.  So some theories on this. The queries to me seem too oddly human. They're in a bunch of different  languages. They're stuff like, you know, the best car insurance in Dallas, Fort Worth area. Okay.  So it's too organic to be just purely automated abuse, in my opinion. And I'm not sure that  the people that are using these proxies are aware that they're using them. I have this  theory that it may be some kind of residential proxy reseller or some kind of, you know,  ultimate anonymous VPN service provider or something. And these people think that they're  getting like these super secret, you know, high privacy stuff. And I'm just sitting here in the  middle reading their traffic, which is a problem. So those are my theories. I'd love to hear more  theories. You'll also intercept some other stuff outside of TLS. Here, for example, are some spam  messages that were being routed. These guys were injecting outlook servers. And then, you know,  you can watch the entire interaction, you can watch as they send their, their hello,  you can see as they confirm different addresses are deliverable, and then they build their message  and shoot it across. You get to see all of that. The good news here is that spam haas is doing God's  work and stopping a lot of that abuse from actually succeeding. This was just a fun finding from the  older data set. So while this project was going on, Belarus had a very tumultuous  political event where a bunch of people went out and protested the recent election. And as a result,  Belarus shut down their internet to news and political websites. And while that's going on,  suddenly I started seeing these guys popping up in UP and proxy. So the top site was SB.by,  which is a news website. And it looks like they were trying to get to the registration and then  solve some captures. The other was photo belted up by which is a stock imaging host, I think,  or something. But they were actually doing command, not command injections, SQL injections,  which I found pretty funny. And then the third one here is mail.rec.gov.by. According to Google  translate, this is the central commission of the Republic of Belarus elections or something  along those lines. And they were just trying to check their mail. It looks like, and then on the  bottom one here, we've got a news outlet, ont.by, and they're trying to get to what appears to be  their exchange server. So I just found that kind of interesting. All right. So that's a lot of  history, a lot of observations, but now comes the cool part. I'm open sourcing all of this. So  anybody that wants to take this, stick it on the internet, play with it, modify it, whatever you  want to do, have at it, it's yours. And this way we can all kind of share the fun and see what's  going on with these campaigns. And if you find really cool stuff, I'd love to hear about it. So  with that open source announcement, let's get some stuff out of the way. First things first,  this project was for fun, it's for research. And it was for me to practice my Golang during COVID.  Second, I apologize for my shitty code. I know it's shitty. I've learned more Golang since then  and learned more design patterns in Golang since then. And I really understand how shitty my code  is. I'm sorry. Like I said, it was a research project. It's not commercial grade software.  It served its purpose and it did well enough to serve that purpose. And that's all I really needed  out of it. Yes, there are bugs. Thank you for noticing. If you open an issue, there's a great  chance I'm not going to address it. Maybe someone else in the community will, but this is not my  top priority anymore. So I would encourage you to learn some Golang and maybe submit a pull request  instead if you'd like to fix that bug. Yes, it's hacky. I know. I am a hacky developer. I'm not  your enterprise leading, scrum running, everything else developer. So it is what it is. If you have  ideas to fix or improve stuff, it's open source. Have at it, boss. Fork away, send pull requests,  whatever. I'll likely accept a pull request. I will likely ignore your issue that you submit to  the GitHub. So some ideas for improvements. If you want to hit the ground running, logging could  be improved. Content injection could be a thing in a world where people are abusing this stuff.  I imagine that you could stick JavaScript in pages or you could tamper with cookies or you  could inject plugs of text that might be indexable that you could maybe turn up on a search engine  later. I don't know. These are just some ideas I've had. There is a memory leak. I know. The  run script actually restarts the binary every hour to get around this because I haven't had  time to actually troubleshoot it. Yes, it runs on screen. I regret nothing. Feel free to properly  demonize it if you'd like. This I think would be the biggest benefit. If you randomize the  SSDP banners and listen on multiple popular exposed UPNP ports, I have a feeling you're  going to see a much more diverse set of attackers show up. My findings may be myopic because I'm  pretending to be a single device and that single device is what's being targeted by these people.  If you were to diversify the target that you paint for your attacker, it's possible you will  also diversify your findings. Some additional ideas for improvement. The cert caching. When I  wrote this, the cert caching was not taking in S&I differences. It works on Google because  all Google servers are just Google. If you were to say have an injection that pointed to some  place that had multiple domains associated with it, your one cloned cert is only going to be the  one that is aligned with that initial request. You could improve that by in the cert cache,  actually using the S&I value, the domain name that was used when that cert was cloned,  improved TLS handling, like I said, it was working. It has stopped working. Improving that would  probably fix the problem. Improved cert cloning. Cloned more fields to better emulate the remote,  the endpoint cert that you're trying to clone. Improved error handling because I didn't really  handle any errors. Anything is an improvement. Improved basically everything else.  You can find most of the information you'll need in the readme file. If there is anything I missed,  please feel free to submit a pull request with the updates you found when deploying the stuff.  It's written in Golang, but it does have Linux dependencies. It will run on any operating  system so long as it's Linux. You can deploy a node in VPS very easily. You could also run it on  Raspberry Pis or Odroid or anything else. Stick it on the DMZ and it will be good to go.  Typically, you start to see abuse within the first 24 to 48 hours of deployment.  It may be even lower than that. The last thing, if you find something cool,  please hit me up on LinkedIn and let me know about it. I'd love to hear if you  find new and interesting trends in some of your deployments. That wraps it up.  Go grab the project, pull it down, hack it up, compile it, deploy it. Let me know what you find.  Go have fun.
DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_0_251.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_252_395.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_396_563.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_564_935.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_936_1139.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_1140_1295.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_1296_2267.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_2268_4343.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_4344_7679.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_7680_9599.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_9600_11615.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_11616_13259.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_13260_15971.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_15972_16979.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_16980_19127.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_19128_23171.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_23172_24743.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_24744_25115.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_25116_26183.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_26184_27743.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_27744_29363.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_29364_29483.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_29484_30779.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_30780_32723.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_32724_32867.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_32868_34211.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_34212_35279.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_35280_36455.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_36456_37295.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_37296_39515.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_39516_40895.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_40896_41603.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_41604_41903.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_41904_42635.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_42636_43211.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_43212_43895.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_43896_44843.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_44844_45863.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_45864_45959.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_45960_46007.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_46008_46079.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_46080_46283.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_46284_46331.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_46332_46499.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_46500_46583.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_46584_46727.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_46728_46775.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_46776_46835.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_46836_46991.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_46992_47867.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_47868_48851.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_48852_49319.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_49320_49643.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_49644_50171.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_50172_50711.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_50712_51191.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_51192_51911.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_51912_52391.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_52392_53327.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_53328_53651.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_53652_54443.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_54444_55151.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_55152_56423.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_56424_57011.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_57012_57215.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_57216_59255.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_59256_59375.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_59376_59855.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_59856_62027.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_62028_63959.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_63960_65591.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_65592_66827.png DEF_CON_29_-_Chad_Seaman_-_UPnProxyPot__Fake_the_F_mHCGNUsrTf0_66828_67540.png
Welcome to my talk. This is UP in Proxy Pot, fake the funk, become a black hat proxy, men in the middle of their TLS and scrape the wire. For begin, I'm Chad Seaman, but around here at DefCon you can just call me Dirt. I am part of the ACMA CERT team. I'm actually a team lead and senior engineer on that team. For those of you unfamiliar with the CERT team, which is probably all of you, you may have heard of some of our research before. We focus on DDoS and emerging threats research, but that typically leads us down the path of malware and botnets and proxies and other good stuff. So before we begin, you're going to see me talk about IoT. A lot of you are going to think IoT means Internet of Things. That's not true for me. It's the Internet of Trash. Whenever you see me say IoT, that's what I mean. So what's this talk about? Well, SSTP and UPnP have been widely vulnerable on IoT devices for nearly 20 years. It's not only possible, but also very easy to turn these devices into proxy servers. When attackers find vulnerable IoT devices susceptible to this kind of attack, they turn these devices into short-lived proxy server and delete their tracks when they're done. If they don't delete their tracks, the tracks will delete themselves. We're going to cover SSTP and UPnP, previous UPnP proxy research campaigns conducted by me, and finally UPnP proxy pot, how it works and findings from a year of geographically distributed deployments. So first things first, SSTP and UPnP. SSTP stands for Simple Service Discovery Protocol. It's a technology that's built for the land, uses broadcast addressing with HTTP over HTTP. It essentially allows machines on a land to announce themselves and or hear announcements from their neighbors or network peers and then expose them to UPnP, which will in turn expose services such as printing and media sharing and network configuration, all that kind of stuff. UPnP is universal plug and play. It's also built for the land. It's good old HTTP and SOAP, SOAP as in XML. It lets machines on a land inquire about the services configuration options offered by that device. It also allows them to access those services and or potentially modify those configurations. So a good example of this is an Xbox, right? An Xbox or a PC game may need you to forward certain ports or certain traffic around the firewall rather than deal with the state of managing that in the net. So that's what UPnP basically enables. It allows your Xbox to go forward and say, hey, poke a hole in the firewall, send everything on UDP, you know, one, two, three, four, over to me. So what is wrong with these technologies? Well, for SSTP, IoT devices are notoriously bad at deploying this correctly. The same is true for UPnP. It's built for the land, but they stick it on the WAN just because, I don't know, just because it was a reflected DDoS vector up and coming most popular MVP, King of the Hill, whatever you want to call it in 2014 and 2015. It's still fairly popular for that, but it's not as popular, mostly due to other vectors becoming more popular, not because it's gotten that much less abused. We're still finding this bullshit everywhere. Older products are still on the internet. Amazingly, newer products and new is there in quotes, but some newer products and by newer I mean within the past few years still have these problems. So 20 years later at a minimum, 14 years, and here we are still having the same old problems. So UPnP, Universal Plug and Play, once again, built for the land, but they seem to just love to stick this stuff on the WAN too. It treats the land as a safe space, which is fine, but the WAN is not a safe space. So when you're listening on the WAN and thinking it's a land, it's a little bit of a problem. It just does whatever it's trusted network peers to help to do, does not require off. There's not really a whole lot protecting it. Information disclosure, it will tell you everything, model numbers, makes, serial numbers. On top of that, it will tell you how to talk to it, what services it exposes, what configurations, what data you can push in and what data you can expect to get out. It facilitates configuration changes on some devices. It makes it very easy to do those changes. In some cases, there are known RCE injections in the UPnP daemons in the soap handling. So your basically soap can get remote command execution on the underlying device. So let's take a quick look at the history here and why I talked about 20 years. The first instance that I could find of somebody exposing something here is 2003, Bjorn Stickler. He came public with a net gear UPnP information disclosure. A couple years later, I'm going to slaughter this guy's name, Armin Hemel, I think, in 2006 gave a talk at the SANE conference and he launched a website called UPnPhacks.org. There's a ton of great info here. His talk really kind of blew the lid off of all the problems that UPnP actually had and all of the potential vectors it could expose people to. And then in 2011, Daniel Garcia gave a talk at DEF CON 19 called UPnP Mapping. It was a great talk. It kind of touched on this proxy and capability and some of the problems with UPnP. I was in the crowd. It freaked me out enough that I think my TP link crowd at the time was actually impacted by this and they went ahead and I went ahead and remoted into my home network and disabled UPnP from the talk while I was in the talk from my phone. So a brief history of UPnP proxy. So UPnP proxy in 2014, like I said, SSDP is the new up and coming DDoS vector starting to see abuse pretty widely. We were at that time we were known as the PLX cert. We were asked to write about it. Start digging into it and put on advisory and all that good stuff. So in 2015, this was happening at the end of 2014. In early 2015, the SSDP research leads me to discover UPnP and it kind of turns on that 2011 talk in my head. I'm like, oh man, I remember this being like a shit show. So in 2016, I decided that since it's been about a decade after the same conference talk and UPnP hacks, it might be fun to revisit this and see how bad this landscape is. Are we talking hundreds of thousands? Are we talking millions? And just talk about and kind of try and bring the fact that this is 10 years later, these things are still a problem and these threats still exist in the real world and everybody just kind of seemed to have forgotten about it. So I start writing that paper. The reason it's relevant is because I had to write a tool chain to test some of these theories and concepts. This is a tool chain here that was for testing the NAT injection capabilities on exposed UPnP devices. So in the top there, you see the SSDP banner that we get back. We take the 192.168.01. We change that to the public facing IP address that we found UPnP responding on. In our SOAP payload, we set that port 5555 is going to point into 192.168.01, which we know is the router at this point on port 80. We then issue that SOAP request via curl. And what we see here is before the injection and after the injection in the scan results there. TCP 80s filtered, you couldn't get to it. But once I opened TCP 55555 and then I hit it in a browser, I am greeted with the admin login page. So that's a little bit of a problem being able to get around the firewall that easy. As I'm doing this research in September of 2016, we get hit with a 620 gigabit per second sustained DDoS attack from a botnet. At that point, the botnet was unknown. It ultimately got named Marai. So as I'm digging into that, I'm inspecting attack sources. I'm seeing lots of IoT. There's a decent overlap with the existing identified UPnP dataset that I had from my decade after disclosure research. I decided that the UPnP info leaks could maybe help. And I start scraping those and poking these devices in general trying to figure out what the heck they are. So it turns out correlation is not causation. The fact that these devices were present in the Marai botnet has nothing to do with Marai. It's just that shitty devices are shitty. And if it's compromisable one way, it's probably compromisable about two or three in the Internet of Trash space that is. So having already written the script to dump the NAT tables as part of the NAT injection testing, I started doing that just to see, you know, maybe there's something weird going on in there that we could figure out. Like I said, it was not related at all. But what I did notice when I did that was there were some really weird entries in some of these devices out in the wild. The entries pointing to DNS servers. They pointed to Akamai CDN servers. They had been pointing at HTTP and HTTPS web servers, which is really interesting. But I have other shit to do. Got a really big botnet. I got to figure out what the hell is going on. So I kind of just stick that in the mental back burner and move on. So on the timeline here, we're down here at the Marai botnet in huge DDoS. So while I'm investigating that, I accidentally uncover the UPM proxy stuff, but I'm too busy dealing with this botnet. 2017 things start to calm down. Marai, at least I have tooling to be able to better track it and handle it. So I start looking back at some of my other research and I decide I'm going to look at what some of those really weird NAT entries were on some of those devices. And I began scanning the entire Internet and dumping all of the NAT tables of all of these exposed UPMP daemons. This is when we uncover the UPM proxy campaigns. So UPM proxy uncovered by the numbers. There were 4.8 million SSDP responders in that data set. 765,000 had exposed UPMP. It's roughly 16%. So of those, 65,000 were actively injected with UPMP entries. That's 9% of the total vulnerable population and 1.3% of the total responders. Of those, 17,599 unique endpoints were identified as being injected in these devices. Typically, if a device had one injection, it had multiples. The most injected destination had 18.8 million instances across 23,236 devices. The second most injected destination had 11 million instances across 59,943 devices. I point this out because it shows two kind of campaigns running simultaneously here. The most injected destination obviously had a lot more instances of injections across a much smaller pool of devices. And then the second most injected destination had a lot less injections but a much larger swath of devices that they were injected on. All in all, there were 15.9 million injections to DNS servers, 9.5 million injections to web servers, and 155,000 injections to HTTPS servers. While I'm doing this research, I'm talking to some fellow researchers and friends and one of the guys goes, hey, I think my friend's working on something very similar. Would you be interested in talking to them? Absolutely I was. I'm very sorry to the researcher I talked to. I don't remember your name and I can't find the email. And Symantec did not give you a shout out on their blog. So thank you for your hard work and I'm sorry. What they ultimately found was that there was an APT group and they were running this inception framework where the attackers were basically using these UP and proxy instances and they were chaining them together. So they would log into their VPS, they would inject a proxy route that pointed to another UP and proxy vulnerable device. They would then use that injection to inject another route, then use that injection to inject another route that ultimately pointed out to their target destination, which was a cloud storage provider for uploading their malware. And then they would use that to upload their malware to the cloud platform. And this is partly to get around detection, right? A lot of times you'll have these lists of known proxies, known endpoints, etc. And when you've got a pool of tens of thousands of home devices that aren't on any of those lists, you're much less likely to set off some alarm bells when you log in and upload a nasty file. So really interesting research. I gave him my tools, he was able to confirm it was what we thought it was. And then I was able to confirm in my data that I could see some of the similar clustering. So what you're seeing here in the graph on the right is two different bubble graphs. The size of the circle is respective to the number of outbound routes found on that device. And then every blue line is pointing in the direction of a relationship. There's an arrow, so where you see the thick blue on one side, that is the tips of the arrows running into one another. And there's clearly two different strategies. The top cluster, you have a larger pool with a handful of routes that go out, and they all point into a smaller pool of devices that may only route to one or two or three things. And in the bottom cluster, you see a centralized, high route out collection, and then they all point out to different endpoints. So it's just a different structure, a different strategy of building that chaining, but the chaining exists and I thought that was pretty cool. So we find all this stuff, but it's not super widespread. I'm just kidding, it's everywhere. So there's 73 brands and over 400 models that we could identify. And it's important there that I say we could identify, because we were only able to successfully fingerprint with confidence about 24% of these based on information leaks. And those information leaks weren't just what came from the UPMP Damage. Sure, it helped a ton because it's a super chatty, it exposes quite a bit of information about the device, like I said. But we would also go so far as to attempt to see what other reports were there, SSH banners, anything like that that we could potentially fingerprint on, we tried. And still, we can only get about 24%. So that is quite a considerable amount. 73 brands and 400 models is a nice chunk, but you got to remember that there's 76% that we couldn't even identify. So who knows what they are. This publication goes live. And you know, the crowd goes mild. Nobody really cared. It didn't really get a lot of attention. I was pretty disappointed. I thought it was a very cool finding. But and I'm downplaying that a little bit. So a couple people cared. The people that needed to care care. So I'll take that as a win. The research did get some industry attention through some some trust groups and work groups and stuff, did get elevated and passed along. And it was ultimately used to help some ISPs support the case internally for cleanup and, you know, sanitation efforts. So progress is made behind the scenes. Some networks start filtering SSDP. That's all good. You'll see that the result of that in the next couple slides. Ultimately, I get an email from a journalist. She's doing some work. And she was recording this new show. Her name is Justine Underhill. And she wanted to talk to me about UP and proxy because the episode she was recording was on IoT and security and everything else. So while we're recording this video, I decided that, you know, she's asking questions about how hard is this? How long does it take? How much does it cost? And I'm like, well, I'll just I'll just show you. So I pull out the laptop, jump on the internet, run Z map and I hit the first 1000 things that respond to my SSDP probe. And then I start dumping their NAT tables. And while I'm sitting there showing her this, I'm like, man, I think we just found something new. Like this, this wasn't in the previous scans. And this is how I accidentally discovered the eternal silence stuff, which it was cool because we, we didn't really have a solid case. I had proposed that attackers could use this to route around the firewall, but we didn't really have a solid proof from our existing scans that that was occurring yet. So these are what the injections look like after they go through our logging process and are converted into JSON. You can see that they are targeting IPs inside the land 192.168.10 space, probably from the information leak from the SSDP banner. And then so like there on 166, you can see that they tried to open a port forward to 139 and port 445. So they're injecting routes into the land space. And they are targeting Samba or SMB. We named it eternal silence because Samba and SMB are clearly being targeted by eternal blue pretty heavily at that point. And the Spanish there, there's in the new port mapping description, there's some Spanish, and I am terrible at Spanish. So my gringo Spanish is galete silenciosa, which go ahead and laugh at me. But it roughly translates to silent cookie. So you PMP eternal silence is discovered and published. Ultimately, 3.5 million SSDP responders. So some of that cleanup effort worked. We found almost a million less devices than we did in our previous research. 227,000 instances of exposed UPNP 45,000 had active eternal silence injections. There's no way to really know what they were up to. But based on what they were targeting, the eternal blue link is an educated guess. And that educated guess is based on if I were evil. That's what I would do, right? I've got this surefire SMB exploit. But everything that's running it on the internet has already been popped. But you know what? If I can find a way around some of these firewalls, I can probably find some devices that are still listening on that that haven't been patched. And now I've got a new place to drop my ransomware. So that's cool. All this research is cool. But we still have problems. The research up to this point, it has been via passive identification. This requires scanning the entire internet regularly to find stuff. It's time consuming. We get lots of hate mail and threats for scanning stuff. People don't like when you scan the internet. Guys, relax. Relax. You're allowed to scan the internet. It's not a crime. Okay. It's still time consuming. It results in a ton of logs because we're dumping all of these net tables. So it ends up with gigs and gigs and gigs of logs per scan. On top of that, it's very time sensitive, right? We know that the attackers can delete their entries. We know that the entries time out. So the odds that we're finding anything at all is pretty surprising, especially when you consider that that that reality. So the real problem here is that we can tell where they're doing stuff and where they're pointing stuff, but we don't actually have visibility into what they're doing with it. So if we see them injecting, you know, port 25, we assume it's spam, but we have no idea. They could be dropping O days against, you know, SNTP servers, no clue. So we need to fix that. And that's where UP and proxy pot kind of enters the fight, if you will. So what is UP and proxy pot, the 50,000 foot view? It listens for SSDP probes and it directs attackers into a fake UP and P instance. The UP and P and emulation is good enough to get to the injection phase. It's not a full implementation. It could be improved, but it's good enough. From there, we offer on the fly proxy capabilities with man in the middle content inspection and logging TLS stripping is also supported. All of this is easy to modify in the sense that if you want to pretend to be a different device, all you have to do is change some XML files and some text files on disk. It doesn't require code changes to change your device profile per se. So it offers session based P cap capabilities. So you can come back later and inspect the traffic that went over the sockets and it's written in go, Lange and bash. So SSDP emulation, the SSDP response that is currently in the project was lifted directly from the most abused device that we discovered during the UP and proxy research. It's stored in a flat file and disk. You can change it without modifying any code. The one gotcha is if you update the SSDP banner and it changes the port on which the UP and P Damon is listening on, this is what the attackers pivot on. So you will need to change the listening port in code that the UP and P Damon listens on. I didn't have a configuration file set up when I wrote this in my initial thing. It's an improvement that could be made. UP and P emulation. So the UP and P responses are lifted also from those same most abused devices. All the HTML and XML stored in flat files updating them requires no code changes. UP and P emulation serves basic files, handles NAT interactions. The attacker supplied SOAP is parsed and handled via Regex. It will respond with proper error payloads if criteria are not met or XML is malformed. Responses must contain attacker supplied data so that these responses use standard printf formatting. So if you need to change your thing and the attacker supplied port needs to be in this chunk of XML, you can just put the %d and it will be there. So on the fly proxy, this is kind of unique because the attacker gets to control their proxy configuration themselves. So we had to support that. So attackers submit their proxy config via SOAP just like they are talking to UP and P. We parse them and then create a session of sorts and then we scrape and log plain text across the proxy session in both directions. If they are proxying to TCP 443, it's a special use case and we assume that connection is a TLS connection and we do some special man in the middle of there. So stripping TLS and this is a hard slide to read. It's very upsetting. So attackers actually do some verification when they are using the TLS connections. The initial deployment saw connections but they would bail before actually pushing data across that connection. So we have to do some verification. Attackers are fingerprinting certs. Initially they were doing this via the subject line. There is an automated cloning process where we began by pulling the domain out of the client hello. We then go forward to the injected endpoint and we get the cert with the respective SNI that was provided in the client hello. We copy the remote cert and we mirror it into a self-signed clone cert and this all happens in real time when they first establish their connection. This allows us to, it was allowing us to bypass their fingerprinting and actually get plain text out of the TLS flows. Literally yesterday as I'm recording this, literally yesterday it stopped working and I don't know why. I don't know if they've changed their fingerprinting. I don't know what is really going on but I have a year and a half, well I have months worth of logs at this point that have this functional and now that it's about to go open source it breaks. So I'm sorry. I hope we can figure it out. So the other feature is the automated p-capping. The project uses GoPacket. It allows us to create p-caps on the fly using Berkeley packet filters that are scoped to the individual sessions, the individual proxy sessions. As attackers interact with the proxy injection the p-caps are automatically collected. If you find something interesting in the logs you can find the associated p-cap and see the entire session easily in whatever your favorite, you know, p-cap packet muncher is, wire shard, TCP dump, whatever. If you run out of disk space on your deployed honeypot this is probably why. That is from a single machine down there at the bottom. You can see that we had 81,100 p-caps that we ultimately collected for different sessions and of those p-caps it resulted in almost five and a half gigs of disk consumed. So this part hurts as well. The initial deployment was for one year and it was four nodes deployed across a single VPS provider. There were geos from Dallas to London to Tokyo. 300 gigs of p-caps and logs were ultimately collected. Hundreds of millions of captured proxy sessions and billions of log lines. And you know I downloaded all that and I destroyed the cluster and then I accidentally lost the backup. So I figured this out literally like a day after I submitted the CFP to DEF CON. Luckily I had a couple months before everything was accepted and approved. So I was able to deploy a smaller four node cluster for the two months between CFP and what you're seeing now. So ultimately four nodes deployed US, UK, India, Japan. 39 gigs of p-caps and logs collected 230,000 captured proxy sessions and 22 million lines of logs. The good news is I did have some notes so not everything was lost from the previous deployment and the trends that I saw in the new data are spot on for the trends I saw in the old data. There's not a whole lot's changed just a lot less data to back up the claims but I promise you it's pretty much identical. So observations. The first thing is that they don't blindly inject their their proxies. They actually come and do some testing. So injections they first come and they insert a test proxy instance. A test proxy instance. Once they confirm that it works then they inject a real proxy. They utilize it and then they attempt to delete it. I say attempt well we'll cover that. So this is the process of an injection. We see them show up with their msearch banner. We respond with our ssdp response here and you see that we point them to 192.168.01 on port 2048 and then etsylinuxigdgatedesk.xml. We see them come back within the same second and they then request that. You can see ssdpn and upnpn there. They request that and from there they think that we are the device they're looking for. Once they have confirmed that then they come back and they attempt to add their port mapping. So in this case they're adding an entry that will force us to listen to on port 22280. It's a tcp socket and any traffic received on that is going to be redirected to port 80 on the host at 74.6231.21. You can see down there the new port mapping description kind of mirrors the external port that they use that's sync and then a number and then the new lease duration is 600 seconds. So this will time out after 600 seconds. Then they come back and they utilize that newly injected proxy. So here you can see they sync 2280. Everything up there between the curly braces is the proxy configuration. We can see the source. We can see where ultimately they're going to point to. So 93191 3976 on port 57388 is going to send traffic to 74 6231 21 on port 80. Ultimately we intercept a get request to YAHU with no headers so it's super easy to spot and YAHU because they move to HTTPS ultimately issues a 301 permanently moved and this is all they really need for their fingerprint. Once that's done we see the attacker come back and they attempt to delete the port mapping but they send us malformed XML. What's interesting here is that the malformed XML apparently has I don't know if they forgot a null at the end of the envelope but it continues as a buffer over read and what we ultimately see here is XML that is not related to this request but it just happened to neighbor it in memory. What's interesting here in this case it's the same injection they just sent us. What isn't that interesting? What's more interesting and there are other instances where there is XML information leakage from the buffer over read from other devices that they may have been talking to recently. In this case at some point they were talking to a D-Link DSL 2730U. If you check that out you can see that it is a popular item on a popular e-commerce website. It's actually a choice item on that e-commerce website and it has 3100 readings. For 1,329 rupees you can buy this device and that's about $19 US I believe and you can inadvertently be a Black Hat Proxy too. These are some of the top talkers or sorry some of the top injected test endpoints. You can see there's Akamai Yahoo a few others in there. That top one is clearly the standout winner the 893910512 and they're going to ip.shtml. That's a special page. It returns your public-facing IP address which here I've clearly modified it and then this UBCIEG plug that they use for some kind of identification I'm assuming. There's also a very large campaign being run against Google. This is predominantly all the TLS traffic. It's very weird. I don't know what it is. Out of the 59,924 intercepted requests going across the TLS sockets all of them 100% targeted Google. This is click fraud SEO. I don't know what it is. This is an example of a caught request. They're searching for a Cisco spark board factory reset. We can see their accept language. We can see their cookies. The user agent they used all that good stuff. We can see even they seem to be coming from Dallas based on the information linkage in the URL but I can't really confirm that. I can't really confirm that. Also here we see the response. We get a 200 okay for their search. We see the cookies. It's not here but we would actually have the full page content and everything. It's disabled in this case because that was a lot of log lines. I mean gigs and gigs and gigs at Google pages. In total they sent 57,237 search terms. There are no really clear patterns. They're from all different geos that they target. They use a ton of different user agents. And each request gets basically one search per session. So you can see that top result there has only shown up 55 times out of 57,000 requests. It's just a search for the word Samsung in quotes which is weird. Some of the funnier searches that were captured 72-hour deodorant, antivirus download now, Marlboro summer camp, leather trousers outfit, fa fa fa fa slot hack. I don't even know what that is. I should probably Google some of these to see but I haven't yet. And like I said they're very geographically distributing their stuff across the Google platform. There are domains in here that I don't even know what country they affiliated with. So did you know there's a dot bj.as.jm.md.ee. They're clearly targeting Google.com the most followed by co.uk but still there's so many that they're hitting it's crazy. This is the user agent profiles. So they sent 293 different user agents. And then you can see there's almost normalized clusters of user agent distribution across the abuse. And these are some of the top talkers. So it's not what you'd expect, right? You would, I guess, I mean, I guess kind of it is what you'd expect if it's a single abuser. But the the nature of the queries almost make it seem organic. It doesn't seem like it has an abuse pattern. It almost seems like real end users. But then it's not real end users showing up and popping holes in this stuff. They're all being ferried through a handful of top talkers. And then you have your outliers. If we look at the top 10, we see that Worldstream, Worldstream, Worldstream, Worldstream, OVH, Worldstream, OVH, Worldstream, OVH and Avast. So let me just put it this way. If you work at Worldstream, find me at the bar. If you work at OVH, find me at the bar. If you work at Avast, find me at the bar and I'll buy your drink if you tell me what the hell is going on there. Because I don't know why Avast would be showing up in this data set, but there they are. So some theories on this. The queries to me seem too oddly human. They're in a bunch of different languages. They're stuff like, you know, the best car insurance in Dallas, Fort Worth area. Okay. So it's too organic to be just purely automated abuse, in my opinion. And I'm not sure that the people that are using these proxies are aware that they're using them. I have this theory that it may be some kind of residential proxy reseller or some kind of, you know, ultimate anonymous VPN service provider or something. And these people think that they're getting like these super secret, you know, high privacy stuff. And I'm just sitting here in the middle reading their traffic, which is a problem. So those are my theories. I'd love to hear more theories. You'll also intercept some other stuff outside of TLS. Here, for example, are some spam messages that were being routed. These guys were injecting outlook servers. And then, you know, you can watch the entire interaction, you can watch as they send their, their hello, you can see as they confirm different addresses are deliverable, and then they build their message and shoot it across. You get to see all of that. The good news here is that spam haas is doing God's work and stopping a lot of that abuse from actually succeeding. This was just a fun finding from the older data set. So while this project was going on, Belarus had a very tumultuous political event where a bunch of people went out and protested the recent election. And as a result, Belarus shut down their internet to news and political websites. And while that's going on, suddenly I started seeing these guys popping up in UP and proxy. So the top site was SB.by, which is a news website. And it looks like they were trying to get to the registration and then solve some captures. The other was photo belted up by which is a stock imaging host, I think, or something. But they were actually doing command, not command injections, SQL injections, which I found pretty funny. And then the third one here is mail.rec.gov.by. According to Google translate, this is the central commission of the Republic of Belarus elections or something along those lines. And they were just trying to check their mail. It looks like, and then on the bottom one here, we've got a news outlet, ont.by, and they're trying to get to what appears to be their exchange server. So I just found that kind of interesting. All right. So that's a lot of history, a lot of observations, but now comes the cool part. I'm open sourcing all of this. So anybody that wants to take this, stick it on the internet, play with it, modify it, whatever you want to do, have at it, it's yours. And this way we can all kind of share the fun and see what's going on with these campaigns. And if you find really cool stuff, I'd love to hear about it. So with that open source announcement, let's get some stuff out of the way. First things first, this project was for fun, it's for research. And it was for me to practice my Golang during COVID. Second, I apologize for my shitty code. I know it's shitty. I've learned more Golang since then and learned more design patterns in Golang since then. And I really understand how shitty my code is. I'm sorry. Like I said, it was a research project. It's not commercial grade software. It served its purpose and it did well enough to serve that purpose. And that's all I really needed out of it. Yes, there are bugs. Thank you for noticing. If you open an issue, there's a great chance I'm not going to address it. Maybe someone else in the community will, but this is not my top priority anymore. So I would encourage you to learn some Golang and maybe submit a pull request instead if you'd like to fix that bug. Yes, it's hacky. I know. I am a hacky developer. I'm not your enterprise leading, scrum running, everything else developer. So it is what it is. If you have ideas to fix or improve stuff, it's open source. Have at it, boss. Fork away, send pull requests, whatever. I'll likely accept a pull request. I will likely ignore your issue that you submit to the GitHub. So some ideas for improvements. If you want to hit the ground running, logging could be improved. Content injection could be a thing in a world where people are abusing this stuff. I imagine that you could stick JavaScript in pages or you could tamper with cookies or you could inject plugs of text that might be indexable that you could maybe turn up on a search engine later. I don't know. These are just some ideas I've had. There is a memory leak. I know. The run script actually restarts the binary every hour to get around this because I haven't had time to actually troubleshoot it. Yes, it runs on screen. I regret nothing. Feel free to properly demonize it if you'd like. This I think would be the biggest benefit. If you randomize the SSDP banners and listen on multiple popular exposed UPNP ports, I have a feeling you're going to see a much more diverse set of attackers show up. My findings may be myopic because I'm pretending to be a single device and that single device is what's being targeted by these people. If you were to diversify the target that you paint for your attacker, it's possible you will also diversify your findings. Some additional ideas for improvement. The cert caching. When I wrote this, the cert caching was not taking in S&I differences. It works on Google because all Google servers are just Google. If you were to say have an injection that pointed to some place that had multiple domains associated with it, your one cloned cert is only going to be the one that is aligned with that initial request. You could improve that by in the cert cache, actually using the S&I value, the domain name that was used when that cert was cloned, improved TLS handling, like I said, it was working. It has stopped working. Improving that would probably fix the problem. Improved cert cloning. Cloned more fields to better emulate the remote, the endpoint cert that you're trying to clone. Improved error handling because I didn't really handle any errors. Anything is an improvement. Improved basically everything else. You can find most of the information you'll need in the readme file. If there is anything I missed, please feel free to submit a pull request with the updates you found when deploying the stuff. It's written in Golang, but it does have Linux dependencies. It will run on any operating system so long as it's Linux. You can deploy a node in VPS very easily. You could also run it on Raspberry Pis or Odroid or anything else. Stick it on the DMZ and it will be good to go. Typically, you start to see abuse within the first 24 to 48 hours of deployment. It may be even lower than that. The last thing, if you find something cool, please hit me up on LinkedIn and let me know about it. I'd love to hear if you find new and interesting trends in some of your deployments. That wraps it up. Go grab the project, pull it down, hack it up, compile it, deploy it. Let me know what you find. Go have fun.
with these technologies? Well, for SSTP, IoT at DEF CON 19 called UPnP Mapping. It was a great talk. It kind of touched on this proxy and instances and they were chaining them together. go so far as to attempt to see what other reports Some networks start filtering SSDP. That's all now I've got a new place to drop my ransomware. they're doing with it. So if we see them this part hurts as well. The initial deployment to delete it. I say attempt well we'll cover we can all kind of share the fun and see what's