Infecting The Embedded Supply Chain
Zach Miller
Alex Kissinger
With a surge in the production of internet of things (IoT) devices, embedded development tools are becoming commonplace and the software they run on is often trusted to run in escalated modes. However, some of the embedded development tools on the market contain serious vulnerabilities that put users at risk. In this talk we discuss the various attack vectors that these embedded development tools expose users to, and why users should not blindly trust their tools. This talk will detail a variety reverse engineering, fuzzing, exploit development and protocol analysis techniques that we used to analyze and exploit the security of a common embedded debugger.
 So we have two more first time speakers coming up here We're gonna give them the Only piece of DEF CON merch that each and every one of you out here could earn but you have to earn it Not only do we give them shots they get the shot the noob shot glass that's our present to them For coming and giving a talk at DEF CON so Help me welcome Zach and Alex to their first talk Infecting the embedded supply chain I was telling them on the walk over here. I haven't read your abstract But just from the title I saw that and I'm like Yeah, we need to do better at that because I I work on an embedded device and yeah, they're poking holes where we've got holes. Alright, help you welcome Zach and Alex Hello, can you guys hear us? Hello everybody This is a infecting the embedded supply chain. So disclaimer We thought we had a 45 minute slot, but we actually have 20 minutes So we're gonna try our best to condense everything within that 20 minutes. So we have a couple of videos Um, which we'll try to do we'll try to get to everything But if we don't make it to everything we'll post everything to our website summer set recon calm and we'll have proof of concept code On github as well. So you guys can check that out I'm Zach Like I said, I work at summer set recon. I do a combination of web application pen testing and reverse engineering I specialize in vulnerability research and exploit development I'm Alex. I'm a barista that occasionally does security things at summer set recon I enjoy making cappuccinos hardware hacking and reverse engineering Just a little bit about summer set recon. We are a security consultancy based out of San Diego We specialize in web applications mobile applications and embedded device security And you can find out more about us at summer set recon calm and our Twitter Cool. So aside from our day-to-day work. We also like to do some cool side projects One of which was an electronic safe lock that we looked at that allowed you to either enter in a pin manually to unlock a safe or you know over Bluetooth on Your favorite mobile device We found some vulnerabilities there including being able to dig deep code pins wirelessly over the air and replay them and also We reversed the wire protocol and created a brute force Device for that We also did work on the hello barbie, which is an IOT doll that kids can talk to and have a full conversation with We looked at all the web services and Reverse the mobile application and pretty much just figured out how that all worked and found some vulnerabilities and made a white paper Which is online as well. So? These are all embedded devices and that's primarily what we focus on on our side research And so when we decided to do this research, we wanted to do something slightly different but touched on it still So we tried to look at what all these products had in common and We thought why not look at their you know the tools used to program and debug them So this is our target we've focused on the SEGR J-Link which is a pretty popular device on both the hardware and the software used to control it The SEGR J-Link debug probe communicates with an embedded device that you're targeting and trying to program like a mobile device for instance over a J-Tag SWD etc It's an in-circuit emulator and in-circuit system programmer so you could program chips it primarily supports arm-and-arm cortex chips But also supports some others like renaissance It communicates with a host Computer over USB and ethernet which is kind of interesting has a cross-platform tool chain and some other cool features And the SEGR J-Links are the most widely used line of debug probes available Today which is a quote from them That's kind of cool Other than hardware they create a ton of software Including like the software package that's used to control the J-Links themselves. They also have a GDB server RTOS plug-in source development kit a couple of real-time analysis visualization tools and a graphical debugger We primarily focused on the software suite used to control these J-Links devices and this package just includes a ton of tools Including J-Link commander which is like your typical command line tool that you would use with most debuggers A GDB server a remote server which allows a client to connect to a J-Link commander session Within the local network a memory viewer and a flashing tool to just flash So typical setup looks like this you have a host PC that has a software suite installed and you're communicating to this J-Link device over USB and ethernet And then you connect it to your embedded target over JTAG or SWD So in terms of attack surface, I mean there's a lot I mean this company makes a lot of different products The hardware debug probes themselves we're interested in firmware and you know how that works and also can you infect the debug probe itself Or can the debug probe infect the end device that you're trying to target And then in addition to that there's a lot of software attack surface as well There's a USB driver that's included with this device as well as a host of different user mode applications that are run And in addition to that which is also a pretty interesting thing they have a custom IDE that's also used to develop applications for the J-Link So hardware yeah so we're interested in how the hardware works extracting firmware how easy is it to do that how does the firmware work The feature differences between each of the devices because SEGR makes a ton of different types of J-Links At different price points of course and we're wondering like are there hardware differences or is it just firmware And what these devices what are the security mechanisms that are on it is the device open can I flash an open source version of firmware Or can I modify firmware and flash the debug probe So this is a typical J-Link EDU it's kind of like the cheapest consumer end J-Link you can get If we focus here and open GIMP you can see that it is also an ARM device So the tool that's used to debug and program ARM embedded devices is also an ARM embedded device which is yeah that's pretty nice Then we started looking for you know debug ports and found a tag connect which is kind of like it's kind of like a J tag header But a proprietary connector that has Pogo pins so we had one of those lying around and we were asking well Can we do it do we have anything that can program or debug this device and we're like why not try debugging a J-Link with a J-Link So it turns out that the security flash bits are set so that's good The flash bit basically means that you can't pre-program the device unless you erase flash Which you can't do because the security bit is set which basically says you can't debug this chip So it refuses to connect and erase we're looking for other ways around this so we need to go deeper So we took a look at some of the other J-Link debug probes and the J-Link mini EDU has an interesting feature in its ARM chip Which is called the backdoor key access which is an opt-in feature That basically if you provide the chip with the proper key it will like willingly disable the security bit And then you can kind of do whatever you want with the device which is pretty cool So then we began really looking at some of the desktop software that was distributed with the Seger J-Link for security vulnerabilities We started out just doing some high-level reverse engineering We noticed that they distributed their software to both Linux and Windows systems In reverse engineering this we determined that they were using cross-compiled code So that eliminated some redundant work where only one of the operating systems versions really needed to be reverse engineered To determine the functionality and vulnerable portions of the code We also noticed that they were using some custom string manipulation code Which was somewhat interesting and we'll talk a little bit about that more later We also noticed that they used a lot of dangerous functions They used string copy, string cat and a lot of other things that don't check the length of the destination buffer From the point of binary protections, Seger opted into most binary protections However they did leave out both PIE and stack canaries in their Linux executables So once we had that we began setting up some fuzzers in order to test the different input vectors See if we could find any vulnerabilities that we could begin to exploit So we tested all input vectors that these applications seemed to accept We tested files, network interfaces and command line arguments that they accepted Since our reversing revealed that there were some really interesting code paths That were somewhat deep and required magic numbers and different things to reach We decided to use a generational fuzzing approach so we used Peach for that We fuzzed and got tons of crashes and began looking into actually exploiting them So the first thing is the custom string formatting We noticed in looking into this some interesting usages such as shown here Where if it were a standard string formatting function it would be a format string vulnerability So we began looking into that a little bit more So the custom string formatting that they implemented used most of the basic format specifiers Such as %s, %n, %d etc But it did not accept the %n that's typically used in format string vulnerabilities So with that we were able to do some of the format string exploits We were able to do arbitrary reads as shown here However due to the fact that it lacked the %n specifier that's used to write bits We were not able to turn this into an arbitrary write I also found in the JLint commander tool which we talked about earlier it's a command line tool There is a feature called command file which basically you can feed JLint commander a script To kind of set up your environment before you start debugging your device And this the file parsing code had a vulnerability which was a traditional stack buffer overflow And so in the image you can see that we kind of just feed a bunch of a's into a payload And then we pass it into JLint.exe and it just seg faults So if you triage that in GDB you can see that it tries to return to address 41, 41, 41, 41 So we went a little bit further with it and we were able to create a POC with like five basic steps Which is take over the return address, get the address of libc Use that pointer to get the address of system and call system with arguments and you got code execution We won't be able to go over the POC because of time But ultimately we got code execution It works on the 32-bit JLint.exe binary on i386 and AMD64 Linux systems And it's a Rop with ASLR bypass, return to libc And reverse shell is very doable but requires roping and libc a lot But the general idea is with this attack that an attacker can create a malicious command file And dummy firmware and send an urgent email with this as an attachment to a developer at a targeted company So developers are typically pressed for time so they'll probably run this with JLint commander And it would pop a shell and return back to the attacker We also found in the same binary we found another file parsing vulnerability in the settings file flag Which is very similar to command file but it's in a completely different area It's in a library called jlinkarm.so and that's where a lot of like the ARM specific code is in For like a JLint software suite And this is in particular a buffer overrun in the BSS segment So you can use this to kind of overwrite function pointers in the BSS segment We also got RCE which is cool So after that we started looking at a really interestingly named executable called JLint remote server So once we saw this we kind of perked up a little bit and began trying to look at the functionality of this executable So running a basic net set we noticed that it was listening on a number of ports And particularly of interest to us was it looked like it was listening as a telnet server So we wanted to figure out if this was actually having a telnet server embedded in the executable Or what was going on there So a little bit of reverse engineering revealed that they actually had embedded a telnet server within their executable So it allows users within the network to connect to this JLint server and do remote debugging So we began fuzzing this and noticed a really interesting crash where we had overwritten the instruction pointer With AAA which is always good We did a little bit of reverse engineering and triage on this and picked up a couple things It was a stack buffer overflow that we were able to trigger remotely We were not able to get the crashes to be consistent due to race conditions within the executable We had a fairly limited amount of stack space to work with if we were going to build a ROP chain We were limited to 48 bytes And again we had ASLR and NX enabled on this executable but PIE was not enabled So we used typical exploitation techniques We used ROP to bypass the NX And then with a ROP chain we did a dereference of the GOT in order to leak the address of libc From there we could calculate the offset of system and then do a return to libc The main issue that we ran into here was we wanted to send arbitrary attack or controlled payloads to system And we had trouble kind of getting the user controlled strings to consistently be passed in the system So we realized that user controlled data was being stored in one of two locations in the executable And due to a race condition we couldn't determine if it was going to be in one location or the other So we tried to figure out if there was some way we could make it more consistent And we came up with the following So we sort of called this space sleds where we were inspired by the not-sled techniques In which nots are propended to a shellcode payload and it improves exploit consistency So here we prepend spaces to the user controlled command string And we hope that that would improve the consistency of our exploit So what we did here is since the two locations where we were seeing the attacker controlled payload string Were 72 bytes apart We prepended 72 spaces to our string and that created an overlap where we could point our argument to system So we have a demo where if we have enough time at the end of this we'll be able to show And then additionally in looking into the Jlink remote server we found some other really interesting functionality So we termed this the tunnel server backdoor And SEGR has this as a documented feature within their application And they said it just provides a tunneling mode from a remote computer into that your network allows remote debugging Even when they're not within the local network, correct So our first thought at that point was I wonder what type of authentication they're using If there's any weaknesses there or anything that we could exploit So as you can kind of see here they use magic numbers and your device serial number to authenticate your device So that was a pretty scary thing So when you run Jlink remote server with your Jlink device attached It sends a magic number and registers your Jlink debugger with the server And then it allows client connections to specify the serial number of the device that they hope to connect to So we looked at the serial numbers and realized that there were 10 billion possibilities Which would make brute forcing all of the connected serial numbers a bit infeasible So we wondered is this actually something that would be a potential attack? Is there some way that an attacker could shrink the space of serial numbers that they would attempt to brute force? So we began looking into is there some format that SEGR uses for their serial numbers? Is there like some smaller range or number that they start at that we could use to shrink the space? So we googled SEGR Jlink and found tons of images that people had posted online that included their device serial number We called some people that we know that also had SEGR Jlinks and asked their device serial number And in the end we got about 30 serial numbers that we were able to analyze And started to notice a few patterns that emerged So the first two digits of the serial numbers were correlated with the model of device Whether you had a Jlink Pro or a Jlink Mini After that two digits for the version where SEGR would release a slightly updated version that would increment And then finally the last five digits are a device unique number that's incremented So doing a little bit of analysis on the numbers that we had We were able to shrink the serial number space and still achieve what we think would be very good coverage With only 100,000 serial numbers And assuming about 10 serial numbers per second could be brute-forced which is about the link that we saw the connections taking When it was done properly via the Jlink application It would take the time from 31 years to brute-force the space to about 3 hours The impact here really is once you make a connection via this server you can do all sorts of things You can flash new firmware, you could read the existing firmware Or make any sort of malicious modifications to the connected devices So we disclosed SEGR and we're really pleased with the response They responded right away and we got a lot of feedback within a week And it was very, we even received a thank you from the founder and CTO of the company So in conclusion we just really realized that these devices are often times lacking And a lot of the protections and security that we would hope from a device piece of the supply chain It was lacking the PIE flag which would have made our exploitation much more difficult As well as having lacking the security features that are just architectural and no authentication in the remote server or anything like that But wait, there's more Okay, before we run out of time We revisited the JLink hardware but instead of from the perspective of trying to manipulate the firmware with a hardware debugging device From the OTA process We realized that if you download a newer version of the JLink software suite JLink Commander will ask you if you want to update your JLink debug probe And so we started from the network perspective and found that there was no firmware blobs being transferred over the network And we thought that was kind of interesting So we eventually figured out where firmware was being stored and how the update process worked And we reversed the USB protocol to be able to flash the JLink devices And it turns out that the JLink device, when it's being flashed over USB, it does do a firmware check But it's not very good And it uses dates to check whether the firmware is valid or not So this could be a bad thing Also, firmware is not signed and can be modified So in the end, we don't have time for the video But to illustrate it, we created a piece of malware that runs on Windows It runs silently in the background And just any JLink device that is connected via USB immediately flashes and completely bricks By flashing firmware that's set way far in the future And you can't fix your device unless you have hardware debugging privileges And that's it Cool Thank you
 So we have two more first time speakers coming up here  We're gonna give them the  Only piece of DEF CON merch that each and every one of you out here could earn but you have to earn it  Not only do we give them shots they get the shot the noob shot glass that's our present to them  For coming and giving a talk at DEF CON so  Help me welcome Zach and Alex to their first talk  Infecting the embedded supply chain I was telling them on the walk over here. I haven't read your abstract  But just from the title I saw that and I'm like  Yeah, we need to do better at that because I  I work on an embedded device and yeah, they're poking holes where we've got holes. Alright, help you welcome Zach and Alex  Hello, can you guys hear us?  Hello everybody  This is a infecting the embedded supply chain. So disclaimer  We thought we had a 45 minute slot, but we actually have 20 minutes  So we're gonna try our best to condense everything within that 20 minutes. So we have a couple of videos  Um, which we'll try to do we'll try to get to everything  But if we don't make it to everything we'll post everything to our website summer set recon calm and we'll have proof of concept code  On github as well. So you guys can check that out  I'm Zach  Like I said, I work at summer set recon. I do a combination of web application pen testing and reverse engineering  I specialize in vulnerability research and exploit development  I'm Alex. I'm a barista that occasionally does security things at summer set recon  I enjoy making cappuccinos hardware hacking and reverse engineering  Just a little bit about summer set recon. We are a security consultancy based out of San Diego  We specialize in web applications mobile applications and embedded device security  And you can find out more about us at summer set recon calm and our Twitter  Cool. So aside from our day-to-day work. We also like to do some cool side projects  One of which was an electronic safe lock that we looked at  that allowed you to either enter in a pin manually to unlock a safe or you know over Bluetooth on  Your favorite mobile device  We found some vulnerabilities there including being able to dig deep code pins wirelessly over the air and replay them and also  We reversed the wire protocol and created a brute force  Device for that  We also did work on the hello barbie, which is an IOT doll that kids can talk to and have a full conversation with  We looked at all the web services and  Reverse the mobile application and pretty much just figured out how that all worked and found some vulnerabilities and made a white paper  Which is online as well. So?  These are all embedded devices and that's primarily what we focus on on our side research  And so when we decided to do this research, we wanted to do something slightly different but touched on it still  So we tried to look at what all these products had in common and  We thought why not look at their you know the tools used to program and debug them  So this is our target we've focused on the SEGR J-Link which is a pretty popular device  on both the hardware and  the software used to control it  The SEGR J-Link debug probe  communicates with an embedded device that you're targeting and trying to program like a mobile device for instance over a J-Tag  SWD etc  It's an in-circuit emulator and in-circuit system programmer so you could program chips it primarily supports arm-and-arm cortex chips  But also supports some others like renaissance  It communicates with a host  Computer over USB and ethernet which is kind of interesting has a cross-platform tool chain and some other cool features  And the SEGR J-Links are the most widely used line of debug probes available  Today which is a quote from them  That's kind of cool  Other than hardware they create a ton of software  Including like the software package that's used to control the J-Links themselves. They also have a GDB server  RTOS plug-in source development kit a couple of real-time analysis visualization tools and a graphical debugger  We primarily focused on the software suite used to control these J-Links devices and this package just includes a ton of tools  Including J-Link commander which is like your typical command line tool that you would use with most debuggers  A GDB server a remote server which allows a client to connect to a J-Link commander session  Within the local network a memory viewer and a flashing tool to just flash  So typical setup looks like this you have a host PC that has a software suite installed and you're communicating to this J-Link device over USB and ethernet  And then you connect it to your embedded target over JTAG or SWD  So in terms of attack surface, I mean there's a lot I mean this company makes a lot of different products  The hardware debug probes themselves we're interested in firmware and you know how that works and also can you infect the debug probe itself  Or can the debug probe infect the end device that you're trying to target  And then in addition to that there's a lot of software attack surface as well  There's a USB driver that's included with this device as well as a host of different user mode applications that are run  And in addition to that which is also a pretty interesting thing they have a custom IDE that's also used to develop applications for the J-Link  So hardware yeah so we're interested in how the hardware works extracting firmware how easy is it to do that how does the firmware work  The feature differences between each of the devices because SEGR makes a ton of different types of J-Links  At different price points of course and we're wondering like are there hardware differences or is it just firmware  And what these devices what are the security mechanisms that are on it is the device open can I flash an open source version of firmware  Or can I modify firmware and flash the debug probe  So this is a typical J-Link EDU it's kind of like the cheapest consumer end J-Link you can get  If we focus here and open GIMP you can see that it is also an ARM device  So the tool that's used to debug and program ARM embedded devices is also an ARM embedded device which is yeah that's pretty nice  Then we started looking for you know debug ports and found a tag connect which is kind of like it's kind of like a J tag header  But a proprietary connector that has Pogo pins so we had one of those lying around and we were asking well  Can we do it do we have anything that can program or debug this device and we're like why not try debugging a J-Link with a J-Link  So it turns out that the security flash bits are set so that's good  The flash bit basically means that you can't pre-program the device unless you erase flash  Which you can't do because the security bit is set which basically says you can't debug this chip  So it refuses to connect and erase we're looking for other ways around this so we need to go deeper  So we took a look at some of the other J-Link debug probes and the J-Link mini EDU has an interesting feature in its ARM chip  Which is called the backdoor key access which is an opt-in feature  That basically if you provide the chip with the proper key it will like willingly disable the security bit  And then you can kind of do whatever you want with the device which is pretty cool  So then we began really looking at some of the desktop software that was distributed with the Seger J-Link for security vulnerabilities  We started out just doing some high-level reverse engineering  We noticed that they distributed their software to both Linux and Windows systems  In reverse engineering this we determined that they were using cross-compiled code  So that eliminated some redundant work where only one of the operating systems versions really needed to be reverse engineered  To determine the functionality and vulnerable portions of the code  We also noticed that they were using some custom string manipulation code  Which was somewhat interesting and we'll talk a little bit about that more later  We also noticed that they used a lot of dangerous functions  They used string copy, string cat and a lot of other things that don't check the length of the destination buffer  From the point of binary protections, Seger opted into most binary protections  However they did leave out both PIE and stack canaries in their Linux executables  So once we had that we began setting up some fuzzers in order to test the different input vectors  See if we could find any vulnerabilities that we could begin to exploit  So we tested all input vectors that these applications seemed to accept  We tested files, network interfaces and command line arguments that they accepted  Since our reversing revealed that there were some really interesting code paths  That were somewhat deep and required magic numbers and different things to reach  We decided to use a generational fuzzing approach so we used Peach for that  We fuzzed and got tons of crashes and began looking into actually exploiting them  So the first thing is the custom string formatting  We noticed in looking into this some interesting usages such as shown here  Where if it were a standard string formatting function it would be a format string vulnerability  So we began looking into that a little bit more  So the custom string formatting that they implemented used most of the basic format specifiers  Such as %s, %n, %d etc  But it did not accept the %n that's typically used in format string vulnerabilities  So with that we were able to do some of the format string exploits  We were able to do arbitrary reads as shown here  However due to the fact that it lacked the %n specifier that's used to write bits  We were not able to turn this into an arbitrary write  I also found in the JLint commander tool which we talked about earlier it's a command line tool  There is a feature called command file which basically you can feed JLint commander a script  To kind of set up your environment before you start debugging your device  And this the file parsing code had a vulnerability which was a traditional stack buffer overflow  And so in the image you can see that we kind of just feed a bunch of a's into a payload  And then we pass it into JLint.exe and it just seg faults  So if you triage that in GDB you can see that it tries to return to address 41, 41, 41, 41  So we went a little bit further with it and we were able to create a POC with like five basic steps  Which is take over the return address, get the address of libc  Use that pointer to get the address of system and call system with arguments and you got code execution  We won't be able to go over the POC because of time  But ultimately we got code execution  It works on the 32-bit JLint.exe binary on i386 and AMD64 Linux systems  And it's a Rop with ASLR bypass, return to libc  And reverse shell is very doable but requires roping and libc a lot  But the general idea is with this attack that an attacker can create a malicious command file  And dummy firmware and send an urgent email with this as an attachment to a developer at a targeted company  So developers are typically pressed for time so they'll probably run this with JLint commander  And it would pop a shell and return back to the attacker  We also found in the same binary we found another file parsing vulnerability in the settings file flag  Which is very similar to command file but it's in a completely different area  It's in a library called jlinkarm.so and that's where a lot of like the ARM specific code is in  For like a JLint software suite  And this is in particular a buffer overrun in the BSS segment  So you can use this to kind of overwrite function pointers in the BSS segment  We also got RCE which is cool  So after that we started looking at a really interestingly named executable called JLint remote server  So once we saw this we kind of perked up a little bit and began trying to look at the functionality of this executable  So running a basic net set we noticed that it was listening on a number of ports  And particularly of interest to us was it looked like it was listening as a telnet server  So we wanted to figure out if this was actually having a telnet server embedded in the executable  Or what was going on there  So a little bit of reverse engineering revealed that they actually had embedded a telnet server within their executable  So it allows users within the network to connect to this JLint server and do remote debugging  So we began fuzzing this and noticed a really interesting crash where we had overwritten the instruction pointer  With AAA which is always good  We did a little bit of reverse engineering and triage on this and picked up a couple things  It was a stack buffer overflow that we were able to trigger remotely  We were not able to get the crashes to be consistent due to race conditions within the executable  We had a fairly limited amount of stack space to work with if we were going to build a ROP chain  We were limited to 48 bytes  And again we had ASLR and NX enabled on this executable but PIE was not enabled  So we used typical exploitation techniques  We used ROP to bypass the NX  And then with a ROP chain we did a dereference of the GOT in order to leak the address of libc  From there we could calculate the offset of system and then do a return to libc  The main issue that we ran into here was we wanted to send arbitrary attack or controlled payloads to system  And we had trouble kind of getting the user controlled strings to consistently be passed in the system  So we realized that user controlled data was being stored in one of two locations in the executable  And due to a race condition we couldn't determine if it was going to be in one location or the other  So we tried to figure out if there was some way we could make it more consistent  And we came up with the following  So we sort of called this space sleds where we were inspired by the not-sled techniques  In which nots are propended to a shellcode payload and it improves exploit consistency  So here we prepend spaces to the user controlled command string  And we hope that that would improve the consistency of our exploit  So what we did here is since the two locations where we were seeing the attacker controlled payload string  Were 72 bytes apart  We prepended 72 spaces to our string and that created an overlap where we could point our argument to system  So we have a demo where if we have enough time at the end of this we'll be able to show  And then additionally in looking into the Jlink remote server we found some other really interesting functionality  So we termed this the tunnel server backdoor  And SEGR has this as a documented feature within their application  And they said it just provides a tunneling mode from a remote computer into that your network allows remote debugging  Even when they're not within the local network, correct  So our first thought at that point was I wonder what type of authentication they're using  If there's any weaknesses there or anything that we could exploit  So as you can kind of see here they use magic numbers and your device serial number to authenticate your device  So that was a pretty scary thing  So when you run Jlink remote server with your Jlink device attached  It sends a magic number and registers your Jlink debugger with the server  And then it allows client connections to specify the serial number of the device that they hope to connect to  So we looked at the serial numbers and realized that there were 10 billion possibilities  Which would make brute forcing all of the connected serial numbers a bit infeasible  So we wondered is this actually something that would be a potential attack?  Is there some way that an attacker could shrink the space of serial numbers that they would attempt to brute force?  So we began looking into is there some format that SEGR uses for their serial numbers?  Is there like some smaller range or number that they start at that we could use to shrink the space?  So we googled SEGR Jlink and found tons of images that people had posted online that included their device serial number  We called some people that we know that also had SEGR Jlinks and asked their device serial number  And in the end we got about 30 serial numbers that we were able to analyze  And started to notice a few patterns that emerged  So the first two digits of the serial numbers were correlated with the model of device  Whether you had a Jlink Pro or a Jlink Mini  After that two digits for the version where SEGR would release a slightly updated version that would increment  And then finally the last five digits are a device unique number that's incremented  So doing a little bit of analysis on the numbers that we had  We were able to shrink the serial number space and still achieve what we think would be very good coverage  With only 100,000 serial numbers  And assuming about 10 serial numbers per second could be brute-forced which is about the link that we saw the connections taking  When it was done properly via the Jlink application  It would take the time from 31 years to brute-force the space to about 3 hours  The impact here really is once you make a connection via this server you can do all sorts of things  You can flash new firmware, you could read the existing firmware  Or make any sort of malicious modifications to the connected devices  So we disclosed SEGR and we're really pleased with the response  They responded right away and we got a lot of feedback within a week  And it was very, we even received a thank you from the founder and CTO of the company  So in conclusion we just really realized that these devices are often times lacking  And a lot of the protections and security that we would hope from a device piece of the supply chain  It was lacking the PIE flag which would have made our exploitation much more difficult  As well as having lacking the security features that are just architectural and no authentication in the remote server or anything like that  But wait, there's more  Okay, before we run out of time  We revisited the JLink hardware but instead of from the perspective of trying to manipulate the firmware with a hardware debugging device  From the OTA process  We realized that if you download a newer version of the JLink software suite  JLink Commander will ask you if you want to update your JLink debug probe  And so we started from the network perspective and found that there was no firmware blobs being transferred over the network  And we thought that was kind of interesting  So we eventually figured out where firmware was being stored and how the update process worked  And we reversed the USB protocol to be able to flash the JLink devices  And it turns out that the JLink device, when it's being flashed over USB, it does do a firmware check  But it's not very good  And it uses dates to check whether the firmware is valid or not  So this could be a bad thing  Also, firmware is not signed and can be modified  So in the end, we don't have time for the video  But to illustrate it, we created a piece of malware that runs on Windows  It runs silently in the background  And just any JLink device that is connected via USB immediately flashes and completely bricks  By flashing firmware that's set way far in the future  And you can't fix your device unless you have hardware debugging privileges  And that's it  Cool  Thank you
DEF_CON_26_-_Zach_and_Alex_-_Infecting_The_Embedded_Supply_Chain_XeiET4FuGjE_0_2003.png DEF_CON_26_-_Zach_and_Alex_-_Infecting_The_Embedded_Supply_Chain_XeiET4FuGjE_2004_2099.png DEF_CON_26_-_Zach_and_Alex_-_Infecting_The_Embedded_Supply_Chain_XeiET4FuGjE_2100_17975.png DEF_CON_26_-_Zach_and_Alex_-_Infecting_The_Embedded_Supply_Chain_XeiET4FuGjE_17976_30743.png DEF_CON_26_-_Zach_and_Alex_-_Infecting_The_Embedded_Supply_Chain_XeiET4FuGjE_30744_32853.png
So we have two more first time speakers coming up here We're gonna give them the Only piece of DEF CON merch that each and every one of you out here could earn but you have to earn it Not only do we give them shots they get the shot the noob shot glass that's our present to them For coming and giving a talk at DEF CON so Help me welcome Zach and Alex to their first talk Infecting the embedded supply chain I was telling them on the walk over here. I haven't read your abstract But just from the title I saw that and I'm like Yeah, we need to do better at that because I I work on an embedded device and yeah, they're poking holes where we've got holes. Alright, help you welcome Zach and Alex Hello, can you guys hear us? Hello everybody This is a infecting the embedded supply chain. So disclaimer We thought we had a 45 minute slot, but we actually have 20 minutes So we're gonna try our best to condense everything within that 20 minutes. So we have a couple of videos Um, which we'll try to do we'll try to get to everything But if we don't make it to everything we'll post everything to our website summer set recon calm and we'll have proof of concept code On github as well. So you guys can check that out I'm Zach Like I said, I work at summer set recon. I do a combination of web application pen testing and reverse engineering I specialize in vulnerability research and exploit development I'm Alex. I'm a barista that occasionally does security things at summer set recon I enjoy making cappuccinos hardware hacking and reverse engineering Just a little bit about summer set recon. We are a security consultancy based out of San Diego We specialize in web applications mobile applications and embedded device security And you can find out more about us at summer set recon calm and our Twitter Cool. So aside from our day-to-day work. We also like to do some cool side projects One of which was an electronic safe lock that we looked at that allowed you to either enter in a pin manually to unlock a safe or you know over Bluetooth on Your favorite mobile device We found some vulnerabilities there including being able to dig deep code pins wirelessly over the air and replay them and also We reversed the wire protocol and created a brute force Device for that We also did work on the hello barbie, which is an IOT doll that kids can talk to and have a full conversation with We looked at all the web services and Reverse the mobile application and pretty much just figured out how that all worked and found some vulnerabilities and made a white paper Which is online as well. So? These are all embedded devices and that's primarily what we focus on on our side research And so when we decided to do this research, we wanted to do something slightly different but touched on it still So we tried to look at what all these products had in common and We thought why not look at their you know the tools used to program and debug them So this is our target we've focused on the SEGR J-Link which is a pretty popular device on both the hardware and the software used to control it The SEGR J-Link debug probe communicates with an embedded device that you're targeting and trying to program like a mobile device for instance over a J-Tag SWD etc It's an in-circuit emulator and in-circuit system programmer so you could program chips it primarily supports arm-and-arm cortex chips But also supports some others like renaissance It communicates with a host Computer over USB and ethernet which is kind of interesting has a cross-platform tool chain and some other cool features And the SEGR J-Links are the most widely used line of debug probes available Today which is a quote from them That's kind of cool Other than hardware they create a ton of software Including like the software package that's used to control the J-Links themselves. They also have a GDB server RTOS plug-in source development kit a couple of real-time analysis visualization tools and a graphical debugger We primarily focused on the software suite used to control these J-Links devices and this package just includes a ton of tools Including J-Link commander which is like your typical command line tool that you would use with most debuggers A GDB server a remote server which allows a client to connect to a J-Link commander session Within the local network a memory viewer and a flashing tool to just flash So typical setup looks like this you have a host PC that has a software suite installed and you're communicating to this J-Link device over USB and ethernet And then you connect it to your embedded target over JTAG or SWD So in terms of attack surface, I mean there's a lot I mean this company makes a lot of different products The hardware debug probes themselves we're interested in firmware and you know how that works and also can you infect the debug probe itself Or can the debug probe infect the end device that you're trying to target And then in addition to that there's a lot of software attack surface as well There's a USB driver that's included with this device as well as a host of different user mode applications that are run And in addition to that which is also a pretty interesting thing they have a custom IDE that's also used to develop applications for the J-Link So hardware yeah so we're interested in how the hardware works extracting firmware how easy is it to do that how does the firmware work The feature differences between each of the devices because SEGR makes a ton of different types of J-Links At different price points of course and we're wondering like are there hardware differences or is it just firmware And what these devices what are the security mechanisms that are on it is the device open can I flash an open source version of firmware Or can I modify firmware and flash the debug probe So this is a typical J-Link EDU it's kind of like the cheapest consumer end J-Link you can get If we focus here and open GIMP you can see that it is also an ARM device So the tool that's used to debug and program ARM embedded devices is also an ARM embedded device which is yeah that's pretty nice Then we started looking for you know debug ports and found a tag connect which is kind of like it's kind of like a J tag header But a proprietary connector that has Pogo pins so we had one of those lying around and we were asking well Can we do it do we have anything that can program or debug this device and we're like why not try debugging a J-Link with a J-Link So it turns out that the security flash bits are set so that's good The flash bit basically means that you can't pre-program the device unless you erase flash Which you can't do because the security bit is set which basically says you can't debug this chip So it refuses to connect and erase we're looking for other ways around this so we need to go deeper So we took a look at some of the other J-Link debug probes and the J-Link mini EDU has an interesting feature in its ARM chip Which is called the backdoor key access which is an opt-in feature That basically if you provide the chip with the proper key it will like willingly disable the security bit And then you can kind of do whatever you want with the device which is pretty cool So then we began really looking at some of the desktop software that was distributed with the Seger J-Link for security vulnerabilities We started out just doing some high-level reverse engineering We noticed that they distributed their software to both Linux and Windows systems In reverse engineering this we determined that they were using cross-compiled code So that eliminated some redundant work where only one of the operating systems versions really needed to be reverse engineered To determine the functionality and vulnerable portions of the code We also noticed that they were using some custom string manipulation code Which was somewhat interesting and we'll talk a little bit about that more later We also noticed that they used a lot of dangerous functions They used string copy, string cat and a lot of other things that don't check the length of the destination buffer From the point of binary protections, Seger opted into most binary protections However they did leave out both PIE and stack canaries in their Linux executables So once we had that we began setting up some fuzzers in order to test the different input vectors See if we could find any vulnerabilities that we could begin to exploit So we tested all input vectors that these applications seemed to accept We tested files, network interfaces and command line arguments that they accepted Since our reversing revealed that there were some really interesting code paths That were somewhat deep and required magic numbers and different things to reach We decided to use a generational fuzzing approach so we used Peach for that We fuzzed and got tons of crashes and began looking into actually exploiting them So the first thing is the custom string formatting We noticed in looking into this some interesting usages such as shown here Where if it were a standard string formatting function it would be a format string vulnerability So we began looking into that a little bit more So the custom string formatting that they implemented used most of the basic format specifiers Such as %s, %n, %d etc But it did not accept the %n that's typically used in format string vulnerabilities So with that we were able to do some of the format string exploits We were able to do arbitrary reads as shown here However due to the fact that it lacked the %n specifier that's used to write bits We were not able to turn this into an arbitrary write I also found in the JLint commander tool which we talked about earlier it's a command line tool There is a feature called command file which basically you can feed JLint commander a script To kind of set up your environment before you start debugging your device And this the file parsing code had a vulnerability which was a traditional stack buffer overflow And so in the image you can see that we kind of just feed a bunch of a's into a payload And then we pass it into JLint.exe and it just seg faults So if you triage that in GDB you can see that it tries to return to address 41, 41, 41, 41 So we went a little bit further with it and we were able to create a POC with like five basic steps Which is take over the return address, get the address of libc Use that pointer to get the address of system and call system with arguments and you got code execution We won't be able to go over the POC because of time But ultimately we got code execution It works on the 32-bit JLint.exe binary on i386 and AMD64 Linux systems And it's a Rop with ASLR bypass, return to libc And reverse shell is very doable but requires roping and libc a lot But the general idea is with this attack that an attacker can create a malicious command file And dummy firmware and send an urgent email with this as an attachment to a developer at a targeted company So developers are typically pressed for time so they'll probably run this with JLint commander And it would pop a shell and return back to the attacker We also found in the same binary we found another file parsing vulnerability in the settings file flag Which is very similar to command file but it's in a completely different area It's in a library called jlinkarm.so and that's where a lot of like the ARM specific code is in For like a JLint software suite And this is in particular a buffer overrun in the BSS segment So you can use this to kind of overwrite function pointers in the BSS segment We also got RCE which is cool So after that we started looking at a really interestingly named executable called JLint remote server So once we saw this we kind of perked up a little bit and began trying to look at the functionality of this executable So running a basic net set we noticed that it was listening on a number of ports And particularly of interest to us was it looked like it was listening as a telnet server So we wanted to figure out if this was actually having a telnet server embedded in the executable Or what was going on there So a little bit of reverse engineering revealed that they actually had embedded a telnet server within their executable So it allows users within the network to connect to this JLint server and do remote debugging So we began fuzzing this and noticed a really interesting crash where we had overwritten the instruction pointer With AAA which is always good We did a little bit of reverse engineering and triage on this and picked up a couple things It was a stack buffer overflow that we were able to trigger remotely We were not able to get the crashes to be consistent due to race conditions within the executable We had a fairly limited amount of stack space to work with if we were going to build a ROP chain We were limited to 48 bytes And again we had ASLR and NX enabled on this executable but PIE was not enabled So we used typical exploitation techniques We used ROP to bypass the NX And then with a ROP chain we did a dereference of the GOT in order to leak the address of libc From there we could calculate the offset of system and then do a return to libc The main issue that we ran into here was we wanted to send arbitrary attack or controlled payloads to system And we had trouble kind of getting the user controlled strings to consistently be passed in the system So we realized that user controlled data was being stored in one of two locations in the executable And due to a race condition we couldn't determine if it was going to be in one location or the other So we tried to figure out if there was some way we could make it more consistent And we came up with the following So we sort of called this space sleds where we were inspired by the not-sled techniques In which nots are propended to a shellcode payload and it improves exploit consistency So here we prepend spaces to the user controlled command string And we hope that that would improve the consistency of our exploit So what we did here is since the two locations where we were seeing the attacker controlled payload string Were 72 bytes apart We prepended 72 spaces to our string and that created an overlap where we could point our argument to system So we have a demo where if we have enough time at the end of this we'll be able to show And then additionally in looking into the Jlink remote server we found some other really interesting functionality So we termed this the tunnel server backdoor And SEGR has this as a documented feature within their application And they said it just provides a tunneling mode from a remote computer into that your network allows remote debugging Even when they're not within the local network, correct So our first thought at that point was I wonder what type of authentication they're using If there's any weaknesses there or anything that we could exploit So as you can kind of see here they use magic numbers and your device serial number to authenticate your device So that was a pretty scary thing So when you run Jlink remote server with your Jlink device attached It sends a magic number and registers your Jlink debugger with the server And then it allows client connections to specify the serial number of the device that they hope to connect to So we looked at the serial numbers and realized that there were 10 billion possibilities Which would make brute forcing all of the connected serial numbers a bit infeasible So we wondered is this actually something that would be a potential attack? Is there some way that an attacker could shrink the space of serial numbers that they would attempt to brute force? So we began looking into is there some format that SEGR uses for their serial numbers? Is there like some smaller range or number that they start at that we could use to shrink the space? So we googled SEGR Jlink and found tons of images that people had posted online that included their device serial number We called some people that we know that also had SEGR Jlinks and asked their device serial number And in the end we got about 30 serial numbers that we were able to analyze And started to notice a few patterns that emerged So the first two digits of the serial numbers were correlated with the model of device Whether you had a Jlink Pro or a Jlink Mini After that two digits for the version where SEGR would release a slightly updated version that would increment And then finally the last five digits are a device unique number that's incremented So doing a little bit of analysis on the numbers that we had We were able to shrink the serial number space and still achieve what we think would be very good coverage With only 100,000 serial numbers And assuming about 10 serial numbers per second could be brute-forced which is about the link that we saw the connections taking When it was done properly via the Jlink application It would take the time from 31 years to brute-force the space to about 3 hours The impact here really is once you make a connection via this server you can do all sorts of things You can flash new firmware, you could read the existing firmware Or make any sort of malicious modifications to the connected devices So we disclosed SEGR and we're really pleased with the response They responded right away and we got a lot of feedback within a week And it was very, we even received a thank you from the founder and CTO of the company So in conclusion we just really realized that these devices are often times lacking And a lot of the protections and security that we would hope from a device piece of the supply chain It was lacking the PIE flag which would have made our exploitation much more difficult As well as having lacking the security features that are just architectural and no authentication in the remote server or anything like that But wait, there's more Okay, before we run out of time We revisited the JLink hardware but instead of from the perspective of trying to manipulate the firmware with a hardware debugging device From the OTA process We realized that if you download a newer version of the JLink software suite JLink Commander will ask you if you want to update your JLink debug probe And so we started from the network perspective and found that there was no firmware blobs being transferred over the network And we thought that was kind of interesting So we eventually figured out where firmware was being stored and how the update process worked And we reversed the USB protocol to be able to flash the JLink devices And it turns out that the JLink device, when it's being flashed over USB, it does do a firmware check But it's not very good And it uses dates to check whether the firmware is valid or not So this could be a bad thing Also, firmware is not signed and can be modified So in the end, we don't have time for the video But to illustrate it, we created a piece of malware that runs on Windows It runs silently in the background And just any JLink device that is connected via USB immediately flashes and completely bricks By flashing firmware that's set way far in the future And you can't fix your device unless you have hardware debugging privileges And that's it Cool Thank you
everybody This is a infecting the embedded supply occasionally does security things at summer set that kids can talk to and have a full products had in common and We thought why not tools and a graphical debugger We primarily focused on the software suite used to control what these devices what are the security some of the desktop software that was distributed they distributed their software to both Linux and Windows systems In reverse engineering this we vulnerabilities that we could begin to exploit So we tested all input vectors that these to use a generational fuzzing approach so we used really realized that these devices are often