IoT Village - The Joy of Coordinating Vulnerability Disclosure
Crob
Bradley, Lisa
Noble, Katie
Santos, Omar
Fogh, Anders
Gruss, Daniel
Under the best of circumstances, coordinating disclosure of vulnerabilities can be a challenge. At times it can feel like everyone involved in CVD has conflicting motivations. The truth is that all of us are aspiring to do the right thing for end-users based on our perspective. The panel will share experiences and show how researchers and technology companies can work together to improve the impact of disclosing vulnerabilities on the technology ecosystem. Join CRob (Red Hat), Lisa Bradley (Dell), Katie Noble (Intel), Omar Santos (Cisco), Anders Fogh (Intel) and Daniel Gruss (TU Graz) for an exciting and engaging dialog between security researchers and industry experts on the Joy of coordinating vulnerability disclosure.
 From your perspective, what does CVD mean to you? And kind of what are your motivations when you're working on an issue and you need to disclose something? So I think the main point of CVD is of course to protect users. And my main thought is a bit historical. So I've been around in the 90s where disclosure was you find a mailing list where other hackers hang out and you zero day everybody. And nobody really cares. And then became responsible disclosure, which is pretty sister for coordinated vulnerability disclosure. The problem with responsible disclosure is that it was really made to shame hackers that didn't follow the desired protocol of industry types. And the current iteration, the coordinated vulnerability disclosure is instead of trying to be coercive, trying to actually work with people that want to submit issues. And it is a much, much improved over that. And it does a much better job of making sure that everybody to a vulnerability disclosure, that means vendors, customers and hackers have their needs met. So that's what I think vulnerability disclosure, coordinated vulnerability disclosure is for me. Awesome. Daniel, another researching perspective. What do you think about coordinated vulnerability disclosure? So we are interested in understanding how things work and sometimes that involves understanding how things work that are not supposed to happen, which we then call vulnerability. And of course, then we want to figure out the truth behind it. And that involves talking to to the vendor. Very often they tell us that we got something slightly wrong. And that is an opportunity to correct these mistakes before we submit the paper or publish the paper. But also, from my perspective, it's the only ethical thing to do to protect the customers, to protect people who are using these products. And for me, I'm not in this game for finding vulnerabilities as a motivation by itself. My main motivation is finding the truth, finding better understanding of how things work. That's awesome. Katie, you've had a lot of experience, monkeying around with CBD. What does the CBD mean to you? Hold on a minute, Lenea. All right. So CBD means to me. So when I think about it, I think of it from a risk-based perspective. And so what we're probing is alluding to here is that my background is government in nature. So I spent about 15 years in the US government, 12 years of that in the US Air Force. And then several other years at Department of Homeland Security, where I ran the vulnerability disclosure profiles programs that most people are familiar with. So like the NIST, NDD program and the Carnegie Mellon CERT CC program and the MITRE CV program. So I was a sponsor for those programs. So in a single year, 2017, we coordinated and disclosed 14,800 vulnerabilities for public disclosure. So it's 14,000 IT vulnerabilities and about 800 ICS vulnerabilities. In the following two years, so 18 and 19, we coordinated and disclosed over 20,000 cybersecurity vulnerabilities. So I've kind of seen things from lots of different perspectives. And when I think about CBD, I think about balancing risk, right? So CBD is really a process. And every time you, when you're going through this process, understand like every organization is going to treat it differently. Because there isn't just standardized like one size fits all kind of here. There's differences that are going to happen across the coordination stack. So if you're looking at digital services, it's going to be different from software, which is going to be different from open source, which is going to be different from hardware or ICS, like all the differences are going to come into play there. But the overarching sort of thing, the big takeaway here is that CBD is balancing risk. It's all about making sure that there is an opportunity for the product vendor to fix the problem before an adversary has the opportunity to take advantage of that. So like it's all about protecting the end user. And I think everybody that I've ever met in the entire ecosystem is all focused on that. Like we may be speaking different languages, we may talk past each other, but I think that everyone is trying to protect the end user. So to me, that's what CBD is about. That's awesome. And remind me to circle back to you later in our talk. Talk a little about some of the psychology behind some of this. Oh yeah, absolutely. I would love to hear my dearest friend Lisa's thoughts on kind of coordinating vulnerability disclosure and kind of what some of your motivations that you've seen or kind of what's behind your motorcycle. Yeah, so I think where I've seen a sort of an evolution of what we do now. I think Heartbleed was sort of the big start of where we paid attention, realized we all needed to come together a little bit, and then Spectre Meltdown sort of brought us even closer. So my thought is that not only do we work with the researchers, but we work across the industry to make sure that we're all doing our best to protect our customers. So like Katie was talking about, it's really our end users that are our focus and making sure that we have the right people involved to be able to solve the issue and then provide a security update so that our customers could get it and be best protected. That I didn't say CBD. Oh, you just did that. I know. Yeah, yeah, yeah. So Omar, kind of you've been doing this for a while and you work for a very large company. So I'm sure you get the opportunity to see a lot of different vulnerabilities. So what what does the whole process meaning? What are some of the motivations you've seen? Yeah, I think that pretty much everybody has summarized very well is protecting the end consumer at the end of the day. But I see it as a very, very complex ecosystem, right that we're trying to actually solve. And Katie, you know, mention in one side hardware, another side software, then if you decompose that then you have open source, which we're going to probably talk about a lot in here. And you have things that, you know, perhaps you cannot even control. I have had a, I guess, a pleasure or not a pleasure in some cases where we're looking at vulnerabilities and even in some cases whenever we try to actually solve the issues, the companies don't even exist anymore. That's another predicament, you know, that in some cases we actually have to take into consideration and it's a fairly complex ecosystem. And at the end of the day, what we have to actually put our heads together is how can we modernize our practice into the way that not only yes, we deal with a single vulnerability, we reproduce it, we actually fix it, we find the patch, but how can we accelerate that process that one everybody's talking that somewhat the correct vocabulary, or at least they can understand each other. And then we also understand the overall risk, right, and how to prioritize things and so on. So it's a fairly complex, we can actually talk about it for quite some time and that's what we're here for. But that's what some of the initial perspective that I want to share. Thank you, I appreciate that. I want to talk about something we touched on in one of our prep calls. We have Anders and Daniel that come at this research angle from slightly different angles. So maybe, and you too, Jelon can figure out who wants to start first, but thinking about the perspective of academia versus a professional bug hunter security researcher, and maybe you can maybe describe some of the particulars in either area, and maybe what are some of the differences you might see between those two different types of research. So from the industry, or rather the bug hunter, the professional or not, there is very often an element of good old fashioned fun. I started out hacking things, not because I wanted to achieve anything with it, just because it was great fun. When you become pro with it, there is different motivations. So if you work for a company and you work on their products, obviously your motivation is to make those products secure. And sometimes my past job, I did some research and that was sponsored by that company, but essentially me doing what I like to do. And their end of it was attention to that company's competences. And my end, of course, was having a bit of fun. So that is probably pretty typical of what hackers get out of hacking. Yeah, from the more academic side, it's more of advancing the field, the knowledge in the field, and there you try to figure out how things work and what the implications there are, what the security implications of certain understandings are. For instance, understanding when you can execute a certain piece of code and that does something that is not intended. What are the implications? And this is interesting from a security perspective than if it enables someone to do something that they shouldn't be allowed to do. Yes, often this involves disclosing a vulnerability to a vendor. But I would say that, for instance, bug bounties that might be very relevant for people from the industry. I think this plays a smaller role in academia because you have to participate in CVD anyway because if you wouldn't, then you would get a lot of problems in the academic community. There's a lot of peer pressure that you participate in this because that's the only ethical approach to handling vulnerabilities. At the same time, if you keep them secret for too long, then there's also the question on why did you keep them secret for so long. Maybe also some perspective that I can share in academia, you see more and more often the fact that academic publications are easier to publish if you have a CVE. I don't think that's a good thing because I think that a CVE does not necessarily describe that a research result really brings you new insights. A CVE is an identifier for a vulnerability, not an identifier that says this is something with a new insight. Also, if you participated in CVD and you mentioned this in the paper, this also gives you bonus points. At least that's my impression to get the paper published. Of course, it's good if you participate in CVD, but my feeling is also that this is going a bit too strong and it's now overemphasized in our community. What about having an icon or a fun name? Of course, I'm really a big fan of logos and names and anything that is fun. But of course, there are multiple layers here. For instance, we had this paper, Hello from the Other Side, which had this name not just by coincidence. It was about a covert channel in the cloud where we send data through the cache from one virtual machine to the other. We really went crazy on that. We built an entire TCP stack on top of that and tunneled an SSH session through the cache cover channel for whatever reason. We then sent a music video through this cache cover channel. Of course, we couldn't just take any music video, so we had to make our own parody of Hello from the Other Side. This is just fun. We are just a bunch of people and we like to have fun. Basically, once you've finished your project, it's really nice to close this up with something nice and funny. Logos on the Other Side also help communicate about the issue. I realized that every time I create slides, my favorite slides are the slides that don't have any words on them, just pictures, logos and icons. Then I can follow the speaker. I don't have to read because I'm very bad at reading. I can't read and listen at the same time. I bet many people can't do that. If there are only icons, symbols and logos, I can follow what the person is saying and look at the images at the same time. I'm always annoyed if I have to speak about the vulnerability and I don't have a logo for it because then I have to put text on a slide. Since Daniel brought it up, let me ask the other panelists, how do you all feel about branded vulnerabilities? Do you like logos? Do you find that helpful to you and your constituents? I'm sorry, Daniel, no. When there's a logo or a PR or everything, it grabs the media's attention so quickly. Sometimes the issue is not that severe or very hard to be able to breach. It gives a little bit of extra fear to our customers. It also encourages us to maybe reprioritize other issues first, which are more severe to our customers. Although I get the point, it's easy to talk about it. They grab the media's attention really quickly. If you put a caveat about the real severity of the issue in there, maybe that would help a little bit. We often have in our team this discussion, should we have a logo, should we have a website or shouldn't we? For most of our papers and most of the vulnerabilities that we discovered, we don't. I think that's also the responsibility of the researcher to assess, is this something that significant that I need this additional PR or is it not? Right. Well, then you're doing it the right way. Maybe, but it's also, I mean, I will have a different view on what is really important than you have. Everyone will have a different view on that. So something that I think is really, really important, you might say, well, it's not really exploitable in our use case. Right. We don't ever want to downplay a researcher's work. I mean, it's important, whatever they do, and we appreciate especially doing CVD with them and not being zero-date. But yeah, it's a more matter of, especially in a bigger industry when you're trying to protect your customers best, how do you approach it? There is a fight for customer attention. And for customers to make the right security decisions, they need to be attentive about the right things. And logos and hype in the press warps that and we have bad security outcomes for our customers due to that. Then of course, there's also the thing that even fixed vulnerabilities sometimes costs a cost of suffering for system administrators running around in basement and patching systems and all the like. And to some extent, I sometimes find over high bucks disrespectful to those people. Yeah, so in my case, I don't have an issue with with logos or anything, even though in a previous call and a couple of exchange Daniel, I alluded into that. Whether it's a logo, as a matter of fact, even for Cisco, we had the first vulnerability with emojis. So we have emojis, logos and everything else. So at the end of the day, if it helps to do an alias for a vulnerability and bring some awareness, I'm perfectly okay with that. The only challenge that I'm seeing or opportunities right is that in some cases whenever we write information and even vendors whenever we, we know we have in my case, we have a research institution at Cisco called Talos and we find vulnerabilities in other people's product right. And yes, yes, yes. So in that case, yes, we also have logos and we put them out there and we put the blog post and so on right. But what we need to do collectively is to make sure that the media is not either downplaying it or over proportion. So we have to have a balance right and we all, you know, say the media, the media, the media, right. So what is our responsibility right there on how we create our security advisories. And it's both ways is, you know, even though we're vendors here and everything, you know, but we have to point fingers to us, you know, how clear is our advisories that we have some collateral that the end consumer actually knows about, you know, what are the implications are and we work with a researcher to make sure that we all understand, you know, what the problem is. And in the previous conversation, we also I also mentioned that the biggest nerd fights in history. It's whenever it comes to CVSS scoring, right. And, you know, whenever it comes to risk and at the end of the day, you know, whether it's CVSS whenever we come up with some new ways and everything we have to have some type of way of saying, yeah, you have to jump right now and fix this vulnerability versus the other 100 criticals that we were also, you know, dependent on and especially with nowadays since it's not so much of a vulnerability coming from like a Cisco or an Intel or Red Hat or anything else is open source right by the time that actually I'm boring you to death in here, probably three more CBEs that are super important has been disclosed. And we don't know about right so so that's the type of balance on the type of, I guess, for a corny way of saying orchestration that needs to take place right in this CBE. I want to add another point here regarding the overhyping and having a logo having a website for something for a vulnerability for research result. It does not necessarily mean that you overhyped it we had experiences where we just put papers on archive without any website without any logo and anything and media picked it up and media reported about it and not necessarily correct in all cases and we have this, for instance, for the, the takeaway paper where we analyzed some site channels on and DCP use and their media picked it up we didn't we intentionally didn't make any website or logo but media picked it up and the reporting. It was not, it was not entirely wrong but it was definitely I would say definitely overhyped. We discussed this also in our team and the conclusion that we had was that in some cases we should have a website, even if it's not a vulnerability that we want to hide because it's not that significant, but just to have a very clear message, which says how relevant is it. I think that's the important part that we also always had on our websites, a short three sentence summary like what is it who has to care about this and what can an attacker do. Yeah, I think that's great. That's great you do that and it's good advice. I think, you know, that I would love to see it be more adopted and in the researcher world. For sure. I think it would help out. Before I move on to my next question, I'm just going to say, when I retire, all of you in vendor world are doomed because I'm going to offer my services to L reg for free writing up crazy headlines. But the touch on a point that Daniel made he invoked the Katie Miseris clause. Let's talk about bug bounties and CVD. Oh, Katie maybe you should take that one. Because our friend Katie and a little trimble here actually has a lot to do with that in her organization. Could you maybe talk about how coordinated disclosure interacts both good and bad with bug bounty. Yeah, so I love Katie Moe. I think that first off let me just say that it is an honor to be confused with Katie Moe. My hair color is like, you know, getting close to hers too now. I don't know. I think I have been accepted to conferences because they thought that I was she. When I showed up, I don't think they were nearly as excited to see me instead of her so but so yeah bug bounties. So bug, tin foil hat really. So I'm trying to be serious here about bug bounties. So what's that is a tool right so they're a tool in a toolkit and they are there to incentivize, but they're a part of a well structured product security portfolio. They're not the whole thing, and there are different motivations that will help people from different perspectives. So in the academic world, for instance, but that is may not weigh as much because a lot of academic institutions don't allow an individual to accept that the heart coal hard cash right. If you are a professional bug hunter, you know that might be how you pay your mortgage and so there's a lot more tied to that. But the problem becomes that like, bug bounties are a wonderful tool and they're, they're great but you have to have a good program in place already to accept that that information to be able to execute on that information and figure out like how do you, how do you actually how there are so many questions about it how do you award how do you manage it like, are we going to tie the payments to see the SS scores are we going to tie the payments to a well thought out proof of concept there's so many pieces that go into it and so I'd say like bug bounty is not a one size it's all it's going to vary from organization to organization and some organizations are going to have different timelines different pieces. It's, yeah it's complex. So, bug bounty is not the end of the world bug bounty is a tool. It's a tool in your toolkit. So it's a great tool. I love the tool because I'm the director of it and tell. But it's not, it's not everything. So vulnerability disclosure is more than bug bounty alone. Yeah, I think when you think about it to like there's a lot of smaller companies that probably, you know, don't have that or don't have the funding, funding for it. But that that doesn't mean that they're not wanting to work with researchers they just can't get the budget to do it. You know, so, so, you know, do your best to try to figure out how to reach out to those companies. Hopefully they have a web page or email address, you know, secure at PISER that security at. I know we struggle a little bit but, you know, there's plenty of us around that could help find the right contacts because we're you sort of come together from all over the place and we're sort of expanding group here I like it. But boundaries are one of the wonderful signs of how the industry has changed back in the 90s. Researchers was often rewarded with that disclosure was rewarded with lawsuits. And now people are the industry are working with the people and have started actually paying the researchers for their efforts. So I very much thumbs up on what boundaries, especially because it shows how the industry has changed on all of you hackers hackers on our days helpers and not the enemy. Yeah, I thought you were going to say t shirts for a minute there in the early days. I like to see it just one to see shirt. Oh, early on the night it was lawsuits, then it became t shirts and then maybe stickers and then. Yeah, and now, and now financial. Awesome stickers. So let's we've touched on it a little bit let's see if we spend some time now we're past the top of the hour here. Let's talk about coordinated vulnerability disclosure inside an open source context. So, I'll go last, but let our esteemed panel here maybe whether it's our researchers are vendor friends kind of describe what your, what works out really well and what was some challenges for you within the open source world, which makes up about 90% of all software now. Jerry open source one. I guess I start. So, so basically, in that stop of mine for me, and it's, you know, it sounds corny whenever people say hey what keeps you up at night and everything else is actually open source right now. It's not so much of the predicament of using open source, we have to use and embrace and contribute to open source and I'm super big fan of that. The challenge when it comes to open source is that it can be anything right like IOT can be anything right so, and it's critical infrastructure for a lot of things. So, to give you some somewhat of a real life you know I guess a realization that we had a while back probably about four or five years ago whenever hardly came, we were looking in the industry right now not only Cisco but a whole bunch of other companies right so what to what to prioritize as far as actually giving funding, probably doing research, and so on so we said, okay so open open SSL and things like that which are actually super important for us. There's a lot of people looking at it right now. So can we look at things that perhaps are actually critical infrastructure that nobody actually has probably taken a look at it right. We're going down the list. And then, and as a matter of fact that number two is perfectly fine because there are two guys that work on open SSL to get paid to do it. Yes, indeed, indeed. And the example that I was going to go is NTP, the network time protocol so NTP is specifically. And there's also two guys that don't get paid, which is bad right is there, not even the full time job right. Well, I guess it's now your fast forward this year is a little bit better now right so it's a lot better. And it's not that the problem is that poor guys that actually are contributing to the code is a matter of scalability is a matter of actually even, you know, running static analysis into the thing didn't even assist, you know, five years ago for these components. Now of course we're modernizing our ways you know even if you submit things of GitHub a lot of things actually happening and we're we're getting we're getting better for sure. It's just that it's getting way more complicated right and more people actually are in not only of course contributed but using it more than contributed, which is the other predicament. And in some cases, we were talking about bug bounties. In some cases actually amazing. I'm a big fan of bug bounties right the challenge is that in some cases, we also don't think about does this affect other vendors, and my actually finding some type of vulnerabilities that you know perhaps yes it's a single injection cross ice scripting which is actually pretty pretty common but I'm doing some fuzzing and I, you know, crash an application. What is the underlying issue right and in some cases actually kind of a commodity you know goes back and forth, and it hasn't been shared. And then two or three months down the road you say oh, but this CBE was not shared with other these vendors that they're also affected. So we go back and, and in some cases you actually see people trying to find the same or found the same CVs and they reported a different way but you know, there were no coordination so those are the things that are that are crucial right for us to be successful and not only among vendors but actually downstream and upstream as well right so especially when it comes to IOT is like number one. Yeah so Omar you brought up a few points you know one is who do you bring in right ahead of time and I think that we're seeing that even if you're the competition we still want to work together in the background here. You know, we, our whole goal is to protect our customers, and we often have the same customers even if we're competitors. So we like to, you know, make sure that we're all working together and ideas is when do you bring someone in and it's not only the researchers. Okay, CROPE, where you got your hand raised go ahead. Can I tell you it's a secret. Yes. Open source has worked with our competitors for 25 plus years. Yeah, yeah, well, I would have to say, we, you know, with with some of those other vendors in the industry, it wasn't always so nice. They were playing with the close source. We're getting better. But I think what you know what would not only the researchers should be thinking who else is affected but when we are the receiving and we should be thinking who else is affected and bring them in. I recently worked on an issue of a very recent one where the we are in Key Base and the researchers were in right with the industry people who were fixing the issue. And there was great, you know, coordination between the pink, you know, amongst the group there. It was pretty awesome to see of how, how much we've evolved. How do you do that though with like open source though when everything's shown. It's super easy. Well, and I'll, I guess I'll chime in before we let other folks. I found it, I found it very interesting how some vulnerabilities have been patched wide in the open under the cover of some other patch. I don't know what yeah, I don't know what you're talking about at all. So, in general, open source, there is no single definition of what open source is you could have two young ladies in Bangladesh that have an amazing idea and they're just trying to get this creativity out there to share with the world. You could have large corporations like a lot of the folks represented here. I mean you can have academics. So open source is a lot of different things and there's really no one definition or moniker that fits that works with every community. But if you're thinking about the types of open source that might make up a product or some kind of a cloud offering, you're going to have the high end of the spectrum or things like the kernel group, which is very mature and organized and then you get you start moving down to like a patchy foundation, these other large, large communities and then you get down to something where it's a single person, a couple people that are just playing around. And it's hard to put any kind of structure or process to all these different models because a lot of people that code for open source are doing it for free for no renumeration from large companies that make a lot of money off of it. But they do it for free because they love kind of adding value and expressing their creativity. And as Daniel mentioned, open source is very creative and some of the larger projects we do have ways to privately take in data. So we take in a private bug report that's well established within the community for a long time. And when you're looking at a less mature community or package or library, they might not have that capability. And that's where larger kind of big brothers and big sisters like a Red Hat or a Susie are canonical kind of step in and try to help mentor these smaller projects to help them get these good practices set up so they can take in a private note. Because sometimes a lot of my team we track between 3,000 and 5,000 vulnerabilities a year out of 450,000 packages. That's a lot. Not all of them really kind of bubble up to the level of they need the attention of like a Spectre meltdown or a Heartbleed or a Blueborn, all these kind of the other big name nonsense things. But they still need to get fixed and what open source is really good at is you identify a problem. The team attacks it collaboratively. They develop a solution and release that update very quickly. And they don't like to spend a lot of time making a big deal out of it because they've already moved on to the next feature, the next big thing. But yeah, it can be challenging, but there are definitely ways to do it. There are methods to do it. There are groups that allow this. I hear a lot of bleep into the background. What's going on, folks? I'm wondering when you're going to change your hat again. Sorry. There we go. Bring out Big Red. All right, let me get this panel back under order. When you're thinking about doing a CVD, what are some things you might want to prepare for? What can you do to help make that coordination very successful? And let me start off with, we'll start off with Lisa because she has a lot of ideas. So from your perspective, what will make you successful when you're getting ready to do this multi-party coordination? So I guess it depends on how familiar you are with it or not. If I was a researcher and I wasn't too familiar, I would potentially utilize cert or something like that if I felt like it was going to cross over more than one company in the industry. Just to have them help. Because I think it could be overbearing of figuring out how if one, you know, if industry, you know, company A wants this date to get it fixed and company B wants another date and company C wants another date. So I think that's one way that people, a researcher could use. But I think the idea is to figure out when you start is what are your rules? What's your embargo date? How many days do you actually want to, you know, give the company or vendor to fix the issue? Basically, it's 90 days. I would prefer that as a company. I think it's respectful to do that, especially when things get more difficult. So think about your rules and what you want to do and how you want to approach it. But I'll pause there because I know we're running out of time to get other people in. Anders, what are your thoughts about how you could make CVD successful? Three things. Use it. It's a tool not only for vendors but also for hackers. Listen to what vendors are saying or what hackers are saying and try to make the best out of it and be aware that there's often a lot of complexities involved with it. Be patient, right? Yeah. Omar, what are your thoughts on how you can make CVD successful? Yeah, I think that I'm going to capitalize on something that Daniel mentioned in a previous call. And in some cases, even if you provide some data to even a vendor or whoever, right, to whenever you're coordinating, you have to make it first easy to understand. But at the same time, in some cases, getting the right people at the right place at the right time to make sure that you understand the technical implications of a given vulnerability and sort of running around because even if you have the 90 days or 60 days or 100 days, getting that streamline and modernizing the way that we actually exchange the information among all the affected parties, that is number one. And Art Manion, which is actually a good friend of mine, he leads the team. We all love art. He's the first one that will tell you we cannot scale. This is a big ecosystem. Having a roll of decks of older vendors that actually use a component is foolish, right? We will never ever be able to actually have a complete thing. But what we have to think about is what Lisa mentioned is that in some cases, we are working with competitors, right? Even more than our companies. For example, you know, I work with Lisa a lot. I work with Juniper even more in some cases more than I work with Cisco whenever it comes to fixing a vulnerability, let's say in BGP, OSPF or whatever the case might be. So that type of thing of, oh, I'm not going to talk to these competitors. They're probably my everything else. It's not, you know, that's 20 years ago, fallacy. And then the last one is that at the end of the day, I have the reality and I tell my guys at Cisco is that whenever I push about the night I publish a CV out there. The number ones that actually are reading that CV are the bad guys, right? And probably at the same time that you're committing an open source code, you know, probably that's going to be the case too. So what we have to do is think about how can we change information to the consumer to downstream and upstream providers in a more modern way. And another sounds corny, but you know, can we actually make it more and we're trying to do that machine readable? Right? So if in the case that we actually have that role of the X of people and everything else, you know, get some tool. And that's what art and you know, the guys in insert that also do and you know, we're creating tools that allows us to do that. And even if you have, I don't know, even two years ago that didn't assist and it wasn't even a thought. So we have to move faster. That's the one thing that I want to say. So as I close, I'll say art manion has the best facial hair in the industry. I love art. I want to thank our panel here. This was a great hour together. I really appreciate your time and expertise. We're going to be hanging out for a little bit on this awesome discord channel that my kids made fun of me all week about because I've had it up. Thank you everybody for coming and your attention here panelists. And thank you to the audience for DEF CON for having us here at the IOT village. Enjoy your day and enjoy the rest of the column. Got some great stuff lined up the rest of the weekend. CBD all the way. Oh yeah. We're out.
 From your perspective, what does CVD mean to you?  And kind of what are your motivations when you're working on an issue and you need to disclose something?  So I think the main point of CVD is of course to protect users.  And my main thought is a bit historical. So I've been around in the 90s where disclosure was you find a mailing list where other hackers hang out and you zero day everybody.  And nobody really cares. And then became responsible disclosure, which is pretty sister for coordinated vulnerability disclosure.  The problem with responsible disclosure is that it was really made to shame hackers that didn't follow the desired protocol of industry types.  And the current iteration, the coordinated vulnerability disclosure is instead of trying to be coercive, trying to actually work with people that want to submit issues.  And it is a much, much improved over that. And it does a much better job of making sure that everybody to a vulnerability disclosure, that means vendors, customers and hackers have their needs met.  So that's what I think vulnerability disclosure, coordinated vulnerability disclosure is for me.  Awesome. Daniel, another researching perspective. What do you think about coordinated vulnerability disclosure?  So we are interested in understanding how things work and sometimes that involves understanding how things work that are not supposed to happen, which we then call vulnerability.  And of course, then we want to figure out the truth behind it. And that involves talking to to the vendor.  Very often they tell us that we got something slightly wrong. And that is an opportunity to correct these mistakes before we submit the paper or publish the paper.  But also, from my perspective, it's the only ethical thing to do to protect the customers, to protect people who are using these products.  And for me, I'm not in this game for finding vulnerabilities as a motivation by itself. My main motivation is finding the truth, finding better understanding of how things work.  That's awesome.  Katie, you've had a lot of experience, monkeying around with CBD.  What does the CBD mean to you?  Hold on a minute, Lenea.  All right. So CBD means to me.  So when I think about it, I think of it from a risk-based perspective.  And so what we're probing is alluding to here is that my background is government in nature.  So I spent about 15 years in the US government, 12 years of that in the US Air Force. And then several other years at Department of Homeland Security, where I ran the vulnerability disclosure profiles programs that most people are familiar with.  So like the NIST, NDD program and the Carnegie Mellon CERT CC program and the MITRE CV program.  So I was a sponsor for those programs. So in a single year, 2017, we coordinated and disclosed 14,800 vulnerabilities for public disclosure.  So it's 14,000 IT vulnerabilities and about 800 ICS vulnerabilities.  In the following two years, so 18 and 19, we coordinated and disclosed over 20,000 cybersecurity vulnerabilities.  So I've kind of seen things from lots of different perspectives.  And when I think about CBD, I think about balancing risk, right?  So CBD is really a process. And every time you, when you're going through this process, understand like every organization is going to treat it differently.  Because there isn't just standardized like one size fits all kind of here.  There's differences that are going to happen across the coordination stack.  So if you're looking at digital services, it's going to be different from software, which is going to be different from open source, which is going to be different from hardware or ICS,  like all the differences are going to come into play there.  But the overarching sort of thing, the big takeaway here is that CBD is balancing risk.  It's all about making sure that there is an opportunity for the product vendor to fix the problem before an adversary has the opportunity to take advantage of that.  So like it's all about protecting the end user. And I think everybody that I've ever met in the entire ecosystem is all focused on that.  Like we may be speaking different languages, we may talk past each other, but I think that everyone is trying to protect the end user.  So to me, that's what CBD is about.  That's awesome. And remind me to circle back to you later in our talk. Talk a little about some of the psychology behind some of this.  Oh yeah, absolutely.  I would love to hear my dearest friend Lisa's thoughts on kind of coordinating vulnerability disclosure and kind of what some of your motivations that you've seen or kind of what's behind your motorcycle.  Yeah, so I think where I've seen a sort of an evolution of what we do now.  I think Heartbleed was sort of the big start of where we paid attention, realized we all needed to come together a little bit, and then Spectre Meltdown sort of brought us even closer.  So my thought is that not only do we work with the researchers, but we work across the industry to make sure that we're all doing our best to protect our customers.  So like Katie was talking about, it's really our end users that are our focus and making sure that we have the right people involved to be able to solve the issue and then provide a security update so that our customers could get it and be best protected.  That I didn't say CBD.  Oh, you just did that.  I know.  Yeah, yeah, yeah.  So Omar, kind of you've been doing this for a while and you work for a very large company. So I'm sure you get the opportunity to see a lot of different vulnerabilities.  So what what does the whole process meaning? What are some of the motivations you've seen?  Yeah, I think that pretty much everybody has summarized very well is protecting the end consumer at the end of the day.  But I see it as a very, very complex ecosystem, right that we're trying to actually solve. And Katie, you know, mention in one side hardware, another side software, then if you decompose that then you have open source, which we're going to probably talk about a lot in here.  And you have things that, you know, perhaps you cannot even control. I have had a, I guess, a pleasure or not a pleasure in some cases where we're looking at vulnerabilities and even in some cases whenever we try to actually solve the issues, the companies don't even exist anymore.  That's another predicament, you know, that in some cases we actually have to take into consideration and it's a fairly complex ecosystem. And at the end of the day, what we have to actually put our heads together is how can we modernize our practice into the way that not only yes, we deal with a single vulnerability,  we reproduce it, we actually fix it, we find the patch, but how can we accelerate that process that one everybody's talking that somewhat the correct vocabulary, or at least they can understand each other.  And then we also understand the overall risk, right, and how to prioritize things and so on. So it's a fairly complex, we can actually talk about it for quite some time and that's what we're here for.  But that's what some of the initial perspective that I want to share.  Thank you, I appreciate that.  I want to talk about something we touched on in one of our prep calls.  We have Anders and Daniel that come at this research angle from slightly different angles.  So maybe, and you too, Jelon can figure out who wants to start first, but thinking about the perspective of academia versus a professional bug hunter security researcher, and maybe you can maybe describe some of the particulars in either area, and maybe what are some of the differences you might see between those two different types of research.  So from the industry, or rather the bug hunter, the professional or not, there is very often an element of good old fashioned fun.  I started out hacking things, not because I wanted to achieve anything with it, just because it was great fun.  When you become pro with it, there is different motivations. So if you work for a company and you work on their products, obviously your motivation is to make those products secure.  And sometimes my past job, I did some research and that was sponsored by that company, but essentially me doing what I like to do.  And their end of it was attention to that company's competences. And my end, of course, was having a bit of fun.  So that is probably pretty typical of what hackers get out of hacking.  Yeah, from the more academic side, it's more of advancing the field, the knowledge in the field, and there you try to figure out how things work and what the implications there are, what the security implications of certain understandings are.  For instance, understanding when you can execute a certain piece of code and that does something that is not intended.  What are the implications? And this is interesting from a security perspective than if it enables someone to do something that they shouldn't be allowed to do.  Yes, often this involves disclosing a vulnerability to a vendor. But I would say that, for instance, bug bounties that might be very relevant for people from the industry.  I think this plays a smaller role in academia because you have to participate in CVD anyway because if you wouldn't, then you would get a lot of problems in the academic community.  There's a lot of peer pressure that you participate in this because that's the only ethical approach to handling vulnerabilities.  At the same time, if you keep them secret for too long, then there's also the question on why did you keep them secret for so long.  Maybe also some perspective that I can share in academia, you see more and more often the fact that academic publications are easier to publish if you have a CVE.  I don't think that's a good thing because I think that a CVE does not necessarily describe that a research result really brings you new insights.  A CVE is an identifier for a vulnerability, not an identifier that says this is something with a new insight.  Also, if you participated in CVD and you mentioned this in the paper, this also gives you bonus points.  At least that's my impression to get the paper published.  Of course, it's good if you participate in CVD, but my feeling is also that this is going a bit too strong and it's now overemphasized in our community.  What about having an icon or a fun name?  Of course, I'm really a big fan of logos and names and anything that is fun.  But of course, there are multiple layers here.  For instance, we had this paper, Hello from the Other Side, which had this name not just by coincidence.  It was about a covert channel in the cloud where we send data through the cache from one virtual machine to the other.  We really went crazy on that.  We built an entire TCP stack on top of that and tunneled an SSH session through the cache cover channel for whatever reason.  We then sent a music video through this cache cover channel.  Of course, we couldn't just take any music video, so we had to make our own parody of Hello from the Other Side.  This is just fun.  We are just a bunch of people and we like to have fun.  Basically, once you've finished your project, it's really nice to close this up with something nice and funny.  Logos on the Other Side also help communicate about the issue.  I realized that every time I create slides, my favorite slides are the slides that don't have any words on them, just pictures, logos and icons.  Then I can follow the speaker. I don't have to read because I'm very bad at reading. I can't read and listen at the same time.  I bet many people can't do that.  If there are only icons, symbols and logos, I can follow what the person is saying and look at the images at the same time.  I'm always annoyed if I have to speak about the vulnerability and I don't have a logo for it because then I have to put text on a slide.  Since Daniel brought it up, let me ask the other panelists, how do you all feel about branded vulnerabilities?  Do you like logos? Do you find that helpful to you and your constituents?  I'm sorry, Daniel, no.  When there's a logo or a PR or everything, it grabs the media's attention so quickly.  Sometimes the issue is not that severe or very hard to be able to breach.  It gives a little bit of extra fear to our customers.  It also encourages us to maybe reprioritize other issues first, which are more severe to our customers.  Although I get the point, it's easy to talk about it.  They grab the media's attention really quickly.  If you put a caveat about the real severity of the issue in there, maybe that would help a little bit.  We often have in our team this discussion, should we have a logo, should we have a website or shouldn't we?  For most of our papers and most of the vulnerabilities that we discovered, we don't.  I think that's also the responsibility of the researcher to assess, is this something that significant that I need this additional PR or is it not?  Right. Well, then you're doing it the right way.  Maybe, but it's also, I mean, I will have a different view on what is really important than you have.  Everyone will have a different view on that.  So something that I think is really, really important, you might say, well, it's not really exploitable in our use case.  Right. We don't ever want to downplay a researcher's work.  I mean, it's important, whatever they do, and we appreciate especially doing CVD with them and not being zero-date.  But yeah, it's a more matter of, especially in a bigger industry when you're trying to protect your customers best, how do you approach it?  There is a fight for customer attention.  And for customers to make the right security decisions, they need to be attentive about the right things.  And logos and hype in the press warps that and we have bad security outcomes for our customers due to that.  Then of course, there's also the thing that even fixed vulnerabilities sometimes costs a cost of suffering for system administrators running around in basement and patching systems and all the like.  And to some extent, I sometimes find over high bucks disrespectful to those people.  Yeah, so in my case, I don't have an issue with with logos or anything, even though in a previous call and a couple of exchange Daniel, I alluded into that.  Whether it's a logo, as a matter of fact, even for Cisco, we had the first vulnerability with emojis.  So we have emojis, logos and everything else.  So at the end of the day, if it helps to do an alias for a vulnerability and bring some awareness, I'm perfectly okay with that.  The only challenge that I'm seeing or opportunities right is that in some cases whenever we write information and even vendors whenever we, we know we have in my case, we have a research institution at Cisco called Talos and we find vulnerabilities in other people's product right.  And yes, yes, yes.  So in that case, yes, we also have logos and we put them out there and we put the blog post and so on right.  But what we need to do collectively is to make sure that the media is not either downplaying it or over proportion.  So we have to have a balance right and we all, you know, say the media, the media, the media, right.  So what is our responsibility right there on how we create our security advisories.  And it's both ways is, you know, even though we're vendors here and everything, you know, but we have to point fingers to us, you know, how clear is our advisories that we have some collateral that the end consumer actually knows about, you know, what are the implications are and we work with a researcher to make sure that we all understand, you know, what the problem is.  And in the previous conversation, we also I also mentioned that the biggest nerd fights in history. It's whenever it comes to CVSS scoring, right.  And, you know, whenever it comes to risk and at the end of the day, you know, whether it's CVSS whenever we come up with some new ways and everything we have to have some type of way of saying, yeah, you have to jump right now and fix this vulnerability versus the other 100  criticals that we were also, you know, dependent on and especially with nowadays since it's not so much of a vulnerability coming from like a Cisco or an Intel or Red Hat or anything else is open source right by the time that actually I'm boring you to death in here, probably three  more CBEs that are super important has been disclosed. And we don't know about right so so that's the type of balance on the type of, I guess, for a corny way of saying orchestration that needs to take place right in this CBE.  I want to add another point here regarding the overhyping and having a logo having a website for something for a vulnerability for research result.  It does not necessarily mean that you overhyped it we had experiences where we just put papers on archive without any website without any logo and anything and media picked it up and media reported about it and not necessarily correct in all cases and we have this, for  instance, for the, the takeaway paper where we analyzed some site channels on and DCP use and their media picked it up we didn't we intentionally didn't make any website or logo but media picked it up and the reporting.  It was not, it was not entirely wrong but it was definitely I would say definitely overhyped.  We discussed this also in our team and the conclusion that we had was that in some cases we should have a website, even if it's not a vulnerability that we want to hide because it's not that significant, but just to have a very clear message, which says how relevant is it.  I think that's the important part that we also always had on our websites, a short three sentence summary like what is it who has to care about this and what can an attacker do.  Yeah, I think that's great. That's great you do that and it's good advice. I think, you know, that I would love to see it be more adopted and in the researcher world.  For sure.  I think it would help out.  Before I move on to my next question, I'm just going to say, when I retire, all of you in vendor world are doomed because I'm going to offer my services to L reg for free writing up crazy headlines.  But the touch on a point that Daniel made he invoked the Katie Miseris clause.  Let's talk about bug bounties and CVD.  Oh, Katie maybe you should take that one.  Because our friend Katie and a little trimble here actually has a lot to do with that in her organization. Could you maybe talk about how coordinated disclosure interacts both good and bad with bug bounty.  Yeah, so I love Katie Moe. I think that first off let me just say that it is an honor to be confused with Katie Moe.  My hair color is like, you know, getting close to hers too now. I don't know.  I think I have been accepted to conferences because they thought that I was she.  When I showed up, I don't think they were nearly as excited to see me instead of her so but so yeah bug bounties.  So bug, tin foil hat really.  So I'm trying to be serious here about bug bounties.  So what's that is a tool right so they're a tool in a toolkit and they are there to incentivize, but they're a part of a well structured product security portfolio.  They're not the whole thing, and there are different motivations that will help people from different perspectives.  So in the academic world, for instance, but that is may not weigh as much because a lot of academic institutions don't allow an individual to accept that the heart coal hard cash right.  If you are a professional bug hunter, you know that might be how you pay your mortgage and so there's a lot more tied to that.  But the problem becomes that like, bug bounties are a wonderful tool and they're, they're great but you have to have a good program in place already to accept that that information to be able to execute on that information and figure out like how do you, how do you actually  how there are so many questions about it how do you award how do you manage it like, are we going to tie the payments to see the SS scores are we going to tie the payments to a well thought out proof of concept there's so many pieces that go into it and so I'd say like bug bounty is not a one size  it's all it's going to vary from organization to organization and some organizations are going to have different timelines different pieces. It's, yeah it's complex. So, bug bounty is not the end of the world bug bounty is a tool.  It's a tool in your toolkit. So it's a great tool. I love the tool because I'm the director of it and tell. But it's not, it's not everything. So vulnerability disclosure is more than bug bounty alone.  Yeah, I think when you think about it to like there's a lot of smaller companies that probably, you know, don't have that or don't have the funding, funding for it.  But that that doesn't mean that they're not wanting to work with researchers they just can't get the budget to do it.  You know, so, so, you know, do your best to try to figure out how to reach out to those companies. Hopefully they have a web page or email address, you know, secure at PISER that security at. I know we struggle a little bit but, you know,  there's plenty of us around that could help find the right contacts because we're you sort of come together from all over the place and we're sort of expanding group here I like it.  But boundaries are one of the wonderful signs of how the industry has changed back in the 90s. Researchers was often rewarded with that disclosure was rewarded with lawsuits.  And now people are the industry are working with the people and have started actually paying the researchers for their efforts.  So I very much thumbs up on what boundaries, especially because it shows how the industry has changed on all of you hackers hackers on our days helpers and not the enemy.  Yeah, I thought you were going to say t shirts for a minute there in the early days. I like to see it just one to see shirt.  Oh, early on the night it was lawsuits, then it became t shirts and then maybe stickers and then.  Yeah, and now, and now financial.  Awesome stickers.  So let's we've touched on it a little bit let's see if we spend some time now we're past the top of the hour here.  Let's talk about coordinated vulnerability disclosure inside an open source context.  So, I'll go last, but let our esteemed panel here maybe whether it's our researchers are vendor friends kind of describe what your, what works out really well and what was some challenges for you within the open source world, which makes up about 90% of all software now.  Jerry open source one.  I guess I start.  So, so basically, in that stop of mine for me, and it's, you know, it sounds corny whenever people say hey what keeps you up at night and everything else is actually open source right now.  It's not so much of the predicament of using open source, we have to use and embrace and contribute to open source and I'm super big fan of that.  The challenge when it comes to open source is that it can be anything right like IOT can be anything right so, and it's critical infrastructure for a lot of things.  So, to give you some somewhat of a real life you know I guess a realization that we had a while back probably about four or five years ago whenever hardly came, we were looking in the industry right now not only Cisco but a whole bunch of other companies right so what to what to prioritize as far as actually  giving funding, probably doing research, and so on so we said, okay so open open SSL and things like that which are actually super important for us.  There's a lot of people looking at it right now. So can we look at things that perhaps are actually critical infrastructure that nobody actually has probably taken a look at it right.  We're going down the list.  And then, and as a matter of fact that number two is perfectly fine because there are two guys that work on open SSL to get paid to do it.  Yes, indeed, indeed.  And the example that I was going to go is NTP, the network time protocol so NTP is specifically.  And there's also two guys that don't get paid, which is bad right is there, not even the full time job right.  Well, I guess it's now your fast forward this year is a little bit better now right so it's a lot better.  And it's not that the problem is that poor guys that actually are contributing to the code is a matter of scalability is a matter of actually even, you know, running static analysis into the thing didn't even assist, you know, five years ago for these components.  Now of course we're modernizing our ways you know even if you submit things of GitHub a lot of things actually happening and we're we're getting we're getting better for sure.  It's just that it's getting way more complicated right and more people actually are in not only of course contributed but using it more than contributed, which is the other predicament.  And in some cases, we were talking about bug bounties.  In some cases actually amazing. I'm a big fan of bug bounties right the challenge is that in some cases, we also don't think about does this affect other vendors, and my actually finding some type of vulnerabilities that you know perhaps yes it's a single injection  cross ice scripting which is actually pretty pretty common but I'm doing some fuzzing and I, you know, crash an application.  What is the underlying issue right and in some cases actually kind of a commodity you know goes back and forth, and it hasn't been shared. And then two or three months down the road you say oh, but this CBE was not shared with other these vendors that they're also affected.  So we go back and, and in some cases you actually see people trying to find the same or found the same CVs and they reported a different way but you know, there were no coordination so those are the things that are that are crucial right for us to be successful and not only among vendors but actually downstream  and upstream as well right so especially when it comes to IOT is like number one.  Yeah so Omar you brought up a few points you know one is who do you bring in right ahead of time and I think that we're seeing that even if you're the competition we still want to work together in the background here.  You know, we, our whole goal is to protect our customers, and we often have the same customers even if we're competitors. So we like to, you know, make sure that we're all working together and ideas is when do you bring someone in and it's not only the researchers.  Okay, CROPE, where you got your hand raised go ahead.  Can I tell you it's a secret.  Yes.  Open source has worked with our competitors for 25 plus years.  Yeah, yeah, well, I would have to say, we, you know, with with some of those other vendors in the industry, it wasn't always so nice.  They were playing with the close source.  We're getting better. But I think what you know what would not only the researchers should be thinking who else is affected but when we are the receiving and we should be thinking who else is affected and bring them in.  I recently worked on an issue of a very recent one where the we are in Key Base and the researchers were in right with the industry people who were fixing the issue. And there was great, you know, coordination between the pink, you know, amongst the group there.  It was pretty awesome to see of how, how much we've evolved.  How do you do that though with like open source though when everything's shown.  It's super easy.  Well, and I'll, I guess I'll chime in before we let other folks.  I found it, I found it very interesting how some vulnerabilities have been patched wide in the open under the cover of some other patch.  I don't know what yeah, I don't know what you're talking about at all.  So, in general, open source, there is no single definition of what open source is you could have two young ladies in Bangladesh that have an amazing idea and they're just trying to get this creativity out there to share with the world.  You could have large corporations like a lot of the folks represented here. I mean you can have academics.  So open source is a lot of different things and there's really no one definition or moniker that fits that works with every community.  But if you're thinking about the types of open source that might make up a product or some kind of a cloud offering, you're going to have the high end of the spectrum or things like the kernel group, which is very mature and organized  and then you get you start moving down to like a patchy foundation, these other large, large communities and then you get down to something where it's a single person, a couple people that are just playing around.  And it's hard to put any kind of structure or process to all these different models because a lot of people that code for open source are doing it for free for no renumeration from large companies that make a lot of money off of it.  But they do it for free because they love kind of adding value and expressing their creativity.  And as Daniel mentioned, open source is very creative and some of the larger projects we do have ways to privately take in data.  So we take in a private bug report that's well established within the community for a long time.  And when you're looking at a less mature community or package or library, they might not have that capability.  And that's where larger kind of big brothers and big sisters like a Red Hat or a Susie are canonical kind of step in and try to help mentor these smaller projects to help them get these good practices set up so they can take in a private note.  Because sometimes a lot of my team we track between 3,000 and 5,000 vulnerabilities a year out of 450,000 packages.  That's a lot. Not all of them really kind of bubble up to the level of they need the attention of like a Spectre meltdown or a Heartbleed or a Blueborn, all these kind of the other big name nonsense things.  But they still need to get fixed and what open source is really good at is you identify a problem.  The team attacks it collaboratively. They develop a solution and release that update very quickly.  And they don't like to spend a lot of time making a big deal out of it because they've already moved on to the next feature, the next big thing.  But yeah, it can be challenging, but there are definitely ways to do it. There are methods to do it. There are groups that allow this.  I hear a lot of bleep into the background. What's going on, folks?  I'm wondering when you're going to change your hat again. Sorry. There we go.  Bring out Big Red.  All right, let me get this panel back under order.  When you're thinking about doing a CVD, what are some things you might want to prepare for?  What can you do to help make that coordination very successful?  And let me start off with, we'll start off with Lisa because she has a lot of ideas.  So from your perspective, what will make you successful when you're getting ready to do this multi-party coordination?  So I guess it depends on how familiar you are with it or not.  If I was a researcher and I wasn't too familiar, I would potentially utilize cert or something like that if I felt like it was going to cross over more than one company in the industry.  Just to have them help. Because I think it could be overbearing of figuring out how if one, you know, if industry, you know, company A wants this date to get it fixed and company B wants another date and company C wants another date.  So I think that's one way that people, a researcher could use. But I think the idea is to figure out when you start is what are your rules?  What's your embargo date? How many days do you actually want to, you know, give the company or vendor to fix the issue?  Basically, it's 90 days. I would prefer that as a company. I think it's respectful to do that, especially when things get more difficult.  So think about your rules and what you want to do and how you want to approach it. But I'll pause there because I know we're running out of time to get other people in.  Anders, what are your thoughts about how you could make CVD successful?  Three things. Use it. It's a tool not only for vendors but also for hackers.  Listen to what vendors are saying or what hackers are saying and try to make the best out of it and be aware that there's often a lot of complexities involved with it.  Be patient, right? Yeah.  Omar, what are your thoughts on how you can make CVD successful?  Yeah, I think that I'm going to capitalize on something that Daniel mentioned in a previous call.  And in some cases, even if you provide some data to even a vendor or whoever, right, to whenever you're coordinating, you have to make it first easy to understand. But at the same time, in some cases, getting the right people at the right place at the right time to make sure that you understand the technical implications of a given vulnerability  and sort of running around because even if you have the 90 days or 60 days or 100 days, getting that streamline and modernizing the way that we actually exchange the information among all the affected parties, that is number one.  And Art Manion, which is actually a good friend of mine, he leads the team.  We all love art.  He's the first one that will tell you we cannot scale. This is a big ecosystem.  Having a roll of decks of older vendors that actually use a component is foolish, right? We will never ever be able to actually have a complete thing.  But what we have to think about is what Lisa mentioned is that in some cases, we are working with competitors, right?  Even more than our companies. For example, you know, I work with Lisa a lot.  I work with Juniper even more in some cases more than I work with Cisco whenever it comes to fixing a vulnerability, let's say in BGP, OSPF or whatever the case might be.  So that type of thing of, oh, I'm not going to talk to these competitors. They're probably my everything else.  It's not, you know, that's 20 years ago, fallacy.  And then the last one is that at the end of the day, I have the reality and I tell my guys at Cisco is that whenever I push about the night I publish a CV out there.  The number ones that actually are reading that CV are the bad guys, right?  And probably at the same time that you're committing an open source code, you know, probably that's going to be the case too.  So what we have to do is think about how can we change information to the consumer to downstream and upstream providers in a more modern way.  And another sounds corny, but you know, can we actually make it more and we're trying to do that machine readable?  Right? So if in the case that we actually have that role of the X of people and everything else, you know, get some tool.  And that's what art and you know, the guys in insert that also do and you know, we're creating tools that allows us to do that.  And even if you have, I don't know, even two years ago that didn't assist and it wasn't even a thought.  So we have to move faster. That's the one thing that I want to say.  So as I close, I'll say art manion has the best facial hair in the industry.  I love art.  I want to thank our panel here. This was a great hour together. I really appreciate your time and expertise.  We're going to be hanging out for a little bit on this awesome discord channel that my kids made fun of me all week about because I've had it up.  Thank you everybody for coming and your attention here panelists.  And thank you to the audience for DEF CON for having us here at the IOT village.  Enjoy your day and enjoy the rest of the column.  Got some great stuff lined up the rest of the weekend.  CBD all the way. Oh yeah.  We're out.
DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_0_131.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_132_203.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_204_1235.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_1236_2207.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_2208_2363.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_2364_4619.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_4620_4667.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_4668_4715.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_4716_4835.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_4836_4883.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_4884_5855.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_5856_6443.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_6444_8759.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_8760_10031.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_10032_10079.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_10080_10259.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_10260_11735.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_11736_19019.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_19020_19379.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_19380_20063.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_20064_20135.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_20136_20363.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_20364_20411.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_20412_21335.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_21336_25547.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_25548_27539.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_27540_31115.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_31116_31283.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_31284_31619.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_31620_35759.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_35760_35831.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_35832_36071.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_36072_36119.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_36120_39083.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_39084_40751.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_40752_41135.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_41136_42623.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_42624_42731.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_42732_42911.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_42912_42971.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_42972_46019.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_46020_54155.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_54156_55967.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_55968_56159.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_56160_56207.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_56208_56507.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_56508_60551.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_60552_61607.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_61608_64187.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_64188_64235.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_64236_64811.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_64812_64943.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_64944_64991.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_64992_65051.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_65052_65111.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_65112_65171.png DEF_CON_Safe_Mode_IoT_Village_-Panel-TheJoyofCoordinatingVulnerabilityDisclosure_65172_65182.png
From your perspective, what does CVD mean to you? And kind of what are your motivations when you're working on an issue and you need to disclose something? So I think the main point of CVD is of course to protect users. And my main thought is a bit historical. So I've been around in the 90s where disclosure was you find a mailing list where other hackers hang out and you zero day everybody. And nobody really cares. And then became responsible disclosure, which is pretty sister for coordinated vulnerability disclosure. The problem with responsible disclosure is that it was really made to shame hackers that didn't follow the desired protocol of industry types. And the current iteration, the coordinated vulnerability disclosure is instead of trying to be coercive, trying to actually work with people that want to submit issues. And it is a much, much improved over that. And it does a much better job of making sure that everybody to a vulnerability disclosure, that means vendors, customers and hackers have their needs met. So that's what I think vulnerability disclosure, coordinated vulnerability disclosure is for me. Awesome. Daniel, another researching perspective. What do you think about coordinated vulnerability disclosure? So we are interested in understanding how things work and sometimes that involves understanding how things work that are not supposed to happen, which we then call vulnerability. And of course, then we want to figure out the truth behind it. And that involves talking to to the vendor. Very often they tell us that we got something slightly wrong. And that is an opportunity to correct these mistakes before we submit the paper or publish the paper. But also, from my perspective, it's the only ethical thing to do to protect the customers, to protect people who are using these products. And for me, I'm not in this game for finding vulnerabilities as a motivation by itself. My main motivation is finding the truth, finding better understanding of how things work. That's awesome. Katie, you've had a lot of experience, monkeying around with CBD. What does the CBD mean to you? Hold on a minute, Lenea. All right. So CBD means to me. So when I think about it, I think of it from a risk-based perspective. And so what we're probing is alluding to here is that my background is government in nature. So I spent about 15 years in the US government, 12 years of that in the US Air Force. And then several other years at Department of Homeland Security, where I ran the vulnerability disclosure profiles programs that most people are familiar with. So like the NIST, NDD program and the Carnegie Mellon CERT CC program and the MITRE CV program. So I was a sponsor for those programs. So in a single year, 2017, we coordinated and disclosed 14,800 vulnerabilities for public disclosure. So it's 14,000 IT vulnerabilities and about 800 ICS vulnerabilities. In the following two years, so 18 and 19, we coordinated and disclosed over 20,000 cybersecurity vulnerabilities. So I've kind of seen things from lots of different perspectives. And when I think about CBD, I think about balancing risk, right? So CBD is really a process. And every time you, when you're going through this process, understand like every organization is going to treat it differently. Because there isn't just standardized like one size fits all kind of here. There's differences that are going to happen across the coordination stack. So if you're looking at digital services, it's going to be different from software, which is going to be different from open source, which is going to be different from hardware or ICS, like all the differences are going to come into play there. But the overarching sort of thing, the big takeaway here is that CBD is balancing risk. It's all about making sure that there is an opportunity for the product vendor to fix the problem before an adversary has the opportunity to take advantage of that. So like it's all about protecting the end user. And I think everybody that I've ever met in the entire ecosystem is all focused on that. Like we may be speaking different languages, we may talk past each other, but I think that everyone is trying to protect the end user. So to me, that's what CBD is about. That's awesome. And remind me to circle back to you later in our talk. Talk a little about some of the psychology behind some of this. Oh yeah, absolutely. I would love to hear my dearest friend Lisa's thoughts on kind of coordinating vulnerability disclosure and kind of what some of your motivations that you've seen or kind of what's behind your motorcycle. Yeah, so I think where I've seen a sort of an evolution of what we do now. I think Heartbleed was sort of the big start of where we paid attention, realized we all needed to come together a little bit, and then Spectre Meltdown sort of brought us even closer. So my thought is that not only do we work with the researchers, but we work across the industry to make sure that we're all doing our best to protect our customers. So like Katie was talking about, it's really our end users that are our focus and making sure that we have the right people involved to be able to solve the issue and then provide a security update so that our customers could get it and be best protected. That I didn't say CBD. Oh, you just did that. I know. Yeah, yeah, yeah. So Omar, kind of you've been doing this for a while and you work for a very large company. So I'm sure you get the opportunity to see a lot of different vulnerabilities. So what what does the whole process meaning? What are some of the motivations you've seen? Yeah, I think that pretty much everybody has summarized very well is protecting the end consumer at the end of the day. But I see it as a very, very complex ecosystem, right that we're trying to actually solve. And Katie, you know, mention in one side hardware, another side software, then if you decompose that then you have open source, which we're going to probably talk about a lot in here. And you have things that, you know, perhaps you cannot even control. I have had a, I guess, a pleasure or not a pleasure in some cases where we're looking at vulnerabilities and even in some cases whenever we try to actually solve the issues, the companies don't even exist anymore. That's another predicament, you know, that in some cases we actually have to take into consideration and it's a fairly complex ecosystem. And at the end of the day, what we have to actually put our heads together is how can we modernize our practice into the way that not only yes, we deal with a single vulnerability, we reproduce it, we actually fix it, we find the patch, but how can we accelerate that process that one everybody's talking that somewhat the correct vocabulary, or at least they can understand each other. And then we also understand the overall risk, right, and how to prioritize things and so on. So it's a fairly complex, we can actually talk about it for quite some time and that's what we're here for. But that's what some of the initial perspective that I want to share. Thank you, I appreciate that. I want to talk about something we touched on in one of our prep calls. We have Anders and Daniel that come at this research angle from slightly different angles. So maybe, and you too, Jelon can figure out who wants to start first, but thinking about the perspective of academia versus a professional bug hunter security researcher, and maybe you can maybe describe some of the particulars in either area, and maybe what are some of the differences you might see between those two different types of research. So from the industry, or rather the bug hunter, the professional or not, there is very often an element of good old fashioned fun. I started out hacking things, not because I wanted to achieve anything with it, just because it was great fun. When you become pro with it, there is different motivations. So if you work for a company and you work on their products, obviously your motivation is to make those products secure. And sometimes my past job, I did some research and that was sponsored by that company, but essentially me doing what I like to do. And their end of it was attention to that company's competences. And my end, of course, was having a bit of fun. So that is probably pretty typical of what hackers get out of hacking. Yeah, from the more academic side, it's more of advancing the field, the knowledge in the field, and there you try to figure out how things work and what the implications there are, what the security implications of certain understandings are. For instance, understanding when you can execute a certain piece of code and that does something that is not intended. What are the implications? And this is interesting from a security perspective than if it enables someone to do something that they shouldn't be allowed to do. Yes, often this involves disclosing a vulnerability to a vendor. But I would say that, for instance, bug bounties that might be very relevant for people from the industry. I think this plays a smaller role in academia because you have to participate in CVD anyway because if you wouldn't, then you would get a lot of problems in the academic community. There's a lot of peer pressure that you participate in this because that's the only ethical approach to handling vulnerabilities. At the same time, if you keep them secret for too long, then there's also the question on why did you keep them secret for so long. Maybe also some perspective that I can share in academia, you see more and more often the fact that academic publications are easier to publish if you have a CVE. I don't think that's a good thing because I think that a CVE does not necessarily describe that a research result really brings you new insights. A CVE is an identifier for a vulnerability, not an identifier that says this is something with a new insight. Also, if you participated in CVD and you mentioned this in the paper, this also gives you bonus points. At least that's my impression to get the paper published. Of course, it's good if you participate in CVD, but my feeling is also that this is going a bit too strong and it's now overemphasized in our community. What about having an icon or a fun name? Of course, I'm really a big fan of logos and names and anything that is fun. But of course, there are multiple layers here. For instance, we had this paper, Hello from the Other Side, which had this name not just by coincidence. It was about a covert channel in the cloud where we send data through the cache from one virtual machine to the other. We really went crazy on that. We built an entire TCP stack on top of that and tunneled an SSH session through the cache cover channel for whatever reason. We then sent a music video through this cache cover channel. Of course, we couldn't just take any music video, so we had to make our own parody of Hello from the Other Side. This is just fun. We are just a bunch of people and we like to have fun. Basically, once you've finished your project, it's really nice to close this up with something nice and funny. Logos on the Other Side also help communicate about the issue. I realized that every time I create slides, my favorite slides are the slides that don't have any words on them, just pictures, logos and icons. Then I can follow the speaker. I don't have to read because I'm very bad at reading. I can't read and listen at the same time. I bet many people can't do that. If there are only icons, symbols and logos, I can follow what the person is saying and look at the images at the same time. I'm always annoyed if I have to speak about the vulnerability and I don't have a logo for it because then I have to put text on a slide. Since Daniel brought it up, let me ask the other panelists, how do you all feel about branded vulnerabilities? Do you like logos? Do you find that helpful to you and your constituents? I'm sorry, Daniel, no. When there's a logo or a PR or everything, it grabs the media's attention so quickly. Sometimes the issue is not that severe or very hard to be able to breach. It gives a little bit of extra fear to our customers. It also encourages us to maybe reprioritize other issues first, which are more severe to our customers. Although I get the point, it's easy to talk about it. They grab the media's attention really quickly. If you put a caveat about the real severity of the issue in there, maybe that would help a little bit. We often have in our team this discussion, should we have a logo, should we have a website or shouldn't we? For most of our papers and most of the vulnerabilities that we discovered, we don't. I think that's also the responsibility of the researcher to assess, is this something that significant that I need this additional PR or is it not? Right. Well, then you're doing it the right way. Maybe, but it's also, I mean, I will have a different view on what is really important than you have. Everyone will have a different view on that. So something that I think is really, really important, you might say, well, it's not really exploitable in our use case. Right. We don't ever want to downplay a researcher's work. I mean, it's important, whatever they do, and we appreciate especially doing CVD with them and not being zero-date. But yeah, it's a more matter of, especially in a bigger industry when you're trying to protect your customers best, how do you approach it? There is a fight for customer attention. And for customers to make the right security decisions, they need to be attentive about the right things. And logos and hype in the press warps that and we have bad security outcomes for our customers due to that. Then of course, there's also the thing that even fixed vulnerabilities sometimes costs a cost of suffering for system administrators running around in basement and patching systems and all the like. And to some extent, I sometimes find over high bucks disrespectful to those people. Yeah, so in my case, I don't have an issue with with logos or anything, even though in a previous call and a couple of exchange Daniel, I alluded into that. Whether it's a logo, as a matter of fact, even for Cisco, we had the first vulnerability with emojis. So we have emojis, logos and everything else. So at the end of the day, if it helps to do an alias for a vulnerability and bring some awareness, I'm perfectly okay with that. The only challenge that I'm seeing or opportunities right is that in some cases whenever we write information and even vendors whenever we, we know we have in my case, we have a research institution at Cisco called Talos and we find vulnerabilities in other people's product right. And yes, yes, yes. So in that case, yes, we also have logos and we put them out there and we put the blog post and so on right. But what we need to do collectively is to make sure that the media is not either downplaying it or over proportion. So we have to have a balance right and we all, you know, say the media, the media, the media, right. So what is our responsibility right there on how we create our security advisories. And it's both ways is, you know, even though we're vendors here and everything, you know, but we have to point fingers to us, you know, how clear is our advisories that we have some collateral that the end consumer actually knows about, you know, what are the implications are and we work with a researcher to make sure that we all understand, you know, what the problem is. And in the previous conversation, we also I also mentioned that the biggest nerd fights in history. It's whenever it comes to CVSS scoring, right. And, you know, whenever it comes to risk and at the end of the day, you know, whether it's CVSS whenever we come up with some new ways and everything we have to have some type of way of saying, yeah, you have to jump right now and fix this vulnerability versus the other 100 criticals that we were also, you know, dependent on and especially with nowadays since it's not so much of a vulnerability coming from like a Cisco or an Intel or Red Hat or anything else is open source right by the time that actually I'm boring you to death in here, probably three more CBEs that are super important has been disclosed. And we don't know about right so so that's the type of balance on the type of, I guess, for a corny way of saying orchestration that needs to take place right in this CBE. I want to add another point here regarding the overhyping and having a logo having a website for something for a vulnerability for research result. It does not necessarily mean that you overhyped it we had experiences where we just put papers on archive without any website without any logo and anything and media picked it up and media reported about it and not necessarily correct in all cases and we have this, for instance, for the, the takeaway paper where we analyzed some site channels on and DCP use and their media picked it up we didn't we intentionally didn't make any website or logo but media picked it up and the reporting. It was not, it was not entirely wrong but it was definitely I would say definitely overhyped. We discussed this also in our team and the conclusion that we had was that in some cases we should have a website, even if it's not a vulnerability that we want to hide because it's not that significant, but just to have a very clear message, which says how relevant is it. I think that's the important part that we also always had on our websites, a short three sentence summary like what is it who has to care about this and what can an attacker do. Yeah, I think that's great. That's great you do that and it's good advice. I think, you know, that I would love to see it be more adopted and in the researcher world. For sure. I think it would help out. Before I move on to my next question, I'm just going to say, when I retire, all of you in vendor world are doomed because I'm going to offer my services to L reg for free writing up crazy headlines. But the touch on a point that Daniel made he invoked the Katie Miseris clause. Let's talk about bug bounties and CVD. Oh, Katie maybe you should take that one. Because our friend Katie and a little trimble here actually has a lot to do with that in her organization. Could you maybe talk about how coordinated disclosure interacts both good and bad with bug bounty. Yeah, so I love Katie Moe. I think that first off let me just say that it is an honor to be confused with Katie Moe. My hair color is like, you know, getting close to hers too now. I don't know. I think I have been accepted to conferences because they thought that I was she. When I showed up, I don't think they were nearly as excited to see me instead of her so but so yeah bug bounties. So bug, tin foil hat really. So I'm trying to be serious here about bug bounties. So what's that is a tool right so they're a tool in a toolkit and they are there to incentivize, but they're a part of a well structured product security portfolio. They're not the whole thing, and there are different motivations that will help people from different perspectives. So in the academic world, for instance, but that is may not weigh as much because a lot of academic institutions don't allow an individual to accept that the heart coal hard cash right. If you are a professional bug hunter, you know that might be how you pay your mortgage and so there's a lot more tied to that. But the problem becomes that like, bug bounties are a wonderful tool and they're, they're great but you have to have a good program in place already to accept that that information to be able to execute on that information and figure out like how do you, how do you actually how there are so many questions about it how do you award how do you manage it like, are we going to tie the payments to see the SS scores are we going to tie the payments to a well thought out proof of concept there's so many pieces that go into it and so I'd say like bug bounty is not a one size it's all it's going to vary from organization to organization and some organizations are going to have different timelines different pieces. It's, yeah it's complex. So, bug bounty is not the end of the world bug bounty is a tool. It's a tool in your toolkit. So it's a great tool. I love the tool because I'm the director of it and tell. But it's not, it's not everything. So vulnerability disclosure is more than bug bounty alone. Yeah, I think when you think about it to like there's a lot of smaller companies that probably, you know, don't have that or don't have the funding, funding for it. But that that doesn't mean that they're not wanting to work with researchers they just can't get the budget to do it. You know, so, so, you know, do your best to try to figure out how to reach out to those companies. Hopefully they have a web page or email address, you know, secure at PISER that security at. I know we struggle a little bit but, you know, there's plenty of us around that could help find the right contacts because we're you sort of come together from all over the place and we're sort of expanding group here I like it. But boundaries are one of the wonderful signs of how the industry has changed back in the 90s. Researchers was often rewarded with that disclosure was rewarded with lawsuits. And now people are the industry are working with the people and have started actually paying the researchers for their efforts. So I very much thumbs up on what boundaries, especially because it shows how the industry has changed on all of you hackers hackers on our days helpers and not the enemy. Yeah, I thought you were going to say t shirts for a minute there in the early days. I like to see it just one to see shirt. Oh, early on the night it was lawsuits, then it became t shirts and then maybe stickers and then. Yeah, and now, and now financial. Awesome stickers. So let's we've touched on it a little bit let's see if we spend some time now we're past the top of the hour here. Let's talk about coordinated vulnerability disclosure inside an open source context. So, I'll go last, but let our esteemed panel here maybe whether it's our researchers are vendor friends kind of describe what your, what works out really well and what was some challenges for you within the open source world, which makes up about 90% of all software now. Jerry open source one. I guess I start. So, so basically, in that stop of mine for me, and it's, you know, it sounds corny whenever people say hey what keeps you up at night and everything else is actually open source right now. It's not so much of the predicament of using open source, we have to use and embrace and contribute to open source and I'm super big fan of that. The challenge when it comes to open source is that it can be anything right like IOT can be anything right so, and it's critical infrastructure for a lot of things. So, to give you some somewhat of a real life you know I guess a realization that we had a while back probably about four or five years ago whenever hardly came, we were looking in the industry right now not only Cisco but a whole bunch of other companies right so what to what to prioritize as far as actually giving funding, probably doing research, and so on so we said, okay so open open SSL and things like that which are actually super important for us. There's a lot of people looking at it right now. So can we look at things that perhaps are actually critical infrastructure that nobody actually has probably taken a look at it right. We're going down the list. And then, and as a matter of fact that number two is perfectly fine because there are two guys that work on open SSL to get paid to do it. Yes, indeed, indeed. And the example that I was going to go is NTP, the network time protocol so NTP is specifically. And there's also two guys that don't get paid, which is bad right is there, not even the full time job right. Well, I guess it's now your fast forward this year is a little bit better now right so it's a lot better. And it's not that the problem is that poor guys that actually are contributing to the code is a matter of scalability is a matter of actually even, you know, running static analysis into the thing didn't even assist, you know, five years ago for these components. Now of course we're modernizing our ways you know even if you submit things of GitHub a lot of things actually happening and we're we're getting we're getting better for sure. It's just that it's getting way more complicated right and more people actually are in not only of course contributed but using it more than contributed, which is the other predicament. And in some cases, we were talking about bug bounties. In some cases actually amazing. I'm a big fan of bug bounties right the challenge is that in some cases, we also don't think about does this affect other vendors, and my actually finding some type of vulnerabilities that you know perhaps yes it's a single injection cross ice scripting which is actually pretty pretty common but I'm doing some fuzzing and I, you know, crash an application. What is the underlying issue right and in some cases actually kind of a commodity you know goes back and forth, and it hasn't been shared. And then two or three months down the road you say oh, but this CBE was not shared with other these vendors that they're also affected. So we go back and, and in some cases you actually see people trying to find the same or found the same CVs and they reported a different way but you know, there were no coordination so those are the things that are that are crucial right for us to be successful and not only among vendors but actually downstream and upstream as well right so especially when it comes to IOT is like number one. Yeah so Omar you brought up a few points you know one is who do you bring in right ahead of time and I think that we're seeing that even if you're the competition we still want to work together in the background here. You know, we, our whole goal is to protect our customers, and we often have the same customers even if we're competitors. So we like to, you know, make sure that we're all working together and ideas is when do you bring someone in and it's not only the researchers. Okay, CROPE, where you got your hand raised go ahead. Can I tell you it's a secret. Yes. Open source has worked with our competitors for 25 plus years. Yeah, yeah, well, I would have to say, we, you know, with with some of those other vendors in the industry, it wasn't always so nice. They were playing with the close source. We're getting better. But I think what you know what would not only the researchers should be thinking who else is affected but when we are the receiving and we should be thinking who else is affected and bring them in. I recently worked on an issue of a very recent one where the we are in Key Base and the researchers were in right with the industry people who were fixing the issue. And there was great, you know, coordination between the pink, you know, amongst the group there. It was pretty awesome to see of how, how much we've evolved. How do you do that though with like open source though when everything's shown. It's super easy. Well, and I'll, I guess I'll chime in before we let other folks. I found it, I found it very interesting how some vulnerabilities have been patched wide in the open under the cover of some other patch. I don't know what yeah, I don't know what you're talking about at all. So, in general, open source, there is no single definition of what open source is you could have two young ladies in Bangladesh that have an amazing idea and they're just trying to get this creativity out there to share with the world. You could have large corporations like a lot of the folks represented here. I mean you can have academics. So open source is a lot of different things and there's really no one definition or moniker that fits that works with every community. But if you're thinking about the types of open source that might make up a product or some kind of a cloud offering, you're going to have the high end of the spectrum or things like the kernel group, which is very mature and organized and then you get you start moving down to like a patchy foundation, these other large, large communities and then you get down to something where it's a single person, a couple people that are just playing around. And it's hard to put any kind of structure or process to all these different models because a lot of people that code for open source are doing it for free for no renumeration from large companies that make a lot of money off of it. But they do it for free because they love kind of adding value and expressing their creativity. And as Daniel mentioned, open source is very creative and some of the larger projects we do have ways to privately take in data. So we take in a private bug report that's well established within the community for a long time. And when you're looking at a less mature community or package or library, they might not have that capability. And that's where larger kind of big brothers and big sisters like a Red Hat or a Susie are canonical kind of step in and try to help mentor these smaller projects to help them get these good practices set up so they can take in a private note. Because sometimes a lot of my team we track between 3,000 and 5,000 vulnerabilities a year out of 450,000 packages. That's a lot. Not all of them really kind of bubble up to the level of they need the attention of like a Spectre meltdown or a Heartbleed or a Blueborn, all these kind of the other big name nonsense things. But they still need to get fixed and what open source is really good at is you identify a problem. The team attacks it collaboratively. They develop a solution and release that update very quickly. And they don't like to spend a lot of time making a big deal out of it because they've already moved on to the next feature, the next big thing. But yeah, it can be challenging, but there are definitely ways to do it. There are methods to do it. There are groups that allow this. I hear a lot of bleep into the background. What's going on, folks? I'm wondering when you're going to change your hat again. Sorry. There we go. Bring out Big Red. All right, let me get this panel back under order. When you're thinking about doing a CVD, what are some things you might want to prepare for? What can you do to help make that coordination very successful? And let me start off with, we'll start off with Lisa because she has a lot of ideas. So from your perspective, what will make you successful when you're getting ready to do this multi-party coordination? So I guess it depends on how familiar you are with it or not. If I was a researcher and I wasn't too familiar, I would potentially utilize cert or something like that if I felt like it was going to cross over more than one company in the industry. Just to have them help. Because I think it could be overbearing of figuring out how if one, you know, if industry, you know, company A wants this date to get it fixed and company B wants another date and company C wants another date. So I think that's one way that people, a researcher could use. But I think the idea is to figure out when you start is what are your rules? What's your embargo date? How many days do you actually want to, you know, give the company or vendor to fix the issue? Basically, it's 90 days. I would prefer that as a company. I think it's respectful to do that, especially when things get more difficult. So think about your rules and what you want to do and how you want to approach it. But I'll pause there because I know we're running out of time to get other people in. Anders, what are your thoughts about how you could make CVD successful? Three things. Use it. It's a tool not only for vendors but also for hackers. Listen to what vendors are saying or what hackers are saying and try to make the best out of it and be aware that there's often a lot of complexities involved with it. Be patient, right? Yeah. Omar, what are your thoughts on how you can make CVD successful? Yeah, I think that I'm going to capitalize on something that Daniel mentioned in a previous call. And in some cases, even if you provide some data to even a vendor or whoever, right, to whenever you're coordinating, you have to make it first easy to understand. But at the same time, in some cases, getting the right people at the right place at the right time to make sure that you understand the technical implications of a given vulnerability and sort of running around because even if you have the 90 days or 60 days or 100 days, getting that streamline and modernizing the way that we actually exchange the information among all the affected parties, that is number one. And Art Manion, which is actually a good friend of mine, he leads the team. We all love art. He's the first one that will tell you we cannot scale. This is a big ecosystem. Having a roll of decks of older vendors that actually use a component is foolish, right? We will never ever be able to actually have a complete thing. But what we have to think about is what Lisa mentioned is that in some cases, we are working with competitors, right? Even more than our companies. For example, you know, I work with Lisa a lot. I work with Juniper even more in some cases more than I work with Cisco whenever it comes to fixing a vulnerability, let's say in BGP, OSPF or whatever the case might be. So that type of thing of, oh, I'm not going to talk to these competitors. They're probably my everything else. It's not, you know, that's 20 years ago, fallacy. And then the last one is that at the end of the day, I have the reality and I tell my guys at Cisco is that whenever I push about the night I publish a CV out there. The number ones that actually are reading that CV are the bad guys, right? And probably at the same time that you're committing an open source code, you know, probably that's going to be the case too. So what we have to do is think about how can we change information to the consumer to downstream and upstream providers in a more modern way. And another sounds corny, but you know, can we actually make it more and we're trying to do that machine readable? Right? So if in the case that we actually have that role of the X of people and everything else, you know, get some tool. And that's what art and you know, the guys in insert that also do and you know, we're creating tools that allows us to do that. And even if you have, I don't know, even two years ago that didn't assist and it wasn't even a thought. So we have to move faster. That's the one thing that I want to say. So as I close, I'll say art manion has the best facial hair in the industry. I love art. I want to thank our panel here. This was a great hour together. I really appreciate your time and expertise. We're going to be hanging out for a little bit on this awesome discord channel that my kids made fun of me all week about because I've had it up. Thank you everybody for coming and your attention here panelists. And thank you to the audience for DEF CON for having us here at the IOT village. Enjoy your day and enjoy the rest of the column. Got some great stuff lined up the rest of the weekend. CBD all the way. Oh yeah. We're out.
Awesome. Daniel, another researching perspective. thing to do to protect the customers, to protect finding the truth, finding better understanding friend Lisa's thoughts on kind of coordinating heads together is how can we modernize our motivations. So if you work for a company and you Also, if you participated in CVD and you customers to make the right security decisions, perfectly okay with that. The only challenge that industry has changed on all of you hackers vulnerability disclosure inside an open source panel here maybe whether it's our researchers are it comes to open source is that it can be finding some type of vulnerabilities that you sisters like a Red Hat or a Susie are canonical what hackers are saying and try to make the best