Security Retrospective of the last year
Meissner, Marcus
Not just Meltdown and Spectre Last year was a quite busy year on the security front, various big issues happened, so its good to tell what security has done there and is doing for openSUSE and SUSE in general here. The talk will give a brief overview of how the SUSE Security Team works and operates. We will look at the reactive work including statistics, and also look at proactive secure development lifecycle activities. I will also highlight some of the big security issues we faced over the last year. - Stack Clash from mid of 2017. Overview of the problem, what we do for mitigations, and our long way for compiler mitigations. - Meltdown and Spectre As we hoped never to have a StackClash like issue again, CPU sidechannel issues surfaced which needed kernel mitigations begin of January. I will give an overview over what these issues are, and how we mitigated them or are still mitigating them. I will also talk about one of my projects done in the last year: - Full PIE enablement for the distribution and its long road to Factory.
 Hi, good morning. My name is Marcus. I work with SUSE and today I would like to introduce you to the work of our security team, our security work, what challenges we face and what interesting things that's happened or have happened in the last years. If you are open SUSE community member or SUSE employee, let me briefly introduce the team that I work in. These are all SUSE employees. Our team lead currently is Andreas Stieger. He has been a project manager before, but as our previous team lead left the company, he's now taken on this role. There's one project manager that's me currently responsible for the reactive work. I'm writing the plural here because we will also likely hire a new project manager in the near future for proactive work again. There are some security engineers. You will be reading about them in our backzilla and by email it's like Alexander Bergmann, Johannes Seegitz, Carol Barbioch, Matthias Gersner and Victor Pereira. We also currently have one working student working for us on, working on a thesis on security and software defined networks. His name is Alexander and he's somewhere around the Greek mafia that hangs around the conference. There he is. We have some open positions to fill because SUSE is growing, so we are looking for security engineers and also soon for a new project manager. What do we do? We do proactive work. Proactive work means making a distribution, a product secure before shipping and we do reactive work, meaning reacting on security issues, responding to incidents, resolving them, releasing updates and so on. At some point in time, one and a half, three years ago, we decided to split those responsibilities because our team has grown and the number of issues has grown. So, Alex, Johannes, Victor, Carol and myself work in the reactive side of things. We still work also on product security, but mostly we are responsible for the response and on the proactive side where we do auditing, releases, approval. Currently we have Andreas and Matthias, but we are still looking for people in that area to build up. So, what do we do? What is reactive security? What do we do for reactive security? So basically, we need to track and react on all security incidents issues reported to us that exist in our products. I will go on the numbers a bit later on. For that, in our sub team, we have so-called incident update managers so that everyone is aware of what he has to do. Basically something really like in the regular ticket handling, we mirror that also for the security incidents. So we have a so-called incident and update managers in a rotating role so everyone knows when to take care of it or who takes care of what. What we do as a security team, we coordinate. We don't fix it ourselves. So coordination means we put the incidents into our tracking systems, then hand over the work of fixing to our packages. They can of course come back to us, ask questions, get help, or we help them with patches and so on. But basically, we rely on our package maintainers or the COS SUSE community to fix the bugs, submit them back to us. The source review is done by the OBS review team externally and by the internal build service reviewers internally. And for the testing itself, we give back our updates to the maintenance, so-called maintenance QR that tests these updates. Internally, we have currently three teams testing. Externally we have mostly rely on open QA during the testing. Internally we also rely on the open QA automated testing framework. But externally we largely rely on the open QA framework and some community members that test the current updates. We take care of the release of the updates, of course, and of documentation. Most of the documentation is automated, but occasionally we need some additional manual documentation with more information, more technical hints, and workarounds to handle. So for incidents that come in, every incident that affects our software gets a bug in our bugzilla, you can look at it. All of those get our public or will get public after the embargo ends. You will notice them if you look at the summary line for the while-prefix and the CENUM and the package name. It's in the summary. References, descriptions, links are all in the regular bugzilla comments. And those bugs, as said, be assigned to the package or bug owner. So it's kind of important if you are a bug owner of a package, you will likely get assigned these bugs. So we rely on you on reacting on that. Internally, we use a ticket system that manages the real flow of the incidents. We call it SMASH. I don't even remember any more what the abbreviation stands for. But it does additional things that bugzilla does not provide. So we rate the incidents. So basically we have simplified rating of low, moderate, critical, important. And we have a CVSS version 3 scoring, common vulnerability scoring system that is used by us and by our customers and users to evaluate how much, how large the impact is of a security issue. So some standards say install all security fixes that has a CVSS rating larger than four or larger than six or something like that. So it's a service we deliver also for our customers that we score every issue with this kind of rating. Internally, as we have a lot of different code streams internally, we track the affected nest. So if something is affected, something is not affected. It's not really that necessary for open zoos or Leap and tumbleweed because these are quite fresh. And for Leap we have currently two code streams, but usually one code stream. So usually the software is affected in Leap or tumbleweed and will get fixes. Internally where we have the software that is 10 years old, it's not always clear if it's affected or not and to keep track of that we use that tool. It's also used for planning and starting updates. So mostly to not lose anything, to not lose sight of the number of bugs that are inside. So how has the number of bugs developed over the last years? So I started a, I have a bugsy query that checks every week on how many bugs were opened in that week and it's running since 2006. And you see that up to the middle, which is around 2013, there were around, it's probably not really about around 10 to 12 bugs open per week and it's accepting some noise and stayed at that rate. And starting with 2013, the rate is growing. And now we are currently with a lot of noise inside around the rate of 30 bugs per week. Why this change? The changes that Linux got more attention on from security researchers, a lot of more people look at it and see fine bugs inside. But mostly this is also coming from automated bug finding. The automated bug finding is usually done as either static code checkers or with fuzzing and a lot of those come out of the fuzzing tools. And notably there's a tool called American Fuzzy Lop that really is easy to use, easy to plug in into your program. And if a code base has not seen any kind of fuzzing and isn't written in a safe language, which unfortunately most of them are, it will find quite a number of bugs very easily. So I will go on some details later on, but ImageMagic is our all-time favorite in that. So to handle that, before we didn't really have that much of a ticket system, we had some tools that helped us, but with a number of occasionally 100 bugs per week, it's very hard to lose, not to lose track of things. So we have written this internal ticketing system called Smash. And basically it gives us all the new issues that we don't know about and we go over them, we analyze all issues, open bug reports, track the affected software and the ratings. And then defer this delegated work of fixing it to the packages. So how does the update submissions work? The package submits the update and the security team puts some metadata on it, additionally to it the summary, the description and the issues in the machine readable format. So basically keeping the same style over our documentation that is then generated because out of that the automated emails are generated and update notices and so on are generated. We check if it's good, if it's building, if some standard checks run on it and then forward it to our QA. QA is both manual and automated, so we have quite a lot of checkers these days. We have open QA and our QA itself tests still the, they test bug fix certification and if not yet automated regression fix, regression testing. The end we release the update and do potentially additional documentation work if not done automatically. So also for the update submissions we have a tool that makes us not lose track. So basically listing what is submitted, what is currently prepared, what is in testing. So here I have the tab open that is used by our QA department that looks at what incidents are the next for testing, they are prioritized to classify it and so on. So without this kind of tooling it would not work anymore. What do we deliver? We deliver the updates of course. And a lot of compliance checking users, they get emails, email notifications, we present them either as email archive update, email archive or we have a web page where you can even search or click on them under this URL. We generate machine readable advisory data that's called CVRF, common vulnerability reporting format. It doesn't have so many external tools yet there is a Python module that uses it that you can hook into your applications. So it's only taking off a bit right now. And we also auto-gain information based on a per CVE ID. So you can look at pages for every CVE ID that we fixed and see what is the rating, what is affected, what is not affected. So basically everything that we worked inside is exposed to the outside in a human readable format but also in a machine readable format. The machine readable format is in so-called OVIM, open vulnerability assessment language format. This is part of the so-called SCAP, security content automation protocol format suite and can be feed into the open SCAP utility or other processing utilities to find automatically if your system is still affected by a CVE or not. That's very helpful if you manage hundreds or thousands of machines for instance. So some of the statistics, I took the full year 2017 so I'm not looking at the last six months but the full year 2017. Overall we processed over 3,000 CVEs declaring them as affected or not affected. We opened 2,500 bucks which the number really surprised me already. I would have thought like a thousand or so but it really was 2,500. Out of that I took the highlights. The highlights were we have 266 kernel issues. The kernel itself also gets fuzzing so more things are also found in the kernel. As I mentioned image matching and graphics magic are really outliers on the number of issues found mostly because they implement a lot of codecs. It's quite old software where people programmed without security in mind at that time. The good thing is both are improving, graphics magic is already based on a better consistency model than image magic and both are continuously improving so less and less bucks are found anymore. Mozilla Firefox here accounted the total number of CVEs. It's a high attention code base so a lot of issues are still found in it. TIFF and Wireshark also got a lot of reports from fuzzing. And there's a long list that has less bucks opened. We released a thousand security updates for the enterprise product code lines and we released 626 for open Zuzo in 2017. And this open Zuzo Leap is inheriting a lot of packages from SLEE. I also looked at the number what of those were SLEE imports. There were 250 updates directly imported from SLEE. So one third of the security fixes for Leap automatically came from our Zuzo Enterprise code base and did not need any open Zuzo community attention. What is not counted here is that also the kernel, the same Git tree is used also for Leap 4203 as for SLEE 12, SBE 3. So the kernel itself is also coming from SLEE but not listed separately here. And in the end we fixed all together 2,300 CVEs in 2017. So without any tooling this is quite impossible and it's really getting really a pipelining feel, so a factory pipeline feel like a car pipeline or something like that. So I don't really know anymore what bugs I opened last week. So was there an image magic bug I don't know. But some bugs shine out of it. Some bugs get more attention than others and we classified those as high impact vulnerabilities. So what are? So this is a name that we invented. Why are these high impact vulnerabilities? Not just because they get a lot of attention but they are high impact. So like worms could exploit them. It's trivial, exploitable, with a root exploit. What I really, they really cause a lot of customer queries because they are difficult to fix. They need some work arounds or something like that. So everything that stands out, that needs special attention, that needs also special reaction times, we classify as so-called high impact vulnerabilities and handle them with more attention, with more manual work, with more manual documentation on top. The code names are probably familiar to you, and Shellshock started that trend with code name vulnerabilities and coming up to Meltdown and Spectra, beginning of this year. So let me tell you about some of those. So the Stacklash, you probably forgot about it already, that was one year ago. The issue was not that something was buggy, so the programs were okay, they were working as intended, the code was okay, the kernel was okay, but there was a flaw in the Linux memory management model. So basically everything exists in the same address space, and most of the data that is allocated grows upwards, except the stack, the stack grows downwards. So the researchers found here, what if those happen to collide? So this is something that was found even earlier, last time was in 2010, that people could explore it in 2010, they thought, yeah, we do one page, we do a little bit of buffer between that one page, so no one will cross that buffer end that protects both address spaces collide. And last year, a security research company, Qualis, found that there are places in common code where more than this one page, this four kilobyte, can be jumped over, so that really you could crash stuff together again. And sadly, one of those places was in G-LipC. In G-LipC, even before the program starts, G-LipC does some of those allocations, those on-stack allocations for the audit, at the library path, and similar things. So it was even exploitable without actually starting the program. So you just took a security program, and during starting, G-LipC was in the start routines already causing an exploitable problem. Yeah, that was kind of hard. It's definitely something that is called a high-impact vulnerability, because it was quite easy to exploit. But it was kind of hard to fix. So foremost, the kernel received a bigger band-aid gap. So currently up to now it was four kilobyte, and now it's one megabyte or two megabytes. It can also be configured now. So if you want to save address space, then you can have a smaller gap if you trust the code. So that killed most of the attacks, but it did not remove the underlying program, actually. So what was additionally added is that GCC now gets code where on such stack allocations that are done by the compiler, it touches every page during this allocation, and so causes a fault if someone jumps over more than four kilobyte or more than eight kilobyte pages. And as we changed this in GCC, we had to actually rebuild all the packages. It's a bit tricky. We don't really want to do that for released code streams, but what we did is that at least for C12 and OpenSuselip 4203, we enabled that also for the updates, and all updates that we have released since then have this feature enabled and are built with this compiler feature. And interesting enough, after fixing the kernel, most of the security researcher community lost interest in that topic. It's just that to avoid finding it again in a year or two, we now switched our distribution to use this compiler option and to avoid this problem altogether. Yeah, and Spectre and Meltdown started, was released begin of January this year. So side channel attacks are not new. I learned about this in university in the 90s. Side channel attacks using speculative executions is, however, quite a new concept that was really brought up in the last year. And researchers are now looking deeply into that topic. So we will, while we have seen some of the issues in begin of January and one other issue just last week, there will likely be more issues coming up because researchers are now looking at it. And with kind of a run through of like half a year, we will like see every half year or also we will see probably new issues. The concept is what a basic problem is that the CPUs speculative execute following instructions to make more out of parallelism or make it more parallel to be faster and quicker. And they take shortcuts during that. So because every correct execution would cost a lot of transistors, die space, and so they take shortcuts and almost all of these shortcuts can cause these side channel attacks. So for instance, if you look at Meltdown, Meltdown, the main problem is the lack of access control on a supervisor bit of the page table entries. So the speculative engine did not look at the supervisor bit. So it accessed page tables, page entries, or memory, even though it was not able or should not have been able to access it. It was a shortcut taken to make it faster to have less transistors, but it caused this kind of security issue. And the general part is that the speculative execution fetches stuff into caches is probably kind of unavoidable to just gain the speed or keep the speed. So we will probably always see some kind of cache-based side channel attacks in this area. What are our problems here? There have been strict embargoes on it for several months. Into itself took like half a year. We were involved since November, around the end of November on that. And our engineers were involved around October. So they were working on it since around October until January. It's not a software bug. The software is fine. But we need software workarounds because we cannot replace all processors in the field. And some of those need CPU microcode changes, especially this speculative bound jump prediction violation that variant 2 was needed some microcode changes to flush the prediction buffers. As these need software workarounds, the patches are not straightforward. So usually a security fix in software is called by, like Oliver said this morning, I just need to add an if somewhere. So usually the security fixes are two liners, four liners or something like that. But this really needed a massive afford, hundreds of lines of code in tricky code paths like the system call, entry path, assembly with also high risk of causing the regressions. And the one thing is that the fixes lose performance. And customers, well, customers, you want security, they also want performance. And the priorities for my team is, of course, we want security. But the customers, they are kind of pissed if their computer takes half an hour longer or performance drops by 20%. So that caused quite some feedback rounds. And after it got public, we fortunately people found for some of the problems faster equivalent of fixing it. So the variant one, at the beginning, they were fencing that stopped all the speculative execution. Now they use a different code, not an if construct, but more a masking. So an end operation on the checks that allows the speculation to continue. And so gives back some of the performance again. So about the details. So currently we have four spectrum variants, or basically four. Meltdown is variant three. But usually the name is Meltdown there. And that issue we fixed fully on the first release using the page table isolation, meaning that the user space and the kernel address space are separate and flipped on, flipped off on transitioning from kernel to user land. The good thing is that there's one issue, that is the problem, and we have one fixed for it. So there's no confusion, is something not fixed, or is there still a partial fix missing? This is different for the variant one of Spectra. That is basically, we can see it as a new bug class. So if you are familiar with it, if construct, so if and speculative execution on secret data, public data, that can happen in a lot of places of code and the compiler cannot easily detect it. So Microsoft has done some work on changing their C compiler to detect such patterns and emit respective boundaries fencing in there for GCC that is not yet done. So basically you have to look at a lot of code and see if it needs a fence there or not. So we are still at upstream and we are still poking, fixing the variant one issue in all the various places that happens throughout the kernel. The variant two was with the jump buffer misprediction. The good thing is that it only is interesting if you transition from privilege to unprivileged or unprivileged to privilege code. So the kernel itself is built now with so-called red pulleys, so no, the kernel has no such gadgets that this can happen anymore. And the microcode helps flushing it once we transition between virtual machines. So that can be considered mostly or largely fixed. And variant four that was released last week, the speculative memory disambiguation that really needs from Intel microcode fixes to really fix it because it's not easily fixable with software. Our software just enables the fixing and low switching on and off the changes. But see what the future will bring on that. There are likely more of those. So this is the fun, the path that the reactive team has. So the not fun part is the massive amount. The more fun paths are these. What we do, however, for proactive security, like making a product secure before we go to shipment. We follow so-called security development life cycle. It's not formalized internally yet, but it's something that we are working on. We have quite some mechanisms in place already for doing that. We have automated checks during build. So our build service runs after building the RPM various scripts that looks for security binaries, service, policy, everything that can cause the user to root transition and looks into our white lists if they have been seen by us and have been white listed. If they have not been white listed, they will be, they will abort the build and the error message will say please contact the security team for white listing the service. So basically then people can, the package can open a bug report. We will look at the code. We will tell them, yeah, there are bugs in there. Fix them first. And then we, in the discussion with the package, we can agree on how to fix it or how, if or if to enable it or not. Yeah, we also look at things that are security-relevant that are not easily catchable by build checks like network services or other system services that could take a look at. Our audits are rolling, rolling as same as TamilVeet. So the same TamilVeet rules, we see breakage, we address the breakage. So we are not going, when the product is being shipped, we have not a milestone and Ludwig's plan that says security audits on the first of April, no, we work all year long on these kind of audits. So as Ludwig takes the open souselip development takes from open souselit TamilVeet, they see the audited packages, the proof packages from our work over time. But we do look at the final product to see if something was enabled that should have been enabled, like we scan for the open network ports and look at the system services running. So if something has escaped our eyes so far. So our, that really the work in TamilVeet is definitely benefiting our work in Sleece. So for Sleece, we don't have that many additional things to review. We have all reviewed them in TamilVeet before. To give you a bit of a number that we kind of handled 80 audit or we opened at least 80 audit requests in 2017 and that number seems to be increasing over time. One thing is also where I was part of it, the activation of, for instance, new compiler flex for Stack Clash, we needed a new compiler flex for the C compiler, minus F stack, minus Clash, minus protection. And I needed to enable it in TamilVeet. And in TamilVeet, any change that you do needs to pass staging because you cannot just enable something and have to think breaks and no one can fix it. So I got a staging project, I had the RPM off-flex change to include it. And then I went and fixed all the bugs that were inside. Fortunately for Stack Clash protection were one or two at most. So it was a very low number. And once I convinced that this is a good thing and once OPCQA is run, I basically told the open-suser factory chief Dominic to add this option also to the regular factory OPCFLEX that was it. So fortunately the Stack Clash protection was an easy transition. Also we changed Stack Protector mode, the heuristics with Stack Canaries from default to the strong mode which brings a bit more, which protects some more functions. It was similar, easy, but also done via staging. They had to fix some packages that stripped out minus F Stack Protector with Z from the OPCFLEX and that didn't know that there could be a F Stack Protector minus strong. So again like five to ten packages I had to fix and then fortunately no compiled problems or no OPCQA problems ended, also went into the distribution. One of the last year's project that was finished last year is the position, independent executable support. Up to then we had only some of the packages were built with that, mostly the security packages. And the question was if we can enable the distribution wide. So there were some challenges on that. One challenge is convincing people it doesn't lose performance. Because on x86 32-bit it uses one register more to enable that, the eBX register. And so the compiled binaries will be slower for an unknown amount. But fortunately as most of the distributions that we currently provide are 64-bit. An additional register usage is not so important anymore and our measurements that we took only showed low digit numbers of percentage problems like one percent or even less. So the allowance on the performance side was taken. And the enabling that in the distribution is a bit tricky. It's not just a C-Flex for the C compiler but it's also a linker-flex. And while previous approaches maybe we just changed the optimization flex and all our packages use those from the global setting, it was not so easy to do this for the position independent executable support. I did a bit of research and lots of discussion with our compiler team and first I said let's change the compiler to do this by default. This did not res... While the compiler guys liked it, the distribution integrators didn't like it. So that was a method that I couldn't go. Then we found the idea of doing a compiler for a JCC wrapper but it was also too ugly for some people. And what we are doing now is there's a compiler default setting file, configuration file that can be plugged into the system and we have one RPM package that just provides this default file. And that puts the PI options into the linker and compiler flex if some conditions are met. And once you install this package called GCC-PI, your compiler will automatically generate PI binaries. So having this package, getting the project config set so that it gets default installed was the way to enable it and then I looked at the fallout and there were like, I think, 50 packages also that needed attention. So mostly packages that built static binaries that were not able to link those position independent executables. For instance Xen builds some BIOS images but I would have... I really wanted to build Xen with PI but build some BIOS images some deep in their make file chain. So I can say that we now compile 99.something percent of packages with position independent secure support. Some of those we don't but these are quite well documented and if people want to build them with PI support they can approach that. Take away from this even security has to go through the staging process and occasionally it's easy, occasionally it's however a bit harder. So our project security guys they also do audits for non-distribution products. So the enterprise distribution is not the only thing that we do these days, we do cloud based on OpenStack, we use storage. There have been some self-talks where you got told about this yesterday. We have through the manager and also some container-based, container stack-based products these days. These are not the ratio of distributions. There are quite large frameworks of lots of components interacting and all these component interaction of course can cause security issues. So this is something that we are also looking at, where we also give security feedback and look at it basically. There we also embedded one of our engineers in the product development team because they work in an agile fashion and in an agile fashion the participation is kind of important. So some of them the outside will not be able to be effective working with them. So we loaned one of our engineers for half a year into this development team and he established code scanning in the continuous integration pipelines and similar things. As we are building up more stuff we can do that more often with the new products that we ship. And that's it basically. We have a landing page for all our resources on our website. There are all the machine readable data, the human readable information and our contact information is displayed. Main contact address is security at Suzy DE. And if you have a question on security topics, feel free to ask me here at the conference or by email. Just feel free. Questions? If I would speak further. But the security more related for the users. Now I think the security in the way that something can reach the root level is quite good. But what about some sandboxing or something? If the user accidentally starts some malicious binary, how to minimize the damage? Yes, we don't have done any specifics on that. So for the web browsers we rely on upstream work from say Chromium, from Firefox that do their sandboxing now. We currently have not been set up to develop this on our own. So we tried several years ago to do form of app armor protection of user space programs. But as they are very flexible in their actions it was kind of hard to confine them. So but fortunately as I said the Chromium web browser is with their sandboxing models are quite advanced. We rely on the use of the programs that we use that they do that already. We are of course looking at such methods like we'll see if we do some more as Linux support or something like that. But so far we are not doing anything on our own on that topic. So there was a lot of work to address spectre in the kernel. Is there need to do work in user space like exerverance such stuff? That's tricky. For user space interesting are parts where you have untrusted code running that can be controlled by an attacker. So for instance the JavaScript engines. And the JavaScript engines were actually affected by that because they have high resolution timers and they have some form of direct memory access so they could get at the cache timing. They could get cache based timings and what web script Chrome and Firefox did they made this high precision timer from JavaScript less precise and they disabled access to these kind of direct memory access pipelines. So if you have sandbox code just in time compiled or something then this is possible that such speculative problems can be exploited also in user land. So just yesterday we got to know why we should probably compile OpenSSH with it and I thought it sounds weird. But then I remember that OpenSSH has already strict sandboxing of their network handling, their PAM handling and so on. And if you can break out of those sandboxes would also be bad. So if you can assume that an attacker can access this network sandbox and with this back to a text he could read at least memory out of other sandboxes and out of other processes of SSH and it would be unfortunate. So why most user land programs don't need it because they just don't run untrusted or they are not called by untrusted code. Because we need to really take a look at the sandboxes and see if we need to protect them in any form. So yes, basically. Is this problem even for such like X server where it runs through and communicates with untrusted applications on the user or something? The idea is to have specific code patterns that you follow and that's kind of hard for the X server. We probably, I mean, one thing we should think about is that these 3D engines, they have this shader language, again a language. So we probably should think about how the shader language can probably be used to speculate with execute or to translate when it is translated into CPU machine code and that could be misused. But yeah, it can be but you see it's a research topic that still has a lot of potential of new research. More questions? Okay, then thank you and talk to you later.
 Hi, good morning.  My name is Marcus.  I work with SUSE and today I would like to introduce you to the work of our security  team, our security work, what challenges we face and what interesting things that's happened  or have happened in the last years.  If you are open SUSE community member or SUSE employee, let me briefly introduce the team  that I work in.  These are all SUSE employees.  Our team lead currently is Andreas Stieger.  He has been a project manager before, but as our previous team lead left the company,  he's now taken on this role.  There's one project manager that's me currently responsible for the reactive work.  I'm writing the plural here because we will also likely hire a new project manager in  the near future for proactive work again.  There are some security engineers.  You will be reading about them in our backzilla and by email it's like Alexander Bergmann,  Johannes Seegitz, Carol Barbioch, Matthias Gersner and Victor Pereira.  We also currently have one working student working for us on, working on a thesis on  security and software defined networks.  His name is Alexander and he's somewhere around the Greek mafia that hangs around the  conference.  There he is.  We have some open positions to fill because SUSE is growing, so we are looking for security  engineers and also soon for a new project manager.  What do we do?  We do proactive work.  Proactive work means making a distribution, a product secure before shipping and we do  reactive work, meaning reacting on security issues, responding to incidents, resolving  them, releasing updates and so on.  At some point in time, one and a half, three years ago, we decided to split those responsibilities  because our team has grown and the number of issues has grown.  So, Alex, Johannes, Victor, Carol and myself work in the reactive side of things.  We still work also on product security, but mostly we are responsible for the response  and on the proactive side where we do auditing, releases, approval.  Currently we have Andreas and Matthias, but we are still looking for people in that area  to build up.  So, what do we do?  What is reactive security?  What do we do for reactive security?  So basically, we need to track and react on all security incidents issues reported to  us that exist in our products.  I will go on the numbers a bit later on.  For that, in our sub team, we have so-called incident update managers so that everyone  is aware of what he has to do.  Basically something really like in the regular ticket handling, we mirror that also for the  security incidents.  So we have a so-called incident and update managers in a rotating role so everyone knows  when to take care of it or who takes care of what.  What we do as a security team, we coordinate.  We don't fix it ourselves.  So coordination means we put the incidents into our tracking systems, then hand over  the work of fixing to our packages.  They can of course come back to us, ask questions, get help, or we help them with patches and  so on.  But basically, we rely on our package maintainers or the COS SUSE community to fix the bugs,  submit them back to us.  The source review is done by the OBS review team externally and by the internal build  service reviewers internally.  And for the testing itself, we give back our updates to the maintenance, so-called maintenance  QR that tests these updates.  Internally, we have currently three teams testing.  Externally we have mostly rely on open QA during the testing.  Internally we also rely on the open QA automated testing framework.  But externally we largely rely on the open QA framework and some community members that  test the current updates.  We take care of the release of the updates, of course, and of documentation.  Most of the documentation is automated, but occasionally we need some additional manual  documentation with more information, more technical hints, and workarounds to handle.  So for incidents that come in, every incident that affects our software gets a bug in our  bugzilla, you can look at it.  All of those get our public or will get public after the embargo ends.  You will notice them if you look at the summary line for the while-prefix and the CENUM and  the package name.  It's in the summary.  References, descriptions, links are all in the regular bugzilla comments.  And those bugs, as said, be assigned to the package or bug owner.  So it's kind of important if you are a bug owner of a package, you will likely get assigned  these bugs.  So we rely on you on reacting on that.  Internally, we use a ticket system that manages the real flow of the incidents.  We call it SMASH.  I don't even remember any more what the abbreviation stands for.  But it does additional things that bugzilla does not provide.  So we rate the incidents.  So basically we have simplified rating of low, moderate, critical, important.  And we have a CVSS version 3 scoring, common vulnerability scoring system that is used  by us and by our customers and users to evaluate how much, how large the impact is of a security  issue.  So some standards say install all security fixes that has a CVSS rating larger than four  or larger than six or something like that.  So it's a service we deliver also for our customers that we score every issue with this  kind of rating.  Internally, as we have a lot of different code streams internally, we track the affected  nest.  So if something is affected, something is not affected.  It's not really that necessary for open zoos or Leap and tumbleweed because these are quite  fresh.  And for Leap we have currently two code streams, but usually one code stream.  So usually the software is affected in Leap or tumbleweed and will get fixes.  Internally where we have the software that is 10 years old, it's not always clear if  it's affected or not and to keep track of that we use that tool.  It's also used for planning and starting updates.  So mostly to not lose anything, to not lose sight of the number of bugs that are inside.  So how has the number of bugs developed over the last years?  So I started a, I have a bugsy query that checks every week on how many bugs were opened  in that week and it's running since 2006.  And you see that up to the middle, which is around 2013, there were around, it's probably  not really about around 10 to 12 bugs open per week and it's accepting some noise and  stayed at that rate.  And starting with 2013, the rate is growing.  And now we are currently with a lot of noise inside around the rate of 30 bugs per week.  Why this change?  The changes that Linux got more attention on from security researchers, a lot of more  people look at it and see fine bugs inside.  But mostly this is also coming from automated bug finding.  The automated bug finding is usually done as either static code checkers or with fuzzing  and a lot of those come out of the fuzzing tools.  And notably there's a tool called American Fuzzy Lop that really is easy to use, easy  to plug in into your program.  And if a code base has not seen any kind of fuzzing and isn't written in a safe language,  which unfortunately most of them are, it will find quite a number of bugs very easily.  So I will go on some details later on, but ImageMagic is our all-time favorite in that.  So to handle that, before we didn't really have that much of a ticket system, we had  some tools that helped us, but with a number of occasionally 100 bugs per week, it's very  hard to lose, not to lose track of things.  So we have written this internal ticketing system called Smash.  And basically it gives us all the new issues that we don't know about and we go over them,  we analyze all issues, open bug reports, track the affected software and the ratings.  And then defer this delegated work of fixing it to the packages.  So how does the update submissions work?  The package submits the update and the security team puts some metadata on it, additionally  to it the summary, the description and the issues in the machine readable format.  So basically keeping the same style over our documentation that is then generated because  out of that the automated emails are generated and update notices and so on are generated.  We check if it's good, if it's building, if some standard checks run on it and then forward  it to our QA.  QA is both manual and automated, so we have quite a lot of checkers these days.  We have open QA and our QA itself tests still the, they test bug fix certification and if  not yet automated regression fix, regression testing.  The end we release the update and do potentially additional documentation work if not done automatically.  So also for the update submissions we have a tool that makes us not lose track.  So basically listing what is submitted, what is currently prepared, what is in testing.  So here I have the tab open that is used by our QA department that looks at what incidents  are the next for testing, they are prioritized to classify it and so on.  So without this kind of tooling it would not work anymore.  What do we deliver?  We deliver the updates of course.  And a lot of compliance checking users, they get emails, email notifications, we present  them either as email archive update, email archive or we have a web page where you can  even search or click on them under this URL.  We generate machine readable advisory data that's called CVRF, common vulnerability reporting  format.  It doesn't have so many external tools yet there is a Python module that uses it that  you can hook into your applications.  So it's only taking off a bit right now.  And we also auto-gain information based on a per CVE ID.  So you can look at pages for every CVE ID that we fixed and see what is the rating, what  is affected, what is not affected.  So basically everything that we worked inside is exposed to the outside in a human readable  format but also in a machine readable format.  The machine readable format is in so-called OVIM, open vulnerability assessment language  format.  This is part of the so-called SCAP, security content automation protocol format suite and  can be feed into the open SCAP utility or other processing utilities to find automatically  if your system is still affected by a CVE or not.  That's very helpful if you manage hundreds or thousands of machines for instance.  So some of the statistics, I took the full year 2017 so I'm not looking at the last six  months but the full year 2017.  Overall we processed over 3,000 CVEs declaring them as affected or not affected.  We opened 2,500 bucks which the number really surprised me already.  I would have thought like a thousand or so but it really was 2,500.  Out of that I took the highlights.  The highlights were we have 266 kernel issues.  The kernel itself also gets fuzzing so more things are also found in the kernel.  As I mentioned image matching and graphics magic are really outliers on the number of  issues found mostly because they implement a lot of codecs.  It's quite old software where people programmed without security in mind at that time.  The good thing is both are improving, graphics magic is already based on a better consistency  model than image magic and both are continuously improving so less and less bucks are found  anymore.  Mozilla Firefox here accounted the total number of CVEs.  It's a high attention code base so a lot of issues are still found in it.  TIFF and Wireshark also got a lot of reports from fuzzing.  And there's a long list that has less bucks opened.  We released a thousand security updates for the enterprise product code lines and we released  626 for open Zuzo in 2017.  And this open Zuzo Leap is inheriting a lot of packages from SLEE.  I also looked at the number what of those were SLEE imports.  There were 250 updates directly imported from SLEE.  So one third of the security fixes for Leap automatically came from our Zuzo Enterprise  code base and did not need any open Zuzo community attention.  What is not counted here is that also the kernel, the same Git tree is used also for  Leap 4203 as for SLEE 12, SBE 3.  So the kernel itself is also coming from SLEE but not listed separately here.  And in the end we fixed all together 2,300 CVEs in 2017.  So without any tooling this is quite impossible and it's really getting really a pipelining  feel, so a factory pipeline feel like a car pipeline or something like that.  So I don't really know anymore what bugs I opened last week.  So was there an image magic bug I don't know.  But some bugs shine out of it.  Some bugs get more attention than others and we classified those as high impact vulnerabilities.  So what are?  So this is a name that we invented.  Why are these high impact vulnerabilities?  Not just because they get a lot of attention but they are high impact.  So like worms could exploit them.  It's trivial, exploitable, with a root exploit.  What I really, they really cause a lot of customer queries because they are difficult  to fix.  They need some work arounds or something like that.  So everything that stands out, that needs special attention, that needs also special  reaction times, we classify as so-called high impact vulnerabilities and handle them with  more attention, with more manual work, with more manual documentation on top.  The code names are probably familiar to you, and Shellshock started that trend with code  name vulnerabilities and coming up to Meltdown and Spectra, beginning of this year.  So let me tell you about some of those.  So the Stacklash, you probably forgot about it already, that was one year ago.  The issue was not that something was buggy, so the programs were okay, they were working  as intended, the code was okay, the kernel was okay, but there was a flaw in the Linux  memory management model.  So basically everything exists in the same address space, and most of the data that is  allocated grows upwards, except the stack, the stack grows downwards.  So the researchers found here, what if those happen to collide?  So this is something that was found even earlier, last time was in 2010, that people could  explore it in 2010, they thought, yeah, we do one page, we do a little bit of buffer  between that one page, so no one will cross that buffer end that protects both address  spaces collide.  And last year, a security research company, Qualis, found that there are places in common  code where more than this one page, this four kilobyte, can be jumped over, so that really  you could crash stuff together again.  And sadly, one of those places was in G-LipC.  In G-LipC, even before the program starts, G-LipC does some of those allocations, those  on-stack allocations for the audit, at the library path, and similar things.  So it was even exploitable without actually starting the program.  So you just took a security program, and during starting, G-LipC was in the start routines  already causing an exploitable problem.  Yeah, that was kind of hard.  It's definitely something that is called a high-impact vulnerability, because it was  quite easy to exploit.  But it was kind of hard to fix.  So foremost, the kernel received a bigger band-aid gap.  So currently up to now it was four kilobyte, and now it's one megabyte or two megabytes.  It can also be configured now.  So if you want to save address space, then you can have a smaller gap if you trust the  code.  So that killed most of the attacks, but it did not remove the underlying program, actually.  So what was additionally added is that GCC now gets code where on such stack allocations  that are done by the compiler, it touches every page during this allocation, and so causes  a fault if someone jumps over more than four kilobyte or more than eight kilobyte pages.  And as we changed this in GCC, we had to actually rebuild all the packages.  It's a bit tricky.  We don't really want to do that for released code streams, but what we did is that at least  for C12 and OpenSuselip 4203, we enabled that also for the updates, and all updates that  we have released since then have this feature enabled and are built with this compiler feature.  And interesting enough, after fixing the kernel, most of the security researcher community  lost interest in that topic.  It's just that to avoid finding it again in a year or two, we now switched our distribution  to use this compiler option and to avoid this problem altogether.  Yeah, and Spectre and Meltdown started, was released begin of January this year.  So side channel attacks are not new.  I learned about this in university in the 90s.  Side channel attacks using speculative executions is, however, quite a new concept that was  really brought up in the last year.  And researchers are now looking deeply into that topic.  So we will, while we have seen some of the issues in begin of January and one other issue  just last week, there will likely be more issues coming up because researchers are now  looking at it.  And with kind of a run through of like half a year, we will like see every half year or  also we will see probably new issues.  The concept is what a basic problem is that the CPUs speculative execute following instructions  to make more out of parallelism or make it more parallel to be faster and quicker.  And they take shortcuts during that.  So because every correct execution would cost a lot of transistors, die space, and so they  take shortcuts and almost all of these shortcuts can cause these side channel attacks.  So for instance, if you look at Meltdown, Meltdown, the main problem is the lack of access control  on a supervisor bit of the page table entries.  So the speculative engine did not look at the supervisor bit.  So it accessed page tables, page entries, or memory, even though it was not able or should  not have been able to access it.  It was a shortcut taken to make it faster to have less transistors, but it caused this  kind of security issue.  And the general part is that the speculative execution fetches stuff into caches is probably  kind of unavoidable to just gain the speed or keep the speed.  So we will probably always see some kind of cache-based side channel attacks in this area.  What are our problems here?  There have been strict embargoes on it for several months.  Into itself took like half a year.  We were involved since November, around the end of November on that.  And our engineers were involved around October.  So they were working on it since around October until January.  It's not a software bug.  The software is fine.  But we need software workarounds because we cannot replace all processors in the field.  And some of those need CPU microcode changes, especially this speculative bound jump prediction  violation that variant 2 was needed some microcode changes to flush the prediction buffers.  As these need software workarounds, the patches are not straightforward.  So usually a security fix in software is called by, like Oliver said this morning, I just  need to add an if somewhere.  So usually the security fixes are two liners, four liners or something like that.  But this really needed a massive afford, hundreds of lines of code in tricky code paths like  the system call, entry path, assembly with also high risk of causing the regressions.  And the one thing is that the fixes lose performance.  And customers, well, customers, you want security, they also want performance.  And the priorities for my team is, of course, we want security.  But the customers, they are kind of pissed if their computer takes half an hour longer  or performance drops by 20%.  So that caused quite some feedback rounds.  And after it got public, we fortunately people found for some of the problems faster equivalent  of fixing it.  So the variant one, at the beginning, they were fencing that stopped all the speculative  execution.  Now they use a different code, not an if construct, but more a masking.  So an end operation on the checks that allows the speculation to continue.  And so gives back some of the performance again.  So about the details.  So currently we have four spectrum variants, or basically four.  Meltdown is variant three.  But usually the name is Meltdown there.  And that issue we fixed fully on the first release using the page table isolation, meaning  that the user space and the kernel address space are separate and flipped on, flipped  off on transitioning from kernel to user land.  The good thing is that there's one issue, that is the problem, and we have one fixed  for it.  So there's no confusion, is something not fixed, or is there still a partial fix missing?  This is different for the variant one of Spectra.  That is basically, we can see it as a new bug class.  So if you are familiar with it, if construct, so if and speculative execution on secret  data, public data, that can happen in a lot of places of code and the compiler cannot  easily detect it.  So Microsoft has done some work on changing their C compiler to detect such patterns and  emit respective boundaries fencing in there for GCC that is not yet done.  So basically you have to look at a lot of code and see if it needs a fence there or not.  So we are still at upstream and we are still poking, fixing the variant one issue in all  the various places that happens throughout the kernel.  The variant two was with the jump buffer misprediction.  The good thing is that it only is interesting if you transition from privilege to unprivileged  or unprivileged to privilege code.  So the kernel itself is built now with so-called red pulleys, so no, the kernel has no such  gadgets that this can happen anymore.  And the microcode helps flushing it once we transition between virtual machines.  So that can be considered mostly or largely fixed.  And variant four that was released last week, the speculative memory disambiguation that  really needs from Intel microcode fixes to really fix it because it's not easily fixable  with software.  Our software just enables the fixing and low switching on and off the changes.  But see what the future will bring on that.  There are likely more of those.  So this is the fun, the path that the reactive team has.  So the not fun part is the massive amount.  The more fun paths are these.  What we do, however, for proactive security, like making a product secure before we go  to shipment.  We follow so-called security development life cycle.  It's not formalized internally yet, but it's something that we are working on.  We have quite some mechanisms in place already for doing that.  We have automated checks during build.  So our build service runs after building the RPM various scripts that looks for security  binaries, service, policy, everything that can cause the user to root transition and  looks into our white lists if they have been seen by us and have been white listed.  If they have not been white listed, they will be, they will abort the build and the error  message will say please contact the security team for white listing the service.  So basically then people can, the package can open a bug report.  We will look at the code.  We will tell them, yeah, there are bugs in there.  Fix them first.  And then we, in the discussion with the package, we can agree on how to fix it or how, if or  if to enable it or not.  Yeah, we also look at things that are security-relevant that are not easily catchable by build checks  like network services or other system services that could take a look at.  Our audits are rolling, rolling as same as TamilVeet.  So the same TamilVeet rules, we see breakage, we address the breakage.  So we are not going, when the product is being shipped, we have not a milestone and  Ludwig's plan that says security audits on the first of April, no, we work all year long  on these kind of audits.  So as Ludwig takes the open souselip development takes from open souselit TamilVeet, they see  the audited packages, the proof packages from our work over time.  But we do look at the final product to see if something was enabled that should have  been enabled, like we scan for the open network ports and look at the system services running.  So if something has escaped our eyes so far.  So our, that really the work in TamilVeet is definitely benefiting our work in Sleece.  So for Sleece, we don't have that many additional things to review.  We have all reviewed them in TamilVeet before.  To give you a bit of a number that we kind of handled 80 audit or we opened at least  80 audit requests in 2017 and that number seems to be increasing over time.  One thing is also where I was part of it, the activation of, for instance, new compiler  flex for Stack Clash, we needed a new compiler flex for the C compiler, minus F stack, minus  Clash, minus protection.  And I needed to enable it in TamilVeet.  And in TamilVeet, any change that you do needs to pass staging because you cannot just enable  something and have to think breaks and no one can fix it.  So I got a staging project, I had the RPM off-flex change to include it.  And then I went and fixed all the bugs that were inside.  Fortunately for Stack Clash protection were one or two at most.  So it was a very low number.  And once I convinced that this is a good thing and once OPCQA is run, I basically told the  open-suser factory chief Dominic to add this option also to the regular factory OPCFLEX  that was it.  So fortunately the Stack Clash protection was an easy transition.  Also we changed Stack Protector mode, the heuristics with Stack Canaries from default to the strong  mode which brings a bit more, which protects some more functions.  It was similar, easy, but also done via staging.  They had to fix some packages that stripped out minus F Stack Protector with Z from the  OPCFLEX and that didn't know that there could be a F Stack Protector minus strong.  So again like five to ten packages I had to fix and then fortunately no compiled problems  or no OPCQA problems ended, also went into the distribution.  One of the last year's project that was finished last year is the position, independent executable  support.  Up to then we had only some of the packages were built with that, mostly the security packages.  And the question was if we can enable the distribution wide.  So there were some challenges on that.  One challenge is convincing people it doesn't lose performance.  Because on x86 32-bit it uses one register more to enable that, the eBX register.  And so the compiled binaries will be slower for an unknown amount.  But fortunately as most of the distributions that we currently provide are 64-bit.  An additional register usage is not so important anymore and our measurements that we took  only showed low digit numbers of percentage problems like one percent or even less.  So the allowance on the performance side was taken.  And the enabling that in the distribution is a bit tricky.  It's not just a C-Flex for the C compiler but it's also a linker-flex.  And while previous approaches maybe we just changed the optimization flex and all our  packages use those from the global setting, it was not so easy to do this for the position  independent executable support.  I did a bit of research and lots of discussion with our compiler team and first I said let's  change the compiler to do this by default.  This did not res...  While the compiler guys liked it, the distribution integrators didn't like it.  So that was a method that I couldn't go.  Then we found the idea of doing a compiler for a JCC wrapper but it was also too ugly  for some people.  And what we are doing now is there's a compiler default setting file, configuration file that  can be plugged into the system and we have one RPM package that just provides this default  file.  And that puts the PI options into the linker and compiler flex if some conditions are met.  And once you install this package called GCC-PI, your compiler will automatically generate  PI binaries.  So having this package, getting the project config set so that it gets default installed  was the way to enable it and then I looked at the fallout and there were like, I think,  50 packages also that needed attention.  So mostly packages that built static binaries that were not able to link those position  independent executables.  For instance Xen builds some BIOS images but I would have...  I really wanted to build Xen with PI but build some BIOS images some deep in their make file  chain.  So I can say that we now compile 99.something percent of packages with position independent  secure support.  Some of those we don't but these are quite well documented and if people want to build  them with PI support they can approach that.  Take away from this even security has to go through the staging process and occasionally  it's easy, occasionally it's however a bit harder.  So our project security guys they also do audits for non-distribution products.  So the enterprise distribution is not the only thing that we do these days, we do cloud  based on OpenStack, we use storage.  There have been some self-talks where you got told about this yesterday.  We have through the manager and also some container-based, container stack-based products  these days.  These are not the ratio of distributions.  There are quite large frameworks of lots of components interacting and all these component  interaction of course can cause security issues.  So this is something that we are also looking at, where we also give security feedback and  look at it basically.  There we also embedded one of our engineers in the product development team because they  work in an agile fashion and in an agile fashion the participation is kind of important.  So some of them the outside will not be able to be effective working with them.  So we loaned one of our engineers for half a year into this development team and he established  code scanning in the continuous integration pipelines and similar things.  As we are building up more stuff we can do that more often with the new products that  we ship.  And that's it basically.  We have a landing page for all our resources on our website.  There are all the machine readable data, the human readable information and our contact  information is displayed.  Main contact address is security at Suzy DE.  And if you have a question on security topics, feel free to ask me here at the conference  or by email.  Just feel free.  Questions?  If I would speak further.  But the security more related for the users.  Now I think the security in the way that something can reach the root level is quite  good.  But what about some sandboxing or something?  If the user accidentally starts some malicious binary, how to minimize the damage?  Yes, we don't have done any specifics on that.  So for the web browsers we rely on upstream work from say Chromium, from Firefox that  do their sandboxing now.  We currently have not been set up to develop this on our own.  So we tried several years ago to do form of app armor protection of user space programs.  But as they are very flexible in their actions it was kind of hard to confine them.  So but fortunately as I said the Chromium web browser is with their sandboxing models  are quite advanced.  We rely on the use of the programs that we use that they do that already.  We are of course looking at such methods like we'll see if we do some more as Linux support  or something like that.  But so far we are not doing anything on our own on that topic.  So there was a lot of work to address spectre in the kernel.  Is there need to do work in user space like exerverance such stuff?  That's tricky.  For user space interesting are parts where you have untrusted code running that can be  controlled by an attacker.  So for instance the JavaScript engines.  And the JavaScript engines were actually affected by that because they have high resolution  timers and they have some form of direct memory access so they could get at the cache timing.  They could get cache based timings and what web script Chrome and Firefox did they made  this high precision timer from JavaScript less precise and they disabled access to these  kind of direct memory access pipelines.  So if you have sandbox code just in time compiled or something then this is possible that such  speculative problems can be exploited also in user land.  So just yesterday we got to know why we should probably compile OpenSSH with it and I thought  it sounds weird.  But then I remember that OpenSSH has already strict sandboxing of their network handling,  their PAM handling and so on.  And if you can break out of those sandboxes would also be bad.  So if you can assume that an attacker can access this network sandbox and with this  back to a text he could read at least memory out of other sandboxes and out of other processes  of SSH and it would be unfortunate.  So why most user land programs don't need it because they just don't run untrusted or  they are not called by untrusted code.  Because we need to really take a look at the sandboxes and see if we need to protect them  in any form.  So yes, basically.  Is this problem even for such like X server where it runs through and communicates with  untrusted applications on the user or something?  The idea is to have specific code patterns that you follow and that's kind of hard for  the X server.  We probably, I mean, one thing we should think about is that these 3D engines, they have  this shader language, again a language.  So we probably should think about how the shader language can probably be used to speculate  with execute or to translate when it is translated into CPU machine code and that could be misused.  But yeah, it can be but you see it's a research topic that still has a lot of potential of  new research.  More questions?  Okay, then thank you and talk to you later.
openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_0_131.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_132_695.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_696_2519.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_2520_3863.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_3864_3911.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_3912_3959.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_3960_4007.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_4008_7247.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_7248_11147.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_11148_13127.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_13128_13907.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_13908_13955.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_13956_14003.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_14004_14183.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_14184_14327.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_14328_14387.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_14388_14435.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_14436_15971.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_15972_16139.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_16140_16259.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_16260_16355.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_16356_16631.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_16632_16739.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_16740_16835.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_16836_16883.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_16884_19367.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_19368_21971.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_21972_22019.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_22020_22079.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_22080_22151.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_22152_22199.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_22200_22475.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_22476_22679.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_22680_22727.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_22728_22871.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_22872_23867.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_23868_24071.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_24072_25631.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_25632_25751.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_25752_31391.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_31392_38519.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_38520_42983.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_42984_43031.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_43032_43079.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_43080_43127.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_43128_45335.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_45336_47411.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_47412_56135.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_56136_58475.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_58476_66959.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_66960_67007.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_67008_67055.png openSUSE_Conference_2018_-_Security_Retrospective__Xjt2uENIPas_67056_67194.png
Hi, good morning. My name is Marcus. I work with SUSE and today I would like to introduce you to the work of our security team, our security work, what challenges we face and what interesting things that's happened or have happened in the last years. If you are open SUSE community member or SUSE employee, let me briefly introduce the team that I work in. These are all SUSE employees. Our team lead currently is Andreas Stieger. He has been a project manager before, but as our previous team lead left the company, he's now taken on this role. There's one project manager that's me currently responsible for the reactive work. I'm writing the plural here because we will also likely hire a new project manager in the near future for proactive work again. There are some security engineers. You will be reading about them in our backzilla and by email it's like Alexander Bergmann, Johannes Seegitz, Carol Barbioch, Matthias Gersner and Victor Pereira. We also currently have one working student working for us on, working on a thesis on security and software defined networks. His name is Alexander and he's somewhere around the Greek mafia that hangs around the conference. There he is. We have some open positions to fill because SUSE is growing, so we are looking for security engineers and also soon for a new project manager. What do we do? We do proactive work. Proactive work means making a distribution, a product secure before shipping and we do reactive work, meaning reacting on security issues, responding to incidents, resolving them, releasing updates and so on. At some point in time, one and a half, three years ago, we decided to split those responsibilities because our team has grown and the number of issues has grown. So, Alex, Johannes, Victor, Carol and myself work in the reactive side of things. We still work also on product security, but mostly we are responsible for the response and on the proactive side where we do auditing, releases, approval. Currently we have Andreas and Matthias, but we are still looking for people in that area to build up. So, what do we do? What is reactive security? What do we do for reactive security? So basically, we need to track and react on all security incidents issues reported to us that exist in our products. I will go on the numbers a bit later on. For that, in our sub team, we have so-called incident update managers so that everyone is aware of what he has to do. Basically something really like in the regular ticket handling, we mirror that also for the security incidents. So we have a so-called incident and update managers in a rotating role so everyone knows when to take care of it or who takes care of what. What we do as a security team, we coordinate. We don't fix it ourselves. So coordination means we put the incidents into our tracking systems, then hand over the work of fixing to our packages. They can of course come back to us, ask questions, get help, or we help them with patches and so on. But basically, we rely on our package maintainers or the COS SUSE community to fix the bugs, submit them back to us. The source review is done by the OBS review team externally and by the internal build service reviewers internally. And for the testing itself, we give back our updates to the maintenance, so-called maintenance QR that tests these updates. Internally, we have currently three teams testing. Externally we have mostly rely on open QA during the testing. Internally we also rely on the open QA automated testing framework. But externally we largely rely on the open QA framework and some community members that test the current updates. We take care of the release of the updates, of course, and of documentation. Most of the documentation is automated, but occasionally we need some additional manual documentation with more information, more technical hints, and workarounds to handle. So for incidents that come in, every incident that affects our software gets a bug in our bugzilla, you can look at it. All of those get our public or will get public after the embargo ends. You will notice them if you look at the summary line for the while-prefix and the CENUM and the package name. It's in the summary. References, descriptions, links are all in the regular bugzilla comments. And those bugs, as said, be assigned to the package or bug owner. So it's kind of important if you are a bug owner of a package, you will likely get assigned these bugs. So we rely on you on reacting on that. Internally, we use a ticket system that manages the real flow of the incidents. We call it SMASH. I don't even remember any more what the abbreviation stands for. But it does additional things that bugzilla does not provide. So we rate the incidents. So basically we have simplified rating of low, moderate, critical, important. And we have a CVSS version 3 scoring, common vulnerability scoring system that is used by us and by our customers and users to evaluate how much, how large the impact is of a security issue. So some standards say install all security fixes that has a CVSS rating larger than four or larger than six or something like that. So it's a service we deliver also for our customers that we score every issue with this kind of rating. Internally, as we have a lot of different code streams internally, we track the affected nest. So if something is affected, something is not affected. It's not really that necessary for open zoos or Leap and tumbleweed because these are quite fresh. And for Leap we have currently two code streams, but usually one code stream. So usually the software is affected in Leap or tumbleweed and will get fixes. Internally where we have the software that is 10 years old, it's not always clear if it's affected or not and to keep track of that we use that tool. It's also used for planning and starting updates. So mostly to not lose anything, to not lose sight of the number of bugs that are inside. So how has the number of bugs developed over the last years? So I started a, I have a bugsy query that checks every week on how many bugs were opened in that week and it's running since 2006. And you see that up to the middle, which is around 2013, there were around, it's probably not really about around 10 to 12 bugs open per week and it's accepting some noise and stayed at that rate. And starting with 2013, the rate is growing. And now we are currently with a lot of noise inside around the rate of 30 bugs per week. Why this change? The changes that Linux got more attention on from security researchers, a lot of more people look at it and see fine bugs inside. But mostly this is also coming from automated bug finding. The automated bug finding is usually done as either static code checkers or with fuzzing and a lot of those come out of the fuzzing tools. And notably there's a tool called American Fuzzy Lop that really is easy to use, easy to plug in into your program. And if a code base has not seen any kind of fuzzing and isn't written in a safe language, which unfortunately most of them are, it will find quite a number of bugs very easily. So I will go on some details later on, but ImageMagic is our all-time favorite in that. So to handle that, before we didn't really have that much of a ticket system, we had some tools that helped us, but with a number of occasionally 100 bugs per week, it's very hard to lose, not to lose track of things. So we have written this internal ticketing system called Smash. And basically it gives us all the new issues that we don't know about and we go over them, we analyze all issues, open bug reports, track the affected software and the ratings. And then defer this delegated work of fixing it to the packages. So how does the update submissions work? The package submits the update and the security team puts some metadata on it, additionally to it the summary, the description and the issues in the machine readable format. So basically keeping the same style over our documentation that is then generated because out of that the automated emails are generated and update notices and so on are generated. We check if it's good, if it's building, if some standard checks run on it and then forward it to our QA. QA is both manual and automated, so we have quite a lot of checkers these days. We have open QA and our QA itself tests still the, they test bug fix certification and if not yet automated regression fix, regression testing. The end we release the update and do potentially additional documentation work if not done automatically. So also for the update submissions we have a tool that makes us not lose track. So basically listing what is submitted, what is currently prepared, what is in testing. So here I have the tab open that is used by our QA department that looks at what incidents are the next for testing, they are prioritized to classify it and so on. So without this kind of tooling it would not work anymore. What do we deliver? We deliver the updates of course. And a lot of compliance checking users, they get emails, email notifications, we present them either as email archive update, email archive or we have a web page where you can even search or click on them under this URL. We generate machine readable advisory data that's called CVRF, common vulnerability reporting format. It doesn't have so many external tools yet there is a Python module that uses it that you can hook into your applications. So it's only taking off a bit right now. And we also auto-gain information based on a per CVE ID. So you can look at pages for every CVE ID that we fixed and see what is the rating, what is affected, what is not affected. So basically everything that we worked inside is exposed to the outside in a human readable format but also in a machine readable format. The machine readable format is in so-called OVIM, open vulnerability assessment language format. This is part of the so-called SCAP, security content automation protocol format suite and can be feed into the open SCAP utility or other processing utilities to find automatically if your system is still affected by a CVE or not. That's very helpful if you manage hundreds or thousands of machines for instance. So some of the statistics, I took the full year 2017 so I'm not looking at the last six months but the full year 2017. Overall we processed over 3,000 CVEs declaring them as affected or not affected. We opened 2,500 bucks which the number really surprised me already. I would have thought like a thousand or so but it really was 2,500. Out of that I took the highlights. The highlights were we have 266 kernel issues. The kernel itself also gets fuzzing so more things are also found in the kernel. As I mentioned image matching and graphics magic are really outliers on the number of issues found mostly because they implement a lot of codecs. It's quite old software where people programmed without security in mind at that time. The good thing is both are improving, graphics magic is already based on a better consistency model than image magic and both are continuously improving so less and less bucks are found anymore. Mozilla Firefox here accounted the total number of CVEs. It's a high attention code base so a lot of issues are still found in it. TIFF and Wireshark also got a lot of reports from fuzzing. And there's a long list that has less bucks opened. We released a thousand security updates for the enterprise product code lines and we released 626 for open Zuzo in 2017. And this open Zuzo Leap is inheriting a lot of packages from SLEE. I also looked at the number what of those were SLEE imports. There were 250 updates directly imported from SLEE. So one third of the security fixes for Leap automatically came from our Zuzo Enterprise code base and did not need any open Zuzo community attention. What is not counted here is that also the kernel, the same Git tree is used also for Leap 4203 as for SLEE 12, SBE 3. So the kernel itself is also coming from SLEE but not listed separately here. And in the end we fixed all together 2,300 CVEs in 2017. So without any tooling this is quite impossible and it's really getting really a pipelining feel, so a factory pipeline feel like a car pipeline or something like that. So I don't really know anymore what bugs I opened last week. So was there an image magic bug I don't know. But some bugs shine out of it. Some bugs get more attention than others and we classified those as high impact vulnerabilities. So what are? So this is a name that we invented. Why are these high impact vulnerabilities? Not just because they get a lot of attention but they are high impact. So like worms could exploit them. It's trivial, exploitable, with a root exploit. What I really, they really cause a lot of customer queries because they are difficult to fix. They need some work arounds or something like that. So everything that stands out, that needs special attention, that needs also special reaction times, we classify as so-called high impact vulnerabilities and handle them with more attention, with more manual work, with more manual documentation on top. The code names are probably familiar to you, and Shellshock started that trend with code name vulnerabilities and coming up to Meltdown and Spectra, beginning of this year. So let me tell you about some of those. So the Stacklash, you probably forgot about it already, that was one year ago. The issue was not that something was buggy, so the programs were okay, they were working as intended, the code was okay, the kernel was okay, but there was a flaw in the Linux memory management model. So basically everything exists in the same address space, and most of the data that is allocated grows upwards, except the stack, the stack grows downwards. So the researchers found here, what if those happen to collide? So this is something that was found even earlier, last time was in 2010, that people could explore it in 2010, they thought, yeah, we do one page, we do a little bit of buffer between that one page, so no one will cross that buffer end that protects both address spaces collide. And last year, a security research company, Qualis, found that there are places in common code where more than this one page, this four kilobyte, can be jumped over, so that really you could crash stuff together again. And sadly, one of those places was in G-LipC. In G-LipC, even before the program starts, G-LipC does some of those allocations, those on-stack allocations for the audit, at the library path, and similar things. So it was even exploitable without actually starting the program. So you just took a security program, and during starting, G-LipC was in the start routines already causing an exploitable problem. Yeah, that was kind of hard. It's definitely something that is called a high-impact vulnerability, because it was quite easy to exploit. But it was kind of hard to fix. So foremost, the kernel received a bigger band-aid gap. So currently up to now it was four kilobyte, and now it's one megabyte or two megabytes. It can also be configured now. So if you want to save address space, then you can have a smaller gap if you trust the code. So that killed most of the attacks, but it did not remove the underlying program, actually. So what was additionally added is that GCC now gets code where on such stack allocations that are done by the compiler, it touches every page during this allocation, and so causes a fault if someone jumps over more than four kilobyte or more than eight kilobyte pages. And as we changed this in GCC, we had to actually rebuild all the packages. It's a bit tricky. We don't really want to do that for released code streams, but what we did is that at least for C12 and OpenSuselip 4203, we enabled that also for the updates, and all updates that we have released since then have this feature enabled and are built with this compiler feature. And interesting enough, after fixing the kernel, most of the security researcher community lost interest in that topic. It's just that to avoid finding it again in a year or two, we now switched our distribution to use this compiler option and to avoid this problem altogether. Yeah, and Spectre and Meltdown started, was released begin of January this year. So side channel attacks are not new. I learned about this in university in the 90s. Side channel attacks using speculative executions is, however, quite a new concept that was really brought up in the last year. And researchers are now looking deeply into that topic. So we will, while we have seen some of the issues in begin of January and one other issue just last week, there will likely be more issues coming up because researchers are now looking at it. And with kind of a run through of like half a year, we will like see every half year or also we will see probably new issues. The concept is what a basic problem is that the CPUs speculative execute following instructions to make more out of parallelism or make it more parallel to be faster and quicker. And they take shortcuts during that. So because every correct execution would cost a lot of transistors, die space, and so they take shortcuts and almost all of these shortcuts can cause these side channel attacks. So for instance, if you look at Meltdown, Meltdown, the main problem is the lack of access control on a supervisor bit of the page table entries. So the speculative engine did not look at the supervisor bit. So it accessed page tables, page entries, or memory, even though it was not able or should not have been able to access it. It was a shortcut taken to make it faster to have less transistors, but it caused this kind of security issue. And the general part is that the speculative execution fetches stuff into caches is probably kind of unavoidable to just gain the speed or keep the speed. So we will probably always see some kind of cache-based side channel attacks in this area. What are our problems here? There have been strict embargoes on it for several months. Into itself took like half a year. We were involved since November, around the end of November on that. And our engineers were involved around October. So they were working on it since around October until January. It's not a software bug. The software is fine. But we need software workarounds because we cannot replace all processors in the field. And some of those need CPU microcode changes, especially this speculative bound jump prediction violation that variant 2 was needed some microcode changes to flush the prediction buffers. As these need software workarounds, the patches are not straightforward. So usually a security fix in software is called by, like Oliver said this morning, I just need to add an if somewhere. So usually the security fixes are two liners, four liners or something like that. But this really needed a massive afford, hundreds of lines of code in tricky code paths like the system call, entry path, assembly with also high risk of causing the regressions. And the one thing is that the fixes lose performance. And customers, well, customers, you want security, they also want performance. And the priorities for my team is, of course, we want security. But the customers, they are kind of pissed if their computer takes half an hour longer or performance drops by 20%. So that caused quite some feedback rounds. And after it got public, we fortunately people found for some of the problems faster equivalent of fixing it. So the variant one, at the beginning, they were fencing that stopped all the speculative execution. Now they use a different code, not an if construct, but more a masking. So an end operation on the checks that allows the speculation to continue. And so gives back some of the performance again. So about the details. So currently we have four spectrum variants, or basically four. Meltdown is variant three. But usually the name is Meltdown there. And that issue we fixed fully on the first release using the page table isolation, meaning that the user space and the kernel address space are separate and flipped on, flipped off on transitioning from kernel to user land. The good thing is that there's one issue, that is the problem, and we have one fixed for it. So there's no confusion, is something not fixed, or is there still a partial fix missing? This is different for the variant one of Spectra. That is basically, we can see it as a new bug class. So if you are familiar with it, if construct, so if and speculative execution on secret data, public data, that can happen in a lot of places of code and the compiler cannot easily detect it. So Microsoft has done some work on changing their C compiler to detect such patterns and emit respective boundaries fencing in there for GCC that is not yet done. So basically you have to look at a lot of code and see if it needs a fence there or not. So we are still at upstream and we are still poking, fixing the variant one issue in all the various places that happens throughout the kernel. The variant two was with the jump buffer misprediction. The good thing is that it only is interesting if you transition from privilege to unprivileged or unprivileged to privilege code. So the kernel itself is built now with so-called red pulleys, so no, the kernel has no such gadgets that this can happen anymore. And the microcode helps flushing it once we transition between virtual machines. So that can be considered mostly or largely fixed. And variant four that was released last week, the speculative memory disambiguation that really needs from Intel microcode fixes to really fix it because it's not easily fixable with software. Our software just enables the fixing and low switching on and off the changes. But see what the future will bring on that. There are likely more of those. So this is the fun, the path that the reactive team has. So the not fun part is the massive amount. The more fun paths are these. What we do, however, for proactive security, like making a product secure before we go to shipment. We follow so-called security development life cycle. It's not formalized internally yet, but it's something that we are working on. We have quite some mechanisms in place already for doing that. We have automated checks during build. So our build service runs after building the RPM various scripts that looks for security binaries, service, policy, everything that can cause the user to root transition and looks into our white lists if they have been seen by us and have been white listed. If they have not been white listed, they will be, they will abort the build and the error message will say please contact the security team for white listing the service. So basically then people can, the package can open a bug report. We will look at the code. We will tell them, yeah, there are bugs in there. Fix them first. And then we, in the discussion with the package, we can agree on how to fix it or how, if or if to enable it or not. Yeah, we also look at things that are security-relevant that are not easily catchable by build checks like network services or other system services that could take a look at. Our audits are rolling, rolling as same as TamilVeet. So the same TamilVeet rules, we see breakage, we address the breakage. So we are not going, when the product is being shipped, we have not a milestone and Ludwig's plan that says security audits on the first of April, no, we work all year long on these kind of audits. So as Ludwig takes the open souselip development takes from open souselit TamilVeet, they see the audited packages, the proof packages from our work over time. But we do look at the final product to see if something was enabled that should have been enabled, like we scan for the open network ports and look at the system services running. So if something has escaped our eyes so far. So our, that really the work in TamilVeet is definitely benefiting our work in Sleece. So for Sleece, we don't have that many additional things to review. We have all reviewed them in TamilVeet before. To give you a bit of a number that we kind of handled 80 audit or we opened at least 80 audit requests in 2017 and that number seems to be increasing over time. One thing is also where I was part of it, the activation of, for instance, new compiler flex for Stack Clash, we needed a new compiler flex for the C compiler, minus F stack, minus Clash, minus protection. And I needed to enable it in TamilVeet. And in TamilVeet, any change that you do needs to pass staging because you cannot just enable something and have to think breaks and no one can fix it. So I got a staging project, I had the RPM off-flex change to include it. And then I went and fixed all the bugs that were inside. Fortunately for Stack Clash protection were one or two at most. So it was a very low number. And once I convinced that this is a good thing and once OPCQA is run, I basically told the open-suser factory chief Dominic to add this option also to the regular factory OPCFLEX that was it. So fortunately the Stack Clash protection was an easy transition. Also we changed Stack Protector mode, the heuristics with Stack Canaries from default to the strong mode which brings a bit more, which protects some more functions. It was similar, easy, but also done via staging. They had to fix some packages that stripped out minus F Stack Protector with Z from the OPCFLEX and that didn't know that there could be a F Stack Protector minus strong. So again like five to ten packages I had to fix and then fortunately no compiled problems or no OPCQA problems ended, also went into the distribution. One of the last year's project that was finished last year is the position, independent executable support. Up to then we had only some of the packages were built with that, mostly the security packages. And the question was if we can enable the distribution wide. So there were some challenges on that. One challenge is convincing people it doesn't lose performance. Because on x86 32-bit it uses one register more to enable that, the eBX register. And so the compiled binaries will be slower for an unknown amount. But fortunately as most of the distributions that we currently provide are 64-bit. An additional register usage is not so important anymore and our measurements that we took only showed low digit numbers of percentage problems like one percent or even less. So the allowance on the performance side was taken. And the enabling that in the distribution is a bit tricky. It's not just a C-Flex for the C compiler but it's also a linker-flex. And while previous approaches maybe we just changed the optimization flex and all our packages use those from the global setting, it was not so easy to do this for the position independent executable support. I did a bit of research and lots of discussion with our compiler team and first I said let's change the compiler to do this by default. This did not res... While the compiler guys liked it, the distribution integrators didn't like it. So that was a method that I couldn't go. Then we found the idea of doing a compiler for a JCC wrapper but it was also too ugly for some people. And what we are doing now is there's a compiler default setting file, configuration file that can be plugged into the system and we have one RPM package that just provides this default file. And that puts the PI options into the linker and compiler flex if some conditions are met. And once you install this package called GCC-PI, your compiler will automatically generate PI binaries. So having this package, getting the project config set so that it gets default installed was the way to enable it and then I looked at the fallout and there were like, I think, 50 packages also that needed attention. So mostly packages that built static binaries that were not able to link those position independent executables. For instance Xen builds some BIOS images but I would have... I really wanted to build Xen with PI but build some BIOS images some deep in their make file chain. So I can say that we now compile 99.something percent of packages with position independent secure support. Some of those we don't but these are quite well documented and if people want to build them with PI support they can approach that. Take away from this even security has to go through the staging process and occasionally it's easy, occasionally it's however a bit harder. So our project security guys they also do audits for non-distribution products. So the enterprise distribution is not the only thing that we do these days, we do cloud based on OpenStack, we use storage. There have been some self-talks where you got told about this yesterday. We have through the manager and also some container-based, container stack-based products these days. These are not the ratio of distributions. There are quite large frameworks of lots of components interacting and all these component interaction of course can cause security issues. So this is something that we are also looking at, where we also give security feedback and look at it basically. There we also embedded one of our engineers in the product development team because they work in an agile fashion and in an agile fashion the participation is kind of important. So some of them the outside will not be able to be effective working with them. So we loaned one of our engineers for half a year into this development team and he established code scanning in the continuous integration pipelines and similar things. As we are building up more stuff we can do that more often with the new products that we ship. And that's it basically. We have a landing page for all our resources on our website. There are all the machine readable data, the human readable information and our contact information is displayed. Main contact address is security at Suzy DE. And if you have a question on security topics, feel free to ask me here at the conference or by email. Just feel free. Questions? If I would speak further. But the security more related for the users. Now I think the security in the way that something can reach the root level is quite good. But what about some sandboxing or something? If the user accidentally starts some malicious binary, how to minimize the damage? Yes, we don't have done any specifics on that. So for the web browsers we rely on upstream work from say Chromium, from Firefox that do their sandboxing now. We currently have not been set up to develop this on our own. So we tried several years ago to do form of app armor protection of user space programs. But as they are very flexible in their actions it was kind of hard to confine them. So but fortunately as I said the Chromium web browser is with their sandboxing models are quite advanced. We rely on the use of the programs that we use that they do that already. We are of course looking at such methods like we'll see if we do some more as Linux support or something like that. But so far we are not doing anything on our own on that topic. So there was a lot of work to address spectre in the kernel. Is there need to do work in user space like exerverance such stuff? That's tricky. For user space interesting are parts where you have untrusted code running that can be controlled by an attacker. So for instance the JavaScript engines. And the JavaScript engines were actually affected by that because they have high resolution timers and they have some form of direct memory access so they could get at the cache timing. They could get cache based timings and what web script Chrome and Firefox did they made this high precision timer from JavaScript less precise and they disabled access to these kind of direct memory access pipelines. So if you have sandbox code just in time compiled or something then this is possible that such speculative problems can be exploited also in user land. So just yesterday we got to know why we should probably compile OpenSSH with it and I thought it sounds weird. But then I remember that OpenSSH has already strict sandboxing of their network handling, their PAM handling and so on. And if you can break out of those sandboxes would also be bad. So if you can assume that an attacker can access this network sandbox and with this back to a text he could read at least memory out of other sandboxes and out of other processes of SSH and it would be unfortunate. So why most user land programs don't need it because they just don't run untrusted or they are not called by untrusted code. Because we need to really take a look at the sandboxes and see if we need to protect them in any form. So yes, basically. Is this problem even for such like X server where it runs through and communicates with untrusted applications on the user or something? The idea is to have specific code patterns that you follow and that's kind of hard for the X server. We probably, I mean, one thing we should think about is that these 3D engines, they have this shader language, again a language. So we probably should think about how the shader language can probably be used to speculate with execute or to translate when it is translated into CPU machine code and that could be misused. But yeah, it can be but you see it's a research topic that still has a lot of potential of new research. More questions? Okay, then thank you and talk to you later.
things that's happened or have happened in the reactive work. I'm writing the plural here the reactive side of things. We still work also security? What do we do for reactive security? So number of bugs developed over the last years? So the statistics, I took the full year 2017 so I'm Yeah, and Spectre and Meltdown started, was some of the issues in begin of January and one transitioning from kernel to user land. The good Microsoft has done some work on changing their C are still poking, fixing the variant one issue in What we do, however, for proactive security, like bug report. We will look at the code. We will tell them, yeah, there are bugs in there. Fix for Stack Clash, we needed a new compiler flex just a C-Flex for the C compiler but it's also a the compiler guys liked it, the distribution integration pipelines and similar things. As we information and our contact information is the root level is quite good. But what about some