Hardening Debian for the Desktop Using Grsecurity

I recently built a desktop system that I think is reasonably secure. It’s running Debian sid, also known as “unstable” — though in the Debian desktop world that just means you get to use the newest software. It’s just about as stable as “stable”, and besides, #yolo. It’s also running a grsecurity-patched Linux kernel and PaX, technologies that make Linux way more secure. Grsecurity protects you against memory corruption attacks, such as buffer overflows.

Last October I traveled to Moscow and interviewed Edward Snowden. Here’s one of the things he told me:

“Something that we haven’t seen that we need to see is a greater hardening of the overall kernels of every operating system through things like grsecurity, but unfortunately there’s a big usability gap between the capabilities that are out there, that are possible, and what is attainable for the average user.”

Since I just set up Debian with a grsec kernel, I figured I’d write a tutorial for how to do it. It’s still a long way before the average user can take advantage of this stuff – it breaks everything, and the user needs to learn how to diagnose and fix it themselves – but I think that it’s well within the capabilities of Linux nerds who are comfortable using a terminal. You can probably also follow along no matter what Linux distribution you’re using. Also, I’m fairly new to grsecurity myself, so if you have tips or suggestions, or if I got something wrong, please post in the comments.

Grsecurity can be used with very little configuration. Just by baking it into your kernel, entire classes of exploits will fail to work against you, but those same exploits will succeed against someone running a vanilla Linux kernel. Grsecurity also lets you enforce access controls, but that’s outside the scope of this article. Just running a grsec kernel gives you a lot of protection for free.

First, I’m going to explain a little bit about what grsec is and how it protects you. Then I’m going to install Debian and upgrade it to sid. Then I’m going to download the Linux source code and the grsecurity patch, verify digital signatures, patch the Linux kernel, and finally compile it. (If you’ve never compiled the Linux kernel before, you’re in for a treat!) Then I’m going to show you how to use paxctl to turn off certain PaX protections on specific binaries so that they can run without crashing. And finally, I’m going to briefly show you how to troubleshoot grsec and PaX, so you can make software work that would otherwise crash, all on your own.

How does grsecurity make Linux more secure?

Memory corruption attacks normally work something like this: The target accepts malicious input into a program (maybe they load a GIF in an image viewing program, or a document in LibreOffice, or a malformed PGP public key in GnuPG). This input includes an exploit – some code that triggers a security bug in whatever software it’s targeting – as well as a small block of malicious code called shellcode. When the buggy software parses the input, the exploit takes over the flow of execution and jumps into the attacker’s shellcode. Shellcode can do many different things, but it commonly gives the attacker a remote shell – basically, letting them run commands on the victim’s computer (with the permissions of the user who was running the program that got hacked).

Grsecurity protects you by being extremely strict about which parts of memory are allowed to get executed. If it sees a program try to execute some code in the wrong part of memory, it kills the process, which will thwart attacks. So if a target using a grsec-patched kernel loads a malicious PDF in a vulnerable PDF reader, and it exploits a bug and tries to jump to shellcode, grsec will kill the process and log this to /var/log/syslog .

Unfortunately, grsec doesn’t know which programs are supposed to jump into user-loaded memory – it turns out quite a few programs do. For example, web browsers are able to load websites that include JavaScript. In order to improve performance, they convert the JavaScript code to machine code and then jump into it to execute it (this is called Just In Time compilation, or JIT). When grsec notices this, it will kill your web browser process.

You can still run a web browser with grsec, but you need to set a PaX flag (more on this below) to disable memory protections on that specific executable binary. This means your browser won’t immediately crash when you open it, but it also means that if an attacker is able to exploit a bug in it, they may be able to succeed in running their shellcode. Likewise, you often need to disable memory protection on programming language interpreters, such as /usr/bin/python2 , /usr/bin/python3 , and /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java .

So it’s not perfect. But even with disabling memory protections on specific binaries, you still get a lot of protection. Most programs work fine without having to disable any protections, and grsec makes it extremely hard for an attacker to escalate privileges. So if someone does manage to hack your web browser and get a shell, grsec will probably prevent them from getting root. Or if you’re a web developer and are running Apache with an old vulnerable WordPress on it that someone on your wifi notices and hacks, grsec will probably prevent them from accessing any data that isn’t readable from the www-data user, even if they come armed with Linux privilege escalation exploits.

Installing Debian sid

Debian doesn’t release sid installation images. Instead you need to install an earlier version of Debian and upgrade to sid. So go and grab the latest stable netinst iso, probably the amd64 one.

You also may want to verify the signature of the iso. As of this writing, the latest stable is 8.2.0, and the iso URL is http://cdimage.debian.org/debian-cd/8.2.0/amd64/iso-cd/debian-8.2.0-amd64-netinst.iso . If you drop the filename part of the path and just load http://cdimage.debian.org/debian-cd/8.2.0/amd64/iso-cd/ , you’ll see several files to downloading, including SHA512 and SHA512.sign . Download both of those. The SHA512 file contains SHA512 checksums of all of the isos for that version of Debian, including debian-8.2.0-amd64-netinst.iso .

Make sure you have the key that Debian uses to sign their releases. You can find its fingerprint here. For Debian 8, the signing key fingerprint is DF9B 9C49 EAA9 2984 3258 9D76 DA87 E80D 6294 BE9B (as of the time of writing), and you can get the key by running this:

$ gpg --recv-keys DF9B9C49EAA9298432589D76DA87E80D6294BE9B

Now you can verify the signature:

$ gpg --verify SHA512SUMS.sign gpg: assuming signed data in 'SHA512SUMS' gpg: Signature made Fri 11 Sep 2015 08:13:34 AM PDT using RSA key ID 6294BE9B gpg: Good signature from "Debian CD signing key <debian-cd@lists.debian.org>" [unknown] gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: DF9B 9C49 EAA9 2984 3258 9D76 DA87 E80D 6294 BE9B

Make sure it says Good signature from "Debian CD signing key <debian-cd@lists.debian.org>" . Great! Now, take a SHA512 checksum of the iso (note that your checksum might be different, if you’ve downloaded a newer version of Debian than I did):

$ sha512sum debian-8.2.0-amd64-netinst.iso 923cd1bfbfa62d78aecaa92d919ee54a95c8fca834b427502847228cf06155e7243875f59279b0bf6bfd1b579cbe2f1bc80528a265dafddee9a9d2a197ef3806 debian-8.2.0-amd64-netinst.iso

Now let’s check to see if that checksum is in the SHA512 file.

$ cat SHA512SUMS | grep 923cd1bfbfa62d78aecaa92d919ee54a95c8fca834b427502847228cf06155e7243875f59279b0bf6bfd1b579cbe2f1bc80528a265dafddee9a9d2a197ef3806 923cd1bfbfa62d78aecaa92d919ee54a95c8fca834b427502847228cf06155e7243875f59279b0bf6bfd1b579cbe2f1bc80528a265dafddee9a9d2a197ef3806 debian-8.2.0-amd64-netinst.iso

Great, the signed SHA512 file shows the same checksum for debian-8.2.0-amd64-netinst.iso that I got manually. This means that I’ve confirmed that the Debian netinst iso I downloaded hasn’t been tampered with.

Finally, burn the iso to a CD or dd it to a USB stick, boot to it, and install Debian.

Note that when it asks for you to come up with a root password, you can leave it blank. If you do this, then Debian will make your user a sudoer. This might be more familiar to you if you’re used to Ubuntu or Mac OS X.

When you get to the “Partition disks” screen, choose “Guided – use entire disk and set up encrypted LVM” in order to set up full disk encryption. You’ll need to come up with a strong passphrase.

When you get to the “Software selection” screen, make sure to select GNOME as your desktop environment. That’s what I’m using, and this tutorial includes setting a PaX flag to make GNOME work.

Reboot into your freshly installed Debian, mount your encrypted hard drive, and login. Now it’s time to upgrade from stable to sid. Open a terminal and edit the /etc/apt/source.list file as root.

It starts out looking like this:

# # deb cdrom:[Debian GNU/Linux 8.2.0 _Jessie_ - Official amd64 NETINST Binary-1 20150906-11:09]/ jessie main #deb cdrom:[Debian GNU/Linux 8.2.0 _Jessie_ - Official amd64 NETINST Binary-1 20150906-11:09]/ jessie main deb http://ftp.us.debian.org/debian/ jessie main deb-src http://ftp.us.debian.org/debian/ jessie main deb http://security.debian.org/ jessie/updates main deb-src http://security.debian.org/ jessie/updates main # jessie-updates, previously known as 'volatile' deb http://ftp.us.debian.org/debian/ jessie-updates main deb-src http://ftp.us.debian.org/debian/ jessie-updates main

We can delete the cdrom comments, and also we can delete the security and updates repositories. Since we’re updating to sid, we get all of the latest versions of all packages, which already includes security updates. And finally, change the “jessie” to “sid”, so that the result file looks like this:

deb http://ftp.us.debian.org/debian/ sid main deb-src http://ftp.us.debian.org/debian/ sid main

Save and exit. Now, update all of the software. This will take awhile. And finally, reboot into sid.

$ sudo apt-get update $ sudo apt-get dist-upgrade $ sudo apt-get autoremove $ sudo reboot

Compiling the grsec-patched Linux kernel

The Debian wiki’s grsecurity page implies that linux-grsec is packaged in sid already, but it doesn’t seem available yet. In the future this will be much simpler, but for now we can compile the Linux kernel ourselves.

Update: Grsecurity is indeed packaged in sid. The package name (right now) is linux-image-4.3.0-1-grsec-amd64 , so you can apt-get install that instead of compiling it yourself if you want.

Start by installing dependencies:

$ sudo apt-get install libncurses5-dev build-essential fakeroot kernel-package gcc-5 gcc-5-plugin-dev make

Download the public keys that are used to sign the Linux kernel source as well as the grsecurity patch. You can find the latest Linux kernel signing key here, and you you can find the latest grsecurity signing key at the bottom of this page.

At the time of writing, here’s how to download the two signing keys:

$ gpg --recv-keys 647F28654894E3BD457199BE38DBBDC86092693E $ gpg --recv-keys DE9452CE46F42094907F108B44D1C0F82525FE49

Now download the Linux source code and the grsecurity patch. You’re going to need to download the latest stable version (not any of the longterm versions), because the grsecurity project only releases that version publicly. Because these versions numbers change constantly, I’m not going to provide direct links.

Download the latest stable Linux source, as well as the associated PGP signature file, from kernel.org. The files I’m downloading are called linux-4.3.3.tar.xz and linux-4.3.3.tar.sign .

Download the corresponding grsecurity patch, as well as the associated PGP signature file, from grsecurity.net. This will be under the header “test kernel patch”. The files I’m downloading are called grsecurity-3.1-4.3.3-201601051958.patch and grsecurity-3.1-4.3.3-201601051958.patch.sig .

If you’re following along with a version of Linux that’s newer than 4.3.3, type the following instructions yourself instead of copying and pasting, so that the commands correspond with the version of Linux and grsec that you downloaded.

Decompress the Linux source code.

$ unxz linux-4.3.3.tar.xz

Verify the signatures. The output of both of these should say “Good signature”.

$ gpg --verify linux-4.3.3.tar.sign $ gpg --verify grsecurity-3.1-4.3.3-201601051958.patch.sig

If all went well, extract the Linux source code and then apply the grsec patch.

$ tar -xf linux-4.3.3.tar $ cd linux-4.3.3/ $ patch -p1 < ../grsecurity-3.1-4.3.3-201601051958.patch

Start with your existing Linux kernel configuration. Note that it’s possible that your config file might be named differently if you currently have a different version of the Linux kernel installed, so make sure you use the right filename.

$ cp /boot/config-4.3.0-1-amd64 .config $ make menuconfig

Navigate to Security options, Grsecurity, and make sure it’s enabled. Then go to Configuration Method and set it to Automatic. Go to Usage Type and change it to Desktop. Go to Virtualization Type and set it to None (unless you’re testing this in a VM, then make sure you choose the right hypervisor). Go to Required Properties and set it to Security .

Now save and exit.

You can use all of your CPU cores to make compiling the Linux kernel faster by running this.

$ export CONCURRENCY_LEVEL="$(grep -c '^processor' /proc/cpuinfo)"

Now compile the kernel. This will probably take a really long time. When you’re done you’ll have a deb file you can install.

$ fakeroot make-kpkg --initrd kernel_image

When this finishes, install the new Linux kernel image.

$ cd .. $ sudo dpkg -i linux-image-4.3.3-grsec_4.3.3-grsec-10.00.Custom_amd64.deb

Now let’s make sure you GRUB boots into this kernel image by default. Figure out what the kernel version string that GRUB will use is called.

$ grep menuentry /boot/grub/grub.cfg | cut -d "'" -f2 | grep "grsec$" Debian GNU/Linux, with Linux 4.3.3-grsec

Now edit /etc/default/grub and change the GRUB_DEFAULT= line be to "Advanced options for Debian GNU\/Linux> followed by the kernel version string. Put the whole things in quotes, like this: GRUB_DEFAULT="Advanced options for Debian GNU\/Linux>Debian GNU/Linux, with Linux 4.3.3-grsec"

Save and exit, and then upgrade grub.

$ sudo update-grub

Before rebooting into the grsec kernel, we need to install PaX tools, which will let us disable memory protections on specific userland binaries.

$ sudo apt-get install paxtest paxctl $ sudo reboot

When you boot into the new kernel, everything will be broken.

This is because grsec is killing the gnome-shell process. In order to make your desktop environment work, you need to disable memory protections on the gnome-shell binary.

Press Ctrl-Alt F1 to switch to tty1 and login without graphics. Then disable memory protections on /usr/bin/gnome-shell using paxctl (more on how this tool works below).

Here’s the command to run.

$ sudo paxctl -cm /usr/bin/gnome-shell

Reboot again, and this time GNOME should work.

$ sudo reboot

Login to your account, and open a terminal to do a few final steps.

Edit /etc/sysctl.conf (as root) and add these lines:

kernel.grsecurity.rwxmap_logging = 0 kernel.grsecurity.grsec_lock = 1

And activate them.

$ sudo sysctl -p

Now, make sure that grsecurity is really working like it should.

$ paxtest blackhat PaXtest - Copyright(c) 2003,2004 by Peter Busser Released under the GNU Public Licence version 2 or later Writing output to /home/micah/paxtest.log It may take a while for the tests to complete Test results: PaXtest - Copyright(c) 2003,2004 by Peter Busser Released under the GNU Public Licence version 2 or later Mode: Blackhat Linux rey 4.3.3-grsec #3 SMP Mon Jan 11 12:42:40 PST 2016 x86_64 GNU/Linux Executable anonymous mapping : Killed Executable bss : Killed Executable data : Killed Executable heap : Killed Executable stack : Killed Executable shared library bss : Killed Executable shared library data : Killed Executable anonymous mapping (mprotect) : Killed Executable bss (mprotect) : Killed Executable data (mprotect) : Killed Executable heap (mprotect) : Killed Executable stack (mprotect) : Killed Executable shared library bss (mprotect) : Killed Executable shared library data (mprotect): Killed Writable text segments : Killed Anonymous mapping randomisation test : 28 bits (guessed) Heap randomisation test (ET_EXEC) : 23 bits (guessed) Heap randomisation test (PIE) : 35 bits (guessed) Main executable randomisation (ET_EXEC) : 28 bits (guessed) Main executable randomisation (PIE) : 28 bits (guessed) Shared library randomisation test : 28 bits (guessed) Stack randomisation test (SEGMEXEC) : 35 bits (guessed) Stack randomisation test (PAGEEXEC) : 35 bits (guessed) Arg/env randomisation test (SEGMEXEC) : 39 bits (guessed) Arg/env randomisation test (PAGEEXEC) : 39 bits (guessed) Randomization under memory exhaustion @~0: 29 bits (guessed) Randomization under memory exhaustion @0 : 28 bits (guessed) Return to function (strcpy) : paxtest: return address contains a NULL byte. Return to function (memcpy) : Killed Return to function (strcpy, PIE) : paxtest: return address contains a NULL byte. Return to function (memcpy, PIE) : Killed

Congratulations, you’re running Debian with a hardened grsecurity kernel! But you’re not done yet. Lots of your software is going to crash as soon as your run it. The next section shows you how to fix that.

Thanks to the Kevin Gallagher and the Freedom of the Press Foundation staff for making a great guide that I mostly based this section off of.

Setting PaX flags so you can actually use your computer

Grsecurity automatically kills processes that use memory in a way that could be an attack. But many programs legitimately use memory this way. In order to use those programs, you need to set PaX flags on their binaries — these flags actually get stored in the binary file’s header. paxctl is a program that lets you view and set these flags.

$ sudo paxctl -h PaX control v0.9 Copyright 2004,2005,2006,2007,2009,2010,2011,2012,2014 PaX Team usage: paxctl options: -p: disable PAGEEXEC -P: enable PAGEEXEC -e: disable EMUTRAMP -E: enable EMUTRAMP -m: disable MPROTECT -M: enable MPROTECT -r: disable RANDMMAP -R: enable RANDMMAP -x: disable RANDEXEC -X: enable RANDEXEC -s: disable SEGMEXEC -S: enable SEGMEXEC -v: view flags -z: restore default flags -q: suppress error messages -Q: report flags in short format -c: convert PT_GNU_STACK into PT_PAX_FLAGS (see manpage!) -C: create PT_PAX_FLAGS (see manpage!)

Earlier we ran paxctl -cm /usr/bin/gnome-shell . The -c modified the binary so that it’s able to accept PaX flags, and the -m disabled MPROTECT — basically meaning that that specific binary is allowed to do stuff that grsec would kill other binaries it caught doing. You can view the flags that are set like this.

$ sudo paxctl -v /usr/bin/gnome-shell PaX control v0.9 Copyright 2004,2005,2006,2007,2009,2010,2011,2012,2014 PaX Team - PaX flags: -----m-x-e-- [/usr/bin/gnome-shell] MPROTECT is disabled RANDEXEC is disabled EMUTRAMP is disabled

For the most part, all you’ll ever need to do is use paxctl -cm on a binary that you trust to run without MPROTECT . You can always run paxctl -M to enable MPROTECT on it again.

There’s also a cool looking project called paxrat, developed by Subgraph, that helps you manage PaX flags. It’s not yet packaged for Debian, and I haven’t yet tested it out, but I look forward to playing with it. For now we can just use paxctl.

To start, let’s set PaX flags for some GRUB binaries.

$ sudo paxctl -cpm /usr/sbin/grub-probe $ sudo paxctl -cpm /usr/sbin/grub-mkdevicemap $ sudo paxctl -cpm /usr/sbin/grub-install $ sudo paxctl -cpm /usr/bin/grub-script-check $ sudo paxctl -cpm /usr/bin/grub-mount

And python interpreters.

$ sudo paxctl -cm /usr/bin/python2 $ sudo paxctl -cm /usr/bin/python3

One issue I’ve run into when trying to set PaX flags with paxctl is an error message that says “Text file busy”. If you run into this issue, you can kill any processes of that binary that are running and then try again. If you have a hard time keeping a process killed, you might try rebooting into recovery mode to set its PaX flags.

How do you know what binaries you should set PaX flags on? Grsec logs its error messages to /var/log/syslog , so let’s start by opening a new terminal and watching all of the grsec error messages that scroll by.

$ sudo tail -f /var/log/syslog | grep grsec

Now, in a new terminal window, try opening Iceweasel by running its binary.

$ iceweasel Gtk-Message: Failed to load module "canberra-gtk-module" Killed

Iceweasel was killed by grsec, and here’s the error message that was logged to /var/log/syslog .

Jan 11 18:18:25 debian kernel: [ 286.128581] grsec: denied resource by requesting 4096 for RLIMIT_CORE against limit 0 for /usr/lib/iceweasel/iceweasel[iceweasel:1991] uuid/euid:1000/1000 gid:egid:1000/1000, parent /bin/bash[1417] uid/euid:1000/1000 gid/egid:1000/1000

The important bits there are grsec: denied resource and /usr/lib/iceweasel/iceweasel . (Note that /usr/bin/iceweasel is a symlink to /usr/lib/iceweasel/iceweasel , so this is in fact the Iceweasel binary.)

If you want to use Iceweasel (and risk turning off its memory protections), then use paxctl to disable MPROTECT .

$ sudo paxctl -cm /usr/lib/iceweasel/iceweasel

Now trying running iceweasel again. It should just open this time!

Now, try opening a new tab. My syslog window throws two more errors.

Jan 11 18:26:48 debian kernel: [ 788.651798] grsec: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /usr/lib/iceweasel/plugin-container[Web Content:2100] uid/euid:1000/1000 gid/egid:1000/1000, parent /usr/lib/iceweasel/iceweasel[Gecko_IOThread:2041] uid/euid:1000/1000 gid/egid:1000/1000 Jan 11 18:26:48 debian kernel: [ 788.818013] grsec: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /usr/lib/iceweasel/plugin-container[Web Content:2114] uid/euid:1000/1000 gid/egid:1000/1000, parent /usr/lib/iceweasel/iceweasel[Gecko_IOThread:2041] uid/euid:1000/1000 gid/egid:1000/1000

It looks like each time you open a tab, Iceweasel tries running /usr/lib/iceweasel/plugin-container in a subprocess, and grsec kills that process. However, Iceweasel appears to work fine without it, so no big deal. Since it hasn’t affected me yet, I’m going to keep MPROTECT enabled on that binary. If you need to run a plugin in Iceweasel and grsec prevents that from working, then you might consider disabling MPROTECT .

Let’s try another: Tor Browser. The easiest way to install Tor Browser is to use Tor Browser Launcher (a piece of software I wrote… ehem.)

Edit /etc/apt/sources.list as root, add the “contrib” repository.

deb http://ftp.us.debian.org/debian/ sid main contrib deb-src http://ftp.us.debian.org/debian/ sid main contrib

Now install Tor Browser Launcher.

$ sudo apt-get update $ sudo apt-get install torbrowser-launcher

Now try launching Tor Browser.

$ torbrowser-launcher

It should pop up a window and download the Tor Browser tarball for the first time. It should also download its PGP signature, verify the signature, extract the tarball, and then try to launch Tor Browser. But when it does, grsec kills the process.

Jan 11 18:38:24 debian kernel: [ 1485.423982] grsec: denied resource overstep by requesting 4096 for RLIMIT_CORE against limit 0 for /home/micah/.local/share/torbrowser/tbb/x86_64/tor-browser_en-US/Browser/firefox[firefox:3954] uid/euid:1000/1000 gid/egid:1000/1000, parent /lib/systemd/systemd[systemd:1] uid/euid:0/0 gid/egid:0/0

Tor Browser is trying to run the /home/micah/.local/share/torbrowser/tbb/x86_64/tor-browser_en-US/Browser/firefox binary, and it fails for the exact same reason that Iceweasel fails. To fix it, disable MPROTECT on that binary.

$ sudo paxctl -cm /home/micah/.local/share/torbrowser/tbb/x86_64/tor-browser_en-US/Browser/firefox $ torbrowser-launcher

This time, Tor Browser will launch without any crashes or grsec errors.

How does Debian with grsecurity compare to Qubes?

I’ve written in the past about how awesome Qubes is, which is also a reasonably secure operating system. So how does it compare to Debian with a grsec kernel?

In Qubes, you separate your computer into different security domains (also known as AppVMs), but in each of these domains you’re basically running a normal, non-hardened OS. If an attacker hacks you, they only have access to data in a single security domain, and the rest of your data remains safe. In Debian/grsec — like in every non-Qubes operating system that’s currently available, including Windows, Mac OS X, Ubuntu, Arch, etc. — you only have one security domain, so if an attacker hacks you it’s basically game over. But grsec makes it harder for an attacker to successfully hack you. (I’m looking forward to the release Subgraph OS, which will be the second desktop operating system designed for compartmentalization, and which will be based on Debian, grsec, and Linux containers.)

For example, let’s say an attacker sends you a malicious PDF that secretly exploits a bug in Evince, a popular Linux PDF viewer. Pretend you’re running Qubes and decide to open this PDF in your “personal” security domain that you use for non-work related stuff (really you should open it in a disposable VM, but maybe you didn’t think it was that risky or something). The PDF exploits the bug in Evince and jumps into the attacker’s shellcode, and now the attacker can run whatever commands they want and access any of your data, but only within your “personal” AppVM. They can’t access any of the files in your “work” domain, or your “email” domain, or the networkless “vault” domain that you use to store your PGP secret keys and password databases.

Now pretend you’re running Debian/grsec. You open the PDF, it exploits the bug in Evince and tries to jump to the attacker’s shellcode, but grsec detects that your Evince process is trying to execute code in an off-limits part of memory and kills the process. Evince immediately closes, grsec logs an error to syslog, and you don’t get hacked at all. In this case, grsec completely prevented the attack, while Qubes only contained it.

But now let’s change the example. Instead of Evince, the attacker is exploiting a bug in Iceweasel. If you’re using Qubes, the attacker only hacks the domain that you’re running Iceweasel in. If you’re using Debian/grsec (and you disabled MPROTECT on /usr/lib/iceweasel/iceweasel like we did above), the attacker’s shellcode will run and they’ll have access to all of your files, be able to log your keystrokes, etc. They might have a very difficult time getting root, which will make it harder for them to install a persistent backdoor, but that’s little consolation.

Hardening your kernel with grsecurity is great because it makes it much harder for attacks against you to succeed, but it still doesn’t make it so you can compartmentalize your computer the way that Qubes does. It’s also important to remember that grsec mostly protects you against memory corruption bugs, but other types of security issues exist too. You might run an ssh server and use a crappy password. You might get tricked into copying and pasting something malicious into your terminal. You might be running buggy software that can be exploited without jumping into forbidden memory space, so grsec won’t catch it.

There’s another big difference between Qubes and Debian/grsec: Usability.

Qubes isn’t terribly hard to use (for experienced Linux nerds), but it does require shifting the paradigm about how you think about operating systems, and working around software that isn’t designed for it. For example, it’s not uncommon to have four or five different web browsers open at the same time, all running in different security domains for different purposes. If someone sends you a link, you can’t just click it. You have to decide which domain you’d like to open it in first, and then copy it into that domain’s clipboard before pasting it into a browser. There’s also still a lot of work to be done to make USB devices usable in Qubes. If you want to do some Android development, it’s not the simplest task yet to get Android Studio installed and communicating with your phone over a USB port in an AppVM. Same with video chat. And same with figuring out a workflow to take a screenshot, crop it, and then tweet it. Many tasks are complicated in Qubes simply because you have to deal with compartmentalization and the limitations of virtualizing all of your software — but this is also what gives Qubes its strength.

Debian with grsec doesn’t require this paradigm shift. Once you get past the initial PaX learning curve and make sure that the software you use most often all works, then it’s just like using any other operating system — but you get the invisible-but-awesome benefits of having a hardened kernel. USB devices, webcams, screenshots, and everything else work just like you’re used to. So if you’re not quite ready for Qubes, or if you can’t run Qubes for your specific work for whatever reason, Debian/grsec might be a good choice for you.

The best of both worlds would be running Qubes, but with AppVMs that have grsecurity-patched Linux kernels. This is entirely possible, but no one has yet succeeded in doing it.