codeblog code is freedom — patching my itch

May 14, 2009

partial NX emulation in Ubuntu

Filed under: Security,Ubuntu — kees @ 9:18 am

Now available for early testing, Ubuntu Karmic Alpha 1 has a kernel feature I’ve long wanted in Ubuntu: NX emulation. Basically, if your hardware lacks NX support, the kernel will emulate the feature using memory segment limits and ordering. This was AFAIBT originally developed by PaX, and a similar version (with histories including work by Solar Designer and maybe OpenBSD?) has been carried in RedHat/Fedora for a while now (under the larger project called “ExecShield”, covering multiple protection technologies).

As more and more of the monolithic ExecShield kernel patch has been taken upstream (many thanks to Arjan van de Ven for pushing them), the patch in RedHat has been shrinking. Recently, Dave Jones split up the remaining pieces into logical chunks small enough that I could actually read it without going cross-eyed. From this, I ported the nx-emulation patches to Ubuntu’s kernel, and now they’re happily live in Karmic.

So, instead of this:

$ ./vulnerable-setuid-program $OVERFLOW_AND_SHELLCODE
# id
uid=0(root) gid=0(root) groups=0(root)

We get this:

$ ./vulnerable-setuid-program $OVERFLOW_AND_SHELLCODE
Segmentation fault (core dumped)
$ dmesg | tail -n1
[170131.763976] vulnerable-set[16278]: general protection ip:80489c5 sp:bfa3e330 error:0 in vulnerable-setuid-program[8048000+1000]

Though, as always, please just use 64bit instead. :)

Update: gave credit to PaX, thanks for the corrections!

© 2009, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License


  1. For clarification: if I run 32bit Jaunty on a 64bit capable processor, is there any kind of NX protection (hardware or software) in use then? Or do I need to run a 64bit installation to get that?

    Comment by oliver — May 14, 2009 @ 9:33 am

  2. Prior to Karmic, if you run 32bit kernels on 64bit machines, you do not get the NX protections. You must be running in “PAE” mode (prior to Karmic only available in the -server 32bit kernel flavor). PAE mode is automatic in 64bit kernels.

    Note that even with PAE (or 64bit), you still need NX support in the CPU (some BIOSes turn it off).

    Comment by kees — May 14, 2009 @ 9:47 am

  3. So can you push these into Debian too?

    Comment by foo — May 14, 2009 @ 8:35 pm

  4. That would be up to Debian — the port I did is against a mostly stock upstream Linux kernel. There’s no reason it couldn’t be carried, but that’s up to the Debian kernel maintainers.

    Comment by kees — May 14, 2009 @ 9:10 pm

  5. Actually, it was designed by the OpenBSD people as part of the
    W^X protection scheme’s adaption to the i386 (and IIRC macppc)

    Comment by mirabilos — May 15, 2009 @ 2:38 am

  6. Actually, it wasn’t designed by the OpenBSD people; get your facts straight. OpenBSD was *several years* late to the game: PaX was developed in 2000.

    As for “NX emulation”, the provided example is highly misleading in the same way that RedHat has been misleading people for years regarding how accurately exec-shield emulates NX.

    Say vulnerable-setuid-program has a vulnerability in bounds checking on a buffer in the .data or .bss sections of the main executable or any library loaded by the executable. If that executable/library doesn’t happen to be located highest in the address space (and its probability of not being located there is n – 1/ n, where n is the number of loaded executables/libraries in the process’ address-space) then the vulnerability is still perfectly exploitable as it would be without any NX. If you get unlucky and it does happen to be mapped highest, just repeat the exploit until it succeeds.

    But of course, details are unimportant. What matters is claiming you have “NX emulation.”


    Comment by spender — May 15, 2009 @ 10:26 am

  7. Thanks for the corrections and links! Sure, I’m not claiming it’s perfect, I’m just claiming the patch carried by RedHat is finally in Ubuntu now. It’s better than not having any NX, but like I ended the post with, I’d prefer everyone just use 64bit. :)

    Comment by kees — May 15, 2009 @ 11:27 am

  8. Also, to clarify: Solar Designer’s non-executable stack via segmentation came first, then some other misc non-executable stack + heap via segmentation patches (RSX, KNoX), then PaX which was the first exact NX emulation via either segmentation or TLB tricks. It’s all documented on


    Comment by spender — May 15, 2009 @ 11:58 am

  9. BTW, yeah, I can totally reproduce NX-emu-dodging via library bss or via PIE program bss and brk regions. That’s unfortunate. :( Thanks for the heads-up!

    Comment by kees — May 15, 2009 @ 1:23 pm

  10. brk too even? That shouldn’t happen. Do you build your kernels with COMPAT_BRK=y?


    Comment by spender — May 15, 2009 @ 1:39 pm

  11. brk only when the exec is PIE and doesn’t get mapped at the top of the executable region limit. I assume this is a bug in the vma randomization side of the patch. I’m gonna go switch to email…

    Comment by kees — May 15, 2009 @ 1:53 pm

Leave a Reply

Your email address will not be published. Required fields are marked *

Powered by WordPress