Handy command line arguments for gpg:
gpg --list-options show-photos --fingerprint 0xdc6dc026
This is nice to examine someone’s PGP photo. You can also include it in --verify-options, depending on how/when you want to see the photo (for example, when doing key signings).
If gpg doesn’t pick the right photo viewer, you can override it with --photo-viewer 'eog %I' or similar.
© 2011, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
This month, I will have been with Canonical for 5 years. It’s been fantastic, but I’ve decided to move on. Next week, I’m going to start working for Google, helping out with ChromeOS, which I’m pretty excited about. I’m sad to be leaving Canonical, but I comfort myself by knowing that I’m not leaving Ubuntu or any other projects I’m involved in. I believe in Ubuntu, I use it everywhere, and I’m friends with so many of its people. And I’m still core-dev, so I’ll continue to break^Wsecure things as much as I can in Ubuntu, and continue working on getting similar stuff into Debian. :)
For nostalgic purposes, I dug up my first security update (sponsored by pitti), and my first Ubuntu Security Notice. I’m proud of Ubuntu’s strong security record and how far the security feature list has come. The Ubuntu Security Team is an awesome group of people, and I’m honored to have worked with them.
I’m looking forward to the new adventures, but I will miss the previous ones.
© 2011, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
Picked up a Doctor Who comic today and saw a nice hat-tip to (or composite ship design plagiarism of) Aliens.
The Colonial Marines ship “Sulaco”, from Aliens, 1986:
The Scavengers ship, from the Doctor Who Spam Filtered story, 2011:
Such a great ship. Not even remotely made to look aerodynamic.
And to make this almost related to Ubuntu and Debian, here was my command line to remove exif data from the image I took with my phone:
mogrify -strip spam-filtered.jpg
© 2011 – 2015, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
The Linux kernel attempts to protect portions of its memory from unexpected modification (through potential future exploits) by setting areas read-only where the compiler has allowed it (CONFIG_DEBUG_RODATA). This, combined with marking function pointer tables “const”, reduces the number of easily writable kernel memory targets for attackers.
However, modules (which are almost the bulk of kernel code) were not handled, and remained read-write, regardless of compiler markings. In 2.6.38, thanks to the efforts of many people (especially Siarhei Liakh and Matthieu Castet), CONFIG_DEBUG_SET_MODULE_RONX was created (and CONFIG_DEBUG_RODATA expanded).
To visualize the effects, I patched Arjan van de Ven’s arch/x86/mm/dump_pagetables.c to be a loadable module so I could look at /sys/kernel/debug/kernel_page_tables without needing to rebuild my kernel with CONFIG_X86_PTDUMP.
Comparing Lucid (2.6.32), Maverick (2.6.35), and Natty (2.6.38), it’s clear to see the effects of the RO/NX improvements, especially in the “Modules” section which has no NX markings at all before 2.6.38:
lucid-amd64# awk '/Modules/,/End Modules/' /sys/kernel/debug/kernel_page_tables | grep NX | wc -l 0 maverick-amd64# awk '/Modules/,/End Modules/' /sys/kernel/debug/kernel_page_tables | grep NX | wc -l 0 natty-amd64# awk '/Modules/,/End Modules/' /sys/kernel/debug/kernel_page_tables | grep NX | wc -l 76
2.6.38’s memory region is much more granular, since each module has been chopped up for the various segment permissions:
lucid-amd64# awk '/Modules/,/End Modules/' /sys/kernel/debug/kernel_page_tables | wc -l 53 maverick-amd64# awk '/Modules/,/End Modules/' /sys/kernel/debug/kernel_page_tables | wc -l 67 natty-amd64# awk '/Modules/,/End Modules/' /sys/kernel/debug/kernel_page_tables | wc -l 155
For example, here’s the large “sunrpc” module. “RW” is read-write, “ro” is read-only, “x” is executable, and “NX” is non-executable:
maverick-amd64# awk '/^'$(awk '/^sunrpc/ {print $NF}' /proc/modules)'/','!/GLB/' /sys/kernel/debug/kernel_page_tables
0xffffffffa005d000-0xffffffffa0096000 228K RW GLB x pte
0xffffffffa0096000-0xffffffffa0098000 8K pte
natty-amd64# awk '/^'$(awk '/^sunrpc/ {print $NF}' /proc/modules)'/','!/GLB/' /sys/kernel/debug/kernel_page_tables
0xffffffffa005d000-0xffffffffa007a000 116K ro GLB x pte
0xffffffffa007a000-0xffffffffa0083000 36K ro GLB NX pte
0xffffffffa0083000-0xffffffffa0097000 80K RW GLB NX pte
0xffffffffa0097000-0xffffffffa0099000 8K pte
The latter looks a whole lot more like a proper ELF (text segment is read-only and executable, rodata segment is read-only and non-executable, and data segment is read-write and non-executable).
Just another reason to make sure you’re using your CPU’s NX bit (via 64bit or 32bit-PAE kernels)! (And no, PAE is not slower in any meaningful way.)
© 2011, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
I’m once again on the program committee for the Linux Security Summit, so I’d love to see people submit talks, attend, etc. It will be held along with the Linux Plumber’s Conference, on September 8th in Santa Rosa, CA, USA.
I’d really like to see more non-LSM developers and end-users show up for this event. We need people interested in defining threats and designing defenses. There is a lot of work to be done on all kinds of fronts and having people voice their opinions and plans can really help us prioritize the areas that need the most attention.
Here’s one of many archives of the announcement, along with the website. We’ve got just under 2 months to get talks submitted (May 27th deadline), with speaker notification quickly after that on June 1st.
Come help us make Linux more secure! :)
© 2011, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
In Ubuntu, the use of ptrace is restricted. The default allowed relationship between the debugger and the debuggee is that parents are allowed to ptrace their descendants. This means that running “gdb /some/program” and “strace /some/program” Just Works. Using gdb‘s “attach” and strace‘s “-p” options need CAP_SYS_PTRACE, care of sudo.
The next most common use-case was that of crash handlers needing to do a live ptrace of a crashing program (in the rare case of Apport being insufficient). For example, KDE applications have a segfault handler that calls out to kdeinit and requests that the crash handling process be started on it, and then sits in a loop waiting to be attached to. While kdeinit is the parent of both the crashing program (debuggee) and the crash handling program (debugger), the debugger cannot attach to the debugee since they are siblings, not parent/descendant. To solve this, a prctl() call was added so that the debugee could declare who’s descendants were going to attach to it. KDE patched their segfault handler to make the prctl() and everything Just Works again.
Breakpad, the crash handler for Firefox and Chromium, was updated to do effectively the same thing, though they had to add code to pass the process id back to the debuggee since they didn’t have it handy like KDE.
Another use-case was Wine, where for emulation to work correctly, they needed to allow all Wine processes to ptrace each other to correctly emulate Windows. For this, they just declared that all descendants of the wine-server could debug a given Wine process, there-by confining their ptrace festival to just Wine programs.
One of the remaining use-cases is that of a debugging IDE that doesn’t directly use ptrace itself. For example, qtcreator will launch a program and then later attach to it by launching gdb and using the “attach” command. This looks a lot like the crash handler use-case, except that the debuggee doesn’t have any idea that it is running under an IDE. A simple solution for this is to have the IDE run its programs with the LD_PRELOAD environment variable aimed at a short library that just calls prctl() with the parent process id, and suddenly the IDE and its descendants (i.e. gdb) can debug the program all day long.
I’ve got an example of this preloadable library written. If it turns out this is generally useful for IDEs, I could package it up like fakeroot and faketime.
© 2011, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
Other people have taken notice of the recent “auto-run” attack research against Linux. I was extremely excited to see Jon Larimer publishing this stuff, since it ultimately did not start with the words, “first we disabled NX, ASLR, and (SELinux|AppArmor) …”
I was pretty disappointed with last year’s Blackhat conference because so many of the presentations just rehashed ancient exploitation techniques, and very few actually showed new ideas. I got tired of seeing mitigation technologies disabled to accomplish an attack. That’s kind of not the point.
Anyway, Jon’s research is a step in the right direction. He defeats ASLR via brute-force, side-steps NX with ret-to-libc, and finds policy holes in AppArmor to accomplish the goal. I was pleased to see “protected by PIE and AppArmor” in his slides — Ubuntu’s hardening of evince was very intentional. It has proven to be a dangerous piece of software, which Jon’s research just further reinforces. He chose to attack the difficult target instead of going after what might have been the easier thumbnailers.
So, because of this research, we can take a step back and think about what could be done to improve the situation from a proactive security perspective. A few things stand out:
Trying to brute-force operational ASLR on a 64bit system, though, would probably not have worked. So, again, I stand by my main recommendation for security: use 64bit. :)
Good stuff; thanks Jon!
© 2011, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
So, I was testing a (closed source) single-player offline game recently and thought this exercise might be fun to document. I didn’t want to spend any time actually earning in-game money since I’d played it before and I wanted to just skip ahead to other aspects of the game. I was curious how straight-forward adjusting my cash might be. So, noting the in-game “bank account number” of 219393 and account balance of 3000, I dived right in.
First up, what’s the memory layout of the heap look like? I looked at the brk and the mmap regions without a mapped library or file, marked with “w” in the permissions column, from /proc/PID/maps:
0827e000-08282000 rw-p 00000000 00:00 0
0a22e000–0b08a000 rw-p 00000000 00:00 0 [heap]
efa59000-efd00000 rw-p 00000000 00:00 0
efd00000-efd21000 rw-p 00000000 00:00 0
Knowing these, I could use gdb’s “find” command, after attaching to the process:
$ gdb /some/cool/game
…
(gdb) attach PID
…
(gdb) find /w 0x0827e000, 0x08282000, 219393
(gdb) find /w 0x0a22e000, 0x0b08a000, 219393
0xaf03d08
0xaf06ca8
No hits in the first region, but I see two hits for the account number value in the second region. Let’s start there and see what’s near them…
(gdb) x/8x 0xaf03d08
0xaf03d08: 0x00035901 0x00000000 0x00000000 0x0af06ce0
0xaf03d18: 0x0af06be0 0x00000059 0x0af03d98 0x0af041e8
(gdb) x/8x 0xaf06ca8
0xaf06ca8: 0x00035901 0x00000bb8 0x00000bb8 0x0820b148
0xaf06cb8: 0x00000001 0x00000000 0x00000000 0x00000000
In that second hit, I see the value 0xBB8, which is 3000, and matches our account balance. Let’s see what happens if we just change both of those to add a bit a few orders of magnitude above the current value…
(gdb) set var *0xaf06cac = 0x00100bb8
(gdb) set var *0xaf06cb0 = 0x00100bb8
(gdb) x/32x 0xaf06cac
0xaf06cac: 0x00100bb8 0x00100bb8 0x0820b148 0x00000001
(gdb) continue
And presto, clicking on the bank account details in-game shows a huge account balance of 1051576 now. No need to reverse-engineer any saved games, whew.
© 2011, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
Recently gcc (4.5) improved its ability to see the size of various structures. As a result, the FORTIFY protections have suddenly gotten a bit stricter. In the past, you used to be able to do things like this:
struct thingy {
int magic;
char data[4];
}
void work(char *input) {
char buffer[1000];
int length;
struct thingy *header;
header = (struct thingy *)buffer;
length = strlen(input);
if (length > sizeof(buffer) - sizeof(*header) - 1) abort();
strcpy(header->data, input);
header->magic = 42;
do_something_fun(header);
}
The problem here is that gcc thinks that header->data is only 4 bytes long. But gcc doesn’t know we intentionally overruled this (and even did length checking), so due to -D_FORTIFY_SOURCE=2, the strcpy() checks kick in when input is more than 4 bytes.
The fix, in this case, is to use memcpy() instead, since we actually know how long our destination is, we can replace the strcpy(...) line with:
memcpy(header->data, input, length + 1); /* take 0-term too */
This kind of header and then data stuff is common for protocol handlers. So far, things like Wine, TFTP, and others have been experiencing problems with the change. Please keep an eye out for it when doing testing.
© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
Want to use a network tarpit? It’s so easy to set up! Thanks to jpds for this whole post. :)
sudo module-assistant auto-install xtables-addons-source sudo iptables -p tcp ... -j TARPIT
Though no such thing exists for IPv6 yet.
Here it is watching over the SSH port:
iptables -N INGRESS-SSH iptables -A INPUT -p tcp --dport 22 -m state --state NEW -j INGRESS-SSH iptables -A INGRESS-SSH -p tcp --dport 22 -m state --state NEW -m recent --name SSH --set iptables -A INGRESS-SSH -p tcp --dport 22 -m state --state NEW -m recent --name SSH --update --rttl --seconds 60 --hitcount 4 -j LOG --log-prefix "[INGRESS SSH TARPIT] " iptables -A INGRESS-SSH -p tcp --dport 22 -m state --state NEW -m recent --name SSH --rcheck --rttl --seconds 60 --hitcount 4 -j TARPIT
© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
Security is more than bug fixing. Security fixing/updating, the thing most people are exposed to, is “reactive security”. However, a large area of security work is “proactive” where defensive abilities are put in place to try and catch problems before they happen, or make classes of vulnerabilities unexploitable. This kind of security is what a lot of people don’t understand, and I think it’s important to point out so the distinction can be clearly seen.
In the Linux kernel, there’s yet another distinction: userspace proactive security and kernel proactive security. Most of the effort in kernel code has been protecting userspace from itself (things like Address Space Layout Randomization), but less attention has been given to protecting the kernel from userspace (currently if a serious enough flaw is found in the kernel, it is usually very easy to exploit it).
One project has taken great strides with proactive security for the Linux kernel: PaX and grsecurity. There hasn’t been a concerted effort to get its pieces upstream and it’s long overdue. People are starting to take proactive kernel security more seriously, though there is still plenty of debate.
While I did my best to push some userspace protections upstream earlier in the year, now it’s time for kernel protections. What to help? Here is the initial list of things to do.
Dan Rosenberg has started the information leaks discussion, and I’ve started the read-only memory discussion. Hopefully this will go somewhere good.
© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
If you’re running a 64bit system, and you’ve got users with access to a video device (/dev/video*), then be sure you update your kernels for CVE-2010-2963. I’ve been slowly making my way through auditing the many uses in the Linux kernel of the copy_from_user() function, and ran into this vulnerability.
Here’s the kernel code from drivers/media/video/v4l2-compat-ioctl32.c:
static int get_microcode32(struct video_code *kp, struct video_code32 __user *up)
{
if (!access_ok(VERIFY_READ, up, sizeof(struct video_code32)) ||
copy_from_user(kp->loadwhat, up->loadwhat, sizeof(up->loadwhat)) ||
get_user(kp->datasize, &up->datasize) ||
copy_from_user(kp->data, up->data, up->datasize))
return -EFAULT;
return 0;
}
Note that kp->data is being used as the target for up->data in the final copy_from_user() without actually verifying that kp->data is pointing anywhere safe. Here’s the caller of get_microcode32:
static long do_video_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
{
union {
struct video_tuner vt;
struct video_code vc;
...
} karg;
void __user *up = compat_ptr(arg);
...
switch (cmd) {
...
case VIDIOCSMICROCODE:
err = get_microcode32(&karg.vc, up);
...
So, the contents of up are totally under control of the caller, and the contents of karg (in our case, the video_code structure) are not initialized at all. So, it seems like a call for VIDIOCSMICROCODE would write video_code->datasize bytes from video_code->data into some random kernel address, just causing an Oops, since we don’t control what is on the kernel’s stack.
But wait, who says we can’t control the contents of the kernel’s stack? In fact, this compat function makes it extremely easy. Let’s look back at the union. Notice the struct video_tuner? That gets populated from the caller’s up memory via this case of the switch (cmd) statement:
...
case VIDIOCSTUNER:
case VIDIOCGTUNER:
err = get_video_tuner32(&karg.vt, up);
...
So, to control the kernel stack, we just need to call this ioctl twice in a row: once to populate the stack via VIDIOCSTUNER with the contents we want (including the future address for video_code->data, which starts at the same location as video_tuner->name[20]), and then again with VIDIOCSMICROCODE.
Tricks involved here are: the definition of the VIDIOCSMICROCODE case in the kernel is wrong, and calling the ioctls without any preparation can trigger other kernel work (memory faults, etc) that may destroy the stack contents. First, we need the real value for the desired case statement. This turns out to be 0x4020761b. Next, we just repeatedly call the setup ioctl in an attempt to get incidental kernel work out of the way so that our last ioctl doing the stack preparation will stick, and then we call the buggy ioctl to trigger the vulnerability.
Since the ioctl already does a multi-byte copy, we can now copy arbitrary lengths of bytes into kernel memory. One method of turning an arbitrary kernel memory write into a privilege escalation is to overwrite a kernel function pointer, and trigger that function. Based on the exploit for CVE-2010-3081, I opted to overwrite the security_ops function pointer table. Their use of msg_queue_msgctl wasn’t very good for the general case since it’s near the end of the table and its offset would depend on kernel versions. Initially I opted for getcap, but in the end used ptrace_traceme, both of which are very near the top the security_ops structure. (Though I need share credit here with Dan Rosenberg as we were working together on improving the reliability of the security_ops overwrite method. He used the same approach for his excellent RDS exploit.)
Here are the steps for one way of taking an arbitrary kernel memory write and turning it into a root escalation:
security_ops with default_security_ops, which will revert the LSM back to the capabilities-only security operations. This, however, means we can calculate where cap_ptrace_traceme is.default_security_ops->ptrace_traceme to point to our supplied function that will actually perform the privilege escalation (thanks to Brad Spengler for his code from Enlightenment).ptrace(PTRACE_TRACEME, 0, NULL, NULL)).default_security_ops->ptrace_traceme to point to cap_ptrace_traceme so the next caller doesn’t Oops the system (since userspace memory will be remapped).Here’s the source for Vyakarana as seen running in Enlightenment using cap_getcap (which is pretty unstable, so you might want to switch it to use ptrace_traceme), and as a stand-alone memory writer.
Conclusions: Keep auditing the kernel for more arbitrary writes; I think there are still many left. Reduce the exploitation surface within the kernel itself (which PaX and grsecurity have been doing for a while now), specifically:
fops, IDT, security_ops, etc) to be writable. These should all be marked correctly, with inline code exceptions being made for updating the global pointers to those tables, leaving the pointer read-only after it gets set. This would stop this particular exploit above, but there are still plenty more targets.© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
The recent CVE-2010-2961 mountall vulnerability got a nice write-up by xorl today. I’ve seen a few public exploits for it, but those that I’ve seen, including the one in xorl’s post, miss a rather important point: udev events can be triggered by regular users without any hardware fiddling. While the bug that kept udev from running inotify correctly on the /dev/.udev/rules.d directory during initial boot kept this vulnerability exposure pretty well minimized, the fact that udev events can be triggered at will made it pretty bad too. If udev had already been restarted, an attacker didn’t have to wait at all, nor have physical access to the system.
While it is generally understood that udev events are related to hardware, it’s important to keep in mind that it also sends events on module loads, and module loads can happen on demand from unprivileged users. For example, say you want to send an X.25 packet, when you call socket(AF_X25, SOCK_STREAM), the kernel will go load net-pf-9, which modules.alias lists as the x25 module. And once loaded, udev sends a “module” event.
(Which, by the way, should serve as a reminder to people to block module loading if you can.)
So, as I mentioned, here’s yet another exploit for the mountall vulnerability: mountall-CVE-2010-2961.py. It writes to the vulnerable udev rule file and then attempts to trigger udev immediately by walking a list of possible socket() AF_* types.
© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
As part of the Ubuntu Security Team’s overall OpenPGP key transition plans, I’ve generated a new GPG key (DC6DC026). If you’ve signed my old key (17063E6D), I’d appreciate it if you could review my signed transition statement and sign my new key too. :) Thanks!
© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
I was asked to write about what I do at Canonical and what I do in the Free Software community at large. There is obviously a great deal of overlap, but I’ll start with the things I’m involved with when I’m wearing my “Ubuntu” hat.
My primary job at Canonical is keeping Ubuntu secure. This means that I, along with the rest of the Ubuntu Security Team, coordinate with other Free Software distributions and upstream projects to publish fixes together so that everyone in the community has the smallest possible window of vulnerability, no matter if they’re running Ubuntu, Debian, RedHat/Fedora, SUSE/openSUSE, Gentoo, etc. Between vendor-sec, oss-security, and the steady stream of new CVEs, there is plenty going on.
In addition to updates, the Security Team works on pro-active security protections. I work on userspace security hardening via patches to gcc and the kernel, and via build-wrapper script packages. Much of this work has been related trying to coordinate these changes with Debian, and to clean up unfinished pieces that were left unsolved by RedHat, who had originally developed many of the hardening features. Things like proper /proc/$pid/maps permissions, real AT_RANDOM implementation, upstreaming executable stack fixing patches, upstreaming kernel NX-emu, etc. Most of the kernel work I’ve done has gotten upstream, but lately some of the more aggressive protections have been hitting frustrating upstream roadblocks.
Besides the hardening work, I also improve and support the AppArmor Mandatory Access Control system, as well as write and improve confinement profiles for processes on Ubuntu. This work ends up improving everyone’s experience with AppArmor, especially now that it has gotten accepted upstream in the Linux kernel.
I audit code from time to time, both “on the clock” with Canonical and in my free time. I’m no Tavis Ormandy, but I try. ;) I’ve found various security issues in Xorg, Koffice, smb4k, libgd2, Inkscape, curl+GnuTLS, hplip, wpa_supplicant, Flickr Drupal module, poppler/xpdf, LimeSurvey, tunapie, and the Linux kernel.
With my Canonical hat off, I do all kinds of random things around the Free Software ecosystem. I’m a sysadmin for kernel.org. In Debian, I maintain a few packages, continue to try to push for security hardening, and contribute to the CVE triage efforts of the Debian Security Team.
I’ve written or maintain several weird projects, including MythTVFS for browsing MythTV recordings, GOPchop for doing non-encoding editing of MPEG2-PS streams, Perl’s Device::SerialPort module, and the TAP paging server Sendpage.
For a selection of things I’ve contributed to other project, I’ve implemented TPM RNG access in rng-tools, made contributions to Inkscape‘s build and print systems, implemented CryptProtect for Wine, wrote a PayPal IPN agent in PHP that actually checks SSL certificates unlike every other implementation I could find, added additional protocol-specific STARTTLS negotiations to OpenSSL, implemented the initial DVD navigation support in MPlayer, updated serial port logic in Scantool for communicating with vehicle CAN interfaces, tried to add support for new types of timeouts in Snort and Ettercap, fixed bugs in mutt, and added HPUX audio support to the Apple ][ emulator XGS.
As you can see, I like making weird/ancient protocols, unfriendly file formats, and security features more accessible to people using Free Software. I’ve done this through patches, convincing people to take those patches, auditing code, testing fixes and features, and doing packaging work.
When I go to conferences, I attend UDS, DefCon, OSCon, and LinuxCon. I’ve presented in the past at OSCon on various topics including security, testing, and video formats, and presented at the Linux Security Summit (miniconf before LinuxCon this year) on the need to upstream various out-of-tree security features available to the Linux kernel.
I love our ecosystem, and I love being part of it. :)
© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
The recent work by MWR Labs does a reasonable job showing Debian’s poor pro-active security and why I am so frustrated about it: we have not been able to move very quickly at getting it enabled. While my hardening-includes package is available to maintainers that want to turn on protections for their builds, it’s still a far cry from having it be distro-wide, and it doesn’t protect people that build stuff by hand. We were able to solve this in Ubuntu very directly a while ago by improving the compiler itself.
Since SSP and FORTIFY_SOURCE can only be confirmed (it’s not possible without source analysis to see if it should have been enabled), it would be nice to see what binaries differed between distros on this. Most of the “SSP disabled” stuff are binaries that lack character arrays on the stack to begin with, and the FORTIFY_SOURCE stuff may have done all compile-time protections. The comments about “other distributions could potentially enable it for a few more binaries” is a bit misleading since, for all but Debian, both SSP and FORTIFY_SOURCE are enabled for all builds.
I did appreciate the nod to Ubuntu for being the only distro without by-default PIE that built Firefox with PIE. Given that Firefox is the #2 most vulnerable piece of software in a desktop distro, it was important to do it. (The #1 most vulnerable is the kernel itself — I’m counting number of fixed CVEs for this stat.)
The kernel analysis by MWR seems rather incomplete. Also, it’s not clear to me which distros were running a PAE kernel, which would change some of the results. I didn’t see any mention of several other userspace protections that the kernel can provide, for example:
And a ton more that only Gentoo Hardened could boast, due to their use of grsecurity.
I’d also be curious to see performance comparisons, too. They compared 4 general-purpose distros against a tuned-specifically-for-security-hardening distro, which seems a bit unfair. How about comparing against vanilla Gentoo instead? I can tell you who would be best then. :)
© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
Dan Rosenberg pointed me to a paper from the 2010 WOOT conference that mentions my work to implement the CryptProtect function in Wine. Their research is awesome, and it was fun to compare my attempts at identifying the blob structure to what they discovered. Looks like I got the structure pretty well, but that was easy; they totally broke the encryption itself. Now those native blobs can be decrypted, opening the door to full NTFS interoperability, offline forensics of Windows encrypted files, etc. (For designers of future symmetric encryption methods: please don’t store the keys (in any form) on disk with the cipher text…)
What I found most alarming about this is a comparison to eCryptfs, and how it is implemented with the user’s login passphrase. In both cases, a hash of the passphrase is used to perform additional work that results in keying the final encryption. In eCryptfs, this hash is calculated to unlock the main key that is used for eCryptfs and is then thrown away (it can always be regenerated when the user logs in). If the user changes their passphrase, they must decrypt and re-encrypt the eCryptfs key (this is done automatically by PAM). Under Windows, to deal with potential user login passphrase changes, they instead decided to store all prior (SHA1) hashes of the user’s passphrases, even lacking a salt. So all the clear-text user login passphrases are recoverable with a standard rainbow table, in parallel. O_o
© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
I think it would be fun to add an achievement system to the Ubuntu Desktop, like is done on Steam and XBox.
The tricky part is tracking various events and finding amusing correlations. For example, if your screen-saver kicks in 40 times in a single 24 hour period, you could earn the “Alternating Current” achievement, indicating that you’re being repeatedly interrupted all day long:
There are all kind of things to track and correlate. Miles moved with the mouse, clicks taken, keys pressed, files opened, applications installed, buddies added, IMs received, sent, etc. There are all kinds of achievements that could be designed that could be used to help people discover how to use Ubuntu, or for just plain humor. “Achievement Unlocked: Application Deficit Disorder” when you uninstall 100 applications you installed in the prior week.
I’ve been told this might all be very easy to implement with the Gnome Activity Journal (Zeitgeist), but I haven’t had a chance to investigate further.
UPDATE: I can easily imagine this being tracked in CouchDB, synced between systems via UbuntuOne, and could be linked to any other remote APIs that people could dream up, including Launchpad, Forums, REVU, Identi.ca, etc.
© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
Starting with GDB version 7, first appearing in Ubuntu with Karmic (Ubuntu 9.10), ASLR is turned off (via the ADDR_NO_RANDOMIZE personality flag) for the debugged process. If you want a more realistic view of how a process will appear in memory, you must “set disable-randomization off” in gdb:
$ gdb /usr/bin/something
...
(gdb) show disable-randomization
Disabling randomization of debuggee's virtual address space is on.
(gdb) start
...
(gdb) ^Z
$ cat /proc/$(pidof /usr/bin/something)/personality
00040000
$ grep 0040000 /usr/include/linux/personality.h
ADDR_NO_RANDOMIZE = 0x0040000, /* disable randomization of VA space */
$ fg
(gdb) set disable-randomization off
(gdb) show disable-randomization
Disabling randomization of debuggee's virtual address space is off.
(gdb) start
...
(gdb) ^Z
$ cat /proc/$(pidof /usr/bin/something)/personality
00000000
© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
I recently learned about the process event connector, and went looking for an example program that could report all the exec()s that happen on my system to help with debugging things like AC plug/unplug scripts, etc.
After cleaning it up and adding some features to do a simple best-effort cmdline reporting, I’ve now got a tool that will report every program run on a system:
$ sudo ./cn_proc sending proc connector: PROC_CN_MCAST_LISTEN... sent Reading process events from proc connector. Hit Ctrl-C to exit event: exec 17514 17514: ls -AF --color=auto (unconfined) event: exec 17516 17516: date +%H:%M (unconfined) event: exec 17518 17518: whoami (unconfined)
Change the values show_event, show_seq, show_cpu, show_security_context to set the reporting defaults. Or, if someone is feeling bored, it would rock to add getopt support instead.
It seems strange to me that only CAP_NET_ADMIN is needed to get access to this information.
© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
I know there are a lot of ways to do this, but I just wanted to note what works for me to catch early boot Oopses in the kernel. In virt-manager, I set up a serial device, as a “pty”. After the VM boots, I pause in Grub, and check the VM details for the serial port (it’ll show something like “/dev/pts/13”). In another terminal, “sudo cat /dev/pts/13”, and I boot the kernel with “console=ttyS0”. When the Oops happens, I get it in the terminal instead of having it scroll off the VM’s screen.
© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
The Call For Participation is open for the 2010 Linux Security Summit, being held just before this year’s LinuxCon.
If you’re interested in helping make Linux more secure, you’ve got ideas to present, want to have your opinion heard, or generally just want to hang out, please join us and/or suggest a topic for discussion (CFP ends June 4th, so please hurry).
I’m hoping to get a chance to discuss what I’m calling the “popular kernel hardening patches” which appear in a lot of distros yet remain missing from the upstream Linux kernel.
© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
I have a camera that takes .MP4 files. MythTV’s Gallery plugin has a static list of extensions it recognizes. MP4 is not one of them. I can rename all my files (which is wrong — they’re MP4s not something else), or I can edit the source and recompile.
Or, since I want to watch them RIGHT NOW, I can binary-edit the shared library and replace something I’ll not have any of (WMV) with MP4.
# strings -a /usr/lib/mythtv/plugins/libmythgallery.so | grep WMV *.WMV # sed -i -e 's/\.WMV/.MP4/' /usr/lib/mythtv/plugins/libmythgallery.so
Ta-da. Just need to make sure I don’t change the size of the binary. :)
(Oddly, I’m not too lazy to open a bug and propose a patch.)
© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
During the Lucid development sprint, Pete showed me the glory that is the OpenOffice presenter console. Since then I’ve switched laptops, and I figured I need to document for myself, and maybe others, how to have your OOo presentation on one screen (e.g. the projector) and the presentation notes, clock, etc, on another (e.g. the main laptop display).
First, install ‘openoffice.org-presenter-console’, and once you have your displays configured, launch OpenOffice and select “Slide Show” / “Slide Show Settings …”. From here, you can configure the displays under “Multiple displays”.
So nice. :)
UPDATE: corrected my typo. thank you! :)
© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
I find it surreal to have people guessing at my motivations when they could just ask me. On top of all that, I find it weird that people spend so much time with in-fighting. I just want my system not to suck.
Some time ago (during in the Ubuntu Karmic development cycle maybe in September 2009), I started having giant problems with my build system. All I/O would start to stall, wait times would surge, and usually my entire system would just go unresponsive with the disk light on solid. This scared the crap out of me, and it wasn’t entirely obvious what was triggering it. No one else seemed to be seeing it. I managed to start tracking things using “latencytop”, and saw stuff like liferea going crazy. As I eliminated more and more things, I eventually settled on it being a problem with umount, and I reported an Ubuntu bug. It seemed to look like an upstream bug that no one else but the reporter could reproduce either.
Since no one else was seeing this issue, and it seemed related to LVM snapshots, I migrated off of snapshots, and started using aufs overlays for my builds. For a while, it seemed like things went away. It didn’t, and I started hitting it again. I opened a new (now famous) bug in Ubuntu, since now snapshots weren’t in the picture, and I didn’t want to confuse the earlier history. I managed to find a relatively minimal test-case too. A few other people commenting on the bug were seeing the problem now too, but it was less pronounced for them.
As an aside, this wasn’t a “just wait a few seconds longer” kind of issue on my system. A single umount would last 30-40 minutes. And when I’m doing parallel builds of security updates, this would turn into my system being unavailable for hours at a stretch.
Since none of the kernel developers I was in contact with were able to track down the root cause, I asked Ted Ts’o in email if he could just quickly peek in on this for me, since I figured he’d be in a good position to confirm or deny it. I didn’t want to start wasting upstream time with this if it wasn’t reproducible (see earlier upstream kernel bug). To my great relief, Ted found a few minutes to check it, and was able to immediately confirm it and give me a viable work-around (“sync; umount …”) for the time being. I confirmed the work-around, and went off to do other things.
A while later, Ted came back to deliver a bit of a rant, the purpose of which was not clear to me, but I ultimately ignored it — I didn’t seem directed at me. I just wanted my system operating normally, and he’d done me a favor to check in on it and got me a work-around.
More time passes, and I eventually get caught in another I/O-wait melt-down. On investigation, it seemed that the in-kernel work-around in the Ubuntu kernel totally back-fired on me in some cases, rendering even the user-space work-around useless. After investigating the Ubuntu-specific work-around, I re-read Ted’s rant in the course of researching what had happened during this bug’s triage.
It seemed that Ted was basically saying:
– this is an upstream problem
– RedHat hasn’t run into it and he didn’t know why
I figured I should confirm for myself if Fedora was affected, so downloaded and installed Fedora to double-check there. Since I was able to reproduce it there, I opened an upstream bug, linking back to the original Ubuntu bug, and then went to open a bug in the Fedora tracker, linking back to upstream.
And it seems to be these actions that everyone has jumped on. I will now bore you with the reality of my motivations: I wanted to fix the bug so no one would end up experiencing the same pain I’d been through over the last 6 months.
The bug was, from my perspective, a serious issue. Since I’d managed to reproduce it in another distro, it was my duty as a Free Software developer to report it to them. And, in what I felt was an unambiguous gesture, I made sure to include the link to the upstream kernel bug. Reproducing it in Ubuntu, in Fedora, and with a stock kernel had me confident that it was an upstream issue. While Ted did correctly suspect the issue was upstream, I really didn’t want to just open an upstream bug and have it be ignored. I wanted some additional proof of reproduction, which I got when I tested it on Fedora.
So, I’m rather saddened that so many people spent so much time questioning my motivations, making fun of Canonical, or doing anything other than trying to just simply solve this problem. I’m totally disinterested in inter-distro fighting. Instead, I continue to assume we’re all on the same team, fighting a philosophical battle against close-source software. And in that regard, I think it’s still true. If I ignore the rants and jeering, I come away thankful for all the people that spent time trying reproduce the issue at Canonical, at RedHat, and in the larger community. I’m hugely thankful that Ted made some time to let me know I wasn’t crazy, and there was actually a problem. I’m thankful for having some work-arounds, and I’m thankful that the root cause was eventually ferreted out, with some possible solutions. I’m even thankful that some people on the LWN thread saw that, far from malicious, I was trying to be helpful with the bug.
I just wanted my filesystem not to eat my computer. And I was hoping other people could maybe help me, since I’m not a filesystem expert. The drama around this bug is pathetic, and now by talking about it for almost 1000 words, I’m just as guilty.
© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
Since the Debian Women Wiki is on hiatus, I found myself without the handy Maintainer Scripts diagrams today. So, I recovered it from Google’s cache, and put it up on the main Debian wiki: MaintainerScripts.
I used a great tool to go from HTML back to MoinMoin syntax: html2wiki
© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
I realize the openssl s_client tool tries to be upper-layer protocol agnostic, but doesn’t everything that uses SSL do commonName checking (HTTP, SMTP, IMAP, FTP, POP, XMPP)? Shouldn’t this be something openssl s_client does by default, maybe with an option to turn it off for less common situations?
Here it doesn’t complain about connecting to “outflux.net” when the cert has a CN for “www.outflux.net”:
echo QUIT | openssl s_client -CApath /etc/ssl/certs \ -connect outflux.net:443 2>/dev/null | egrep "subject=|Verify"
subject=/CN=www.outflux.net
Verify return code: 0 (ok)
© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
I don’t like unconditionally clearing /tmp on boot, since I’m invariably working on something in there when my system locks up. But I do like /tmp getting cleaned up from time to time. As a compromise, I’ve set TMPTIME=7 in /etc/default/rcS so that only stuff older than 7 days is deleted when I reboot.
© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
I’ve finally soldered on the LEDs for the Cylon DorkShield I designed and got built in a recent DorkBotPDX PCB run. I’m having trouble with the programmer, but I got the shield mostly working:
© 2010 – 2015, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
I’m never sure when my commits to Linux actually make it into an official release. Luckily, since releases are tagged, I can use “git tag --contains COMMIT” to find them. So, in a stunning display of inefficiency, here’s what I use to find my commits:
git log --author='Kees Cook' --pretty=oneline | \
while read commit name; do \
echo $(git tag --contains $commit | head -n1): "$name"; \
done
Which lets me know where my code is with respect to releases:
v2.6.33: x86, mm: Report state of NX protections during boot v2.6.33: sysctl: require CAP_SYS_RAWIO to set mmap_min_addr v2.6.32: proc: fix reported unit for RLIMIT_CPU v2.6.31: modules: sysctl to block module loading ...
© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
Powered by WordPress