codeblog code is freedom — patching my itch

November 7, 2010

security is more than bug fixing

Filed under: Blogging,Debian,Security,Ubuntu,Ubuntu-Server — kees @ 12:20 pm

Security is more than bug fixing. Security fixing/updating, the thing most people are exposed to, is “reactive security”. However, a large area of security work is “proactive” where defensive abilities are put in place to try and catch problems before they happen, or make classes of vulnerabilities unexploitable. This kind of security is what a lot of people don’t understand, and I think it’s important to point out so the distinction can be clearly seen.

In the Linux kernel, there’s yet another distinction: userspace proactive security and kernel proactive security. Most of the effort in kernel code has been protecting userspace from itself (things like Address Space Layout Randomization), but less attention has been given to protecting the kernel from userspace (currently if a serious enough flaw is found in the kernel, it is usually very easy to exploit it).

One project has taken great strides with proactive security for the Linux kernel: PaX and grsecurity. There hasn’t been a concerted effort to get its pieces upstream and it’s long overdue. People are starting to take proactive kernel security more seriously, though there is still plenty of debate.

While I did my best to push some userspace protections upstream earlier in the year, now it’s time for kernel protections. What to help? Here is the initial list of things to do.

Dan Rosenberg has started the information leaks discussion, and I’ve started the read-only memory discussion. Hopefully this will go somewhere good.

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

October 25, 2010

Jettison Jaunty

Filed under: Blogging,Security,Ubuntu,Ubuntu-Server — kees @ 10:07 pm

Jaunty Jackalope (Ubuntu 9.04) went End-Of-Life on Saturday.

Looking back through my build logs, it seems my desktop did 223 builds, spending 19 hours, 18 minutes, and 23 seconds doing builds during the development cycle of Jaunty. Once released, it performed an additional 99 builds, taking 18 hours, 3 minutes, and 37 seconds for security updates. As before, these times obviously don’t include patch hunting/development, failed builds, testing, stuff done on my laptop or the porting machines, etc.

Combined devel/security build standings per current release:

dapper: 59:19:10
hardy: 189:32:51
karmic: 57:44:27
lucid: 36:07:05
maverick: 13:54:15

Looking at the build histories, Gutsy and Jaunty had about the same amount of builds (around 19 hours) during development, but Intrepid was a whopping 70 hours. This was related to all the default compiler flag testing there. I rebuilt the entire “main” component multiple times that release. Jaunty was a nice return to normalcy.

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

October 19, 2010

CVE-2010-2963 v4l compat exploit

Filed under: Blogging,Debian,Security,Ubuntu,Ubuntu-Server,Vulnerabilities — kees @ 3:41 pm

If you’re running a 64bit system, and you’ve got users with access to a video device (/dev/video*), then be sure you update your kernels for CVE-2010-2963. I’ve been slowly making my way through auditing the many uses in the Linux kernel of the copy_from_user() function, and ran into this vulnerability.

Here’s the kernel code from drivers/media/video/v4l2-compat-ioctl32.c:

static int get_microcode32(struct video_code *kp, struct video_code32 __user *up)
{
        if (!access_ok(VERIFY_READ, up, sizeof(struct video_code32)) ||
                copy_from_user(kp->loadwhat, up->loadwhat, sizeof(up->loadwhat)) ||
                get_user(kp->datasize, &up->datasize) ||
                copy_from_user(kp->data, up->data, up->datasize))
                        return -EFAULT;
        return 0;
}

Note that kp->data is being used as the target for up->data in the final copy_from_user() without actually verifying that kp->data is pointing anywhere safe. Here’s the caller of get_microcode32:

static long do_video_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
{
        union {
                struct video_tuner vt;
                struct video_code vc;
...
        } karg;
        void __user *up = compat_ptr(arg);
...
        switch (cmd) {
...
        case VIDIOCSMICROCODE:
                err = get_microcode32(&karg.vc, up);
...

So, the contents of up are totally under control of the caller, and the contents of karg (in our case, the video_code structure) are not initialized at all. So, it seems like a call for VIDIOCSMICROCODE would write video_code->datasize bytes from video_code->data into some random kernel address, just causing an Oops, since we don’t control what is on the kernel’s stack.

But wait, who says we can’t control the contents of the kernel’s stack? In fact, this compat function makes it extremely easy. Let’s look back at the union. Notice the struct video_tuner? That gets populated from the caller’s up memory via this case of the switch (cmd) statement:

...
        case VIDIOCSTUNER:
        case VIDIOCGTUNER:
                err = get_video_tuner32(&karg.vt, up);
...

So, to control the kernel stack, we just need to call this ioctl twice in a row: once to populate the stack via VIDIOCSTUNER with the contents we want (including the future address for video_code->data, which starts at the same location as video_tuner->name[20]), and then again with VIDIOCSMICROCODE.

Tricks involved here are: the definition of the VIDIOCSMICROCODE case in the kernel is wrong, and calling the ioctls without any preparation can trigger other kernel work (memory faults, etc) that may destroy the stack contents. First, we need the real value for the desired case statement. This turns out to be 0x4020761b. Next, we just repeatedly call the setup ioctl in an attempt to get incidental kernel work out of the way so that our last ioctl doing the stack preparation will stick, and then we call the buggy ioctl to trigger the vulnerability.

Since the ioctl already does a multi-byte copy, we can now copy arbitrary lengths of bytes into kernel memory. One method of turning an arbitrary kernel memory write into a privilege escalation is to overwrite a kernel function pointer, and trigger that function. Based on the exploit for CVE-2010-3081, I opted to overwrite the security_ops function pointer table. Their use of msg_queue_msgctl wasn’t very good for the general case since it’s near the end of the table and its offset would depend on kernel versions. Initially I opted for getcap, but in the end used ptrace_traceme, both of which are very near the top the security_ops structure. (Though I need share credit here with Dan Rosenberg as we were working together on improving the reliability of the security_ops overwrite method. He used the same approach for his excellent RDS exploit.)

Here are the steps for one way of taking an arbitrary kernel memory write and turning it into a root escalation:

  • overwrite security_ops with default_security_ops, which will revert the LSM back to the capabilities-only security operations. This, however, means we can calculate where cap_ptrace_traceme is.
  • overwrite default_security_ops->ptrace_traceme to point to our supplied function that will actually perform the privilege escalation (thanks to Brad Spengler for his code from Enlightenment).
  • trigger the function (in this case, call ptrace(PTRACE_TRACEME, 0, NULL, NULL)).
  • restore default_security_ops->ptrace_traceme to point to cap_ptrace_traceme so the next caller doesn’t Oops the system (since userspace memory will be remapped).

Here’s the source for Vyakarana as seen running in Enlightenment using cap_getcap (which is pretty unstable, so you might want to switch it to use ptrace_traceme), and as a stand-alone memory writer.

Conclusions: Keep auditing the kernel for more arbitrary writes; I think there are still many left. Reduce the exploitation surface within the kernel itself (which PaX and grsecurity have been doing for a while now), specifically:

  • Block userspace memory access while in kernel mode. This would stop the ability to make the kernel start executing functions that live in userspace — a clear privilege violation. This protection would stop the current exploit above, but the exploit could be adjusted to use kernel memory instead.
  • Keep function pointers read-only. There is no reason for these function pointer tables (fops, IDT, security_ops, etc) to be writable. These should all be marked correctly, with inline code exceptions being made for updating the global pointers to those tables, leaving the pointer read-only after it gets set. This would stop this particular exploit above, but there are still plenty more targets.
  • Randomize the kernel stack location on a per-syscall basis. This will stop exploits that depend on a stable kernel stack location (as this exploit does).

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

October 13, 2010

mountall umask

Filed under: Blogging,Debian,Security,Ubuntu,Ubuntu-Server,Vulnerabilities — kees @ 9:13 am

The recent CVE-2010-2961 mountall vulnerability got a nice write-up by xorl today. I’ve seen a few public exploits for it, but those that I’ve seen, including the one in xorl’s post, miss a rather important point: udev events can be triggered by regular users without any hardware fiddling. While the bug that kept udev from running inotify correctly on the /dev/.udev/rules.d directory during initial boot kept this vulnerability exposure pretty well minimized, the fact that udev events can be triggered at will made it pretty bad too. If udev had already been restarted, an attacker didn’t have to wait at all, nor have physical access to the system.

While it is generally understood that udev events are related to hardware, it’s important to keep in mind that it also sends events on module loads, and module loads can happen on demand from unprivileged users. For example, say you want to send an X.25 packet, when you call socket(AF_X25, SOCK_STREAM), the kernel will go load net-pf-9, which modules.alias lists as the x25 module. And once loaded, udev sends a “module” event.

(Which, by the way, should serve as a reminder to people to block module loading if you can.)

So, as I mentioned, here’s yet another exploit for the mountall vulnerability: mountall-CVE-2010-2961.py. It writes to the vulnerable udev rule file and then attempts to trigger udev immediately by walking a list of possible socket() AF_* types.

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

September 27, 2010

new RSA4096 key

Filed under: Blogging,Debian,Ubuntu — kees @ 11:50 am

As part of the Ubuntu Security Team’s overall OpenPGP key transition plans, I’ve generated a new GPG key (DC6DC026). If you’ve signed my old key (17063E6D), I’d appreciate it if you could review my signed transition statement and sign my new key too. :) Thanks!

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

September 14, 2010

my part in the ecosystem

I was asked to write about what I do at Canonical and what I do in the Free Software community at large. There is obviously a great deal of overlap, but I’ll start with the things I’m involved with when I’m wearing my “Ubuntu” hat.

My primary job at Canonical is keeping Ubuntu secure. This means that I, along with the rest of the Ubuntu Security Team, coordinate with other Free Software distributions and upstream projects to publish fixes together so that everyone in the community has the smallest possible window of vulnerability, no matter if they’re running Ubuntu, Debian, RedHat/Fedora, SUSE/openSUSE, Gentoo, etc. Between vendor-sec, oss-security, and the steady stream of new CVEs, there is plenty going on.

In addition to updates, the Security Team works on pro-active security protections. I work on userspace security hardening via patches to gcc and the kernel, and via build-wrapper script packages. Much of this work has been related trying to coordinate these changes with Debian, and to clean up unfinished pieces that were left unsolved by RedHat, who had originally developed many of the hardening features. Things like proper /proc/$pid/maps permissions, real AT_RANDOM implementation, upstreaming executable stack fixing patches, upstreaming kernel NX-emu, etc. Most of the kernel work I’ve done has gotten upstream, but lately some of the more aggressive protections have been hitting frustrating upstream roadblocks.

Besides the hardening work, I also improve and support the AppArmor Mandatory Access Control system, as well as write and improve confinement profiles for processes on Ubuntu. This work ends up improving everyone’s experience with AppArmor, especially now that it has gotten accepted upstream in the Linux kernel.

I audit code from time to time, both “on the clock” with Canonical and in my free time. I’m no Tavis Ormandy, but I try. ;) I’ve found various security issues in Xorg, Koffice, smb4k, libgd2, Inkscape, curl+GnuTLS, hplip, wpa_supplicant, Flickr Drupal module, poppler/xpdf, LimeSurvey, tunapie, and the Linux kernel.

With my Canonical hat off, I do all kinds of random things around the Free Software ecosystem. I’m a sysadmin for kernel.org. In Debian, I maintain a few packages, continue to try to push for security hardening, and contribute to the CVE triage efforts of the Debian Security Team.

I’ve written or maintain several weird projects, including MythTVFS for browsing MythTV recordings, GOPchop for doing non-encoding editing of MPEG2-PS streams, Perl’s Device::SerialPort module, and the TAP paging server Sendpage.

For a selection of things I’ve contributed to other project, I’ve implemented TPM RNG access in rng-tools, made contributions to Inkscape‘s build and print systems, implemented CryptProtect for Wine, wrote a PayPal IPN agent in PHP that actually checks SSL certificates unlike every other implementation I could find, added additional protocol-specific STARTTLS negotiations to OpenSSL, implemented the initial DVD navigation support in MPlayer, updated serial port logic in Scantool for communicating with vehicle CAN interfaces, tried to add support for new types of timeouts in Snort and Ettercap, fixed bugs in mutt, and added HPUX audio support to the Apple ][ emulator XGS.

As you can see, I like making weird/ancient protocols, unfriendly file formats, and security features more accessible to people using Free Software. I’ve done this through patches, convincing people to take those patches, auditing code, testing fixes and features, and doing packaging work.

When I go to conferences, I attend UDS, DefCon, OSCon, and LinuxCon. I’ve presented in the past at OSCon on various topics including security, testing, and video formats, and presented at the Linux Security Summit (miniconf before LinuxCon this year) on the need to upstream various out-of-tree security features available to the Linux kernel.

I love our ecosystem, and I love being part of it. :)

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

September 7, 2010

cross-distro default security protection review

Filed under: Blogging,Debian,Security,Ubuntu,Ubuntu-Server — kees @ 11:06 am

The recent work by MWR Labs does a reasonable job showing Debian’s poor pro-active security and why I am so frustrated about it: we have not been able to move very quickly at getting it enabled. While my hardening-includes package is available to maintainers that want to turn on protections for their builds, it’s still a far cry from having it be distro-wide, and it doesn’t protect people that build stuff by hand. We were able to solve this in Ubuntu very directly a while ago by improving the compiler itself.

Since SSP and FORTIFY_SOURCE can only be confirmed (it’s not possible without source analysis to see if it should have been enabled), it would be nice to see what binaries differed between distros on this. Most of the “SSP disabled” stuff are binaries that lack character arrays on the stack to begin with, and the FORTIFY_SOURCE stuff may have done all compile-time protections. The comments about “other distributions could potentially enable it for a few more binaries” is a bit misleading since, for all but Debian, both SSP and FORTIFY_SOURCE are enabled for all builds.

I did appreciate the nod to Ubuntu for being the only distro without by-default PIE that built Firefox with PIE. Given that Firefox is the #2 most vulnerable piece of software in a desktop distro, it was important to do it. (The #1 most vulnerable is the kernel itself — I’m counting number of fixed CVEs for this stat.)

The kernel analysis by MWR seems rather incomplete. Also, it’s not clear to me which distros were running a PAE kernel, which would change some of the results. I didn’t see any mention of several other userspace protections that the kernel can provide, for example:

  • symlink and hardlink protections (Gentoo Hardened and Ubuntu 10.10 only)
  • PTRACE protections (Gentoo Hardened and Ubuntu 10.10 only)

And a ton more that only Gentoo Hardened could boast, due to their use of grsecurity.

I’d also be curious to see performance comparisons, too. They compared 4 general-purpose distros against a tuned-specifically-for-security-hardening distro, which seems a bit unfair. How about comparing against vanilla Gentoo instead? I can tell you who would be best then. :)

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

August 12, 2010

CryptProtect broken

Dan Rosenberg pointed me to a paper from the 2010 WOOT conference that mentions my work to implement the CryptProtect function in Wine. Their research is awesome, and it was fun to compare my attempts at identifying the blob structure to what they discovered. Looks like I got the structure pretty well, but that was easy; they totally broke the encryption itself. Now those native blobs can be decrypted, opening the door to full NTFS interoperability, offline forensics of Windows encrypted files, etc. (For designers of future symmetric encryption methods: please don’t store the keys (in any form) on disk with the cipher text…)

What I found most alarming about this is a comparison to eCryptfs, and how it is implemented with the user’s login passphrase. In both cases, a hash of the passphrase is used to perform additional work that results in keying the final encryption. In eCryptfs, this hash is calculated to unlock the main key that is used for eCryptfs and is then thrown away (it can always be regenerated when the user logs in). If the user changes their passphrase, they must decrypt and re-encrypt the eCryptfs key (this is done automatically by PAM). Under Windows, to deal with potential user login passphrase changes, they instead decided to store all prior (SHA1) hashes of the user’s passphrases, even lacking a salt. So all the clear-text user login passphrases are recoverable with a standard rainbow table, in parallel. O_o

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

July 23, 2010

Achievement Unlocked

Filed under: Blogging,Debian,General,Ubuntu — kees @ 5:45 am

I think it would be fun to add an achievement system to the Ubuntu Desktop, like is done on Steam and XBox.

The tricky part is tracking various events and finding amusing correlations. For example, if your screen-saver kicks in 40 times in a single 24 hour period, you could earn the “Alternating Current” achievement, indicating that you’re being repeatedly interrupted all day long:

achievement unlocked: alternating current

There are all kind of things to track and correlate. Miles moved with the mouse, clicks taken, keys pressed, files opened, applications installed, buddies added, IMs received, sent, etc. There are all kinds of achievements that could be designed that could be used to help people discover how to use Ubuntu, or for just plain humor. “Achievement Unlocked: Application Deficit Disorder” when you uninstall 100 applications you installed in the prior week.

I’ve been told this might all be very easy to implement with the Gnome Activity Journal (Zeitgeist), but I haven’t had a chance to investigate further.

UPDATE: I can easily imagine this being tracked in CouchDB, synced between systems via UbuntuOne, and could be linked to any other remote APIs that people could dream up, including Launchpad, Forums, REVU, Identi.ca, etc.

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

July 3, 2010

gdb turns off ASLR

Filed under: Blogging,Debian,Ubuntu,Ubuntu-Server — kees @ 10:05 am

Starting with GDB version 7, first appearing in Ubuntu with Karmic (Ubuntu 9.10), ASLR is turned off (via the ADDR_NO_RANDOMIZE personality flag) for the debugged process. If you want a more realistic view of how a process will appear in memory, you must “set disable-randomization off” in gdb:

$ gdb /usr/bin/something
...
(gdb) show disable-randomization
Disabling randomization of debuggee's virtual address space is on.
(gdb) start
...
(gdb) ^Z
$ cat /proc/$(pidof /usr/bin/something)/personality
00040000
$ grep 0040000 /usr/include/linux/personality.h
    ADDR_NO_RANDOMIZE =     0x0040000,  /* disable randomization of VA space */
$ fg
(gdb) set disable-randomization off
(gdb) show disable-randomization
Disabling randomization of debuggee's virtual address space is off.
(gdb) start
...
(gdb) ^Z
$ cat /proc/$(pidof /usr/bin/something)/personality
00000000

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

July 1, 2010

reporting all execs

Filed under: Blogging,Debian,Ubuntu,Ubuntu-Server — kees @ 2:24 pm

I recently learned about the process event connector, and went looking for an example program that could report all the exec()s that happen on my system to help with debugging things like AC plug/unplug scripts, etc.

After cleaning it up and adding some features to do a simple best-effort cmdline reporting, I’ve now got a tool that will report every program run on a system:

$ sudo ./cn_proc 
sending proc connector: PROC_CN_MCAST_LISTEN... sent
Reading process events from proc connector.
Hit Ctrl-C to exit
event: exec 17514 17514: ls -AF --color=auto (unconfined)
event: exec 17516 17516: date +%H:%M (unconfined)
event: exec 17518 17518: whoami (unconfined)

Change the values show_event, show_seq, show_cpu, show_security_context to set the reporting defaults. Or, if someone is feeling bored, it would rock to add getopt support instead.

It seems strange to me that only CAP_NET_ADMIN is needed to get access to this information.

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

June 21, 2010

kvm kernel console

Filed under: Blogging,Debian,Ubuntu,Ubuntu-Server — kees @ 12:50 pm

I know there are a lot of ways to do this, but I just wanted to note what works for me to catch early boot Oopses in the kernel. In virt-manager, I set up a serial device, as a “pty”. After the VM boots, I pause in Grub, and check the VM details for the serial port (it’ll show something like “/dev/pts/13”). In another terminal, “sudo cat /dev/pts/13”, and I boot the kernel with “console=ttyS0”. When the Oops happens, I get it in the terminal instead of having it scroll off the VM’s screen.

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

May 29, 2010

Linux Security Summit 2010

Filed under: Blogging,Debian,Security,Ubuntu,Ubuntu-Server — kees @ 5:22 pm

The Call For Participation is open for the 2010 Linux Security Summit, being held just before this year’s LinuxCon.

If you’re interested in helping make Linux more secure, you’ve got ideas to present, want to have your opinion heard, or generally just want to hang out, please join us and/or suggest a topic for discussion (CFP ends June 4th, so please hurry).

I’m hoping to get a chance to discuss what I’m calling the “popular kernel hardening patches” which appear in a lot of distros yet remain missing from the upstream Linux kernel.

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

May 27, 2010

too lazy to recompile

Filed under: Blogging,Debian,Multimedia,Ubuntu — kees @ 9:13 pm

I have a camera that takes .MP4 files. MythTV’s Gallery plugin has a static list of extensions it recognizes. MP4 is not one of them. I can rename all my files (which is wrong — they’re MP4s not something else), or I can edit the source and recompile.

Or, since I want to watch them RIGHT NOW, I can binary-edit the shared library and replace something I’ll not have any of (WMV) with MP4.

# strings -a /usr/lib/mythtv/plugins/libmythgallery.so | grep WMV 
*.WMV
# sed -i -e 's/\.WMV/.MP4/' /usr/lib/mythtv/plugins/libmythgallery.so

Ta-da. Just need to make sure I don’t change the size of the binary. :)

(Oddly, I’m not too lazy to open a bug and propose a patch.)

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

May 19, 2010

openoffice presenter console

Filed under: Blogging,Debian,Ubuntu — kees @ 5:38 pm

During the Lucid development sprint, Pete showed me the glory that is the OpenOffice presenter console. Since then I’ve switched laptops, and I figured I need to document for myself, and maybe others, how to have your OOo presentation on one screen (e.g. the projector) and the presentation notes, clock, etc, on another (e.g. the main laptop display).

First, install ‘openoffice.org-presenter-console’, and once you have your displays configured, launch OpenOffice and select “Slide Show” / “Slide Show Settings …”. From here, you can configure the displays under “Multiple displays”.

So nice. :)

UPDATE: corrected my typo. thank you! :)

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

May 17, 2010

yay for barriers

Filed under: Blogging,Debian,Ubuntu,Ubuntu-Server — kees @ 12:13 pm

I find it surreal to have people guessing at my motivations when they could just ask me. On top of all that, I find it weird that people spend so much time with in-fighting. I just want my system not to suck.

Some time ago (during in the Ubuntu Karmic development cycle maybe in September 2009), I started having giant problems with my build system. All I/O would start to stall, wait times would surge, and usually my entire system would just go unresponsive with the disk light on solid. This scared the crap out of me, and it wasn’t entirely obvious what was triggering it. No one else seemed to be seeing it. I managed to start tracking things using “latencytop”, and saw stuff like liferea going crazy. As I eliminated more and more things, I eventually settled on it being a problem with umount, and I reported an Ubuntu bug. It seemed to look like an upstream bug that no one else but the reporter could reproduce either.

Since no one else was seeing this issue, and it seemed related to LVM snapshots, I migrated off of snapshots, and started using aufs overlays for my builds. For a while, it seemed like things went away. It didn’t, and I started hitting it again. I opened a new (now famous) bug in Ubuntu, since now snapshots weren’t in the picture, and I didn’t want to confuse the earlier history. I managed to find a relatively minimal test-case too. A few other people commenting on the bug were seeing the problem now too, but it was less pronounced for them.

As an aside, this wasn’t a “just wait a few seconds longer” kind of issue on my system. A single umount would last 30-40 minutes. And when I’m doing parallel builds of security updates, this would turn into my system being unavailable for hours at a stretch.

Since none of the kernel developers I was in contact with were able to track down the root cause, I asked Ted Ts’o in email if he could just quickly peek in on this for me, since I figured he’d be in a good position to confirm or deny it. I didn’t want to start wasting upstream time with this if it wasn’t reproducible (see earlier upstream kernel bug). To my great relief, Ted found a few minutes to check it, and was able to immediately confirm it and give me a viable work-around (“sync; umount …”) for the time being. I confirmed the work-around, and went off to do other things.

A while later, Ted came back to deliver a bit of a rant, the purpose of which was not clear to me, but I ultimately ignored it — I didn’t seem directed at me. I just wanted my system operating normally, and he’d done me a favor to check in on it and got me a work-around.

More time passes, and I eventually get caught in another I/O-wait melt-down. On investigation, it seemed that the in-kernel work-around in the Ubuntu kernel totally back-fired on me in some cases, rendering even the user-space work-around useless. After investigating the Ubuntu-specific work-around, I re-read Ted’s rant in the course of researching what had happened during this bug’s triage.

It seemed that Ted was basically saying:
– this is an upstream problem
– RedHat hasn’t run into it and he didn’t know why

I figured I should confirm for myself if Fedora was affected, so downloaded and installed Fedora to double-check there. Since I was able to reproduce it there, I opened an upstream bug, linking back to the original Ubuntu bug, and then went to open a bug in the Fedora tracker, linking back to upstream.

And it seems to be these actions that everyone has jumped on. I will now bore you with the reality of my motivations: I wanted to fix the bug so no one would end up experiencing the same pain I’d been through over the last 6 months.

The bug was, from my perspective, a serious issue. Since I’d managed to reproduce it in another distro, it was my duty as a Free Software developer to report it to them. And, in what I felt was an unambiguous gesture, I made sure to include the link to the upstream kernel bug. Reproducing it in Ubuntu, in Fedora, and with a stock kernel had me confident that it was an upstream issue. While Ted did correctly suspect the issue was upstream, I really didn’t want to just open an upstream bug and have it be ignored. I wanted some additional proof of reproduction, which I got when I tested it on Fedora.

So, I’m rather saddened that so many people spent so much time questioning my motivations, making fun of Canonical, or doing anything other than trying to just simply solve this problem. I’m totally disinterested in inter-distro fighting. Instead, I continue to assume we’re all on the same team, fighting a philosophical battle against close-source software. And in that regard, I think it’s still true. If I ignore the rants and jeering, I come away thankful for all the people that spent time trying reproduce the issue at Canonical, at RedHat, and in the larger community. I’m hugely thankful that Ted made some time to let me know I wasn’t crazy, and there was actually a problem. I’m thankful for having some work-arounds, and I’m thankful that the root cause was eventually ferreted out, with some possible solutions. I’m even thankful that some people on the LWN thread saw that, far from malicious, I was trying to be helpful with the bug.

I just wanted my filesystem not to eat my computer. And I was hoping other people could maybe help me, since I’m not a filesystem expert. The drama around this bug is pathetic, and now by talking about it for almost 1000 words, I’m just as guilty.

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

April 29, 2010

Intrepid Inactive

Filed under: Blogging,Security,Ubuntu,Ubuntu-Server — kees @ 5:00 pm

Intrepid is now officially at end-of-life.

Looking back through my build logs, it seems my desktop did 1340 builds, spending 70 hours, 38 minutes, and 49 seconds doing builds during the development cycle of Intrepid. Once released, it performed an additional 123 builds, taking 19 hours, 29 minutes, and 48 seconds for security updates. As before, these times obviously don’t include patch hunting/development, failed builds, testing, stuff done on my laptop or the porting machines, etc.

Thank you Intrepid! You were the first release to carry the full set of by-default hardened compiler flags.

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

April 7, 2010

Debian maintainer scripts diagrams

Filed under: Blogging,Debian,Ubuntu — kees @ 2:27 pm

Since the Debian Women Wiki is on hiatus, I found myself without the handy Maintainer Scripts diagrams today. So, I recovered it from Google’s cache, and put it up on the main Debian wiki: MaintainerScripts.

I used a great tool to go from HTML back to MoinMoin syntax: html2wiki

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

March 10, 2010

openssl client does not check commonName

Filed under: Blogging,Debian,Security,Ubuntu,Ubuntu-Server — kees @ 10:47 pm

I realize the openssl s_client tool tries to be upper-layer protocol agnostic, but doesn’t everything that uses SSL do commonName checking (HTTP, SMTP, IMAP, FTP, POP, XMPP)? Shouldn’t this be something openssl s_client does by default, maybe with an option to turn it off for less common situations?

Here it doesn’t complain about connecting to “outflux.net” when the cert has a CN for “www.outflux.net”:

echo QUIT | openssl s_client -CApath /etc/ssl/certs \
  -connect outflux.net:443 2>/dev/null | egrep "subject=|Verify"
subject=/CN=www.outflux.net
    Verify return code: 0 (ok)

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

Clearing /tmp on boot

Filed under: Blogging,Debian,Ubuntu,Ubuntu-Server — kees @ 3:48 pm

I don’t like unconditionally clearing /tmp on boot, since I’m invariably working on something in there when my system locks up. But I do like /tmp getting cleaned up from time to time. As a compromise, I’ve set TMPTIME=7 in /etc/default/rcS so that only stuff older than 7 days is deleted when I reboot.

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

March 8, 2010

Cylon Dorkshield

Filed under: Blogging,Debian,Embedded,Ubuntu — kees @ 8:42 pm

I’ve finally soldered on the LEDs for the Cylon DorkShield I designed and got built in a recent DorkBotPDX PCB run. I’m having trouble with the programmer, but I got the shield mostly working:

Cylon DorkShield

© 2010 – 2015, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

February 28, 2010

egosurfing with git

Filed under: Blogging,Debian,Ubuntu,Ubuntu-Server — kees @ 12:19 pm

I’m never sure when my commits to Linux actually make it into an official release. Luckily, since releases are tagged, I can use “git tag --contains COMMIT” to find them. So, in a stunning display of inefficiency, here’s what I use to find my commits:

git log --author='Kees Cook' --pretty=oneline | \
while read commit name; do \
    echo $(git tag --contains $commit | head -n1): "$name"; \
done

Which lets me know where my code is with respect to releases:

v2.6.33: x86, mm: Report state of NX protections during boot
v2.6.33: sysctl: require CAP_SYS_RAWIO to set mmap_min_addr
v2.6.32: proc: fix reported unit for RLIMIT_CPU
v2.6.31: modules: sysctl to block module loading
...

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

February 25, 2010

fun with process scheduling

Filed under: Debian,Ubuntu,Ubuntu-Server — kees @ 11:04 am

In an attempt to force an exec to totally stall so I could attach gdb to a process that does a setuid transition without freaking it out (i.e. can’t launch “sudo gdb …”), I briefly played with process priority, scheduling, and CPU affinity (pinning a process to a CPU).

So far, the best attempt at stalling the process was to set CPU affinity, set its scheduling policy (IDLE), drop the priority value fully (19), and then run a CPU hog with the same CPU affinity with a very high priority (-20).

Staller: sudo schedtool -a 0 -N -n -20 -e /tmp/spin-cpu
Stallee: schedtool -a 0 -D -n 19 -e /usr/bin/sudo -s

The hope was to then run “sudo gdb /usr/bin/sudo $(pidof sudo)“, but it seems that this still isn’t enough to make the exec of sudo totally stall while the “spin-cpu” process is running — it certainly executes more slowly, but I want it to stop.

I have this feeling like I’m missing something obvious about how to accomplish this gdb session sanely. I wonder if I can get gdb to switch targets across a child exec, i.e. run gdb as root on a program that fully drops privs, but then execs sudo. Guess it’s time to go read the gdb manual some more…

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

February 18, 2010

data mining for NX bit

Filed under: Blogging,Debian,Security,Ubuntu,Ubuntu-Server — kees @ 11:15 am

9% of Ubuntu systems that were used to report bugs that included their /proc/cpuinfo file need to fix their BIOS settings to gain the NX bit.

Check for yourself. (Run it with --verbose for useful details.)

Out of 7511 Ubuntu bugs Brian Murray collected for me that included /proc/cpuinfo files, there were 7270 unique contents (which surprised me — I was expecting this to be much lower).

  • 5 (0.07%) were non-x86.
  • 1 (0.02%) had corrupted contents (likely due to a search/replace in apport gone awry).
  • 5670 (77.99%) had NX (this also surprised me — I was not expecting it to be so high).
  • 337 (4.64%) lacked PAE, and so cannot have NX (I didn’t expect this to be so low; Ubuntu bug reporters must have relatively recent hardware overall).
  • 595 (8.18%) had PAE and correctly lacked NX (I didn’t expect this to be so high — PAE without NX is a bit more common than I’d hoped; hopefully these systems are running 32bit kernels to at least get the partial NX emulation).
  • 662 (9.1%) had PAE but incorrectly lacked NX.

It’s this last group of systems I’m hoping to get fixed through education.

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

February 9, 2010

easy example of filesystem capabilities

Filed under: Blogging,Debian,Networking,Security,Ubuntu,Ubuntu-Server — kees @ 11:15 am

Instead of using setuid programs, the goal for the future is to use fine-grained capabilities. For example, here is /bin/ping:

$ ls -la /bin/ping
-rwsr-xr-x 1 root root 35680 2009-11-05 00:41 /bin/ping
$ ./ping 127.0.0.1
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.041 ms

$ sudo setcap CAP_NET_RAW=ep /bin/ping
$ getcap /bin/ping
/bin/ping = cap_net_raw+ep
$ ./ping 127.0.0.1
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.041 ms

$ dmesg | tail -n1
[212275.772124] warning: `/bin/ping’ has both setuid-root and effective capabilities. Therefore not raising all capabilities.

The best part is that the kernel will choose the set of least privileges when both setuid and capabilities exist. Easy way to transition!

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

February 8, 2010

rng-tools with TPM

Filed under: Blogging,Debian,Security,Ubuntu,Ubuntu-Server — kees @ 7:32 pm

In Ubuntu, I uploaded an rng-tools that supports the RNG in TPM devices (my patch is waiting in Debian). This hardware is available on a bunch of systems, including several Thinkpads and the Intel Q35, Q45 and newer main boards.

While most TPM RNGs aren’t really heavy-duty hardware RNGs, they are at least a mild source of randomness. I’ll be using an entropy key eventually, but for now, the TPM can supplement my collected entropy.

/etc/default/rng-tools:

HRNGDEVICE=/dev/null
RNGDOPTIONS=”–hrng=tpm –fill-watermark=90% –feed-interval=1″

After it’s been running a bit:

Feb 8 19:10:51 linux rngd[13143]: stats: bits received from HRNG source: 6180064
Feb 8 19:10:51 linux rngd[13143]: stats: bits sent to kernel pool: 6166144
Feb 8 19:10:51 linux rngd[13143]: stats: entropy added to kernel pool: 4624608
Feb 8 19:10:51 linux rngd[13143]: stats: FIPS 140-2 successes: 309
Feb 8 19:10:51 linux rngd[13143]: stats: FIPS 140-2 failures: 0
Feb 8 19:10:51 linux rngd[13143]: stats: FIPS 140-2(2001-10-10) Monobit: 0
Feb 8 19:10:51 linux rngd[13143]: stats: FIPS 140-2(2001-10-10) Poker: 0
Feb 8 19:10:51 linux rngd[13143]: stats: FIPS 140-2(2001-10-10) Runs: 0
Feb 8 19:10:51 linux rngd[13143]: stats: FIPS 140-2(2001-10-10) Long run: 0
Feb 8 19:10:51 linux rngd[13143]: stats: FIPS 140-2(2001-10-10) Continuous run: 0
Feb 8 19:10:51 linux rngd[13143]: stats: HRNG source speed: (min=5.207; avg=6.145; max=6.200)Kibits/s
Feb 8 19:10:51 linux rngd[13143]: stats: FIPS tests speed: (min=66.925; avg=75.789; max=112.861)Mibits/s
Feb 8 19:10:51 linux rngd[13143]: stats: Lowest ready-buffers level: 0
Feb 8 19:10:51 linux rngd[13143]: stats: Entropy starvations: 308
Feb 8 19:10:51 linux rngd[13143]: stats: Time spent starving for entropy: (min=3150263; avg=3178447.994; max=3750848)us

And now the kernel entropy pool is high:

$ echo $(cat /proc/sys/kernel/random/entropy_avail)/$(cat /proc/sys/kernel/random/poolsize)
3968/4096

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

January 24, 2010

Google is wardriving

Filed under: Blogging,Debian,General,Networking,Security,Ubuntu,Web — kees @ 8:28 pm

So, a while back, Google started providing location services. This seemed pretty cool, but I kind of ignored it until recently when I was playing with my Android’s location API. With the GPS off, and no cell towers visible (my basement gets terrible cell service), my phone knew within about 500 feet of where it actually was. All I was connected to was my wifi.

Bottom line: it seems that Google, among other methods, is likely wardriving while photographing for Street View. They are now able to pinpoint wifi access points if they happened to see it while driving through your city.

I’m really rather astonished that no one is freaking out about this; I’m a bit unnerved. I implemented the location-of-your-wifi API quickly, so I could terrify myself further. You can do lookups via my location website too, if you want.

UPDATE: yeah, it would seem to be crowd-sourced wifi and cell tower triangulation data. I should say “Google is WarCrowding”.

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

January 19, 2010

Using huludesktop on MythTV

Filed under: Blogging,Security,Ubuntu — kees @ 11:29 am

Based on a friend’s recommendation, I decided I would install Hulu Desktop for my MythTV system.

The MythTV wiki instructions were very good. However, I didn’t like that it was a closed-source binary doing network traffic. (While “system” doesn’t show up in “readelf -r” output, that doesn’t mean it isn’t doing direct syscalls, or manually finding the “system” offset in the libc library, or is vulnerable to overflows, and on and on.) So, to put my mind at ease, I decided to confine it in an AppArmor profile:

#include <tunables/global>

/usr/bin/huludesktop {
#include <abstractions/gnome>
#include <abstractions/audio>
#include <abstractions/nameservice>

/etc/huludesktop/** r,
@{HOME}/.huludesktop rwkl,
@{HOME}/.local/share/.huludesktop.data rwkl,
@{HOME}/.macromedia/Flash_Player/macromedia.com/support/flashplayer/sys/*.hulu.com/** r,
@{HOME}/.macromedia/Flash_Player/#SharedObjects/ r,
@{HOME}/.macromedia/Flash_Player/#SharedObjects/*/*.hulu.com/ rw,
@{HOME}/.macromedia/Flash_Player/#SharedObjects/*/*.hulu.com/** rwkl,

# MythTV is already managing the screensaver
deny /usr/bin/xdg-screensaver x,
}

Additionally, I disabled its executable stack, which seems to serve no purpose:
$ sudo execstack -c /usr/bin/huludesktop

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

December 25, 2009

happy holidays 2009!

Filed under: Blogging,Debian,Ubuntu,Ubuntu-Server — kees @ 9:19 pm

Happy Holidays!

Or, as my firewall reported this morning:

Dec 25 04:39:18 firewall kernel: [1054241.000084] sd 2:0:0:0: [sda] ABORT operation started
...
Dec 25 04:40:10 firewall kernel: [1054292.072173] raid1: sdb2: redirecting sector 3490144 to another mirror

I guess I’m getting my firewall a new hard drive for Christmas. :)

© 2009, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

December 16, 2009

headache empathy

Filed under: Blogging,Debian,Multimedia,Ubuntu — kees @ 9:20 pm

Run “gstreamer-properties“, click the “Video” tab, change Default Input Plugin to “Custom”, and add this Pipeline:

v4l2src ! ffmpegcolorspace ! vertigotv ! ffmpegcolorspace

Now when Empathy video-chatting with a friend, you can give them a headache!

Or give yourself a headache by trying it directly from the command line:

gst-launch v4l2src ! ffmpegcolorspace ! vertigotv ! ffmpegcolorspace ! xvimagesink

Feel free to replace “vertigotv” with any other or more of the video effects listed in “gst-inspect effectv“.

Here’s me with edgetv ! vertigotv:

Edge Vertigo

© 2009 – 2015, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

« Newer PostsOlder Posts »

Powered by WordPress