codeblog code is freedom — patching my itch

June 21, 2010

kvm kernel console

Filed under: Blogging,Debian,Ubuntu,Ubuntu-Server — kees @ 12:50 pm

I know there are a lot of ways to do this, but I just wanted to note what works for me to catch early boot Oopses in the kernel. In virt-manager, I set up a serial device, as a “pty”. After the VM boots, I pause in Grub, and check the VM details for the serial port (it’ll show something like “/dev/pts/13”). In another terminal, “sudo cat /dev/pts/13”, and I boot the kernel with “console=ttyS0”. When the Oops happens, I get it in the terminal instead of having it scroll off the VM’s screen.

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

May 29, 2010

Linux Security Summit 2010

Filed under: Blogging,Debian,Security,Ubuntu,Ubuntu-Server — kees @ 5:22 pm

The Call For Participation is open for the 2010 Linux Security Summit, being held just before this year’s LinuxCon.

If you’re interested in helping make Linux more secure, you’ve got ideas to present, want to have your opinion heard, or generally just want to hang out, please join us and/or suggest a topic for discussion (CFP ends June 4th, so please hurry).

I’m hoping to get a chance to discuss what I’m calling the “popular kernel hardening patches” which appear in a lot of distros yet remain missing from the upstream Linux kernel.

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

May 17, 2010

yay for barriers

Filed under: Blogging,Debian,Ubuntu,Ubuntu-Server — kees @ 12:13 pm

I find it surreal to have people guessing at my motivations when they could just ask me. On top of all that, I find it weird that people spend so much time with in-fighting. I just want my system not to suck.

Some time ago (during in the Ubuntu Karmic development cycle maybe in September 2009), I started having giant problems with my build system. All I/O would start to stall, wait times would surge, and usually my entire system would just go unresponsive with the disk light on solid. This scared the crap out of me, and it wasn’t entirely obvious what was triggering it. No one else seemed to be seeing it. I managed to start tracking things using “latencytop”, and saw stuff like liferea going crazy. As I eliminated more and more things, I eventually settled on it being a problem with umount, and I reported an Ubuntu bug. It seemed to look like an upstream bug that no one else but the reporter could reproduce either.

Since no one else was seeing this issue, and it seemed related to LVM snapshots, I migrated off of snapshots, and started using aufs overlays for my builds. For a while, it seemed like things went away. It didn’t, and I started hitting it again. I opened a new (now famous) bug in Ubuntu, since now snapshots weren’t in the picture, and I didn’t want to confuse the earlier history. I managed to find a relatively minimal test-case too. A few other people commenting on the bug were seeing the problem now too, but it was less pronounced for them.

As an aside, this wasn’t a “just wait a few seconds longer” kind of issue on my system. A single umount would last 30-40 minutes. And when I’m doing parallel builds of security updates, this would turn into my system being unavailable for hours at a stretch.

Since none of the kernel developers I was in contact with were able to track down the root cause, I asked Ted Ts’o in email if he could just quickly peek in on this for me, since I figured he’d be in a good position to confirm or deny it. I didn’t want to start wasting upstream time with this if it wasn’t reproducible (see earlier upstream kernel bug). To my great relief, Ted found a few minutes to check it, and was able to immediately confirm it and give me a viable work-around (“sync; umount …”) for the time being. I confirmed the work-around, and went off to do other things.

A while later, Ted came back to deliver a bit of a rant, the purpose of which was not clear to me, but I ultimately ignored it — I didn’t seem directed at me. I just wanted my system operating normally, and he’d done me a favor to check in on it and got me a work-around.

More time passes, and I eventually get caught in another I/O-wait melt-down. On investigation, it seemed that the in-kernel work-around in the Ubuntu kernel totally back-fired on me in some cases, rendering even the user-space work-around useless. After investigating the Ubuntu-specific work-around, I re-read Ted’s rant in the course of researching what had happened during this bug’s triage.

It seemed that Ted was basically saying:
– this is an upstream problem
– RedHat hasn’t run into it and he didn’t know why

I figured I should confirm for myself if Fedora was affected, so downloaded and installed Fedora to double-check there. Since I was able to reproduce it there, I opened an upstream bug, linking back to the original Ubuntu bug, and then went to open a bug in the Fedora tracker, linking back to upstream.

And it seems to be these actions that everyone has jumped on. I will now bore you with the reality of my motivations: I wanted to fix the bug so no one would end up experiencing the same pain I’d been through over the last 6 months.

The bug was, from my perspective, a serious issue. Since I’d managed to reproduce it in another distro, it was my duty as a Free Software developer to report it to them. And, in what I felt was an unambiguous gesture, I made sure to include the link to the upstream kernel bug. Reproducing it in Ubuntu, in Fedora, and with a stock kernel had me confident that it was an upstream issue. While Ted did correctly suspect the issue was upstream, I really didn’t want to just open an upstream bug and have it be ignored. I wanted some additional proof of reproduction, which I got when I tested it on Fedora.

So, I’m rather saddened that so many people spent so much time questioning my motivations, making fun of Canonical, or doing anything other than trying to just simply solve this problem. I’m totally disinterested in inter-distro fighting. Instead, I continue to assume we’re all on the same team, fighting a philosophical battle against close-source software. And in that regard, I think it’s still true. If I ignore the rants and jeering, I come away thankful for all the people that spent time trying reproduce the issue at Canonical, at RedHat, and in the larger community. I’m hugely thankful that Ted made some time to let me know I wasn’t crazy, and there was actually a problem. I’m thankful for having some work-arounds, and I’m thankful that the root cause was eventually ferreted out, with some possible solutions. I’m even thankful that some people on the LWN thread saw that, far from malicious, I was trying to be helpful with the bug.

I just wanted my filesystem not to eat my computer. And I was hoping other people could maybe help me, since I’m not a filesystem expert. The drama around this bug is pathetic, and now by talking about it for almost 1000 words, I’m just as guilty.

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

April 29, 2010

Intrepid Inactive

Filed under: Blogging,Security,Ubuntu,Ubuntu-Server — kees @ 5:00 pm

Intrepid is now officially at end-of-life.

Looking back through my build logs, it seems my desktop did 1340 builds, spending 70 hours, 38 minutes, and 49 seconds doing builds during the development cycle of Intrepid. Once released, it performed an additional 123 builds, taking 19 hours, 29 minutes, and 48 seconds for security updates. As before, these times obviously don’t include patch hunting/development, failed builds, testing, stuff done on my laptop or the porting machines, etc.

Thank you Intrepid! You were the first release to carry the full set of by-default hardened compiler flags.

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

March 10, 2010

openssl client does not check commonName

Filed under: Blogging,Debian,Security,Ubuntu,Ubuntu-Server — kees @ 10:47 pm

I realize the openssl s_client tool tries to be upper-layer protocol agnostic, but doesn’t everything that uses SSL do commonName checking (HTTP, SMTP, IMAP, FTP, POP, XMPP)? Shouldn’t this be something openssl s_client does by default, maybe with an option to turn it off for less common situations?

Here it doesn’t complain about connecting to “outflux.net” when the cert has a CN for “www.outflux.net”:

echo QUIT | openssl s_client -CApath /etc/ssl/certs \
  -connect outflux.net:443 2>/dev/null | egrep "subject=|Verify"
subject=/CN=www.outflux.net
    Verify return code: 0 (ok)

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

Clearing /tmp on boot

Filed under: Blogging,Debian,Ubuntu,Ubuntu-Server — kees @ 3:48 pm

I don’t like unconditionally clearing /tmp on boot, since I’m invariably working on something in there when my system locks up. But I do like /tmp getting cleaned up from time to time. As a compromise, I’ve set TMPTIME=7 in /etc/default/rcS so that only stuff older than 7 days is deleted when I reboot.

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

February 28, 2010

egosurfing with git

Filed under: Blogging,Debian,Ubuntu,Ubuntu-Server — kees @ 12:19 pm

I’m never sure when my commits to Linux actually make it into an official release. Luckily, since releases are tagged, I can use “git tag --contains COMMIT” to find them. So, in a stunning display of inefficiency, here’s what I use to find my commits:

git log --author='Kees Cook' --pretty=oneline | \
while read commit name; do \
    echo $(git tag --contains $commit | head -n1): "$name"; \
done

Which lets me know where my code is with respect to releases:

v2.6.33: x86, mm: Report state of NX protections during boot
v2.6.33: sysctl: require CAP_SYS_RAWIO to set mmap_min_addr
v2.6.32: proc: fix reported unit for RLIMIT_CPU
v2.6.31: modules: sysctl to block module loading
...

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

February 25, 2010

fun with process scheduling

Filed under: Debian,Ubuntu,Ubuntu-Server — kees @ 11:04 am

In an attempt to force an exec to totally stall so I could attach gdb to a process that does a setuid transition without freaking it out (i.e. can’t launch “sudo gdb …”), I briefly played with process priority, scheduling, and CPU affinity (pinning a process to a CPU).

So far, the best attempt at stalling the process was to set CPU affinity, set its scheduling policy (IDLE), drop the priority value fully (19), and then run a CPU hog with the same CPU affinity with a very high priority (-20).

Staller: sudo schedtool -a 0 -N -n -20 -e /tmp/spin-cpu
Stallee: schedtool -a 0 -D -n 19 -e /usr/bin/sudo -s

The hope was to then run “sudo gdb /usr/bin/sudo $(pidof sudo)“, but it seems that this still isn’t enough to make the exec of sudo totally stall while the “spin-cpu” process is running — it certainly executes more slowly, but I want it to stop.

I have this feeling like I’m missing something obvious about how to accomplish this gdb session sanely. I wonder if I can get gdb to switch targets across a child exec, i.e. run gdb as root on a program that fully drops privs, but then execs sudo. Guess it’s time to go read the gdb manual some more…

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

February 18, 2010

data mining for NX bit

Filed under: Blogging,Debian,Security,Ubuntu,Ubuntu-Server — kees @ 11:15 am

9% of Ubuntu systems that were used to report bugs that included their /proc/cpuinfo file need to fix their BIOS settings to gain the NX bit.

Check for yourself. (Run it with --verbose for useful details.)

Out of 7511 Ubuntu bugs Brian Murray collected for me that included /proc/cpuinfo files, there were 7270 unique contents (which surprised me — I was expecting this to be much lower).

  • 5 (0.07%) were non-x86.
  • 1 (0.02%) had corrupted contents (likely due to a search/replace in apport gone awry).
  • 5670 (77.99%) had NX (this also surprised me — I was not expecting it to be so high).
  • 337 (4.64%) lacked PAE, and so cannot have NX (I didn’t expect this to be so low; Ubuntu bug reporters must have relatively recent hardware overall).
  • 595 (8.18%) had PAE and correctly lacked NX (I didn’t expect this to be so high — PAE without NX is a bit more common than I’d hoped; hopefully these systems are running 32bit kernels to at least get the partial NX emulation).
  • 662 (9.1%) had PAE but incorrectly lacked NX.

It’s this last group of systems I’m hoping to get fixed through education.

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

February 9, 2010

easy example of filesystem capabilities

Filed under: Blogging,Debian,Networking,Security,Ubuntu,Ubuntu-Server — kees @ 11:15 am

Instead of using setuid programs, the goal for the future is to use fine-grained capabilities. For example, here is /bin/ping:

$ ls -la /bin/ping
-rwsr-xr-x 1 root root 35680 2009-11-05 00:41 /bin/ping
$ ./ping 127.0.0.1
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.041 ms

$ sudo setcap CAP_NET_RAW=ep /bin/ping
$ getcap /bin/ping
/bin/ping = cap_net_raw+ep
$ ./ping 127.0.0.1
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.041 ms

$ dmesg | tail -n1
[212275.772124] warning: `/bin/ping’ has both setuid-root and effective capabilities. Therefore not raising all capabilities.

The best part is that the kernel will choose the set of least privileges when both setuid and capabilities exist. Easy way to transition!

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

February 8, 2010

rng-tools with TPM

Filed under: Blogging,Debian,Security,Ubuntu,Ubuntu-Server — kees @ 7:32 pm

In Ubuntu, I uploaded an rng-tools that supports the RNG in TPM devices (my patch is waiting in Debian). This hardware is available on a bunch of systems, including several Thinkpads and the Intel Q35, Q45 and newer main boards.

While most TPM RNGs aren’t really heavy-duty hardware RNGs, they are at least a mild source of randomness. I’ll be using an entropy key eventually, but for now, the TPM can supplement my collected entropy.

/etc/default/rng-tools:

HRNGDEVICE=/dev/null
RNGDOPTIONS=”–hrng=tpm –fill-watermark=90% –feed-interval=1″

After it’s been running a bit:

Feb 8 19:10:51 linux rngd[13143]: stats: bits received from HRNG source: 6180064
Feb 8 19:10:51 linux rngd[13143]: stats: bits sent to kernel pool: 6166144
Feb 8 19:10:51 linux rngd[13143]: stats: entropy added to kernel pool: 4624608
Feb 8 19:10:51 linux rngd[13143]: stats: FIPS 140-2 successes: 309
Feb 8 19:10:51 linux rngd[13143]: stats: FIPS 140-2 failures: 0
Feb 8 19:10:51 linux rngd[13143]: stats: FIPS 140-2(2001-10-10) Monobit: 0
Feb 8 19:10:51 linux rngd[13143]: stats: FIPS 140-2(2001-10-10) Poker: 0
Feb 8 19:10:51 linux rngd[13143]: stats: FIPS 140-2(2001-10-10) Runs: 0
Feb 8 19:10:51 linux rngd[13143]: stats: FIPS 140-2(2001-10-10) Long run: 0
Feb 8 19:10:51 linux rngd[13143]: stats: FIPS 140-2(2001-10-10) Continuous run: 0
Feb 8 19:10:51 linux rngd[13143]: stats: HRNG source speed: (min=5.207; avg=6.145; max=6.200)Kibits/s
Feb 8 19:10:51 linux rngd[13143]: stats: FIPS tests speed: (min=66.925; avg=75.789; max=112.861)Mibits/s
Feb 8 19:10:51 linux rngd[13143]: stats: Lowest ready-buffers level: 0
Feb 8 19:10:51 linux rngd[13143]: stats: Entropy starvations: 308
Feb 8 19:10:51 linux rngd[13143]: stats: Time spent starving for entropy: (min=3150263; avg=3178447.994; max=3750848)us

And now the kernel entropy pool is high:

$ echo $(cat /proc/sys/kernel/random/entropy_avail)/$(cat /proc/sys/kernel/random/poolsize)
3968/4096

© 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

December 25, 2009

happy holidays 2009!

Filed under: Blogging,Debian,Ubuntu,Ubuntu-Server — kees @ 9:19 pm

Happy Holidays!

Or, as my firewall reported this morning:

Dec 25 04:39:18 firewall kernel: [1054241.000084] sd 2:0:0:0: [sda] ABORT operation started
...
Dec 25 04:40:10 firewall kernel: [1054292.072173] raid1: sdb2: redirecting sector 3490144 to another mirror

I guess I’m getting my firewall a new hard drive for Christmas. :)

© 2009, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

December 9, 2009

install from official repositories only

Filed under: Debian,Security,Ubuntu,Ubuntu-Server — kees @ 10:02 am

As quickly pointed out by Rick, don’t install random software that isn’t in the official distribution archive unless you really know what you’re doing (and copy/pasting commands from a website doesn’t count). You’re just asking to be made part of a botnet.

© 2009, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

October 24, 2009

karmic and log rotation

Filed under: Debian,Ubuntu,Ubuntu-Server — kees @ 10:00 am

In Ubuntu’s Karmic and and Debian’s Lenny, sysklogd was replaced with rsyslog. This is fine, since rsyslog will have converted your /etc/syslog.conf to /etc/rsyslog.d/50-default.conf. However, if you modified the (maddeningly strange sysklogd-specific) log file rotation in /etc/cron.daily/sysklogd or /etc/cron.weekly/sysklogd, you’ll want to review the new (sane) /etc/logrotate.d/rsyslog. (Note also that savelog uses .0 as the first rotated file extension, and logrotate uses .1.)

© 2009, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

October 22, 2009

TPM as RNG

Filed under: Debian,Security,Ubuntu,Ubuntu-Server — kees @ 10:43 pm

I was reminded about some TPM coding I’d done to get random bytes from the pRNG on my TPM-enabled system from Matt Domsch’s recent post. I’m not fully convinced that the pRNG of the TPM is an appropriate source of entropy, but it does pass my simple FIPS-140-2 test.

I had to find the Intel TPM docs to figure out how to enable TPM on my system. It was under “Advanced / Peripherals”. I was expecting it under “Security”, like every other BIOS I’d seen. After that:

$ sudo apt-get install trousers tpm-tools
...
$ sudo modprobe tpm_tis
$ dmesg | grep -i tpm
[676618.167313] tpm_tis 00:07: 1.2 TPM (device-id 0xFE, rev-id 70)
$ sudo service trousers start
...
$ tpm_version
TPM 1.2 Version Info:
Chip Version: 1.2.2.16
Spec Level: 2
Errata Revision: 1
TPM Vendor ID: WEC
TPM Version: 01010000
Manufacturer Info: 57454300
$ ./tpm-getrand | hexdump -C
00000000 61 07 23 ff 71 3e 25 e8 f0 d5 de a7 a3 07 21 dc |a.#.q>%.......!.|

I could run rngd with a named pipe, but it’d be nice to have a new driver that could run a command instead to get the next 20000 bits.

UPDATE: I’ve implemented this in rngd now.

© 2009, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

September 12, 2009

uninstall sun-java6

Filed under: Blogging,Debian,Ubuntu,Ubuntu-Server — kees @ 11:59 am

With the vrms meme raging on Planet Ubuntu, I noticed some people still have sun-java6 installed. I’ve been using openjdk-6 since Hardy, and everything I use works fine with it (e.g. Vuze, Catan, Eclipse, FreeMind, and even Facebook’s photo uploader thing). Given the Ubuntu Tech Board’s “remove sun-java6 from the archive” Agenda item, it’s possible sun-java6 (being redundant and orphaned) would go away in Ubuntu Karmic (sun-java5 is already gone because Sun will drop support for it in October, and sun-java6 is not far behind).

I’ve heard some noises about openjdk-6 not working for people, but I haven’t actually seen any direct technical evidence of something working in one and not the other. Seeing as Jaunty’s OpenJDK was certified by Sun, it would be very interesting to find failures, as this would indicate that Sun’s certifications are missing something.

Looking for examples of failures in Google, I couldn’t find anything with obvious test-cases that failed with openjdk-6. I suspect I’m just not trying hard enough, but I’m curious what other people have run into.

Test for yourself:

sudo apt-get purge sun-java6-jdk sun-java6-jre sun-java6-bin sun-java6-plugin
sudo apt-get install openjdk-6-jdk icedtea6-plugin

(Don’t forget to restart your browser.)

© 2009 – 2016, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

July 31, 2009

blocking module loading

Filed under: Blogging,Debian,Security,Ubuntu,Ubuntu-Server — kees @ 11:47 am

New for Linux 2.6.31 (and Ubuntu 9.10) is the ability to throw a one-way toggle to block module loading via /proc/sys/kernel/modules_disabled:

# uname -a
Linux sec-karmic-amd64 2.6.31-4-generic #23-Ubuntu SMP Mon Jul 27 18:39:59 UTC 2009 x86_64 GNU/Linux
# lsmod | head -n3
Module Size Used by
binfmt_misc 10220 1
ppdev 8200 0
# cat /proc/sys/kernel/modules_disabled
0
# modprobe usb-storage
# lsmod | head -n3
Module Size Used by
usb_storage 65600 0
binfmt_misc 10220 1
# echo 1 > /proc/sys/kernel/modules_disabled
# rmmod usb-storage
ERROR: Removing 'usb_storage': Operation not permitted
# modprobe zlib_deflate
FATAL: Error inserting zlib_deflate (/lib/modules/2.6.31-4-generic/kernel/lib/zlib_deflate/zlib_deflate.ko): Operation not permitted
# echo 0 > /proc/sys/kernel/modules_disabled
bash: echo: write error: Invalid argument
# cat /proc/sys/kernel/modules_disabled
1

The intent is for this to allow paranoid server admins (or other people not expecting to hot-plug new hardware or kernel services) the ability to block module loading without compiling a monolithic kernel.

This kind of thing used to be available through the “lcap” utility modifying the global capability bounding set (which was removed in 2.6.25), but init could always be made to turn it back on.

Combined with the removal of /dev/kmem and the hardening of /dev/mem, this closes another kernel rootkit door. It’s not a cure-all, but it’s another layer.

Now we just have to figure out ways to stamp out unexpected ioport-triggered DMA access.

© 2009, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

July 16, 2009

Diminished Dapper

Filed under: Blogging,Security,Ubuntu,Ubuntu-Server — kees @ 10:18 am

Dapper has reached it’s half-way support age. This means that only shipped server packages are getting official support. If you’re still running Dapper, upgrade to Hardy (and then consider upgrading through Intrepid to Jaunty too, since Intrepid and newer has awesome compiler defaults).

Looking back through my build logs, I can see that my desktop spent 51 hours, 48 minutes, and 37 seconds building 389 security updates. As before, these times obviously don’t include patch hunting/development, failed builds, testing, stuff done on my laptop or the porting machines, etc.

Current combined devel/security build standings per release:

hardy: 172:23:11
intrepid: 85:34:53
jaunty: 20:11:53

Thank you Dapper Desktop! I am reminded what you look like each time I start a Dapper VM. If not for that, my memory of you would have diminished long ago.

© 2009, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

April 17, 2009

Goodbye Gutsy

Filed under: Blogging,Security,Ubuntu,Ubuntu-Server — kees @ 3:05 pm

(In reading my prior EOL posts, I realize I should have said “Enough Edgy” or something like that to be appropriately alliterative.)

Gutsy is now officially at end-of-life.

Looking back through my build logs, I can see that my desktop spent 25 hours, 6 minutes, and 47 seconds building 208 security updates. (And 18 hours, 23 minutes, 45 seconds doing 335 builds during the Gutsy devel window.) As before, these times obviously don’t include patch hunting/development, failed builds, testing, stuff done on my laptop or the porting machines, etc.

Current combined devel/security build standings per release:

dapper: 51:27:29
hardy: 171:21:40
intrepid: 84:23:19
jaunty: 18:43:09

Thank you Gutsy! You were much nicer than Feisty, especially for wifi on my poor laptop.

© 2009, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

April 9, 2009

CODEGATE 2009 ran Ubuntu Intrepid

Filed under: Blogging,Security,Ubuntu,Ubuntu-Server — kees @ 9:05 pm

I’m pleased to hear than the CODEGATE 2009 International Hacking Contest ran Ubuntu Intrepid for both the qualifying rounds and the final match. The host machine was running the 32bit -server kernel, so NX was active (along with all the other memory-corruption protections). From what I can see the compiled challenges were stack-protected and fortified, so the contestants had to work around that as well as the randomized stack, heap, and library locations. Sounds like it was a fun contest in a real-world situation. All that’s left now is for 64bit to become the standard. And PIE too. I can’t wait to hear more. :)

$ nm cracktris | grep _chk
         U __fprintf_chk@@GLIBC_2.3.4
         U __printf_chk@@GLIBC_2.3.4
         U __sprintf_chk@@GLIBC_2.3.4
         U __stack_chk_fail@@GLIBC_2.4
         U __strcat_chk@@GLIBC_2.3.4
         U __strcpy_chk@@GLIBC_2.3.4

© 2009, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

January 24, 2009

debugging hangs in MIMEDefang

Filed under: Blogging,Debian,Ubuntu,Ubuntu-Server — kees @ 3:21 pm

I spent a while this morning hunting down the root cause of a 100% CPU MIMEDefang process. (Turned out to be one specific email triggering one of my custom SpamAssassin rules that seems to get processed really inefficiently.) I wasn’t able to find any documentation on how to debug where in MIMEDefang an email was getting stuck, and even jacking up the debugging output of my mimedefang-filter script seemed to imply it was happening late in the filtering process. After digging through mimedefang.pl, I found a “status reporting” subsystem and tracked it back out into configuration in /etc/default/mimedefang:

# Uncomment next line to have busy slaves send status updates to the
# multiplexor. NOTE: Consumes one extra file descriptor per slave, plus
# a bit of CPU time.
# MX_STATUS_UPDATES=yes

Turning this on, and then using “md-mx-ctrl busyslaves” (I command I didn’t know existed until today either) I got to see exactly where it was stuck:

3 B 13380 (Running SpamAssassin j951iX1t025805)

Now I’m having fun running md-mx-ctl with histo, barstatus, and load. :)

© 2009, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

January 13, 2009

ETOOMANYCERTS

Filed under: Debian,Networking,Ubuntu,Ubuntu-Server — kees @ 10:46 pm

After upgrading an Ubuntu mail server from Hardy to Intrepid, two users could no longer connect via SSL to send email though sendmail. One was using msmtp and the other was using Outlook Express. The msmtp issue was tracked down as a supposed deficiency in msmtp. However, this left Outlook, which is neigh-impossible to debug. From the Debian msmtp bug linked from the Ubuntu bug, it seemed that the root cause was the server sending too much data during the initial connection. Packet captures of an Outlook connection seemed to back this up: Outlook negotiated STARTTLS fine, and then just never responded to the SSL handshake.

It seems that something (openssl? sendmail?) changed between Hardy and Intrepid so that instead of using the /etc/ssl/certs/ca-certificates.crt file just for verification, its contents were now being sent during the SSL handshake. (I reduced the number of configured certs with “sudo dpkg-reconfigure ca-certificates“, and checked on the size of the handshake with “openssl s_client -connect server:port | wc -l“.) It spewed 143 certs sent at every connection. Unsurprisingly, it seems some clients were choking on it (I would like to note that Thunderbird behaved correctly).

In the end, I configured my sendmail’s CAfile (“confCACERT”) to aim at just a single CA (the CA used to sign the server’s SSL key), and that fixed both msmtp and Outlook. How fun.

© 2009 – 2010, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

January 1, 2009

happy new year, RIP md5

Filed under: Blogging,Debian,Security,Ubuntu,Ubuntu-Server — kees @ 10:59 am

Welcome to 2009! Really, seriously, everyone can stop using MD5 now. However, be sure not to overreact. The forged CA research is a great read, but there’s no need to break existing certs. For a calm reaction, please read this.

© 2009, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

December 3, 2008

bogosec run on intrepid main

Filed under: Blogging,Security,Ubuntu,Ubuntu-Server — kees @ 10:16 am

Care of Mike Owens and Dustin Kirkland, bogosec has been uploaded to Jaunty (in the NEW queue at the moment). It is a source-code analyzer framework with plugins for lintian, rats, and flawfinder. Out of curiousity, I ran it on all of Intrepid main. Highest 5 scores were:

  1. 0.717338929043293 lsscsi
  2. 0.612729234088457 nevow
  3. 0.561151781356762 powertop
  4. 0.431034482758621 language-pack-tk-base
  5. 0.431034482758621 language-pack-se-base

As Dustin reminded me, bogosec seems biased against smaller code bases. In the case of the lang packs, the score is entirely from lintian. Both lsscsi and powertop deal mostly with input from kernel strings, so while they scored highly, I doubt either is actually vulnerable to very much. I haven’t looked at nevow yet. Also, both rats and flawfinder yell about things that are mitigated by compiler flags (e.g. -D_FORTIFY_SOURCE=2) so those warnings are less interesting too.

Really, this all boils down to “we need better code analyzers”. The best tool will be one that predicts CVE counts (I would expect the Linux kernel to be at the top, since it has the all-time highest number of CVEs filed against it).

To get closer to reality, I think just doing a normal package build and scanning for stderr output would be meaningful (gcc has plenty of built-in checks already). Steve Beattie suggested writing a plugin for sparse, too.

© 2008, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

November 21, 2008

make your BIOS love security

Filed under: Security,Ubuntu,Ubuntu-Server — kees @ 1:39 pm

There’s this great CPU feature called “nx” — it protects your computer from intrusion by blocking execution of memory regions that weren’t expected to be executable (i.e. stack/heap data). You really want this enabled. Unfortunately for you, it seems some BIOSes default to disabling it. On Dell laptops, look under “Security” / “CPU XD Support”: you want it enabled. In an American Megatrends BIOS, I found it under “CPU Features” / “Execute Disable Bit”: you want it enabled.

As far as making use of the CPU feature once it’s not disabled in the BIOS, you’re already using it if you’re running a 64bit kernel. And if you’re using 32bit, you can start using it if you install the -server flavor of the 32bit kernel. As a bonus, you get to address all your physical RAM if you do this too (since -server’s “PAE” mode is the kernel mode that allows “nx” to work). For Ubuntu Jaunty, I’m hoping to get some element of the system (installer? jockey?) to make the right kernel selection for a given system.

If “pae” is in your /proc/cpuinfo flags:

$ grep --color pae /proc/cpuinfo
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov \
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc \
arch_perfmon pebs bts rep_good nopl pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr lahf_lm

(note the “nx” in there too, since my BIOS isn’t set to disable it)

Then you almost certainly want to use -server kernel flavor:

sudo apt-get install linux-server linux-restricted-modules-server

© 2008, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

November 3, 2008

days since last incident…

Filed under: Blogging,Debian,Networking,Security,Ubuntu,Ubuntu-Server — kees @ 11:15 am

If I made one of those work-site signs that tracked “Days since last incident”, and made one for “Days since last in-the-wild remote-root worm” for Windows and Linux, what would they each say? 0 and 7304 respectively?

Update: while the post was tongue-in-cheek (everyone suffers when any large subset of computers is being attacked), I should lower the Linux days count to 2783 (for L10n on March 23, 2001, which is slightly newer than Ramen on January 17, 2001). Thanks for everyone’s comments. :)

© 2008, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

October 30, 2008

how to drain your entropy and have fun with ssh fingerprint ASCII-art

Filed under: Blogging,Debian,General,Ubuntu,Ubuntu-Server — kees @ 10:48 am

SSH’s new “VisualHostKey” option (in Ubuntu Intrepid and Debian Lenny) is great fun. Normally it is disabled, but it seems that “ssh-keygen” turns it on when generating new keys. In celebration of the Ubuntu release, here is a script to entertain yourself with RSA ASCII-art, care of SSH and your system’s entropy pool:


#!/bin/sh
set -e
DIR=$(mktemp -t -d rsa-art-XXXXXX)
trap "rm -f $DIR/key*; rmdir $DIR" EXIT HUP INT QUIT TERM

while :
do
    ART=$(ssh-keygen -t rsa -f $DIR/key -N "" | tail -n 11)
    rm -f $DIR/key
    /bin/echo -e "\x1Bc"
    echo "$ART"
done

Makes me feel like I’m watching Life. (Use control-C to stop it.)

© 2008, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

October 21, 2008

Feisty Finale

Filed under: Blogging,Security,Ubuntu,Ubuntu-Server — kees @ 3:59 pm

Feisty is now officially at end-of-life.

Looking back through my build logs, I can see that my desktop spent 34 hours, 44 minutes, and 46 seconds building 255 security updates. (And 25 hours, 40 minutes, 13 seconds doing 249 builds during the Feisty devel window.) As before, these times obviously don’t include patch hunting/development, failed builds, testing, stuff done on my laptop or the porting machines, etc.

As a correction to the Edgy EOL post, my desktop actually spent 50:59:40 doing 322 security builds and 04:14:23 doing 84 devel builds.

Current standings:

dapper: 49:42:36
gutsy: 43:14:36
hardy: 166:11:15
intrepid: 70:24:52

As mentioned, these numbers are mixed devel/security times.

Thank you Feisty! You were much more stable than Edgy, even if we didn’t see eye-to-eye about wifi connectivity.

© 2008, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

September 4, 2008

all PIE distro

Filed under: Blogging,Security,Ubuntu,Ubuntu-Server — kees @ 2:00 pm

Major props to NCommander for taking on the painful experiment of getting the entire Ubuntu Intrepid archive rebuilt with PIE on amd64. After getting all the other hardening defaults enabled for Intrepid, PIE is the last on the original list for enabling “by default”. Due to the overhead of PIE on i386, it’s really only an option on architectures with lots of general registers.

© 2008, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

September 3, 2008

kvm disk image filesystem growth notes

Filed under: Blogging,Debian,Ubuntu,Ubuntu-Server — kees @ 12:14 pm

Here are my notes on growing a KVM disk image’s root filesystem. I had a few 4G partitions that really needed to be bigger. This shows how to get a report on the sizes of the disk images, convert them to raw, work on the partition tables, grow the root filesystem, and rebuild the swap partition with the original UUID. With some work, it could probably become fully scripted, but since the partition layout may not always be the same from VM to VM, the “fdisk” step needs human interaction to delete and rebuild the partition table. Note that the method below also maintains the sparseness of the images.

# Look for files to change
for i in /vmware/*/*{vmdk,qcow2}; do qemu-img info $i; done
...

# Pick one...
cd dir...
ORIG=64bit-Ubuntu-7.10-desktop.vmdk
SIZE=8G


ORIG_TYPE=$(echo $ORIG | awk -F. '{print $NF}')
TARGET_TYPE="qcow2"
TARGET_BASE=$(basename "$ORIG" ."$ORIG_TYPE")
TARGET_RAW="$TARGET_BASE".raw
TARGET="$TARGET_BASE"."$TARGET_TYPE"

qemu-img convert -f "$ORIG_TYPE" "$ORIG" -O raw "$TARGET_RAW"

trunc "$TARGET_RAW" "$SIZE"

sudo kpartx -a "$TARGET_RAW"
SWAP_PART=$(for i in /dev/mapper/loop0p*; do sudo vol_id $i | \
    grep -q ^ID_FS_TYPE=swap && echo $i; done | head -n 1)
UUID=$(sudo vol_id "$SWAP_PART" | grep ^ID_FS_UUID= | cut -d= -f2)
sudo kpartx -d "$TARGET_RAW"

# use losetup otherwise fdisk doesn't know cylinder count
sudo losetup /dev/loop0 "$TARGET_RAW"
# FIXME: Need to automate fdisk (detect swap partition size, etc)
# I'm deleting the swap and growing the root partition, then re-adding swap
sudo fdisk /dev/loop0
sudo losetup -d /dev/loop0

sudo kpartx -a "$TARGET_RAW"
sudo e2fsck -f /dev/mapper/loop0p1
sudo resize2fs /dev/mapper/loop0p1
sudo mkswap -U "$UUID" "$SWAP_PART"
sudo kpartx -d "$TARGET_RAW"

qemu-img convert -f raw "$TARGET_RAW" -O "$TARGET_TYPE" "$TARGET"
rm "$TARGET_RAW"
# FIXME: change disk image path
sudo vi /etc/libvirt/qemu/THING
# FIXME: have the daemon notice the file change
sudo /etc/init.d/libvirt-bin restart
if [ "$ORIG" != "$TARGET" ]; then rm "$ORIG"; fi

The “trunc” command above is based on my network backups post, but is now a general script I use:

#!/usr/bin/perl
# Copyright (C) 2006-2008 Kees Cook <kees@outflux.net>, License: GPLv3
use strict;
use warnings;

my $filename = $ARGV[0];
die "Need valid size also\n" unless ($ARGV[1] =~ /^(\d+)([KMG])$/);
my $size       = $1 + 0;
my $multiplier = $2;

$size *= 1024 if $multiplier =~ /^[KMG]$/;
$size *= 1024 if $multiplier =~ /^[MG]$/;
$size *= 1024 if $multiplier =~ /^[G]$/;

#die "Not trunc'ing existing file\n" if (-e $filename);
die "$filename: $!\n" if (!open(FILE,">>$filename"));
die "seek: $!\n" if (!(seek(FILE,$size,0)));
die "truncate: $!\n" if (!(truncate(FILE,$size)));
die "close: $!\n" if (!(close(FILE)));

© 2008, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

« Newer PostsOlder Posts »

Powered by WordPress