Recently I was trying to help debug a stack overflow crash in wpa_supplicant. The trouble with a stack crash is that you end up without a useful call history since the stack is left partially wrecked. The compiler code for detecting stack overflows (SSP), sets up a canary value between the local variables of the function and the stack frame. When the function exits, it tests this canary value and aborts if it doesn’t match what it is expecting. So, logically, to catch the stack overflow, gdb needs to be set up in a way to watch the canary location too. Since the canary is only valid while in the function, gdb must be set up to have a memory watch only when the function is called.
Here is the function preamble:
0x08081940 <wpa_driver_wext_get_scan_results+0>: push %ebp 0x08081941 <wpa_driver_wext_get_scan_results+1>: mov %esp,%ebp 0x08081943 <wpa_driver_wext_get_scan_results+3>: push %edi 0x08081944 <wpa_driver_wext_get_scan_results+4>: push %esi 0x08081945 <wpa_driver_wext_get_scan_results+5>: push %ebx
Save registers, prepare %ebp.
0x08081946 <wpa_driver_wext_get_scan_results+6>: mov $0x1000,%ebx 0x08081951 <wpa_driver_wext_get_scan_results+17>: mov 0x8(%ebp),%eax 0x08081954 <wpa_driver_wext_get_scan_results+20>: mov 0xc(%ebp),%edx 0x08081957 <wpa_driver_wext_get_scan_results+23>: lea 0xffffffb0(%ebp),%esi
Make room for local variables, copy some function arguments and local variables into registers.
0x0808195a <wpa_driver_wext_get_scan_results+26>: mov %gs:0x14,%ecx 0x08081961 <wpa_driver_wext_get_scan_results+33>: mov %ecx,0xffffffec(%ebp) 0x08081964 <wpa_driver_wext_get_scan_results+36>: xor %ecx,%ecx
Here’s the stack canary getting set, and the register cleared. It’s saved at %ebp minus 0x14 (0xffffffec signed is -0x14):
(gdb) printf "0x%x\n", 0-0xffffffec 0x14
Now for the function play-out:
0x08081a37 <wpa_driver_wext_get_scan_results+247>: mov 0xffffffec(%ebp),%edx 0x08081a3a <wpa_driver_wext_get_scan_results+250>: xor %gs:0x14,%edx 0x08081a41 <wpa_driver_wext_get_scan_results+257>: jne 0x8081eae <wpa_driver_wext_get_scan_results+1390>
There is the canary check.
0x08081a47 <wpa_driver_wext_get_scan_results+263>: add $0xec,%esp 0x08081a4d <wpa_driver_wext_get_scan_results+269>: pop %ebx 0x08081a4e <wpa_driver_wext_get_scan_results+270>: pop %esi 0x08081a4f <wpa_driver_wext_get_scan_results+271>: pop %edi 0x08081a50 <wpa_driver_wext_get_scan_results+272>: pop %ebp 0x08081a51 <wpa_driver_wext_get_scan_results+273>: ret ... 0x08081eae <wpa_driver_wext_get_scan_results+1390>: call 0x804bdc8 <__stack_chk_fail@plt>
Release local stack, pop saved registers and return. Nearer the end is the call to __stack_chk_fail when the canary doesn’t match.
So, to watch the canary, we need to set up a memory watch after it as been set, and tear it down before we leave the function. Respectively, we can use addresses 0x08081964 and 0x08081a3a (in bold above):
(gdb) br *0x08081964 Breakpoint 1 at 0x8081964 (gdb) br *0x08081a3a Breakpoint 2 at 0x8081a3a
At the first breakpoint, we set a memory watch using a gdb-local variable, based on %ebp (we can’t use %ebp directly since it will change in lower function calls):
(gdb) commands 1 Type commands for when breakpoint 1 is hit, one per line. End with a line saying just "end". >silent >set variable $cow = (unsigned long*)($ebp - 0x14) >watch *$cow >cont >end
Since I couldn’t find an easy way to track the memory watch number that was created during the first breakpoint, I just built a gdb counter, and deleted the memory watch when leaving, since I could predict gdb’s numbering (first watch will be “3”, following our breakpoints 1 and 2):
(gdb) set variable $count = 3 (gdb) commands 2 Type commands for when breakpoint 2 is hit, one per line. End with a line saying just "end". >silent >delete $count >set variable $count = $count + 1 >cont >end
Now we can run, and wait for the canary to get overwritten:
(gdb) cont Continuing. Hardware watchpoint 3: *$cow Hardware watchpoint 4: *$cow ... Hardware watchpoint 12: *$cow Hardware watchpoint 13: *$cow Hardware watchpoint 13: *$cow Old value = 4278845440 New value = 4278845546 0x0804eae6 in ?? ()
We see the canary value is 0xFF0A0000 getting it’s little-endian first byte overwritten to FF0A006A. We catch it before it has wrecked the stack, and we can see very clearly where we are:
(gdb) bt #0 hexstr2bin (hex=0x080a239d "6151663870517a74", buf=0x080a2395 "aQf8pQzt00000000j", len=8) at ../src/utils/common.c:88 #1 0x08082297 in wpa_driver_wext_get_scan_results (priv=0xb7dd816c, results=0x080a239d, max_size=0x79) at ../src/drivers/driver_wext.c:1383 ... (gdb) x/1i $eip 0x804eae6 <hexstr2bin +54>: addl $0x1,0xfffffff0(%ebp)
On a closer look at the source, we realize wext_get_scan_custom got inlined into the function (it was static and only called from one place, so the compiler optimized it). Further tracking in the source shows that the “16” value passed in should actually be “8” (the limit of the destination, not the source, buffer size).
© 2007, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
Thank you so much for this!
This was immensely helpful to me in tracking down a bug. I was about to re-write a massive function, but this saved me :)
Comment by milo — August 27, 2009 @ 10:37 pm