{"id":1369,"date":"2019-11-20T21:09:29","date_gmt":"2019-11-21T05:09:29","guid":{"rendered":"https:\/\/outflux.net\/blog\/?p=1369"},"modified":"2020-08-03T14:51:50","modified_gmt":"2020-08-03T21:51:50","slug":"experimenting-with-clang-cfi-on-upstream-linux","status":"publish","type":"post","link":"https:\/\/outflux.net\/blog\/archives\/2019\/11\/20\/experimenting-with-clang-cfi-on-upstream-linux\/","title":{"rendered":"experimenting with Clang CFI on upstream Linux"},"content":{"rendered":"

While much of the work on kernel Control Flow Integrity (CFI) is focused on arm64 (since kernel CFI is available on Android<\/a>), a significant portion is in the core kernel itself (and especially the build system). Recently I got a sane build and boot on x86 with everything enabled, and I’ve been picking through some of the remaining pieces. I figured now would be a good time to document everything I do to get a build working in case other people want to play with it and find stuff that needs fixing.<\/p>\n

First, everything is based on Sami Tolvanen’s upstream port of Clang’s forward-edge CFI<\/a>, which includes his Link Time Optimization (LTO) work, which CFI requires. This tree also includes his backward-edge CFI work on arm64 with Clang’s Shadow Call Stack (SCS).<\/p>\n

On top of that, I’ve got a few x86-specific patches that get me far enough to boot a kernel without warnings pouring across the console. Along with that are general linker script cleanups, CFI cast fixes, and x86 crypto fixes, all in various states of getting upstreamed. The resulting tree is here<\/a>.<\/p>\n

On the compiler side, you need a very recent Clang and LLD (i.e. “Clang 10”, or what I do is build from the latest git). For example, here’s how to get started. First, checkout, configure, and build Clang (leave out “--depth=1<\/code>” if you want the full git history):<\/p>\n

\r\n# Check out latest LLVM\r\nmkdir -p $HOME\/src\r\ncd $HOME\/src\r\ngit clone --depth=1 https:\/\/github.com\/llvm\/llvm-project.git\r\nmkdir -p llvm-build\r\ncd llvm-build\r\n# Configure\r\nmkdir -p $HOME\/bin\/clang-release\r\ncmake -G Ninja \\\r\n      -DCMAKE_BUILD_TYPE=Release \\\r\n      -DLLVM_ENABLE_PROJECTS='clang;lld;compiler-rt' \\\r\n      -DCMAKE_INSTALL_PREFIX=\"$HOME\/bin\/clang-release\" \\\r\n      ..\/llvm-project\/llvm\r\n# Build!\r\nninja install\r\n<\/pre>\n
Then checkout, configure, and build the CFI tree. (This assumes you’ve already got a checkout of Linus’s tree.)<\/p>\n
\r\n# Check out my branch\r\ncd ..\/linux\r\ngit remote add kees https:\/\/git.kernel.org\/pub\/scm\/linux\/kernel\/git\/kees\/linux.git\r\ngit fetch kees\r\ngit checkout kees\/kspp\/cfi\/x86 -b test\/cfi\r\n# Use the above built Clang path first for the needed binaries: clang, ld.lld, and llvm-ar.\r\nPATH=\"$HOME\/bin\/clang-release\/bin:$PATH\"\r\n# Configure (this uses \"defconfig\" but you could use \"menuconfig\"), but you must\r\n# include CC and LD in the make args or your .config won't know about Clang.\r\nmake defconfig CC=clang LD=ld.lld\r\n# Enable LTO and CFI.\r\nscripts\/config \\\r\n     -e CONFIG_LTO \\\r\n     -e CONFIG_THINLTO \\\r\n     -d CONFIG_LTO_NONE \\\r\n     -e CONFIG_LTO_CLANG \\\r\n     -e CONFIG_CFI_CLANG \\\r\n     -e CONFIG_CFI_PERMISSIVE \\\r\n     -e CONFIG_CFI_CLANG_SHADOW\r\n# Enable LKDTM if you want runtime fault testing:\r\nscripts\/config -e CONFIG_LKDTM\r\n# Build!\r\nmake -j$(getconf _NPROCESSORS_ONLN) CC=clang LD=ld.lld\r\n<\/pre>\n
Do not be alarmed by various warnings, such as:<\/p>\n
\r\nld.lld: warning: cannot find entry symbol _start; defaulting to 0x1000\r\nllvm-ar: error: unable to load 'arch\/x86\/kernel\/head_64.o': file too small to be an archive\r\nllvm-ar: error: unable to load 'arch\/x86\/kernel\/head64.o': file too small to be an archive\r\nllvm-ar: error: unable to load 'arch\/x86\/kernel\/ebda.o': file too small to be an archive\r\nllvm-ar: error: unable to load 'arch\/x86\/kernel\/platform-quirks.o': file too small to be an archive\r\nWARNING: EXPORT symbol \"page_offset_base\" [vmlinux] version generation failed, symbol will not be versioned.\r\nWARNING: EXPORT symbol \"vmalloc_base\" [vmlinux] version generation failed, symbol will not be versioned.\r\nWARNING: EXPORT symbol \"vmemmap_base\" [vmlinux] version generation failed, symbol will not be versioned.\r\nWARNING: \"__memcat_p\" [vmlinux] is a static (unknown)\r\nno symbols\r\n<\/pre>\n
Adjust your .config<\/code> as you want (but, again, make sure the CC and LD args are pointed at Clang and LLD respectively). This should(!) result in a happy bootable x86 CFI-enabled kernel. If you want to see what a CFI failure looks like, you can poke LKDTM:<\/p>\n
\r\n# Log into the booted system as root, then:\r\ncat <(echo CFI_FORWARD_PROTO) >\/sys\/kernel\/debug\/provoke-crash\/DIRECT\r\ndmesg\r\n<\/pre>\n
Here’s the CFI splat I see on the console:<\/p>\n
\r\n[   16.288372] lkdtm: Performing direct entry CFI_FORWARD_PROTO\r\n[   16.290563] lkdtm: Calling matched prototype ...\r\n[   16.292367] lkdtm: Calling mismatched prototype ...\r\n[   16.293696] ------------[ cut here ]------------\r\n[   16.294581] CFI failure (target: lkdtm_increment_int$53641d38e2dc4a151b75cbe816cbb86b.cfi_jt+0x0\/0x10):\r\n[   16.296288] WARNING: CPU: 3 PID: 2612 at kernel\/cfi.c:29 __cfi_check_fail+0x38\/0x40\r\n...\r\n[   16.346873] ---[ end trace 386b3874d294d2f7 ]---\r\n[   16.347669] lkdtm: Fail: survived mismatched prototype function call!\r\n<\/pre>\n
The claim of “Fail: survived …” is due to CONFIG_CFI_PERMISSIVE=y<\/code>. This allows the kernel to warn but continue with the bad call anyway. This is handy for debugging. In a production kernel that would be removed and the offending kernel thread would be killed. If you run this again with the config disabled, there will be no continuation from LKDTM. :)<\/p>\n
Enjoy! And if you can figure out before me why there is still CFI instrumentation in the KPTI entry handler, please let me know and help us fix it<\/a>. ;)<\/p>\n
© 2019 – 2020, Kees Cook<\/a>. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License<\/a>.
\"CC<\/a> <\/p>\n","protected":false},"excerpt":{"rendered":"
While much of the work on kernel Control Flow Integrity (CFI) is focused on arm64 (since kernel CFI is available on Android), a significant portion is in the core kernel itself (and especially the build system). Recently I got a sane build and boot on x86 with everything enabled, and I’ve been picking through some […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5,20,18,21,6,14,19],"tags":[],"_links":{"self":[{"href":"https:\/\/outflux.net\/blog\/wp-json\/wp\/v2\/posts\/1369"}],"collection":[{"href":"https:\/\/outflux.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/outflux.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/outflux.net\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/outflux.net\/blog\/wp-json\/wp\/v2\/comments?post=1369"}],"version-history":[{"count":18,"href":"https:\/\/outflux.net\/blog\/wp-json\/wp\/v2\/posts\/1369\/revisions"}],"predecessor-version":[{"id":1436,"href":"https:\/\/outflux.net\/blog\/wp-json\/wp\/v2\/posts\/1369\/revisions\/1436"}],"wp:attachment":[{"href":"https:\/\/outflux.net\/blog\/wp-json\/wp\/v2\/media?parent=1369"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/outflux.net\/blog\/wp-json\/wp\/v2\/categories?post=1369"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/outflux.net\/blog\/wp-json\/wp\/v2\/tags?post=1369"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}