{"id":76,"date":"2006-01-16T16:50:46","date_gmt":"2006-01-16T23:50:46","guid":{"rendered":"http:\/\/outflux.net\/blog\/?p=76"},"modified":"2006-01-16T17:29:17","modified_gmt":"2006-01-17T00:29:17","slug":"kernelorg-and-sendfile","status":"publish","type":"post","link":"https:\/\/outflux.net\/blog\/archives\/2006\/01\/16\/kernelorg-and-sendfile\/","title":{"rendered":"kernel.org and sendfile"},"content":{"rendered":"

The “sendfile<\/a>” system call is a way to send file contents directly out to a network socket. This saves time in userspace (so it doesn’t have to copy buffer contents around), and was one of the reasons I upgraded kernel.org<\/a>‘s Apache to version 2.x at the end of 2003 (because version 1.x doesn’t have sendfile support). A few weeks ago, one of the other kernel.org admins discovered that files greater than 2G were not being delivered by Apache.<\/p>\n

I had a lot of fun tracking down the issue. The “amount to send” argument in the sendfile call is a “size_t”, which is basically an “unsigned long”. Having a 2G limit didn’t make sense, since even with 32 bits, that should be a 4G limit. However, the kernel.org servers are both 64bit, so as it turns out, “size_t” is a full 64 bits. After writing a quick test, I was able to verify that it was, indeed, a 31 bit limit on both 64 bit and 32 bit kernels. Peter Anvin took it from here, and tracked down the origin of the problem: filesystem operations greater than 31 bits in offset were being rejected deep in the kernel. He suggested truncating the request<\/a> instead of returning a failure.<\/p>\n

Seems as though Linus decided to limit the size of filesystem calls to make sure there aren’t security problems (signed vs unsigned overflows) in the various filesystem drivers, while people using the Linux kernel migrate more from 32bit to 64bit systems. Personally, I don’t agree with this, but from a practical stand-point, it hardly makes a difference. Instead of sending all 4G out the pipe and returning to user space, it just returns twice, sending 2G per call.<\/p>\n

This should be fixed<\/a> in 2.6.16. Until then, we could patch Apache to keep it’s offset request under 31 bits, but we’ll probably just tell people to use FTP, since vsftpd<\/a> doesn’t use sendfile yet.<\/p>\n

© 2006, Kees Cook<\/a>. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License<\/a>.
\"CC<\/a> <\/p>\n","protected":false},"excerpt":{"rendered":"

The “sendfile” system call is a way to send file contents directly out to a network socket. This saves time in userspace (so it doesn’t have to copy buffer contents around), and was one of the reasons I upgraded kernel.org‘s Apache to version 2.x at the end of 2003 (because version 1.x doesn’t have sendfile […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11,7,6],"tags":[],"_links":{"self":[{"href":"https:\/\/outflux.net\/blog\/wp-json\/wp\/v2\/posts\/76"}],"collection":[{"href":"https:\/\/outflux.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/outflux.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/outflux.net\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/outflux.net\/blog\/wp-json\/wp\/v2\/comments?post=76"}],"version-history":[{"count":0,"href":"https:\/\/outflux.net\/blog\/wp-json\/wp\/v2\/posts\/76\/revisions"}],"wp:attachment":[{"href":"https:\/\/outflux.net\/blog\/wp-json\/wp\/v2\/media?parent=76"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/outflux.net\/blog\/wp-json\/wp\/v2\/categories?post=76"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/outflux.net\/blog\/wp-json\/wp\/v2\/tags?post=76"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}