{"id":1139,"date":"2018-04-19T17:34:51","date_gmt":"2018-04-20T00:34:51","guid":{"rendered":"https:\/\/outflux.net\/blog\/?p=1139"},"modified":"2020-11-30T22:05:41","modified_gmt":"2020-12-01T06:05:41","slug":"uefi-booting-and-raid1","status":"publish","type":"post","link":"https:\/\/outflux.net\/blog\/archives\/2018\/04\/19\/uefi-booting-and-raid1\/","title":{"rendered":"UEFI booting and RAID1"},"content":{"rendered":"

I spent some time yesterday building out a UEFI server that didn’t have on-board hardware RAID for its system drives. In these situations, I always use Linux’s md<\/code> RAID1 for the root filesystem (and\/or \/boot<\/code>). This worked well for BIOS booting since BIOS just transfers control blindly to the MBR of whatever disk it sees (modulo finding a “bootable partition” flag, etc, etc). This means that BIOS doesn’t really care what’s on the drive, it’ll hand over control to the GRUB code in the MBR.<\/p>\n

With UEFI, the boot firmware is actually examining the GPT partition table, looking for the partition marked with the “EFI System Partition” (ESP) UUID. Then it looks for a FAT32 filesystem there, and does more things like looking at NVRAM boot entries, or just running BOOT\/EFI\/BOOTX64.EFI<\/code> from the FAT32. Under Linux, this .EFI<\/code> code is either GRUB itself, or Shim which loads GRUB.<\/p>\n

So, if I want RAID1 for my root filesystem, that’s fine (GRUB will read md<\/code>, LVM, etc), but how do I handle \/boot\/efi<\/code> (the UEFI ESP)? In everything I found answering this question, the answer was “oh, just manually make an ESP on each drive in your RAID and copy the files around, add a separate NVRAM entry (with efibootmgr<\/code>) for each drive, and you’re fine!” I did not like this one bit since it meant things could get out of sync between the copies, etc.<\/p>\n

The current implementation of Linux’s md<\/code> RAID puts metadata at the front<\/em> of a partition. This solves more problems than it creates, but it means the RAID isn’t “invisible” to something that doesn’t know about the metadata. In fact, mdadm<\/code> warns about this pretty loudly:<\/p>\n

# mdadm --create \/dev\/md0 --level 1 --raid-disks 2 \/dev\/sda1 \/dev\/sdb1\r\nmdadm: Note: this array has metadata at the start and\r\n    may not be suitable as a boot device.  If you plan to\r\n    store '\/boot' on this device please ensure that\r\n    your boot-loader understands md\/v1.x metadata, or use\r\n    --metadata=0.90\r\n<\/blockquote><\/pre>\n
Reading from the mdadm man page:<\/p>\n
   -e, --metadata=\r\n...\r\n          1, 1.0, 1.1, 1.2 default\r\n                 Use  the new version-1 format superblock.  This has fewer\r\n                 restrictions.  It can easily be moved between hosts  with\r\n                 different  endian-ness,  and  a recovery operation can be\r\n                 checkpointed and restarted.  The  different  sub-versions\r\n                 store  the  superblock  at  different  locations  on  the\r\n                 device, either at the end (for 1.0), at  the  start  (for\r\n                 1.1)  or  4K from the start (for 1.2).  \"1\" is equivalent\r\n                 to \"1.2\" (the commonly preferred 1.x format).   \"default\"\r\n                 is equivalent to \"1.2\".\r\n<\/blockquote><\/pre>\n
First we toss a FAT32 on the RAID (mkfs.fat -F32 \/dev\/md0<\/code>), and looking at the results, the first 4K is entirely zeros, and file<\/code> doesn’t see a filesystem:<\/p>\n
# dd if=\/dev\/sda1 bs=1K count=5 status=none | hexdump -C\r\n00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|\r\n*\r\n00001000  fc 4e 2b a9 01 00 00 00  00 00 00 00 00 00 00 00  |.N+.............|\r\n...\r\n# file -s \/dev\/sda1\r\n\/dev\/sda1: Linux Software RAID version 1.2 ...\r\n<\/blockquote><\/pre>\n
So, instead, we’ll use --metadata 1.0<\/code> to put the RAID metadata at the end:<\/p>\n
# mdadm --create \/dev\/md0 --level 1 --raid-disks 2 --metadata 1.0 \/dev\/sda1 \/dev\/sdb1\r\n...\r\n# mkfs.fat -F32 \/dev\/md0\r\n# dd if=\/dev\/sda1 bs=1 skip=80 count=16 status=none | xxd\r\n00000000: 2020 4641 5433 3220 2020 0e1f be77 7cac    FAT32   ...w|.\r\n# file -s \/dev\/sda1\r\n\/dev\/sda1: ... FAT (32 bit)\r\n<\/blockquote><\/pre>\n
Now we have a visible FAT32 filesystem on the ESP. UEFI should be able to boot whatever disk hasn’t failed, and grub-install<\/code> will write to the RAID mounted at \/boot\/efi<\/code>.<\/p>\n
However, we’re left with a new problem: on (at least) Debian and Ubuntu, grub-install<\/code> attempts to run efibootmgr<\/code> to record which disk UEFI should boot from. This fails, though, since it expects a single disk<\/a>, not a RAID set. In fact, it returns nothing<\/em>, and tries to run efibootmgr<\/code> with an empty -d<\/code> argument:<\/p>\n
Installing for x86_64-efi platform.\r\nefibootmgr: option requires an argument -- 'd'\r\n...\r\ngrub-install: error: efibootmgr failed to register the boot entry: Operation not permitted.\r\nFailed: grub-install --target=x86_64-efi  \r\nWARNING: Bootloader is not properly installed, system may not be bootable\r\n<\/blockquote><\/pre>\n
Luckily my UEFI boots without NVRAM entries, and I can disable the NVRAM writing via the “Update NVRAM variables to automatically boot into Debian?” debconf prompt when running: dpkg-reconfigure -p low grub-efi-amd64<\/code><\/p>\n
So, now my system will boot with both or either drive present, and updates from Linux to \/boot\/efi<\/code> are visible on all RAID members at boot-time. HOWEVER<\/strong> there is one nasty risk with this setup: if UEFI writes anything to one of the drives (which this firmware did<\/em> when it wrote out a “boot variable cache” file), it may lead to corrupted results once Linux mounts the RAID (since the member drives won’t have identical block-level copies of the FAT32 any more).<\/p>\n
To deal with this “external write” situation, I see some solutions:<\/p>\n