codeblog code is freedom — patching my itch

April 23, 2006

grub, yaird, mdadm, and missing drives

Filed under: General — kees @ 7:54 pm

This is basically a rant. I spent all my energy tracking down the problems, so I never did get things actually fixed. :P

I have my machines configured for software RAID between my primary and secondary drives. I always have. LILO supported this configuration back in RedHat 5.2 days. I’ve been doing RAID1 for a long time now. About a year ago, I changed my preference for boot loaders to GRUB, and just kind of assumed it handled mirroring. Well, much to my surprise, grub totally and completely does not handle mirrored configurations. Even the proclaimed fix didn’t work.

As a result of this “discovery”, I’ve switched back to LILO, which, I think, is a pain in the ass because it doesn’t actually have any filesystem-smarts built into it. (i.e. I have to re-run “lilo” every time I change a kernel or initrd.) I may see if another fix works as expected, but I don’t have a lot of hope considering the device map in the filesystem is the same for both grub drives, which is what causes the problems in the first place. (“Ieee! Where did the other drive go?!”)

So, moving forward, assuming my bootloader works, all kernels from 2.6.13 forward don’t support devfs, and the older initrd tools can’t handle that. Debian invented “yaird”. I had assumed they used the /sys filesystem and did other smart things. As it turns out, it’s fairly brain-dead. I booted without one of my mirrored drives, and yaird totally freaked out. As I discovered while digging through the initrd yaird generated, it just statically builds device nodes, based on what the running system used to look like.

There are two problems with this:

  1. DM devices (LVM, crypto, etc) are dynamically assigned. They may not have the same numbers after rebooting. This is mostly worked around by waiting for stuff to show up in /sys, so I’ll only complain about Ubuntu’s practice of encoding the major/minor numbers for the root device. (e.g. 0xFF00 — my root partition may not always be detected first) I don’t understand this, since the loader handles string-based paths for the root partition. But that’s not the bug I ran into for this rant.
  2. If a device goes missing, yaird assumes this is a bad thing. It has no concept of quorum. It could be argued that it shouldn’t, but in that case, it shouldn’t drop me to a prompt every time a device goes missing. It should only do that in “debug” mode. (I should send my patch for that in.)

While digging to open a Debian Bug report against yaird, I discovered that yaird, while annoyingly dropping me to a prompt (which I can “exit” out of), isn’t the real problem. The real problem is that “mdadm” incorrectly thinks it can’t start up the mirror with only 1 drive. There’s actually a counting bug where it just flat out thinks it needs 2 drives to start. Once I found this, I got pissed, “What? How could this bug exist?”

I proceeded to find the current source for mdadm, so I could write a patch to fix it. Only then did I discover that Debian’s version of mdadm is 5 REVISIONS BEHIND (including a major version jump)! AAAGGGh!

At this point I got in line reporting how old mdadm is, installed a work-around-mdadm patch to my yaird templates, and switched back to LILO. Ugh. And before someone yells “Run Gentoo!”, I checked already. The Gentoo mdadm version is old too. But at least they have a masked ebuild of the modern versions.

I hate choosing between stability and bleeding edge, but I don’t usually complain because I recognize the costs associated with stabilizing new stuff. But, come on, the mdadm 2.x series came out in AUGUST. That’s 8 months ago. I think that’s pretty stable! *sob*

I wish I had enough time to be a Debian maintainer instead of just sitting here and moaning, but hopefully my bug reports will do some good. :)

© 2006, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

1 Comment

  1. Hey, Kees,

    I had to figure out this problem for Linux cubes we’re installing into client sites at work. Actually, the second fix you point to is the proper way to configure GRUB for RAID-1. You should find it works beautifully.

    The problem is because GRUB probes the BIOS directly to build its drive numbering scheme. With the GRUB config (/boot/grub.conf or /boot/grub/menu.lst) being mirrored onto both drives, that one file tells GRUB to boot off of, say, (hd0), even when GRUB is running off of (hd1) because (hd0) is down. So when installing GRUB onto the second drive, you alias (hd0) to (hd1) with the command “device (hd0) /dev/sdb”, then install GRUB onto that drive or partition [with “root (hd0,0)” and “setup (hd0)”].

    At work we use SUSE Linux 10.0 (unfortunately), and its installer, even though it supports RAID and GRUB, doesn’t take this into account, so you have to install GRUB onto the second drive manually. I think this is the situation for most distributions. I don’t know if Fedora’s installer (anaconda) configures GRUB properly when booting from a RAID-1 device, but I hope it would (I use Fedora at home, but don’t have the funds to run RAID-1 on my personal machines).

    At work I have a post-install script that hardware techs run on every new Linux cube which automates (among other things) the installing of GRUB onto their secondary drives.

    It’s not at all that GRUB doesn’t support this situation, because it does with device aliasing. It’s that (most or all) distribution installers don’t support this situation.

    Comment by Derek P. Moore — April 24, 2006 @ 7:33 am

Powered by WordPress