RAID Boot

Revision as of 21:52, 12 August 2006

Historically when the kernel boots it uses a mechanism called 'autodetect' to identify partitions which are used in RAID arrays: it assumes that all partitions of type 0xfd are so used. It then attempts to automatically assemble and start these arrays.

This approach can cause problems in several situations (see various mailing list archives for the debates) and kernel autodetect is correspondingly deprecated.

The recommended approach now is to use the initramfs system.

This system is documented in more detail than you're likely to care about in the file Documentation/filesystems/ramfs-rootfs-initramfs.txt in the kernel source tree, but in brief it allows you to store in the kernel image a nonswappable in-memory filesystem (the 'rootfs') which is uncompressed onto / as the kernel boots; the kernel runs /init and leaves it to find the root filesystem, chroot into it, and execute init. It's sort of like the old initrd system, only your image never gets out of sync with the kernel, it's much easier to build it (the kernel build system can put it together for you), the kernel can always find it, and there's no overcomplicated scheme for switching to the real root filesystem: you can just chroot.

This approach provides a great deal of flexibility: you can get your root filesystem from LVM layered over a RAID array stored on a dozen network block devices on machines in Gautemala, San Diego, and Tokyo if you really need to (although that particular combination might be a bit slow without careful use of 'write-mostly'). I've even heard that some people have a C compiler on there, and recompile third-party modules for the running kernel on the fly from the source code!

But this flexibility comes with a price, and getting the thing working in the first place is a bit full of traps. If init isn't PID 1, you're in trouble; if you've left anything in the rootfs before chrooting, you've lost the memory it was in forever: if you mess up population you've got a useless kernel image; and populating it is sort of like working on an embedded system, because unless you want a 20Mb kernel image you'd better use small tools, like busybox, and preferably a small libc, like uClibc.

But it's useful if you want to mount RAID arrays before booting: e.g., examining the partitions and assembling arrays with a defined UUID, while leaving you with enough emergency repair facilities to figure out what's wrong if assembly fails.

mdadm comes with such a script, so here's another. It has a number of improvements over the mdadm variation:

• It handles LVM2 as well as md (obviously if you boot off RAID you still have to boot off RAID1, but /boot can be a RAID1 filesystem of its own now, with / in LVM, on RAID, or both at once; you don't even need md on the machine anymore)
• It fscks / before mounting it
• If anything goes wrong, it drops you into an emergency shell in the rootfs, from where you have all the power of ash with hardly any builtin commands, lvm and mdadm to diagnose your problem!
• it supports a number of arguments: 'rescue', to drop into /bin/ash instead of init after mounting the real root filesystem, 'emergency', to drop into a shell on the initramfs before doing *anything*, 'trace', to turn on shell tracing early in the init script execution, so if something's failing with a bizarre error message, you can tell what it is.
• It supports root= and init= arguments, although for arcane reasons to do with LILO suckage you need to pass the root argument as `root=LABEL=/dev/some/device', or LILO will helpfully transform it into a device number, which is rarely useful if the device name is, say, /dev/emergency-volume-group/root. It gets the default volume group name from a file `vgname' which you have to arrange to put on the initramfs (sticking it in the usr/ subdirectory in the kernel tree will do).

Right now, if you don't pass root=, it tries to mount /dev/raid/root after initializing all the RAID arrays and LVM VGs it can. (I've heard people suggest that this is not ideal, and it should try to initialize as little as possible, but I haven't heard a good reason for it yet: neither mdadm nor vgscan will initialize partial arrays/VGs by default, after all.)

• It doesn't waste memory. initramfs isn't like initrd: if you just chroot into the new root filesystem, the data in the initramfs *stays around*, in *nonswappable* kernel memory. And it's not gzipped by that point, either!

The downsides:

• It needs busybox 1.2 or later, and a 2.6.12+ kernel with sysfs and hotplug support; this is because it populates /dev with the `mdev' mini-udev tool inside busybox, and switches root filesystems with the `switch_root' tool, which chroots only after erasing the entire contents of the initramfs (taking great care not to recurse off that filesystem!)
• If you link against uClibc you'll need mdadm 2.5.2 or later: earlier versions will crash.
• if you link against uClibc (recommended), you need a CVS uClibc too (i.e., one newer than 0.9.27).
• It doesn't try to e.g. set up the network: changing the script to do that isn't likely to be terribly difficult.
• The init script's got a few too many things hardwired still, like the type of the root filesystem. I expect it's short enough to easily hack up if you need to  :)
• You need an /etc/mdadm.conf (if using md) and an /etc/lvm/lvm.conf, both taken by default from the system you built the kernel on: personally I'd recommend a really simple one with no device= lines, like

DEVICE partitions
ARRAY /dev/md0 UUID=some:long:uuid:here
ARRAY /dev/md1 UUID=another:long:uuid:here
ARRAY /dev/md2 UUID=yetanother:long:uuid:here
...

(I might change this to use --homehost in future, whereupon you'd only need to provide a file giving the hostname; but --homehost isn't widely-enough available yet, and I haven't tried it myself.)

Here's the init script:

#!/bin/sh
#
# init --- locate and mount root filesystem
#          By Nix <nix@esperi.org.uk>.
#
#          Placed in the public domain.
#

export PATH=/sbin:/bin

/bin/mount -t proc proc /proc
/bin/mount -t sysfs sysfs /sys
CMDLINE=`cat /proc/cmdline`

# Populate /dev from /sys
/bin/mount -t tmpfs tmpfs /dev
/sbin/mdev -s

INIT_ARGS=

# If there is a forced root filesystem or init, accept the forcing
for param in $CMDLINE; do
    case "$param" in
        init=*) eval "$param";;
        -b|single|s|S|[1-5]) INIT_ARGS="$INIT_ARGS $param";;
        trace) echo "Tracing init script.";
               set -x;;
        rescue) echo "Rescue boot mode: invoking ash.";
                init=/bin/ash;
                INIT_ARGS="-";;
        emergency) echo "Emergency boot mode. Dropping to a minimal shell.";
                   echo "Reboot with Ctrl-Alt-Delete.";
                   exec /bin/sh;;
        root=LABEL=*) root="`echo $param | cut -d= -f3-`";;
    esac
done

# Assemble the RAID arrays.
if [[ -x /sbin/mdadm ]]; then
    /sbin/mdadm --assemble --scan --auto=md --run
fi

FAILED=
# Scan for volume groups.
/sbin/lvm vgscan --ignorelockingfailure --mknodes && /sbin/lvm vgchange -ay --ignorelockingfailure

[[ -z $root ]] && root="/dev/`cat /etc/vgname`/root"

fsck -a $root

if [[ $? -eq 4 ]]; then
    echo "Filesystem errors left uncorrected."
    echo
    echo "Dropping to a minimal shell.  Reboot with Ctrl-Alt-Delete."

    exec /bin/sh
fi

if [[ -n $root ]]; then
    /bin/mount -o rw -t ext3 $root /new-root
fi

if /bin/mountpoint /new-root >/dev/null; then :; else
    echo "No root filesystem given to the kernel or found on the root RAID array."
    echo "Append the correct 'root=' boot option."
    echo
    echo "Dropping to a minimal shell.  Reboot with Ctrl-Alt-Delete."

    exec /bin/sh
fi

if [[ -z "$init" ]]; then
    init=/sbin/init
fi

# Unmount everything and switch root filesystems for good:
# exec the real init and begin the real boot process.
/bin/umount -l /proc
/bin/umount -l /sys
/bin/umount -l /dev

echo "Switching to /new-root and running '$init'"
exec switch_root /new-root $init $INIT_ARGS

Here's the initramfs source script, by default named usr/initramfs; you'll need to adjust it to pick up tools from the right place. You can omit `mdadm' if you like, so you can use the same init script on machines with root on LVM and with root on LVM on RAID, changing only the initramfs source script.

#
# Files needed for early userspace.
# Placed in the public domain.
#

# Binaries.
dir /bin 0755 0 0
file /bin/busybox /usr/i686-pc-linux-uclibc/bin/busybox 0755 0 0
slink /bin/sh /bin/busybox 0755 0 0
slink /bin/msh /bin/busybox 0755 0 0
slink /bin/[ /bin/busybox 0755 0 0
slink /bin/[[ /bin/busybox 0755 0 0
slink /bin/test /bin/busybox 0755 0 0
slink /bin/mount /bin/busybox 0755 0 0
slink /bin/umount /bin/busybox 0755 0 0
slink /bin/cat /bin/busybox 0755 0 0
slink /bin/ls /bin/busybox 0755 0 0
slink /bin/mountpoint /bin/busybox 0755 0 0
slink /bin/echo /bin/busybox 0755 0 0
slink /bin/false /bin/busybox 0755 0 0
slink /bin/true /bin/busybox 0755 0 0
slink /bin/mkdir /bin/busybox 0755 0 0
dir /sbin 0755 0 0
slink /sbin/mdev /bin/busybox 0755 0 0
slink /sbin/fsck /bin/busybox 0755 0 0
slink /sbin/e2fsck /bin/busybox 0755 0 0
slink /sbin/fsck.ext2 /bin/busybox 0755 0 0
slink /sbin/fsck.ext3 /bin/busybox 0755 0 0
slink /sbin/switch_root /bin/busybox 0755 0 0
file /sbin/lvm /usr/i686-pc-linux-uclibc/sbin/lvm 0755 0 0
file /init usr/init 0755 0 0
# (optional)
file /sbin/mdadm /usr/i686-pc-linux-uclibc/sbin/mdadm 0755 0 0

# Supporting directories
dir /proc 0755 0 0
dir /sys 0755 0 0
dir /new-root 0755 0 0
dir /etc 0755 0 0
dir /etc/lvm 0755 0 0

# Files you must provide.
file /etc/lvm/lvm.conf /etc/lvm/lvm.conf 0644 0 0
file /etc/mdadm.conf /etc/mdadm.conf 0644 0 0
file /etc/vgname usr/vgname.loki 0644 0 0

# initial device files required (mdev creates the rest)
dir /dev 0755 0 0
nod /dev/console 0600 0 0 c 5 1
nod /dev/null 0666 0 0 c 1 3

Here's the busybox config file I used for all this. You will need to change the CROSS_COMPILER_PREFIX and the EXTRA_CFLAGS_OPTIONS, and you might want to build in more tools as well for use when things go really wrong, in emergency mode:

HAVE_DOT_CONFIG=y
CONFIG_FEATURE_BUFFERS_USE_MALLOC=y
CONFIG_FEATURE_DEVPTS=y
CONFIG_STATIC=y
CONFIG_LFS=y
USING_CROSS_COMPILER=y
CROSS_COMPILER_PREFIX="/usr/bin/i686-pc-linux-uclibc-"
EXTRA_CFLAGS_OPTIONS="-march=pentium3 -fomit-frame-pointer"
CONFIG_INSTALL_NO_USR=y
CONFIG_INSTALL_APPLET_SYMLINKS=y
PREFIX="./_install"
CONFIG_CAT=y
CONFIG_CUT=y
CONFIG_ECHO=y
CONFIG_FALSE=y
CONFIG_LS=y
CONFIG_MKDIR=y
CONFIG_TEST=y
CONFIG_TRUE=y
CONFIG_FEATURE_AUTOWIDTH=y
CONFIG_E2FSCK=y
CONFIG_FSCK=y
FDISK_SUPPORT_LARGE_DISKS=y
CONFIG_MDEV=y
CONFIG_MOUNT=y
CONFIG_SWITCH_ROOT=y
CONFIG_UMOUNT=y
CONFIG_MOUNTPOINT=y
CONFIG_FEATURE_SH_IS_MSH=y
CONFIG_MSH=y
CONFIG_FEATURE_SH_EXTRA_QUIET=y
CONFIG_FEATURE_SH_STANDALONE_SHELL=y
CONFIG_FEATURE_COMMAND_EDITING=y
CONFIG_FEATURE_COMMAND_EDITING_VI=y
CONFIG_FEATURE_COMMAND_HISTORY=15
CONFIG_FEATURE_COMMAND_TAB_COMPLETION=y
CONFIG_FEATURE_IPC_SYSLOG_BUFFER_SIZE=0
CONFIG_MD5_SIZE_VS_SPEED=2