View source for Tweaking, tuning and troubleshooting

{| style="border:1px solid #aaaaaa; background-color:#f9f9f9;width:100%; font-family: Verdana, sans-serif;"
|- padding:5px;padding-top:0.5em;font-size: 95%; 
| Back to [[Detecting, querying and testing]] <span style="float:right; padding-left:5px;">Forward to [[Reconstruction]]</span>
|}
=Tweaking, tuning and troubleshooting=

==Autodetection==

[[Autodetect|In-kernel autodetection]] was a way to allow the RAID devices to be automatically recognized
by the kernel at boot-time, right after the ordinary partition detection is done. Modern kernels do not recognise raid arrays and in order to boot off a version 1.2 array, you must use an initramfs to assemble the array.

It is possible to boot off a raid array without an initramfs but the following is necessary

# You must use metadata 0.9 or 1.0 that goes at the end of the array
# The array must be raid-1 - a mirror
# The kernel will not realise it's an array, so boot the partition as read-only, then remount / as the mirror read-write once the array has started.

== Booting on RAID ==

[[LegacyBoot|LILO and (legacy) Grub 1]]

Pretty much all modern linux systems use Grub 2. Your install program should set it up correctly, but if you have to set it up manually, make sure that the raid driver is loaded. Also make sure when linux is loaded that the domdadm option is passed. An example boot entry is

 menuentry 'Gentoo GNU/Linux, with Linux 4.4.6-gentoo' --class gentoo --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.4.6-gentoo-advanced-ab538350-d249-413b-86ef-4bd5280600b8' {
        load_video
        insmod gzio
        insmod part_gpt
        insmod diskfilter
        insmod mdraid1x
        insmod ext2
        set root='mduuid/69270eaca840f6e70199064bd5863c5d'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint='mduuid/69270eaca840f6e70199064bd5863c5d'  ab538350-d249-413b-86ef-4bd5280600b8
        else
          search --no-floppy --fs-uuid --set=root ab538350-d249-413b-86ef-4bd5280600b8
        fi
        echo    'Loading Linux 4.4.6-gentoo ...'
        linux   /boot/vmlinuz-4.4.6-gentoo root=UUID=ab538350-d249-413b-86ef-4bd5280600b8 ro  domdadm
        echo    'Loading initial ramdisk ...'
        initrd  /boot/initramfs-genkernel-x86_64-4.4.6-gentoo
 }


== Converting the root filesystem to RAID ==

The time-honoured way of coping with this sort of thing is to have a small
/boot partition at the start of the drive. This, however, means that your boot
details are not protected by raid, unless you go to the trouble of manually 
copying them every time they change, or you mess about with old metadata
formats.

And if you're doing a new install, most modern distros will set up raid for you.
The ones that won't and expect you to do your own disk setup, will come with
raid support enabled so you can create a raid device before installing.

=== Method 2016 ===

This method assumes you are adding a new drive, and will set up a degraded array
before converting it to a full working array. It's easier if you're adding two drives
and can set up a fully working array. Note that, by default, a system will not boot from an array
that has become degraded. [TODO: document how to make it boot. Hopefully it will boot from an array
that has been set up in degraded mode]

* First, make sure your kernel has raid compiled in, and that mdadm is installed. If you're
not using grub2, upgrade now.

* Add the new disk. If it's the same size as the original, and you plan to mirror everything,
then make sure you can afford to lose a little disk space. Install grub2 on that drive, too.

* Plan and create your new partitioning scheme. It doesn't have to be the same as on the old
disk, but if the new disk is larger and you use the extra space, you will not be able to raid
everything.

* Create your arrays using your new disk. Use a command similar to the following - note the
use of a named array - "root" - and the word "missing" which tells the create command to create
a mirror with just one active device.

 mdadm --create /dev/md/root --level raid1 --raid-disks 2 missing /dev/sdb1

* Create file systems on your new arrays with a command like the following:

 mkfs.ext4 /dev/md/root

* Mount your new file system and copy the contents of your root file system to the new filesystem

 mount /dev/md/root /mnt/newroot
 cp -ax / /mnt/newroot

Note options ax to copy everything including permissions and links but not follow any mount points.

* Run grub2-mkconfig, check that everything is okay, that it has detected /mnt/newroot as a boot partition,
and that when booting from it it loads the raid driver and passed the domdadm option to linux. Add this configuration to grub
(making sure it's in both /boot/grub and /mnt/newroot/boot/grub),
and reboot to boot on the raid drive.

* make sure that grub is reading its configuration at boot time from your new drive [TODO: How?]

* Copy the data from all the old partitions to the new ones the same way as with root, and update fstab or whatever. Reboot the system to make sure you have a working system all on the new drive.

* If the new partitions are the same size as the old ones, you can now add them in and let the mirrors rebuild.

 mdadm --grow /dev/mdroot --add /dev/sda1

It would be wise to just do root first, and then reboot to make sure we have not messed up grub. If we didn't successfully switch it to the
new drive then the old drive no longer exists and it won't be able to find its configuration.

=== Method 1 (2011) ===

This method assumes you have a spare disk you can install the system
on, which is not part of the RAID you will be configuring.

* First, install a normal system on your extra disk.

* Get the kernel you plan on running, get the raid-patches and the tools, and make your system boot with this new RAID-aware kernel. Make sure that RAID-support is in the kernel, and is not loaded as modules.

* Ok, now you should configure and create the RAID you plan to use for the root filesystem. This is standard procedure, as described elsewhere in this document.

* Just to make sure everything's fine, try rebooting the system to see if the new RAID comes up on boot. It should.

* Put a filesystem on the new array (using mke2fs), and mount it under /mnt/newroot

*  Now, copy the contents of your current root-filesystem (the spare disk) to the new root-filesystem (the array). There are lots of ways to do this, one of them is

      cd /
      find . -xdev | cpio -pm /mnt/newroot

another way to copy everything from / to /mnt/newroot could be

    cp -ax / /mnt/newroot

*  You should modify the /mnt/newroot/etc/fstab file to use the correct device (the /dev/md? root device) for the root filesystem.

*  Now, unmount the current /boot filesystem, and mount the boot device on /mnt/newroot/boot instead. This is required for LILO to run successfully in the next step.

*  Update /mnt/newroot/etc/lilo.conf to point to the right devices. The boot device must still be a regular disk (non-RAID device), but the root device should point to your new RAID. When done, run

      lilo -r /mnt/newroot

complete with no errors.

* Reboot the system, and watch everything come up as expected  :)

If you're doing this with IDE disks, be sure to tell your BIOS that
all disks are "auto-detect" types, so that the BIOS will allow your
machine to boot even when a disk is missing.

=== Method 2 (2011) ===

This method requires that your kernel and raidtools understand the
failed-disk directive in the /etc/raidtab file - if you are working on
a really old system this may not be the case, and you will need to
upgrade your tools and/or kernel first.

You can only use this method on RAID levels 1 and above, as the method
uses an array in "degraded mode" which in turn is only possible if the
RAID level has redundancy. The idea is to install a system on a disk
which is purposely marked as failed in the RAID, then copy the system
to the RAID which will be running in degraded mode, and finally making
the RAID use the no-longer needed "install-disk", zapping the old
installation but making the RAID run in non-degraded mode.

*  First, install a normal system on one disk (that will later become  part of your RAID). It is important that this disk (or partition) is not the smallest one. If it is, it will not be possible to add it to the RAID later on!

* Then, get the kernel, the patches, the tools etc. etc. You know the drill. Make your system boot with a new kernel that has the RAID support you need, compiled into the kernel.

*  Now, set up the RAID with your current root-device as the failed-disk in the /etc/raidtab file. Don't put the failed-disk as the first disk in the raidtab, that will give you problems with  starting the RAID. Create the RAID, and put a filesystem on it.  If using mdadm, you can create a degraded array just by running something like
 mdadm -C /dev/md0 --level raid1 --raid-disks 2 missing /dev/hdc1
note the missing parameter.

*  Try rebooting and see if the RAID comes up as it should

*  Copy the system files, and reconfigure the system to use the RAID as root-device, as described in the previous section.

*  When your system successfully boots from the RAID, you can modify the <tt>/etc/mdamd.conf</tt> file to include the previously failed-disk as a normal raid-disk. Now use <code>mdadm /dev/md0 --add /dev/hd??</code> to add the disk to your RAID.

*  You should now have a system that can boot from a non-degraded RAID.

== Making the system boot on RAID (2011) ==

For the kernel to be able to mount the root filesystem, all support
for the device on which the root filesystem resides, must be present
in the kernel. Therefore, in order to mount the root filesystem on a
RAID device, the kernel must have RAID support.

The normal way of ensuring that the kernel can see the RAID device is
to simply compile a kernel with all necessary RAID support compiled
in. Make sure that you compile the RAID support into the kernel, and
not as loadable modules. The kernel cannot load a module (from the
root filesystem) before the root filesystem is mounted.

However, since RedHat-6.0 ships with a kernel that has new-style RAID
support as modules, I here describe how one can use the standard
RedHat-6.0 kernel and still have the system boot on RAID.


===Booting with RAID as module===

You will have to instruct LILO to use a RAM-disk in order to achieve
this. Use the mkinitrd command to create a ramdisk containing all
kernel modules needed to mount the root partition. This can be done
as:

   mkinitrd --with=<module> <ramdisk name> <kernel>

For example:

   mkinitrd --preload raid5 --with=raid5 raid-ramdisk 2.2.5-22

This will ensure that the specified RAID module is present at boot-
time, for the kernel to use when mounting the root device.

===Modular RAID on Debian GNU/Linux after move to RAID===

Debian users may encounter problems using an initrd to mount their
root filesystem from RAID, if they have migrated a standard non-RAID
Debian install to root on RAID.

If your system fails to mount the root filesystem on boot (you will
see this in a "kernel panic" message), then the problem may be that
the initrd filesystem does not have the necessary support to mount the
root filesystem from RAID.

Debian seems to produce its initrd.img files on the assumption that
the root filesystem to be mounted is the current one.  This will
usually result in a kernel panic if the root filesystem is moved to
the raid device and you attempt to boot from that device using the
same initrd image.  The solution is to use the mkinitrd command but
specifying the proposed new root filesystem.  For example, the
following commands should create and set up the new initrd on a Debian
system:

  % mkinitrd -r /dev/md0 -o /boot/initrd.img-2.4.22raid
  % mv /initrd.img /initrd.img-nonraid
  % ln -s /boot/initrd.img-raid /initrd.img"


== Converting a non-RAID RedHat System to run on Software RAID (2011) ==

This section was written and contributed by Mark Price, IBM. The text
has undergone minor changes since his original work.

Notice: the following information is provided "AS IS" with no
representation or warranty of any kind either express or implied. You
may use it freely at your own risk, and no one else will be liable for
any damages arising out of such usage.

[[Convert RedHat to Raid|Converting a non-RAID RedHat System to run on Software RAID]]

==Sharing spare disks between different arrays==

When running mdadm in the follow/monitor mode you can make different
arrays share spare disks. That will surely make you save storage space
without losing the comfort of fallback disks.

In the world of software RAID, this is a brand new never-seen-before
feature: for securing things to the point of spare disk areas, you
just have to provide one single idle disk for a bunch of arrays.

With mdadm is running as a daemon, you have an agent polling arrays at
regular intervals. Then, as a disk fails on an array without a spare
disk, mdadm removes an available spare disk from another array and
inserts it into the array with the failed disk. The reconstruction
process begins now in the degraded array as usual.

To declare shared spare disks, just use the spare-group parameter when
invoking mdadm as a daemon.

== Pitfalls (2011) ==

Never NEVER never re-partition disks that are part of a running RAID.
If you must alter the partition table on a disk which is a part of a
RAID, stop the array first, then repartition.

It is easy to put too many disks on a bus. A normal Fast-Wide SCSI bus
can sustain 10 MB/s which is less than many disks can do alone today.
Putting six such disks on the bus will of course not give you the
expected performance boost. It is becoming equally easy to saturate
the PCI bus - remember, a normal 32-bit 33 MHz PCI bus has a
theoretical maximum bandwidth of around 133 MB/sec, considering
command overhead etc. you will see a somewhat lower real-world
transfer rate. Some disks today has a throughput in excess of 30
MB/sec, so just four of those disks will actually max out your PCI
bus! When designing high-performance RAID systems, be sure to take the
whole I/O path into consideration - there are boards with more PCI
busses, with 64-bit and 66 MHz busses, and with PCI-X.

More SCSI controllers will only give you extra performance, if the
SCSI busses are nearly maxed out by the disks on them.  You will not
see a performance improvement from using two 2940s with two old SCSI
disks, instead of just running the two disks on one controller.

If you forget the persistent-superblock option, your array may not
start up willingly after it has been stopped.  Just re-create the
array with the option set correctly in the raidtab. Please note that
this will destroy the information on the array!

If a RAID-5 fails to reconstruct after a disk was removed and re-
inserted, this may be because of the ordering of the devices in the
raidtab. Try moving the first "device ..." and "raid-disk ..." pair to
the bottom of the array description in the raidtab file.

{| style="border:1px solid #aaaaaa; background-color:#f9f9f9;width:100%; font-family: Verdana, sans-serif;"
|- padding:5px;padding-top:0.5em;font-size: 95%; 
| Back to [[Detecting, querying and testing]] <span style="float:right; padding-left:5px;">Forward to [[Reconstruction]]</span>
|}