Monitoring your system

Many of the horror stories that come to the linux raid mailing list are down to a simple lack of monitoring. Okay, it's not unknown for several disks to fail simultaneously, and if your raid array consists of a bunch of drives all bought at the same time, for the array, the odds of that happening are painfully high - batches of disks tend to have similar lifetimes.

But all too often, an array has been running in a degraded state for months, and then a disk fails and tips the array over the edge. The author's brother told him of a raid array, bought and placed in a colo facility, where a technician just happened to walk past and spot two red lights! The raid 6 array had two failed drives! This should never have happened, and of course there was a mad panic while they tried to safely replace the dead drives.

Monitoring Tools

/proc/mdstat

You should get to know /proc/mdstat, looking at it often. This will tell you the state of your arrays, and very importantly it will tell you whether any drives have failed, and whether any arrays are degraded. Check, and check regularly!

xosview

xosview is a venerable utility, and one of the author's favourites. It is capable of displaying the state of raid arrays, but unfortunately currently the code is broken - it reads mdstat, and doesn't understand the current output. It is currently (2016) being updated to read the status directly from /sys, and should hopefully soon be able to display raid status correctly. The author leaves xosview running permanently on his desktop to provide an overview of system performance.