Timeout Mismatch

From Linux Raid Wiki
Jump to: navigation, search
Back to Asking for help Forward to What's all this with USB?

Most cheap modern desktop drives do not support some form of managed error recovery. This seems to have started when the typical drive size hit 1TB. So for drives over 1TB, you should buy drives that are explicitly suitable for RAID. Most drives of 1TB or less are okay, but you should check first. RAID-rated drives aren't that much more expensive. (For some strange reason, every 2 1/2" laptop drive I've come across does support it!)

Also, in 2019, a new technology called shingled magnetic recording (SMR) started becoming mainstream. Whereas drive usage limits on conventional drives are advisory, burst limits especially on SMR drives are mandatory, and interfere with raid operation. This is the problem with the latest WD Red drives, that are sold as suitable for RAID but RAID will rapidly overwhelm the burst limit.

For a quick summary of the problem, when the OS tries to read from the disk, it sends the command and waits. What should happen is that drive returns the data successfully.

The proper sequence of events when something goes wrong is that the drive can't read the data, and it returns an error to the OS. The raid code then calculates what the data should be, and writes it back to the disk. Glitches like this are normal and, provided the disk isn't failing, this will correct the problem.

Unfortunately, with desktop drives, they can take over two minutes to give up, while the linux kernel will give up after 30 seconds. At which point, the RAID code recomputes the block and tries to write it back to the disk. The disk is still trying to read the data and fails to respond, so the raid code assumes the drive is dead and kicks it from the array. This is how a single error with these drives can easily kill an array.

SMR drives are even worse. It has been reported that it can take them 10 minutes or more to recover if burst limits are exceeded. That is, assuming the firmware doesn't crash, which seems to be the case quite often, in which case the drive will just fail.

To check whether this is the case, look at the output you got from smartctl, and see whether SCT Error Recovery Control is supported. If it isn't, this is your problem. On WD disks it may be called TLER. To just look at this parameter, you can use the command

smartctl -l scterc /dev/sdx

For SMR drives, the drive should report that the trim command is supported. Unfortunately, some (many?) cheaper SMR drives do not, and due to the nature of SMR drives that don't support trim will have problems, leading to exactly the grief many have reported - the drive stalling for ever almost as it has to rewrite masses of data. Note, however, that SMR drives come in at least three types - DM (device managed) which may or may not support trimming, and HM (host managed) which shouldn't be a problem as they leave it to the computer to sort out. The latest (2020-on) WD Reds seem to be the worst of all worlds here as most people will associate them with being good raid drives while they are now SMR (bad) and probably don't support trim (appalling).

69     14          1   Deterministic data after trim supported
69      5          1   Trimmed LBA range(s) returning zeroed data supported

Unfortunately, some drives do not. The reason for this may be down to the ATA specification. If I have my facts right, version 4 of the ATA specification postdates these problematic drives. but is required for reporting these capabilities. The drives in question may stick to the v3 specification rather than the provisional v4 spec.

ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)

After a lot of bad publicity WD have re-positioned their WD Red line as pure SMR. Red Plus is what the Reds used to be - all CMR, and Red Pro are unchanged - all CMR. Other manufacturers - Seagate, Hitachi - are moving various drive ranges over to SMR but are being far less secretive and are telling customers what is going on. The message here has to be "avoid WD, avoid SMR.".

The following script was posted to the mailing list by Brad Campbell. Make sure it runs on every boot - the cheaper drives especially forget any settings you may make when the system is shut down. It will increase the timeout for all non-ERC drives. It also sets the timeout for ERC drives as many older desktop drives that do support it have inappropriate settings. Note that it does nothing for SMR drives.

for i in /dev/sd? ; do
    if smartctl -l scterc,70,70 $i > /dev/null ; then
        echo -n $i " is good "
        echo 180 > /sys/block/${i/\/dev\/}/device/timeout
        echo -n $i " is  bad "
    smartctl -i $i | egrep "(Device Model|Product:)"
    blockdev --setra 1024 $i

WARNING: This does not work for all drives, although it seems to be older 2010-era ones that fail. The smartctl command attempts to set the ERC timeout to 7 seconds. This should either succeed and return 0, or fail and return an error code. Unfortunately, for drives that do not support SCT at all, the attempt to set ERC fails but returns 0, fooling the script. Whenever you get a new drive, you should make sure it behaves as expected.

[TODO: Discuss assembling a broken array with --force]

The following links-to-email have been collected by Phil Turmel as background reading to the problem. Read the entire threads if you have time.


[TODO: link to gmane if/when it comes back]

External links about SMR drives

Back to Asking for help Forward to What's all this with USB?
Personal tools