Choosing your hardware, and what is a device?
What is a device?
As far as linux is concerned, all storage is a "block device". This means that data is written in chunks, typically 4KB in size, and that these chunks may be written to random locations on the device. Block devices are often layered, which means that one device writes to another, which may then write to a third, and so on. One typically thinks of a device as a hard disk, but as far as linux is concerned that is just one of many.
A typical raid system will have several hard drives as the bottom layer. These will then be partitioned (another block device) and the partitions will be combined into a raid array (yet another block device). The array will then have lvm over that (yet another block device) before this is partitioned (yet another device layer).
From this it can be seen that linux doesn't particularly what devices you use to make up an array - you can combine arrays, you can use a network storage device, you can use SSDs, SATA or PATA disks, USB, whatever. However, certain choices may or may not be wise.
(In contrast to block devices, linux also has character devices, which just read or write a continuous stream of characters. Not suitable for storage because you have to start at the beginning and end at the end, so you would have to rewrite everything every time you wanted to change just a little bit - like an old-fashioned tape drive, for example).
When creating your arrays, make sure you have it clear in your mind (and well documented for your successor, who is quite likely to be you six months down the line!) what devices fit where in the hierarchy.
Do NOT use these as part of your array. They have a timeout/disconnect mechanism which interacts very badly with the raid code. An array which includes USB is likely to fall over in a heap. USB is okay in a recovery scenario when you are copying from a drive as the disconnect scenario won't apply. Always copy from USB to SATA, as writing to USB is usually much slower than reading.
Solid State Disks
These are very fast disks, and if you build an array from them, you are best off using raid-1 (mirror). This is a scenario where using the other forms of redundant raid (ie 5 or 6) is actually likely to slow access down!
A new facility (2016) of raid is to use an SSD as a journal. [TODO: Improve my understanding of journalling] This flushes writes to the SSD, partly to speed the system up for the user, and partly to optimise writes to slower rotating media, as it allows changes to accumulate in the journal until a full stripe can be flushed to rotating media in one go.
Serial ATA drives
All modern drives are serial ATA. Not all drives are suitable for RAID, though (at least in their default state). When buying disk drives, make sure they support SCT/ERC, otherwise known as TLER. Most drives of 1TB or less are okay, manufacturers seem to have removed this support from their cheaper drives as capacities went over 1TB.
Other devices such as network storage will be added when I or other contributors gain experience of them. Obsolete devices such as parallel ATA will only be mentioned in passing - note that it is not wise to put more than one PATA drive per cable/controller.
Selecting a motherboard
For home systems, many motherboards nowadays come with five or six SATA ports. Bearing in mind you will probably want to use one port for a DVD-ROM, you will need at least three ports to have a single mirror, or five for two mirrors or a raid-6. An add-in card to add more SATA ports isn't that expensive.
One thing to watch out for - does the motherboard support legacy IDE? Mine does, and it will only boot from CD-ROM in this mode - which disables the fifth SATA port. However, it's a BIOS board, most boards today will be UEFI.
Desktop and Enterprise drives
Disk drives now tend to come in two varieties, desktop drives from which most of the features needed for a decent raid have been deleted, and enterprise drives, which have the features but are designed to run 24/7. So if you want to run raid on a desktop system it's rather difficult to find a drive that is suitable.
If you're going to buy desktop drives (raid drives are a bit more expensive, but not much) you MUST read the Timeout Mismatch section.
For example, Western Digital make "Green" and "Purple" drives.
Green drives are designed to be eco-friendly, and shut down as much as possible. This is great for a laptop that caches everything in ram until it absolutely has to flush everything to disk, or if you've got an SSD in front of them so they can stay idle for as long as possible, but in normal use they'll cause problems and wear out far faster than they should - linux usage is almost guaranteed to stress them in unsuitable ways.
Read the hdparm man page - "hdparm -J" in particular, to disable the greens from head-parking.
[Todo: document hdparm -S to disable spindown, and add this to the script in "Timeout Mismatch"]
Purple drives are meant for CCTV systems. Again, it's highly unlikely their default configuration is linux-friendly, but I know nothing about them.
TLER and SCT/ERC
TLER (Time Limited Error Recovery) is a WD creation, which means that drives will return within 7 seconds. Having introduced it, WD subsequently disabled it on most desktop drives, although it is enabled by default on enterprise drives.
SCT/ERC is the generic specification implemented by TLER.
If it's available this feature needs to be enabled. If it isn't enabled or available, the linux defaults will interact badly with the drive, and a single drive failure will usually take down the array.
This command will tell you what the drive is capable of. If possible, it would be wise to see the output of it on the drive(s) you are thinking of buying. The following is the output from my laptop's Toshiba drive. Note especially where it says SCT Error Recovery Control is supported.
crappit:/home/anthony # smartctl -x /dev/sda smartctl 6.2 2013-11-07 r3856 [x86_64-linux-4.1.27-27-default] (SUSE RPM) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Device Model: ST2000LM003 HN-M201RAD Serial Number: S321J9DG805231 LU WWN Device Id: 5 0004cf 2106b38eb Firmware Version: 2BC10001 User Capacity: 2,000,398,934,016 bytes [2.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ATA8-ACS T13/1699-D revision 6 SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Tue Sep 20 00:05:59 2016 BST SMART support is: Available - device has SMART capability. SMART support is: Enabled AAM feature is: Disabled APM feature is: Disabled Rd look-ahead is: Enabled Write cache is: Enabled ATA Security is: Disabled, NOT FROZEN [SEC1] Wt Cache Reorder: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (22740) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 379) minutes. SCT capabilities: (0x003f) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported.