https://raid.wiki.kernel.org/api.php?action=feedcontributions&user=Julien+Robin&feedformat=atomLinux Raid Wiki - User contributions [en]2024-03-29T02:30:34ZUser contributionsMediaWiki 1.19.24https://raid.wiki.kernel.org/index.php/Irreversible_mdadm_failure_recoveryIrreversible mdadm failure recovery2019-05-05T16:23:51Z<p>Julien Robin: About recovery of an array which is split in the different arrangements, reminder that overlay should be used in case the approach finally brings insufficient results.</p>
<hr />
<div>= Introduction =<br />
<br />
If you feel comfortable using overlays, this is always a good idea for your first attempts (so that no accidental write can happen to the real members until the correct parameter set is proven to be found, and right). You will find some help about discovering and using overlay into this guide.<br />
<br />
Every information and situation you can find here has been seriously tested on mdadm 3.4-4 and 4.1-1 before being published.<br />
<br />
Going here is normally not needed, unless mdadm conventional ways won't have any chance to work anymore (we believe that you explored others solution before going here). This section is about searching right parameters to play --create over an existing array, in order to retrieve a correctly configured access to any still available data (be it the entire filesystem if still available).<br />
<br />
Be warned: playing mdadm --create erases and overwrites at least the RAID array information area of every involved RAID members. A misuse of mdadm --create can be the reason why you are here. Without the --assume-clean parameter, one or several member data area can be entirely reconstructed (sometimes it's fine, sometimes it's not, sometimes it's recoverable, sometimes it's not - so it's better to avoid it). Without the --readonly parameter, mounting and browsing the filesystem can write things, and it can be a problem if some of the members finally aren't at the right position into the array.<br />
<br />
If the correct RAID array information is still available (then are you sure nothing else can be done with conventional ways?), to save some time, please keep the output of cat /proc/mdstat (for getting the members orders), and the output of mdadm --examine against your differents RAID members. It should be done BEFORE calling any mdadm --create command. Creation and Update time are provided into the output, so you can easily know if it contains your original array information, or if it's already lost/overwritten.<br />
<br />
If you can't manage to be sure about the position of each disk into the array, and you have a large number of disks, you are encouraged to find a way to remember the position of as much disks as possible into the array, and prepare a listing of all possible combinations (starting with the most likely ones) because 4 disk could already give you a lot of work, and above 4 disks, an huge amount of attempts may be needed. Needing you each time to verify if your data is correctly available by checking the integrity of files that are big enough to be spread on every members.<br />
<br />
Members can be entire drives or partitions inside of it. If you erased a GPT table describing a partition by mistake, or removed the partition by mistake, please create it again, unformatted, at the same place (and same size) it was when used as RAID member. The content of the partition may be completely or mostly untouched (depending whether it has just been marked deleted, or partially overwritten).<br />
<br />
= Simple using of overlays =<br />
== First step and commands ==<br />
Writing on an overlay associated with a disk or partition won't really write the data to the disk or partition (but to an overlay file instead).<br />
Reading on an overlay associated with a disk or partition will show the content of the partition, but will include what has been written to the overlay file.<br />
This clever mechanism will give you some security during your attempts, as --create attempts will overwrite RAID array informations of each member, and if something accidentally attempts to write into the array or members, the real ones are safe.<br />
<br />
mdadm should do really few writes, so you can truncate the overlay to a tiny size compared to your RAID members.<br />
<br />
If my members are /dev/sdc1, and /dev/sdd1:<br />
mkdir overlay-files<br />
cd overlay_files<br />
truncate -s1G overlay-sdc1<br />
truncate -s1G overlay-sdd1<br />
This creates two 1GB sized files : overlay-sdc1 and overlay-sdd1<br />
<br />
== Loop devices ==<br />
The overlay system will use loop devices. How does loop devices works?<br />
*Make some file available as a block device on /dev/loop0 (as read/write)<br />
losetup /dev/loop0 /address/of/someFile<br />
*Disable the block device<br />
losetup -d /dev/loop0<br />
You can use 0, 1, 2, ... and so on. losetup -a will show you currently active loop devices. losetup -f can be used to find an available number.<br />
<br />
For our overlay files:<br />
losetup /dev/loop0 overlay-sdc1<br />
losetup /dev/loop1 overlay-sdd1<br />
<br />
== Enabling overlay devices ==<br />
I'm not sure to understand everything into those commands. But it works:<br />
<br />
size1=$(blockdev --getsize /dev/sdc1);<br />
size2=$(blockdev --getsize /dev/sdd1);<br />
<br />
echo 0 $size1 snapshot /dev/sdc1 /dev/loop0 P 8 | dmsetup create sdc1<br />
echo 0 $size2 snapshot /dev/sdc1 /dev/loop1 P 8 | dmsetup create sdd1<br />
<br />
According to some software able to list block devices (gnome-disks, photorec), /dev/mapper/sdc1 and /dev/mapper/sdd1 are available, showing the size of my RAID members ($size1 and $size2). It's using the true /dev/sdc1 and /dev/sdd1 for reading, and /dev/loop0 and /dev/loop1 for writing (and reading back) changes.<br />
<br />
Some disk usage monitor like bmw-ng ('n' for disks, 't' for displaying sums) shows that playing read and write with the /dev/mapper devices (or RAID array using it) instead of the real ones seems to work, without writing anything to the real devices. If you exceeded the maximum write amount permitted by the size of your overlays files, write operations won't work (Input/output error).<br />
<br />
For monitoring your overlay devices, you can use dmsetup status.<br />
<br />
== Stopping overlay devices ==<br />
For removing the overlays when you finished or screwed them:<br />
dmsetup remove sdc1<br />
dmsetup remove sdd1<br />
losetup -d /dev/loop0<br />
losetup -d /dev/loop1<br />
rm overlay-sdc1<br />
rm overlay-sdd1<br />
<br />
= Situation analysis =<br />
<br />
A RAID array member has 2 area: RAID array information (which may be lost if you are here), and the Data area. As long as the Data area is not erased, you can consider the member as useful for the following steps. If Data area is erased, it should be considered as missing for the following steps (and if the missing member is mandatory, don't quit this page before taking a look at the "Unlucky situations" section: may be some files can be recovered).<br />
<br />
You are about to recover entire access to your filesystem as long as:<br />
*Your array is redundant and you didn't lose/overwrite any member data area<br />
*Your array isn't redundant but hopefully, you didn't lose/overwrite any member data area<br />
*Your array is redundant, you have lost/overwritten one member data area, but hopefully there is still enough others members left to get your data back<br />
*The data inside your members are arranged in one consistent and predictable way all over the array<br />
<br />
If you aren't lucky, don't quit this page before taking a look at the "Unlucky situations" section: there's still several things to verify before aborting.<br />
<br />
= Parameters recovery =<br />
<br />
Playing --create over an existing array needs you to know:<br />
*The number of members in your array.<br />
*The RAID level of your array<br />
*The chunk size of your array if using RAID0, RAID4, RAID5, RAID6, or RAID10 (generally 512K on modern mdadm versions, at least for 3.4-4 and 4.1-1)<br />
*The metadata version of your array (depends of the mdadm version initially used to create the array - 1.2 on modern versions, at least for 3.4-4 and 4.1-1)<br />
*The layout of your array if using RAID5, RAID6, or RAID10 (generally left-symmetric by default for RAID5 and 6, and can be near=2 for RAID10 if your 4 members are A, A, B, B)<br />
*The data offset of your array (depends on the size of your array members used at initial array creation - even if members have been replaced by bigger ones after, and grow --size=max used! It also depends on the version of mdadm you were using at the initial creation)<br />
*The position of the different members into the array (if it changed on the motherboard since initial creation, you may need to try several ones. If not, you probably used alphabetical order)<br />
*Is any of your member missing, or having his Data area overwritten by something like a wrong command? Which one? Because it should be declared as "missing" when running --create. If you don't know which one, you will have to do more attempts.<br />
*--assume-clean will give you the ability to make several attempts without having any member rebuilt/overwritten<br />
*--readonly will give you the ability to mount the filesystem without causing any update to any byte of the data into the array<br />
<br />
If nothing changed on your system (members size, mdadm version...) and you used default parameters, then letting mdadm choosing default ones should be fine and apply the right parameters.<br />
<br />
== Data-offset ==<br />
If you changed mdadm version since initial array creation, data-offset is likely to have changed. I have seen 8192s when created with tiny disks (5000 MiB) into mdadm 3.4-4 (while 4.1-1 defaults to 10240s for the same size, and 18432s for 10000 MiB), and 262144s for 8TB drives with mdadm 3.4-4. Didn't try creating with 8TB members on mdadm 4.1-1. May be some formula can be found into the different code versions?<br />
<br />
= Example of command =<br />
== The easy one ==<br />
Using Debian 9, mdadm 3.4-4, a RAID array was created with with 3 x 8TB members, using:<br />
<br />
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1<br />
<br />
If nothing changed and I'm still using the same mdadm version, typing the same command again (but appending --assume-clean --readonly for safety) will be fine: <br />
<br />
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/mapper/sdb1 /dev/mapper/sdc1 /dev/mapper/sdd1 --assume-clean --readonly<br />
<br />
Gnome-disks shows the file system again and I can mount it (trying read-only before, mount -o ro,noload /dev/mdXX /media/raid-volume) : everything is fine and every member is in the right place into the array. Then, do not forget to do a filesystem check and repair after you ensured that you are using the correct parameters, in case some brutal interruption happened previously on the array.<br />
<br />
== A more precise one ==<br />
For creating the array again after having upgraded from mdadm 3.4-4 to mdadm 4.1-1, data offset should be specified because the automatically selected value for a given size changed. <br />
<br />
Also, disk position changed on my motherboard. After a precise analysis of the previous version defaults parameters for this array, in this case, the working command for finding back my array finally was:<br />
<br />
mdadm --create /dev/md0 --level=5 --chunk=512K --metadata=1.2 --layout left-symmetric --data-offset=262144s --raid-devices=3 /dev/mapper/sdd1 /dev/mapper/sde1 /dev/mapper/sdb1 --assume-clean --readonly<br />
<br />
== One facultative member is missing ==<br />
If the second disk is missing, type missing instead of its block device name. On a redundant RAID array, data will still be available. Of course, once your array is back working, you will have to --add the missing member soon or late for safety, but after you ensured that the parameters are OK and your data available. The member you will add will be rebuilt with consistent data. <br />
<br />
== The array was created with 2 disks, and grown to 3 ==<br />
Having the array initially created with 2 disks before having the 3rd one added and the array grown, give the exact same final arrangement compared to a 3 disks array creation.<br />
<br />
= In case of accidental rebuild =<br />
If you forgot the --assume-clean parameter, a rebuild process may have rewritten some of your members.<br />
But if you were using --readonly, then you are fine (in both case, I didn't see any rebuild process starting).<br />
<br />
*In case of RAID 0, there is no possible rebuild process. That's fine.<br />
*In case of RAID 1, only the 1st member is read, the others ones are overwritten with a copy of it.<br />
*In case of RAID 5, the last RAID member given in parameter will be overwritten by reading and processing data of the others given members. If your original array was already using RAID5, and the member count is correct, it's likely to be fine.<br />
*In case of RAID 6, every member is rewritten by reading and computing data from others ones. If you are wrong about the position of 2 disks out of 4, you can recover, see below. If its more, it likely to be lost (but I didn't try with array bigger than 4 disks)<br />
*In case of RAID 10, with 4 members and default parameters, 1st drive is read and copied to the 2nd one, while 3rd drive is read and copied to the 4th one.<br />
<br />
In case of RAID 5:<br />
*Even with wrong order or wrong chunk size, and any wrong RAID5 or RAID4-like layout, if the others members data is fine, and the amount of members is correct, the parity calculation will be correct. Although the computer may not interpret the data the right manner (which bit is parity or data) until you find the right parameters. <br />
*But if you forget the --assume-clean parameter in others cases (like incorrect amount of disks, or RAID level, or reading a lost member to calculate and rewrite over a good one), consider the overwritten disk as missing.<br />
<br />
In case of RAID6:<br />
*Driving some tests about RAID6, having the members accidentally rebuilt with 2 out of 4 disks in the wrong position (1243 - for example - instead of 1234) an erroneous reconstruction is applied on all of the 4 members. Trying to access the array filesystem, the files were incorrect. But --creating the RAID again, with the 4 disks back at the correct position (and --assume-clean and --readonly) restored the access to the correct data. Replaying the same command after ensuring the position are right, and without --assume-clean and --readonly also restored the original data over all of the 4 disks (sha1sum controlled, RAID information excluded).<br />
*Driving another test, it seems that if you have accidentally rebuilt your array with 2 out of 4 disks in the wrong position (still 1243 instead of 1234 for example), mounting the array with only disk 1 and 2 won't be fine. Nothing more with 1, 2, and 3: the 4 disks should be used. This is probably due to the fact that the erroneous reconstruction is done over all of the members, so all of them are needed to revert this error.<br />
<br />
In others cases:<br />
In case correct data seems to be erased from only one of your members, if it wasn't mandatory (if remaining fine members are enough), you will be able to resync it later by --adding it as a new one, but only once your correct parameters are found, and your array back working. If the overwritten member was mandatory, hope you played mdadm --stop. Please find a way to verify if the data is really gone, and if it's gone, go to the "Unlucky situations" section.<br />
<br />
= Help needed, no combination is giving back access to the data =<br />
If you have 3 members and you are sure that data is still available on both of them, try every order: <br />
*Disk 1, Disk 2, Disk 3 <br />
*Disk 1, Disk 3, Disk 2<br />
*Disk 2, Disk 1, Disk 3<br />
*Disk 2, Disk 3, Disk 1<br />
*Disk 3, Disk 1, Disk 2 <br />
*Disk 3, Disk 2, Disk 1<br />
<br />
If it's not working, you should find a way to ensure data-offset (and may be others values) are correct, by creating a reconstitution of your original array creation environment to find what default values mdadm probably used. If your values are good, your data are back.<br />
If one of the members has its data missing, and you know which one (if not, its longer), check every combination:<br />
<br />
Member 1/3 is missing: X23, X32, 2X3, 23X, 3X2, 32X<br />
<br />
Member 2/3 is missing: 1X3, 13X, X13, X31, 31X, 3X1<br />
<br />
Member 3/3 is missing: 12X, 1X2, 21X, 2X1, X12, X21<br />
<br />
The numbers stand for the different members block device names, X stands for the "missing" word, in order to represent the parameters order into --create command.<br />
<br />
If you have 4 members, there is 24 possible combinations:<br />
*1234, 1243, 1324, 1342, 1423, 1432<br />
*2134, 2143, 2314, 2341, 2413, 2431<br />
*3124, 3142, 3214, 3241, 3412, 3421<br />
*4123, 4132, 4213, 4231, 4312, 4321<br />
<br />
If you are using RAID6 with 4 members, as 2 disks out of 4 are optional, let's save some time, try the 12 following attempts :<br />
*1st disk in 1st place, 2nd disk in the others places : 12xx 1x2x 1xx2<br />
*1st disk in 2nd place, 2nd disk in the others places : 21xx x12x x1x2<br />
*1st disk in 3rd place, 2nd disk in the others places : 2x1x x21x xx12<br />
*1st disk in 4th place, 2nd disk in the others places : 2xx1 x2x1 xx21<br />
<br />
Still about the RAID6 with 4 disks, if the disk number 2 and disk number 4 have lost their data or aren't available for any reason, you will have to use disk numbers 1 and 3 to do the same position attempts (instead of disk 1 and 2). If you are sure 2 disks are badly overwritten but you don't know which ones, 12, 13, 14, 23, 24, 34 are the possible couples: up to 6 x 12 try may be required to find back your data.<br />
<br />
= Unlucky situations =<br />
<br />
== The RAID array is split into 2 arrangements ==<br />
*Did the reshape/conversion actually started? If you are sure that the answer is no (stuck at 0% with no disk activity, backup file is there but contains nothing, nothing started and it's completely stuck), then things will be easy. Find the parameters set of the unchanged arrangement.<br />
<br />
*If the reshape/conversion actually started and moved some data, this kind of interruption can be restarted just by assembling the array again, and mount/use it as nothing happened. If it doesn't work, see if there is no parameters to force things to continue: don't follow this guide unless RAID information of your members have already been erased, or if you're sure that nothing else worked.<br />
<br />
*If a member failed during reshaping, and your array is redundant, don't play --create! You can add a new member: reshaping will finish in degraded mode, and the last member will be integrated and rebuilt just after.<br />
<br />
*If you have no other choice that playing --create with 2 different arrangements over 2 half of the array, unless someone have a better idea, you're falling into forensics : you'll need to find 2 sets of parameters, and probably need to use a file recovery scanner tool to run twice, with both parameters set on the array. The second set of parameters will probably be the same as the first one, with 1 more member (and raid-devices count set to one more). <br />
<br />
In this case, having done some test with 2 sets of parameters and 2 photorec passes, shown that almost every photos are recoverable but names and folders structure was seriously lost. In order to preserve a maximum of future possible attempts, please use overlays for keeping your RAID members information area untouched, in case this approach finally proves to give insufficient results for your case: you may try to know where the separation is between the 2 arrangements and concatenate the resulting data of both arrangements, or anything else to get the original filesystem started again.<br />
<br />
Remember that when it is possible, having the array cleanly restarted and finishing the conversion can be much better.<br />
<br />
== Mandatory member lost ==<br />
So, your RAID is not redundant (or not enough), and a mandatory member (or the data area into it) is screwed.<br />
<br />
*Is your RAID member partially screwed/overwritten? If mdadm --create (or anything else) started to overwrite it, but you played mdadm --stop before it's entirely overwritten (or stopped anything that was overwriting the data into it), some of the data is still available. Find back your correct parameters (but you won't see any ext4 partition so it will be more difficult to know it you have the right ones) and use some file recovery scanner tool over your resulting array. You may still have the surprise to see some files that aren't completely erased.<br />
<br />
*Also, remember that having mdadm --create accidentally overwriting disks should be avoided, but if it accidentally happens, in some case, it may be recoverable (see the section about it). If you aren' sure, you can always try to use the member as if it's OK: if it is, your filesystem (or some parts of it) could be made available again.<br />
<br />
*If one of the disks physically failed, but the data is still enclosed into it, see if RAW data into it can be rescued and cloned by some company, or if available sectors can be rescued by dd-rescue. See [[Replacing a failed drive]] for more detailed information about drive failure, including failure in non-redundant arrays.<br />
<br />
= Making sure that none of my arrays will never be lost anymore in the future =<br />
<br />
You can't, because it won't always be your fault. We believe that having succeeded or not into your recovery, you are probably interested into clever backup approaches for securing the recovered or future data.<br />
<br />
Indeed, a defective power supply can destroy some of your disks. A violent shock on the case containing all the disks can cause several of them to fail. A fire incident too. The server can be stolen (with disks inside of it). A ransomware having access to some part of the filesystem, even through the network file sharing, can also destroy all of the writable files. A wrong dd, shred, or you swapped the wrong "failed" disk, opened it to play with what's inside, before realizing it wasn't the failed one. Data memory module malfunction during a rebuild (it can screw the data of one of your members without letting you know what happened). And may be others things.<br />
<br />
A free open source versioned continuous copy system like Syncthing (in case files are being encrypted, the previous version is kept and timestamped) isn't officially meant to protect you in this kind of cases, but still does it pretty well. Conventional backups systems and solutions could also be used. In case of malicious attack, still remember to use a backup system that is different enough of the system you are trying to backup (not the same password, not the same config, not the same network, not the same place: it would be like a RAID1 on 2 partitions of the same disk).</div>Julien Robinhttps://raid.wiki.kernel.org/index.php/Irreversible_mdadm_failure_recoveryIrreversible mdadm failure recovery2019-05-05T14:38:36Z<p>Julien Robin: </p>
<hr />
<div>= Introduction =<br />
<br />
If you feel comfortable using overlays, this is always a good idea for your first attempts (so that no accidental write can happen to the real members until the correct parameter set is proven to be found, and right). You will find some help about discovering and using overlay into this guide.<br />
<br />
Every information and situation you can find here has been seriously tested on mdadm 3.4-4 and 4.1-1 before being published.<br />
<br />
Going here is normally not needed, unless mdadm conventional ways won't have any chance to work anymore (we believe that you explored others solution before going here). This section is about searching right parameters to play --create over an existing array, in order to retrieve a correctly configured access to any still available data (be it the entire filesystem if still available).<br />
<br />
Be warned: playing mdadm --create erases and overwrites at least the RAID array information area of every involved RAID members. A misuse of mdadm --create can be the reason why you are here. Without the --assume-clean parameter, one or several member data area can be entirely reconstructed (sometimes it's fine, sometimes it's not, sometimes it's recoverable, sometimes it's not - so it's better to avoid it). Without the --readonly parameter, mounting and browsing the filesystem can write things, and it can be a problem if some of the members finally aren't at the right position into the array.<br />
<br />
If the correct RAID array information is still available (then are you sure nothing else can be done with conventional ways?), to save some time, please keep the output of cat /proc/mdstat (for getting the members orders), and the output of mdadm --examine against your differents RAID members. It should be done BEFORE calling any mdadm --create command. Creation and Update time are provided into the output, so you can easily know if it contains your original array information, or if it's already lost/overwritten.<br />
<br />
If you can't manage to be sure about the position of each disk into the array, and you have a large number of disks, you are encouraged to find a way to remember the position of as much disks as possible into the array, and prepare a listing of all possible combinations (starting with the most likely ones) because 4 disk could already give you a lot of work, and above 4 disks, an huge amount of attempts may be needed. Needing you each time to verify if your data is correctly available by checking the integrity of files that are big enough to be spread on every members.<br />
<br />
Members can be entire drives or partitions inside of it. If you erased a GPT table describing a partition by mistake, or removed the partition by mistake, please create it again, unformatted, at the same place (and same size) it was when used as RAID member. The content of the partition may be completely or mostly untouched (depending whether it has just been marked deleted, or partially overwritten).<br />
<br />
= Simple using of overlays =<br />
== First step and commands ==<br />
Writing on an overlay associated with a disk or partition won't really write the data to the disk or partition (but to an overlay file instead).<br />
Reading on an overlay associated with a disk or partition will show the content of the partition, but will include what has been written to the overlay file.<br />
This clever mechanism will give you some security during your attempts, as --create attempts will overwrite RAID array informations of each member, and if something accidentally attempts to write into the array or members, the real ones are safe.<br />
<br />
mdadm should do really few writes, so you can truncate the overlay to a tiny size compared to your RAID members.<br />
<br />
If my members are /dev/sdc1, and /dev/sdd1:<br />
mkdir overlay-files<br />
cd overlay_files<br />
truncate -s1G overlay-sdc1<br />
truncate -s1G overlay-sdd1<br />
This creates two 1GB sized files : overlay-sdc1 and overlay-sdd1<br />
<br />
== Loop devices ==<br />
The overlay system will use loop devices. How does loop devices works?<br />
*Make some file available as a block device on /dev/loop0 (as read/write)<br />
losetup /dev/loop0 /address/of/someFile<br />
*Disable the block device<br />
losetup -d /dev/loop0<br />
You can use 0, 1, 2, ... and so on. losetup -a will show you currently active loop devices. losetup -f can be used to find an available number.<br />
<br />
For our overlay files:<br />
losetup /dev/loop0 overlay-sdc1<br />
losetup /dev/loop1 overlay-sdd1<br />
<br />
== Enabling overlay devices ==<br />
I'm not sure to understand everything into those commands. But it works:<br />
<br />
size1=$(blockdev --getsize /dev/sdc1);<br />
size2=$(blockdev --getsize /dev/sdd1);<br />
<br />
echo 0 $size1 snapshot /dev/sdc1 /dev/loop0 P 8 | dmsetup create sdc1<br />
echo 0 $size2 snapshot /dev/sdc1 /dev/loop1 P 8 | dmsetup create sdd1<br />
<br />
According to some software able to list block devices (gnome-disks, photorec), /dev/mapper/sdc1 and /dev/mapper/sdd1 are available, showing the size of my RAID members ($size1 and $size2). It's using the true /dev/sdc1 and /dev/sdd1 for reading, and /dev/loop0 and /dev/loop1 for writing (and reading back) changes.<br />
<br />
Some disk usage monitor like bmw-ng ('n' for disks, 't' for displaying sums) shows that playing read and write with the /dev/mapper devices (or RAID array using it) instead of the real ones seems to work, without writing anything to the real devices. If you exceeded the maximum write amount permitted by the size of your overlays files, write operations won't work (Input/output error).<br />
<br />
For monitoring your overlay devices, you can use dmsetup status.<br />
<br />
== Stopping overlay devices ==<br />
For removing the overlays when you finished or screwed them:<br />
dmsetup remove sdc1<br />
dmsetup remove sdd1<br />
losetup -d /dev/loop0<br />
losetup -d /dev/loop1<br />
rm overlay-sdc1<br />
rm overlay-sdd1<br />
<br />
= Situation analysis =<br />
<br />
A RAID array member has 2 area: RAID array information (which may be lost if you are here), and the Data area. As long as the Data area is not erased, you can consider the member as useful for the following steps. If Data area is erased, it should be considered as missing for the following steps (and if the missing member is mandatory, don't quit this page before taking a look at the "Unlucky situations" section: may be some files can be recovered).<br />
<br />
You are about to recover entire access to your filesystem as long as:<br />
*Your array is redundant and you didn't lose/overwrite any member data area<br />
*Your array isn't redundant but hopefully, you didn't lose/overwrite any member data area<br />
*Your array is redundant, you have lost/overwritten one member data area, but hopefully there is still enough others members left to get your data back<br />
*The data inside your members are arranged in one consistent and predictable way all over the array<br />
<br />
If you aren't lucky, don't quit this page before taking a look at the "Unlucky situations" section: there's still several things to verify before aborting.<br />
<br />
= Parameters recovery =<br />
<br />
Playing --create over an existing array needs you to know:<br />
*The number of members in your array.<br />
*The RAID level of your array<br />
*The chunk size of your array if using RAID0, RAID4, RAID5, RAID6, or RAID10 (generally 512K on modern mdadm versions, at least for 3.4-4 and 4.1-1)<br />
*The metadata version of your array (depends of the mdadm version initially used to create the array - 1.2 on modern versions, at least for 3.4-4 and 4.1-1)<br />
*The layout of your array if using RAID5, RAID6, or RAID10 (generally left-symmetric by default for RAID5 and 6, and can be near=2 for RAID10 if your 4 members are A, A, B, B)<br />
*The data offset of your array (depends on the size of your array members used at initial array creation - even if members have been replaced by bigger ones after, and grow --size=max used! It also depends on the version of mdadm you were using at the initial creation)<br />
*The position of the different members into the array (if it changed on the motherboard since initial creation, you may need to try several ones. If not, you probably used alphabetical order)<br />
*Is any of your member missing, or having his Data area overwritten by something like a wrong command? Which one? Because it should be declared as "missing" when running --create. If you don't know which one, you will have to do more attempts.<br />
*--assume-clean will give you the ability to make several attempts without having any member rebuilt/overwritten<br />
*--readonly will give you the ability to mount the filesystem without causing any update to any byte of the data into the array<br />
<br />
If nothing changed on your system (members size, mdadm version...) and you used default parameters, then letting mdadm choosing default ones should be fine and apply the right parameters.<br />
<br />
== Data-offset ==<br />
If you changed mdadm version since initial array creation, data-offset is likely to have changed. I have seen 8192s when created with tiny disks (5000 MiB) into mdadm 3.4-4 (while 4.1-1 defaults to 10240s for the same size, and 18432s for 10000 MiB), and 262144s for 8TB drives with mdadm 3.4-4. Didn't try creating with 8TB members on mdadm 4.1-1. May be some formula can be found into the different code versions?<br />
<br />
= Example of command =<br />
== The easy one ==<br />
Using Debian 9, mdadm 3.4-4, a RAID array was created with with 3 x 8TB members, using:<br />
<br />
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1<br />
<br />
If nothing changed and I'm still using the same mdadm version, typing the same command again (but appending --assume-clean --readonly for safety) will be fine: <br />
<br />
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/mapper/sdb1 /dev/mapper/sdc1 /dev/mapper/sdd1 --assume-clean --readonly<br />
<br />
Gnome-disks shows the file system again and I can mount it (trying read-only before, mount -o ro,noload /dev/mdXX /media/raid-volume) : everything is fine and every member is in the right place into the array. Then, do not forget to do a filesystem check and repair after you ensured that you are using the correct parameters, in case some brutal interruption happened previously on the array.<br />
<br />
== A more precise one ==<br />
For creating the array again after having upgraded from mdadm 3.4-4 to mdadm 4.1-1, data offset should be specified because the automatically selected value for a given size changed. <br />
<br />
Also, disk position changed on my motherboard. After a precise analysis of the previous version defaults parameters for this array, in this case, the working command for finding back my array finally was:<br />
<br />
mdadm --create /dev/md0 --level=5 --chunk=512K --metadata=1.2 --layout left-symmetric --data-offset=262144s --raid-devices=3 /dev/mapper/sdd1 /dev/mapper/sde1 /dev/mapper/sdb1 --assume-clean --readonly<br />
<br />
== One facultative member is missing ==<br />
If the second disk is missing, type missing instead of its block device name. On a redundant RAID array, data will still be available. Of course, once your array is back working, you will have to --add the missing member soon or late for safety, but after you ensured that the parameters are OK and your data available. The member you will add will be rebuilt with consistent data. <br />
<br />
== The array was created with 2 disks, and grown to 3 ==<br />
Having the array initially created with 2 disks before having the 3rd one added and the array grown, give the exact same final arrangement compared to a 3 disks array creation.<br />
<br />
= In case of accidental rebuild =<br />
If you forgot the --assume-clean parameter, a rebuild process may have rewritten some of your members.<br />
But if you were using --readonly, then you are fine (in both case, I didn't see any rebuild process starting).<br />
<br />
*In case of RAID 0, there is no possible rebuild process. That's fine.<br />
*In case of RAID 1, only the 1st member is read, the others ones are overwritten with a copy of it.<br />
*In case of RAID 5, the last RAID member given in parameter will be overwritten by reading and processing data of the others given members. If your original array was already using RAID5, and the member count is correct, it's likely to be fine.<br />
*In case of RAID 6, every member is rewritten by reading and computing data from others ones. If you are wrong about the position of 2 disks out of 4, you can recover, see below. If its more, it likely to be lost (but I didn't try with array bigger than 4 disks)<br />
*In case of RAID 10, with 4 members and default parameters, 1st drive is read and copied to the 2nd one, while 3rd drive is read and copied to the 4th one.<br />
<br />
In case of RAID 5:<br />
*Even with wrong order or wrong chunk size, and any wrong RAID5 or RAID4-like layout, if the others members data is fine, and the amount of members is correct, the parity calculation will be correct. Although the computer may not interpret the data the right manner (which bit is parity or data) until you find the right parameters. <br />
*But if you forget the --assume-clean parameter in others cases (like incorrect amount of disks, or RAID level, or reading a lost member to calculate and rewrite over a good one), consider the overwritten disk as missing.<br />
<br />
In case of RAID6:<br />
*Driving some tests about RAID6, having the members accidentally rebuilt with 2 out of 4 disks in the wrong position (1243 - for example - instead of 1234) an erroneous reconstruction is applied on all of the 4 members. Trying to access the array filesystem, the files were incorrect. But --creating the RAID again, with the 4 disks back at the correct position (and --assume-clean and --readonly) restored the access to the correct data. Replaying the same command after ensuring the position are right, and without --assume-clean and --readonly also restored the original data over all of the 4 disks (sha1sum controlled, RAID information excluded).<br />
*Driving another test, it seems that if you have accidentally rebuilt your array with 2 out of 4 disks in the wrong position (still 1243 instead of 1234 for example), mounting the array with only disk 1 and 2 won't be fine. Nothing more with 1, 2, and 3: the 4 disks should be used. This is probably due to the fact that the erroneous reconstruction is done over all of the members, so all of them are needed to revert this error.<br />
<br />
In others cases:<br />
In case correct data seems to be erased from only one of your members, if it wasn't mandatory (if remaining fine members are enough), you will be able to resync it later by --adding it as a new one, but only once your correct parameters are found, and your array back working. If the overwritten member was mandatory, hope you played mdadm --stop. Please find a way to verify if the data is really gone, and if it's gone, go to the "Unlucky situations" section.<br />
<br />
= Help needed, no combination is giving back access to the data =<br />
If you have 3 members and you are sure that data is still available on both of them, try every order: <br />
*Disk 1, Disk 2, Disk 3 <br />
*Disk 1, Disk 3, Disk 2<br />
*Disk 2, Disk 1, Disk 3<br />
*Disk 2, Disk 3, Disk 1<br />
*Disk 3, Disk 1, Disk 2 <br />
*Disk 3, Disk 2, Disk 1<br />
<br />
If it's not working, you should find a way to ensure data-offset (and may be others values) are correct, by creating a reconstitution of your original array creation environment to find what default values mdadm probably used. If your values are good, your data are back.<br />
If one of the members has its data missing, and you know which one (if not, its longer), check every combination:<br />
<br />
Member 1/3 is missing: X23, X32, 2X3, 23X, 3X2, 32X<br />
<br />
Member 2/3 is missing: 1X3, 13X, X13, X31, 31X, 3X1<br />
<br />
Member 3/3 is missing: 12X, 1X2, 21X, 2X1, X12, X21<br />
<br />
The numbers stand for the different members block device names, X stands for the "missing" word, in order to represent the parameters order into --create command.<br />
<br />
If you have 4 members, there is 24 possible combinations:<br />
*1234, 1243, 1324, 1342, 1423, 1432<br />
*2134, 2143, 2314, 2341, 2413, 2431<br />
*3124, 3142, 3214, 3241, 3412, 3421<br />
*4123, 4132, 4213, 4231, 4312, 4321<br />
<br />
If you are using RAID6 with 4 members, as 2 disks out of 4 are optional, let's save some time, try the 12 following attempts :<br />
*1st disk in 1st place, 2nd disk in the others places : 12xx 1x2x 1xx2<br />
*1st disk in 2nd place, 2nd disk in the others places : 21xx x12x x1x2<br />
*1st disk in 3rd place, 2nd disk in the others places : 2x1x x21x xx12<br />
*1st disk in 4th place, 2nd disk in the others places : 2xx1 x2x1 xx21<br />
<br />
Still about the RAID6 with 4 disks, if the disk number 2 and disk number 4 have lost their data or aren't available for any reason, you will have to use disk numbers 1 and 3 to do the same position attempts (instead of disk 1 and 2). If you are sure 2 disks are badly overwritten but you don't know which ones, 12, 13, 14, 23, 24, 34 are the possible couples: up to 6 x 12 try may be required to find back your data.<br />
<br />
= Unlucky situations =<br />
<br />
== The RAID array is split into 2 arrangements ==<br />
*Did the reshape/conversion actually started? If you are sure that the answer is no (stuck at 0% with no disk activity, backup file is there but contains nothing, nothing started and it's completely stuck), then things will be easy. Find the parameters set of the unchanged arrangement.<br />
<br />
*If the reshape/conversion actually started and moved some data, this kind of interruption can be restarted just by assembling the array again, and mount/use it as nothing happened. If it doesn't work, see if there is no parameters to force things to continue: don't follow this guide unless RAID information of your members have already been erased, or if you're sure that nothing else worked.<br />
<br />
*If a member failed during reshaping, and your array is redundant, don't play --create! You can add a new member: reshaping will finish in degraded mode, and the last member will be integrated and rebuilt just after.<br />
<br />
*If you have no other choice that playing --create with 2 different arrangements over 2 half of the array, unless someone have a better idea, you're falling into forensics : you'll need to find 2 sets of parameters, and probably need to use a file recovery scanner tool with both parameters set on the array. Be brave! The second set of parameters will probably be the same as the first one, with 1 more member (and raid-devices count set to one more). <br />
<br />
== Mandatory member lost ==<br />
So, your RAID is not redundant (or not enough), and a mandatory member (or the data area into it) is screwed.<br />
<br />
*Is your RAID member partially screwed/overwritten? If mdadm --create (or anything else) started to overwrite it, but you played mdadm --stop before it's entirely overwritten (or stopped anything that was overwriting the data into it), some of the data is still available. Find back your correct parameters (but you won't see any ext4 partition so it will be more difficult to know it you have the right ones) and use some file recovery scanner tool over your resulting array. You may still have the surprise to see some files that aren't completely erased.<br />
<br />
*Also, remember that having mdadm --create accidentally overwriting one of the disks should be avoided, but if it accidentally happens, in some case, calculated data overwritten to the disk may be the same that data which was already into it. In such case, there is a chance that your data is totally fine. You can always try to use the member as if it's OK: if it is, your file system will be made available again.<br />
<br />
*If one of the disks physically failed, but the data is still enclosed into it, see if RAW data into it can be rescued and cloned by some company, or if available sectors can be rescued by dd-rescue. See [[Replacing a failed drive]] for more detailed information about drive failure, including failure in non-redundant arrays.<br />
<br />
= Making sure that none of my arrays will never be lost anymore in the future =<br />
<br />
You can't, because it won't always be your fault. We believe that having succeeded or not into your recovery, you are probably interested into clever backup approaches for securing the recovered or future data.<br />
<br />
Indeed, a defective power supply can destroy some of your disks. A violent shock on the case containing all the disks can cause several of them to fail. A fire incident too. The server can be stolen (with disks inside of it). A ransomware having access to some part of the filesystem, even through the network file sharing, can also destroy all of the writable files. A wrong dd, shred, or you swapped the wrong "failed" disk, opened it to play with what's inside, before realizing it wasn't the failed one. Data memory module malfunction during a rebuild (it can screw the data of one of your members without letting you know what happened). And may be others things.<br />
<br />
A free open source versioned continuous copy system like Syncthing (in case files are being encrypted, the previous version is kept and timestamped) isn't officially meant to protect you in this kind of cases, but still does it pretty well. Conventional backups systems and solutions could also be used. In case of malicious attack, still remember to use a backup system that is different enough of the system you are trying to backup (not the same password, not the same config, not the same network, not the same place: it would be like a RAID1 on 2 partitions of the same disk).</div>Julien Robinhttps://raid.wiki.kernel.org/index.php/Irreversible_mdadm_failure_recoveryIrreversible mdadm failure recovery2019-05-05T14:12:17Z<p>Julien Robin: Including overlays simple explanation - and make usage of it into the guide, for an absolute safety following it.</p>
<hr />
<div>= Introduction =<br />
<br />
If you feel comfortable using overlays, this is always a good idea for your first attempts (so that no accidental write can happen to the real members until the correct parameter set is proven to be found, and right). You will find some help about discovering and using overlay into this guide.<br />
<br />
Every information and situation you can find here has been seriously tested on mdadm 3.4-4 and 4.1-1 before being published.<br />
<br />
Going here is normally not needed, unless mdadm conventional ways won't have any chance to work anymore (we believe that you explored others solution before going here). This section is about searching right parameters to play --create over an existing array, in order to retrieve a correctly configured access to any still available data (be it the entire filesystem if still available).<br />
<br />
Be warned: playing mdadm --create erases and overwrites at least the RAID array information area of every involved RAID members. A misuse of mdadm --create can be the reason why you are here. Without the --assume-clean parameter, one or several member data area can be entirely reconstructed (sometimes it's fine, sometimes it's not, sometimes it's recoverable, sometimes it's not - so it's better to avoid it). Without the --readonly parameter, mounting and browsing the filesystem can write things, and it can be a problem if some of the members finally aren't at the right position into the array.<br />
<br />
If the correct RAID array information is still available (then are you sure nothing else can be done with conventional ways?), to save some time, please keep the output of cat /proc/mdstat (for getting the members orders), and the output of mdadm --examine against your differents RAID members. It should be done BEFORE calling any mdadm --create command. Creation and Update time are provided into the output, so you can easily know if it contains your original array information, or if it's already lost/overwritten.<br />
<br />
If you can't manage to be sure about the position of each disk into the array, and you have a large number of disks, you are encouraged to find a way to remember the position of as much disks as possible into the array, and prepare a listing of all possible combinations (starting with the most likely ones) because 4 disk could already give you a lot of work, and above 4 disks, an huge amount of attempts may be needed. Needing you each time to verify if your data is correctly available by checking the integrity of files that are big enough to be spread on every members.<br />
<br />
Members can be entire drives or partitions inside of it. If you erased a GPT table describing a partition by mistake, or removed the partition by mistake, please create it again, unformatted, at the same place (and same size) it was when used as RAID member. The content of the partition may be completely or mostly untouched (depending whether it has just been marked deleted, or partially overwritten).<br />
<br />
= Simple using of overlays =<br />
== First step and commands ==<br />
Writing on an overlay associated with a disk or partition won't really write the data to the disk or partition (but to an overlay file instead).<br />
Reading on an overlay associated with a disk or partition will show the content of the partition, but will include what has been written to the overlay file.<br />
This clever mechanism will give you some security during your attempts, as --create attempts will overwrite RAID array informations of each member, and if something accidentally attempts to write into the array or members, the real ones are safe.<br />
<br />
mdadm should do really few writes, so you can truncate the overlay to a tiny size compared to your RAID members.<br />
<br />
If my members are /dev/sdc1, and /dev/sdd1:<br />
mkdir overlay-files<br />
cd overlay_files<br />
truncate -s1G overlay-sdc1<br />
truncate -s1G overlay-sdd1<br />
This creates two 1GB sized files : overlay-sdc1 and overlay-sdd1<br />
<br />
== Loop devices ==<br />
The overlay system will use loop devices. How does loop devices works?<br />
*Make some file available as a block device on /dev/loop0 (as read/write)<br />
losetup /dev/loop0 /address/of/someFile<br />
*Disable the block device<br />
losetup -d /dev/loop0<br />
You can use 0, 1, 2, ... and so on. losetup -a will show you currently active loop devices. losetup -f can be used to find an available number.<br />
<br />
For our overlay files:<br />
/losetup /dev/loop0 overlay-sdc1<br />
/losetup /dev/loop1 overlay-sdd1<br />
<br />
== Enabling overlay devices ==<br />
I'm not sure to understand everything into those commands. But it works:<br />
<br />
size1=$(blockdev --getsize /dev/sdc1);<br />
size2=$(blockdev --getsize /dev/sdd1);<br />
<br />
echo 0 $size1 snapshot /dev/sdc1 /dev/loop0 P 8 | dmsetup create sdc1<br />
echo 0 $size2 snapshot /dev/sdc1 /dev/loop1 P 8 | dmsetup create sdd1<br />
<br />
According to some software able to list block devices (gnome-disks, photorec), /dev/mapper/sdc1 and /dev/mapper/sdd1 are available, showing the size of my RAID members ($size1 and $size2). It's using the true /dev/sdc1 and /dev/sdd1 for reading, and /dev/loop0 and /dev/loop1 for writing (and reading back) changes.<br />
<br />
Some disk usage monitor like bmw-ng ('n' for disks, 't' for displaying sums) shows that playing read and write with the /dev/mapper devices (or RAID array using it) instead of the real ones seems to work, without writing anything to the real devices. If you exceeded the maximum write amount permitted by the size of your overlays files, write operations won't work (Input/output error).<br />
<br />
For monitoring your overlay devices, you can use dmsetup status.<br />
<br />
== Stopping overlay devices ==<br />
For removing the overlays when you finished or screwed them:<br />
dmsetup remove sdc1<br />
dmsetup remove sdd1<br />
losetup -d /dev/loop0<br />
losetup -d /dev/loop1<br />
rm overlay-sdc1<br />
rm overlay-sdd1<br />
<br />
= Situation analysis =<br />
<br />
A RAID array member has 2 area: RAID array information (which may be lost if you are here), and the Data area. As long as the Data area is not erased, you can consider the member as useful for the following steps. If Data area is erased, it should be considered as missing for the following steps (and if the missing member is mandatory, don't quit this page before taking a look at the "Unlucky situations" section: may be some files can be recovered).<br />
<br />
You are about to recover entire access to your filesystem as long as:<br />
*Your array is redundant and you didn't lose/overwrite any member data area<br />
*Your array isn't redundant but hopefully, you didn't lose/overwrite any member data area<br />
*Your array is redundant, you have lost/overwritten one member data area, but hopefully there is still enough others members left to get your data back<br />
*The data inside your members are arranged in one consistent and predictable way all over the array<br />
<br />
If you aren't lucky, don't quit this page before taking a look at the "Unlucky situations" section: there's still several things to verify before aborting.<br />
<br />
= Parameters recovery =<br />
<br />
Playing --create over an existing array needs you to know:<br />
*The number of members in your array.<br />
*The RAID level of your array<br />
*The chunk size of your array if using RAID0, RAID4, RAID5, RAID6, or RAID10 (generally 512K on modern mdadm versions, at least for 3.4-4 and 4.1-1)<br />
*The metadata version of your array (depends of the mdadm version initially used to create the array - 1.2 on modern versions, at least for 3.4-4 and 4.1-1)<br />
*The layout of your array if using RAID5, RAID6, or RAID10 (generally left-symmetric by default for RAID5 and 6, and can be near=2 for RAID10 if your 4 members are A, A, B, B)<br />
*The data offset of your array (depends on the size of your array members used at initial array creation - even if members have been replaced by bigger ones after, and grow --size=max used! It also depends on the version of mdadm you were using at the initial creation)<br />
*The position of the different members into the array (if it changed on the motherboard since initial creation, you may need to try several ones. If not, you probably used alphabetical order)<br />
*Is any of your member missing, or having his Data area overwritten by something like a wrong command? Which one? Because it should be declared as "missing" when running --create. If you don't know which one, you will have to do more attempts.<br />
*--assume-clean will give you the ability to make several attempts without having any member rebuilt/overwritten<br />
*--readonly will give you the ability to mount the filesystem without causing any update to any byte of the data into the array<br />
<br />
If nothing changed on your system (members size, mdadm version...) and you used default parameters, then letting mdadm choosing default ones should be fine and apply the right parameters.<br />
<br />
== Data-offset ==<br />
If you changed mdadm version since initial array creation, data-offset is likely to have changed. I have seen 8192s when created with tiny disks (5000 MiB) into mdadm 3.4-4 (while 4.1-1 defaults to 10240s for the same size, and 18432s for 10000 MiB), and 262144s for 8TB drives with mdadm 3.4-4. Didn't try creating with 8TB members on mdadm 4.1-1. May be some formula can be found into the different code versions?<br />
<br />
= Example of command =<br />
== The easy one ==<br />
Using Debian 9, mdadm 3.4-4, a RAID array was created with with 3 x 8TB members, using:<br />
<br />
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1<br />
<br />
If nothing changed and I'm still using the same mdadm version, typing the same command again (but appending --assume-clean --readonly for safety) will be fine: <br />
<br />
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/mapper/sdb1 /dev/mapper/sdc1 /dev/mapper/sdd1 --assume-clean --readonly<br />
<br />
Gnome-disks shows the file system again and I can mount it (trying read-only before, mount -o ro,noload /dev/mdXX /media/raid-volume) : everything is fine and every member is in the right place into the array. Then, do not forget to do a filesystem check and repair after you ensured that you are using the correct parameters, in case some brutal interruption happened previously on the array.<br />
<br />
== A more precise one ==<br />
For creating the array again after having upgraded from mdadm 3.4-4 to mdadm 4.1-1, data offset should be specified because the automatically selected value for a given size changed. <br />
<br />
Also, disk position changed on my motherboard. After a precise analysis of the previous version defaults parameters for this array, in this case, the working command for finding back my array finally was:<br />
<br />
mdadm --create /dev/md0 --level=5 --chunk=512K --metadata=1.2 --layout left-symmetric --data-offset=262144s --raid-devices=3 /dev/mapper/sdd1 /dev/mapper/sde1 /dev/mapper/sdb1 --assume-clean --readonly<br />
<br />
== One facultative member is missing ==<br />
If the second disk is missing, type missing instead of its block device name. On a redundant RAID array, data will still be available. Of course, once your array is back working, you will have to --add the missing member soon or late for safety, but after you ensured that the parameters are OK and your data available. The member you will add will be rebuilt with consistent data. <br />
<br />
== The array was created with 2 disks, and grown to 3 ==<br />
Having the array initially created with 2 disks before having the 3rd one added and the array grown, give the exact same final arrangement compared to a 3 disks array creation.<br />
<br />
= In case of accidental rebuild =<br />
If you forgot the --assume-clean parameter, a rebuild process may have rewritten some of your members.<br />
But if you were using --readonly, then you are fine (in both case, I didn't see any rebuild process starting).<br />
<br />
*In case of RAID 0, there is no possible rebuild process. That's fine.<br />
*In case of RAID 1, only the 1st member is read, the others ones are overwritten with a copy of it.<br />
*In case of RAID 5, the last RAID member given in parameter will be overwritten by reading and processing data of the others given members. If your original array was already using RAID5, and the member count is correct, it's likely to be fine.<br />
*In case of RAID 6, every member is rewritten by reading and computing data from others ones. If you are wrong about the position of 2 disks out of 4, you can recover, see below. If its more, it likely to be lost (but I didn't try with array bigger than 4 disks)<br />
*In case of RAID 10, with 4 members and default parameters, 1st drive is read and copied to the 2nd one, while 3rd drive is read and copied to the 4th one.<br />
<br />
In case of RAID 5:<br />
*Even with wrong order or wrong chunk size, and any wrong RAID5 or RAID4-like layout, if the others members data is fine, and the amount of members is correct, the parity calculation will be correct. Although the computer may not interpret the data the right manner (which bit is parity or data) until you find the right parameters. <br />
*But if you forget the --assume-clean parameter in others cases (like incorrect amount of disks, or RAID level, or reading a lost member to calculate and rewrite over a good one), consider the overwritten disk as missing.<br />
<br />
In case of RAID6:<br />
*Driving some tests about RAID6, having the members accidentally rebuilt with 2 out of 4 disks in the wrong position (1243 - for example - instead of 1234) an erroneous reconstruction is applied on all of the 4 members. Trying to access the array filesystem, the files were incorrect. But --creating the RAID again, with the 4 disks back at the correct position (and --assume-clean and --readonly) restored the access to the correct data. Replaying the same command after ensuring the position are right, and without --assume-clean and --readonly also restored the original data over all of the 4 disks (sha1sum controlled, RAID information excluded).<br />
*Driving another test, it seems that if you have accidentally rebuilt your array with 2 out of 4 disks in the wrong position (still 1243 instead of 1234 for example), mounting the array with only disk 1 and 2 won't be fine. Nothing more with 1, 2, and 3: the 4 disks should be used. This is probably due to the fact that the erroneous reconstruction is done over all of the members, so all of them are needed to revert this error.<br />
<br />
In others cases:<br />
In case correct data seems to be erased from only one of your members, if it wasn't mandatory (if remaining fine members are enough), you will be able to resync it later by --adding it as a new one, but only once your correct parameters are found, and your array back working. If the overwritten member was mandatory, hope you played mdadm --stop. Please find a way to verify if the data is really gone, and if it's gone, go to the "Unlucky situations" section.<br />
<br />
= Help needed, no combination is giving back access to the data =<br />
If you have 3 members and you are sure that data is still available on both of them, try every order: <br />
*Disk 1, Disk 2, Disk 3 <br />
*Disk 1, Disk 3, Disk 2<br />
*Disk 2, Disk 1, Disk 3<br />
*Disk 2, Disk 3, Disk 1<br />
*Disk 3, Disk 1, Disk 2 <br />
*Disk 3, Disk 2, Disk 1<br />
<br />
If it's not working, you should find a way to ensure data-offset (and may be others values) are correct, by creating a reconstitution of your original array creation environment to find what default values mdadm probably used. If your values are good, your data are back.<br />
If one of the members has its data missing, and you know which one (if not, its longer), check every combination:<br />
<br />
Member 1/3 is missing: X23, X32, 2X3, 23X, 3X2, 32X<br />
<br />
Member 2/3 is missing: 1X3, 13X, X13, X31, 31X, 3X1<br />
<br />
Member 3/3 is missing: 12X, 1X2, 21X, 2X1, X12, X21<br />
<br />
The numbers stand for the different members block device names, X stands for the "missing" word, in order to represent the parameters order into --create command.<br />
<br />
If you have 4 members, there is 24 possible combinations:<br />
*1234, 1243, 1324, 1342, 1423, 1432<br />
*2134, 2143, 2314, 2341, 2413, 2431<br />
*3124, 3142, 3214, 3241, 3412, 3421<br />
*4123, 4132, 4213, 4231, 4312, 4321<br />
<br />
If you are using RAID6 with 4 members, as 2 disks out of 4 are optional, let's save some time, try the 12 following attempts :<br />
*1st disk in 1st place, 2nd disk in the others places : 12xx 1x2x 1xx2<br />
*1st disk in 2nd place, 2nd disk in the others places : 21xx x12x x1x2<br />
*1st disk in 3rd place, 2nd disk in the others places : 2x1x x21x xx12<br />
*1st disk in 4th place, 2nd disk in the others places : 2xx1 x2x1 xx21<br />
<br />
Still about the RAID6 with 4 disks, if the disk number 2 and disk number 4 have lost their data or aren't available for any reason, you will have to use disk numbers 1 and 3 to do the same position attempts (instead of disk 1 and 2). If you are sure 2 disks are badly overwritten but you don't know which ones, 12, 13, 14, 23, 24, 34 are the possible couples: up to 6 x 12 try may be required to find back your data.<br />
<br />
= Unlucky situations =<br />
<br />
== The RAID array is split into 2 arrangements ==<br />
*Did the reshape/conversion actually started? If you are sure that the answer is no (stuck at 0% with no disk activity, backup file is there but contains nothing, nothing started and it's completely stuck), then things will be easy. Find the parameters set of the unchanged arrangement.<br />
<br />
*If the reshape/conversion actually started and moved some data, this kind of interruption can be restarted just by assembling the array again, and mount/use it as nothing happened. If it doesn't work, see if there is no parameters to force things to continue: don't follow this guide unless RAID information of your members have already been erased, or if you're sure that nothing else worked.<br />
<br />
*If a member failed during reshaping, and your array is redundant, don't play --create! You can add a new member: reshaping will finish in degraded mode, and the last member will be integrated and rebuilt just after.<br />
<br />
*If you have no other choice that playing --create with 2 different arrangements over 2 half of the array, unless someone have a better idea, you're falling into forensics : you'll need to find 2 sets of parameters, and probably need to use a file recovery scanner tool with both parameters set on the array. Be brave! The second set of parameters will probably be the same as the first one, with 1 more member (and raid-devices count set to one more). <br />
<br />
== Mandatory member lost ==<br />
So, your RAID is not redundant (or not enough), and a mandatory member (or the data area into it) is screwed.<br />
<br />
*Is your RAID member partially screwed/overwritten? If mdadm --create (or anything else) started to overwrite it, but you played mdadm --stop before it's entirely overwritten (or stopped anything that was overwriting the data into it), some of the data is still available. Find back your correct parameters (but you won't see any ext4 partition so it will be more difficult to know it you have the right ones) and use some file recovery scanner tool over your resulting array. You may still have the surprise to see some files that aren't completely erased.<br />
<br />
*Also, remember that having mdadm --create accidentally overwriting one of the disks should be avoided, but if it accidentally happens, in some case, calculated data overwritten to the disk may be the same that data which was already into it. In such case, there is a chance that your data is totally fine. You can always try to use the member as if it's OK: if it is, your file system will be made available again.<br />
<br />
*If one of the disks physically failed, but the data is still enclosed into it, see if RAW data into it can be rescued and cloned by some company, or if available sectors can be rescued by dd-rescue. See [[Replacing a failed drive]] for more detailed information about drive failure, including failure in non-redundant arrays.<br />
<br />
= Making sure that none of my arrays will never be lost anymore in the future =<br />
<br />
You can't, because it won't always be your fault. We believe that having succeeded or not into your recovery, you are probably interested into clever backup approaches for securing the recovered or future data.<br />
<br />
Indeed, a defective power supply can destroy some of your disks. A violent shock on the case containing all the disks can cause several of them to fail. A fire incident too. The server can be stolen (with disks inside of it). A ransomware having access to some part of the filesystem, even through the network file sharing, can also destroy all of the writable files. A wrong dd, shred, or you swapped the wrong "failed" disk, opened it to play with what's inside, before realizing it wasn't the failed one. Data memory module malfunction during a rebuild (it can screw the data of one of your members without letting you know what happened). And may be others things.<br />
<br />
A free open source versioned continuous copy system like Syncthing (in case files are being encrypted, the previous version is kept and timestamped) isn't officially meant to protect you in this kind of cases, but still does it pretty well. Conventional backups systems and solutions could also be used. In case of malicious attack, still remember to use a backup system that is different enough of the system you are trying to backup (not the same password, not the same config, not the same network, not the same place: it would be like a RAID1 on 2 partitions of the same disk).</div>Julien Robinhttps://raid.wiki.kernel.org/index.php/Irreversible_mdadm_failure_recoveryIrreversible mdadm failure recovery2019-05-05T03:04:36Z<p>Julien Robin: word exactitude improvement</p>
<hr />
<div>== Introduction ==<br />
<br />
If you feel comfortable using overlays, this is always a good idea (so that no accidental write can happen to the real members).<br />
<br />
Every information and situation you can find here has been seriously tested on mdadm 3.4-4 and 4.1-1 before being published.<br />
<br />
Going here is normally not needed, unless mdadm conventional ways won't have any chance to work anymore (we believe that you explored others solution before going here). This section is about searching right parameters to play --create over an existing array, in order to retrieve a correctly configured access to any still available data (be it the entire filesystem if still available).<br />
<br />
Be warned: playing mdadm --create erases and overwrites at least the RAID array information area of every involved RAID members. A misuse of mdadm --create can be the reason why you are here. Without the --assume-clean parameter, one or several member data area can be entirely reconstructed (sometimes it's fine, sometimes it's not, sometimes it's recoverable, sometimes it's not - so it's better to avoid it). Without the --readonly parameter, mounting and browsing the filesystem can write things, and it can be a problem if some of the members finally aren't at the right position into the array.<br />
<br />
If the correct RAID array information is still available (then are you sure nothing else can be done with conventional ways?), to save some time, please keep the output of cat /proc/mdstat (for getting the members orders), and the output of mdadm --examine against your differents RAID members. It should be done BEFORE calling any mdadm --create command. Creation and Update time are provided into the output, so you can easily know if it contains your original array information, or if it's already lost/overwritten.<br />
<br />
If you can't manage to be sure about the position of each disk into the array, and you have a large number of disks, you are encouraged to find a way to remember the position of as much disks as possible into the array, and prepare a listing of all possible combinations (starting with the most likely ones) because 4 disk could already give you a lot of work, and above 4 disks, an huge amount of attempts may be needed. Needing you each time to verify if your data is correctly available by checking the integrity of files that are big enough to be spread on every members.<br />
<br />
Members can be entire drives or partitions inside of it. If you erased a GPT table describing a partition by mistake, or removed the partition by mistake, please create it again, unformatted, at the same place (and same size) it was when used as RAID member. The content of the partition may be completely or mostly untouched (depending whether it has just been marked deleted, or partially overwritten).<br />
<br />
== Situation analysis ==<br />
<br />
A RAID array member has 2 area: RAID array information (which may be lost if you are here), and the Data area. As long as the Data area is not erased, you can consider the member as useful for the following steps. If Data area is erased, it should be considered as missing for the following steps (and if the missing member is mandatory, don't quit this page before taking a look at the "Unlucky situations" section: may be some files can be recovered).<br />
<br />
You are about to recover entire access to your filesystem as long as:<br />
*Your array is redundant and you didn't lose/overwrite any member data area<br />
*Your array isn't redundant but hopefully, you didn't lose/overwrite any member data area<br />
*Your array is redundant, you have lost/overwritten one member data area, but hopefully there is still enough others members left to get your data back<br />
*The data inside your members are arranged in one consistent and predictable way all over the array<br />
<br />
If you aren't lucky, don't quit this page before taking a look at the "Unlucky situations" section: there's still several things to verify before aborting.<br />
<br />
== Parameters recovery ==<br />
<br />
Playing --create over an existing array needs you to know:<br />
*The number of members in your array.<br />
*The RAID level of your array<br />
*The chunk size of your array if using RAID0, RAID4, RAID5, RAID6, or RAID10 (generally 512K on modern mdadm versions, at least for 3.4-4 and 4.1-1)<br />
*The metadata version of your array (depends of the mdadm version initially used to create the array - 1.2 on modern versions, at least for 3.4-4 and 4.1-1)<br />
*The layout of your array if using RAID5, RAID6, or RAID10 (generally left-symmetric by default for RAID5 and 6, and can be near=2 for RAID10 if your 4 members are A, A, B, B)<br />
*The data offset of your array (depends on the size of your array members used at initial array creation - even if members have been replaced by bigger ones after, and grow --size=max used! It also depends on the version of mdadm you were using at the initial creation)<br />
*The position of the different members into the array (if it changed on the motherboard since initial creation, you may need to try several ones. If not, you probably used alphabetical order)<br />
*Is any of your member missing, or having his Data area overwritten by something like a wrong command? Which one? Because it should be declared as "missing" when running --create. If you don't know which one, you will have to do more attempts.<br />
*--assume-clean will give you the ability to make several attempts without having any member rebuilt/overwritten<br />
*--readonly will give you the ability to mount the filesystem without causing any update to any byte of the data into the array<br />
<br />
If nothing changed on your system (members size, mdadm version...) and you used default parameters, then letting mdadm choosing default ones should be fine and apply the right parameters.<br />
<br />
== Data-offset ==<br />
If you changed mdadm version since initial array creation, data-offset is likely to have changed. I have seen 8192s when created with tiny disks (5000 MiB) into mdadm 3.4-4 (while 4.1-1 defaults to 10240s for the same size, and 18432s for 10000 MiB), and 262144s for 8TB drives with mdadm 3.4-4. Didn't try creating with 8TB members on mdadm 4.1-1. May be some formula can be found into the different code versions?<br />
<br />
= Example of command =<br />
== The easy one ==<br />
Using Debian 9, mdadm 3.4-4, a RAID array was created with with 3 x 8TB members, using:<br />
<br />
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1<br />
<br />
If nothing changed and I'm still using the same mdadm version, typing the same command again (but appending --assume-clean --readonly for safety) will be fine: <br />
<br />
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1 --assume-clean --readonly<br />
<br />
Gnome-disks shows the file system again and I can mount it (trying read-only before, mount -o ro,noload /dev/mdXX /media/raid-volume) : everything is fine and every member is in the right place into the array. Then, do not forget to do a filesystem check and repair after you ensured that you are using the correct parameters, in case some brutal interruption happened previously on the array.<br />
<br />
== A more precise one ==<br />
For creating the array again after having upgraded from mdadm 3.4-4 to mdadm 4.1-1, data offset should be specified because the automatically selected value for a given size changed. <br />
<br />
Also, disk position changed on my motherboard. After a precise analysis of the previous version defaults parameters for this array, in this case, the working command for finding back my array finally was:<br />
<br />
mdadm --create /dev/md0 --level=5 --chunk=512K --metadata=1.2 --layout left-symmetric --data-offset=262144s --raid-devices=3 /dev/sdd1 /dev/sde1 /dev/sdb1 --assume-clean --readonly<br />
<br />
== One facultative member is missing ==<br />
If the second disk is missing, type missing instead of its block device name. On a redundant RAID array, data will still be available. Of course, once your array is back working, you will have to --add the missing member soon or late for safety, but after you ensured that the parameters are OK and your data available. The member you will add will be rebuilt with consistent data. <br />
<br />
== The array was created with 2 disks, and grown to 3 ==<br />
Having the array initially created with 2 disks before having the 3rd one added and the array grown, give the exact same final arrangement compared to a 3 disks array creation.<br />
<br />
= In case of accidental rebuild =<br />
If you forgot the --assume-clean parameter, a rebuild process may have rewritten some of your members.<br />
But if you were using --readonly, then you are fine (in both case, I didn't see any rebuild process starting).<br />
<br />
*In case of RAID 0, there is no possible rebuild process. That's fine.<br />
*In case of RAID 1, only the 1st member is read, the others ones are overwritten with a copy of it.<br />
*In case of RAID 5, the last RAID member given in parameter will be overwritten by reading and processing data of the others given members. If your original array was already using RAID5, and the member count is correct, it's likely to be fine.<br />
*In case of RAID 6, every member is rewritten by reading and computing data from others ones. If you are wrong about the position of 2 disks out of 4, you can recover, see below. If its more, it likely to be lost (but I didn't try with array bigger than 4 disks)<br />
*In case of RAID 10, with 4 members and default parameters, 1st drive is read and copied to the 2nd one, while 3rd drive is read and copied to the 4th one.<br />
<br />
In case of RAID 5:<br />
*Even with wrong order or wrong chunk size, and any wrong RAID5 or RAID4-like layout, if the others members data is fine, and the amount of members is correct, the parity calculation will be correct. Although the computer may not interpret the data the right manner (which bit is parity or data) until you find the right parameters. <br />
*But if you forget the --assume-clean parameter in others cases (like incorrect amount of disks, or RAID level, or reading a lost member to calculate and rewrite over a good one), consider the overwritten disk as missing.<br />
<br />
In case of RAID6:<br />
*Driving some tests about RAID6, having the members accidentally rebuilt with 2 out of 4 disks in the wrong position (1243 - for example - instead of 1234) an erroneous reconstruction is applied on all of the 4 members. Trying to access the array filesystem, the files were incorrect. But --creating the RAID again, with the 4 disks back at the correct position (and --assume-clean and --readonly) restored the access to the correct data. Replaying the same command after ensuring the position are right, and without --assume-clean and --readonly also restored the original data over all of the 4 disks (sha1sum controlled, RAID information excluded).<br />
*Driving another test, it seems that if you have accidentally rebuilt your array with 2 out of 4 disks in the wrong position (still 1243 instead of 1234 for example), mounting the array with only disk 1 and 2 won't be fine. Nothing more with 1, 2, and 3: the 4 disks should be used. This is probably due to the fact that the erroneous reconstruction is done over all of the members, so all of them are needed to revert this error.<br />
<br />
In others cases:<br />
In case correct data seems to be erased from only one of your members, if it wasn't mandatory (if remaining fine members are enough), you will be able to resync it later by --adding it as a new one, but only once your correct parameters are found, and your array back working. If the overwritten member was mandatory, hope you played mdadm --stop. Please find a way to verify if the data is really gone, and if it's gone, go to the "Unlucky situations" section.<br />
<br />
= Help needed, no combination is giving back access to the data =<br />
If you have 3 members and you are sure that data is still available on both of them, try every order: <br />
*Disk 1, Disk 2, Disk 3 <br />
*Disk 1, Disk 3, Disk 2<br />
*Disk 2, Disk 1, Disk 3<br />
*Disk 2, Disk 3, Disk 1<br />
*Disk 3, Disk 1, Disk 2 <br />
*Disk 3, Disk 2, Disk 1<br />
<br />
If it's not working, you should find a way to ensure data-offset (and may be others values) are correct, by creating a reconstitution of your original array creation environment to find what default values mdadm probably used. If your values are good, your data are back.<br />
If one of the members has its data missing, and you know which one (if not, its longer), check every combination:<br />
<br />
Member 1/3 is missing: X23, X32, 2X3, 23X, 3X2, 32X<br />
<br />
Member 2/3 is missing: 1X3, 13X, X13, X31, 31X, 3X1<br />
<br />
Member 3/3 is missing: 12X, 1X2, 21X, 2X1, X12, X21<br />
<br />
The numbers stand for the different members block device names, X stands for the "missing" word, in order to represent the parameters order into --create command.<br />
<br />
If you have 4 members, there is 24 possible combinations:<br />
*1234, 1243, 1324, 1342, 1423, 1432<br />
*2134, 2143, 2314, 2341, 2413, 2431<br />
*3124, 3142, 3214, 3241, 3412, 3421<br />
*4123, 4132, 4213, 4231, 4312, 4321<br />
<br />
If you are using RAID6 with 4 members, as 2 disks out of 4 are optional, let's save some time, try the 12 following attempts :<br />
*1st disk in 1st place, 2nd disk in the others places : 12xx 1x2x 1xx2<br />
*1st disk in 2nd place, 2nd disk in the others places : 21xx x12x x1x2<br />
*1st disk in 3rd place, 2nd disk in the others places : 2x1x x21x xx12<br />
*1st disk in 4th place, 2nd disk in the others places : 2xx1 x2x1 xx21<br />
<br />
Still about the RAID6 with 4 disks, if the disk number 2 and disk number 4 have lost their data or aren't available for any reason, you will have to use disk numbers 1 and 3 to do the same position attempts (instead of disk 1 and 2). If you are sure 2 disks are badly overwritten but you don't know which ones, 12, 13, 14, 23, 24, 34 are the possible couples: up to 6 x 12 try may be required to find back your data.<br />
<br />
= Unlucky situations =<br />
<br />
== The RAID array is split into 2 arrangements ==<br />
*Did the reshape/conversion actually started? If you are sure that the answer is no (stuck at 0% with no disk activity, backup file is there but contains nothing, nothing started and it's completely stuck), then things will be easy. Find the parameters set of the unchanged arrangement.<br />
<br />
*If the reshape/conversion actually started and moved some data, this kind of interruption can be restarted just by assembling the array again, and mount/use it as nothing happened. If it doesn't work, see if there is no parameters to force things to continue: don't follow this guide unless RAID information of your members have already been erased, or if you're sure that nothing else worked.<br />
<br />
*If a member failed during reshaping, and your array is redundant, don't play --create! You can add a new member: reshaping will finish in degraded mode, and the last member will be integrated and rebuilt just after.<br />
<br />
*If you have no other choice that playing --create with 2 different arrangements over 2 half of the array, unless someone have a better idea, you're falling into forensics : you'll need to find 2 sets of parameters, and probably need to use a file recovery scanner tool with both parameters set on the array. Be brave! The second set of parameters will probably be the same as the first one, with 1 more member (and raid-devices count set to one more). <br />
<br />
== Mandatory member lost ==<br />
So, your RAID is not redundant (or not enough), and a mandatory member (or the data area into it) is screwed.<br />
<br />
*Is your RAID member partially screwed/overwritten? If mdadm --create (or anything else) started to overwrite it, but you played mdadm --stop before it's entirely overwritten (or stopped anything that was overwriting the data into it), some of the data is still available. Find back your correct parameters (but you won't see any ext4 partition so it will be more difficult to know it you have the right ones) and use some file recovery scanner tool over your resulting array. You may still have the surprise to see some files that aren't completely erased.<br />
<br />
*Also, remember that having mdadm --create accidentally overwriting one of the disks should be avoided, but if it accidentally happens, in some case, calculated data overwritten to the disk may be the same that data which was already into it. In such case, there is a chance that your data is totally fine. You can always try to use the member as if it's OK: if it is, your file system will be made available again.<br />
<br />
*If one of the disks physically failed, but the data is still enclosed into it, see if RAW data into it can be rescued and cloned by some company, or if available sectors can be rescued by dd-rescue. See [[Replacing a failed drive]] for more detailed information about drive failure, including failure in non-redundant arrays.<br />
<br />
= Making sure that none of my arrays will never be lost anymore in the future =<br />
<br />
You can't, because it won't always be your fault. We believe that having succeeded or not into your recovery, you are probably interested into clever backup approaches for securing the recovered or future data.<br />
<br />
Indeed, a defective power supply can destroy some of your disks. A violent shock on the case containing all the disks can cause several of them to fail. A fire incident too. The server can be stolen (with disks inside of it). A ransomware having access to some part of the filesystem, even through the network file sharing, can also destroy all of the writable files. A wrong dd, shred, or you swapped the wrong "failed" disk, opened it to play with what's inside, before realizing it wasn't the failed one. Data memory module malfunction during a rebuild (it can screw the data of one of your members without letting you know what happened). And may be others things.<br />
<br />
A free open source versioned continuous copy system like Syncthing (in case files are being encrypted, the previous version is kept and timestamped) isn't officially meant to protect you in this kind of cases, but still does it pretty well. Conventional backups systems and solutions could also be used. In case of malicious attack, still remember to use a backup system that is different enough of the system you are trying to backup (not the same password, not the same config, not the same network, not the same place: it would be like a RAID1 on 2 partitions of the same disk).</div>Julien Robinhttps://raid.wiki.kernel.org/index.php/Irreversible_mdadm_failure_recoveryIrreversible mdadm failure recovery2019-05-05T03:01:54Z<p>Julien Robin: Adding RAID6 mistakes recoveries, more info about overwritten disks during a rebuild process if --assume-clean was forgotten, and adding --readonly flag (which works with --create)</p>
<hr />
<div>== Introduction ==<br />
<br />
If you feel comfortable using overlays, this is always a good idea (so that no accidental write can happen to the real members).<br />
<br />
Every information and situation you can find here has been seriously tested on mdadm 3.4-4 and 4.1-1 before being published.<br />
<br />
Going here is normally not needed, unless mdadm conventional ways won't have any chance to work anymore (we believe that you explored others solution before going here). This section is about searching right parameters to play --create over an existing array, in order to retrieve a correctly configured access to any still available data (be it the entire filesystem if still available).<br />
<br />
Be warned: playing mdadm --create erases and overwrites at least the RAID array information area of every involved RAID members. A misuse of mdadm --create can be the reason why you are here. Without the --assume-clean parameter, one or several member data area can be entirely reconstructed (sometimes it's fine, sometimes it's not, sometimes it's recoverable, sometimes it's not - so it's better to avoid it). Without the --readonly parameter, mounting and browsing the filesystem can write things, and it can be a problem if some of the members finally aren't at the right position into the array.<br />
<br />
If the correct RAID array information is still available (then are you sure nothing else can be done with conventional ways?), to save some time, please keep the output of cat /proc/mdstat (for getting the members orders), and the output of mdadm --examine against your differents RAID members. It should be done BEFORE calling any mdadm --create command. Creation and Update time are provided into the output, so you can easily know if it contains your original array information, or if it's already lost/overwritten.<br />
<br />
If you can't manage to be sure about the position of each disk into the array, and you have a large number of disks, you are encouraged to find a way to remember the position of as much disks as possible into the array, and prepare a listing of all possible combinations (starting with the most likely ones) because 4 disk could already give you a lot of work, and above 4 disks, an huge amount of attempts may be needed. Needing you each time to verify if your data is correctly available by checking the integrity of files that are big enough to be spread on every members.<br />
<br />
Members can be entire drives or partitions inside of it. If you erased a GPT table describing a partition by mistake, or removed the partition by mistake, please create it again, unformatted, at the same place (and same size) it was when used as RAID member. The content of the partition may be completely or mostly untouched (depending whether it has just been marked deleted, or partially overwritten).<br />
<br />
== Situation analysis ==<br />
<br />
A RAID array member has 2 area: RAID array information (which may be lost if you are here), and the Data area. As long as the Data area is not erased, you can consider the member as useful for the following steps. If Data area is erased, it should be considered as missing for the following steps (and if the missing member is mandatory, don't quit this page before taking a look at the "Unlucky situations" section: may be some files can be recovered).<br />
<br />
You are about to recover entire access to your filesystem as long as:<br />
*Your array is redundant and you didn't lose/overwrite any member data area<br />
*Your array isn't redundant but hopefully, you didn't lose/overwrite any member data area<br />
*Your array is redundant, you have lost/overwritten one member data area, but hopefully there is still enough others members left to get your data back<br />
*The data inside your members are arranged in one consistent and predictable way all over the array<br />
<br />
If you aren't lucky, don't quit this page before taking a look at the "Unlucky situations" section: there's still several things to verify before aborting.<br />
<br />
== Parameters recovery ==<br />
<br />
Playing --create over an existing array needs you to know:<br />
*The number of members in your array.<br />
*The RAID level of your array<br />
*The chunk size of your array if using RAID0, RAID4, RAID5, RAID6, or RAID10 (generally 512K on modern mdadm versions, at least for 3.4-4 and 4.1-1)<br />
*The metadata version of your array (depends of the mdadm version initially used to create the array - 1.2 on modern versions, at least for 3.4-4 and 4.1-1)<br />
*The layout of your array if using RAID5, RAID6, or RAID10 (generally left-symmetric by default for RAID5 and 6, and can be near=2 for RAID10 if your 4 members are A, A, B, B)<br />
*The data offset of your array (depends on the size of your array members used at initial array creation - even if members have been replaced by bigger ones after, and grow --size=max used! It also depends on the version of mdadm you were using at the initial creation)<br />
*The position of the different members into the array (if it changed on the motherboard since initial creation, you may need to try several ones. If not, you probably used alphabetical order)<br />
*Is any of your member missing, or having his Data area overwritten by something like a wrong command? Which one? Because it should be declared as "missing" when running --create. If you don't know which one, you will have to do more attempts.<br />
*--assume-clean will give you the ability to make several attempts without having the last member rebuilt<br />
*--readonly will give you the ability to mount the filesystem without causing any update to any byte of the data into the array<br />
<br />
If nothing changed on your system (members size, mdadm version...) and you used default parameters, then letting mdadm choosing default ones should be fine and apply the right parameters.<br />
<br />
== Data-offset ==<br />
If you changed mdadm version since initial array creation, data-offset is likely to have changed. I have seen 8192s when created with tiny disks (5000 MiB) into mdadm 3.4-4 (while 4.1-1 defaults to 10240s for the same size, and 18432s for 10000 MiB), and 262144s for 8TB drives with mdadm 3.4-4. Didn't try creating with 8TB members on mdadm 4.1-1. May be some formula can be found into the different code versions?<br />
<br />
= Example of command =<br />
== The easy one ==<br />
Using Debian 9, mdadm 3.4-4, a RAID array was created with with 3 x 8TB members, using:<br />
<br />
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1<br />
<br />
If nothing changed and I'm still using the same mdadm version, typing the same command again (but appending --assume-clean --readonly for safety) will be fine: <br />
<br />
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1 --assume-clean --readonly<br />
<br />
Gnome-disks shows the file system again and I can mount it (trying read-only before, mount -o ro,noload /dev/mdXX /media/raid-volume) : everything is fine and every member is in the right place into the array. Then, do not forget to do a filesystem check and repair after you ensured that you are using the correct parameters, in case some brutal interruption happened previously on the array.<br />
<br />
== A more precise one ==<br />
For creating the array again after having upgraded from mdadm 3.4-4 to mdadm 4.1-1, data offset should be specified because the automatically selected value for a given size changed. <br />
<br />
Also, disk position changed on my motherboard. After a precise analysis of the previous version defaults parameters for this array, in this case, the working command for finding back my array finally was:<br />
<br />
mdadm --create /dev/md0 --level=5 --chunk=512K --metadata=1.2 --layout left-symmetric --data-offset=262144s --raid-devices=3 /dev/sdd1 /dev/sde1 /dev/sdb1 --assume-clean --readonly<br />
<br />
== One facultative member is missing ==<br />
If the second disk is missing, type missing instead of its block device name. On a redundant RAID array, data will still be available. Of course, once your array is back working, you will have to --add the missing member soon or late for safety, but after you ensured that the parameters are OK and your data available. The member you will add will be rebuilt with consistent data. <br />
<br />
== The array was created with 2 disks, and grown to 3 ==<br />
Having the array initially created with 2 disks before having the 3rd one added and the array grown, give the exact same final arrangement compared to a 3 disks array creation.<br />
<br />
= In case of accidental rebuild =<br />
If you forgot the --assume-clean parameter, a rebuild process may have rewritten some of your members.<br />
But if you were using --readonly, then you are fine (in both case, I didn't see any rebuild process starting).<br />
<br />
*In case of RAID 0, there is no possible rebuild process. That's fine.<br />
*In case of RAID 1, only the 1st member is read, the others ones are overwritten with a copy of it.<br />
*In case of RAID 5, the last RAID member given in parameter will be overwritten by reading and processing data of the others given members. If your original array was already using RAID5, and the member count is correct, it's likely to be fine.<br />
*In case of RAID 6, every member is rewritten by reading and computing data from others ones. If you are wrong about the position of 2 disks out of 4, you can recover, see below. If its more, it likely to be lost (but I didn't try with array bigger than 4 disks)<br />
*In case of RAID 10, with 4 members and default parameters, 1st drive is read and copied to the 2nd one, while 3rd drive is read and copied to the 4th one.<br />
<br />
In case of RAID 5:<br />
*Even with wrong order or wrong chunk size, and any wrong RAID5 or RAID4-like layout, if the others members data is fine, and the amount of members is correct, the parity calculation will be correct. Although the computer may not interpret the data the right manner (which bit is parity or data) until you find the right parameters. <br />
*But if you forget the --assume-clean parameter in others cases (like incorrect amount of disks, or RAID level, or reading a lost member to calculate and rewrite over a good one), consider the overwritten disk as missing.<br />
<br />
In case of RAID6:<br />
*Driving some tests about RAID6, having the members accidentally rebuilt with 2 out of 4 disks in the wrong position (1243 - for example - instead of 1234) an erroneous reconstruction is applied on all of the 4 members. Trying to access the array filesystem, the files were incorrect. But --creating the RAID again, with the 4 disks back at the correct position (and --assume-clean and --readonly) restored the access to the correct data. Replaying the same command after ensuring the position are right, and without --assume-clean and --readonly also restored the original data over all of the 4 disks (sha1sum controlled, RAID information excluded).<br />
*Driving another test, it seems that if you have accidentally rebuilt your array with 2 out of 4 disks in the wrong position (still 1243 instead of 1234 for example), mounting the array with only disk 1 and 2 won't be fine. Nothing more with 1, 2, and 3: the 4 disks should be used. This is probably due to the fact that the erroneous reconstruction is done over all of the members, so all of them are needed to revert this error.<br />
<br />
In others cases:<br />
In case correct data seems to be erased from only one of your members, if it wasn't mandatory (if remaining fine members are enough), you will be able to resync it later by --adding it as a new one, but only once your correct parameters are found, and your array back working. If the overwritten member was mandatory, hope you played mdadm --stop. Please find a way to verify if the data is really gone, and if it's gone, go to the "Unlucky situations" section.<br />
<br />
= Help needed, no combination is giving back access to the data =<br />
If you have 3 members and you are sure that data is still available on both of them, try every order: <br />
*Disk 1, Disk 2, Disk 3 <br />
*Disk 1, Disk 3, Disk 2<br />
*Disk 2, Disk 1, Disk 3<br />
*Disk 2, Disk 3, Disk 1<br />
*Disk 3, Disk 1, Disk 2 <br />
*Disk 3, Disk 2, Disk 1<br />
<br />
If it's not working, you should find a way to ensure data-offset (and may be others values) are correct, by creating a reconstitution of your original array creation environment to find what default values mdadm probably used. If your values are good, your data are back.<br />
If one of the members has its data missing, and you know which one (if not, its longer), check every combination:<br />
<br />
Member 1/3 is missing: X23, X32, 2X3, 23X, 3X2, 32X<br />
<br />
Member 2/3 is missing: 1X3, 13X, X13, X31, 31X, 3X1<br />
<br />
Member 3/3 is missing: 12X, 1X2, 21X, 2X1, X12, X21<br />
<br />
The numbers stand for the different members block device names, X stands for the "missing" word, in order to represent the parameters order into --create command.<br />
<br />
If you have 4 members, there is 24 possible combinations:<br />
*1234, 1243, 1324, 1342, 1423, 1432<br />
*2134, 2143, 2314, 2341, 2413, 2431<br />
*3124, 3142, 3214, 3241, 3412, 3421<br />
*4123, 4132, 4213, 4231, 4312, 4321<br />
<br />
If you are using RAID6 with 4 members, as 2 disks out of 4 are optional, let's save some time, try the 12 following attempts :<br />
*1st disk in 1st place, 2nd disk in the others places : 12xx 1x2x 1xx2<br />
*1st disk in 2nd place, 2nd disk in the others places : 21xx x12x x1x2<br />
*1st disk in 3rd place, 2nd disk in the others places : 2x1x x21x xx12<br />
*1st disk in 4th place, 2nd disk in the others places : 2xx1 x2x1 xx21<br />
<br />
Still about the RAID6 with 4 disks, if the disk number 2 and disk number 4 have lost their data or aren't available for any reason, you will have to use disk numbers 1 and 3 to do the same position attempts (instead of disk 1 and 2). If you are sure 2 disks are badly overwritten but you don't know which ones, 12, 13, 14, 23, 24, 34 are the possible couples: up to 6 x 12 try may be required to find back your data.<br />
<br />
= Unlucky situations =<br />
<br />
== The RAID array is split into 2 arrangements ==<br />
*Did the reshape/conversion actually started? If you are sure that the answer is no (stuck at 0% with no disk activity, backup file is there but contains nothing, nothing started and it's completely stuck), then things will be easy. Find the parameters set of the unchanged arrangement.<br />
<br />
*If the reshape/conversion actually started and moved some data, this kind of interruption can be restarted just by assembling the array again, and mount/use it as nothing happened. If it doesn't work, see if there is no parameters to force things to continue: don't follow this guide unless RAID information of your members have already been erased, or if you're sure that nothing else worked.<br />
<br />
*If a member failed during reshaping, and your array is redundant, don't play --create! You can add a new member: reshaping will finish in degraded mode, and the last member will be integrated and rebuilt just after.<br />
<br />
*If you have no other choice that playing --create with 2 different arrangements over 2 half of the array, unless someone have a better idea, you're falling into forensics : you'll need to find 2 sets of parameters, and probably need to use a file recovery scanner tool with both parameters set on the array. Be brave! The second set of parameters will probably be the same as the first one, with 1 more member (and raid-devices count set to one more). <br />
<br />
== Mandatory member lost ==<br />
So, your RAID is not redundant (or not enough), and a mandatory member (or the data area into it) is screwed.<br />
<br />
*Is your RAID member partially screwed/overwritten? If mdadm --create (or anything else) started to overwrite it, but you played mdadm --stop before it's entirely overwritten (or stopped anything that was overwriting the data into it), some of the data is still available. Find back your correct parameters (but you won't see any ext4 partition so it will be more difficult to know it you have the right ones) and use some file recovery scanner tool over your resulting array. You may still have the surprise to see some files that aren't completely erased.<br />
<br />
*Also, remember that having mdadm --create accidentally overwriting one of the disks should be avoided, but if it accidentally happens, in some case, calculated data overwritten to the disk may be the same that data which was already into it. In such case, there is a chance that your data is totally fine. You can always try to use the member as if it's OK: if it is, your file system will be made available again.<br />
<br />
*If one of the disks physically failed, but the data is still enclosed into it, see if RAW data into it can be rescued and cloned by some company, or if available sectors can be rescued by dd-rescue. See [[Replacing a failed drive]] for more detailed information about drive failure, including failure in non-redundant arrays.<br />
<br />
= Making sure that none of my arrays will never be lost anymore in the future =<br />
<br />
You can't, because it won't always be your fault. We believe that having succeeded or not into your recovery, you are probably interested into clever backup approaches for securing the recovered or future data.<br />
<br />
Indeed, a defective power supply can destroy some of your disks. A violent shock on the case containing all the disks can cause several of them to fail. A fire incident too. The server can be stolen (with disks inside of it). A ransomware having access to some part of the filesystem, even through the network file sharing, can also destroy all of the writable files. A wrong dd, shred, or you swapped the wrong "failed" disk, opened it to play with what's inside, before realizing it wasn't the failed one. Data memory module malfunction during a rebuild (it can screw the data of one of your members without letting you know what happened). And may be others things.<br />
<br />
A free open source versioned continuous copy system like Syncthing (in case files are being encrypted, the previous version is kept and timestamped) isn't officially meant to protect you in this kind of cases, but still does it pretty well. Conventional backups systems and solutions could also be used. In case of malicious attack, still remember to use a backup system that is different enough of the system you are trying to backup (not the same password, not the same config, not the same network, not the same place: it would be like a RAID1 on 2 partitions of the same disk).</div>Julien Robinhttps://raid.wiki.kernel.org/index.php/Irreversible_mdadm_failure_recoveryIrreversible mdadm failure recovery2019-05-04T19:01:49Z<p>Julien Robin: /* Making sure that none of my arrays will never be lost anymore in the future */</p>
<hr />
<div>== Introduction ==<br />
<br />
If you feel comfortable using overlays, this is always a good idea (so that no accidental write can happen to the real members).<br />
<br />
Every information and situation you can find here has been seriously tested on mdadm 3.4-4 and 4.1-1 before being published.<br />
<br />
Going here is normally not needed, unless mdadm conventional ways won't have any chance to work anymore (we believe that you explored others solution before going here). This section is about searching right parameters to play --create over an existing array, in order to retrieve a correctly configured access to any still available data (be it the entire filesystem if still available).<br />
<br />
Be warned: Playing mdadm --create erases and overwrites at least the RAID array information area of every involved RAID members. A misuse of mdadm --create can be the reason why you are here. Without the --assume-clean parameter, one member data area can be entirely reconstructed (sometimes it's fine, sometimes it's not - so it's better to avoid it). And if you write anything to the array before checking everything, it can cause problems if create parameters were incorrect: this is why it needs to be so careful.<br />
<br />
If the correct RAID array information is still available (then are you sure nothing else can be done with conventional ways?), to save some time, please keep the output of cat /proc/mdstat (for getting the members orders), and the output of mdadm --examine against your differents RAID members. It should be done BEFORE calling any mdadm --create command. Creation and Update time are provided into the output, so you can easily know if it contains your original array information, or if it's already lost/overwritten.<br />
<br />
Members can be entire drives or partitions inside of it. If you erased a GPT table describing a partition by mistake, or removed the partition by mistake, please create it again, unformatted, at the same place (and same size) it was when used as RAID member. The content of the partition may be completely or mostly untouched.<br />
<br />
== Situation analysis ==<br />
<br />
A RAID array member has 2 area: RAID array information (which may be lost if you are here), and the Data area. As long as the Data area is not erased, you can consider the member as useful for the following steps. If Data area is erased, it should be considered as missing for the following steps (and if the missing member is mandatory, don't quit this page before taking a look at the "Unlucky situations" section: may be some files can be recovered).<br />
<br />
You are about to recover entire access to your filesystem as long as:<br />
*Your array is redundant and you didn't lose/overwrite any member data area<br />
*Your array isn't redundant but hopefully, you didn't lose/overwrite any member data area<br />
*Your array is redundant, you have lost/overwritten one member data area, but hopefully there is still enough others members left to get your data back<br />
*The data inside your members are arranged in one consistent and predictable way all over the array<br />
<br />
If you aren't lucky, don't quit this page before taking a look at the "Unlucky situations" section: there's still several things to verify before aborting.<br />
<br />
== Parameters recovery ==<br />
<br />
Playing --create over an existing array needs you to know:<br />
*The number of members in your array.<br />
*The RAID level of your array<br />
*The chunk size of your array if using RAID0, RAID4, RAID5, RAID6, or RAID10 (generally 512K on modern mdadm versions, at least for 3.4-4 and 4.1-1)<br />
*The metadata version of your array (depends of the mdadm version initially used to create the array - 1.2 on modern versions, at least for 3.4-4 and 4.1-1)<br />
*The layout of your array if using RAID5, RAID6, or RAID10 (generally left-symmetric by default for RAID5 and 6, and can be near=2 for RAID10 if your 4 members are A, A, B, B)<br />
*The data offset of your array (depends on the size of your array members used at initial array creation - even if members have been replaced by bigger ones after, and grow --size=max used! It also depends on the version of mdadm you were using at the initial creation)<br />
*The position of the different members into the array (if it changed on the motherboard since initial creation, you may need to try several ones. If not, you probably used alphabetical order)<br />
*Is any of your member missing, or having his Data area overwritten by something like a wrong command? Which one? Because it should be declared as "missing" when running --create. If you don't know which one, you will have to do more attempts.<br />
*--assume-clean will give you the ability to make several attempts without having the last member rebuilt<br />
<br />
If nothing changed on your system (members size, mdadm version...) and you used default parameters, then letting mdadm choosing default ones should be fine and apply the right parameters.<br />
<br />
== Data-offset ==<br />
If you changed mdadm version since initial array creation, data-offset is likely to have changed. I have seen 8192s when created with tiny disks (5000 MiB) into mdadm 3.4-4 (while 4.1-1 defaults to 10240s for the same size, and 18432s for 10000 MiB), and 262144s for 8TB drives with mdadm 3.4-4. Didn't try creating with 8TB members on mdadm 4.1-1. May be some formula can be found into the different code versions?<br />
<br />
= Example of command =<br />
== The easy one ==<br />
Using Debian 9, mdadm 3.4-4, a RAID array was created with with 3 x 8TB members, using:<br />
<br />
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1<br />
<br />
If nothing changed and I'm still using the same mdadm version, typing the same command again (but appending --assume-clean for safety) will be fine: <br />
<br />
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1 --assume-clean<br />
<br />
Gnome-disks shows the file system again and I can mount it (trying read-only before, mount -o ro,noload /dev/mdXX /media/raid-volume) : everything is fine and every member is in the right place into the array. Then, do not forget to do a filesystem check and repair after you ensured that you are using the correct parameters, in case some brutal interruption happened previously on the array.<br />
<br />
== A more precise one ==<br />
For creating the array again after having upgraded from mdadm 3.4-4 to mdadm 4.1-1, data offset should be specified because the automatically selected value for a given size changed. <br />
<br />
Also, disk position changed on my motherboard. After a precise analysis of the previous version defaults parameters for this array, in this case, the working command for finding back my array finally was:<br />
<br />
mdadm --create /dev/md0 --level=5 --chunk=512K --metadata=1.2 --layout left-symmetric --data-offset=262144s --raid-devices=3 /dev/sdd1 /dev/sde1 /dev/sdb1 --assume-clean <br />
<br />
== One facultative member is missing ==<br />
If the second disk is missing, type missing instead of its block device name. On a redundant RAID array, data will still be available. Of course, once your array is back working, you will have to --add the missing member soon or late for safety, but after you ensured that the parameters are OK and your data available. The member you will add will be rebuilt with consistent data. <br />
<br />
== The array was created with 2 disks, and grown to 3 ==<br />
Having the array initially created with 2 disks before having the 3rd one added and the array grown, give the exact same final arrangement compared to a 3 disks array creation.<br />
<br />
= About accidental rebuild =<br />
If you forget the --assume-clean parameter, in case of RAID 5 for example, the last RAID member given in parameter will be rewritten by reading and processing data of the others given members. <br />
<br />
In case of RAID 5:<br />
*Even with wrong order or wrong chunk size, and any wrong RAID5 or RAID4-like layout, if the others members data is fine, and the amount of members is correct, the parity calculation will be correct. Although the computer may not interpret the data the right manner (which bit is parity or data) until you find the right parameters. <br />
*But if you forget the --assume-clean parameter in others cases (like incorrect amount of disks, or RAID level, or reading a lost member to calculate and rewrite over a good one), consider the overwritten disk as missing.<br />
<br />
In case correct data seems to be erased from one of your members, if it wasn't mandatory (if remaining fine members are enough), you will be able to resync it later by --adding it as a new one, but only once your correct parameters are found, and your array back working. If the overwritten member was mandatory, hope you played mdadm --stop. Please find a way to verify if the data is really gone, and if it's gone, go to the "Unlucky situations" section.<br />
<br />
= Help needed, no combination is giving back access to the data =<br />
If you have 3 members and you are sure that data is still available on both of it, try every order: 123 132 213 231 312 321. <br />
<br />
If it's not working, you should find a way to ensure data-offset (and may be others values) are correct, by creating a reconstitution of your original array creation environment to find what default values mdadm probably used. If your values are good, your data are back.<br />
If one of the members has its data missing, and you know which one (if not, its longer), check every combination:<br />
<br />
Member 1/3 is missing: X23, X32, 2X3, 23X, 3X2, 32X<br />
<br />
Member 2/3 is missing: 1X3, 13X, X13, X31, 31X, 3X1<br />
<br />
Member 3/3 is missing: 12X, 1X2, 21X, 2X1, X12, X21<br />
<br />
The numbers stand for the different members block device names, X stands for the "missing" word, in order to represent the parameters order into --create command.<br />
<br />
= Unlucky situations =<br />
<br />
== The RAID array is split into 2 arrangements ==<br />
*Did the reshape/conversion actually started? If you are sure that the answer is no (stuck at 0% with no disk activity, backup file is there but contains nothing, nothing started and it's completely stuck), then things will be easy. Find the parameters set of the unchanged arrangement.<br />
<br />
*If the reshape/conversion actually started and moved some data, this kind of interruption can be restarted just by assembling the array again, and mount/use it as nothing happened. If it doesn't work, see if there is no parameters to force things to continue: don't follow this guide unless RAID information of your members have already been erased, or if you're sure that nothing else worked.<br />
<br />
*If a member failed during reshaping, and your array is redundant, don't play --create! You can add a new member: reshaping will finish in degraded mode, and the last member will be integrated and rebuilt just after.<br />
<br />
*If you have no other choice that playing --create with 2 different arrangements over 2 half of the array, unless someone have a better idea, you're falling into forensics : you'll need to find 2 sets of parameters, and probably need to use a file recovery scanner tool with both parameters set on the array. Be brave! The second set of parameters will probably be the same as the first one, with 1 more member (and raid-devices count set to one more). <br />
<br />
== Mandatory member lost ==<br />
So, your RAID is not redundant (or not enough), and a mandatory member (or the data area into it) is screwed.<br />
<br />
*Is your RAID member partially screwed/overwritten? If mdadm --create (or anything else) started to overwrite it, but you played mdadm --stop before it's entirely overwritten (or stopped anything that was overwriting the data into it), some of the data is still available. Find back your correct parameters (but you won't see any ext4 partition so it will be more difficult to know it you have the right ones) and use some file recovery scanner tool over your resulting array. You may still have the surprise to see some files that aren't completely erased.<br />
<br />
*Also, remember that having mdadm --create accidentally overwriting one of the disks should be avoided, but if it accidentally happens, in some case, calculated data overwritten to the disk may be the same that data which was already into it. In such case, there is a chance that your data is totally fine. You can always try to use the member as if it's OK: if it is, your file system will be made available again.<br />
<br />
*If one of the disks physically failed, but the data is still enclosed into it, see if RAW data into it can be rescued and cloned by some company, or if available sectors can be rescued by dd-rescue. See [[Replacing a failed drive]] for more detailed information about drive failure, including failure in non-redundant arrays.<br />
<br />
= Making sure that none of my arrays will never be lost anymore in the future =<br />
<br />
You can't, because it won't always be your fault. We believe that having succeeded or not into your recovery, you are probably interested into clever backup approaches for securing the recovered or future data.<br />
<br />
Indeed, a defective power supply can destroy some of your disks. A violent shock on the case containing all the disks can cause several of them to fail. A fire incident too. The server can be stolen (with disks inside of it). A ransomware having access to some part of the filesystem, even through the network file sharing, can also destroy all of the writable files. A wrong dd, shred, or you swapped the wrong "failed" disk, opened it to play with what's inside, before realizing it wasn't the failed one. Data memory module malfunction during a rebuild (it can screw the data of one of your members without letting you know what happened). And may be others things.<br />
<br />
A free open source versioned continuous copy system like Syncthing (in case files are being encrypted, the previous version is kept and timestamped) isn't officially meant to protect you in this kind of cases, but still does it pretty well. Conventional backups systems and solutions could also be used. In case of malicious attack, still remember to use a backup system that is different enough of the system you are trying to backup (not the same password, not the same config, not the same network, not the same place: it would be like a RAID1 on 2 partitions of the same disk).</div>Julien Robinhttps://raid.wiki.kernel.org/index.php/Irreversible_mdadm_failure_recoveryIrreversible mdadm failure recovery2019-05-04T18:30:55Z<p>Julien Robin: </p>
<hr />
<div>== Introduction ==<br />
<br />
If you feel comfortable using overlays, this is always a good idea (so that no accidental write can happen to the real members).<br />
<br />
Every information and situation you can find here has been seriously tested on mdadm 3.4-4 and 4.1-1 before being published.<br />
<br />
Going here is normally not needed, unless mdadm conventional ways won't have any chance to work anymore (we believe that you explored others solution before going here). This section is about searching right parameters to play --create over an existing array, in order to retrieve a correctly configured access to any still available data (be it the entire filesystem if still available).<br />
<br />
Be warned: Playing mdadm --create erases and overwrites at least the RAID array information area of every involved RAID members. A misuse of mdadm --create can be the reason why you are here. Without the --assume-clean parameter, one member data area can be entirely reconstructed (sometimes it's fine, sometimes it's not - so it's better to avoid it). And if you write anything to the array before checking everything, it can cause problems if create parameters were incorrect: this is why it needs to be so careful.<br />
<br />
If the correct RAID array information is still available (then are you sure nothing else can be done with conventional ways?), to save some time, please keep the output of cat /proc/mdstat (for getting the members orders), and the output of mdadm --examine against your differents RAID members. It should be done BEFORE calling any mdadm --create command. Creation and Update time are provided into the output, so you can easily know if it contains your original array information, or if it's already lost/overwritten.<br />
<br />
Members can be entire drives or partitions inside of it. If you erased a GPT table describing a partition by mistake, or removed the partition by mistake, please create it again, unformatted, at the same place (and same size) it was when used as RAID member. The content of the partition may be completely or mostly untouched.<br />
<br />
== Situation analysis ==<br />
<br />
A RAID array member has 2 area: RAID array information (which may be lost if you are here), and the Data area. As long as the Data area is not erased, you can consider the member as useful for the following steps. If Data area is erased, it should be considered as missing for the following steps (and if the missing member is mandatory, don't quit this page before taking a look at the "Unlucky situations" section: may be some files can be recovered).<br />
<br />
You are about to recover entire access to your filesystem as long as:<br />
*Your array is redundant and you didn't lose/overwrite any member data area<br />
*Your array isn't redundant but hopefully, you didn't lose/overwrite any member data area<br />
*Your array is redundant, you have lost/overwritten one member data area, but hopefully there is still enough others members left to get your data back<br />
*The data inside your members are arranged in one consistent and predictable way all over the array<br />
<br />
If you aren't lucky, don't quit this page before taking a look at the "Unlucky situations" section: there's still several things to verify before aborting.<br />
<br />
== Parameters recovery ==<br />
<br />
Playing --create over an existing array needs you to know:<br />
*The number of members in your array.<br />
*The RAID level of your array<br />
*The chunk size of your array if using RAID0, RAID4, RAID5, RAID6, or RAID10 (generally 512K on modern mdadm versions, at least for 3.4-4 and 4.1-1)<br />
*The metadata version of your array (depends of the mdadm version initially used to create the array - 1.2 on modern versions, at least for 3.4-4 and 4.1-1)<br />
*The layout of your array if using RAID5, RAID6, or RAID10 (generally left-symmetric by default for RAID5 and 6, and can be near=2 for RAID10 if your 4 members are A, A, B, B)<br />
*The data offset of your array (depends on the size of your array members used at initial array creation - even if members have been replaced by bigger ones after, and grow --size=max used! It also depends on the version of mdadm you were using at the initial creation)<br />
*The position of the different members into the array (if it changed on the motherboard since initial creation, you may need to try several ones. If not, you probably used alphabetical order)<br />
*Is any of your member missing, or having his Data area overwritten by something like a wrong command? Which one? Because it should be declared as "missing" when running --create. If you don't know which one, you will have to do more attempts.<br />
*--assume-clean will give you the ability to make several attempts without having the last member rebuilt<br />
<br />
If nothing changed on your system (members size, mdadm version...) and you used default parameters, then letting mdadm choosing default ones should be fine and apply the right parameters.<br />
<br />
== Data-offset ==<br />
If you changed mdadm version since initial array creation, data-offset is likely to have changed. I have seen 8192s when created with tiny disks (5000 MiB) into mdadm 3.4-4 (while 4.1-1 defaults to 10240s for the same size, and 18432s for 10000 MiB), and 262144s for 8TB drives with mdadm 3.4-4. Didn't try creating with 8TB members on mdadm 4.1-1. May be some formula can be found into the different code versions?<br />
<br />
= Example of command =<br />
== The easy one ==<br />
Using Debian 9, mdadm 3.4-4, a RAID array was created with with 3 x 8TB members, using:<br />
<br />
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1<br />
<br />
If nothing changed and I'm still using the same mdadm version, typing the same command again (but appending --assume-clean for safety) will be fine: <br />
<br />
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1 --assume-clean<br />
<br />
Gnome-disks shows the file system again and I can mount it (trying read-only before, mount -o ro,noload /dev/mdXX /media/raid-volume) : everything is fine and every member is in the right place into the array. Then, do not forget to do a filesystem check and repair after you ensured that you are using the correct parameters, in case some brutal interruption happened previously on the array.<br />
<br />
== A more precise one ==<br />
For creating the array again after having upgraded from mdadm 3.4-4 to mdadm 4.1-1, data offset should be specified because the automatically selected value for a given size changed. <br />
<br />
Also, disk position changed on my motherboard. After a precise analysis of the previous version defaults parameters for this array, in this case, the working command for finding back my array finally was:<br />
<br />
mdadm --create /dev/md0 --level=5 --chunk=512K --metadata=1.2 --layout left-symmetric --data-offset=262144s --raid-devices=3 /dev/sdd1 /dev/sde1 /dev/sdb1 --assume-clean <br />
<br />
== One facultative member is missing ==<br />
If the second disk is missing, type missing instead of its block device name. On a redundant RAID array, data will still be available. Of course, once your array is back working, you will have to --add the missing member soon or late for safety, but after you ensured that the parameters are OK and your data available. The member you will add will be rebuilt with consistent data. <br />
<br />
== The array was created with 2 disks, and grown to 3 ==<br />
Having the array initially created with 2 disks before having the 3rd one added and the array grown, give the exact same final arrangement compared to a 3 disks array creation.<br />
<br />
= About accidental rebuild =<br />
If you forget the --assume-clean parameter, in case of RAID 5 for example, the last RAID member given in parameter will be rewritten by reading and processing data of the others given members. <br />
<br />
In case of RAID 5:<br />
*Even with wrong order or wrong chunk size, and any wrong RAID5 or RAID4-like layout, if the others members data is fine, and the amount of members is correct, the parity calculation will be correct. Although the computer may not interpret the data the right manner (which bit is parity or data) until you find the right parameters. <br />
*But if you forget the --assume-clean parameter in others cases (like incorrect amount of disks, or RAID level, or reading a lost member to calculate and rewrite over a good one), consider the overwritten disk as missing.<br />
<br />
In case correct data seems to be erased from one of your members, if it wasn't mandatory (if remaining fine members are enough), you will be able to resync it later by --adding it as a new one, but only once your correct parameters are found, and your array back working. If the overwritten member was mandatory, hope you played mdadm --stop. Please find a way to verify if the data is really gone, and if it's gone, go to the "Unlucky situations" section.<br />
<br />
= Help needed, no combination is giving back access to the data =<br />
If you have 3 members and you are sure that data is still available on both of it, try every order: 123 132 213 231 312 321. <br />
<br />
If it's not working, you should find a way to ensure data-offset (and may be others values) are correct, by creating a reconstitution of your original array creation environment to find what default values mdadm probably used. If your values are good, your data are back.<br />
If one of the members has its data missing, and you know which one (if not, its longer), check every combination:<br />
<br />
Member 1/3 is missing: X23, X32, 2X3, 23X, 3X2, 32X<br />
<br />
Member 2/3 is missing: 1X3, 13X, X13, X31, 31X, 3X1<br />
<br />
Member 3/3 is missing: 12X, 1X2, 21X, 2X1, X12, X21<br />
<br />
The numbers stand for the different members block device names, X stands for the "missing" word, in order to represent the parameters order into --create command.<br />
<br />
= Unlucky situations =<br />
<br />
== The RAID array is split into 2 arrangements ==<br />
*Did the reshape/conversion actually started? If you are sure that the answer is no (stuck at 0% with no disk activity, backup file is there but contains nothing, nothing started and it's completely stuck), then things will be easy. Find the parameters set of the unchanged arrangement.<br />
<br />
*If the reshape/conversion actually started and moved some data, this kind of interruption can be restarted just by assembling the array again, and mount/use it as nothing happened. If it doesn't work, see if there is no parameters to force things to continue: don't follow this guide unless RAID information of your members have already been erased, or if you're sure that nothing else worked.<br />
<br />
*If a member failed during reshaping, and your array is redundant, don't play --create! You can add a new member: reshaping will finish in degraded mode, and the last member will be integrated and rebuilt just after.<br />
<br />
*If you have no other choice that playing --create with 2 different arrangements over 2 half of the array, unless someone have a better idea, you're falling into forensics : you'll need to find 2 sets of parameters, and probably need to use a file recovery scanner tool with both parameters set on the array. Be brave! The second set of parameters will probably be the same as the first one, with 1 more member (and raid-devices count set to one more). <br />
<br />
== Mandatory member lost ==<br />
So, your RAID is not redundant (or not enough), and a mandatory member (or the data area into it) is screwed.<br />
<br />
*Is your RAID member partially screwed/overwritten? If mdadm --create (or anything else) started to overwrite it, but you played mdadm --stop before it's entirely overwritten (or stopped anything that was overwriting the data into it), some of the data is still available. Find back your correct parameters (but you won't see any ext4 partition so it will be more difficult to know it you have the right ones) and use some file recovery scanner tool over your resulting array. You may still have the surprise to see some files that aren't completely erased.<br />
<br />
*Also, remember that having mdadm --create accidentally overwriting one of the disks should be avoided, but if it accidentally happens, in some case, calculated data overwritten to the disk may be the same that data which was already into it. In such case, there is a chance that your data is totally fine. You can always try to use the member as if it's OK: if it is, your file system will be made available again.<br />
<br />
*If one of the disks physically failed, but the data is still enclosed into it, see if RAW data into it can be rescued and cloned by some company, or if available sectors can be rescued by dd-rescue. See [[Replacing a failed drive]] for more detailed information about drive failure, including failure in non-redundant arrays.<br />
<br />
= Making sure that none of my arrays will never be lost anymore in the future =<br />
<br />
You can't, because it won't always be your fault. We believe that having succeeded or not into your recovery, you are probably interesting into clever backup approaches for securing the recovered or future data.<br />
<br />
Indeed, a defective power supply can destroy some of your disks. A violent shock on the case containing all the disks can cause several of them to fail. A fire incident too. The server can be stolen (with disks inside of it). A ransomware having access to some part of the filesystem, even through the network file sharing, can also destroy all of the writable files. A wrong dd, shred, or you swapped the wrong "failed" disk, opened it to play with what's inside, before realizing it wasn't the failed one. Data memory module malfunction during a rebuild (it can screw the data of one of your members without letting you know what happened). And may be others things.<br />
<br />
A free open source versioned continuous copy system like Syncthing (in case files are being encrypted, the previous version is kept and timestamped) isn't officially meant to protect you in this kind of cases, but still does it pretty well. Conventional backups systems and solutions could also be used. In case of malicious attack, still remember to use a backup system that is different enough of the system you are trying to backup (not the same password, not the same config, not the same network, not the same place: it would be like a RAID1 on 2 partitions of the same disk).</div>Julien Robinhttps://raid.wiki.kernel.org/index.php/Irreversible_mdadm_failure_recoveryIrreversible mdadm failure recovery2019-05-04T18:12:01Z<p>Julien Robin: </p>
<hr />
<div>== Introduction ==<br />
<br />
If you feel comfortable using overlays, this is always a good idea (so that no accidental write can happen to the real members).<br />
<br />
Every information and situation you can find here has been seriously tested on mdadm 3.4-4 and 4.1-1 before being published.<br />
<br />
Going here is normally not needed, unless mdadm conventional ways won't have any chance to work anymore (we believe that you explored others solution before going here). This section is about searching right parameters to play --create over and existing array, in order to retrieve a correctly configured access to any still available data (be it the entire filesystem if still available).<br />
<br />
Be warned: Playing mdadm --create erases and overwrites at least the RAID array information area of every involved RAID members. A misuse of mdadm --create can be the reason why you are here. Without the --assume-clean parameter, one member data area can be entirely reconstructed (sometimes it's fine, sometimes it's not - so it's better to avoid it). And if you write anything to the array before checking everything, it can cause problems if create parameters were incorrect: this is why it needs to be so careful.<br />
<br />
If the correct RAID array information is still available (then are you sure nothing else can be done with conventional ways?), to save some time, please keep the output of cat /proc/mdstat (for getting the members orders), and the output of mdadm --examine against your differents RAID members. It should be done BEFORE calling any mdadm --create command. Creation and Update time are provided into the output, so you can easily know if it contains your original array information, or if it's already lost/overwritten.<br />
<br />
Members can be entire drives or partitions inside of it. If you erased a GPT table describing a partition by mistake, or removed the partition by mistake, please create it again, unformatted, at the same place (and same size) it was when used as RAID member. The content of the partition may be completely or mostly untouched.<br />
<br />
== Situation analysis ==<br />
<br />
A RAID array member has 2 area: RAID array information (which may be lost if you are here), and the Data area. As long as the Data area is not erased, you can consider the member as useful for the following steps. If Data area is erased, it should be considered as missing for the following steps (and if the missing member is mandatory, don't quit this page before taking a look at the "Unlucky situations" section: may be some files can be recovered).<br />
<br />
You are about to recover entire access to your filesystem as long as:<br />
*Your array is redundant and you didn't lose/overwrite any member data area<br />
*Your array isn't redundant but hopefully, you didn't lose/overwrite any member data area<br />
*Your array is redundant, you have lost/overwritten one member data area, but hopefully there is still enough others members left to get your data back<br />
*The data inside your members are arranged in one consistent and predictable way all over the array<br />
<br />
If you aren't lucky, don't quit this page before taking a look at the "Unlucky situations" section: there's still several things to verify before aborting.<br />
<br />
== Parameters recovery ==<br />
<br />
Playing --create over an existing array needs you to know:<br />
*The number of members in your array.<br />
*The RAID level of your array<br />
*The chunk size of your array if using RAID0, RAID4, RAID5, RAID6, or RAID10 (generally 512K on modern mdadm versions, at least for 3.4-4 and 4.1-1)<br />
*The metadata version of your array (depends of the mdadm version initially used to create the array - 1.2 on modern versions, at least for 3.4-4 and 4.1-1)<br />
*The layout of your array if using RAID5, RAID6, or RAID10 (generally left-symmetric by default for RAID5 and 6, and can be near=2 for RAID10 if your 4 members are A, A, B, B)<br />
*The data offset of your array (depends on the size of your array members used at initial array creation - even if members have been replaced by bigger ones after, and grow --size=max used! It also depends on the version of mdadm you were using at the initial creation)<br />
*The position of the different members into the array (if it changed on the motherboard since initial creation, you may need to try several ones. If not, you probably used alphabetical order)<br />
*Is any of your member missing, or having his Data area overwritten by something like a wrong command? Which one? Because it should be declared as "missing" when running --create. If you don't know which one, you will have to do more attempts.<br />
*--assume-clean will give you the ability to make several attempts without having the last member rebuilt<br />
<br />
If nothing changed on your system (members size, mdadm version...) and you used default parameters, then letting mdadm choosing default ones should be fine and apply the right parameters.<br />
<br />
== Data-offset ==<br />
If you changed mdadm version since initial array creation, data-offset is likely to have changed. I have seen 8192s when created with tiny disks (5000 MiB) into mdadm 3.4-4 (while 4.1-1 defaults to 10240s for the same size, and 18432s for 10000 MiB), and 262144s for 8TB drives with mdadm 3.4-4. Didn't try creating with 8TB members on mdadm 4.1-1. May be some formula can be found into the different code versions?<br />
<br />
= Example of command =<br />
== The easy one ==<br />
Using Debian 9, mdadm 3.4-4, a RAID array was created with with 3 x 8TB members, using:<br />
<br />
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1<br />
<br />
If nothing changed and I'm still using the same mdadm version, typing the same command again (but appending --assume-clean for safety) will be fine: <br />
<br />
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1 --assume-clean<br />
<br />
Gnome-disks shows the file system again and I can mount it (trying read-only before, mount -o ro,noload /dev/mdXX /media/raid-volume) : everything is fine and every member is in the right place into the array. Then, do not forget to do a filesystem check and repair after you ensured that you are using the correct parameters, in case some brutal interruption happened previously on the array.<br />
<br />
== A more precise one ==<br />
For creating the array again after having upgraded from mdadm 3.4-4 to mdadm 4.1-1, data offset should be specified because the automatically selected value for a given size changed. <br />
<br />
Also, disk position changed on my motherboard. After a precise analysis of the previous version defaults parameters for this array, in this case, the working command for finding back my array finally was:<br />
<br />
mdadm --create /dev/md0 --level=5 --chunk=512K --metadata=1.2 --layout left-symmetric --data-offset=262144s --raid-devices=3 /dev/sdd1 /dev/sde1 /dev/sdb1 --assume-clean <br />
<br />
== One facultative member is missing ==<br />
If the second disk is missing, type missing instead of its block device name. On a redundant RAID array, data will still be available. Of course, once your array is back working, you will have to --add the missing member soon or late for safety, but after you ensured that the parameters are OK and your data available. The member you will add will be rebuilt with consistent data. <br />
<br />
== The array was created with 2 disks, and grown to 3 ==<br />
Having the array initially created with 2 disks before having the 3rd one added and the array grown, give the exact same final arrangement compared to a 3 disks array creation.<br />
<br />
= About accidental rebuild =<br />
If you forget the --assume-clean parameter, in case of RAID 5 for example, the last RAID member given in parameter will be rewritten by reading and processing data of the others given members. <br />
<br />
In case of RAID 5:<br />
*Even with wrong order or wrong chunk size, and any wrong RAID5 or RAID4-like layout, if the others members data is fine, and the amount of members is correct, the parity calculation will be correct. Although the computer may not interpret the data the right manner (which bit is parity or data) until you find the right parameters. <br />
*But if you forget the --assume-clean parameter in others cases (like incorrect amount of disks, or RAID level, or reading a lost member to calculate and rewrite over a good one), consider the overwritten disk as missing.<br />
<br />
In case correct data seems to be erased from one of your members, if it wasn't mandatory (if remaining fine members are enough), you will be able to resync it later by --adding it as a new one, but only once your correct parameters are found, and your array back working. If the overwritten member was mandatory, hope you played mdadm --stop. Please find a way to verify if the data is really gone, and if it's gone, go to the "Unlucky situations" section.<br />
<br />
= Help needed, no combination is giving back access to the data =<br />
If you have 3 members and you are sure that data is still available on both of it, try every order: 123 132 213 231 312 321. <br />
<br />
If it's not working, you should find a way to ensure data-offset (and may be others values) are correct, by creating a reconstitution of your original array creation environment to find what default values mdadm probably used. If your values are good, your data are back.<br />
If one of the members has its data missing, and you know which one (if not, its longer), check every combination:<br />
<br />
Member 1/3 is missing: X23, X32, 2X3, 23X, 3X2, 32X<br />
<br />
Member 2/3 is missing: 1X3, 13X, X13, X31, 31X, 3X1<br />
<br />
Member 3/3 is missing: 12X, 1X2, 21X, 2X1, X12, X21<br />
<br />
The numbers stand for the different members block device names, X stands for the "missing" word, in order to represent the parameters order into --create command.<br />
<br />
= Unlucky situations =<br />
<br />
== The RAID array is split into 2 arrangements ==<br />
*Did the reshape/conversion actually started? If you are sure that the answer is no (stuck at 0% with no disk activity, backup file is there but contains nothing, nothing started and it's completely stuck), then things will be easy. Find the parameters set of the unchanged arrangement.<br />
<br />
*If the reshape/conversion actually started and moved some data, this kind of interruption can be restarted just by assembling the array again, and mount/use it as nothing happened. If it doesn't work, see if there is no parameters to force things to continue: don't follow this guide unless RAID information of your members have already been erased, or if you're sure that nothing else worked.<br />
<br />
*If a member failed during reshaping, and your array is redundant, don't play --create! You can add a new member: reshaping will finish in degraded mode, and the last member will be integrated and rebuilt just after.<br />
<br />
*If you have no other choice that playing --create with 2 different arrangements over 2 half of the array, unless someone have a better idea, you're falling into forensics : you'll need to find 2 sets of parameters, and probably need to use a file recovery scanner tool with both parameters set on the array. Be brave! The second set of parameters will probably be the same as the first one, with 1 more member (and raid-devices count set to one more). <br />
<br />
== Mandatory member lost ==<br />
So, your RAID is not redundant (or not enough), and a mandatory member (or the data area into it) is screwed.<br />
<br />
*Is your RAID member partially screwed/overwritten? If mdadm --create (or anything else) started to overwrite it, but you played mdadm --stop before it's entirely overwritten (or stopped anything that was overwriting the data into it), some of the data is still available. Find back your correct parameters (but you won't see any ext4 partition so it will be more difficult to know it you have the right ones) and use some file recovery scanner tool over your resulting array. You may still have the surprise to see some files that aren't completely erased.<br />
<br />
*Also, remember that having mdadm --create accidentally overwriting one of the disks should be avoided, but if it accidentally happens, in some case, calculated data overwritten to the disk may be the same that data which was already into it. In such case, there is a chance that your data is totally fine. You can always try to use the member as if it's OK: if it is, your file system will be made available again.<br />
<br />
*If one of the disks physically failed, but the data is still enclosed into it, see if RAW data into it can be rescued and cloned by some company, or if available sectors can be rescued by dd-rescue. See [[Replacing a failed drive]] for more detailed information about drive failure, including failure in non-redundant arrays.<br />
<br />
= Making sure that none of my arrays will never be lost anymore in the future =<br />
<br />
You can't, because it won't always be your fault. We believe that having succeeded or not into your recovery, you are probably interesting into clever backup approaches for securing the recovered or future data.<br />
<br />
Indeed, a defective power supply can destroy some of your disks. A violent shock on the case containing all the disks can cause several of them to fail. A fire incident too. The server can be stolen (with disks inside of it). A ransomware having access to some part of the filesystem, even through the network file sharing, can also destroy all of the writable files. A wrong dd, shred, or you swapped the wrong "failed" disk, opened it to play with what's inside, before realizing it wasn't the failed one. Data memory module malfunction during a rebuild (it can screw the data of one of your members without letting you know what happened). And may be others things.<br />
<br />
A free open source versioned continuous copy system like Syncthing (in case files are being encrypted, the previous version is kept and timestamped) isn't officially meant to protect you in this kind of cases, but still does it pretty well. Conventional backups systems and solutions could also be used. In case of malicious attack, still remember to use a backup system that is different enough of the system you are trying to backup (not the same password, not the same config, not the same network, not the same place: it would be like a RAID1 on 2 partitions of the same disk).</div>Julien Robinhttps://raid.wiki.kernel.org/index.php/Irreversible_mdadm_failure_recoveryIrreversible mdadm failure recovery2019-05-04T18:11:15Z<p>Julien Robin: English language corrections, and better phrasing, sentence retouching for more precision</p>
<hr />
<div>== Introduction ==<br />
<br />
If you feel comfortable using overlays, this is always a good idea (so that no accidental write can happen to the real members).<br />
<br />
Every information and situation you can find here has been seriously tested on mdadm 3.4-4 and 4.1-1 before being published.<br />
<br />
Going here is normally not needed, unless mdadm conventional ways won't have any chance to work anymore (we believe that you explored others solution before going here). This section is about searching right parameters to play --create over and existing array, in order to retrieve a correctly configured access to any still available data (be it the entire filesystem if still available).<br />
<br />
Be warned: Playing mdadm --create erases and overwrites at least the RAID array information area of every involved RAID members. A misuse of mdadm --create can be the reason why you are here. Without the --assume-clean parameter, one member data area can be entirely reconstructed (sometimes it's fine, sometimes it's not - so it's better to avoid it). And if you write anything to the array before checking everything, it can cause problems if create parameters were incorrect: this is why it needs to be so careful.<br />
<br />
If the correct RAID array information is still available (then are you sure nothing else can be done with conventional ways?), to save some time, please keep the output of cat /proc/mdstat (for getting the members orders), and the output of mdadm --examine against your differents RAID members. It should be done BEFORE calling any mdadm --create command. Creation and Update time are provided into the output, so you can easily know if it contains your original array information, or if it's already lost/overwritten.<br />
<br />
Members can be entired drives or partitions inside of it. If you erased a GPT table describing a partition by mistake, or removed the partition by mistake, please create it again, unformatted, at the same place (and same size) it was when used as RAID member. The content of the partition may be completely or mostly untouched.<br />
<br />
== Situation analysis ==<br />
<br />
A RAID array member has 2 area: RAID array information (which may be lost if you are here), and the Data area. As long as the Data area is not erased, you can consider the member as useful for the following steps. If Data area is erased, it should be considered as missing for the following steps (and if the missing member is mandatory, don't quit this page before taking a look at the "Unlucky situations" section: may be some files can be recovered).<br />
<br />
You are about to recover entire access to your filesystem as long as:<br />
*Your array is redundant and you didn't lose/overwrite any member data area<br />
*Your array isn't redundant but hopefully, you didn't lose/overwrite any member data area<br />
*Your array is redundant, you have lost/overwritten one member data area, but hopefully there is still enough others members left to get your data back<br />
*The data inside your members are arranged in one consistent and predictable way all over the array<br />
<br />
If you aren't lucky, don't quit this page before taking a look at the "Unlucky situations" section: there's still several things to verify before aborting.<br />
<br />
== Parameters recovery ==<br />
<br />
Playing --create over an existing array needs you to know:<br />
*The number of members in your array.<br />
*The RAID level of your array<br />
*The chunk size of your array if using RAID0, RAID4, RAID5, RAID6, or RAID10 (generally 512K on modern mdadm versions, at least for 3.4-4 and 4.1-1)<br />
*The metadata version of your array (depends of the mdadm version initially used to create the array - 1.2 on modern versions, at least for 3.4-4 and 4.1-1)<br />
*The layout of your array if using RAID5, RAID6, or RAID10 (generally left-symmetric by default for RAID5 and 6, and can be near=2 for RAID10 if your 4 members are A, A, B, B)<br />
*The data offset of your array (depends on the size of your array members used at initial array creation - even if members have been replaced by bigger ones after, and grow --size=max used! It also depends on the version of mdadm you were using at the initial creation)<br />
*The position of the different members into the array (if it changed on the motherboard since initial creation, you may need to try several ones. If not, you probably used alphabetical order)<br />
*Is any of your member missing, or having his Data area overwritten by something like a wrong command? Which one? Because it should be declared as "missing" when running --create. If you don't know which one, you will have to do more attempts.<br />
*--assume-clean will give you the ability to make several attempts without having the last member rebuilt<br />
<br />
If nothing changed on your system (members size, mdadm version...) and you used default parameters, then letting mdadm choosing default ones should be fine and apply the right parameters.<br />
<br />
== Data-offset ==<br />
If you changed mdadm version since initial array creation, data-offset is likely to have changed. I have seen 8192s when created with tiny disks (5000 MiB) into mdadm 3.4-4 (while 4.1-1 defaults to 10240s for the same size, and 18432s for 10000 MiB), and 262144s for 8TB drives with mdadm 3.4-4. Didn't try creating with 8TB members on mdadm 4.1-1. May be some formula can be found into the different code versions?<br />
<br />
= Example of command =<br />
== The easy one ==<br />
Using Debian 9, mdadm 3.4-4, a RAID array was created with with 3 x 8TB members, using:<br />
<br />
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1<br />
<br />
If nothing changed and I'm still using the same mdadm version, typing the same command again (but appending --assume-clean for safety) will be fine: <br />
<br />
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1 --assume-clean<br />
<br />
Gnome-disks shows the file system again and I can mount it (trying read-only before, mount -o ro,noload /dev/mdXX /media/raid-volume) : everything is fine and every member is in the right place into the array. Then, do not forget to do a filesystem check and repair after you ensured that you are using the correct parameters, in case some brutal interruption happened previously on the array.<br />
<br />
== A more precise one ==<br />
For creating the array again after having upgraded from mdadm 3.4-4 to mdadm 4.1-1, data offset should be specified because the automatically selected value for a given size changed. <br />
<br />
Also, disk position changed on my motherboard. After a precise analysis of the previous version defaults parameters for this array, in this case, the working command for finding back my array finally was:<br />
<br />
mdadm --create /dev/md0 --level=5 --chunk=512K --metadata=1.2 --layout left-symmetric --data-offset=262144s --raid-devices=3 /dev/sdd1 /dev/sde1 /dev/sdb1 --assume-clean <br />
<br />
== One facultative member is missing ==<br />
If the second disk is missing, type missing instead of its block device name. On a redundant RAID array, data will still be available. Of course, once your array is back working, you will have to --add the missing member soon or late for safety, but after you ensured that the parameters are OK and your data available. The member you will add will be rebuilt with consistent data. <br />
<br />
== The array was created with 2 disks, and grown to 3 ==<br />
Having the array initially created with 2 disks before having the 3rd one added and the array grown, give the exact same final arrangement compared to a 3 disks array creation.<br />
<br />
= About accidental rebuild =<br />
If you forget the --assume-clean parameter, in case of RAID 5 for example, the last RAID member given in parameter will be rewritten by reading and processing data of the others given members. <br />
<br />
In case of RAID 5:<br />
*Even with wrong order or wrong chunk size, and any wrong RAID5 or RAID4-like layout, if the others members data is fine, and the amount of members is correct, the parity calculation will be correct. Although the computer may not interpret the data the right manner (which bit is parity or data) until you find the right parameters. <br />
*But if you forget the --assume-clean parameter in others cases (like incorrect amount of disks, or RAID level, or reading a lost member to calculate and rewrite over a good one), consider the overwritten disk as missing.<br />
<br />
In case correct data seems to be erased from one of your members, if it wasn't mandatory (if remaining fine members are enough), you will be able to resync it later by --adding it as a new one, but only once your correct parameters are found, and your array back working. If the overwritten member was mandatory, hope you played mdadm --stop. Please find a way to verify if the data is really gone, and if it's gone, go to the "Unlucky situations" section.<br />
<br />
= Help needed, no combination is giving back access to the data =<br />
If you have 3 members and you are sure that data is still available on both of it, try every order: 123 132 213 231 312 321. <br />
<br />
If it's not working, you should find a way to ensure data-offset (and may be others values) are correct, by creating a reconstitution of your original array creation environment to find what default values mdadm probably used. If your values are good, your data are back.<br />
If one of the members has its data missing, and you know which one (if not, its longer), check every combination:<br />
<br />
Member 1/3 is missing: X23, X32, 2X3, 23X, 3X2, 32X<br />
<br />
Member 2/3 is missing: 1X3, 13X, X13, X31, 31X, 3X1<br />
<br />
Member 3/3 is missing: 12X, 1X2, 21X, 2X1, X12, X21<br />
<br />
The numbers stand for the different members block device names, X stands for the "missing" word, in order to represent the parameters order into --create command.<br />
<br />
= Unlucky situations =<br />
<br />
== The RAID array is split into 2 arrangements ==<br />
*Did the reshape/conversion actually started? If you are sure that the answer is no (stuck at 0% with no disk activity, backup file is there but contains nothing, nothing started and it's completely stuck), then things will be easy. Find the parameters set of the unchanged arrangement.<br />
<br />
*If the reshape/conversion actually started and moved some data, this kind of interruption can be restarted just by assembling the array again, and mount/use it as nothing happened. If it doesn't work, see if there is no parameters to force things to continue: don't follow this guide unless RAID information of your members have already been erased, or if you're sure that nothing else worked.<br />
<br />
*If a member failed during reshaping, and your array is redundant, don't play --create! You can add a new member: reshaping will finish in degraded mode, and the last member will be integrated and rebuilt just after.<br />
<br />
*If you have no other choice that playing --create with 2 different arrangements over 2 half of the array, unless someone have a better idea, you're falling into forensics : you'll need to find 2 sets of parameters, and probably need to use a file recovery scanner tool with both parameters set on the array. Be brave! The second set of parameters will probably be the same as the first one, with 1 more member (and raid-devices count set to one more). <br />
<br />
== Mandatory member lost ==<br />
So, your RAID is not redundant (or not enough), and a mandatory member (or the data area into it) is screwed.<br />
<br />
*Is your RAID member partially screwed/overwritten? If mdadm --create (or anything else) started to overwrite it, but you played mdadm --stop before it's entirely overwritten (or stopped anything that was overwriting the data into it), some of the data is still available. Find back your correct parameters (but you won't see any ext4 partition so it will be more difficult to know it you have the right ones) and use some file recovery scanner tool over your resulting array. You may still have the surprise to see some files that aren't completely erased.<br />
<br />
*Also, remember that having mdadm --create accidentally overwriting one of the disks should be avoided, but if it accidentally happens, in some case, calculated data overwritten to the disk may be the same that data which was already into it. In such case, there is a chance that your data is totally fine. You can always try to use the member as if it's OK: if it is, your file system will be made available again.<br />
<br />
*If one of the disks physically failed, but the data is still enclosed into it, see if RAW data into it can be rescued and cloned by some company, or if available sectors can be rescued by dd-rescue. See [[Replacing a failed drive]] for more detailed information about drive failure, including failure in non-redundant arrays.<br />
<br />
= Making sure that none of my arrays will never be lost anymore in the future =<br />
<br />
You can't, because it won't always be your fault. We believe that having succeeded or not into your recovery, you are probably interesting into clever backup approaches for securing the recovered or future data.<br />
<br />
Indeed, a defective power supply can destroy some of your disks. A violent shock on the case containing all the disks can cause several of them to fail. A fire incident too. The server can be stolen (with disks inside of it). A ransomware having access to some part of the filesystem, even through the network file sharing, can also destroy all of the writable files. A wrong dd, shred, or you swapped the wrong "failed" disk, opened it to play with what's inside, before realizing it wasn't the failed one. Data memory module malfunction during a rebuild (it can screw the data of one of your members without letting you know what happened). And may be others things.<br />
<br />
A free open source versioned continuous copy system like Syncthing (in case files are being encrypted, the previous version is kept and timestamped) isn't officially meant to protect you in this kind of cases, but still does it pretty well. Conventional backups systems and solutions could also be used. In case of malicious attack, still remember to use a backup system that is different enough of the system you are trying to backup (not the same password, not the same config, not the same network, not the same place: it would be like a RAID1 on 2 partitions of the same disk).</div>Julien Robinhttps://raid.wiki.kernel.org/index.php/Linux_RaidLinux Raid2019-05-04T16:57:23Z<p>Julien Robin: /* When Things Go Wrogn */ : added the link to a new page with information about recovering what remains of a definitely failed array</p>
<hr />
<div>== Introduction ==<br />
<br />
This site is the Linux-raid kernel list community-managed reference for Linux software RAID as implemented in recent version 4 kernels and earlier.<br />
It should replace many of the unmaintained and out-of-date documents ''out there'' such as the '''Software RAID HOWTO''' and the '''Linux RAID FAQ'''.<br />
<br />
Where possible, information should be tagged with the minimum kernel/software version required to use the feature. Some of the information on these pages are unfortunately quite old, but we are in the process of updating the info (aren't we always...)<br />
<br />
== Mailing list ==<br />
<br />
Linux RAID issues are discussed in the linux-raid mailing list to be found at http://vger.kernel.org/vger-lists.html#linux-raid<br />
<br />
This follows kernel.org conventions. You should use "reply to all" unless explicitly requested. Extraneous material should be trimmed. Replies should be in-line or at the bottom. (Please read posters' .sig's, as this may well say "I read the list - please do not cc me".)<br />
<br />
And please use an email client that threads correctly!<br />
<br />
== Help wanted ==<br />
This site was created by [[David Greaves]] and [[Nick Yeates]]. But life moved on and having tried to provide up-to-date info, the info became out of date again. [[Keld Simonsen]] updated a lot of the information, and made good ratings for Google.<br />
<br />
As of September 2016 [[User:Anthony Youngman|Wol]] is updating it to mdadm 3.3 and the 4.x kernels (mdadm 4.0 was released in January 2017). Please contact Wol, Keld or Nick if you want to help. Please read the [[editing guidelines]].<br />
<br />
As of June 2018, the main site is pretty up-to-date. There are, however, a lot of holes in the information available, any help you can give to the editor(s) would be gratefully received - offers of articles, case studies of raid implementations that aren't covered at present, hardware to practice on to avoid corrupting live systems, etc etc.<br />
<br />
== Overview ==<br />
<br />
* [[What is RAID and why should you want it?]]<br />
* [[Choosing your hardware, and what is a device?]]<br />
* [[What do you want in your stack?]]<br />
* [[RAID and filesystems]]<br />
* [[Setting up a (new) system]]<br />
* [[Converting an existing system]]<br />
* [[A guide to mdadm]]<br />
* [[Scrubbing the drives]]<br />
* [[Monitoring your system]]<br />
<br />
[TODO: discuss layering things on top of raid, ie partitioning an array, LVM, or a btrfs filesystem]<br />
<br />
The 2016 rewrite is not covering LVM (at the moment) so for LVM you will find all the old stuff in the archaeology section. Also all the performance data is 2011 vintage, so that has been relegated to the archaeology section too.<br />
<br />
== When Things Go Wrogn ==<br />
<br />
Don't panic, Mister Mainwaring!<br />
<br />
RAID is very good at protecting your data. In fact, NEARLY ALL data lost as reported to the raid mailing list, is down to user error while attempting to recover a failed array.<br />
<br />
In particular NEVER NEVER NEVER use "mdadm --create" on an already-existing array unless you are being guided by an expert. It is the single most effective way of turning a simple recovery exercise into a major forensic problem - it may not be quite as effective as "dd if=/dev/random of=/dev/sda", but it's pretty close ...<br />
<br />
The simplest things are sometimes the best. If an array fails to start after a crash or reboot and you can't get it to assemble, always try an "mdadm /dev/mdN --stop", and then try to assemble it again. Problems at boot often leave you with a partially assembled array that then refuses to do anything. A "stop" followed by an "assemble" can never do any harm, and may well fix the problem. Be very careful with "--force", though, as it may trigger a resync which could destroy the contents of a drive and make recovery difficult or impossible.<br />
<br />
Also, make sure you are using the latest mdadm (see [[A guide to mdadm]]).<br />
<br />
* [[Asking for help]]<br />
* [[Timeout Mismatch|Timeout Mismatch (Why you shouldn't use desktop drives)]]<br />
* [[ What's all this with USB? ]]<br />
* [[The Badblocks controversy]]<br />
* [[Easy Fixes|Easy Fixes (Scary problems that tend to resolve themselves)]]<br />
* [[Replacing a failed drive]]<br />
* [[Assemble Run|My array won't assemble / run]]<br />
* [[Recovering a damaged RAID]]<br />
* [[Advanced data recovery|mdadm says my array doesn't exist! (WIP)]]<br />
* [[Irreversible mdadm failure recovery]]<br />
<br />
In addition to reading this, it is probably worth while going to the software archaeology section and reading "RAID Recovery" and "Recovering a failed software RAID". Just be aware these are old pages, and things may have changed. And that everything that is relevant in 2016 should have been copied into the above pages.<br />
<br />
== Areas Of Interest ==<br />
<br />
* [[RAID Creation]]<br />
* [[Drive Data Sheets]]<br />
<br />
== Hardware RAID ==<br />
Proper hardware RAID systems are presented to linux as a block device and there's no coverage of them (yet) in this wiki.<br />
<br />
* [[Hardware Raid Setup using MegaCli]]<br />
<br />
BIOS / firmware RAID aka [https://ata.wiki.kernel.org/index.php/SATA_RAID_FAQ fake raid cards]:<br />
* offer a few performance benefits (like CPU, bus and RAM offloading), but may often be much slower than SW raid (link?)<br />
* if the 'raid' card or motherboard dies then you often have to find an exact replacement and this can be tricky for older cards<br />
* if drives move to other machines the data can't easily be read<br />
* there is usually no monitoring or reporting on the array - if a problem occurs then it may not show up unless the machine is rebooted *and* someone is actually watching the BIOS boot screen (or until multiple errors occur and your data is lost)<br />
* you are entrusting your data to unpatchable software written into a BIOS that has probably not been tested, has no support mechanism and almost no community.<br />
* having seen how many bugs the kernel works around in various BIOSes it would be optimistic to think that the BIOS RAID has no bugs.<br />
<br />
Given the point of RAID is usually to reduce risk it is fair to say that using fakeraid is a terrible idea and it's better to focus energy on either true HW raid or in-kernel SW raid .... but there is nothing stopping you :)<br />
<br />
== Kernel Programming ==<br />
<br />
This section is meant to be the home for a variety of things. With Neil Brown stepping down as maintainer (early 2016), the development process doesn't seem to be quite so "robust". Not a surprise, the new maintainers need to gain the experience Neil had of the subsystem. So this section will house documentation about how the internals of the raid subsystem works.<br />
<br />
RIP Shaohua Li. [https://lwn.net/Articles/775861/ Shaohua Li (LWN)] Shaohua took over from Neil, but sadly died over Christmas 2018. Jens Axboe has stepped up as temporary maintainer.<br />
<br />
But documentation without specification isn't much use. There was a philosophy (famously espoused by Microsoft, especially in their Office Open XML Specification) that "the code is the documentation" or "the code is the specification". This is great for coders - one of its features is that it eliminates all bugs at a stroke! If the code is the specification, then the system <em>has</em> to behave as specified. So this section will also house documentation about how the internals of the raid subsystem are supposed to work.<br />
<br />
Then, of course, we want as many people helping with the system as possible. So this section will also contain a list of projects that people can do, and some advice on help for them on where to start. They needn't be work on the kernel itself, or mdadm, there are utilities already out there (Phil's lsdrv, Brad's timeout script) and there are plenty more that would be appreciated.<br />
<br />
[[Programming projects]]<br />
<br />
== Archaeology ==<br />
<br />
This section is where all the old pages have been moved. Some of them may have been edited before being moved but the information here is mostly out-of-date, such as lilo, raidtools, etc. It may well be of interest to people running old systems, but shouldn't be in the main section where it may confuse people.<br />
<br />
[[Valley Of The Kings|RAID Archaeology]]<br />
<br />
== External links ==<br />
* [[wikipedia:How to edit a page|Editing pages]]<br />
* [http://en.wikipedia.org/wiki/RAID Wikipedia RAID] including description of specific Linux RAID types<br />
* [http://www.snia.org/tech_activities/standards/curr_standards/ddf/ Common RAID Disk Drive Format (DDF) standard] specifying standard RAID levels from SNIA<br />
* [http://wiki.kernelnewbies.org/ Kernel Newbies] basic information about working on kernel<br />
* [http://kernel.org/pub/linux/kernel/people/hpa/raid6.pdf The mathematics of RAID6]<br />
* [https://ata.wiki.kernel.org/index.php/SATA_RAID_FAQ FAQ about hardware/fake raid cards]<br />
* [http://linuxmafia.com/faq/Hardware/sata.html HW RAID support in Linux]<br />
* [http://www.runmapglobal.com/blog/fault-tolerant-dedicated-servers/ Fault-Tolerant RAID]<br />
* [http://marc.info/?l=linux-raid&r=1&w=2 linux-raid mailing list archives]<br />
<br />
<br><br />
See [[Spam Blocks]] for the spam restrictions on this site.</div>Julien Robinhttps://raid.wiki.kernel.org/index.php/Irreversible_mdadm_failure_recoveryIrreversible mdadm failure recovery2019-05-04T16:54:03Z<p>Julien Robin: Creation of a page summarizing complete and tested procedures to recover from catastrophic situation in which mdadm conventional recover feature failed</p>
<hr />
<div>== Introduction ==<br />
<br />
If you feel comfortable using overlays, this is always a good idea (so that no accidental write can happen to the real members).<br />
<br />
Every information and situation you can found here has been seriously tested on mdadm 3.4-4 and 4.1-1 before being published.<br />
<br />
Going here is normally not needed, unless mdadm conventional ways won't have any chance to work anymore (we believe that you explored others solution before going here). This section is about searching right parameters to play --create over and existing array, in order to retrieve a correctly configured access to any still available data (be it the entire filesystem if still available).<br />
<br />
Be warned : Playing mdadm --create erase and overwrite at least the RAID array information area of every involved RAID members. A misuse of mdadm --create can be the reason why you are here. Without the --assume-clean parameter, one member data area can be entirely reconstructed (sometimes it's fine, sometimes it's not - so it's better to avoid it). And if you write anything to the array before checking everything, it can cause problems if create parameters were incorrect : this is why it needs to be so careful.<br />
<br />
If the correct RAID array information is still available, to save some time, keeping the output of cat /proc/mdstat (for getting the members orders), and the output of mdadm --examine against your different raid members, should if possible, be done BEFORE calling any mdadm --create command. Creation and Update time are provided into the output, so you can easily know if it contains your original array information, or if it's already lost/overwritten. If you're not sure whether you used a partition on a disk as member, or the entire disk without partition as member, --examine can help you too.<br />
<br />
== Situation analysis ==<br />
<br />
A RAID array member has 2 area : RAID array information (which may be lost if you are here), and the Data area. As long as the Data area is not erased, you can consider the member as useful for the following steps. If Data area is erased, it should be considered as missing for the following steps (and if the missing member is mandatory, don't quit this page before taking a look at the "Unlucky situations" section : may be some files can be recovered)<br />
<br />
You are about to recover entire access to your filesystem as long as :<br />
*Your array is redundant and you didn't lose/overwrite any member data area<br />
*Your array isn't redundant but hopefully, you didn't lose/overwrite any member data area<br />
*Your array is redundant, you have lost/overwritten one member data area, but hopefully there is still enough others members left to get your data back<br />
*The data inside your members are arranged in one consistent and predictable way all over the array<br />
<br />
If you aren't lucky, don't quit this page before taking a look at the "Unlucky situations" section : there's still several things to verify before aborting<br />
<br />
== Parameters recovery ==<br />
<br />
Playing --create over an existing array needs you to know :<br />
*The number of disk in your array.<br />
*The RAID level of your array<br />
*The chunk size of your array (if using RAID0, RAID4, RAID5, RAID6, and RAID10 - generally 512K on modern mdadm versions, at least for 3.4-4 and 4.1-1)<br />
*The metadata version of your array (depends of the mdadm version initially used to create the array - 1.2 on modern versions, at least for 3.4-4 and 4.1-1)<br />
*The layout of your array in case of RAID5, RAID6, and RAID10 (generally left-symmetric by default for RAID5 and 6, and can be near=2 for RAID10 if your 4 members are A, A, B, B)<br />
*The data offset of your array (depends on the size of your array members used at initial array creation - even if members have been replaced by bigger ones after, and grow --size=max used! It also depends on the version of mdadm you were using at the initial creation)<br />
*The position of the different disks into the array (if it changed on the motherboard since initial creation, you may need to try several ones. If not, you probably used alphabetical order)<br />
*Is any of your disk missing, or having his Data area overwritten by something like a wrong command ? Which one ? Because it should be declared as "missing" when running --create. if you don't know which one, you will have to do more attempts.<br />
*--assume-clean will give you the ability to make several attempts without having the last member rebuilt<br />
<br />
If nothing changed on your system (members size, mdadm version...) and you used default parameters, then letting mdadm choosing default ones should be fine and apply the right parameters.<br />
<br />
== data-offset ==<br />
If you changed mdadm version since initial creation, data-offset is likely to have changed. I have seen 8192s when created with tiny disks (5000 MiB) into mdadm 3.4-4 (while 4.1-1 defaults to 10240s for the same size, and 18432s for 10000 MiB with mdadm 4.1-1), and 262144s for 8TB drives with mdadm 3.4-4. Didn't try creating with 8TB members on mdadm 4.1-1. May be some formula can be found into the different code versions?<br />
<br />
= Example of command =<br />
== The easy one ==<br />
Using Debian 9, mdadm 3.4-4, a RAID array was created with with 3 x 8TB members, using :<br />
<br />
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1<br />
<br />
If nothing changed and I'm still using the same mdadm version, typing the same command again (but appending --assume-clean for safety) will be fine : <br />
<br />
mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1 --assume-clean<br />
<br />
Gnome-disks shows the file system again and I can mount it (trying read-only before, mount -o ro,noload /dev/mdXX /media/raid-volume) : everything is fine and every member is in the right place into the array. Then, do not forget to do a filesystem check and repair after you ensured that your are using the correct parameters, in case some brutal interruption happened previously on the array.<br />
<br />
== A more precise one ==<br />
For creating back the array again after having upgraded to mdadm 4.1-1, data offset should be specified because the automatically selected value for a given size changed. <br />
<br />
Also, disk position changed on my motherboard. After a precise analysis of the previous version defaults parameters for this array, in this case, the working command for finding back my array finally was :<br />
<br />
mdadm --create /dev/md0 --level=5 --chunk=512K --metadata=1.2 --layout left-symmetric --data-offset=262144s --raid-devices=3 /dev/sdd1 /dev/sde1 /dev/sdb1 --assume-clean <br />
<br />
== One facultative member is missing ==<br />
If the second disk is missing, type missing instead of its block device name. On a redundant RAID array, data will still be available. Of course, once your array is back working, you will have to --add the missing member soon or late for safety, but after you ensured that the parameters are OK and your data available. The member you will add will be rebuilt with consistent data. <br />
<br />
== The array was created with 2 disk, and grown to 3 ==<br />
Having the array initially created with 2 disks before having the 3rd one added and the array grown, give the exact same final arrangement compared to a 3 disk array creation.<br />
<br />
= About accidental rebuild =<br />
If you forget the --assume-clean parameter, in case of RAID 5 for example, the last RAID member given in parameter will be rewritten by reading and processing data of the others given members. <br />
<br />
In case of RAID 5 : <br />
*Even with wrong order or wrong chunk size, and any wrong RAID5 or RAID4-like layout, if the others members data is fine, and the amount of members is correct, the parity calculation will be correct. Although the computer may not interpret the data the right manner (which bit is parity or data) until you find the right parameters. <br />
*But if you forget the --assume-clean parameter in others cases (like incorrect amount of disks, or RAID level, or reading a lost member to calculate and rewrite over a good one), consider the overwritten disk as missing.<br />
<br />
In case correct data seems to be erased from one of your member, if it wasn't mandatory (if remaining fine members are enough), you will be able to resync it later. If the overwritten member was mandatory, hope you played mdadm --stop. Please find a way to verify if the data is really gone, and if it's gone, go to the "Unlucky situations" section.<br />
<br />
= Help needed, no combination is giving back access to the data =<br />
If you have 3 members and you are sure that data is still available on both of it, try every order : 123 132 213 231 312 321. <br />
<br />
If it's not working, you should find a way to ensure data-offset (and may be others values) are correct, by creating a reconstitution of your original array creation environment to find what default values mdadm probably used. If your values are good, your data are back.<br />
If one of the members has its data missing, and you know which one (if not, its longer), check every combination :<br />
<br />
Member 1/3 is missing : X23 X32 2X3 23X 3X2 32X<br />
<br />
Member 2/3 is missing : 1X3 13X X13 X31 31X 3X1<br />
<br />
Member 3/3 is missing : 12X 1X2 21X 2X1 X12 X21<br />
<br />
(1 2 and 3 stands for your members, X for missing)<br />
<br />
= Unlucky situations =<br />
<br />
== The RAID array is split into 2 arrangements ==<br />
*Did the reshape/conversion actually started ? If you are sure that the answer is no (stuck at 0% with no disk activity, backup file is there but contains nothing, nothing started and it's completely stuck), then things will be easy. Find the parameters set of the unchanged arrangement.<br />
<br />
*If the reshape/conversion actually started and moved some data, this kind of interruption can be restarted just by assembling the array again, and mount/use it as nothing happened. If it doesn't work, see if there is no parameters to force things to continue : don't follow this guide unless RAID information of your members have already been erased, or if you're sure that nothing else worked.<br />
<br />
*If a member failed during reshaping, and your array is redundant, don't play --create! You can add a new member : reshaping will finish in degraded mode, and the last member will be integrated and rebuilt just after.<br />
<br />
*If you have no other choice that playing --create with 2 different arrangements over 2 half of the array, unless someone have a better idea, you're falling into forensics : you'll need to find 2 sets of parameters, and probably need to use a file recovery scanner tool with both parameters set on the array. Be brave! The second set of parameters will probably be the same as the first one, with 1 more member (and raid-devices count set to one more). <br />
<br />
== Mandatory member lost ==<br />
So your RAID is not redundant (or not enough), and a mandatory member (or the data area into it) is screwed.<br />
<br />
*Is your RAID member partially screwed/overwritten ? If mdadm --create (or anything else) started to overwrite it, but you played mdadm --stop before it's entirely overwritten (or stopped anything that was overwriting the data into it), some of the data is still available. Find back your correct parameters (but you won't see any ext4 partition so it will be more difficult to know it you have the right ones) and use some file recovery scanner tool over your resulting array. You may still have the surprise to see some files that aren't completely erased.<br />
<br />
*Also, remember that having mdadm --create accidentally overwriting one of the disks should be avoided, but if it accidentally happen, in some case, calculated data overwritten to the disk may be the same that data which was already into it. In such case, there is a chance that your data is totally fine. You can always try to use the member as if it's OK : if it is, your file system will be made available again.<br />
<br />
*If one of the disk physically failed, but the data is still enclosed into it, see if RAW data into it can be rescued and cloned by some company, or if available sectors can be rescued by dd-rescue.<br />
See [[Replacing a failed drive]] for more detailed information about drive failure, including failure in non redundant arrays.<br />
<br />
= Making sure that none of my arrays will never be lost anymore in the future =<br />
<br />
You can't, because it won't always be your fault.<br />
<br />
Remember that a defective power supply can destroy some of your disks. A fire too. The server can be stolen (with disks inside of it). A ransomware having access to some part of the filesystem trough the network can also destroy all of the accessible files. A wrong dd, shred, or you swapped the wrong "failed" disk, opened it to play with what's inside, because you thought it was the broken one. Data memory module malfunction during a rebuild (it can screw the data of one of your members without letting you know what happened).<br />
<br />
A free open source versioned continuous copy system like Syncthing (in case files are being encrypted, the previous version is kept and timestamped) isn't officially meant to protect you in this kind of cases, but still does it pretty well. Conventional backups systems could also be used. In case of malicious attack, still remember to use a backup system that is way different of the system you are trying to backup (not the same password, not the same config, not the same network, not the same place : it would be like a RAID1 on 2 partitions of the same disk).</div>Julien Robin