Can't access drive - Need raid5/linux help

[imt17]

I had actually tried this last night. It won;t kill it. From what I read if a status is D during the process it will remain that way forever. Its impossible to kill a process with a D status.

What I then tried was just pulling the USB. That of course solved the issue and then I tried again. This time I used the -v so I can see where in the copying it stopped. I tried again. But what I noticed is that this time it couldn;t have stopped in the same place, since the amount of files copies to the USB is far less. This AM I tried a third time. First I went into Ubtunu and formatted the partition here as FAT32 to test that out. I did a copy again but it again froze. This time even smaller amount of files copied before it failed. Since the amount of files transferred keeps shrinking could this be a memory issue? USB drive is 3TB but figure the size of the drive shouldn't matter. Thoughts?

[imt17]

I am testing out a plan. I am now using the 1TB replacement drive I bought (for the bad drive in the Lacie). I formatted it with FAT and using it in my USB to SATA dock. I also did a search to see if there is a way to pause a process. I see that if I do <ctl>+z, this will suspend the copying. I then can do fg to resume.

I have another terminal window running top. This way I can see memory usage for used and free. I see memory usage jus significantly when copying and the free space, which isn;t much left, drops lower and lower then may rebound etc. I am thinking that the IO of the USB was getting overwhelmed and then the memory was getting used up on the NAS to process the files. So I have started the copy again and will pause every 10-15min and let the memory usage drop and the free space build and then do it again.

For example, I stopped the process prior to me typing this email. Right now the memory used is down to 42422444 and free is up to 89740k. Now I restart the process again and used jumps to 508000 and free is down to 3936. But this fluctuates. can get to like 4800k in free and down to 3700k, etc. So watching to make sure it doesn't plummet below 3000 fr free.

[imt17]

Same issue. I was letting it go since the free memory seemed stable and then it crashed again. Went into D state and then get the IO error. I then did <ctk>+z and tried to start it back up again. Got errors that drive doesn;t exists and then saw an IO error i=on the screen. I then went to see all partitions and the USB were all gone so I'm guessing when it hits the D state the partition is kicked which is why it freezes. Any thoughts?

[Jocko]

Hi imt17,

Other idea and I think better :

- unplug your USB disk and use directly the sata port of the fail disk (sdd)

- run udevstart

- in /proc/partitions you should see the new device sdd and at least sdd1

- mount it (same cmdline than with your WD USB drive)

Note: try to use only one telnet window, and think that top cmd uses some resources

[imt17]

That was a great idea. Thanks.

This time it finished. Surprised it took so long. Its like 676gb, which I figured would have taken less then 2 hrs. I gave up and checked back in the AM and saw I gt the prompt back so new it was completed and didn't hand. I unmounted /usb and pulled the drive. I put it in my usb to sata dock and checked out the files. Seems like everything is here.

I am now copying everything off of the 3.5" 1gb drive to the 3TB WD USB Drive. Will take half the day today.

So after this is done I'm assuming next steps would involve putting the drive back into the Lacie, Formatting it and somehow getting it part of the array so that it uses this 5th disk and rebuilds the array.

Then I have to get the Lacie up and running again.

Just need to know exactly what commands and in what order I need to run.

[Jocko]

Hi imt17,

Before replacing the fail disk, I want to check the other raids which use partitions 5, 7, 8 and 9 :

So do

mdadm --examine --scan --verbose

and try to assemble them (md0 with partitions 7; md1 /partitions 8; md2 /partitions 9; md3 /partitions 5)
with this cmdline

Code: Select all

mdadm --assemble /dev/md? --force --run --uuid=???????????

(replace '?' by values got in the previous output)

Post the content of /proc/mdstat after assembling these raid devices.

If all are the status "clean,degraded", try to reboot the nas. (I'd rather you do it as overwrite raid array is not fully safe but we should do it if your NAS still fails to start)

[imt17]

Code: Select all

root@fvdw-sta-kirkwood:/ # mdadm --examine --scan --verbose
ARRAY /dev/md/4  level=raid5 metadata=1.0 num-devices=5 UUID=36d39884:a2d4762b:3e519be3:fe529c84 name=(none):4
   devices=/dev/sde2,/dev/sda2,/dev/sdb2,/dev/sdc2
ARRAY /dev/md3 level=raid1 num-devices=5 UUID=b2ab486e:87e9a97c:3d186b3c:53958f34
   devices=/dev/sde5,/dev/sda5,/dev/sdb5,/dev/sdc5
ARRAY /dev/md0 level=raid1 num-devices=5 UUID=da0c61be:48b2e856:4c17d85b:cd614625
   devices=/dev/sde7,/dev/sda7,/dev/sdb7,/dev/sdc7
ARRAY /dev/md1 level=raid1 num-devices=5 UUID=7f16912a:3fca4e8e:3d186b3c:53958f34
   devices=/dev/sde8,/dev/sda8,/dev/sdb8,/dev/sdc8
ARRAY /dev/md2 level=raid1 num-devices=5 UUID=6931896d:781a3429:3d186b3c:53958f34
   devices=/dev/sde9,/dev/sda9,/dev/sdb9,/dev/sdc9


Code: Select all

root@fvdw-sta-kirkwood:/ # mdadm --assemble /dev/md3 --force --run --uuid=b2ab486e:87e9a97c:3d186b3c:53958f34
mdadm: ignoring /dev/sda5 as it reports /dev/sde5 as failed
mdadm: ignoring /dev/sdb5 as it reports /dev/sde5 as failed
mdadm: ignoring /dev/sdc5 as it reports /dev/sde5 as failed
mdadm: /dev/md3 has been started with 1 drive (out of 5).

Code: Select all

root@fvdw-sta-kirkwood:/ # mdadm --assemble /dev/md0 --force --run --uuid=da0c61be:48b2e856:4c17d85b:cd614625
mdadm: ignoring /dev/sda7 as it reports /dev/sde7 as failed
mdadm: ignoring /dev/sdb7 as it reports /dev/sde7 as failed
mdadm: ignoring /dev/sdc7 as it reports /dev/sde7 as failed
mdadm: /dev/md0 has been started with 1 drive (out of 5).


Code: Select all

root@fvdw-sta-kirkwood:/ # mdadm --assemble /dev/md1 --force --run --uuid=7f16912a:3fca4e8e:3d186b3c:53958f34
mdadm: ignoring /dev/sdc8 as it reports /dev/sda8 as failed
mdadm: ignoring /dev/sde8 as it reports /dev/sda8 as failed
mdadm: /dev/md1 has been started with 2 drives (out of 5).


Code: Select all

root@fvdw-sta-kirkwood:/ # mdadm --assemble /dev/md2 --force --run --uuid=6931896d:781a3429:3d186b3c:53958f34
mdadm: ignoring /dev/sdc9 as it reports /dev/sda9 as failed
mdadm: ignoring /dev/sde9 as it reports /dev/sda9 as failed
mdadm: /dev/md2 has been started with 2 drives (out of 5).

Guessing thats not good. I did umount md4 earlier. Is that why I have the errors?

[Jocko]

Indeed, It is not a good new and explain why the NAS fails to start.

md0 (partitions 7) contains the kernel so without kernel the nas can not start.

Please to post all these outputs

Code: Select all

cat /proc/mdstat

Code: Select all

cat /proc/partitions

Code: Select all

ls -l /dev[abce][5789]

Code: Select all

mdadm --examine /dev/sd[abce]5
mdadm --examine /dev/sd[abce]7
mdadm --examine /dev/sd[abce]8
mdadm --examine /dev/sd[abce]9

Code: Select all

mdadm --detail /dev/md[0123]

[imt17]

Code: Select all

root@fvdw-sta-kirkwood:/ # ls -l /md4cat /proc/mdstat
ls: /md4cat: No such file or directory
-r--r--r--    1 root     root             0 Dec  7 12:18 /proc/mdstat


Code: Select all

root@fvdw-sta-kirkwood:/ # cat /proc/partitions
major minor  #blocks  name

  31        0        500 mtdblock0
  31        1          4 mtdblock1
   8        0  976762584 sdc
   8        1       1024 sdc1
   8        2  974737408 sdc2
   8        3        934 sdc3
   8        4       1024 sdc4
   8        5     256000 sdc5
   8        6       8033 sdc6
   8        7      16065 sdc7
   8        8     843413 sdc8
   8        9     875543 sdc9
   8       10       8033 sdc10
   8       16  976762584 sdb
   8       17       1024 sdb1
   8       18  974737408 sdb2
   8       19        934 sdb3
   8       20       1024 sdb4
   8       21     256000 sdb5
   8       22       8033 sdb6
   8       23      16065 sdb7
   8       24     843413 sdb8
   8       25     875543 sdb9
   8       26       8033 sdb10
   8       32  976762584 sda
   8       33       1024 sda1
   8       34  974738119 sda2
   8       35        934 sda3
   8       36       1024 sda4
   8       37     256000 sda5
   8       38       8033 sda6
   8       39      16065 sda7
   8       40     843413 sda8
   8       41     875543 sda9
   8       42       8033 sda10
   8       48  976762584 sde
   8       49       1024 sde1
   8       50  974737408 sde2
   8       51        934 sde3
   8       52       1024 sde4
   8       53     256000 sde5
   8       54       8033 sde6
   8       55      16065 sde7
   8       56     843413 sde8
   8       57     875543 sde9
   8       58       8033 sde10
   9        0      16000 md0
   9        4 3898947584 md4
   9        3     255936 md3
   9        1     843328 md1
   9        2     875456 md2


Code: Select all

root@fvdw-sta-kirkwood:/ # ls -l /dev[abce][5789]
ls: /dev[abce][5789]: No such file or directory


Code: Select all

root@fvdw-sta-kirkwood:/ # mdadm --examine /dev/sd[abce]5
/dev/sda5:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : b2ab486e:87e9a97c:3d186b3c:53958f34
  Creation Time : Thu Jan  1 00:00:37 1970
     Raid Level : raid1
  Used Dev Size : 255936 (249.98 MiB 262.08 MB)
     Array Size : 255936 (249.98 MiB 262.08 MB)
   Raid Devices : 5
  Total Devices : 2
Preferred Minor : 3

    Update Time : Wed Dec  2 08:26:11 2015
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 3
  Spare Devices : 0
       Checksum : cad0d0d9 - correct
         Events : 178


      Number   Major   Minor   RaidDevice State
this     0       8       21        0      active sync   /dev/sdb5

   0     0       8       21        0      active sync   /dev/sdb5
   1     1       8       37        1      active sync   /dev/sda5
   2     2       0        0        2      faulty removed
   3     3       0        0        3      faulty removed
   4     4       0        0        4      faulty removed
/dev/sdb5:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : b2ab486e:87e9a97c:3d186b3c:53958f34
  Creation Time : Thu Jan  1 00:00:37 1970
     Raid Level : raid1
  Used Dev Size : 255936 (249.98 MiB 262.08 MB)
     Array Size : 255936 (249.98 MiB 262.08 MB)
   Raid Devices : 5
  Total Devices : 2
Preferred Minor : 3

    Update Time : Wed Dec  2 08:26:11 2015
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 3
  Spare Devices : 0
       Checksum : cad0d0eb - correct
         Events : 178


      Number   Major   Minor   RaidDevice State
this     1       8       37        1      active sync   /dev/sda5

   0     0       8       21        0      active sync   /dev/sdb5
   1     1       8       37        1      active sync   /dev/sda5
   2     2       0        0        2      faulty removed
   3     3       0        0        3      faulty removed
   4     4       0        0        4      faulty removed
/dev/sdc5:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : b2ab486e:87e9a97c:3d186b3c:53958f34
  Creation Time : Thu Jan  1 00:00:37 1970
     Raid Level : raid1
  Used Dev Size : 255936 (249.98 MiB 262.08 MB)
     Array Size : 255936 (249.98 MiB 262.08 MB)
   Raid Devices : 5
  Total Devices : 1
Preferred Minor : 3

    Update Time : Tue Dec  1 20:33:22 2015
          State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 3
  Spare Devices : 0
       Checksum : cad02996 - correct
         Events : 174


      Number   Major   Minor   RaidDevice State
this     2       8       21        2      active sync   /dev/sdb5

   0     0       0        0        0      removed
   1     1       0        0        1      faulty removed
   2     2       8       21        2      active sync   /dev/sdb5
   3     3       0        0        3      faulty removed
   4     4       0        0        4      faulty removed
/dev/sde5:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : b2ab486e:87e9a97c:3d186b3c:53958f34
  Creation Time : Thu Jan  1 00:00:37 1970
     Raid Level : raid1
  Used Dev Size : 255936 (249.98 MiB 262.08 MB)
     Array Size : 255936 (249.98 MiB 262.08 MB)
   Raid Devices : 5
  Total Devices : 1
Preferred Minor : 3

    Update Time : Tue Dec  1 22:54:18 2015
          State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 3
  Spare Devices : 0
       Checksum : cad04aae - correct
         Events : 180


      Number   Major   Minor   RaidDevice State
this     4       8       21        4      active sync   /dev/sdb5

   0     0       0        0        0      removed
   1     1       0        0        1      faulty removed
   2     2       0        0        2      faulty removed
   3     3       0        0        3      faulty removed
   4     4       8       21        4      active sync   /dev/sdb5


Code: Select all

root@fvdw-sta-kirkwood:/ # mdadm --examine /dev/sd[abce]7
/dev/sda7:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : da0c61be:48b2e856:4c17d85b:cd614625
  Creation Time : Thu Jan  1 00:00:15 1970
     Raid Level : raid1
  Used Dev Size : 16000 (15.63 MiB 16.38 MB)
     Array Size : 16000 (15.63 MiB 16.38 MB)
   Raid Devices : 5
  Total Devices : 2
Preferred Minor : 0

    Update Time : Wed Dec  2 08:26:10 2015
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 3
  Spare Devices : 0
       Checksum : 3bc2a810 - correct
         Events : 730


      Number   Major   Minor   RaidDevice State
this     0       8       23        0      active sync   /dev/sdb7

   0     0       8       23        0      active sync   /dev/sdb7
   1     1       8       39        1      active sync   /dev/sda7
   2     2       0        0        2      faulty removed
   3     3       0        0        3      faulty removed
   4     4       0        0        4      faulty removed
/dev/sdb7:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : da0c61be:48b2e856:4c17d85b:cd614625
  Creation Time : Thu Jan  1 00:00:15 1970
     Raid Level : raid1
  Used Dev Size : 16000 (15.63 MiB 16.38 MB)
     Array Size : 16000 (15.63 MiB 16.38 MB)
   Raid Devices : 5
  Total Devices : 2
Preferred Minor : 0

    Update Time : Wed Dec  2 08:26:10 2015
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 3
  Spare Devices : 0
       Checksum : 3bc2a822 - correct
         Events : 730


      Number   Major   Minor   RaidDevice State
this     1       8       39        1      active sync   /dev/sda7

   0     0       8       23        0      active sync   /dev/sdb7
   1     1       8       39        1      active sync   /dev/sda7
   2     2       0        0        2      faulty removed
   3     3       0        0        3      faulty removed
   4     4       0        0        4      faulty removed
/dev/sdc7:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : da0c61be:48b2e856:4c17d85b:cd614625
  Creation Time : Thu Jan  1 00:00:15 1970
     Raid Level : raid1
  Used Dev Size : 16000 (15.63 MiB 16.38 MB)
     Array Size : 16000 (15.63 MiB 16.38 MB)
   Raid Devices : 5
  Total Devices : 1
Preferred Minor : 0

    Update Time : Tue Dec  1 20:33:22 2015
          State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 3
  Spare Devices : 0
       Checksum : 3bc200cc - correct
         Events : 726


      Number   Major   Minor   RaidDevice State
this     2       8       23        2      active sync   /dev/sdb7

   0     0       0        0        0      removed
   1     1       0        0        1      faulty removed
   2     2       8       23        2      active sync   /dev/sdb7
   3     3       0        0        3      faulty removed
   4     4       0        0        4      faulty removed
/dev/sde7:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : da0c61be:48b2e856:4c17d85b:cd614625
  Creation Time : Thu Jan  1 00:00:15 1970
     Raid Level : raid1
  Used Dev Size : 16000 (15.63 MiB 16.38 MB)
     Array Size : 16000 (15.63 MiB 16.38 MB)
   Raid Devices : 5
  Total Devices : 1
Preferred Minor : 0

    Update Time : Tue Dec  1 22:54:17 2015
          State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 3
  Spare Devices : 0
       Checksum : 3bc221dd - correct
         Events : 730


      Number   Major   Minor   RaidDevice State
this     3       8       23        3      active sync   /dev/sdb7

   0     0       0        0        0      removed
   1     1       0        0        1      faulty removed
   2     2       0        0        2      faulty removed
   3     3       8       23        3      active sync   /dev/sdb7
   4     4       0        0        4      faulty removed


Code: Select all

root@fvdw-sta-kirkwood:/ # mdadm --examine /dev/sd[abce]8
/dev/sda8:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 7f16912a:3fca4e8e:3d186b3c:53958f34
  Creation Time : Thu Jan  1 00:00:35 1970
     Raid Level : raid1
  Used Dev Size : 843328 (823.70 MiB 863.57 MB)
     Array Size : 843328 (823.70 MiB 863.57 MB)
   Raid Devices : 5
  Total Devices : 2
Preferred Minor : 1

    Update Time : Wed Dec  2 08:26:10 2015
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 3
  Spare Devices : 0
       Checksum : 4f25b5af - correct
         Events : 244


      Number   Major   Minor   RaidDevice State
this     0       8       24        0      active sync   /dev/sdb8

   0     0       8       24        0      active sync   /dev/sdb8
   1     1       0        0        1      faulty removed
   2     2       8       40        2      active sync   /dev/sda8
   3     3       0        0        3      faulty removed
   4     4       0        0        4      faulty removed
/dev/sdb8:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 7f16912a:3fca4e8e:3d186b3c:53958f34
  Creation Time : Thu Jan  1 00:00:35 1970
     Raid Level : raid1
  Used Dev Size : 843328 (823.70 MiB 863.57 MB)
     Array Size : 843328 (823.70 MiB 863.57 MB)
   Raid Devices : 5
  Total Devices : 2
Preferred Minor : 1

    Update Time : Wed Dec  2 08:26:10 2015
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 3
  Spare Devices : 0
       Checksum : 4f25b5c3 - correct
         Events : 244


      Number   Major   Minor   RaidDevice State
this     2       8       40        2      active sync   /dev/sda8

   0     0       8       24        0      active sync   /dev/sdb8
   1     1       0        0        1      faulty removed
   2     2       8       40        2      active sync   /dev/sda8
   3     3       0        0        3      faulty removed
   4     4       0        0        4      faulty removed
/dev/sdc8:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 7f16912a:3fca4e8e:3d186b3c:53958f34
  Creation Time : Thu Jan  1 00:00:35 1970
     Raid Level : raid1
  Used Dev Size : 843328 (823.70 MiB 863.57 MB)
     Array Size : 843328 (823.70 MiB 863.57 MB)
   Raid Devices : 5
  Total Devices : 1
Preferred Minor : 1

    Update Time : Tue Dec  1 20:33:22 2015
          State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 3
  Spare Devices : 0
       Checksum : 4f250e52 - correct
         Events : 229


      Number   Major   Minor   RaidDevice State
this     1       8       24        1      active sync   /dev/sdb8

   0     0       0        0        0      removed
   1     1       8       24        1      active sync   /dev/sdb8
   2     2       0        0        2      faulty removed
   3     3       0        0        3      faulty removed
   4     4       0        0        4      faulty removed
/dev/sde8:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 7f16912a:3fca4e8e:3d186b3c:53958f34
  Creation Time : Thu Jan  1 00:00:35 1970
     Raid Level : raid1
  Used Dev Size : 843328 (823.70 MiB 863.57 MB)
     Array Size : 843328 (823.70 MiB 863.57 MB)
   Raid Devices : 5
  Total Devices : 1
Preferred Minor : 1

    Update Time : Tue Dec  1 22:54:17 2015
          State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 3
  Spare Devices : 0
       Checksum : 4f252f73 - correct
         Events : 239


      Number   Major   Minor   RaidDevice State
this     4       8       24        4      active sync   /dev/sdb8

   0     0       0        0        0      removed
   1     1       0        0        1      faulty removed
   2     2       0        0        2      faulty removed
   3     3       0        0        3      faulty removed
   4     4       8       24        4      active sync   /dev/sdb8


Code: Select all

root@fvdw-sta-kirkwood:/ # mdadm --examine /dev/sd[abce]9
/dev/sda9:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 6931896d:781a3429:3d186b3c:53958f34
  Creation Time : Thu Jan  1 00:00:36 1970
     Raid Level : raid1
  Used Dev Size : 875456 (855.08 MiB 896.47 MB)
     Array Size : 875456 (855.08 MiB 896.47 MB)
   Raid Devices : 5
  Total Devices : 2
Preferred Minor : 2

    Update Time : Wed Dec  2 08:26:10 2015
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 3
  Spare Devices : 0
       Checksum : 71f27ce2 - correct
         Events : 3192540


      Number   Major   Minor   RaidDevice State
this     0       8       25        0      active sync   /dev/sdb9

   0     0       8       25        0      active sync   /dev/sdb9
   1     1       0        0        1      faulty removed
   2     2       8       41        2      active sync   /dev/sda9
   3     3       0        0        3      faulty removed
   4     4       0        0        4      faulty removed
/dev/sdb9:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 6931896d:781a3429:3d186b3c:53958f34
  Creation Time : Thu Jan  1 00:00:36 1970
     Raid Level : raid1
  Used Dev Size : 875456 (855.08 MiB 896.47 MB)
     Array Size : 875456 (855.08 MiB 896.47 MB)
   Raid Devices : 5
  Total Devices : 2
Preferred Minor : 2

    Update Time : Wed Dec  2 08:26:10 2015
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 3
  Spare Devices : 0
       Checksum : 71f27cf6 - correct
         Events : 3192540


      Number   Major   Minor   RaidDevice State
this     2       8       41        2      active sync   /dev/sda9

   0     0       8       25        0      active sync   /dev/sdb9
   1     1       0        0        1      faulty removed
   2     2       8       41        2      active sync   /dev/sda9
   3     3       0        0        3      faulty removed
   4     4       0        0        4      faulty removed
/dev/sdc9:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 6931896d:781a3429:3d186b3c:53958f34
  Creation Time : Thu Jan  1 00:00:36 1970
     Raid Level : raid1
  Used Dev Size : 875456 (855.08 MiB 896.47 MB)
     Array Size : 875456 (855.08 MiB 896.47 MB)
   Raid Devices : 5
  Total Devices : 1
Preferred Minor : 2

    Update Time : Tue Dec  1 20:33:22 2015
          State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 3
  Spare Devices : 0
       Checksum : 71f1d596 - correct
         Events : 3192534


      Number   Major   Minor   RaidDevice State
this     1       8       25        1      active sync   /dev/sdb9

   0     0       0        0        0      removed
   1     1       8       25        1      active sync   /dev/sdb9
   2     2       0        0        2      faulty removed
   3     3       0        0        3      faulty removed
   4     4       0        0        4      faulty removed
/dev/sde9:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 6931896d:781a3429:3d186b3c:53958f34
  Creation Time : Thu Jan  1 00:00:36 1970
     Raid Level : raid1
  Used Dev Size : 875456 (855.08 MiB 896.47 MB)
     Array Size : 875456 (855.08 MiB 896.47 MB)
   Raid Devices : 5
  Total Devices : 1
Preferred Minor : 2

    Update Time : Tue Dec  1 22:54:17 2015
          State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 3
  Spare Devices : 0
       Checksum : 71e4a753 - correct
         Events : 2756398


      Number   Major   Minor   RaidDevice State
this     4       8       25        4      active sync   /dev/sdb9

   0     0       0        0        0      removed
   1     1       0        0        1      faulty removed
   2     2       0        0        2      faulty removed
   3     3       0        0        3      faulty removed
   4     4       8       25        4      active sync   /dev/sdb9


Code: Select all

root@fvdw-sta-kirkwood:/ # mdadm --detail /dev/md[0123]
/dev/md0:
        Version : 0.90
  Creation Time : Thu Jan  1 00:00:15 1970
     Raid Level : raid1
     Array Size : 16000 (15.63 MiB 16.38 MB)
  Used Dev Size : 16000 (15.63 MiB 16.38 MB)
   Raid Devices : 5
  Total Devices : 1
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Tue Dec  1 22:54:17 2015
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           UUID : da0c61be:48b2e856:4c17d85b:cd614625
         Events : 0.730

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       0        0        1      removed
       2       0        0        2      removed
       3       8       55        3      active sync   /dev/sde7
       4       0        0        4      removed
/dev/md1:
        Version : 0.90
  Creation Time : Thu Jan  1 00:00:35 1970
     Raid Level : raid1
     Array Size : 843328 (823.70 MiB 863.57 MB)
  Used Dev Size : 843328 (823.70 MiB 863.57 MB)
   Raid Devices : 5
  Total Devices : 2
Preferred Minor : 1
    Persistence : Superblock is persistent

    Update Time : Wed Dec  2 08:26:10 2015
          State : clean, degraded
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : 7f16912a:3fca4e8e:3d186b3c:53958f34
         Events : 0.244

    Number   Major   Minor   RaidDevice State
       0       8       40        0      active sync   /dev/sda8
       1       0        0        1      removed
       2       8       24        2      active sync   /dev/sdb8
       3       0        0        3      removed
       4       0        0        4      removed
/dev/md2:
        Version : 0.90
  Creation Time : Thu Jan  1 00:00:36 1970
     Raid Level : raid1
     Array Size : 875456 (855.08 MiB 896.47 MB)
  Used Dev Size : 875456 (855.08 MiB 896.47 MB)
   Raid Devices : 5
  Total Devices : 2
Preferred Minor : 2
    Persistence : Superblock is persistent

    Update Time : Wed Dec  2 08:26:10 2015
          State : clean, degraded
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : 6931896d:781a3429:3d186b3c:53958f34
         Events : 0.3192540

    Number   Major   Minor   RaidDevice State
       0       8       41        0      active sync   /dev/sda9
       1       0        0        1      removed
       2       8       25        2      active sync   /dev/sdb9
       3       0        0        3      removed
       4       0        0        4      removed
/dev/md3:
        Version : 0.90
  Creation Time : Thu Jan  1 00:00:37 1970
     Raid Level : raid1
     Array Size : 255936 (249.98 MiB 262.08 MB)
  Used Dev Size : 255936 (249.98 MiB 262.08 MB)
   Raid Devices : 5
  Total Devices : 1
Preferred Minor : 3
    Persistence : Superblock is persistent

    Update Time : Tue Dec  1 22:54:18 2015
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           UUID : b2ab486e:87e9a97c:3d186b3c:53958f34
         Events : 0.180

    Number   Major   Minor   RaidDevice State
       0       0        0        0      removed
       1       0        0        1      removed
       2       0        0        2      removed
       3       0        0        3      removed
       4       8       53        4      active sync   /dev/sde5


[Jocko]

:shocked :shocked :shocked
What mess in your RAID devices! for example in md2 or md3 sdb is the 3th RAID member whereas it should be the 2th.
Think to be happy to have succeeded to backup your data...

First please to repost these output
You made an error when you pasted this command

Code: Select all

cat /proc/mdstat

and try this command (I made a syntax error)

Code: Select all

ls -l /dev/sd[abce][5789]

Currently I see only two explanations:
-either previously you tried several times to start the nas after swapping some disks at each boot(=> and devices does not have the same order in the RAID arrays)
-or in the fvdw-sl console device nodes in /dev are not update.