Another LaCie 5big Network 2 problem

[Mijzelf]

It's not over yet. A 'failed raid member' is no more than a raid member which isn't 'known to be good'. But the problem could be as little as a 'last changed' stamp which was never written to the raid header. If you didn't copy gigabytes of data to the array, while this member was dropped, the array can be re-created, with possibly only minor filesystem inconsistencies, or maybe completely sane.

The command to do so is:

Code: Select all

mdadm --create --assume-clean --metadata=1.0 --chunk=512 --level=raid5  --layout=left-symmetric --raid-devices=5 /dev/md0 /dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2 /dev/sde2

It's *essential* here that the sequence of the disks is right, in the command line. You've got only 4 disks, so you have to exchange the missing disks by the missing keyword. Assuming sdd is missing, this would become

Code: Select all

mdadm --create --assume-clean --metadata=1.0 --chunk=512 --level=raid5  --layout=left-symmetric --raid-devices=5 /dev/md0 /dev/sda2 /dev/sdb2 /dev/sdc2 missing /dev/sde2

--assume-clean tells mdadm to assume the disks are already 'clean', which means it won't do an attempt to generate the parity data.

You'll have to examine each member to find it's role in the array.

mdadm --examine /dev/sdc2
/dev/sdc2:
<snip>
Device Role : Active device 0

This one is the 'zero'th' of the array, or in normal speak the first one.

(BTW, this is from your posting here, and now I realize that sdc2 is supposed to be device 2. Your posting shows sdc2-0 sdb2-1 sda2-2. So even in the Lacie you'll have to ask the exact role of the members, before re-creating the array.)

@fvdw: is /dev populated automatically, in the kernel used by tenhouse? If yes, then Lacie sells boxes with pre-mangled raid arrays. Nice for data recovery.

/Edit: Do I see the book 'Applying UML and patterns' there? Nice reading before sleep.

[tenhouse]

Good morning

Ok I did the "--examin" command and what I got is this

sda2= 2
sdb2 = 1
sdc2 = 0
sdd2 = 4

So I guess "3" is missing.
What I don't get how the command should be since I don't have an sde2 , but I guess like this:

Code: Select all

mdadm --create --assume-clean --metadata=1.0 --chunk=512 --level=raid5  --layout=left-symmetric --raid-devices=5 /dev/md0 /dev/sdc2 /dev/sdb2 /dev/sda2 missing /dev/sdd2


Is this correct?

edit: you see correct - it's not mine but I read this kind of stuff too ;)

[Mijzelf]

Is this correct?

Yes. If you didn't power off the system in between, you can apply it. (I don't know if the disks are necessarily found in the same sequence, each boot.)

[tenhouse]

Done.

Code: Select all

Gentoo-20121221 ~ # mdadm --create --assume-clean --metadata=1.0 --chunk=512 --level=raid5  --layout=left-symmetric --raid-devices=5 /dev/md0 /dev/sdc2 /dev/sdb2 /dev/sda2 missing /dev/sdd2
mdadm: /dev/sdc2 appears to be part of a raid array:
    level=raid5 devices=5 ctime=Tue Jul 10 15:36:48 2012
mdadm: /dev/sdb2 appears to be part of a raid array:
    level=raid5 devices=5 ctime=Tue Jul 10 15:36:48 2012
mdadm: /dev/sda2 appears to be part of a raid array:
    level=raid5 devices=5 ctime=Tue Jul 10 15:36:48 2012
mdadm: /dev/sdd2 appears to be part of a raid array:
    level=raid5 devices=5 ctime=Tue Jul 10 15:36:48 2012
Continue creating array? y
mdadm: array /dev/md0 started.
Gentoo-20121221 ~ # mkdir /tmp/nas
Gentoo-20121221 ~ # mount -o ro /dev/md0 /tmp/nas
mount: mount /dev/md0 on /tmp/nas failed: Structure needs cleaning



edit:
and from /var/log/messages

Code: Select all

Sep 12 09:03:39 Gentoo-20121221 kernel: [ 3423.748064] md: bind<sdc2>               
Sep 12 09:03:39 Gentoo-20121221 kernel: [ 3423.764884] md: bind<sdb2>                                                                   
Sep 12 09:03:39 Gentoo-20121221 kernel: [ 3423.768797] md: bind<sda2>                       
Sep 12 09:03:39 Gentoo-20121221 kernel: [ 3423.778039] md: bind<sdd2>                                               
Sep 12 09:03:39 Gentoo-20121221 kernel: [ 3423.779514] bio: create slab <bio-1> at 1                                 
Sep 12 09:03:39 Gentoo-20121221 kernel: [ 3423.779533] md/raid:md0: device sdd2 operational as raid disk 4           
Sep 12 09:03:39 Gentoo-20121221 kernel: [ 3423.779535] md/raid:md0: device sda2 operational as raid disk 2         
Sep 12 09:03:39 Gentoo-20121221 kernel: [ 3423.779537] md/raid:md0: device sdb2 operational as raid disk 1             
Sep 12 09:03:39 Gentoo-20121221 kernel: [ 3423.779539] md/raid:md0: device sdc2 operational as raid disk 0           
Sep 12 09:03:39 Gentoo-20121221 kernel: [ 3423.779971] md/raid:md0: allocated 5277kB                                 
Sep 12 09:03:39 Gentoo-20121221 kernel: [ 3423.780023] md/raid:md0: raid level 5 active with 4 out of 5 devices, algorithm 2
Sep 12 09:03:39 Gentoo-20121221 kernel: [ 3423.780025] RAID conf printout:                                           
Sep 12 09:03:39 Gentoo-20121221 kernel: [ 3423.780027]  --- level:5 rd:5 wd:4                                         
Sep 12 09:03:39 Gentoo-20121221 kernel: [ 3423.780030]  disk 0, o:1, dev:sdc2                                         
Sep 12 09:03:39 Gentoo-20121221 kernel: [ 3423.780031]  disk 1, o:1, dev:sdb2                                       
Sep 12 09:03:39 Gentoo-20121221 kernel: [ 3423.780033]  disk 2, o:1, dev:sda2                                                   
Sep 12 09:03:39 Gentoo-20121221 kernel: [ 3423.780035]  disk 4, o:1, dev:sdd2
Sep 12 09:03:39 Gentoo-20121221 kernel: [ 3423.780081] md0: detected capacity change from 0 to 11994073661440                       
Sep 12 09:03:39 Gentoo-20121221 kernel: [ 3423.782296]  md0: unknown partition table                               
Sep 12 09:04:22 Gentoo-20121221 kernel: [ 3466.631683] XFS (md0): Mounting Filesystem                 
Sep 12 09:04:22 Gentoo-20121221 kernel: [ 3466.759153] XFS (md0): log has mismatched uuid - can't recover
Sep 12 09:04:22 Gentoo-20121221 kernel: [ 3466.759166] XFS (md0): failed to find log head
Sep 12 09:04:22 Gentoo-20121221 kernel: [ 3466.759168] XFS (md0): log mount/recovery failed: error 117
Sep 12 09:04:22 Gentoo-20121221 kernel: [ 3466.759211] XFS (md0): log mount failed


[Mijzelf]

That is an xfs error. So there are at least 'some' filesystem problems. You can have a look with dmesg, to see if it gives any further information, and you can cross your fingers and look if xfs_repair can do it's job.
Or, depending on the nature and the irreplaceability of the data, you can let PhotoRec do low-level recovery on the assembled array before attempting to repair it.

/edit:

md0: detected capacity change from 0 to 11994073661440

I like that message. You can't fool the kernel. It's detects everything, no matter how trivial. :mrgreen:

Hm, it seems the log is corrupted, which means the volume can't roll back the last changes, to make the filesystem consistent. So a full fsck is needed.

[tenhouse]

I did

Code: Select all

xfs_repair -L /dev/md0


because of the corrupt log. Now I'm waiting. Waiting for my doom most likely.

edit:

omfg. I'ts done. I could mount it I see my folders (just checked 2 of them).
I start copying away right now. And cross my finger once more. If this really works....

edit2:
I think I best format my external hds with ext3 first

Wow nice after I plugged in my external usb drive it looks like gentoo crashed. Thanks

[Mijzelf]

Congratulations!

Wow nice after I plugged in my external usb drive it looks like gentoo crashed. Thanks

You mounted your array read only, I hope? Next time you'll only have to assemble it, not create. (Well, if you mounted it read-write, you still have only to assemble it, but maybe you'll have to repair the filesystem again.)

[tenhouse]

To be honest I don't remeber how I mounted. Maybe I did it with what you said or maybe I did it stupid (my way).
Anyway the gentoo didn't crash... and it looks like one of my two external HDs is broken (sure why not), because the other is working perfectly smooth - even with NTFS as FS.
I'm copying (with rsync) the most important stuff away right now, so far it looks like it's gonna work.
But you really shouldn't congratulate just yet... I'm only happy when I suceeded in copying everything (important) away.
Will post progress here :)

/edit: yeah gentoo completley froze... anyway I rebooted and did the assemble again (the disks keep the same order by the way) and I could it mount (as readonly) without an error this time :) Backup job is running andrunning... but takes it's time with USB 2.0 :P I don't care as long as it works... looks pretty ok until now.

[fvdw]

@fvdw: is /dev populated automatically, in the kernel used by tenhouse? If yes, then Lacie sells boxes with pre-mangled raid arrays. Nice for data recovery.

in the 164 standalone kernel there is no automatic update yet for the dev nodes.
So in case of the 5big you need to updtae them using udevstart (upload by tftp) it will be include in the next (linux 3.9.5) standalone kernel

[tenhouse]

Hello Guys

I'm not finished copying (have to buy a new external HD first, since one is broken) but it looks like everything will be fine - ~3 TB of Data is already backed up!

Thank you guys so much! Especially to you Mijzelf. Do you have a "Donate-Button" for yourself or something smililar where I could "spend you a couple of beers" ;) ?

By the way for everyone who reads this thread and wonders whats an easy way to backup, I recommend using rsync:

Code: Select all

rsync -Pavzh /dir/to/mounted/raid /target/dir


With this command you can always resume copying if you ever have an system-error or something similar.

Greetings!