Issue after upgrading lacie firmware

[Mijzelf]

So, I understand then that disk A is 'kaputt',

At least that is mdadm's judgement. But reading carefully I think it's wrong. The raid header on each member contains the state of the array. I think that is done to prevent you could pull a disk, update the degraded array, and add the (now obsolete) disk again, killing the filesystem.
But that status is written by the raid manager in the kernel. (Of course. The member itself is just a partition) So how can sda2 contain a raid status where sdb2 is missing, while sdb2 has the opposite? I think the raid manager writes this state as soon as the state changes. So if you pull a disk from an assembled array, that disk is marked missing on all other members. So the only way I can think of to get your situation is when the array was assembled with sdb2 missing, and later it was assembled with sda2 missing. The 'Update times' seem to agree:

Code: Select all

    Update Time : Sat May 23 13:42:53 2015
    Update Time : Mon Jun 15 20:46:21 2015
    Update Time : Mon Jun 15 20:46:21 2015
    Update Time : Mon Jun 15 20:46:21 2015

The array was assembled with sdb2 missing at May 23, and was reassembled with sda2 missing at Jun 15. Maybe at June 15 your problems started? When you made any significant change to the filesystem between May 23 and Jun 15, the filesystem is damaged on the new array.
(BTW, the box should have refused to assemble the array with sda2 missing. As sdb2 was missing before, the array was down. Maybe you forced something using the webinterface on June 15?)

mounting /dev/md4 on /mountpoint failed: Input/output error

AFAIK there are 2 ways you can get this error. A hardware error, causing a sector to be unreadable, or an error in the filesystem, pointing to a sector outside the array. A look at dmesg might tell you. When the above story is true, both could happen. The filesystem is likely damaged, and possibly sdb2 is dying.

If sdb2 is dying, you can try to reassemble the array with sdb2 missing. If the filesystem is damaged, you could do the same, but if you made significant changes to the filesystem after Jun 15, this might not help. But it won't hurt to try, of course. As long as you mount the array read only it won't cause further damager. Maybe the array can be assembled this way:

Code: Select all

mdadm --assemble /dev/md4 /dev/sd[acd]2 --force

But I'm not sure if --force will cause mdadm to ignore the status records on the members. Maybe it will be necessary to re-create the array. But that's for later.

If you are tempted to repair the filesystem, it is desireable to generate a low-level copy of the disks (or filesystem) first, as it can bring further damage. Unless you have good backups, of course.

[Jocko]

Hi turmeric,

I am sorry but last days I was busy and I did not read your last posts.

About mount issue, for xfs file system you need to use this mount command: viewtopic.php?f=26&t=1574&start=40#p19163

[turmeric]

Hi Jocko & Mijzelf,

thank you very much for your inputs and replies.

@Mijzelf - it might well be that around May 23rd the firmware update was run... However, booted-up the NAS obviously also later, even though did not work - just thought it might heel itself. Obviously this did not happen :sob

So what I did again is

Code: Select all

mdadm --assemble /dev/md4 /dev/sd[bcd]2
mdadm: /dev/md4 has been started with 3 drives (out of 4).

Then...

Code: Select all

 tftp -l mount.tar -r mount.tar -g 192.168.1.100
mount.tar            100% |*****************************************************************************************************************************************************************************|   199k  0:00:00 ETA
root@fvdw-sta-kirkwood:/ # tar -xvf mount.tar
usr/
usr/sbin/
usr/sbin/mount
usr/lib/
usr/lib/libblkid.so.1
usr/lib/libblkid.so.1.1.0
usr/lib/libblkid.so


And finally

Code: Select all

/usr/sbin/mount -o ro /dev/md4 /mountpoint
mount: /dev/md4: can't read superblock


Again, the output I get here does not sound really encouraging to me....

So, to your statement Mijzelf

If you are tempted to repair the filesystem, it is desireable to generate a low-level copy of the disks (or filesystem) first, as it can bring further damage. Unless you have good backups, of course.

Unfortunately no backup of the NAS - as it was supposed to be my insurace...

So I continue with the description in the link provided by Jocko...

I run

Code: Select all

dmesg
[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 3.14.2 (root@fvdwsl-5big2.local) (gcc version 4.8.1 (GCC) ) #24 PREEMPT Mon May 25 14:17:19 CEST 2015
[    0.000000] CPU: Feroceon 88FR131 [56251311] revision 1 (ARMv5TE), cr=00053177
[    0.000000] CPU: VIVT data cache, VIVT instruction cache
[    0.000000] Machine: LaCie 5Big Network v2
[    0.000000] Ignoring unrecognised tag 0x41000403
[    0.000000] Memory policy: Data cache writeback
[    0.000000] On node 0 totalpages: 131072
[    0.000000] free_area_init_node: node 0, pgdat c0946ac4, node_mem_map dfbfb000
[    0.000000]   Normal zone: 1024 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 131072 pages, LIFO batch:31
[    0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
[    0.000000] pcpu-alloc: [0] 0
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 130048
[    0.000000] Kernel command line: console=ttyS0,115200 netconsole=6666@192.168.1.2/,6666@192.168.1.100/
[    0.000000] PID hash table entries: 2048 (order: 1, 8192 bytes)
[    0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
[    0.000000] Memory: 510020K/524288K available (5780K kernel code, 212K rwdata, 1548K rodata, 1917K init, 232K bss, 14268K reserved)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
[    0.000000]     fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
[    0.000000]     vmalloc : 0xe0800000 - 0xff000000   ( 488 MB)
[    0.000000]     lowmem  : 0xc0000000 - 0xe0000000   ( 512 MB)
[    0.000000]     modules : 0xbf000000 - 0xc0000000   (  16 MB)
[    0.000000]       .text : 0xc0008000 - 0xc073020c   (7329 kB)
[    0.000000]       .init : 0xc0731000 - 0xc0910784   (1918 kB)
[    0.000000]       .data : 0xc0912000 - 0xc0947320   ( 213 kB)
[    0.000000]        .bss : 0xc094732c - 0xc0981568   ( 233 kB)
[    0.000000] Preemptible hierarchical RCU implementation.
[    0.000000] NR_IRQS:114
[    0.000011] sched_clock: 32 bits at 166MHz, resolution 6ns, wraps every 25769803770ns
[    3.455573] Calibrating delay loop... 1196.85 BogoMIPS (lpj=5984256)
[    3.545546] pid_max: default: 32768 minimum: 301
[    3.545684] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    3.545704] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    3.546240] CPU: Testing write buffer coherency: ok
[    3.546547] Setting up static identity map for 0x57d5c0 - 0x57d618
[    3.549207] VFP support v0.3: not present
[    3.549741] xor: measuring software checksum speed
[    3.645574]    arm4regs  :  1089.200 MB/sec
[    3.745539]    8regs     :   654.400 MB/sec
[    3.845531]    32regs    :   905.600 MB/sec
[    3.845548] xor: using function: arm4regs (1089.200 MB/sec)
[    3.845565] pinctrl core: initialized pinctrl subsystem
[    3.845680] NET: Registered protocol family 16
[    3.846032] DMA: preallocated 256 KiB pool for atomic coherent allocations
[    3.848769] Kirkwood: MV88F6281-A0, TCLK=166666667.
[    3.848795] Feroceon L2: Enabling L2
[    3.848833] Feroceon L2: Cache support initialised.
[    3.849928] mv_usb_init 1
[    3.849947] mv_usb_init 2 devnum = 0
[    3.849958] init function mv_usb_init finished
[    3.849967] initial MPP regs: 01112222 00003311 33331100 33003333 00000033 00000000 00000000
[    3.849998]   final MPP regs: 01112222 00003311 33331100 33003333 00000033 00000000 00000000
[    3.850182]  orion-ehci registered
[    3.852020] orion_gpio_is_valid: invalid GPIO 7
[    3.875804] bio: create slab <bio-0> at 0
[    4.046706] raid6: int32x1     95 MB/s
[    4.215696] raid6: int32x2    123 MB/s
[    4.385641] raid6: int32x4    120 MB/s
[    4.555634] raid6: int32x8    128 MB/s
[    4.555644] raid6: using algorithm int32x8 (128 MB/s)
[    4.555652] raid6: using intx1 recovery algorithm
[    4.556484] SCSI subsystem initialized
[    4.556942] libata version 3.00 loaded.
[    4.557811] usbcore: registered new interface driver usbfs
[    4.557992] usbcore: registered new interface driver hub
[    4.558186] usbcore: registered new device driver usb
[    4.560670] cfg80211: Calling CRDA to update world regulatory domain
[    4.560915] Switched to clocksource orion_clocksource
[    4.563201] NET: Registered protocol family 2
[    4.563672] TCP established hash table entries: 4096 (order: 2, 16384 bytes)
[    4.563732] TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
[    4.563784] TCP: Hash tables configured (established 4096 bind 4096)
[    4.563837] TCP: reno registered
[    4.563850] UDP hash table entries: 256 (order: 0, 4096 bytes)
[    4.563872] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[    4.564035] NET: Registered protocol family 1
[    4.564310] RPC: Registered named UNIX socket transport module.
[    4.564320] RPC: Registered udp transport module.
[    4.564328] RPC: Registered tcp transport module.
[    4.564335] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    4.564346] PCI: CLS 0 bytes, default 32
[    4.709780] NetWinder Floating Point Emulator V0.97 (double precision)
[    4.709905] gpio-usb-probe-4b err=0
[    4.709921] gpio-usb gpio-usb: USB GPIO's initialized
[    4.711392] futex hash table entries: 256 (order: -1, 3072 bytes)
[    4.809797] VFS: Disk quotas dquot_6.5.2
[    4.809847] Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[    4.810288] NFS: Registering the id_resolver key type
[    4.810336] Key type id_resolver registered
[    4.810346] Key type id_legacy registered
[    4.810363] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    4.810377] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[    4.810470] NTFS driver 2.1.30 [Flags: R/W].
[    4.810506] jffs2: version 2.2. (NAND) (SUMMARY)  © 2001-2006 Red Hat, Inc.
[    4.810650] fuse init (API version 7.22)
[    4.811213] SGI XFS with ACLs, security attributes, realtime, large block/inode numbers, no debug enabled
[    4.811623] msgmni has been set to 996
[    4.812736] async_tx: api initialized (async)
[    4.812936] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    4.812948] io scheduler noop registered
[    4.812957] io scheduler deadline registered
[    4.812976] io scheduler cfq registered (default)
[    4.813358] mv_xor mv_xor.0: Marvell shared XOR driver
[    4.845629] mv_xor mv_xor.0: Marvell XOR: ( xor cpy )
[    4.885551] mv_xor mv_xor.0: Marvell XOR: ( xor cpy )
[    4.885796] mv_xor mv_xor.1: Marvell shared XOR driver
[    4.925551] mv_xor mv_xor.1: Marvell XOR: ( xor cpy )
[    4.965550] mv_xor mv_xor.1: Marvell XOR: ( xor cpy )
[    4.971415] Serial: 8250/16550 driver, 2 ports, IRQ sharing disabled
[    4.992510] serial8250.0: ttyS0 at MMIO 0xf1012000 (irq = 33, base_baud = 10416666) is a 16550A
[    5.540252] console [ttyS0] enabled
[    5.553096] brd: module loaded
[    5.560767] loop: module loaded
[    5.565606] sata_mv sata_mv.0: version 1.28
[    5.565670] sata_mv sata_mv.0: cannot get optional clkdev
[    5.571141] sata_mv sata_mv.0: slots 32 ports 2
[    5.578913] scsi0 : sata_mv
[    5.582252] scsi1 : sata_mv
[    5.585444] ata1: SATA max UDMA/133 irq 21
[    5.589782] ata2: SATA max UDMA/133 irq 21
[    5.596483] m25p80 spi0.0: mx25l4005a (512 Kbytes)
[    5.601429] Creating 2 MTD partitions on "spi_flash":
[    5.606601] 0x000000000000-0x00000007d000 : "u-boot"
[    5.612716] 0x00000007e000-0x00000007f000 : "u-boot-env"
[    5.619384] tun: Universal TUN/TAP device driver, 1.6
[    5.624412] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[    5.630841] ipddp.c:v0.01 8/28/97 Bradford W. Johnson <johns393@maroon.tc.umn.edu>
[    5.639103] ipddp0: Appletalk-IP Encap. mode by Bradford W. Johnson <johns393@maroon.tc.umn.edu>
[    5.655394] libphy: orion_mdio_bus: probed
[    5.659684] mv643xx_eth: MV-643xx 10/100/1000 ethernet driver version 1.4
[    5.747197] mv643xx_eth_port mv643xx_eth_port.0 eth0: port 0 with MAC address 00:d0:4b:8f:15:d4
[    5.837226] mv643xx_eth_port mv643xx_eth_port.1 eth1: port 0 with MAC address 00:d0:4b:8f:9e:38
[    5.846054] PPP generic driver version 2.4.2
[    5.850622] PPP Deflate Compression module registered
[    5.855725] PPP MPPE Compression module registered
[    5.860492] NET: Registered protocol family 24
[    5.865096] usbcore: registered new interface driver rtl8187
[    5.870786] pegasus: v0.9.3 (2013/04/25), Pegasus/Pegasus II USB Ethernet driver
[    5.878376] usbcore: registered new interface driver pegasus
[    5.884205] usbcore: registered new interface driver asix
[    5.889751] usbcore: registered new interface driver ax88179_178a
[    5.896008] usbcore: registered new interface driver cdc_ether
[    5.901988] usbcore: registered new interface driver net1080
[    5.907821] usbcore: registered new interface driver cdc_subset
[    5.913886] usbcore: registered new interface driver zaurus
[    5.919665] usbcore: registered new interface driver ipheth
[    5.925415] usbcore: registered new interface driver cdc_ncm
[    5.932062] aoe: cannot create debugfs directory
[    5.936922] aoe: AoE v85 initialised.
[    5.941300] rtc-mv rtc-mv: rtc core: registered rtc-mv as rtc0
[    5.947349] i2c /dev entries driver
[    5.951809] at24 0-0050: 512 byte 24c04 EEPROM, writable, 16 bytes/write
[    5.974358] md: linear personality registered for level -1
[    5.979913] md: raid0 personality registered for level 0
[    5.985202] md: raid1 personality registered for level 1
[    5.990559] md: raid10 personality registered for level 10
[    5.996188] md: raid6 personality registered for level 6
[    6.001471] md: raid5 personality registered for level 5
[    6.006790] md: raid4 personality registered for level 4
[    6.012271] device-mapper: ioctl: 4.27.0-ioctl (2013-10-30) initialised: dm-devel@redhat.com
[    6.020918] device-mapper: multipath: version 1.6.0 loaded
[    6.026421] device-mapper: multipath round-robin: version 1.0.0 loaded
[    6.032921] device-mapper: multipath queue-length: version 0.1.0 loaded
[    6.039530] device-mapper: multipath service-time: version 0.2.0 loaded
[    6.046192] device-mapper: dm-log-userspace: version 1.3.0 loaded
[    6.052258] device-mapper: raid: Loading target version 1.5.2
[    6.062139] ipip: IPv4 over IPv4 tunneling driver
[    6.067626] TCP: cubic registered
[    6.070974] NET: Registered protocol family 10
[    6.076393] sit: IPv6 over IPv4 tunneling driver
[    6.081850] NET: Registered protocol family 17
[    6.086501] NET: Registered protocol family 5
[    6.090944] Key type dns_resolver registered
[    6.095561] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl F300)
[    6.103047] ata1.15: Port Multiplier 1.2, 0x11ab:0x4140 r193, 4 ports, feat 0x5/0x1f
[    6.111132] netpoll: netconsole: local port 6666
[    6.115784] netpoll: netconsole: local IPv4 address 192.168.1.2
[    6.121679] netpoll: netconsole: interface 'eth0'
[    6.127769] netpoll: netconsole: remote port 6666
[    6.132455] netpoll: netconsole: remote IPv4 address 192.168.1.100
[    6.140292] netpoll: netconsole: remote ethernet address ff:ff:ff:ff:ff:ff
[    6.148126] netpoll: netconsole: device eth0 not up yet, forcing it
[    6.158552] ata1.00: hard resetting link
[    6.236311] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    6.506281] ata1.01: hard resetting link
[    7.015776] ata1.02: hard resetting link
[    7.525768] ata1.03: hard resetting link
[    8.075557] ata1.01: ATA-8: ST2000DL001-9VT156, CC41, max UDMA/133
[    8.081710] ata1.01: 3907029168 sectors, multi 0: LBA48
[    8.145558] ata1.01: configured for UDMA/133
[    8.185550] ata1.02: ATA-8: ST2000DL001-9VT156, CC41, max UDMA/133
[    8.191705] ata1.02: 3907029168 sectors, multi 0: LBA48
[    8.255560] ata1.02: configured for UDMA/133
[    8.295552] ata1.03: ATA-8: ST2000DL001-9VT156, CC41, max UDMA/133
[    8.301710] ata1.03: 3907029168 sectors, multi 0: LBA48
[    8.365557] ata1.03: configured for UDMA/133
[    8.369987] ata1: EH complete
[    8.385791] scsi 0:1:0:0: Direct-Access     ATA      ST2000DL001-9VT1 CC41 PQ: 0 ANSI: 5
[    8.394349] fvdw1 disk index before get_sd_name = 0
[    8.399258] device-name= sata_mv
[    8.402467] fvdw2 disk index after get_sd_name = 2
[    8.407265] fvdw3 prefix[0]=s  prefix[1]=d
[    8.411339] fvdw4 diskname = sdc
[    8.415031] sd 0:1:0:0: [sdc] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
[    8.423133] sd 0:1:0:0: Attached scsi generic sg0 type 0
[    8.428514] sd 0:1:0:0: [sdc] 4096-byte physical blocks
[    8.434214] scsi 0:2:0:0: Direct-Access     ATA      ST2000DL001-9VT1 CC41 PQ: 0 ANSI: 5
[    8.442557] sd 0:1:0:0: [sdc] Write Protect is off
[    8.447358] sd 0:1:0:0: [sdc] Mode Sense: 00 3a 00 00
[    8.447468] sd 0:1:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    8.456967] fvdw1 disk index before get_sd_name = 1
[    8.461822] device-name= sata_mv
[    8.465027] fvdw2 disk index after get_sd_name = 1
[    8.470777] fvdw3 prefix[0]=s  prefix[1]=d
[    8.474853] fvdw4 diskname = sdb
[    8.478608] sd 0:2:0:0: [sdb] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
[    8.486860] sd 0:2:0:0: Attached scsi generic sg1 type 0
[    8.492318] sd 0:2:0:0: [sdb] 4096-byte physical blocks
[    8.497976] scsi 0:3:0:0: Direct-Access     ATA      ST2000DL001-9VT1 CC41 PQ: 0 ANSI: 5
[    8.506393] sd 0:2:0:0: [sdb] Write Protect is off
[    8.511244] sd 0:2:0:0: [sdb] Mode Sense: 00 3a 00 00
[    8.511361] sd 0:2:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    8.520978] fvdw1 disk index before get_sd_name = 2
[    8.525999] device-name= sata_mv
[    8.529215] fvdw2 disk index after get_sd_name = 0
[    8.533983] fvdw3 prefix[0]=s  prefix[1]=d
[    8.538110]  sdc: sdc1 < sdc5 sdc6 sdc7 sdc8 sdc9 sdc10 > sdc2
[    8.546423] fvdw4 diskname = sda
[    8.550316] sd 0:3:0:0: [sda] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
[    8.558655] sd 0:3:0:0: [sda] 4096-byte physical blocks
[    8.564003] sd 0:3:0:0: Attached scsi generic sg2 type 0
[    8.570743] sd 0:1:0:0: [sdc] Attached SCSI disk
[    8.575801] sd 0:3:0:0: [sda] Write Protect is off
[    8.580669] sd 0:3:0:0: [sda] Mode Sense: 00 3a 00 00
[    8.580781] sd 0:3:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    8.634909]  sdb: sdb1 < sdb5 sdb6 sdb7 sdb8 sdb9 sdb10 > sdb2
[    8.643546] sd 0:2:0:0: [sdb] Attached SCSI disk
[    8.669300]  sda: sda1 < sda5 sda6 sda7 sda8 sda9 sda10 > sda2
[    8.677837] sd 0:3:0:0: [sda] Attached SCSI disk
[    8.750649] mv643xx_eth_port mv643xx_eth_port.0 eth0: link up, 1000 Mb/s, full duplex, flow control disabled
[    8.760473] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    8.769631] console [netcon0] enabled
[    8.773290] netconsole: network logging started
[    8.778333] input: gpio-keys as /devices/platform/gpio-keys/input/input0
[    8.785412] rtc-mv rtc-mv: setting system clock to 2015-06-19 23:08:03 UTC (1434755283)
[    9.085528] ata2: SATA link up 3.0 Gbps (SStatus 123 SControl F300)
[    9.092470] ata2.15: Port Multiplier 1.2, 0x11ab:0x4140 r193, 4 ports, feat 0x5/0x1f
[    9.108033] ata2.00: hard resetting link
[    9.456255] ata2.01: hard resetting link
[    9.965777] ata2.02: hard resetting link
[   10.475771] ata2.03: hard resetting link
[   10.865554] ata2.01: ATA-8: ST2000DL001-9VT156, CC41, max UDMA/133
[   10.871726] ata2.01: 3907029168 sectors, multi 0: LBA48
[   10.935559] ata2.01: configured for UDMA/133
[   10.975550] ata2.02: ATA-8: ST2000DL001-9VT156, CC41, max UDMA/133
[   10.981727] ata2.02: 3907029168 sectors, multi 0: LBA48
[   11.045558] ata2.02: configured for UDMA/133
[   11.050007] ata2: EH complete
[   11.065786] scsi 1:1:0:0: Direct-Access     ATA      ST2000DL001-9VT1 CC41 PQ: 0 ANSI: 5
[   11.074387] fvdw1 disk index before get_sd_name = 3
[   11.079319] device-name= sata_mv
[   11.082552] fvdw2 disk index after get_sd_name = 4
[   11.087377] fvdw3 prefix[0]=s  prefix[1]=d
[   11.091467] fvdw4 diskname = sde
[   11.095190] sd 1:1:0:0: [sde] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
[   11.103276] sd 1:1:0:0: Attached scsi generic sg3 type 0
[   11.108658] sd 1:1:0:0: [sde] 4096-byte physical blocks
[   11.114370] scsi 1:2:0:0: Direct-Access     ATA      ST2000DL001-9VT1 CC41 PQ: 0 ANSI: 5
[   11.122716] sd 1:1:0:0: [sde] Write Protect is off
[   11.127680] sd 1:1:0:0: [sde] Mode Sense: 00 3a 00 00
[   11.128069] sd 1:1:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   11.137244] fvdw1 disk index before get_sd_name = 4
[   11.142116] device-name= sata_mv
[   11.145343] fvdw2 disk index after get_sd_name = 3
[   11.151133] fvdw3 prefix[0]=s  prefix[1]=d
[   11.155237] fvdw4 diskname = sdd
[   11.159002] sd 1:2:0:0: [sdd] 3907029168 512-byte logical blocks: (2.00 TB/1.81 TiB)
[   11.167330] sd 1:2:0:0: Attached scsi generic sg4 type 0
[   11.172881] sd 1:2:0:0: [sdd] 4096-byte physical blocks
[   11.178522] sd 1:2:0:0: [sdd] Write Protect is off
[   11.183408] sd 1:2:0:0: [sdd] Mode Sense: 00 3a 00 00
[   11.183517] sd 1:2:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   11.221839]  sde: sde1 < sde5 sde6 sde7 sde8 sde9 sde10 > sde2
[   11.230536] sd 1:1:0:0: [sde] Attached SCSI disk
[   11.262165]  sdd: sdd1 < sdd5 sdd6 sdd7 sdd8 sdd9 sdd10 > sdd2
[   11.270735] sd 1:2:0:0: [sdd] Attached SCSI disk
[   11.280756] Freeing unused kernel memory: 1916K (c0731000 - c0910000)
[   15.363993] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[   15.374226] ehci-orion: EHCI orion driver
[   15.378447] orion-ehci orion-ehci.0: EHCI Host Controller
[   15.383867] orion-ehci orion-ehci.0: new USB bus registered, assigned bus number 1
[   15.391588] orion-ehci orion-ehci.0: irq 19, io mem 0xf1050000
[   15.415530] orion-ehci orion-ehci.0: USB 2.0 started, EHCI 1.00
[   15.421575] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[   15.428379] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   15.435604] usb usb1: Product: EHCI Host Controller
[   15.440475] usb usb1: Manufacturer: Linux 3.14.2 ehci_hcd
[   15.445884] usb usb1: SerialNumber: orion-ehci.0
[   15.451204] hub 1-0:1.0: USB hub found
[   15.455010] hub 1-0:1.0: 1 port detected
[   15.465570] usbcore: registered new interface driver usb-storage
[   15.775694] usb 1-1: new high-speed USB device number 2 using orion-ehci
[   15.926617] usb 1-1: New USB device found, idVendor=0424, idProduct=2512
[   15.933327] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[   15.950141] hub 1-1:1.0: USB hub found
[   15.955225] hub 1-1:1.0: 2 ports detected
[  160.836047] random: nonblocking pool is initialized
[  758.344377] md: md4 stopped.
[  758.363281] md: bind<sdc2>
[  758.366582] md: bind<sdd2>
[  758.369687] md: bind<sdb2>
[  758.375476] md/raid:md4: device sdb2 operational as raid disk 1
[  758.381489] md/raid:md4: device sdd2 operational as raid disk 3
[  758.387439] md/raid:md4: device sdc2 operational as raid disk 2
[  758.394596] md/raid:md4: allocated 0kB
[  758.398510] md/raid:md4: raid level 5 active with 3 out of 4 devices, algorithm 2
[  758.406012] RAID conf printout:
[  758.406022]  --- level:5 rd:4 wd:3
[  758.406032]  disk 1, o:1, dev:sdb2
[  758.406040]  disk 2, o:1, dev:sdc2
[  758.406048]  disk 3, o:1, dev:sdd2
[  758.406223] md4: detected capacity change from 0 to 5995018321920
[  874.757635]  md4: unknown partition table
[  874.847704] XFS (md4): correcting sb_features alignment problem
[  874.856209] XFS (md4): Mounting Filesystem
[  874.860574] attempt to access beyond end of device
[  874.865367] md4: rw=32, want=236328163009256, limit=11709020160
[  874.871344] XFS (md4): metadata I/O error: block 0xd6f07034e2e0 ("xlog_bread_noalign") error 5 numblks 8
[  874.880832] XFS (md4): empty log check failed
[  874.885189] XFS (md4): log mount/recovery failed: error 5
[  874.890758] XFS (md4): log mount failed


So it starts getting interesting only at the end... (at least from what I understand)...

Code: Select all

[  874.757635]  md4: unknown partition table
[  874.847704] XFS (md4): correcting sb_features alignment problem
[  874.856209] XFS (md4): Mounting Filesystem
[  874.860574] attempt to access beyond end of device
[  874.865367] md4: rw=32, want=236328163009256, limit=11709020160
[  874.871344] XFS (md4): metadata I/O error: block 0xd6f07034e2e0 ("xlog_bread_noalign") error 5 numblks 8
[  874.880832] XFS (md4): empty log check failed
[  874.885189] XFS (md4): log mount/recovery failed: error 5
[  874.890758] XFS (md4): log mount failed

What I just read out of it - problem with the filesystem... What do I do with that? Start feeling slightly worried...

[Jocko]

Hi turmeric,

It seems that you have an issue with your fs on md4.

So try to restore it by using xfs_repair. You need to download it here viewtopic.php?f=7&t=1271&p=18801#p18801

[turmeric]

Hi Jocko,

you're testing my skills here ;) ... While I'm even proud that somehow I manage to upload the script, I do not manage to execute due to missing shared library...

Code: Select all

root@fvdw-sta-kirkwood:/ # tftp -l /sbin/xfs_repair -r xfs_repair -g 192.168.1.100
xfs_repair           100% |*****************************************************************************************************************************************************************************|   622k  0:00:00 ETA
root@fvdw-sta-kirkwood:/ # chmod 755 /sbin/xfs_repair
root@fvdw-sta-kirkwood:/ # xfs_repair -n
xfs_repair: error while loading shared libraries: librt.so.1: cannot open shared object file: No such file or directory


How can I bring the script to work?

Furthermore, I read that by running xfs_repair -n I can do kind of a check.

For my better understanding:
- once executing xfs-repair for good - what is the risk that I just loose my data?
- what command would I actually have to enter to execute?

Many thanks for your help here. This is really most appreciated

[fvdw]

you also need this package that contains the shared libs you need

Unpack the zip file, put the tar archive in the tftp folder of the fvdw-sl console and upload the tar archive (glibc-mini-mkfs.xfs-25feb14.tar) to the nas using tftp

Code: Select all

cd /
tftp -r glibc-mini-mkfs.xfs-25feb14.tar -r glibc-mini-mkfs.xfs-25feb14.tar -g ip-pc

(replace ip-pc by actual ip address of the pc running the console)

and extract it

Code: Select all

tar -xvf glibc-mini-mkfs.xfs-25feb14.tar

,
now xfs_repair should work

[turmeric]

Thanks fvdw - that did the trick

Code: Select all

root@fvdw-sta-kirkwood:/ # cd /
root@fvdw-sta-kirkwood:/ # tftp -r glibc-mini-mkfs.xfs-25feb14.tar -r glibc-mini-mkfs.xfs-25feb14.tar -g 192.168.1.100
glibc-mini-mkfs.xfs- 100% |*****************************************************************************************************************************************************************************|  2519k  0:00:00 ETA
root@fvdw-sta-kirkwood:/ # tar -xvf glibc-mini-mkfs.xfs-25feb14.tar
./
./usr/
./usr/lib/
./usr/lib/libuuid.so
./usr/lib/libuuid.so.1.3.0
./usr/lib/libgcc_s.so.1
./usr/lib/libuuid.so.1
./bin/
./sbin/
./sbin/mkfs.xfs
./lib/
./lib/libutil.so.1
./lib/libc-2.17.so
./lib/librt.so.1
./lib/librt-2.17.so
./lib/libc.so.6
./lib/libm-2.17.so
./lib/libpthread.so.0
./lib/ld-linux.so.3
./lib/ld-2.17.so
./lib/libutil-2.17.so
./lib/libpthread-2.17.so
./lib/libm.so.6
root@fvdw-sta-kirkwood:/ # xfs_repair -n
Usage: xfs_repair [options] device


Question - what exact command do I need to enter when it refers to device?

[fvdw]

as device you should enter /dev/md4 (if md4 is your raid array)

It should not be mounted when running xfs_repair
I think with the -n option it will only check it and will do no changes

[turmeric]

Well knowing that this is only analyzing, but not repairing, I run

Code: Select all

 mdadm --assemble /dev/md4 /dev/sd[bcd]2
mdadm: /dev/md4 has been started with 3 drives (out of 4).
root@fvdw-sta-kirkwood:/ # xfs_repair -n /dev/md4


I get the following:

Code: Select all

Phase 1 - find and verify superblock...
bad primary superblock - bad sector size !!!

attempting to find secondary superblock...
(...)found candidate secondary superblock...
error reading superblock 175 -- seek to offset 6012954214400 failed
unable to verify superblock, continuing...
.found candidate secondary superblock...
error reading superblock 175 -- seek to offset 6012954214400 failed
unable to verify superblock, continuing...

(...)found candidate secondary superblock...
error reading superblock 175 -- seek to offset 6012942745600 failed
unable to verify superblock, continuing...
found candidate secondary superblock...
error reading superblock 175 -- seek to offset 6012942745600 failed
unable to verify superblock, continuing...

(...) found candidate secondary superblock...
verified secondary superblock...
would write modified primary superblock
Primary superblock would have been modified.
Cannot proceed further in no_modify mode.
Exiting now.



So next step is now that I would run just - or any additional options required?

Code: Select all

 xfs_repair /dev/md4

When I do this, what are the odds that I loose my data?

[fvdw]

About risk of loss of data when you run xfs_repair and let it repair the errors
To be honest I don't know, I never used xfs.
Our firmware used ext3.

But I think that with these errors your data will also not be accesible (or partly)
You could do some investigation on the internet what risks there are or hope that the other experts here comment on your question.

sorry that I can not help you better