Failed Drive Replacement

Failed Drive Replacement

Postby hvymetal86 » Fri Jun 09, 2017 10:54 pm

I've been using the fvdw firmware on my 5big vs2 for over a year now and I love it. I've kept it up to date with patches too. Today the front indicator started blinking red, logged on and found that one of the 5 drives has died and is no longer detected (sdc). Its blinking red correspondingly on the center drive as well.

I know when I was setting up the 5 disk RAID 5 array that I had to partition each drive correctly as part of the process with specific partition details. My thread here was for getting help with that: viewtopic.php?t=2405

I had a few questions before replacing the drive now.
1) Do I need to prep the replacement as before in the thread above? My experience with RAID arrays outside of this NAS is that no prep is needed and the array rebuilds onto it right after the swap
2) Confirmation that using a larger drive is ok. All the disks in there now are 2TB, but I have good 3TB drives lying around. I know the extra space of the 3TB drive would be "wasted" but I'm not really using them right now and so it would save money
3) If I use a larger drive for the replacement and later on go on to replace all the other drives (including the OS one) with the same size of larger drive, is there a way to expand the RAID 5 array to use all the space of the larger drives WITHOUT wiping the data?

Thanks!

PS: it was already confirmed earlier my unit can support larger than 2TB drives.
hvymetal86
Donator VIP
Donator VIP
 
Posts: 121
Joined: Mon May 11, 2015 3:42 pm

Re: Failed Drive Replacement

Postby Jocko » Sat Jun 10, 2017 7:42 am

Hi hvymetal,

What is your fvdw-sl version on the 5big2

With version 17.0 will be easy to replace sdc as it allows to initialize the disk from the web-interface

otherwise: yes you can use bigger size on one component in a raid; yes you can add a gpt partition on a raid using md-dos partitions on the other components.
Jocko
Site Admin - expert
 
Posts: 11529
Joined: Tue Apr 12, 2011 4:48 pm
Location: Orleans, France

Re: Failed Drive Replacement

Postby hvymetal86 » Sat Jun 10, 2017 1:22 pm

Current Firmware Version : 5big2 fvdw-sl 16.1 patched

Patch Log file:
Date Patch Status
2015-05-19 08:30:21 fvdw-sl-16.1-afpfix-remoteshr-11oct15.zip(fvdw-sl-16.1-afpfix-remoteshr-11oct15.tar) OK
2015-05-19 08:30:41 fvdw-sl-16.1-fix-autowakeup-03oct15.zip(fvdw-sl-16.1-fix-autowakeup-03oct15.tar) OK
2015-05-19 08:31:03 fvdw-sl-16.1-warning-iesupport-03dec15.zip(fvdw-sl-16.1-warning-iesupport-03dec15.tar) OK
2016-08-20 12:33:11 fvdw-sl-16.1-USB-fix-17feb16.zip(fvdw-sl-16.1-USB-fix-17feb16.tar) OK
2016-08-20 12:42:52 fvdw-sl-16.1-samba-patch-25-mar-2016.zip(fvdw-sl-16.1-samba-patch-25-mar-2016.tar) OK
2017-04-01 12:29:54 fvdw-sl-16.1-fix-USB-menu-04nov16.zip(fvdw-sl-16.1-fix-USB-menu-04nov16.tar) OK
2017-04-01 12:36:25 fvdw-sl-16.1-Root-CA-Chain-04dec2016.zip(fvdw-sl-16.1-Root-CA-Chain-04dec2016.tar) OK

I did see the announcement in my NAS' management about 17.0 and when reading saw about disk initialization, but wasn't sure if it was a good idea to upgrade firmware with a degraded array. All 5 disks are part of the RAID5 array including the OS disk sda.

Expanding the RAID can wait until later (have to get all larger disks first), but making sure I understand your response about that, the added space would be a new separate GPT partition, not the existing one expanded?
hvymetal86
Donator VIP
Donator VIP
 
Posts: 121
Joined: Mon May 11, 2015 3:42 pm

Re: Failed Drive Replacement

Postby Jocko » Sat Jun 10, 2017 5:07 pm

You can update the firmware without issue as the process changes nothing on the data partitions

hvymetal86 wrote:Expanding the RAID can wait until later (have to get all larger disks first), but making sure I understand your response about that, the added space would be a new separate GPT partition, not the existing one expanded?
the firmware will build only one data partition with size around 2,7 TB, so as long as you did not change the other disks you lose around .9 TB when you will include it on the raid

The firmware can not handle an additional data partition on the disk
Jocko
Site Admin - expert
 
Posts: 11529
Joined: Tue Apr 12, 2011 4:48 pm
Location: Orleans, France

Re: Failed Drive Replacement

Postby hvymetal86 » Sun Jun 11, 2017 2:49 am

Applied the 16.1 pre-upgrade patch successfully:
Code: Select all
Patch Log file:
Date   Patch   Status
2015-05-19 08:30:21   fvdw-sl-16.1-afpfix-remoteshr-11oct15.zip(fvdw-sl-16.1-afpfix-remoteshr-11oct15.tar)   OK
2015-05-19 08:30:41   fvdw-sl-16.1-fix-autowakeup-03oct15.zip(fvdw-sl-16.1-fix-autowakeup-03oct15.tar)   OK
2015-05-19 08:31:03   fvdw-sl-16.1-warning-iesupport-03dec15.zip(fvdw-sl-16.1-warning-iesupport-03dec15.tar)   OK
2016-08-20 12:33:11   fvdw-sl-16.1-USB-fix-17feb16.zip(fvdw-sl-16.1-USB-fix-17feb16.tar)   OK
2016-08-20 12:42:52   fvdw-sl-16.1-samba-patch-25-mar-2016.zip(fvdw-sl-16.1-samba-patch-25-mar-2016.tar)   OK
2017-04-01 12:29:54   fvdw-sl-16.1-fix-USB-menu-04nov16.zip(fvdw-sl-16.1-fix-USB-menu-04nov16.tar)   OK
2017-04-01 12:36:25   fvdw-sl-16.1-Root-CA-Chain-04dec2016.zip(fvdw-sl-16.1-Root-CA-Chain-04dec2016.tar)   OK
2017-06-10 22:40:52   fvdw-sl-16.1-upgrade-patch-30may2017.zip(fvdw-sl-16.1-upgrade-patch-30may2017.tar)   OK

Then tried the web gui firmware update to 17.0 and it failed:
Code: Select all
=> Firmware upgrade step 1 started:
Search firmware version information for 5big2 ...
The latest version available from http://plugout.net is: fvdw-sl 17.0

Get the bound upgrade files list
Retrieve the firmware package
Try to download part 1 on 5...
Attempt 1: OK, got successful the firmware package part1 file
Try to download part 2 on 5...
Attempt 1: OK, got successful the firmware package part2 file
Try to download part 3 on 5...
Attempt 1: OK, got successful the firmware package part3 file
Try to download part 4 on 5...
Attempt 1: OK, got successful the firmware package part4 file
Try to download part 5 on 5...
Attempt 1: OK, got successful the firmware package part5 file
Extract firmware package...
Extract error : some files are missing
Error 303
hvymetal86
Donator VIP
Donator VIP
 
Posts: 121
Joined: Mon May 11, 2015 3:42 pm

Re: Failed Drive Replacement

Postby Jocko » Sun Jun 11, 2017 8:09 am

Hi Hvymetal

I think understand why you got this message

I did a test on my 5big2 and got no issue
Code: Select all
=> Firmware upgrade step 1 started:
Search firmware version information for 5big2 ...
The latest version available from http://plugout.net is: fvdw-sl 17.0

Get the bound upgrade files list
Retrieve the firmware package
Try to download part 1 on 5...
Attempt 1: OK, got successful the firmware package part1 file
Try to download part 2 on 5...
Attempt 1: OK, got successful the firmware package part2 file
Try to download part 3 on 5...
Attempt 1: OK, got successful the firmware package part3 file
Try to download part 4 on 5...
Attempt 1: OK, got successful the firmware package part4 file
Try to download part 5 on 5...
Attempt 1: OK, got successful the firmware package part5 file
Extract firmware package...
OK, extract successful the firmware package
Retrieve kernel image file for kirkwood...
Attempt 1: OK, got successful the kernel image file

OK: Retrieve the latest firmware image file successful


The error code 303 means at least one file is missing after extracting them. This may happen if there is no enough space on /tmp. For example, this occurs if you installed some addons previously without restarting the nas
(in next version we add a check on the free space)

Please to post
Code: Select all
df -h /tmp
and try again to upgrade the nas after rebooting it
Jocko
Site Admin - expert
 
Posts: 11529
Joined: Tue Apr 12, 2011 4:48 pm
Location: Orleans, France

Re: Failed Drive Replacement

Postby hvymetal86 » Sun Jun 11, 2017 1:36 pm

Before reboot:
Code: Select all
root@HvyMtlNAS:/ # df -h /tmp
Filesystem                Size      Used Available Use% Mounted on
/dev/sda5               739.9M      6.3M    725.9M   1% /rw_fs
After reboot:
Code: Select all
root@HvyMtlNAS:/ # df -h /tmp
Filesystem                Size      Used Available Use% Mounted on
/dev/sda5               739.9M      1.2M    731.0M   0% /rw_fs
Firmware Update:
Code: Select all
Firmware upgrade step 1 started:
Search firmware version information for 5big2 ...
The latest version available from http://plugout.net is: fvdw-sl 17.0

Get the bound upgrade files list
Retrieve the firmware package
Try to download part 1 on 5...
Attempt 1: OK, got successful the firmware package part1 file
Try to download part 2 on 5...
Attempt 1: OK, got successful the firmware package part2 file
Try to download part 3 on 5...
Attempt 1: OK, got successful the firmware package part3 file
Try to download part 4 on 5...
Attempt 1: OK, got successful the firmware package part4 file
Try to download part 5 on 5...
Attempt 1: OK, got successful the firmware package part5 file
Extract firmware package...
OK, extract successful the firmware package
Retrieve kernel image file for kirkwood...
Attempt 1: OK, got successful the kernel image file

OK: Retrieve the latest firmware image file successful


=> Firmware upgrade step 2 started:
Firmware package uploaded via fvdw-sl server
package type: new configuration (version > 16.1)
Check Image file... OK
Update basic firmware image for 5big2 platform, ... OK
Check kernel version... new version must be installed
Check DATABASE file... OK
Check active partition... OK
Write fs image on /dev/sda2...(please wait) OK
verify fs ... OK
Write new kernel image... OK
Update DATABASE... OK
switch boot part to upgraded partition... OK

OK: Firmware upgrade successful
Firmware status after reboot:
Code: Select all
Current Firmware Version :   fvdw-sl 17.0 (5big2)

You have the latest version

But my shares are all offline and in disk setup it doesn't show the drives in an array, degraded or not.
https://drive.google.com/file/d/0B5x7lw ... sp=sharing
hvymetal86
Donator VIP
Donator VIP
 
Posts: 121
Joined: Mon May 11, 2015 3:42 pm

Re: Failed Drive Replacement

Postby Jocko » Sun Jun 11, 2017 1:41 pm

This may happen only if you do not have a regular partitions table on your disks or the firmware failed to assemble the raid
(Note your raid array settings is still here)

So post
Code: Select all
fdisk -l
and
Code: Select all
file -bs /dev/sda8
file -bs /dev/sdb8
file -bs /dev/sdc8
file -bs /dev/sde8


Post the content of
Code: Select all
cat /boot.log
cat /etc/mdadm.conf
ls -l /etc/mdadm.conf
cat /rw_fs/etc/mdadm.conf
Jocko
Site Admin - expert
 
Posts: 11529
Joined: Tue Apr 12, 2011 4:48 pm
Location: Orleans, France

Re: Failed Drive Replacement

Postby hvymetal86 » Sun Jun 11, 2017 7:43 pm

Code: Select all
root@HvyMtlNAS:/ # fdisk -l

Disk /dev/sdb: 2000.3 GB, 2000398934016 bytes
256 heads, 63 sectors/track, 242251 cylinders
Units = cylinders of 16128 * 512 = 8257536 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sdb1               1      242252  1953514583+ ee EFI GPT

Disk /dev/sda: 2000.3 GB, 2000398934016 bytes
256 heads, 63 sectors/track, 242251 cylinders
Units = cylinders of 16128 * 512 = 8257536 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sda1               1      242252  1953514583+ ee EFI GPT

Disk /dev/sde: 2000.3 GB, 2000398934016 bytes
256 heads, 63 sectors/track, 242251 cylinders
Units = cylinders of 16128 * 512 = 8257536 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sde1               1      242252  1953514583+ ee EFI GPT

Disk /dev/sdd: 2000.3 GB, 2000398934016 bytes
256 heads, 63 sectors/track, 242251 cylinders
Units = cylinders of 16128 * 512 = 8257536 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sdd1               1      242252  1953514583+ ee EFI GPT

I replaced sdc8 with sdd8 because sdc is the failed drive and is currently out of the NAS. Though funny thing about the drive, I just finished doing a long test with Segate's SeaTools and it passed... but the NAS couldn't detected even with a reboot before I started this thread:
Code: Select all
root@HvyMtlNAS:/ # file -bs /dev/sda8
Linux rev 1.0 ext3 filesystem data, UUID=bd535352-1c00-4e7c-a4dd-7b2ca7731449 (needs journal recovery) (large files)

root@HvyMtlNAS:/ # file -bs /dev/sdb8
data

root@HvyMtlNAS:/ # file -bs /dev/sdd8
data

root@HvyMtlNAS:/ # file -bs /dev/sde8
Linux rev 268435463.0 ext4 filesystem data, UUID=bfd3825d-1c00-4e6c-a4dd-7b2ca7731449 (needs journal recovery) (large files)

Code: Select all
root@HvyMtlNAS:/ # cat /boot.log
start boot log
detect and set platform
5big2
kirkwood
UIMAGE-466-KIRKWOOD-7
Current kernel: 4.6.6 #7 PREEMPT Sun May 7 11:50:52 CEST 2017
5big2
 mount dev/pts
update dev nodes
booting using sda2 root file system...
make dev node for buttons...
make dev node for tun device...
enable IP forwarding...
start buttons control daemon
device = 5big2
source = buttons-nwsp2
buttons-nwsp2 daemon started
Sun Jun 11 14:24:19 GMT+1 2017
create temporary passwd file
run udevstart to update dev nodes when necessary
inserting kernel modules:
modprobe: module 'iscsi_trgt' not found
create temporary group file
configure loopback network interface
setting reboot and standby
5big2
rebootd-nwsp2
5big2
standbyd-nwsp2
start fan
starting php based setup routines step 1
 * Initialize the volatile db file...    [ OK ]
 * Starting udevd...                                     [ OK ]
 * Starting RAID monitor:                                [ OK ]
 * Starting restore RAID devices...
   - Assembling device /dev/md0:  [ Fail ]

Warning: file_get_contents(/sys/block/md0/md/sync_action): failed to open stream: No such file or directory in /etc/finc/dm_restore_md.finc on line 52
 * Finishing restore RAID devices...     [ Fail ]
 * Found database configuration file...  [ OK ]
 * Updating Disks database...                    [ OK ]
 * Starting mount of volumes...
 * Finishing mount of volumes...                 [ OK ]
 * Generating Hosts File...                      [ OK ]
 * Configuring System Hostname...                [ OK ]
 * Configuring LAN interface...                  [ OK ]
 * Initializing Timezone...                      [ OK ]
 * Starting web server...                                [ OK ]
 * Starting mount of internal USB ...

Warning: Illegal string offset 'vol' in /etc/finc/dm_mount_internal_USB.finc on line 21
 * Finishing mount internal USB ...      [ OK ]
 * Configuring Disks...                                  [ OK ]
 * Configuring Samba...                                  [ OK ]
 * Configuring System Users...
        Root password: use default password
        Set users, linux and samba accounts      [ OK ]
 * Starting Fvdw-sl Discovery Daemon...  [ OK ]
start rpcbind service
starting php based setup routines step 2
 * Starting dropbear...                                  [ OK ]
 * Kill temporary dropbear...                    [ OK ]
starting php based setup routines step 3
 * Starting daemon update Hosts File...  [ OK ]
 * Starting Disk Temperature Guard...    [ OK ]
 * Starting mount of remote shares...
 * Finishing mount shares...                     [ OK ]
 * Starting NTP client...                                [ OK ]

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 31

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 32

Warning: Illegal string offset 'volid' in /etc/finc/dm_get_volume_addons.finc on line 32

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 31

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 32

Warning: Illegal string offset 'volid' in /etc/finc/dm_get_volume_addons.finc on line 32

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 31

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 32

Warning: Illegal string offset 'volid' in /etc/finc/dm_get_volume_addons.finc on line 32

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 31

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 32

Warning: Illegal string offset 'volid' in /etc/finc/dm_get_volume_addons.finc on line 32

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 31

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 32

Warning: Illegal string offset 'volid' in /etc/finc/dm_get_volume_addons.finc on line 32

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 31

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 32

Warning: Illegal string offset 'volid' in /etc/finc/dm_get_volume_addons.finc on line 32

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 31

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 32

Warning: Illegal string offset 'volid' in /etc/finc/dm_get_volume_addons.finc on line 32

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 31

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 32

Warning: Illegal string offset 'volid' in /etc/finc/dm_get_volume_addons.finc on line 32

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 31

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 32

Warning: Illegal string offset 'volid' in /etc/finc/dm_get_volume_addons.finc on line 32
 * Starting Transmission Client...
Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 31

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 32

Warning: Illegal string offset 'volid' in /etc/finc/dm_get_volume_addons.finc on line 32

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 31

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 32

Warning: Illegal string offset 'volid' in /etc/finc/dm_get_volume_addons.finc on line 32
                 [ OK ]

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 31

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 32

Warning: Illegal string offset 'volid' in /etc/finc/dm_get_volume_addons.finc on line 32

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 31

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 32

Warning: Illegal string offset 'volid' in /etc/finc/dm_get_volume_addons.finc on line 32

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 31

Warning: Illegal string offset 'vol' in /etc/finc/dm_get_volume_addons.finc on line 32

Warning: Illegal string offset 'volid' in /etc/finc/dm_get_volume_addons.finc on line 32
starting php based setup routines step banner


*** fvdw-sl NAS firmware
    This is fvdw-sl version: fvdw-sl 17.0
    built on: May 28 2017

    LAN IP address: 192.168.1.128 (DHCP)

    Port configuration:

    LAN   -> eth0
php based setup routines finished
web permission on resolv.conf
set device tuning for dms performance
move smbd en nmbd db files away from ram disk to prevent hanging samba server
LED settings

Send a boot mail notification to [REDACTED]@gmail.com
else loop1 finished
rcS finished

Code: Select all
root@HvyMtlNAS:/ # cat /etc/mdadm.conf
CREATE owner=root group=root mode=0666 auto=yes metadata=1.0
PROGRAM /usr/bin/mdadm-events
DEVICE /dev/sd* /dev/se*
ARRAY /dev/md0 level=raid5 num-devices=5 spares=1 UUID=c6b0476f:07a34c67:9b30df64:77fdfc96

MAILADDR [REDACTED]@gmail.com

Code: Select all
root@HvyMtlNAS:/ # ls -l /etc/mdadm.conf
lrwxrwxrwx  1 root root 21 2017-05-30 17:12 /etc/mdadm.conf -> /rw_fs/etc/mdadm.conf

Code: Select all
root@HvyMtlNAS:/ # cat /rw_fs/etc/mdadm.conf
CREATE owner=root group=root mode=0666 auto=yes metadata=1.0
PROGRAM /usr/bin/mdadm-events
DEVICE /dev/sd* /dev/se*
ARRAY /dev/md0 level=raid5 num-devices=5 spares=1 UUID=c6b0476f:07a34c67:9b30df64:77fdfc96

MAILADDR [REDACTED]@gmail.com
Last edited by hvymetal86 on Mon Jun 12, 2017 12:41 am, edited 1 time in total.
hvymetal86
Donator VIP
Donator VIP
 
Posts: 121
Joined: Mon May 11, 2015 3:42 pm

Re: Failed Drive Replacement

Postby Jocko » Sun Jun 11, 2017 7:56 pm

So the boot script failed to assemble md0

This is not normal:
Code: Select all
root@HvyMtlNAS:/ # file -bs /dev/sdb8
data

root@HvyMtlNAS:/ # file -bs /dev/sdd8
data


Can you post
Code: Select all
mdadm --examine /dev/sd[abce]8
Jocko
Site Admin - expert
 
Posts: 11529
Joined: Tue Apr 12, 2011 4:48 pm
Location: Orleans, France

Next

Return to Lacie 5big Network vs2

Who is online

Users browsing this forum: No registered users and 10 guests