Lacie 5Big2 Failed After Firmware Update

[unlucky1]

It did stop a few times and I told it to "retry" each time it did. Okay, that means I need to restart it all again :( I'll try after breakfast ;) Good morning, by the way :)

[Mijzelf]

I think it's a protocol issue. When connecting to ftpd, you get a 'command channel'. For each data transfer action (read/write files, ask directory listing) a 'data channel' is opened over a different port. I think ftpd will fork a new process for each 'data channel', which means that for each copied file a new instance of ftpd is created.

If for some reason the protocol is not handled to it's end, I think the ftpd process will just wait forever. So if winscp only says 'okay, thanks' at the end of a data transfer, instead of 'okay, thanks, goodbye', ftpd just waits forever for that goodbye.
But that's only a theory. At least it's clear busybox ftpd is not yet ready for the enterprise marked. :mrgreen:

How far are you? Is it worthwhile to compile a statically linked 'enterprise' server?

[unlucky1]

Just got back. Had a family thing to do today. I'm going to reboot and restart now. I still have a lot to copy over and then somehow verify that I have it all. After that, I want to set it up again as a RAID but with 2 disks fall-over in a way so that they're readable independently. I suppose that's not a raid but I just don't trust RAIDs anymore. It's the second time they've failed me. But I think it would be good to have a box that does an auto-load-balance/back-up of every file I copy to it. Is that even possible?

[Mijzelf]

I just done't trust RAIDs anymore. It's the second time they've failed me. But I think it would be good to have a box that does an auto-load-balance/back-up of every file I copy to it. Is that even possible?

*Any* storage solution will fail, if you wait long enough. You'll always need independent backups.

auto-load-balance/back-up of every file I copy to it. I'm not sure what you exactly mean. It sounds like raid1, mirroring. Using raid1 you have disks which are a copy of each other. But the volume size can never exceed the size of a single disk.

Or do you mean you just want to be able to mount each 'raid member' independently? In that case you can create an mhddfs volume. One virtual volume, automatically loadbalancing over 5 'real' volumes. But no automatic redundancy.

Are you planning to use the same hardware? In that case you just don't have enough space to be fully redundant. 3+TB files on 5TB disks. You just can't store it twice.

But if you would use new, bigger disks, you could use 4 2TB disks to build 2 2 disk mhddfs 4TB volumes. Then you could use one volume to store your data, and let rsync, or even better, rsnapsnot, sync it to the other volume at night.

This way you'll have all your files twice, on 4 independently mountable disks. I think it's about the best you can get using a single 5big2. But remember that theft, lightning, a failing power supply, ... can still kill all your disks at once. You still need backups.

[unlucky1]

WinSCP has a copy new/changed files only so I'm going through and verifying the directories before I transfer new stuff. And I'm keeping an eye on ftpd. It's growing but slowly. It's up to process number 1225 now which is good because it's been running a couple hours and it seems to fail at around 8000ish.

Once I get to the point where I can rebuild the Lacie, I'd really like your help. I don't know if I should use your software and build it better that way. Or really what I should do. I know that I need to have multiple back-ups and I plan to do so but if I do the rsnapshot thing, I could also change out 2 of the disks every week or so and keep a rotating back-up for disaster recovery. Except I'd need to put on and off the rails which would suck....

The truth is I really don't know what's best and I'd value your opinion on it.

[Mijzelf]

it seems to fail at around 8000ish.

I suppose that's 8192. (2^13). When only the number of processes is the limit, and not the available memory, you could increase this number:

Code: Select all

echo 32768 >/proc/sys/kernel/pid_max

I could also change out 2 of the disks every week or so and keep a rotating back-up for disaster recovery.

That is not a good idea. Nor the connectors on the disks, neither the connectors in the Lacie are designed for that. But you could use the e-sata connectors for that.

The truth is I really don't know what's best

"Best". That's a complicated word. I could tell you what's best, but you would react: "What! 1 million euro's a week?"

My personal preference against backups is to automate erverything, as I'm just too unorganized to rely on daily/weekly human initiated actions, and to have at least one offsite backup. And I'm not a fan of raid. Raid is overrated.
So far I haven't had the need to create big volumes (bigger than a disk), but if I had, I would have used mhddfs or something similar, and not raid.
I *think* your best option is a 2 disk mhddfs volume to contain the data, a 2 or 3 disks volume containing rsnapshot backups (And we should investigate if "the big rsnapshot trick"® (usage of hardlinks, so that each individual file is only one time present in numerous complete backups) works on a mhddfs volume.)
Further an offsite NAS which gets a weekly backup over the internet. Of course you need some bandwidth for it. How much depends on how changes in your data set can be expected in a week.

This can not be done on Lacie firmware. It can be done on fvdw's firmware, if he is willing to spend some time on it.
And it can be done on any Linux distro which could run on your box, like Debian, Arch, Ubuntu, ... But for all those cases first some serious time should be invested to get that distro running on the Lacie.

[fvdw]

intersting comments !

...It can be done on fvdw's firmware, if he is willing to spend some time on it.

yes for sure we will look into this, now that we have support for mutiple disks nas devices we should have these kind of options. Only it will not be on short notice as holiday also takes time :whistle

[fvdw]

For what it is worth our firmware has rsync incl daemon on board and a task planner.
This can be used quite easily for automating the backups from one disk to the other.
It is the way i planned to set up my 5big2 to automatically backup the important folders every day or more.

You can have a look at the fvdw-sl firmware here
http://www.plugout.net/web/usr/htdocs/home.php

Look in the "file share" menu for backup and rsync server...