FreeNas Box Mutation - Looking for tips and comments

[PotatoMasher]

Hello!

I have been a FreeNas user for the past two years (or so, been using it since the first versions of 7). My box and I have been through various crashes, loss of data, etc... so I have been experimenting a lot with FreeNas. But my box is filling fast, mainly because my whole family is using it, and the time has come to upgrade it!

Before going further, here is a description of my actual setup:
Mobo: A8N32-SLI Deluxe
CPU: AMD Athlon 64 x2 4200+
RAM: 4gb DDR (don't laugh, it's powerful enough for my needs)
HDD: 4x 2TB WD Caviar Green, set up in Raid-Z

My main goal, if possible, is to come up with a setup that will evolve through the years and adapt to the growing need of my family to store lots of stuff. I have been looking around, reading posts, reviews of products and have been compiling pros and cons for potential solutions. Here they are, along with the underlying questions:

Question 0: Is there a way to make sure I don't lose data while doing a backup of my array? Since all of my solutions include creating a JBOD to contain the existing data, can I just move the data from the existing pool to the temporary JBOD with a single command line that will check the data integrity?

Solution 0)
"Drop FreeNAS and go for Ubuntu Server with mdadm"
This solution is the more "painful" one, but it fits my needs (or so I read...). Mdadm seems to have the capability to expand an existing array with parity without having to destroy it beforehand.
Pros:
-Array with parity
-Expandable array
Cons:
-Modular --> Lots of tweaks to get what FreeNAS had built-in
-Not sure if it will run from USB stick
-Loooong resilver downtime when expanding array

Question 1: Is FreeNAS able to use an equivalent to mdadm in terms of array extensibility?

Solution 1)
"Upgrade existing RAID-Z after destroying it"
This solution is the more by-the-book one, which means that I will need to destroy my existing array (after backing it up, of course) to build it again with more drives.
Pros:
-FreeNAS rocks
-Still using the good old Raid-Z
Cons:
-Doesn't fix the problem, I will have to add another vdev when I will need more space --> More expensive

Question 2: What is the end-user benefit of using Raid-Z2 compared to Raid-Z, except from the double drive failure capacity?

Solution 2)
"Buy a hardware solution"
This solution would mean that I would have to buy some sort of SAS card that supports hardware raid expansion. I have not really explored that solution yet, but I have been reading more and more about some basic SAS cards since my motherboard currently only have 4 SATA plugs.
Pros:
-FreeNAS still rocks (if the SAS card is supported)
-Easy array expansion depending on the card I choose to buy
Cons:
-The usual hardware raid concerns: If the card breaks, so does your array if you don't find an identical card to replace the old one
-The price tag on some SAS + Raid cards

Question 3: I get it that it would be easier in a long-term view to get a SAS card instead of a PCI-e SATA port duplicator, but is there a list of "officially supported cards" for the current FreeNAS version?

Thank you for the time you spent going through my first wall of text on those new forums, and if you have any clue, comment, input or any answer to my questions, feel free to reply!

PotatoMasher
-His family's IT guy-

 

[JaimieV]

You have an additional question -1 that you haven't mentioned - "Shouldn't I be backing up my NAS on a regular basis already?", with the answer "Oh yes, absolutely, because as we all know having data on a RAID is not a backup, it's just for resilience and continued service through disk failure".

Answer 0: Put your backup onto a RAIDZ1. Or directly onto the new array, whatever that will be - 5x3Tb or some such? You haven't mentioned how much actual data you have. Note that as this is a family NAS you probably have two classes of data - important stuff, and stuff that can be downloaded again. The important stuff might be small enough to simply copy onto a couple of workstations (two copies!), while putting the downloadable stuff onto something nonresilient.

Answer 1: It's not a FreeNAS question, it's a ZFS question - and no, there's no way to directly extend a ZFS RAIDZ array, you can only add another (preferably resilient) vdev to it.

Answer 2: Nothing greater than the answer you have in the question - RAIDZ2 allows two drives to go dead before you lose all your data. If you have a live backup, RAIDZ1 isn't a problem (until you get to many drives - exact value of many being rather context dependent, could be as few as 4)

Solution 2 needs commenting, as if you're using a hardware RAID card then you're foregoing the ZFS RAID system entirely, and only using the data scrubbing part of ZFS. This may be enough for your needs.

Question 3: Dunno about supported SAS, except there's surely a list of FreeBSD SAS cards kicking around. But there's a misconception to clear up: A PCIe SATA card would not be a port duplicator, it'd be a set of raw SATA ports just like the ones on the motherboard (and functionally equivalent to getting a SAS card). A port duplicator is a device that hangs off one SATA port and gives you eg four SATA ports from it.

My own domestic config is two FreeNASes - one with a RAIDZ1, the other with a stripe set that the first is backed up to. Consider that as part of your planning. Build the new primary array, replicate/rsync the old array content to it, move the old 4x2tB into another box, set that up (maybe with a fifth 2Tb) as a stripe or RAIDZ1, sort out a backup strategy to get data from primary to backup automatedly.

 

[PotatoMasher]

Hi!

Thank you for your quick answer. First of all, let me answer your question: I currently have a 5.6 tb array, and about 5.3 of it is currently in use.
Here is another batch of questions, mainly to add precision to your previous answers:

1) You said to back up my NAS. Since an array have a n disk failure capacity, which means that n disk can go wrong without me losing data, isn't that some kind of financial overkill to add another array to back-up the data? (Considering that this array can also fail)
1.1) More importantly, is this backup solely to protect against double disk failure (in the hypothesis of a Raid-Z1 filesystem)? I understand that this back-up array would be for critical user data only, if so would setting up a permanent mirrored 2x2tb array be enough "protection"?
2) Can this backup array be in the same box? (My answer would be yes, but that wouldn't prevent total data loss due to power spark/motherboard fusion)

The important edge I see to a SAS card compared to a PCIe SATA card is the number of drives I can add versus the number of PCIe slots I use on my motherboard. I will look around for a list of officially supported SAS cards before buying anything.

So far, from my reflection and your answers, I have come up with a setup a bit like that:

Primary array (containing everything)
-----------------------------------------
5x2tb Raid-Z1 array [Can be extended with a minimum of 2 drives at a time --> Adding vdevs]
OR
5x2tb Hardware Raid array [Can be extended with a single drive at a time]

"Backup" array (only user data and pictures)
-----------------------------------------
2x2tb Mirrored array --> Backup of important stuff (pictures, documents)

Last question form this batch:
3) Is hardware raid via a SAS card the only logical solution for extending an array with a single disk at a time, knowing that FreeNAS doesn't seem to support mdadm? (unless I'm wrong and I haven't stumbled across that particular manual article)?

Thanks again!

PotatoMasher

 

[JaimieV]

1) A RAID array is useful for continued operation in case of one failed disk (well, one for RAID5/Z1, more if RAID6/Z2 etc).
It does not protect you from other problems. Specifically,
* Accidental/wilful but wrong deletion of files (note - can be recovered from poorly with snapshots)
* Rolling back to an older version of a file on accidental save over (note - can be recovered from poorly with snapshots)
* Failure of the host machine (mobo die, psu zap's rest, flood, lightning, fire...)
* Filesystem data or structure corruption (bad luck, power transients, bad RAM, buggy OS/fs driver, even dodgy SATA cable)
* Fully degraded array (loss of one more than your redundant disks, loss of a SATA card (counts as losing several disks))

So for (1.1) no, it's for other reasons too.

2) You could put the backup array into the same machine, but that would not protect against many of those things so I would advise against. Best is another machine in a different location in the house. I use HP Microservers - they're cheaper than a 4xSATA USB disk box! My primary is always on in the basement (nice and cool, up a hill so very little flood risk), while the secondary is in a cupboard.

At the very least, pick out your important stuff and use something like Crashplan to backup over t'Internet to a relation's computer. That doesn't help you with the original need to backup and rebuild the array though!

Again, it's not FreeNAS that doesn't support mdadm, it's the way ZFS works that you can't expand a vdev after the initial build. mdadm is a solution to a different problem, and requires co-operating filesystems too.

Your possible model looks okay... (it's not what I'd do, but it's effective), but if you're considering going SAS for hardware RAID, then the budget for buying SAS disks on top of the RAID card will be considerably more than building out a new (eg) 4x3Tb SATA RAIDZ1 for the primary and using the old disks as backup, won't it? Or were you considering using SAS-SATA converter blobs? I'd be rather wary of trusting my data to that...

 

[PotatoMasher]

Thanks again for a very fast and concise answer!

First, let me try to confirm something. SAS cards do work with SAS -to-> SATA cables, right? So no need to buy some special drives to use with the card, assuming you use the "converter blobs". It will only cost me between 200$ and 300$ for a good PCIe SAS card on which I will be able to hook 4 SATA drives to each SAS connector using the provided cables, and do some extendable raid-5 with them. The drivers to the SAS cards I was looking at seems to be integrated in FreeBSD 8.3 (Manpage of the driver here) so I will need to upgrade my software to FreeNAS 8.3 too.

My question concerning mdadm is more a question about if FreeNAS would be able to use and manage a mdadm array (if it can, it would certainly be a good option for the main storage array). The answer so far seems to be no :D

Note so self: Buy/convert another machine to be a backup server.
I already have a Ubuntu server dedicated to run my Minecraft server and a partimage server, I will try to find a way to make it contain the backup mirrored array.
1) Can the second server be used to do more tasks than to maintain the mirrored (Raid 1) backup of important data?
2) Is there a better (disk-space-efficient and safe failure-wise) way to store a backup of the important data than Raid 1?

Since my final solution will probably (since it seems to be more cost-effective as a home user with limited budget) include a SAS card with SAS to SATA cables, I will redirect my research on the "How to use" rather than the "What to choose".

The final setup will probably look like

Main array (Existing box with SAS card)
4x 2tb Drives in Hardware Raid 5 (The existing ones, at which I will dynamically add the drives used to temporarily back-up all of the original data in a JBOD)

6x 2tb Drives will be the final setup once everything is expanded​

Backup array (Containing important data, on a Ubuntu server)
2x 2tb Drives in Raid 1 (New ones)

3) What is the best mechanism to do a daily backup of important data from one server to another?

Thanks again for your precious answers.