I'm relatively close to noob-dom on FreeNAS, so the experts will have to correct this where I get it wrong, but I have some comments that seem correct to me and that I think would be useful to other newbies.
(1) It is somewhat amazing that we have a computer-world where an individual can even contemplate having and managing tens of terabytes of data. Wow. This was the exclusive realm of governments and enormous corporations not all that long ago. (yes, I'm old)
I have a very active data-life for a single human. There are a dozen PCs in my house, eight of which I can touch from my seat in my office. The backup on all of these machines together is a few TB. To have tens of TB to back up, you're hosting a lot of music, pictures, and video. It would be good for you to ponder what of that data is really, really irreplaceable. You're spending a lot of money (for a single human, not an organization) and considering spending a lot more to keep those bits in your house. You're also spending a lot of your personal time managing that data and introducing the possibility of a keyboard mistake wiping out a lot of it. Have you considered something like Amazon S3 Glacier?
The point here is that (1) humans are pack rats and (2) different kinds of data are actually worth different amount of worry and money spent on keeping it.
(2) ZFS and by extension FreeNAS is an industrial grade solution to managing huge amounts of data that happens to fit within the systems that we can buy. Along with that industrial strength ability comes some complex and non-obvious issues. This is not as simple as making a new copy on another disk. ZFS solves a couple of specific problems in keeping large chunks of data, but you pay for that with the potential for losing large chunks of data with no hope of recovery if you type the wrong thing.
ZFS is an attack on two items - the need to use an entire trained human or maybe several of them to manage what data is in what disk, and the threat of silent corruption of data. In an earlier life, I employed a storage administrator to keep the storage up to date for my department. There is a lot of moving data from volume to volume as you exceed one disk's capacity and bringing more storage on line or off as disks die or data grows. ZFS solves this by internally administering data on multiple disks invisible to the user. The silent corruption issue is also a biggie. Data rots. CDs and DVDs rot and lose bits, and disks corrupt and die. ZFS is a suite of approaches to detect the inevitable bit-rot, and correct it by storing additional checking and correcting info. All of the data in a Zdev and Zpool is knit together with this detect-and-correct web of stuff, so you pay for your fixing bit-rot with the need to preserve the entire pool by losing no more than the allowable number of dead disks at once. ZFS is not fragile on the scale of individual bits, bytes, or disks, but it is prone to losing 100% all of it if you do the wrong thing. This is counter-intuitive to even some computer-savvy folks, and even some professionals.
So as advice:
- consider the value of your data in terms of how hard/expensive it would be to get it back if you simply lost it
- consider the value of your time spent managing your data
- consider the bet you're making in wagering that you have/will learn enough about ZFS to manage it properly and avoid the sudden total death of your data
After that, if you're still up for the bet, what jgreco said.
- go read "ZFS Best Practices". Now read it again.
- make zdevs in groups of no more than 12 disks per zdev. I like 7 or 11 disks per in RAIDZ3, but that's just me.
- worry about where you'll back up your ZFS. I use my ZFS as a backup medium, not as a working medium. If you are using your ZFS server as a working medium, you simply must have a backup for IT. Proper backup practice uses at least three levels; working, backup and backup-backup, and even better three rotating backup levels exclusive of the working store level. There is whole set of best practices in the area of backups. In any case, if you have less than two copies of your data *and a way to tell which one is right* and *you have tested that you actually CAN get the data back out of your backup* then you have no protection against a finger slip on the keyboard, a cosmic ray in your memory, or a random read error on the disk corrupting your data. How much was that data worth?
Here's my dim reaction to your questions. Bear in mind that I have a lot of professional computer background, but have only been tinkering with ZFS and FreeNAS for a year or two, and then only incidentally.
>> I am running out of space and am studying migrating to FreeNAS; I will use the 16 RE4 disks in there too;
EGADS that's a lot of data for one person. You really need all that stuff??
>> The problem at hand is: which disks to buy, 4TB or 6TB?
>> ... similar disks ... almost the same price ... benefit extra from more space ... more than 1 VDEV ... 'an investment in the future'
>> If I need to expand ... 3 years warranty ... 5 years warranty;
I find it helpful to think data-centric. Preserving data costs money and time, and all data corrupts over time. The most stable data preservation media that humans have come up with are (1) stone (hieroglyphics) (2) ceramic (cuniform clay tablets) (3) paper-ish (papyrus and similar). If these all seem very non-dense and slow, well, they are. Data storage is a trade off of stability versus cost versus access time versus density.
I have come to think of computer hardware as being a fungible, rotting medium where the data I want to keep is copied into new storage as the new storage comes on line. Three years is actually a very comprehensive warranty, as in three years new storage becomes available and affordable that makes keeping the old stuff on line questionable. Think of flowing new hardware under your data to keep the data afloat.
>> I am only starting to learn FreeNAS. It is fairly complex to understand what to do for the combination of optimum price/speed/reliability:
The solutions to speed/price/reliability are non-obvious in several ways. The need for speed is somewhat overrated. If you're streaming video, yep, you have a need for speed. That's a highly specialized need, and IMHO not a fundamental issue about how to store masses of data. Why base your data backup scheme on the need for super-speed for pieces of it at a time? They're two conflicting needs, and frankly, it is very cheap to build a separate very fast computer with fast data streaming needs that is backed up by slower, but more certain-to-survive bulk data storage.
It's worth remembering that "Good, fast, cheap; you may pick any two." appears to be a universal law. The only relief from this is that the definitions of "good", "fast" and "cheap" are defined by you, and that the computer industry continuously makes improvement possible. But at any moment, the maxim is true. And remember that "optimum" necessarily implies that none of them are fully realized.
>> I am noobly assuming only 1 zpool, and as big as possible a single VDEV RAID-Z2 ... single VDEV, in order not to sacrifice too many disks to
>> parity;
Not the best choice, IMHO. The intricacies of ZFS, managing pools and devs, and the failure rates/resilver rates indicate that under 12 drives more or less is best for a VDEV. See "ZFS best practices". I personally value my data very highly, and it is worth the extra price of one additional drive to get to RAIDZ3 from Z2 to ensure one more disk worth of immunity. My personal choices are a 4-drive or an 8-drive set of data with three drives of parity, giving a cost of 17% more or 10% more to go from RAIDZ2 to RAIDZ3, and get another disk-failure worth of immunity. Going to massive numbers of drives in RAIDZ3 increases the chance that you could have multiple disk failures while resilvering to fix the first failure. Keep numbers of disks in a Vdev modest.
>> But it might very well be that the expert wisdom in here says 1 VDEV is a bad idea ... optimum number of VDEV's
One giant Vdev is a bad idea. Buy using smaller, modest sized Vdevs, you compartmentalize the number of disks put at risk by one failing disk.
>> reading this thread: ... scared me. Destroying a whole pool(?) I thought ZFS was so safe this can never happen?
Rightly so. ZFS is a solution to certain common and difficult problems. It introduces non-obvious pitfalls that you have to go educate yourself against. That is, there are new and unusual ways to shoot yourself in the foot.
Fortunately, the practice of keeping a backup of your data, even when you use ZFS, innoculates you against this to a degree. You'll see this over and over: ZFS is not a backup. If you use ZFS for your live, working storage, you have no backups. ZFS is a protection against certain predictable problems with multiple disk data storage, including the failure of a disk. If you need to prevent data loss, the only good way is with a well-thought-out backup strategy, and this largely means at least one, preferably two entire backup stores, at least one of which is not in the same physical location as the other data.
It gets deep, doesn't it? :)
>> Speed/reliability/price, should I create 1 big VDEV with 4TB disks, or are there valid reasons to use more than 1 VDEV
(1) Pick any two :( (2) there are valid reasons to use modest sized VDEVs and more of them.
>> I've also read something about 'mirrored zpools', ... Does that mean you have a redundant zpool? Does that make sense? Is it relevant?
I have less experience with this terminology, but I think it may fall under the heading of "backup theory". FreeNAS and ZFS offer the function of replicating a zpool to a backup machine, and that's a good mechanism for doing backups of your ZFS pool(s). Maybe that's involved. I tend to gloss this over in my mind as "backup stuff, go think about your backup strategy", but that may not be correct.
Down at the bottom of this, you're about to learn a whole lot about storage systems and how to keep the data you already have. That's GOOD!
