setup for as much storage now, but expandability soon?

[digity]

I'm putting together a FreeNAS build and I need as much storage as possible (performance is not a priority). On hand, I have 10 x 2TB HDDs (mixed 3Gb/s & 6Gb/s, green, 5400 & 7200 rpm) and in due time will need more space which will force me to add more and bigger capacity HDDs. I figured for the time being I put the 10 HDDs in RAIDZ3 (for 14TB of space). Is this the best setup considering I need as much space now and will need to add more later?

 

[nojohnny101]

If you truly need as much space, go with a single raidz2 vdev. You have backups correct?

 

[digity]

This build will be the only backup server. Eventually I will build another backup server and store it off site for redundancy.

 

[digity]

Also, I was following that rule of thumb that says raidz2 for 6+ drives, raidz3 for 9+ drives. That's why I chose raidz3 since I have 10 drives. I assume raidz3 won't be a good idea anymore once I pass 12 drives

 

[gpsguy]

As @nojohnny101 said, you could put all 10 disks in a single RAIDz2 vdev.

When you need to expand do you plan to add additional disks or replace the existing disks? If the latter and assuming you put all 10 disks in a single vdev, you would have to replace all 10 disks with larger ones before your volume would expand.

If you can't afford to buy 10 disks at a time, you might consider starting with 2x6 RAIDz2 vdev's. You'll lose more space due to parity, but if you do in-place disk upgrades to expand your storage, you could see gains by doing 6 disks at a time.

 

[Spearfoot]

Also, I was following that rule of thumb that says raidz2 for 6+ drives, raidz3 for 9+ drives. That's why I chose raidz3 since I have 10 drives. I assume raidz3 won't be a good idea anymore once I pass 12 drives

Those guidelines aren't hard-and-fast... you can certainly set up a 10-disk RAIDZ2 array if you wish.

Your situation is a bit of a quandary, with conflicting requirements: maximum storage with easy expandability.

You can only expand a pool by adding vdevs (where a vdev is one or more disks in a stripe, mirror, or RAIDZ configuration). While not considered good practice, you can use different types of vdevs in a pool: so you could start out with a 10-disk RAIDZ2 array now, then add, say a pair of mirrored drives as an additional vdev later.

Everything @gpsguy said is on-point. Another option is to start out with 5 vdevs, where each vdev is 2 mirrored drives. You could expand the pool by adding additional mirrored pair vdevs or by replacing both drives in an existing mirrored vdev with larger-capacity drives. In the latter case, if you replace both 2TB drives in a mirror with 8TB drives, for example, you will gain 6TB of usable space.

 

[digity]

Ok, my storage requirements aside, adding vdevs to a pool is better practice than adding drives to a RAIDZ# vdev?

 

[Spearfoot]

Ok, my storage requirements aside, adding vdevs to a pool is better practice than adding drives to a RAIDZ# vdev?

You cannot add drives to a vdev.
The only way to expand a pool is to either add more vdevs, or replace each disk in a vdev with larger disks.
So the advantage of using 5 vdevs of 2-disk mirrors is that you can easily expand the pool a vdev at a time - i.e., by replacing both small disks in a mirrored pair with larger disks. Instead of having to replace all 10 disks in a raidz2 vdev. See?

 

[nojohnny101]

@Spearfoot gave a good explanation.

Have you see this though:
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Gives a good concise overview of FN and the varying terms in regards to storage. Should help to clear things up in addition to the great advice you've gotten on here.

 

[digity]

You cannot add drives to a vdev.
The only way to expand a pool is to either add more vdevs, or replace each disk in a vdev with larger disks.
So the advantage of using 5 vdevs of 2-disk mirrors is that you can easily expand the pool a vdev at a time - i.e., by replacing both small disks in a mirrored pair with larger disks. Instead of having to replace all 10 disks in a raidz2 vdev. See?

Got it, thanks!

Sent from my Nexus 6P using Tapatalk

 

[digity]

So the advantage of using 5 vdevs of 2-disk mirrors is that you can easily expand the pool a vdev at a time - i.e., by replacing both small disks in a mirrored pair with larger disks. Instead of having to replace all 10 disks in a raidz2 vdev. See?

So if one of the drives in a vdev fails, it does not bring down the whole pool, just that vdev, correct? Just replace the bad drive and resilver?

If both drives in a vdev fails, then the whole pool does go down, correct?

Sent from my Nexus 9 using Tapatalk

 

[Spearfoot]

So if one of the drives in a vdev fails, it does not bring down the whole pool, just that vdev, correct? Just replace the bad drive and resilver?

Correct. If a single drive in one of the mirrored pairs were to fail, the vdev (and pool) would be 'degraded'. This doesn't take town the vdev or pool, but it means they are at risk: if the other drive in that particular vdev fails too, the entire pool will fail.

When you replace the failed drive, that particular vdev - and only that vdev - would resilver. This is another advantage of mirrors: resilvering only involves the drives in a mirror, whereas with RAIDZ - whether RAIDZ1, RAIDZ2, or RAIDZ3 - every drive in the RAIDZ array is involved in resilvering.

If both drives in a vdev fails, then the whole pool does go down, correct?

Correct. A pool fails if any of its vdevs fail; so if you lose both drives in one of your mirrored-pair vdevs, you would lose that vdev and thus your pool as well.

 

[digity]

During volume/pool creation FreeNAS Corral suggested 3 pools of 3 x 2TB in RAIDZ1 (resulting in 10.92 TB total) so I went with that (and put the remaining 2TB drive in spare). I put three different model HDDs in each vdev so two of the same aren't together for added protection against 2 drives failing together/during resilvering.

Any further suggestions or objections (I can still redo all of this)?

 

[cobrakiller58]

people don't recommend raidZ1 much anymore because the odds are uncomfortably high for most people who like quickly accessing their data that a second drive will fail while resilvering the first failed drive. If you don't mind the potential risk of loosing the pool while fetching your back-up but I personally would not take the chance.

 

[gpsguy]

I would recreate the pool now. We discussed other options earlier in the thread. The responders in this thread wouldn't recommend 3 x (3x2TB RAIDz1) vdev's.

 

[digity]

So double drive failure is a high occurrence with RAIDZ1 during resilvering because of the nature of resilvering itself? Is this just a ZFS RAID thing or does this affect RAID5 arrays during rebuilding too?

 

[cobrakiller58]

There are a number of good articles listed on google why raid5 is no longer appropriate with the size of today's drives.

 

[digity]

I searched and I see that RAID5/RAIDZ1 (vis a vis MTBF/URE) is believed to be a killer, but the deaths don't add up. There doesn't appear to be much out there to justify fear over facts. Am I missing something?