24 drive raid setup recommendations?

[jacknasty]

I'm building a 24 drive raid with Freenas. I've heard that it is unwise to make pools larger than 11 drives SO I was thinking of creating 3 pools of 8. I had a few questions:

1) Does this make sense?
2) Would it be possible/easy to view them as a single logical volume? That would be immeasurably preferable to three distinct volumes.
3) Am I not considering something that is important and that I simply haven't thought of?

Thanks!

 

[survive]

Hi jacknasty,

First thing I would do is read through cyberjock's excellent doc here:

http://forums.freenas.org/showthrea...explaining-VDev-zpool-ZIL-and-L2ARC-for-noobs!

It explains the "rules" for constructing zfs pools and all sorts of other things you need to know to make a successful setup.

That said, 12 disks is about as "big" as you want to make a virtual device, so your plan to make 3 is a wise one. One thing you need to know is you are really talking about 2 things when you make a zfs file system....there's the pool and the virtual devices (vdevs) that the pool is constructed on. In your case above you would actually make 3 8-drive vdevs and add all 3 into the pool to create your one large volume.

How you construct your pool really depends on what you need out of it. Do you want maximum speed? maximum capacity? somewhere between the 2? Post back with what you plan to use it for and we can make a better recommendation.

-Will

 

[jacknasty]

Thank you so much for the detailed response! I'll read that link as soon as I finish this post.

I'm building this system to host a about 50+ terabytes of my music/video/etc. It will have at most one or two concurrent connections so throughput and availability are not nearly as important as reliability in this case. It will be a multimedia server.

Thanks again! Reading the link now.

 

[jgreco]

Two vdevs of 11 disks in RAIDZ3 leaves space for two warm spares.

That gives optimal sizing for RAIDZ3, and with 4TB drives, gives you 32TB vdevs, or 64TB for 22 drives worth of space. With that many drives, my cynical nature says you're guaranteed to have one fail unless you arrange for spares, in which case you're guaranteed to have no failures, making you think you wasted money.

 

[jacknasty]

Oh, I forgot to mention these are 4TB drives and I want at least 80TB if possible after parity.

 

[cyberjock]

There's no combination of 4TB drives that will give you at least 80TB. That's 20 drives of actual space and only 4 parity. Far from anything I'd come close to recommending.

I have 24 drives in my server, and I think jgreco's recommendation of 2 vdevs of 11 drives on a RAIDZ3 is the best option I'd recommend.

 

[titan_rw]

How about this alternative:

4 vdevs of 6 drives in z2? This is also 16 drive capacity. Instead of the 'warm spares' of the double z3, this 'integrates' the spares into the pool. Also 'optimal' number of drives per vdev if you subscribe to that sort of thing. Likely to be faster as well. This is 33% parity.


Another option for more capacity, but less redundancy:

3 vdevs of 8 drives in z2. This is 18 drive capacity. 'non optimal' number of drives per vdev, but for media storage, this is unlikely to matter. This is 25% parity. With a cold spare or two kept in a drawer, I'd probably consider this.


The 2 vdevs of 11 drives in z3 would be 27% parity, not counting the warm spares.

Note the parity numbers don't reflect any differences in raidZ levels, only the grand total of redundant data assuming the 'correct' drives fail. For example, 2 vdevs of 3 drives in z1 is 33% parity. One vdev of 6 drives in z2 is also 33% parity, but is 'better' from a resiliency perspective. However it's also potentially 'slower'.

 

[jgreco]

4 vdevs of 6 drives in z2? This is also 16 drive capacity. Instead of the 'warm spares' of the double z3, this 'integrates' the spares into the pool. Also 'optimal' number of drives per vdev if you subscribe to that sort of thing. Likely to be faster as well. This is 33% parity.

It integrates the spares into the pool, yes. It also means that a triple drive failure can dump your entire pool, though you have to suffer some seriously bad luck in order to land all three failed drives in a single vdev.

From my perspective, this is less desirable because we'll be pulling the drive for RMA. With 2x11+2, an array temporarily becomes 2x11+1 while that failed drive is out for several weeks, and indeed can even suffer another failure/rebuild-on-spare event, all while continuing to run at full non-degraded mode (except during the rebuild of course).

When a 4x6 fails a drive, you either need a cold spare standing by, or you need to suffer reduced redundancy (and the performance hit of degraded mode) while the drive is out for RMA; if you lose a second drive, you have an approx. 1-in-4 chance of actually losing redundancy entirely.

From an operational perspective (not relevant to home users I know), not having to dispatch someone to replace a drive at 3AM is a very attractive thing. With the two warm spares, it means that a remote administrator can virtually swap in a disk twice without anyone being on site, so if you think about it like that, it is kind of like RAIDZ4 with the ability to suffer a disk failure and not stay degraded pending a physical disk swap.

I am not convinced that what is probably a mild performance gain offsets that, but I enjoy the debate so don't hesitate to tell me I'm full of it (just tell me why!).

Another option for more capacity, but less redundancy:

3 vdevs of 8 drives in z2. This is 18 drive capacity. 'non optimal' number of drives per vdev, but for media storage, this is unlikely to matter. This is 25% parity. With a cold spare or two kept in a drawer, I'd probably consider this.

I agree that the speed issue is unlikely to be a problem for media storage, but then again, in this era, even a fairly small slow server ought to be able to serve up media without being too stressed. Once you're talking more than six or eight disks, the server ought to be sufficiently large and there ought to be sufficient IOPS to handle media retrieval.

If absolute storage is the overriding concern, I'd say push out to 12-drive vdevs at RAIDZ3. There's anecdotal evidence of people having success with vdevs up to 18 or even 24 drives, but it is also pretty easy to find reports of failures, and trying to find a way to back up a 24-disk pool so you can blast it and reload it would ... be annoying to the average user. But since your average storage chassis will be in a 12, 24, or 36 disk version, 12 is really a compelling number of disks to be working with.

 

[titan_rw]

I agree, in a datacenter environment, warm spares would be good.

For myself in a home environment, I don't consider it necessary. I'd rather 'do something else' with the drive(s). Either increase vdev drive count, or increase raidz level (if not already at z3), or rearrange vdev config some other way.

Yes, I agree on the triple drive failure being a concern with z2. Not purely the chance that 3 drives die completely, but that 3 drives could develop separate issues within a reasonable time frame. But in the case I mentioned, the vdevs would be only 6 disks wide. I'd be comfortable with z2 with 6 wide vdevs.

I don't know how your rma process works, but (as a plain old consumer) I've always had good luck with advanced replacements from the various drive manufacturers. My experience has been 1-2 days to get the replacement(s), plus whatever resilver time. Like I said, in a home environment, I'm always "at" the server, so getting local access is as simple as walking into the correct room.

I agree, 12 drive z3 vdevs is also interesting. I wonder what the differences would be between:

2 vdevs of 12 drives in z3 VS 3 vdevs of 8 drives in z2? Both are 18 drive capacity.

I imagine the z3 setup would possibly be slightly better for resiliency? As any 3 drives in half the pool can die, as opposed to any 2 in a third of the pool? (more flexibility on 'failure groups')?

 

[jgreco]

Bad case RMA process might work like this: Problem occurs. Obtain replacement drive (from inventory, or from vendor, whatever). Pay on-site smarthands $*** (~$200/hr) to take FedEx-overnight-shipped drive from shipping area into cage and coordinate via phone to make sure they're actually removing the right drive tray. On fourth attempt, 30 minutes later, they finally get it right and then go to get a screwdriver. 15 minutes later they return, manage to replace the drive in the tray, and get it back in to the server, which sees it. Since one hour and five minutes have elapsed, they bill for two hours (~$400). So add the cost of drive. Plus the cost of FedEx overnight. Now approaching $700-$800. You don't actually make the mistake of asking them to ship the drive, which would turn into a quagmire of additional hours and costs. You just have them leave it (or even discard it) for someone to pick up at a later date, at which point it gets RMA'd and maybe given away because the failure rate of "recertified" drives is substantially higher than the failure rate of a new drive, so you don't want to have to go through that whole thing again because you deployed a LESS reliable drive. And note that this whole process still probably took ~24 hours to get the drive replaced.

So my improved process works more like this: engineer sufficient spare capacity. In event of failure, sit in front of monitor and type some commands, right when problem is noticed, problem fixed immediately. Drop an e-mail asking any colleagues in the area if they'd like a case of beer in exchange for an hour's worth of time. UPS Ground them a drive. Fix happens a week later. Don't really care what happens next, maybe have them ship it to RMA center for us, or let them keep it and RMA it themselves (most techies can find a use for free storage).

So a few points:

1) The cost of doing things in a crisis at a faraway data center absolutely dwarfs the value of the drive.

2) You don't want to redeploy someone else's "fixed" drive back into that environment. As a result, advance replacement is mostly useless unless you like gambling on the reliability of recertified drives.

3) Planning for failure can save your butt. ;-)

 

[titan_rw]

Yea, not using 're-certified' drives limits you to buying brand new drives in the event of drive failure. I agree that might be a good option for enterprise / corporate environments.

I wouldn't be able to afford that personally. I have no problem with re-certified drives here. The ones I have continue to work just fine.

Definitely if the server is remote, warm spares would be a god send. Makes the actual replacement much less time sensitive. Being my server is local, and I'm ok with re-certified drives, and getting an advanced shipment for an rma is only $15 or so, I don't bother with warm spares. If it was an emergency, there's lots of computer stores locally I could pickup new drives for use as replacements.