Fusion pool metadata vdev size for a 45 drive Supermicro box

[Chris Moore]

Thanks in advance!

Sorry I missed out on this conversation when it was happening. What did you decide to do in the end? Have you received the hardware yet? Done any testing with it? I am curious to find out results with the metadata vdev for similar reasons to your original question. I am expecting to get new hardware in the near future and I am trying to decide how to configure it.

 

[Constantin]

I've been thinking about the safety issue for the past couple of days and I think I would rather have a 3-way mirror for the metadata, just so it would match the redundancy of the rest of the pool (raidz2). I wouldn't technically need this for the backup box, but the bucket of money only comes by once. Also we set the backup boxes up to be able take over for the production boxes if need be.

Seeing that the whole pool goes when the metadata sVDEV goes, I concur in general with adding more sVDEV metadata redundancy to your pool. Depending on how you play the probabilities, I'd consider a 3- or a 4-way mirror for production metadata with that many drives. That's an awful lot of data to back up if the pool crashes.

I'd also monitor those SSDs carefully to see how many DWPD they're experiencing and then pulling and replacing them well in advance of their Intel-designated retirement. Time value of money and all that. Plus, have some cold but qualified SSD and HDD spares on hand.

As for the rest of the discussion, I cannot speak to the technical merits of your use case as I do not have the requisite experience with all that. I have had hardware RAID bite me several times in the butt, so that's one reason I'm now on FreeNAS vs. continuing to rely on RAID5, BTRFS, etc. At least with FreeNAS I know I am as close to pool protection as I am going to get at the present time. Unlike RAID5, BTRFS, etc. which may give the illusion of protection until you discover an issue with the hardware RAID controller, the BTRFS implementation, etc.

 

[ornias]

Seeing that the whole pool goes when the metadata sVDEV goes, I concur in general with adding more sVDEV metadata redundancy to your pool. Depending on how you play the probabilities, I'd consider a 3- or a 4-way mirror for production metadata with that many drives. That's an awful lot of data to back up if the pool crashes.

My 2 golden ruls are:
- Always have n+2 redundancy (in this case: minimum of a 3 way mirror), because you want to be able to repair corrupted (metadata)blocks during a rebuild.
- Regarding failure chances: The SSD mirror equivalent of a raidz HDD pool, is one less disk of redundancy. So RaidZ3->triple mirror.
This basically leads to conclude: Always use a threeway mirror for metadata as a minimum. Being paranoid is always an option to go fourway.