36 Drive Config Best practice

[DomCon]

Hi Everyone,

Require assistance in creating the best drive (pool) configuration for redundancy with the following system specs.

System will be used for media storage, high performance drive config is not required.

1. Current recommendation is no more than 11 drives per vdev in RAID-Z3 config, please confirm this is true?
2. Any other recommendations?

System Specs:

TrueNAS-12.0-U3
Supermicro X8DT6-A-IS018
196GB 12x SAMSUNG 16GB PC3L-8500R DDR3 REGISTERED ECC
2x Intel(R) Xeon(R) CPU E5603 @ 1.60GHz
36x 3TB ATA Hitachi HUA72303 - Dedicated to Storage
SSD used for OS

EMC ISILSON NL400 Series re-purposed

Thanks,

DC

 

[Ericloewe]

1. Current recommendation is no more than 11 drives per vdev in RAID-Z3 config, please confirm this is true?

Well, that's a rule of thumb. If you're dealing with large files and relaxed performance requirements, you can go for 12-wide, it's only marginally worse than 11-wide and lets you use the multiple-of-12 bays you have in many servers.

That said... 36x 3 TB? That seems wasteful.

 

[NugentS]

I guess he doesn't have to worry about power - or the discs are very cheap

 

[Arwen]

Yes, the system board and amount of disks does seem to be in the class of house warmer. (Maybe good in the Canadian winters...)

On the subject of disk layout, instead of 3 x 11 or 12 wide RAID-Z3, you could go 4 x 9 wide RAID-Z2. Reduce the rebuild time, and add another vDev for more IOPS. Here are the rough data disk figures;

3 x 11 wide RAID-Z3 = 24 data disks (does leave 3 disk slots un-used, for drive replacement? Or backup disks?)
3 x 12 wide RAID-Z3 = 27 data disks
4 x 9 wide RAID-Z2 = 28 data disks

Those are rough figures and a case could be made for narrower RAID-Z2 over wider RAID-Z3. As long as you keep up with disk failure change outs, narrower RAID-Z2 should be fine.

Of course, if their are external factors that end up causing more drive failures than expected, (like heat or poor power supplies), that could impact reliability, where RAID-Z3 could be a better choice.

 

[DomCon]

Thank you everyone for the advice, I like the 4x9 config, going to do the math for the total disk space for each scenario.

@Ericloewe, for my knowledge, what’s your thinking with the comment “that seems wasteful”? All hardware was free, so trying to use it the best I can for large storage files.

Thanks,

Dom

 

[jgreco]

Well, that's a rule of thumb. If you're dealing with large files and relaxed performance requirements, you can go for 12-wide, it's only marginally worse than 11-wide and lets you use the multiple-of-12 bays you have in many servers.

11 wide with a spare is also quite reasonable. This would give you three spares in a 36 drive config.

 

[WI_Hedgehog]

I guess he doesn't have to worry about power - or the discs are very cheap

"There is no such thing as a free puppy." That's a heck of a drive enclosure.

How big of an array are those HUA72303's rated for? 36 is quite a few... shimmy shimmy shake.

"It’s important to know the number of drives that will be used in a NAS array to help ensure that the drive can handle vibration levels and still deliver the expected performance." srouce: NAS Drive Selection Guide -Seagate

 

[jgreco]

shimmy shimmy shake.

 

[Ericloewe]

@Ericloewe, for my knowledge, what’s your thinking with the comment “that seems wasteful”? All hardware was free, so trying to use it the best I can for large storage files.

Mostly power. Very roughly speaking, that's going to be three times as much power as 12 10 TB disks, plus power for expanders and the like. "Free" has a quality all its own, but the hidden costs can add up.

 

[WI_Hedgehog]

I hope his hard drives turn out better than Cindy Lou Who (Taylor Momsen).

 

[DomCon]

Thanks for all the feed back, hardware being used is all repurposed enterprise SAN hardware.