Limit on number of disks in a pool?

SubnetMask

Contributor
Joined
Jul 27, 2017
Messages
129
I've been searching around, and maybe my 'search-fu' is weak, while the number '10' seems to ring a bell, I haven't found anything that gives a 'solid' disk per pool limit.

Currently, I have an eight disk RAIDZ2 pool as my 'main' datastore - this datastore is home to two VMDK files attached to the same VM - one is for general data and the other is for media (movies, music, etc). As such, I don't need to be able to push 10,000 IOPS or 1.2GB/s from it (wouldn't complain if I could though) - capacity and reliability are most important for this particular pool. My pool that VMs boot off is a 10 disk stripe/mirror.

That being said, I'm in the process of migrating this pool to a 10 disk RAIDZ2 by way of migrating the VMDKs off to another pool and once empty, deleting and re-creating it as a larger pool, since we can't do live pool size migrations of this nature (Although I just read it's on the horizon....... somewhere... out there). But while I'm waiting for the data to be moved off, I've been thinking about making it a Z3 instead of a Z2 (Again, size and reliability are the most important). But I don't want to do this if 10 is the limit - a 10 disk Z2 and 11 disk Z3 will have roughly the same capacity and the 10 disk Z2 (or 11 disk Z3) is where I want to be in terms of capacity. Is there any reason I couldn't do an 11 disk RAIDZ3 for this pool? If it matters with regards to the pool size, the FreeNAS 'Controller' is currently a Poweredge R620 with two E5-2640's, 96GB RAM and two SAS2 LSI HBAs, and based on testing so far and some possible further testing, will almost certainly be migrating over to a FC630 with a single E5-2699v3, 256GB RAM and two LSI 9300 SAS3 HBAs.
 
Last edited:

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,703
s there any reason I couldn't do an 11 disk RAIDZ3 for this pool?
11 is fine, but you need to consider resilver and scrub times... if you're happy with slightly slower/longer times for those two things, no issue... the "limit" which people say is good to stop at is 12, but most keep it under 10. The technical limit isn't really there, so you can go as high as you see fit if you're prepared to deal with the resilver/scrub and loss of IO/performance for that many disks (the whole pool is capable of as many IOs as the slowest disk in each VDEV added together).
 

SubnetMask

Contributor
Joined
Jul 27, 2017
Messages
129
Thanks - The one resilver I've had due to a failed disk really wasn't bad - I've got my resilver set to give priority to the resilver all the time - I'll sacrifice some performance degradation -if it's even ends up being noticeable - in return for a faster resilver. Scrubs really have never been a noticeable issue. All disks (once this migration is done) are SAS3 (but currently linked to the HBA via two SAS2 expanders) HGST Helium 8TB SAS disks.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
I've been searching around, and maybe my 'search-fu' is weak, while the number '10' seems to ring a bell, I haven't found anything that gives a 'solid' disk per pool limit.
The limit on the number of disks in a vdev is "anecdotally" 10 (11 if it is RAIDz3) but that is really more about performance. I have seen a vdev with 45 drives in it. The performance was HORRIBLE, but it works. I strongly advise against it.

The limit on drives in a pool, is that there is no limit. Drives are grouped into vdevs and vdevs are grouped into pools, here is a general guide on how that works:

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://www.ixsystems.com/community...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Overview of ZFS Pools in FreeNAS from the iXsystems blog:
https://www.ixsystems.com/blog/zfs-pools-in-freenas/

Terminology and Abbreviations Primer
https://www.ixsystems.com/community/threads/terminology-and-abbreviations-primer.28174/

Where I used to work, we had a server with 124 drives in a pool, but there was another department with a system that had around 300 drives. It just requires the correct hardware to support it.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
two LSI 9300 SAS3 HBAs
Why two HBAs? Each of them can control 1024 drives by way of SAS expanders. Here are links to a couple of informative videos that might help your understanding:


 

SubnetMask

Contributor
Joined
Jul 27, 2017
Messages
129
Why two HBAs? Each of them can control 1024 drives by way of SAS expanders. Here are links to a couple of informative videos that might help your understanding:



Yes, I'm aware - I have two HBAs for redundancy. One HBA is connected to the first expander in my enclosures and the other is connected to the other expander (I have the dual expander EL2 backplanes in my enclosures)
 

Jessep

Patron
Joined
Aug 19, 2018
Messages
379
Wouldn't it make more sense to have more vdevs in one pool?
More performance, more redundancy, easier upgrades (only replace drives in single vdev) etc.

6 drive RaidZ2 vdev is pretty common, add as needed.
 
Top