itsScientific
Cadet
- Joined
- Feb 11, 2016
- Messages
- 2
I'm trying to understand my [planned] zpool expansion options. Part of that comes down to how ZFS handles its "dynamic striping" across vdevs. Solaris documentation leaves that pretty vague, to the effect of "does good." I think my question boils down to: Which most affects the amount of data striped to each vdev, the % free or the raw amount free in comparison with available vdevs?
The plan starts simply - 2-way mirror vdevs. I'm optimizing for speed, ease of management, and ideally, incremental expansion capabilities. One of the arguments for mirror vdevs is that you can easily expand in two ways - resilver with larger drives (quick with 2 per vdev), and/or add more mirror vdevs over time.
This got me thinking, surely, it would be bad if you took a 80% full zpool and merely added one vdev - ZFS would hit that new, empty vdev hard because the rest of the vdevs are quite full. Perhaps even until the new vdev hit 80% (?). In that case, you'd lose most benefits of striping to the high number of vdevs that a 2-way mirror layout gives you - not a slick means of expansion. It's incremental, but nullifies a key advantage of a mirror-based pool.
Then I figured, when I do add new capacity, the drives will likely be an increase on the originals. Perhaps a resilver-based increase + adding back the smaller drives will be optimal?
Say I have two vdevs, each 2 TB (2x2TB mirror) = 4 TB zpool. At the point I need more capacity, I'd primarily get it by individually resilvering drives to 4 TB each. If the drives used to be 80% full, they're now down to 40%. Great! Now I have an 8 TB pool. But I still only have 2 vdevs and no striping speed increase as capacity increased. But hey, why let the old 2 TB drives go to waste? If I add them back as clean vdevs, I'd now have 4 vdevs to stripe over, 12 TB of storage, and roughly equal TB free (i.e. about 2 TB) per vdev, though not equal % free. If ZFS prioritizes % full, I'd lose some benefit, as data is more heavily distributed to the empty 2TB-based vdevs; however, if it prioritizes available capacity, the drives will quickly be in step, with equal striping across 4 vdevs.
…I’m considering all this because ZFS does not provide a way to re-balance data on vdevs. I'm ultimately stuck with "rebuild the zpool if you want to optimize striping in a new layout." I'd much prefer a "re-balance because I changed the zpool layout" feature, but I get it - that's damn complicated. So what's the best middle ground?
Secondly, without me digging in to the code, are there any guidelines that explain what ZFS attempts to optimize about dynamic striping - percentage full, speed of particular vdev, error conditions, etc...?
(Good link on performance basics : http://constantin.glez.de/blog/2010/06/closer-look-zfs-vdevs-and-performance)
The plan starts simply - 2-way mirror vdevs. I'm optimizing for speed, ease of management, and ideally, incremental expansion capabilities. One of the arguments for mirror vdevs is that you can easily expand in two ways - resilver with larger drives (quick with 2 per vdev), and/or add more mirror vdevs over time.
This got me thinking, surely, it would be bad if you took a 80% full zpool and merely added one vdev - ZFS would hit that new, empty vdev hard because the rest of the vdevs are quite full. Perhaps even until the new vdev hit 80% (?). In that case, you'd lose most benefits of striping to the high number of vdevs that a 2-way mirror layout gives you - not a slick means of expansion. It's incremental, but nullifies a key advantage of a mirror-based pool.
Then I figured, when I do add new capacity, the drives will likely be an increase on the originals. Perhaps a resilver-based increase + adding back the smaller drives will be optimal?
Say I have two vdevs, each 2 TB (2x2TB mirror) = 4 TB zpool. At the point I need more capacity, I'd primarily get it by individually resilvering drives to 4 TB each. If the drives used to be 80% full, they're now down to 40%. Great! Now I have an 8 TB pool. But I still only have 2 vdevs and no striping speed increase as capacity increased. But hey, why let the old 2 TB drives go to waste? If I add them back as clean vdevs, I'd now have 4 vdevs to stripe over, 12 TB of storage, and roughly equal TB free (i.e. about 2 TB) per vdev, though not equal % free. If ZFS prioritizes % full, I'd lose some benefit, as data is more heavily distributed to the empty 2TB-based vdevs; however, if it prioritizes available capacity, the drives will quickly be in step, with equal striping across 4 vdevs.
…I’m considering all this because ZFS does not provide a way to re-balance data on vdevs. I'm ultimately stuck with "rebuild the zpool if you want to optimize striping in a new layout." I'd much prefer a "re-balance because I changed the zpool layout" feature, but I get it - that's damn complicated. So what's the best middle ground?
Secondly, without me digging in to the code, are there any guidelines that explain what ZFS attempts to optimize about dynamic striping - percentage full, speed of particular vdev, error conditions, etc...?
(Good link on performance basics : http://constantin.glez.de/blog/2010/06/closer-look-zfs-vdevs-and-performance)