HOW to Expand(?)/Rebuild Storage Pool (4 vDevs Raidz1 vs. 2 vDevs Raidz2)

[G3orgios]

Hello Community,

After quite a few years of operation, my TrueNas system has reached about 90% of space occupancy (...quite over the recommended 80%!) and thus I need to consider HOW to expand (or rebuild?) it. Here are the existing specs:

TrueNAS 13.0
Model: Supermicro SuperStorage 6049P-E1CR36L
CPU: 2 x Intel Xeon silver 4114 @ 2.2GHz
RAM: 8 x 16GB ECC DDR4
Boot: 2 x 128GB Kingston 2.5" (Raid1)
Jails: 2 x 960GB Samsung PM963 NVMe M.2 (Raid1)
Storage: 12 x 12TB HGST 3.5" SAS 4Kn (3vdev x 4disks, Raidz1)

The main point is to expand the capacity. The "easy" way of doing so is to make another vDev of 4 x 12TB SAS 4Kn disks Raidz1 and be done. HOWEVER, I am considering switching to Raidz2 and just two (2) vDevs (all data are replicated in another machine too, if needed to rebuilt). I think of this to have lower disk-failure risk (Raidz2), since the available space is the same:

• 4vdevs x 4x12TB disks each, Raidz1 = 4 x 36TB = 144TB
• 2vdevs x 8x12TB disks each, Raidz2 = 2 x 72TB = 144TB (I am aware that this has slower performance than the 4 vDevs but this is not critical)

What are your thoughts on this?

Also, is vDev expansion (add drives) released or not yet? If so, does it makes sense to switch to "legacy" Raid design with just 2 vDevs, expanding them by an extra drive on each when needed? Are vDevs become "unstable" over an amount of disks? The top priorities are DATA INTEGRITY and CAPACITY. R/W performance is NOT critical.

Thank you.

George

 

[Morris]

Hello Community,

After quite a few years of operation, my TrueNas system has reached about 90% of space occupancy (...quite over the recommended 80%!) and thus I need to consider HOW to expand (or rebuild?) it. Here are the existing specs:

TrueNAS 13.0
Model: Supermicro SuperStorage 6049P-E1CR36L
CPU: 2 x Intel Xeon silver 4114 @ 2.2GHz
RAM: 8 x 16GB ECC DDR4
Boot: 2 x 128GB Kingston 2.5" (Raid1)
Jails: 2 x 960GB Samsung PM963 NVMe M.2 (Raid1)
Storage: 12 x 12TB HGST 3.5" SAS 4Kn (3vdev x 4disks, Raidz1)

The main point is to expand the capacity. The "easy" way of doing so is to make another vDev of 4 x 12TB SAS 4Kn disks Raidz1 and be done. HOWEVER, I am considering switching to Raidz2 and just two (2) vDevs (all data are replicated in another machine too, if needed to rebuilt). I think of this to have lower disk-failure risk (Raidz2), since the available space is the same:

• 4vdevs x 4x12TB disks each, Raidz1 = 4 x 36TB = 144TB
• 2vdevs x 8x12TB disks each, Raidz2 = 2 x 72TB = 144TB (I am aware that this has slower performance than the 4 vDevs but this is not critical)

What are your thoughts on this?

Also, is vDev expansion (add drives) released or not yet? If so, does it makes sense to switch to "legacy" Raid design with just 2 vDevs, expanding them by an extra drive on each when needed? Are vDevs become "unstable" over an amount of disks? The top priorities are DATA INTEGRITY and CAPACITY. R/W performance is NOT critical.

Thank you.

George

The first option is fast and simple if your have the slots and ports. Just add the vdev and you are done. The second option requires deleting the pool and then creating the new one and then copying the data over from your other system. That will take a long time.

 

[danb35]

is vDev expansion (add drives) released or not yet?

It continues to be some distance in the future. I do think we're going to see it, but I wouldn't delay in doing any required work/expansion to wait for it--I'd be surprised if we see it within the next year.

 

[G3orgios]

Since there is not much (any?) light for the vDev-expand feature on the horizon, I guess I will have to stick to adding a 4th Raidz1 vDev and be done with it, leaving Raidz2 for the future!

Still, Raidz2 is my preferred choice (not done initially due to available disks vs. needed capacity). Is there any rule of thumb on how many Disks a vDev can effectively support? I still believe that a "big" Raidz2 or even Raidz3 vDev is fine when CAPACITY and DATA SAFETY are the priorities (not speed), as long as this vDev can be expanded! Otherwise, we have to stick to keep adding vDevs...

 

[Etorix]

Boot: 2 x 128GB Kingston 2.5" (Raid1)
Jails: 2 x 960GB Samsung PM963 NVMe M.2 (Raid1)

I hope you mean "mirror" here, not hardware RAID1.

(all data are replicated in another machine too, if needed to rebuilt). I think of this to have lower disk-failure risk (Raidz2),

You are correct that raidz2 is safer than raidz1, and that the only way to migrate is to destroy the pool, rebuilt it with the new geometry and restore all data from backup.

Also, is vDev expansion (add drives) released or not yet? If so, does it makes sense to switch to "legacy" Raid design with just 2 vDevs, expanding them by an extra drive on each when needed?

Not sure what you mean by "legacy" here.
If or when vdev expansion becomes available, it could allow to change a 4-wide raidz1 into 5-wide raidz-1, or 8-wide raidz2 into 9-wide raidz2, but NOT to change raidz1 into raidz2.

Are vDevs become "unstable" over an amount of disks?

The general advice is not to go beyond 10-12 wide. So starting with 8-wide raidz2 vdevs and expanding these one drive at a time until, say, 20-wide is not a viable strategy, if that's what you're thinking. Once the data is on 8-wide raidz2, the path to increase capacity is to add further 8-wide raidz2 vdevs. (Or replace drives by larger ones.)

The top priorities are DATA INTEGRITY and CAPACITY. R/W performance is NOT critical.

With these priorities, especially data integrity, the logical course would be to migrate to raidz2 now. Adding a fourth raidz1 vdev and waiting to migrate only means there will be even more data held in not-so-secure raidz1 and even more data to restore in order to migrate to raidz2.

 

[G3orgios]

I hope you mean "mirror" here, not hardware RAID1.

Raid1 = Mirror, right?

You are correct that raidz2 is safer than raidz1, and that the only way to migrate is to destroy the pool, rebuilt it with the new geometry and restore all data from backup.

+1!

If or when vdev expansion becomes available, it could allow to change a 4-wide raidz1 into 5-wide raidz-1, or 8-wide raidz2 into 9-wide raidz2, but NOT to change raidz1 into raidz2.

Indeed, "just" having the ability to add disks if fine, WHEN it will be available!

The general advice is not to go beyond 10-12 wide. So starting with 8-wide raidz2 vdevs and expanding these one drive at a time until, say, 20-wide is not a viable strategy, if that's what you're thinking. Once the data is on 8-wide raidz2, the path to increase capacity is to add further 8-wide raidz2 vdevs. (Or replace drives by larger ones.)

With these priorities, especially data integrity, the logical course would be to migrate to raidz2 now. Adding a fourth raidz1 vdev and waiting to migrate only means there will be even more data held in not-so-secure raidz1 and even more data to restore in order to migrate to raidz2.

Right you are stating "migrate to raidz2 now"! I am also in favour to switch now than incur more pain later (more disks/data to deal with). I am inclining to switch to problably 3 x 6 or 8 disks vDevs Raidz2 (probably 8 for better cost efficiency!).

Thank you for the feedback!

 

[sretalla]

Raid1 = Mirror, right?

In technologies that aren't ZFS, yes, but in ZFS, what you would talk about is Mirrors.