Replacing pool with smaller drives

[Ziggy]

I currently have a 4TB mirrored pool on spinning rust that is only using 16% of its capacity and I would like to replace them with two 2TB SSDs. My FreeNAS Mini 2.2 has two other pools and thus I cannot attach the two new 2TB SSDs in order to create another pool to which I might be able to replicate the data from 4TB to 2TB. Given I can only find answers that tell me this is impossible without destroying the pool and starting from scratch, I am wondering if the following might work:

1) I have a separate machine running ZFS on Ubuntu in the same network.
2) Install the 2x2 SSDs in a new pool (Pool2) on Ubuntu.
2) 'ZFS send' a fresh snapshot of the existing 4TB pool (Pool1) to Pool2.
3) Disconnect Pool2 from Ubuntu.
4) Disconnect Pool1 from TrueNAS and import Pool2.
5) Point TrueNAS to Pool2 as the pool for Apps and VMs as is currently the case for Pool1.

Thanks in advance.
TrueNAS-SCALE-22.12.4.2

 

[danb35]

Simpler, but removes redundancy until the process is complete:

• Detach one disk from the 4 TB mirror
• Add one of the 2 TB SSDs to that pool as a new vdev
• Remove the other 4 TB disk from that pool
• Add the second SSD as a mirror of the first

 

[Etorix]

Simple and with redundancy:

• Export the other pool (do NOT wipe the drives when asked)
• Remove its drives to free two slots
• Add SSDs and create a pool
• Replicate from the old HDD pool to the new SSD pool
• Export the HDD pool
• Put back the other pool and import

 

[danb35]

Simple and with redundancy:

If you're going to remove a different pool, then there's a much simpler method available, which still keep redundancy:

• Export the other pool and remove its drives
• Add the 2x 2 TB SSDs as a new mirror vdev to Pool1
• Remove the existing 2x 3 TB spinner vdev from that pool
• Reinstall the other drives and re-import that pool

 

[Ziggy]

Simpler, but removes redundancy until the process is complete:

• Detach one disk from the 4 TB mirror
• Add one of the 2 TB SSDs to that pool as a new vdev
• Remove the other 4 TB disk from that pool
• Add the second SSD as a mirror of the first

Thanks guys. Appreciate the quick response. I’m a little confused, so I’ll reply to each in turn. First up, danb35:
I thought you couldn’t directly replace a larger drive with a smaller one (btw, my bad: it’s a 3TB RAID1 mirror, not 4TB, but I guess that wont impact my query), but you’re not suggesting replacing - and then resilvering, as would be the normal process with substitute drives of the same size or larger - right? You’re saying that by adding a 2TB SSD vdev the data on the original 3TB mirrored vdev is copied to the SSD, and that adding the second SSD effectively runs a resilvering process from the first SSD once the second 3TB HD has been removed. Anyway, it doesn’t matter if it works. The pool is only a workhorse for apps and VMs and worst case scenario I would start from scratch, but obviously would prefer not to have to do this.

 

[Ziggy]

Simple and with redundancy:

• Export the other pool (do NOT wipe the drives when asked)
• Remove its drives to free two slots
• Add SSDs and create a pool
• Replicate from the old HDD pool to the new SSD pool
• Export the HDD pool
• Put back the other pool and import

Next up, Etorix;
I have never exported a pool previously. Am I doing this to a - for example - temporarily USB attached drive with a temporary pool created on that disk for this purpose? And then once the new SSD pool has been created, importing that exported pool to the SSD pool? I don’t understand bullet points 5 and 6 if I’ve already exported and imported the relevant 3TB pool. Hope that makes sense.

 

[danb35]

I thought you couldn’t directly replace a larger drive with a smaller one

That's correct.

You’re saying that by adding a 2TB SSD vdev the data on the original 3TB mirrored vdev is copied to the SSD

Close. It's not when adding the second vdev that the data is moved over; it's when you remove the first one. That's a fairly recent addition to ZFS that landed (in TrueNAS) with CORE 12 and I think has always been in SCALE--you can remove vdevs from your pool, as long as the pool consists only of single-disk or mirror vdevs. And then when you add the second SSD as a mirror of the first, the data is copied in a resilver operation.

I have never exported a pool previously.

Exporting a pool is simply what removes it from the system--it does not include copying that pool's data anywhere else.

 

[Etorix]

"Exporting" is ZFS parlance for "unmounting" a pool. No need for a temporary storage. Under Storage>Pool, click on the gear and select "Export"—do NOT wipe the drives, and do not delete configurations. Remove the drives and use their slots. When you're done, and you have two free slots (not necessarily those where the exported pool was), plug back the drives of the exported pool and "Import".

You have two pools, A and B, with two drives each (A1, A2, B1, B2). You want to reduce A, using SSDs C1 and C2 instead.

1. Export pool B.
2. Remove B1, B2.
3. Plug in C1, C2. Make pool C.
4. Replicate from pool A to pool C.
5. Export pool A, and remove A1, A2.
6. Put back B1, B2 and import pool B.

You end up with C1, C2, B1, B2 in the NAS, and A1, A2 which still hold a copy of pool A the time you did the transfer. You still have to redo the configuration of the shares which used pool A so they now point to pool C (or you rename 'C' to 'A').

In @danb35's procedure:

1. Export pool B.
2. Remove B1, B2.
3. Plug in C1, C2. Add a vdev to pool A with C1,C2 (Storage>Pool>(gear)>Add vdevs)
4. Remove the A1,A2 vdev (Storage>pool>(gear)>Status and then (…)>Remove)
5. Wait for ZFS to move the data and automatically offline A1, A2. Remove A1, A2.
6. Put back B1, B2 and import pool B.

Pool A keeps all of its configuration, which is easier, but A1, A2 no longer contain a backup of pool A as it was.

In either case it cannot hurt to wait for the upcoming hotfix update, or to apply the mitigation against the silent corruption issue, before moving that data.

 

[Ziggy]

Thanks guys. Learning that exporting is unmounting is very helpful. I was confused on that. As for exporting and later re-importing the pool on spinning rust to the new SSD vdev, my problem there is I don’t have spare slots. There’s another pool of 8TB mirrors that occupies the other main slots in the mini, and internally I have a small 256GB SSD mirrored pool for documents on the sata connectors there. Now, as I write, I’m wondering if I could temporarily export that pool (all my documents are well backed up externally and I would do a fresh copy before proceeding), attach the two new 2TB SSDs here, create a pool and import the 3TB pool (with only 16% used in case anyone is newly puzzled) to this pool. Once this is done I can then remove the 3TB pool and put the 2TB SSD pool in their slots, followed by re-importing my 256GB documents pool. Then I’d have to redo the config for the apps and VMs. Would all that work?

 

[Ziggy]

Update: again, thanks for the help. i went with exporting / detaching the pool, inserting the two new 2TB SSDs, creating a new pool, and then just redoing the VM and couple of apps I had on the old pool. Felt it would be cleaner that way and start with fresh snapshots.