Migrating Pool in Degraded State

[Chris Tobey]

Hi everyone,

I would like some help with suggestions and possibly commands to use to accomplish a migration.

I currently have a server with 12 SAS bays. My ZFS is configured with 12 disks as single zpool with two 6-drive RAID-Z2 vdevs (6 x 8 TB and 6 x 3 TB). I have purchased six new 16 TB drives that I would like to create a new pool on, copy everything over, and then wipe the old disks for other uses, leaving me with a single zpool with one 6-drive RAID-Z2 vdev, and 6 spare bays.

So my problem is that I only have 12 bays, and they are all used. Other info:

1. To further complicate things, the zpool is also hardware encrypted (created in FreeNAS 11.x if that makes a difference).
2. Running TrueNAS 12.0.
3. Current usage sitting at 94% (~34.5 TiB).
4. All data is backed up offsite, so data loss would not be catastrophic, but would like to be avoided.

I believe what I can do is remove two drives from each vdev, add four new drives, created a new zpool in a degraded state, replicate the old to new, remove the old drives, then install the new remaining two drives and resilver. So the steps would be something like:

1. Pull two drives from each 6-drive vdev, I'll still have a fully functional pool, but with zero redundancy.
2. Create two fake drives on the OS.
   1. Commands?
3. Then I put in four of my new 16 TB drives.
4. Create my new pool with four real drives and two fake drives in RAID-Z2 (with encryption).
   1. Commands?
5. Replicate old pool to new pool (scary part).
   1. Is there a way to make this safer? Mounting the old zpool read only so if a drive does die, I can simply replace it with one of the parity drives and start again?
   2. Commands?
6. Remove the other 8 old drives.
7. Add my remaining two new drives.
8. Resilver.

Any help or suggestions, especially for the commands (as I am not sure the GUI is well-suited to this) is appreciated.

 

[HoneyBadger]

While what you're suggesting will work, man is it fraught with potential peril.

You're talking about cutting out all redundancy/recovery points out of three different RAIDZ2 vdevs, with the penalty for failure being ~34.5TiB of data needing to be restored from an offsite backup. That's well beyond my tolerance for risk - but as always, you're free to make your own decision.

In your shoes, I would be looking at borrowing/renting/buying-and-reselling an inexpensive SAS JBOD shelf (Dell MD series, Lenovo SA120, etc) and external SAS HBA just for the built and transfer process. (And then probably making some excuse to keep it around, but that last part might just be my own obsession.)

 

[Chris Tobey]

With a 1 Gbps connection I would be looking at ~ 4 days to re-download my backup, so I would like to avoid that if possible.
I am hoping to mitigate the data loss by importing the existing pool as read-only in the degraded state, so even if a drive were to fail, I could simply swap it with one of the other two from that vdev and re-import it read-only again... is that a thing?

 

[HoneyBadger]

I suppose since you aren't overwriting the two removed disks per "source vdev" you have that as a fallback - and importing as read-only is a good safeguard as well, although it means you can't write to the pool for the entire duration of the migration.

I would definitely suggest a full scrub of the pool before exporting and re-importing as read-only, as well as a full scrub of the new pool after the "missing" Z2 drives are added. Since you're never truly destroying the previous vdevs, you can always remount them and try again if a file fails the checksum at the end of the road.

But again - given the amount of data to be shifted here, I would be looking at the value of my time against the cost of the JBOD shelf. If it fails out halfway through the zfs send/recv (during which I can't save any files to my NAS) and I have to redo it, I'd be a little annoyed. If it does it a second time, I'd be - well, "more annoyed" is the polite way to put it.

I'd almost suggest using a separate chassis (an old ATX tower with a sufficiently strong PSU is fine) and running some SAS breakout cables through an open PCI slot cover from your 12-bay across and into it. A little bit of a physical hack-job certainly, but much less of a logistical/data-integrity one.

 

[Scharbag]

You should also fully test your new drives. This will both take time (days) and hardware. I think you will likely need to figure a way to run more disks in your system.

Too bad we are not yet able to remove a top level RaidZ vDev... Cause that would make it easier.

Cheers,

 

[Chris Tobey]

As it turns out, I do have an additional SAS connector on my LSI card, so I could crudely add four additional drives, assuming I get the power somewhere.

So I think a "better" setup would be something like:

1. Remove one drive from the old zpool RAID-Z2 vdev A.
2. Remove one drive from the old zpool RAID-Z2 vdev B.
3. Add six new 16TB SAS drives, two internally, four externally.
4. Create new zpool with one RAID-Z2 vdev.
5. Replicate old zpool to new zpool.
6. Scrub.
7. Backup.
8. Destroy old zpool.
9. Remove old drives.
10. Re-arrange new drives inside chassis.

 

[Chris Tobey]

This is working, and so far I am on step 5 (replicate) which has been complicated by the old pool being GELI encrypted and wanting encryption on the new pool. So far it is moving forward though.

 

[Chris Tobey]

Look what you have made me do!

 

[Scharbag]

View attachment 43207
Look what you have made me do!

That is a TrueNAS right of passage!!

More than a few of us have likely been there before - I know I have. :D