The great migration has begun

[Constantin]

Backing up the server to enable transition to TrueNAS from FreeNAS 11.3U5. Just a quick note of thanks to the iXSystems team for making the ZFS send / receive process so simple and easy to use. Setting up a replication task was far simpler than I imagined. I had visions of having to exchange SSH keys manually, setting up tunnels between the machines, etc. and the wizard was not only very helpful but comprehensive as well.

Also looks like the ZFS send is based on a single-threaded process /protocol like SMB? My Mini XL on the receiving end keeps maxing out one of its 8 cores 100%. Then the package is digested (all cores bump up) followed by yet another bout of all cores but one lying idle while one is maxed out. It's fun seeing that process dance from core to core in the CPU dashboard.

 

[Constantin]

One question I have re: the SLOG over-provision feature in System -> Advanced. I presume I have to set that value here first and then attach a SLOG of my choice?

 

[Samuel Tai]

One question I have re: the SLOG over-provision feature in System -> Advanced. I presume I have to set that value here first and then attach a SLOG of my choice?

Yea, verily.

 

[Constantin]

That's another neat "upgrade" vs. having to manually format / partition / whatever a SSD to over-provision it. Very nice!

6TB or so per day. A bit slower than I expected via 10GbE-SR but quick enough.

The challenge at the end of the week is

• nuking the extant pool on the server that the data is being pulled from right now
• nuking the extant ifconfig in all its forms and setting up a bridge across ix0 and ix1.
• setting up the small files and metadata-only VDEVs.
• enabling ZFS native encryption for the pool as it's moved back
• reconfiguring Time Machine to use SMB for my modern macs

 

[Constantin]

Having wiped the main server clean (including snapshots), the pool was rebuilt with a 3-way mirror for the sVDEV, now all the data is hurtling back. I couldn't make it work using the GUI, so I simply launched a bunch of tmux sessions to SCP it from one server to the other. Not as fast as running a snapshot with ZFS-Send, but good enough. On the other hand, using SCP allows me to granularly send stuff back, i.e. avoid my time-machine backup since I will rebuild it from scratch with SMB instead of AFP.

I am somewhat disappointed that the snapshot-deletion tool in the GUI is dependent on said GUI session to stay active until the task has completed. This is somewhat counterintuitive as many other tasks inside TrueNAS function just fine, whether the customer is keeping associated windows active or not. Also, the browser "window" is locked while this happens (i.e. no updates, no progress bar, nothing other than the swirling blue "please wait" tail chasing itself).

I would have preferred a snapshot deletion experience where the GUI notifies the subsystem and then keeps working until the task is complete unless the customer interrupts the process. For extra points, show a progress bar!

One of the side effects is the two servers heating the room that they sit in. The plug load (with heavy activity) on the main server is now approaching 150W.

Lastly, I disagree with sVDEVs not being assignable re: their purpose. For example, being able to reserve a % of the space on the sVDEV for metadata, dedup, and small files to ensure that if stuff migrates out to the main pool due to crowding, the most important data sets (for a sysadmin) have adequate "space" to keep growing. In my case, that would be prioritizing metadata over small files, reserving 600GB exclusively for metadata use and leaving the balance for small files.

 

[Constantin]

One more thing.... while removing GELI encryption from an extant pool is a possibility (see Patrick M. Hausen's article on that), I decided to bite the bullet, nuke the pool, and start over. On the one hand, this will cost me a lot of time (re-setting all the applicable permissions, etc.) but on balance it likely makes sense as a way to eliminate all cruft and start over.

 

[Constantin]

Made the transition from AFP to SMB and the support for Apple file systems has improved dramatically. (Thank you @anodos!) I have finally turned off AFP and am running a SMB-only machine here now. However, I did run into the same issue as other Mac users that permissions, etc. were borked completely until I nuked the local (Mac OS) nsmb.conf file (thank you @thedude123!). That fixed permissions issues I could not explain or fix.

I did rebuild the pool a second time to redo everything as a SMB-intended share and am now copying everything back from backup for the second time. I also removed attempts at compression from anything but shares that may actually have compressible data. The rest got the newish ZSTD.

Will say that the fusion pool is amazingly fast. Metadata operations like browsing directories is at DAS speeds, downloading files reminds me of DAS arrays with HDDs. It's simply awesome. I cannot wait to run my first backup to see what happens to rsync next.

Next up is getting SSL deployed across the home as I got my Raspberry Pi-based certificate-authority deployed. Took a while to get set up... now how to make nice with ACME and other services to pull certificates and allow secure connections to my NAS, the Pi's, and so on.