Ok, just over a month later, I am up and running with 12 5TB, drives in Z3. All worked out very well, with 0 broken keyboards. I did a bit of mix and match to make it all work. I moved to a Supermicro board, new case, Xeon, and RAM, basically a whole new machine. For anyone reading this at a later date who wanted to move all data to a new machine while preserving jails and all data, here is what I did.
- Installed all drives and did hardware setup.
- Installed an identical FreeNAS on the new system to a spare flash drive.
- Tested all new drives thoroughly. Solnet-array, badblocks, and SMART (before and after). The solnet-array test gives a great idea on which drives may not be performing up to par. I returned two drives that were testing much slower than the others. Another drive failed badblocks with 3 read errors, and it too was returned. Then I had to wait for the new drives, and then retest them. You can easily find several how to's and pointers in this forum.
- Made two identically named raidZ arrays on the new system once I was satisfied all drives were 100 percent functional.
- Connected the two machines via the 2nd LAN ports with a patch/crossover cable (10.0.0.1, 10.0.0.2).
- Set up replication according to the FreeNAS 9.3 docs, for both volumes (zfs0, and zfs1), to the root of the volumes on the second system. You might have to move the system volume a few times. Do the jail volume last, if applicable.
- Waited about a week for the data to replicate (21TB). Then for shits and gigs, I did a rsync of my main data pool just to be absolutely sure I had everything. A few new items had downloaded over the week I was replicating, so it was not in vain.
- Exported/detached the volumes on both the first and second systems. (Keep share data, and for the love of almighty, DO NOT CHECK DESTROY)
- Shutdown. Removed the flash drive(s) from the first system, and placed into the second new system.
- Boot up, then configure your network to use the original IP. Then used the GUI to import the volumes.
- Reboot. Then verified all data was there, jails started, plugins ran, UPS was running (I had to change port to ugen 1.3).
To my delight this worked like a charm. All my data was there, shares, config, everything just like I were still running on the old system.
A big thanks goes out to all of you who offered pointers. dpasseg, JGreco, Cyberjock, and everyone else who I sniped how to's from. Thanks to everyone in the FreeNAS community, its a beautiful project. Keep up the great work.