Also, update:
Today I did some more with the server. I set up the snapshot schedule, the scrub schedule, etc. But much more importantly, I moved the jail SSD over. And I want to talk about that for a moment, because I really developed an appreciation for this.
As a lot of you know, many of us run our jails on a separate pool, often on a single (or mirrored) SSD, even though we have ample space in our data pools to run the jails if we so chose. We often don't have super convincing reasons for doing this; at the end of the day, "it just seems like a smart thing to do" somehow. Well let me tell you how nice that was today. My jails are configured with VIMAGE, with specific fixed IPs outside of the DHCP range of the router, and with specific MAC addresses that I put into the config. Well let me tell you how awesome that was today as I moved my jail ssd over to the new box ("Daneel" by the way, is the name I settled on, another robot from Asimov's work). Here were the steps I followed:
1) I went into the old server storage screen, highlighted the ssd with the jails on it, and clicked the icon with the red X in it, and was careful to make sure "erase the data" was *NOT* checked, and proceeded to "detach" the pool. I really hate this language. We have vocabulary in FreeNAS that does not really match the common vocabulary in ZFS at large, and that makes it hard to understand what you are or are not doing. It took me a few minutes to convince myself that this thing called "detach the pool without clicking the DELETE DATA button" was code for "export zpool". There are a few similar vocabulary liberties we take in FreeNAS in other places, but I will not dwell on those. If we have ZFS language, we should be using it, and a tooltip to help the novice user. "Exporting the pool" and "completely thermonuclearly blowing away the collection of devices that constitute a pool" are two completely separate--and unrelated--procedures, and they should not be part of the same user dialogue, particularly one which uses non-standard terminology, where the difference between one and the other is a checkbox. And don't tell me "the whole screen turns crimson red and warns you that you are about to delete the data if you click the checkbox". I'm not buying it. It was either Jordan, or Xin, or Suraj, one of the guys at ixSystems that once told me, "If you have to put an arrow or a flashing red screen on it in your GUI so the user doesn't screw it up, then your GUI is doing it wrong; the problem is the GUI not the user's stupidity." Indeed. Since I don't write GUIs (rather, I write the computationally-intensive stuff GUIs call), this bit of wisdom was new to me, and I remembered it. Anyway, as for why to "detach": The purpose to "exporting the zpool" ("detach" in FreeNASese) is so that all caches and whatnot are flushed through and committed, and the pool's metadata now indicates to the next system that imports it that everything is good and fine, and the last transactions completed, and we are ready to import. If you try to import a pool that was not properly exported, as many of you have personally experienced, there can be issues.
2) Put the SSD into the new box. I clicked "import volume" in the GUI, boom, it's imported, 2 seconds. All my jails are there in the pool.
3) Now, I just go to Jails->config, and say the jail "root" is ssd/jails. Boom done. All my jails are now in the "jails" tab in the GUI.
4) Magically, all the jails are right as I left them, FreeNAS is aware of the entire jail config, everything is good.
5) Of course, if you "added storage" from your pool into the jails, you'll have to set that up again, as that will not carry over.
6) Now I fire up the jails. And everything simply is EXACTLY as it was before, and the beauty of having it on the separate SSD, with static IPs (which of course carry over), specific MACs (which carry over), etc., is that all of my port forwarding rules, all of my static DHCP allotments, all of that stuff, totally seamless now, I don't have to do *A SINGLE THING*. Don't have to reconfigure the firewall to forward the mumble server port to the new box, don't have to reconfigure the other ports to forward to new boxes, don't have to do ANYTHING. It just works.
7) Of course, you lose all of your periodic snapshot tasks that went with the jail's dataset(s), so you'll have to reset those.
Now of course, you can have a similar experience if your jails are simply on your main pool, by zfs sending over your datasets and resetting the jail root and whatever. But the "just move the jail ssd to the new box and import the pool" method really has fewer moving parts, and the transfer speed is infinity, instead of 1Gbps or whatever.
I still have the side panels off the box. I think this is a design flaw in the R5 case, definitely, with standard connectors, you are smashing your SSD's connections on the left panel, and you are smashing your HDD's connections on the right panel, when you close it. Simply making the case 0.5" wider totally solves this problem. I guess I'll just order some 90 degree connectors or something and lick my wounds. Overall though, the case is pimp.