Esteemed Community:
Many of you with bootable USB drives have been reporting that you are having difficulties getting FreeNAS 9.3 to boot, even on recommended hardware, and that the process dies at the infamous "mountroot" prompt. We have had this report now on at least three different recommended SuperMicro models (X10SLL, X10SLM, and X9DRD-7LN4F), and since I was personally affected as a friend of mine asked for a new FreeNAS build on bare metal (X10SLL), and I had the problem personally, we put in some effort to track it down. In the bug reports, I had called this a "fix", but, (after Jordan yelled at me for the hubris of calling it a "fix") we're going to downgrade to "workaround" until we get more clarity on what's going on, and the senseis at iXsystems can do more investigation.
But the good news is: many/most of you using recommended hardware getting the mountroot error on boot should now be able to boot, for the foreseeable future, using the following guide. Your hardware is fine; the problem has more to do with the peculiarities of the booting sequence, and the fact that you are using the USB bus to boot.
NOTE: Even if you're not using recommended/server grade hardware, there's a good chance this will work. This problem is one that will probably be ubiquitous across many motherboard models and chipsets. In any case, you can't hurt anything by trying this, if you're having a mountroot issue. Go ahead and try, and if it works, reply with your motherboard make and model so that others might search and find the discussion.
As is usual for me, I'll explain both what to do, and try to give some insight as to what the hell is going on while I am doing it. A disclaimer though: we are talking about parts of the process that I, personally, do not have a very good grip on, so I may say some horsecrap in my attempts to explain what's going on. Even so, the workaround process I outline below should solve your problem.
So here we go.
First of all, if I understand Cyberjock correctly, when you boot, this is a multi-staged process. At first, the BIOS recognizes your boot devices, and reads sectors specifically designated to get the initial stages of the operating system going. GRUB (which can then hand off to a variety of operating systems) then takes over, and at this point, your boot USB device is re-recognized in the context of a FreeBSD device (like /dev/ad0).
Here's the problem. Apparently, for whatever reason, there is a small bit of time/lag that occurs between when the GRUB takes over, and when your thumb drive is *actually* accessible in a /dev/da0 context. If, at the point that access is attempted, and it's not available, THAT is when you drop to mountroot, and then you're screwed.
So here is what you do:
Step 1: GRUB loads.
View attachment 5985
Step 2: When you get to the GRUB screen (this is the screen that pauses for 5 seconds with the FreeNAS 9.3 option highlighted), IMMEDIATELY HIT THE ESCAPE KEY to stop GRUB.
View attachment 5986
Step 3: Having done this, hit "e" to edit. You will now see a list of what we call loader tunables that over-ride default behaviors in the loader, and you will be in a rudimentary text editor similar to "nano" (if you are familiar with it).
View attachment 5987
Step 4: You will need to add your OWN tunable here, manually. That tunable is as follows:
set kFreeBSD.kern.cam.boot_delay="50000"
This is shown below. Note: do not scroll down too far, you want this entry to be in the "Normal Bootup" entry in this file. Doesn't matter really where within that, but do not go so far that you are outside of that zone.
View attachment 5988
This number 50000 is almost certainly much, much more than you actually need, but we think it would be impossible to need *more* than this, so let's just start with this number.
Step 5: At this time, press F10 to BOOT. For this one boot (and this one boot only), your tunable will be active. When prompted, select "normal boot". Now, the GRUB will pause for 50000 milliseconds (i.e., 50 seconds), which is obviously ample (obscenely ample) time to let the USB bus do whatever it has to do.
View attachment 5989
And you should boot now!!!! woo hoo!
Step 6: NOW, once you're booted, the first thing you're going to want to do is to go into your FreeNAS GUI, and go to system->tunables and add a tunable, with category "loader". The name of the loader tunable is exactly as above minus the kFreeBSD, in other words, kern.cam.boot_delay, and the value for the loader tunable will be 50000. Hit Save/OK (whatever), and now this should be a persistent setting. (In fact, you should actually see it, if you were to reboot and start the GRUB process and go into the edit screen).
Play with the numbers (i.e., lower it by 10000 at a time). More than likely, 10000 will not be enough, but 20000 or 30000 will.
You're all set! At least until such time as the devs determine if this is actually the appropriate long-term fix, or the devs figure something else out.
Please, if this has fixed your problem, let us know right here. Thanks to Cyberjock for working jointly on this with me.