Updates kick USB out of boot order

[joeschmuck]

You seem very new to FreeNAS so don't take this the wrong way but maybe you should be running FreeNAS 9.2.1.9 vice 9.3-Beta. The beta still has lots of issues with it and I only run it on a test system, not my real FreeNAS machine. And then many people are running the beta in a virtual machine as well (I do the same). It's still under development and if you're not ready to find and identify problems and possible solutions, it probably isn't for you. I think the one positive thing is you discovered the upgrade/boot issue so you could address it. And I have no idea if in the future that specific problem isn't addressed, I have no UEFI experience. My next MB purchase will certainly use that technology.

 

[indivision]

I appreciate the perspective. I've been using FreeNAS for about 2 years. Maybe not too long. But, I've also been developing software professionally for 15+. I don't assess much significant risk in using this beta. Worst case scenario, it nukes the main server. Everything is rsync'd and replicated in other places. But, having gone through drive restores, testing multiple NAS os', mounting with live ubuntu cds to access data directly, etc. I feel that the mechanics of ZFS are actually pretty resilient and easy to use in disaster scenarios. Just as long as you don't make bad decisions setting up the volumes/pools.

Other than this boot issue, I haven't found any issues at all. Seems very stable. As far as betas go, I think that using a product as it would normally be used is far more valuable than poking around trying to emulate things people might do. If that is all people are doing right now, we are bound to find new issues once it is called a release anyway.

UEFI seems promising on paper. And maybe it's because we're kind of in a transition to it. But, my experience with it has been frustrating. Installing Windows 8 with UEFI was a beast.

 

[RobertT]

I use this system: http://www.asrockrack.com/general/productdetail.asp?Model=C2750D4I

I was using the beta for a week or so, and noticed an upgrade today which I applied. Later when I checked back on the server it had changed the boot order in Bios. When I fixed that, the boot proceeded for some time but there were many errors, and no ip. I finally resolved the issue by selecting the previous beta version in the boot menu.

I have the same exact motherboard and have been applying updates nearly every day. I have not had any issues with booting.

 

[Pasquale61]

I'll add my name to the list of people having this problem. I didn't post anything earlier because I really thought I was doing something wrong in my BIOS settings and I kept trying things. I thought I had it resolved only to find out it was back on the next update...and then again on the next one. I didn't have this problem when I was on 9.2.1.x. Other than this, I have not have not run into any other problems with 9.3-Beta

Here is my setup for reference:

MB/CPU: ASRock C2750D4I
Disk: 5 HGST Deskstar NAS 4TB H3IKNAS40003272SN (RAIDZ1)
Boot Drive: SanDisk Cruzer 8GB USB
RAM: 32GB ECC (CT2KIT102472BD160B x 2)
PS: ST45SF-G
Case: DS380B

 

[RoboKaren]

I've been having issues with my USB flash drive getting kicked out of the boot drive order as well. It's to the point that I have to nurse each upgrade/reboot to make sure that it reboots properly.

What usually happens is:

0. Upgrade and automatic reboot
1. A warning flashes "THIS IS A NAS DISK AND IS NOT BOOTABLE" (i.e., the BIOS tried to boot off one of the SATA disks)
2. Try to do a CTRL-ALT-DELETE soft reboot and the BIOS then hangs with a USB initialization error
3. Hard power off and power on and everything goes great, it boots off the USB flash drive and everything is hunky dory.


It's not a major pain in the ass but it does mean that I have to manual observe each upgrade cycle. This is with a 16GB Lexar USB flash drive as my boot drive.

I wonder if it's not so much the motherboard or CPU, but that the upgrade process leaves the USB drive in an unknown state that confused the BIOS at soft reboot.

Karen

 

[Pasquale61]

I've been having issues with my USB flash drive getting kicked out of the boot drive order as well. It's to the point that I have to nurse each upgrade/reboot to make sure that it reboots properly.

What usually happens is:

0. Upgrade and automatic reboot
1. A warning flashes "THIS IS A NAS DISK AND IS NOT BOOTABLE" (i.e., the BIOS tried to boot off one of the SATA disks)
2. Try to do a CTRL-ALT-DELETE soft reboot and the BIOS then hangs with a USB initialization error
3. Hard power off and power on and everything goes great, it boots off the USB flash drive and everything is hunky dory.


It's not a major pain in the ass but it does mean that I have to manual observe each upgrade cycle. This is with a 16GB Lexar USB flash drive as my boot drive.

I wonder if it's not so much the motherboard or CPU, but that the upgrade process leaves the USB drive in an unknown state that confused the BIOS at soft reboot.

Karen

I find it hard to believe that FreeNAS can somehow alter my BIOS settings, but strangely enough the updates seem to somehow affect the boot order. (I wouldn't have believed it if I haven't seen it first hand!) I refrained from posting anything earlier because it just didn't seem possible and I really believed that I just needed a different BIOS setting for the new boot format. It's interesting that some others are seeing something similar to what I'm experiencing.

For what it's worth, I did not have the problem prior to 9.3. Also, I can reboot, power off, etc, with no problem, until I do an update (Using the System, Update, Apply Pending Updates.) What happens is that my USB boot device no longer shows up in my BIOS, and it tries to boot from one of the SATA drives. I can't even change the boot order because the USB drive doesn't even show up in BIOS! I have tried just about every setting, including legacy and UEFI. This last update, after fiddling around with BIOS settings, rebooting, powering the server on and off via the IPMI, it still didn't work. It got to the point where I needed to physically unplug the power and reset everything just to get the USB drive to show up again so I can select it as the first and only device in the boot order.

None of this really bothers me since this is in my home, but I'm just bringing it up since I'm seeing that I am not the only one having the problem.

My motherboard is the ASRock C2750D4I, running latest BIOS (2.50) and latest BMC (00.19.00)

 

[solarisguy]

A guess...

A BIOS identifies the USB memory device by its filesystem label or something similar. That gets a new value with an update.

That is why the problem is seen only with certain BIOSes.

 

[jkh]

It must be something odd like that, because frankly, there's nothing FreeNAS can do to affect the BIOS boot order. It does not have access to that configuration information, much less try to set it, so the best theory I've seen so far is above (maybe something to do with how a new boot environment is created for each new install?). I guess we could try to create a test version of FreeNAS which didn't try to create a new boot environment with each upgrade, or make it a secret tunable that folks who see this problem could set. It wouldn't "solve the problem" since you really *want* a new boot environment created before an upgrade so that you can roll-back in case of difficulty, but it would at least narrow down the cause!

 

[RoboKaren]

A guess...

A BIOS identifies the USB memory device by its filesystem label or something similar. That gets a new value with an update.

That is why the problem is seen only with certain BIOSes.

But isn't the filesystem "label" that's seen by the BIOS just the GRUB bootloader? That should remain constant.

My guess was that the upgrader was leaving the USB flash drive in an unknown state (data written but not flushed, etc.) before the soft reset. It's only with a hard power-down and power-up that the flash drive remembers who it was and what it was doing.

 

[sef]

The fact that the thumb drive isn't showing up in the BIOS _at all_ is strange -- but that would also explain why it's not booting from it, of course.

I am curious if some thumb drives are more susceptible to this than others.

(I was playing with that motherboard this weekend, btw. But a) using SSDs not thumb drives, and b) using FreeBSD for the moment, not FreeNAS.)

 

[Pasquale61]

I was thinking the same thing about certain types of flash drives being susceptible. I can try some testing tonight when I get home and see if I can figure out a pattern.

 

[solarisguy]

I wonder whether the thread Won't POST with 9.3 on USB https://forums.freenas.org/index.php?threads/wont-post-with-9-3-on-usb.25234/ is related to this one.

As in: jnyl42's BIOS tries to read some place of the USB device (that place that gets written to with the updates), and gets stuck...

Those who are seeing the issue(s), could you please post details about the BIOS (and a little about the motherboard). Possibly there would be some commonality, after all not everybody is experiencing the issue.

P.S.
I am aware that the above is just speculation with no supporting evidence :)

 

[Pasquale61]

OK, I did some testing tonight and this is what I observed:

1. Updating to any newer version causes it to boot to the BIOS setup screen. This happens because all of a sudden no USB drives are visible anywhere in BIOS and no boot devices are selected. (Boot Option #1 shows “Disabled”) Selecting “Discard Changes and Exit Setup” immediately comes back to BIOS screen.
2. Pressing Ctrl-Alt –Delete brings it back to BIOS screen after POST. Still no USB drive in BIOS.
3. While on the BIOS screen, pulling the USB drive out and plugging it back in, and then Selecting “Save Changes and Exit Setup” (even though Boot Option#1 is still set to “Disabled” and no other drives are selected for boot), the system boots fine into GRUB. I figured out that re-seating the USB drive somehow resets something and therefore does not require a complete shutdown.
4. From this point forward, I can reboot as often as I want until I attempt a System Update again.

5. Activating a previous boot version using the GUI works fine, as well as going back to an existing newer version.
6. If I activate a previous boot version, and then apply an update again, the problem appears again.

Like someone mentioned earlier, it seems that the update is writing something to the USB drive that the BIOS does not like. (But only the first time...go figure!) The other strange thing is that someone earlier mentioned that they have the same motherboard that some of us have and they do not have this problem. On the other hand, RoboKaren has a different motherboard altogether; not sure what the original poster in this thread has.

I did not get a chance to try another type of thumb drive yet but I'll try to find something tomorrow and see if it makes any difference. Another thought that comes to mind is to disconnect my APC UPS that is plugged into another USB port in case that somehow has anything to do with this.

 

[Pasquale61]

So, unplugging the UPS from the USB port and making the thumb drive the only thing plugged in didn't make any difference.

@solarisguy - I think you're right that POST has something to do with it, but I'm not exactly sure what. I can tell when it's not going to work because it takes a long time before I see the BIOS logo screen before it boots into the BIOS setup. During this time I only have a black screen and I'm guessing it's looking for the boot drive during POST. No errors or messages during this though, which is the fault of the BIOS. I have the ASRock C2750D4I with the latest BIOS (2.50)

Regarding the other thread, "Won't POST with 9.3 on USB," it may be related to this...especially because he is seeing "This is a NAS drive, system halted" which sounds like it's trying to boot from a drive out of his storage pool.

 

[cyberjock]

Is this problem happening after an upgrade via the WebGUI?

If I'm not mistaken the WebGUI does the update, then initiates a "reboot" command. This isn't as "clean" as a "shutdown -r". It could be that this difference is the reason why the USB device isn't showing up on reboot. The hardware is stuck in some inappropriate loop or other problem and isn't usable until a power cycle. We've seen this in the past with "reboot" and IMO FreeNAS should never be coded to do "reboot" when "shutdown -r" is the recommended reboot. This might be easily testable....

do this from the CLI to upgrade:


# freenas-update check
# freenas-update update
# reboot

(notice if the problem occurs)

Try the first two commands again but replace them with "shutdown -r now".

 

[jnyl42]

As in: jnyl42's BIOS tries to read some place of the USB device (that place that gets written to with the updates), and gets stuck...

Those who are seeing the issue(s), could you please post details about the BIOS (and a little about the motherboard). Possibly there would be some commonality, after all not everybody is experiencing the issue.

My board has a legacy Award BIOS, with the latest update installed (from 2010)

Regarding the other thread, "Won't POST with 9.3 on USB," it may be related to this...especially because he is seeing "This is a NAS drive, system halted" which sounds like it's trying to boot from a drive out of his storage pool.

This is only when I boot with no USB drive attached. I was testing to make sure it was really the USB software causing the crash and not some other hardware problem. Otherwise the BIOS crashes immediately after displaying the CPU type.

 

[BigJ]

RoboKaren's description is exactly what occurs for me. Saything that, I'm sure that the earlier updates worked without a hitch but I do have a terrible memory.

Is this problem happening after an upgrade via the WebGUI?

If I'm not mistaken the WebGUI does the update, then initiates a "reboot" command. This isn't as "clean" as a "shutdown -r". It could be that this difference is the reason why the USB device isn't showing up on reboot. The hardware is stuck in some inappropriate loop or other problem and isn't usable until a power cycle. We've seen this in the past with "reboot" and IMO FreeNAS should never be coded to do "reboot" when "shutdown -r" is the recommended reboot. This might be easily testable....

do this from the CLI to upgrade:


# freenas-update check
# freenas-update update
# reboot

(notice if the problem occurs)

Try the first two commands again but replace them with "shutdown -r now".

It worked fine when updating from the CLI using 'reboot'. This surprised me as the last three updates (at least) had the reboot failure as previously described.

 

[Pasquale61]

@cyberjock Yes, this is all using the WebGUI for me. I will definitely try your suggestion later today and let you know what happens.

 

[Pasquale61]

I have good news/bad news.

The good news is that if I follow cyberjocks recommendation to test updating using the CLI, everything works great and I have no problem on the reboot. The bad news is that it doesn't seem to matter whether I use "reboot" or "shutdown -r now" after running the update. (I backed out by de-activating and deleting the updated boot version between tests, as I did before.) This leads me to believe that there is something different in the WebGUI update process that may be causing this?? Is there some type of update log that I can look at to compare what it's doing different to try and help?

 

[cyberjock]

I wonder if this is a timing issue. What I mean is something isn't finishing before the "reboot" command is executed via the WebGUI. I'm curious to know if the WebGUI was changed to run "shutdown -r now" or have a 10 second delay if that would change the outcome.