Updates kick USB out of boot order

[Pasquale61]

I was kind of wondering the same thing. I was trying to read my screen during the process but it was going too fast, but I "thought" I saw it trying to do a scrub on the boot volume towards the end of the update process. Maybe it doesn't finish this through the WebGUI? I don't know...

 

[Dave Genton]

Has anyone else run into this problem? Any work arounds? Anyone using this same motherboard and have a solution?

I have recently upgraded my FreeNAS server to 9.3 Beta after using it in a test machine for awhile playing around with it. Since upgrading it I have also found that each time I update it I must re-enter the bios and select the boot device all over again. Its been booting for a year on its own just fine but since 9.3 each daily update requires me to intervene again as well.

 

[solarisguy]

@Dave Genton, could you please describe your BIOS ?

 

[indivision]

A guess...

A BIOS identifies the USB memory device by its filesystem label or something similar. That gets a new value with an update.

That is why the problem is seen only with certain BIOSes.

That has been my theory as well. It seems like it looks for the same USB. But, if it "looks" different according to some id or label it thinks its a new USB and puts it at the bottom of the boot order.

It must be something odd like that, because frankly, there's nothing FreeNAS can do to affect the BIOS boot order. It does not have access to that configuration information, much less try to set it, so the best theory I've seen so far is above (maybe something to do with how a new boot environment is created for each new install?). I guess we could try to create a test version of FreeNAS which didn't try to create a new boot environment with each upgrade, or make it a secret tunable that folks who see this problem could set. It wouldn't "solve the problem" since you really *want* a new boot environment created before an upgrade so that you can roll-back in case of difficulty, but it would at least narrow down the cause!

I definitely wouldn't want to lose the ability to roll back. That's a great feature. Do the new boot environments have any kind of a label associated with them? A partition label? If that is being changed based on the update id each time, would it be possible to just keep this theoretical label the same at all times?

But isn't the filesystem "label" that's seen by the BIOS just the GRUB bootloader? That should remain constant.

My guess was that the upgrader was leaving the USB flash drive in an unknown state (data written but not flushed, etc.) before the soft reset. It's only with a hard power-down and power-up that the flash drive remembers who it was and what it was doing.

That's an interesting idea. In fact, I have two different FreeNAS machines. One of them requires BIOS work on every update. But, the other one has an issue similar to that. It works most of the time. But, every now and then it gets stuck after reboot and reset doesn't work. Only a hard power-down brings it back online. I hadn't mentioned that second machine to start this thread because powering down after a stalled reboot hasn't been as annoying as having to go into BIOS on every update. :)

Is this problem happening after an upgrade via the WebGUI?

If I'm not mistaken the WebGUI does the update, then initiates a "reboot" command. This isn't as "clean" as a "shutdown -r". It could be that this difference is the reason why the USB device isn't showing up on reboot. The hardware is stuck in some inappropriate loop or other problem and isn't usable until a power cycle. We've seen this in the past with "reboot" and IMO FreeNAS should never be coded to do "reboot" when "shutdown -r" is the recommended reboot. This might be easily testable....

do this from the CLI to upgrade:


# freenas-update check
# freenas-update update
# reboot

(notice if the problem occurs)

Try the first two commands again but replace them with "shutdown -r now".

Interesting. I will try this once there is a new update queued and report back.

 

[indivision]

# freenas-update check
# freenas-update update
# reboot

(notice if the problem occurs)

Try the first two commands again but replace them with "shutdown -r now".

That worked! I tried "shutdown -r now" first and it rebooted fine without dropping the USB out of the boot order. This suggests to me that this is something that can be fixed from within FreeNAS.

NOTE: I entered these lines individually and there was a delay between as I was doing other things. So, timing could be a factor or even the entire issue. When there is another update I will run the same commands but "reboot" from shell to see what happens.

 

[solarisguy]

@indivision, thank you for the test.

However, in my opinion the problem with BIOS is real: it reads from somewhere on the USB device.

 

[indivision]

Would there be a drawback for FreeNAS to use "shutdown -r now" after updates rather than "reboot"?

 

[cyberjock]

Would there be a drawback for FreeNAS to use "shutdown -r now" after updates rather than "reboot"?

Doubtful. reboot just tells all the process "do your shit now! I'm shuttin'er down!" while a shutdown actually shuts down the processes cleanly.

 

[Pasquale61]

The only problem is that (at least for me) I had the same results with "reboot" as I did with "shutdown -r now" - Both update/reboot methods worked fine in CLI. This tells me there's something in the WebGUI (possibly timing like you said) that is different than updating via CLI. Is it possible that this problem only exists with slower thumb drives and it's not quite done doing whatever it needs to do before rebooting? (Like a scrub of the boot device or something?)

BTW, I agree on using "shutdown -r now" I'm not as familiar with FreeBSD as I am with Linux, but on all my Linux servers I have always used the shutdown command, probably out of habit more than anything.

 

[indivision]

Doubtful. reboot just tells all the process "do your shit now! I'm shuttin'er down!" while a shutdown actually shuts down the processes cleanly.

The words went by too quickly. But, I was watching the output while shutting down and I did see some reference to the USB. Maybe too quick of a shutdown leaves some status of the drive off?

The only problem is that (at least for me) I had the same results with "reboot" as I did with "shutdown -r now" - Both update/reboot methods worked fine in CLI. This tells me there's something in the WebGUI (possibly timing like you said) that is different than updating via CLI. Is it possible that this problem only exists with slower thumb drives and it's not quite done doing whatever it needs to do before rebooting? (Like a scrub of the boot device or something?)

BTW, I agree on using "shutdown -r now" I'm not as familiar with FreeBSD as I am with Linux, but on all my Linux servers I have always used the shutdown command, probably out of habit more than anything.

I see. I haven't tried "reboot" from CLI yet. But, I suspect that it will work if yours did. A timing issue like that makes sense to me. Though, I don't think that my USB drives are particularly slow... I wonder if "shutdown" would fix it either way since it would presumably wait for processes to finish before shutting down.

Can anyone with technical knowledge of FreeNAS verify what the WebGUI reboot after update does differently than the CLI update process cyberjock wrote?

 

[Pasquale61]

@indivision Your original post said that you opened a bug report and it was closed as a "user configuration issue." Since I'm kind of new here, I don't know what the protocol is to re-open it, but I think there's enough evidence here that proves that this is more than a user config problem. It seems that it doesn't matter whether the hardware is older or newer, and I think the same is true with the BIOS.

I don't know how wide-spread this and if it's enough to be considered a show-stopper to get out of BETA. I realize that testing the updating process takes a lot of time, considering having to revert back for every iteration, so I don't mind helping the developers with any testing if it helps find a solution...

I just want to mention that I really like how the WebGUI is setup to manage multiple boot images, as I'm sure others do. The CLI update method works fine as a work-around for me, especially since it's just a home NAS, but I would hope to think that this will eventually be addressed.

 

[indivision]

I agree. It seems to me that this is something fixable from within FreeNAS. I don't know the ticket policy either. It's still closed as of now. I assume that a dev would have to re-open.

I did add a comment there with a link back to this discussion. Maybe this testing will be helpful to the team.

I suspect that this issue is more widespread than realized. It wasn't a big deal in the past because updates were infrequent. I know I didn't mind going into the BIOS once every few months. But, it becomes an issue with frequent updates.

 

[indivision]

The only problem is that (at least for me) I had the same results with "reboot" as I did with "shutdown -r now" - Both update/reboot methods worked fine in CLI.

UPDATE: I tried updating via the CLI again but with "reboot" and it did not work. USB has been kicked out of the order again.

So, for me, it appears that "reboot" is the problem and that "shutdown -r now" fixes it. I'll keep testing to make sure that "shutdown" works consistently...

Added new issue report specific to this happening from CLI: https://bugs.freenas.org/issues/6923

 

[craigyb]

I hate to add this, but it happened to me about a week ago as well, ran a system update and my system didn't come back. I went to the console and my USB drive was not visible and PC was in BIOS mode.

I had to power cycle my server to get it to recognise the USB stick.

Subsequent updates have been ok.

 

[Pasquale61]

UPDATE: I tried updating via the CLI again but with "reboot" and it did not work. USB has been kicked out of the order again.

So, for me, it appears that "reboot" is the problem and that "shutdown -r now" fixes it. I'll keep testing to make sure that "shutdown" works consistently...

Interesting. I tried the "reboot" method again this morning with the latest update and it still worked OK. One thing I did was uninstalled Plex (so no jails, etc) and stopped the CIFS service. I only tried this to see if somehow the shutdown of these services was affecting the timing.

 

[sef]

The update mechanism via the GUI (not the manual upgrade, but new shiny! upgrade process) does

Executing: /bin/sleep 3 && /sbin/shutdown -r now &​

which is to say it is using shutdown.

shutdown(8) simply does a bit of notification, and then invokes /sbin/reboot.

 

[Dave Genton]

@Dave Genton, could you please describe your BIOS ?

Sorry, I also have a ASUS MB but its the X79-Deluxe used on my FreeNAS machine with 6x2TB sata drives, 32GB ram, and using Kingston 8gb DataTraveler USB drives for booting FreeNAS. I am using 2 now since 9.3 beta vs single in the past of course. I am never losing the device in bios, it just moves it so its not the boot device stating the same error as robokaren and a few others, being its a data drive and not bootable, even after disabling all other sata devices from being bootable in the bios. After reading this I downloaded and upgraded to newest bios available but same results both yesterday and today. Sounds like this thread is getting to the root cause however so I'll continue to monitor.. My test FreeNAS is a VM so it doesn't see this issue like the production box.

 

[Pasquale61]

@sef, is there anything I can do to make it sleep more than 3 seconds, just to test to see if it helps?

 

[sef]

You'd have to change
/usr/local/www/freenasUI/middleware/notifier.py

 

[Pasquale61]

Thanks @sef. Tried setting it to 10...no luck.