[SOLVED] Freenas crashed after firmware upgrade

[Ptera]

Both systems are identical (Based on recommended equipment).
Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz
Supermicro X9SCM-B LGA1155/ Intel C204 PCH/ DDR3/ SATA3/ V&2GbE/ MicroATX Server Motherboard
With 16335MB error correcting RAM
4) 4 TB WD RED SATA drives
1) 16GB USB drive to boot from.
Both servers were running 9.2.1.5

Upgraded both of my Freenas servers with 9.2.1.8 and they both appeared to crash.
I used the Upgrade Firmware option of the FreeNas gui
One is local to my office. The other is in the next town over.
After what seemed a painful amount of time the FreeNas server local came back up.
But there are no ping responses from the server the next town over.
So I am driving over there to plug in a monitor to see what it is doing (or not doing).

Ok so I reported this problem before - when a reboot is called for, the last line on the console is...
da0: <pny USB 2.0 FD 1.00 > s/n ############## detached

Why are we detaching the device we are shutting down from?

reset or power cycle the system and we get this...

This is a NAS data disk and can not boot system: System Halted

The only fix I have found is to reload a fresh copy of FreeNAS on the USB stick.

This is not a convenient way to upgrade a server.
Is this a bad USB stick?

Really would like to see a resolution to this before I commit sensitive data to this type of storage device.
Thanks

 

[solarisguy]

Two comments. The first one is about Unix in general. The Unix flavours, I am familiar with, do unmount all the filesystems upon a shutdown. Including root (/). And no, you are not shutting down from the disk/USB/whatever, the shutdown is being executed by the image that is already loaded into RAM. Since at this point in time no writing to any filesystem (as opposed to logging to a console) should be happening, such unmounting is Generally Recognized As Safe. Programmers do make mistakes..., thus I cannot comment on your particular circumstances. I might be mistaken, your report appears to me the first one, may be there was something else at play.

Unfortunately, some USB memory devices are getting occasional hiccups and transient problems that are very difficult to troubleshoot and replicate. Henceforth, the usual recommendation is to use those USB memory devices that are known to be of better quality and IPMI on the server. Corollary: since any device can fail, when your servers are remote, have two of boot devices (one uninitialized) and use IPMI to manage boot sequence as required...

P.S. I am very well aware that my attitude runs contrary to prevailing trends. However, wherever my personal opinion counts, I try to avoid in-place upgrades. In-place upgrades tend to make restoration of the previous state unnecessarily more complicated and time consuming. And such a restoration might be required for a variety of reasons.

 

[cyberjock]

I have the X9SCM-F board and the same CPU. I had no problems upgrading to 9.2.1.8. I can't vouch for your exact problem, but I will say that I personally do not consider PNY appropriate for a boot device.

P.S. I am very well aware that my attitude runs contrary to prevailing trends. However, wherever my personal opinion counts, I try to avoid in-place upgrades. In-place upgrades tend to make restoration of the previous state unnecessarily more complicated and time consuming. And such a restoration might be required for a variety of reasons.

You are going to hate 9.3. In-place upgrades from 9.3+ are all that will exist. FreeNAS is trying to get away from releasing images and instead doing patches that are self-installable and stuff. You'll have to allow the install, but it won't be like you can do a fresh install after 9.3 with patches applied. :)

 

[solarisguy]

[...] You are going to hate 9.3. In-place upgrades from 9.3+ are all that will exist. FreeNAS is trying to get away from releasing images and instead doing patches that are self-installable and stuff. You'll have to allow the install, but it won't be like you can do a fresh install after 9.3 with patches applied. :)

I hope that I will love 9.3 ;) To upgrade, I would just get a new USB memory device, install 9.3, add all the patches, read the configuration file and do anything that is still left to be done AND keep the old 9.3 installation in a safe place for a moment.

 

[cyberjock]

I don't think you understand. There may be small patches at regular intervals. I'm thinking weekly or more frequently. If you are that hardcore, go for it. I wish you luck. :P

 

[Ptera]

Ok so instead of the $9.00 PNY thumb drive I bought the $30.00 Sansdisk thumb drive - loaded the image on to it and installed it. System back on line but it was version 9.2.1.5 So I wanted it to be the same version as the primary nas server so I tried the upgrade again. It is now 17 minutes later and still no response from the server. I am guessing I will drive over to that location and the screen will be frozen after unmounting the USB drive.
Then I will try to reboot it and I will get the freenas message of death....
This is a NAS data disk and can not boot system: System Halted
Well this is Friday - I will drive over there Monday. Good thing I have not committed these to the production environment.

 

[Ptera]

So before we left here I took screen shots of the Bios config of the FreeNas server here to compare with the Bios config of the server there. There was no difference.

As I suspected the screen showed that it stopped in the same place as it did when we used the PNY USB thumb drive.

And we power cycled the server we got the same message
"This is a NAS data disk and can not boot system. System halted."

I had originally thought that was because it was trying to boot from the SATA drives in the system. But this time I went into the bios and set the USB as the only boot device to boot from. I got the same message again. Then I unplugged the USB drive and rebooted to make sure it was not trying to boot from the SATA drives and the system went to the UEIF shell prompt. So it was really trying to boot from the USB. I then brought the thumb drive back and plugged into a linux Mint system to see what was on the thumb drive. A full 16gig FreeBSD ZFS partition!

Which appears to me that the boot image on the USB was changed to a ZFS storage partition.
Then...
Here is what the USB looks like when the image is freshly installed and plugged into the Linux Mint System.

So this has happened to me more than once and now with two different USB drives.
So in my eyes this is a serious bug.
FreeNAS over writing the boot drive with a storage partition.

 

[Ericloewe]

For readability, I highly recommend you read this:

http://community.linuxmint.com/tutorial/view/85

 

[Ptera]

Nice tutorial but does not offer any resolution to this issue.

 

[mjws00]

Why not just back up the config. Install fresh 9.2.1.8. Restore config? Unless there is a very strong reason to fight this. I'd fix it and move on.

I ran a few updates with from the update firmware button. I do remember it taking a little longer than I thought it would. But beyond that nothing notable. I can't see any incentive to worry about a 9.2.1.5->9.2.1.8 upgrade glitch for devs. They will have their hands full with the 9.3 migration. :)

Good luck. Y0u caught the attention of some great minds who would have pointed you in the right direction if the problem wasn't obscure.

 

[solarisguy]

I have no idea how to solve the failed upgrade.

I recommend that you use mjws00's idea about installing (not upgrading) 9.2.1.8, reading the saved configuration, and adding required customizations. That could be saving you time...

 

[Ptera]

OK but there is something about that process that I do not under stand. I use the USB image, And with a new USB image the system comes up with the original config. I never had to restore or back up the config. Are we saying to do a regular install not the USB image?

Besides this is not just a failed upgrade. Rebooting the system yields the same results.

 

[mjws00]

The only way the config can survive is with the upgrade procedure. If you burn a clean full install on a new/zeroed USB. It is not possible for there to be anything beyond a bare config. Where would the bits come from?

I'm not sure if you are burning the ISO and then installing? For me it is much faster to just grab the FreeNAS-9.2.1.8-RELEASE-x64.img.xz file, extract and burn it with win32diskimager. Throw it in and the system is live. Takes all of 5 minutes. Hit the restore config button and in 30 more seconds you are back where you were, but on 9.2.1.8. Sorry for other os's use the manual, the same logic exists.

You'll have to clarify, "Regular Install not the USB image". The USB image IS a regular install. The ISO is also a regular install.

Your reboot issue is interesting, because you are on common known good hardware. This has to be a specific config issue, or procedure.

 

[Ptera]

That is what I did. Both the original install and each time after this issue occurred.
Downloaded the FreeNAS-9.2.1.8-RELEASE-x64.img.xz file, extract with 7-zip and burn it with win32diskimager to the USB. Throw it in and the system is live. And I never had to restore a config. The system boots up with the config that was there before the USB was overwritten with the 16G ZFS partition.
I have done this again to the USB drive sitting here on my desk.
I am going to create two of them, go out to the site, plug in the USB drive and turn the system on. If it does like it did the last time the system will boot up to the config that was there before. I will issue a reboot command and I am sure I will get the same thing displayed above. The USB with be overwritten with the ZFS partition. Second thought I am going to go get that server and bring it back here so I can repeat any process I need to get this resolved.

 

[mjws00]

I'd re-download and redo it. I'd also test the usb locally by throwing it in any old suitable local box. No point wasting time off site.

You should be able to verify that the old config is not loading by pulling the drives. If the old usb is not physically attached, and there are no drives, it is physically impossible for the old config to load. When you re-attach the pool, nothing should change unless you have a config issue in the bios. Is it possible that a fresh install and your old config look similar, or do you have many shares and users set up that persist?

 

[Ptera]

Yes this unit receives replication tasks from NS1 here. If the config was gone with the new USB drive I would not be able to get to the server with the assigned public address with out reprogramming it since the default behavior is DHCP and there is no DHCP server on our core to hand it the address.

 

[jkh]

I think you're running into some weird issue with USB support on your system - obviously there is no general issue with booting FreeNAS off of a USB stick or 90% of the people on this Forum wouldn't be able to use FreeNAS at all. In your particular case, however, I'd just get a SATA DOM (or a pair of them, with 9.3) and install (INSTALL, not just grab the pre-installed .img!) FreeNAS onto them. We don't use USB sticks in any of our production systems, not even the most minimal FreeNAS Mini comes with a USB stick. SATA DOMs. Because life is already short enough. :)

 

[Ptera]

Turns out there is a FreeNAS image installed on the 16Gig solid state drive that was supposed to be the cache drive.
I guess it was booting from the USB drive and then loading the config from the Solid State Drive.

 

[mjws00]

Bingo. Glad you got it.

 

[Ptera]

I wish I did - reloaded the image on the USB drive and now I get...
Mounting from ufs:/dev/ufs/FreeNASs1a failed with error 19
Reading the other threads for this but no luck yet getting it to work.