Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

Unable to boot after upgrading to 11.2u1

Joined
Dec 27, 2018
Messages
12
Thanks
0
#1
Since the server rebooted for the 11.2u1 update using the GUI, I have not been able to get my system back operational. It gets to the following screen and just hangs.

IMG_1942.JPG

Troubleshooting steps:
- Tried booting with a fresh install on another usb drive. Same issue
- Tried a multitude of bios changes but nothing seems to get past this point

The system started on 11.1u2 and kept up with subsequent updates. Successfully upgraded to 11.2 when that was officially released. Multiple reboots were successful while 11.2 was installed.

Specs:
Lenovo TS140
16 Gigs of ram
32 GB SanDisk USB 3.0 Flash Drive (FreeNas Boot Drive)

Any thoughts?
 

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
16,139
Thanks
3,867
#2
Have you tried 11.2-Release again?
 

Ericloewe

Not-very-passive-but-aggressive
Moderator
Joined
Feb 15, 2014
Messages
16,139
Thanks
3,867
#4
That raises a few questions. I mean, clearly it finds the bootloader and it gets as far as finding the boot pool...

Have you tried disconnecting all non-boot devices?
 

jlpellet

FreeNAS Experienced
Joined
Mar 21, 2012
Messages
155
Thanks
20
#5
I updated 5 systems from 11.2 release & had a similar problem with 1 of them (other 4 fine). System got slightly further in process but just stopped with error msg. Rebooted & chose 11.2 release boot from bootloader & 11.2R booted fine. Made 11.2R in the GUI & rebooted. At this point 11.2R boot failed & boot 11.2U1 failed. Tried to boot from 11.1U6 (earlier install) & it failed as well. I'm assuming the system did not like going back to the GRUB bootloader. Tried a 11.2U1 iso fresh install. Install stated successful but failed the same on reboot. Tried 2 different boot SSDs on 2 different ports with same error. Installed 11.1U6 then changed trains & updated to 11.1U1 with same failure on reboot. Going to try an 11.2 release fresh next.
 
Joined
Dec 27, 2018
Messages
12
Thanks
0
#6
All non boot devices are disconnected. At this point I have a brand new install on a usb drive connected to it.

Unfortunately where I am at in the process does not allow me to select anything. From what I am seeing I got from post directly into the Screenshot posted above. The only changes that happen on that screen is during the probing of boot block the ... increments and then it outputs the zfs pools and the ufs pools.
 
Joined
Jan 1, 2019
Messages
8
Thanks
1
#7
I'm freezing at the exact same point. I also have a 'TS140', although all that remains of the original is the motherboard (C226). 11.2 I had no issues with. I've been working on tidying up/modding my case a bit for cable management so the machine has powered cycled quite a few times since 11.2 came out. A re-install of 11.2U1 did not change anything. BIOS reset had not benefits. Leaving it sit here for 12+ hours in case it was just crazy slow did nothing to help. I was booting from mirrored 32GB Sandisk Cruzers. I did a clean install with ALL hardware possible to remove eliminated leaving me with 1 8GB stick of RAM (1600mhz, ECC, DDR3L), CPU (E3-1245v3), onboard LAN (Intel), onboard video (Intel), onboard USB controller, 2 new USB sticks (the ones bundled with the WD 10tb Easystores), and no HDDs. That had the same exact result. I'm burning an 11.2-Release image now to give a try next. I'm also going to burn a Linux live disc just to verify other systems are still booting properly and that the update didn't somehow cause issues in the hardware.
 
Joined
Dec 27, 2018
Messages
12
Thanks
0
#8
I’m not alone!
I did not have success with fresh 11.2 release loaded on another flash drive.

I was thinking of trying to load Ubuntu on a flash drive tomorrow as a test as well.

I can’t find anything on FreeBSD forums about failure to boot into the zfs pool.
 
Joined
Jan 1, 2019
Messages
8
Thanks
1
#9
I’m not alone!
I did not have success with fresh 11.2 release loaded on another flash drive.

I was thinking of trying to load Ubuntu on a flash drive tomorrow as a test as well.s

I can’t find anything on FreeBSD forums about failure to boot into the zfs pool.
A fresh set of flash drives booted into 11.1u6 just fine for me, and I can import my pools successfully. 11.2's bootloader is the issue I think, but you see how minimal the output we get is so there's not much to go off of there. I've conceded to rolling back to 11.1u6 for now until 11.2 is debugged. I'm still mostly confused by how 11.2 worked perfectly for a month but fresh installing it doesn't work. I'll probably try doing an in place update from 11.1 to 11.2 to test that as well since that worked fine before. My other idea is swapping over to my SSDs as boot drives (I really don't need an L2ARC on each of my arrays, lol) as I've found a few posts are online about issues with the bootloader caused by slow read speeds.
 
Joined
Jun 26, 2017
Messages
8
Thanks
2
#10
I had this experience too - boot configuration completely broken by upgrade to 11.2-U1. The boot loader hangs for around 5 minutes, and then when it does finally load, the error on my system showed that it was trying to use my mechanical storage pool as the boot device instead of freenas-boot/ROOT/default

I tried upgrade installs to the pen drive of 11.2 and 11.2-U1 but neither could resolve it - I have rebuilt 11.2 on a blank pendrive to get my system back into service. I've been happy with 11.2, but would currently advise against upgrading to 11.2-U1 - I'm sure it works for some people, but equally I've noticed a few people left with unbootable systems after upgrade.
 

jlpellet

FreeNAS Experienced
Joined
Mar 21, 2012
Messages
155
Thanks
20
#11
After trying 11.2 upgrade, 11.2U1 fresh install, & 11.1U6 upgrades last week that all failed to work with the boot failure noted, I left the system at 11.1U6 last week. Tried the 11.2U1 upgrade whis morning from 11.1U6 & it worked correctly. Don't know if random or if something changed in last week but wanted to update with today's succcess.
 
Joined
Jan 1, 2019
Messages
8
Thanks
1
#12
Soo.... I accidentally found a method to boot 11.2U1... lol

I pulled out my working 11.1u6.3 flash drives, and did a fresh install of 11.2U1 to my SSD mirror. Shocker, it didn't boot. It froze at the same exact point it had in prior attempts. On the upside, this ruled out the USB controller and pretty well confirmed for me this is a bug in 11.2u1 and not something in my hardware. I didn't feel like wiping the SSDs right now in case I wanted to try and dive into the files and try to find a useful log, and didn't want to open my case so I got lazy and just swapped the boot order so that it now goes USB FDD > Sata 1 & 3 (my SSDs) > DVD. I stuck the flash drives back in, booted up, it went into Grub like you'd expect and I selected to boot 11.1u6.3 like normal. I then went back to my kitchen to prep dinner, but Plex never came up so I pulled up FreeNAS on my phone. It loaded to the 11.2 interface, so I refreshed thinking it was an old cached page but it didn't change o_O. Head over to my desktop... and WTF IT IS RUNNING 11.2U1.

The workaround for the broken 11.2U1 bootloader is apparently a pretty obvious one, just don't use it! Get it to use GRUB still by having a dual FreeNAS install media setup and GRUB set as the earlier boot device. Why it loaded from the SSDs that came later in my boot order and wasn't even listed in GRUB I've no clue, but for now it is working so I'm not going to complain. I'm no GRUB expert by any extent to explain why this works.

Edit: all right... this leads to lots of errors. Part of the system thinks it is 11.1, part thinks it is 11.2
Here's an example when trying to get to the plugin list.
"404 Client Error: Not Found for url: http://pkg.cdn.trueos.org/iocage/11.1-RELEASE/All "

Edit 2: 11.2U1 is now listed on the roadmap as due at the end of the month. They must have pulled the update. We very well may have received an accidentally released alpha update as a holiday gift.
 
Last edited:

jlpellet

FreeNAS Experienced
Joined
Mar 21, 2012
Messages
155
Thanks
20
#14
I believe, from the blog, that the release last week reflects a security fix out-of-sequence update. Note releases do not appear on the roadmap after they are issued. Last week's update was named "11.2-RELEASE-U1" & the next scheduled release is "11.2-U1" and, until last week's release, both were listed on the roadmap separately. Confusing but not, I think, an inadvertent release.
 
Joined
Jan 1, 2019
Messages
8
Thanks
1
#16
Anyone get their system up and running yet?
As of a few minutes ago sort of, but I'm doing some more testing now. I just did another strip down, so that all that is attached is the ODD and two SSDs. Clean installing 11.2 gave the same exact freeze point we've had. Clean installing 11.1_u6.3 it installed and booted properly. Installing 11.2 clean over top of it again gave the standard boot freeze (I wanted to confirm it wasn't old 11.2u1 issues that for some reason weren't eliminated during the clean install). What I've just done now though, is clean install 11.1u6.3 then upgraded to 11.2u1 directly through the upgrade process in the UI... and it booted o_O. I'm going to try rebooting a few times to see what happens. If all goes well, I'll start rebuilding my jails while leaving my working 11.1u6.3 flash drives at the ready to swap back in should it spontaneously die again.

Edit: Welp, it died with a reboot. Setup it up again, died with a shutdown and cold boot. So I can install it and get a first boot coming out of the 11.2 installer, but after that it either has a very low chance to successfully boot or will never boot. I'll try again once 11.2u2 comes out next week, but if that doesn't work I'll decide between staying with 11.1 or leaving FreeNAS. If I stay with 11.1, I'll probably move FreeNAS into a VM and stop using jails.
 
Last edited:
Joined
Jan 1, 2019
Messages
8
Thanks
1
#17
I believe, from the blog, that the release last week reflects a security fix out-of-sequence update. Note releases do not appear on the roadmap after they are issued. Last week's update was named "11.2-RELEASE-U1" & the next scheduled release is "11.2-U1" and, until last week's release, both were listed on the roadmap separately. Confusing but not, I think, an inadvertent release.
I had never seen them as separate listings on the roadmap, but I may have just missed it. They are definitely recognizing it as U1 now though and the one due end of this month is now U2.
 
Joined
Jan 21, 2019
Messages
1
Thanks
0
#18
As of a few minutes ago sort of, but I'm doing some more testing now. I just did another strip down, so that all that is attached is the ODD and two SSDs. Clean installing 11.2 gave the same exact freeze point we've had. Clean installing 11.1_u6.3 it installed and booted properly.
Hey.. maybe you can help me out. I want to copy your solution of downgrading to get my freeNAS box back up and running... but.. I'm a freeNAS noob, and I installed 11.2 in December. This was my first ever freeNAS install. I started getting similar boot problems as everyone else here when I was trying to reinstall 11.2 this week to replace what I suspected to be a failing USB flash drive. For me it also finds the boot pool, but takes like 10min trying during loading zfs boot pool, and then I get "middlewared failed to load within 4 minutes" error. Tried two different flash drives with the freshest 11.2 iso, and also installed 11.2 on a drive in a VM. Tried to boot from that drive on my hardware and same thing happens.

If I make a boot flash drive for 11.1, do you have any idea if my config backup and jails from 11.2 will work? Jails are not on the boot device, so maybe? But I've never had 11.1 installed before since I just made my first freeNAS box like 5 weeks ago, so I'm somewhat doubtful I can upload my configuration from 11.2 to a fresh install of 11.1.

That'll teach me to download the latest "stable" release when I try out a new OS!!
 
Last edited:
Joined
Jan 1, 2019
Messages
8
Thanks
1
#19
Hey.. maybe you can help me out. I want to copy your solution of downgrading to get my freeNAS box back up and running... but.. I'm a freeNAS noob, and I installed 11.2 in December. This was my first ever freeNAS install. I started getting similar boot problems as everyone else here when I was trying to reinstall 11.2 this week to replace what I suspected to be a failing USB flash drive. For me it also finds the boot pool, but takes like 10min trying during loading zfs boot pool, and then I get "middlewared failed to load within 4 minutes" error. Tried two different flash drives with the freshest 11.2 iso, and also installed 11.2 on a drive in a VM. Tried to boot from that drive on my hardware and same thing happens.

If I make a boot flash drive for 11.1, do you have any idea if my config backup and jails from 11.2 will work? Jails are not on the boot device, so maybe? But I've never had 11.1 installed before since I just made my first freeNAS box like 5 weeks ago, so I'm somewhat doubtful I can upload my configuration from 11.2 to a fresh install of 11.1.

That'll teach me to download the latest "stable" release when I try out a new OS!!
If you setup your Jails to run based on 11.2-Release, they will not work in 11.1. if for some reason you'd selected 11.1 or lower they can be loaded up in 11.1. Your config from 11.2 definitely can't be loaded in 11.1.

I actually might have changed my personal plan/extended it. I'm debating switching back to esxi and virtualizing FreeNAS. I'd guess this will fix the boot issue, but I'd need a second HBA to pass through all of my HDDs.

11.2 is messier than other releases on the past in my experience. They switched bootloaders for some reason, and the new one seems to be unreliable.
 
Joined
Jan 1, 2019
Messages
8
Thanks
1
#20
https://redmine.ixsystems.com/issues/37388

Went to report the bug, saw it was already reported. I've not tried again since disabling PXE. I THINK it was disabled the first time around as I typically disable it (and any unneeded features) before my first install on a machine, but it's worth a try. There is also a new BIOS that came out in Oct 2018 that updates UEFI Network Stack, so possibly that will fix it as well if I'm not already on it. I've tried so many things at this point I can't keep any of the numbers or anything straight now, lol.
 
Top