WebGUI Missing after Update

[cshake]

Short problem description:
Hardware is FreeNAS Mini, less than a year old, came with 9.3 and had been solid for many months, no issues at all. Updated in the 9.3-STABLE track on 2016-01-21 to the version that was released on 2016-01-19 or so (from a December version). WebGUI didn't come back after the reboot, but cifs shares and jail (Plex) are working fine. Messages on boot show that nginx is failing to start with errors on binding to its ipv6 listen address.

Timeline/detailed description:
I got emails from my server telling me that a new update was ready. I waited until the night of Jan 20 to tell it to apply the updates, then as it started to run I went to bed. At some point overnight my desktop crashed (where I'd had the web browser open), and when I checked it in the morning the server was still up and I had another email that an update was ready - apparently it hadn't succeeded. I opened the WebGUI again and went to the update tab; it showed that one was ready so I told it to apply. It jumped quickly to 5/5 on installing and then spit out a message to the console at the bottom saying that it had a problem with creating and/or deleting the boot image/partition/revision/whatever (going from memory here), but I figured that was due to it creating that boot entry when I first tried the update, so I ignored it. I then told it to verify my installation, which failed on one file (/etc/resolv.conf), and since I'd never had network issues and this running with a static IP on my local network, I figured that didn't really matter, and told it to reboot.

I waited the usual few minutes for it to come back, it didn't, so I left for work. When I got home from work, still on the rebooting screen (uh oh, it failed...), and attempts to refresh the page now give me "can't establish a connection". Time for troubleshooting...

Here's where I should point out my difficulties: I never enabled sshd since I didn't ever need it, so I don't have easy console access (but being a mini, it has IPMI, so that's good). Second, and this is where I slap myself, I don't have any configuration backups. I had been under the (apparently mistaken) impression that rebooting to a previous boot snapshot would fix any software issues, and if there was a hardware issue I figured I'd have to rebuild ground up anyway. So yeah, on with the issue...

I connected via IPMI to the server and the console screen was there, but for some reason it wouldn't take input (let me type, but didn't select the options I chose), refreshed a bunch of times, and threw "getty returning too quickly, sleeping for 30sec", then just hung. I have not updated the firmware on the mini, and was using a slightly out of date java (v8 update 66) with Pale Moon browser, but hadn't ever had that issue before. Told the server to graceful power cycle, and selected the last working December boot snapshot in grub, expecting all to work again (right?). When it booted, I kept screencaps of the boot messages, and had this as a part:

Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nging: [emerg] bind() to [fe80::d250:99ff:fe64:aa2e]:433 failed (49: Can't assign requested address)
nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
Starting nginx.
nging: [emerg] bind() to [fe80::d250:99ff:fe64:aa2e]:433 failed (49: Can't assign requested address)
/etc/rc: WARNING: failed to start nginx​

As expected, the WebGUI was not available. To note here, I don't actually care about ipv6 right now because my ISP doesn't use it and I'm only using ipv4 with NAT in my house currently (yes, always better to have, and I want to have it eventually). Would disabling the listen line for the ipv6 address and leaving only the ipv4 listen line fix this issue?

To test that hypothesis, I restarted it again and went to the newest boot image, single-user mode, and checked in the nginx.conf, but it's obviously the fallback and not the live version used by FreeNAS, since it was configured only for port 80 and didn't have anything about certificates or ssl (which I use). This was enough bashing my head against the wall and I didn't know where to go, so I gave up and let it boot fully (still no nginx server, but jails and cifs are fine, and that's what I really need anyway).

Now today, I got home from work and thought that maybe the IPMI console giving getty errors could be due to browser and/or java version, so I updated java on my desktop and opened the dusty Firefox I still had. I saw the same screen, it showed getty errors, but I repeatedly hit "9" "<enter>" and after a few seconds I have the console. I looked at nginx.conf and saw that it's the active one (is set to listen on that ip and port, certificates in place, etc), but before screwing around in it I figured I should really ask for some help to see if I'm missing something obvious.

With the working console, I did scp the config file (/data/freenas-v1.db) to my linux laptop that had a ssh server running, so now I have that at least, but if the server won't start nginx with it I don't feel like just reinstalling from factory and uploading the config will fix it. I also checked /etc/resolv.conf (since it supposedly failed verification earlier) and it has two correct nameservers and the search is my domain, so I don't see anything wrong with it. The search domain isn't technically what I'm using at my house, but it's one I own and have a VPS using in a datacenter, and regardless that shouldn't stop the webserver from starting.

So, sorry about the wall of text, but I wanted to give the most info I could. Does anyone have any ideas?

 

[m0nkey_]

Sounds like you had a bad update. Next time you boot up, try selecting the previous boot environment to see if you get the GUI back. If you do, delete the new boot environment and apply the update again.

 

[DrKK]

Doesn't that box come with a DOM? Have you ever heard of a DOM taking a shit like that, m0nkey? I agree it sounds like a badly written update image, but, on a DOM?

 

[m0nkey_]

I'm not saying the DOM is at fault here, a bad update can do some weird shit (remember I had an update that caused my FN to go back to a November update, although I was already on the December update?) Best way to find out if it is just that is to roll back and see if the GUI comes back.

 

[Ericloewe]

Best way to find out if it is just that is to roll back and see if the GUI comes back.

Yeah, start with the easy stuff.

remember I had an update that caused my FN to go back to a November update, although I was already on the December update?

Supposedly, that stops being possible starting now. Some stupid CDN issue was at fault, so now FreeNAS grabs a manifest from an iX server and goes to the CDN. If the CDN is acting up, it'll revert to downloading straight from iX's server.

Of course, there might be a weird issue in this new process that could be causing this particular error.

 

[cshake]

I got the GUI back by going to shell, editing nginx.conf to comment out the ipv6 listen line then `service restart nginx` and it came up fine. This was on the newest boot environment - I'd tried just going back to the previous a few times and it had the same nginx error. I went back to the GUI and verified the install, which flagged nginx.conf as not matching checksum (as expected), then changed the ipv6 listen address in the general tab from the predefined ip to "::" (leaving the static ipv4) and saved, which I'm assuming updated the config file. Also enabled ssh so I should be able to fix this easier in the future if the same problem pops up after a reboot.

I'm guessing that somewhere in the last update or somehow in the last 2 weeks (server showed an uptime of 14 days before I rebooted it for that update) that either the ipv6 address changed (no idea how, my router doesn't do ipv6 DHCP) or something before nginx in the boot order bound on port 443 for that address so nginx couldn't. I'm much more of a *nix person than BSD so I'm kinda just fumbling in the dark here.

Since the GUI is back, my current plan is to wait until the next stable release update comes out and hope for the best.

 

[kingcharles]

Exactly the same thing happened to me!
But I used a keyboard and screen directly connected to my box to adjust the nginx.conf file. I had to resort to ALT-F2 because the console menu was not responding to keyboard entry.
Anyway, thanks for pointing me to the right direction!
Root cause for my situation was probably me fiddling with the interface settings during a router problem. I lost internet connectivity from my FreeNAS box and tried to fix it by resetting the interface but later figured out the problem was the router. Some time after that I did an update and found that the GUI was unreachable...
Most likely I corrupted the config by the interface reset earlier.