cshake
Cadet
- Joined
- Jan 21, 2016
- Messages
- 7
Short problem description:
Hardware is FreeNAS Mini, less than a year old, came with 9.3 and had been solid for many months, no issues at all. Updated in the 9.3-STABLE track on 2016-01-21 to the version that was released on 2016-01-19 or so (from a December version). WebGUI didn't come back after the reboot, but cifs shares and jail (Plex) are working fine. Messages on boot show that nginx is failing to start with errors on binding to its ipv6 listen address.
Timeline/detailed description:
I got emails from my server telling me that a new update was ready. I waited until the night of Jan 20 to tell it to apply the updates, then as it started to run I went to bed. At some point overnight my desktop crashed (where I'd had the web browser open), and when I checked it in the morning the server was still up and I had another email that an update was ready - apparently it hadn't succeeded. I opened the WebGUI again and went to the update tab; it showed that one was ready so I told it to apply. It jumped quickly to 5/5 on installing and then spit out a message to the console at the bottom saying that it had a problem with creating and/or deleting the boot image/partition/revision/whatever (going from memory here), but I figured that was due to it creating that boot entry when I first tried the update, so I ignored it. I then told it to verify my installation, which failed on one file (/etc/resolv.conf), and since I'd never had network issues and this running with a static IP on my local network, I figured that didn't really matter, and told it to reboot.
I waited the usual few minutes for it to come back, it didn't, so I left for work. When I got home from work, still on the rebooting screen (uh oh, it failed...), and attempts to refresh the page now give me "can't establish a connection". Time for troubleshooting...
Here's where I should point out my difficulties: I never enabled sshd since I didn't ever need it, so I don't have easy console access (but being a mini, it has IPMI, so that's good). Second, and this is where I slap myself, I don't have any configuration backups. I had been under the (apparently mistaken) impression that rebooting to a previous boot snapshot would fix any software issues, and if there was a hardware issue I figured I'd have to rebuild ground up anyway. So yeah, on with the issue...
I connected via IPMI to the server and the console screen was there, but for some reason it wouldn't take input (let me type, but didn't select the options I chose), refreshed a bunch of times, and threw "getty returning too quickly, sleeping for 30sec", then just hung. I have not updated the firmware on the mini, and was using a slightly out of date java (v8 update 66) with Pale Moon browser, but hadn't ever had that issue before. Told the server to graceful power cycle, and selected the last working December boot snapshot in grub, expecting all to work again (right?). When it booted, I kept screencaps of the boot messages, and had this as a part:
To test that hypothesis, I restarted it again and went to the newest boot image, single-user mode, and checked in the nginx.conf, but it's obviously the fallback and not the live version used by FreeNAS, since it was configured only for port 80 and didn't have anything about certificates or ssl (which I use). This was enough bashing my head against the wall and I didn't know where to go, so I gave up and let it boot fully (still no nginx server, but jails and cifs are fine, and that's what I really need anyway).
Now today, I got home from work and thought that maybe the IPMI console giving getty errors could be due to browser and/or java version, so I updated java on my desktop and opened the dusty Firefox I still had. I saw the same screen, it showed getty errors, but I repeatedly hit "9" "<enter>" and after a few seconds I have the console. I looked at nginx.conf and saw that it's the active one (is set to listen on that ip and port, certificates in place, etc), but before screwing around in it I figured I should really ask for some help to see if I'm missing something obvious.
With the working console, I did scp the config file (/data/freenas-v1.db) to my linux laptop that had a ssh server running, so now I have that at least, but if the server won't start nginx with it I don't feel like just reinstalling from factory and uploading the config will fix it. I also checked /etc/resolv.conf (since it supposedly failed verification earlier) and it has two correct nameservers and the search is my domain, so I don't see anything wrong with it. The search domain isn't technically what I'm using at my house, but it's one I own and have a VPS using in a datacenter, and regardless that shouldn't stop the webserver from starting.
So, sorry about the wall of text, but I wanted to give the most info I could. Does anyone have any ideas?
Hardware is FreeNAS Mini, less than a year old, came with 9.3 and had been solid for many months, no issues at all. Updated in the 9.3-STABLE track on 2016-01-21 to the version that was released on 2016-01-19 or so (from a December version). WebGUI didn't come back after the reboot, but cifs shares and jail (Plex) are working fine. Messages on boot show that nginx is failing to start with errors on binding to its ipv6 listen address.
Timeline/detailed description:
I got emails from my server telling me that a new update was ready. I waited until the night of Jan 20 to tell it to apply the updates, then as it started to run I went to bed. At some point overnight my desktop crashed (where I'd had the web browser open), and when I checked it in the morning the server was still up and I had another email that an update was ready - apparently it hadn't succeeded. I opened the WebGUI again and went to the update tab; it showed that one was ready so I told it to apply. It jumped quickly to 5/5 on installing and then spit out a message to the console at the bottom saying that it had a problem with creating and/or deleting the boot image/partition/revision/whatever (going from memory here), but I figured that was due to it creating that boot entry when I first tried the update, so I ignored it. I then told it to verify my installation, which failed on one file (/etc/resolv.conf), and since I'd never had network issues and this running with a static IP on my local network, I figured that didn't really matter, and told it to reboot.
I waited the usual few minutes for it to come back, it didn't, so I left for work. When I got home from work, still on the rebooting screen (uh oh, it failed...), and attempts to refresh the page now give me "can't establish a connection". Time for troubleshooting...
Here's where I should point out my difficulties: I never enabled sshd since I didn't ever need it, so I don't have easy console access (but being a mini, it has IPMI, so that's good). Second, and this is where I slap myself, I don't have any configuration backups. I had been under the (apparently mistaken) impression that rebooting to a previous boot snapshot would fix any software issues, and if there was a hardware issue I figured I'd have to rebuild ground up anyway. So yeah, on with the issue...
I connected via IPMI to the server and the console screen was there, but for some reason it wouldn't take input (let me type, but didn't select the options I chose), refreshed a bunch of times, and threw "getty returning too quickly, sleeping for 30sec", then just hung. I have not updated the firmware on the mini, and was using a slightly out of date java (v8 update 66) with Pale Moon browser, but hadn't ever had that issue before. Told the server to graceful power cycle, and selected the last working December boot snapshot in grub, expecting all to work again (right?). When it booted, I kept screencaps of the boot messages, and had this as a part:
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nging: [emerg] bind() to [fe80::d250:99ff:fe64:aa2e]:433 failed (49: Can't assign requested address)
nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
Starting nginx.
nging: [emerg] bind() to [fe80::d250:99ff:fe64:aa2e]:433 failed (49: Can't assign requested address)
/etc/rc: WARNING: failed to start nginx
As expected, the WebGUI was not available. To note here, I don't actually care about ipv6 right now because my ISP doesn't use it and I'm only using ipv4 with NAT in my house currently (yes, always better to have, and I want to have it eventually). Would disabling the listen line for the ipv6 address and leaving only the ipv4 listen line fix this issue?nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nging: [emerg] bind() to [fe80::d250:99ff:fe64:aa2e]:433 failed (49: Can't assign requested address)
nginx: configuration file /usr/local/etc/nginx/nginx.conf test failed
Starting nginx.
nging: [emerg] bind() to [fe80::d250:99ff:fe64:aa2e]:433 failed (49: Can't assign requested address)
/etc/rc: WARNING: failed to start nginx
To test that hypothesis, I restarted it again and went to the newest boot image, single-user mode, and checked in the nginx.conf, but it's obviously the fallback and not the live version used by FreeNAS, since it was configured only for port 80 and didn't have anything about certificates or ssl (which I use). This was enough bashing my head against the wall and I didn't know where to go, so I gave up and let it boot fully (still no nginx server, but jails and cifs are fine, and that's what I really need anyway).
Now today, I got home from work and thought that maybe the IPMI console giving getty errors could be due to browser and/or java version, so I updated java on my desktop and opened the dusty Firefox I still had. I saw the same screen, it showed getty errors, but I repeatedly hit "9" "<enter>" and after a few seconds I have the console. I looked at nginx.conf and saw that it's the active one (is set to listen on that ip and port, certificates in place, etc), but before screwing around in it I figured I should really ask for some help to see if I'm missing something obvious.
With the working console, I did scp the config file (/data/freenas-v1.db) to my linux laptop that had a ssh server running, so now I have that at least, but if the server won't start nginx with it I don't feel like just reinstalling from factory and uploading the config will fix it. I also checked /etc/resolv.conf (since it supposedly failed verification earlier) and it has two correct nameservers and the search is my domain, so I don't see anything wrong with it. The search domain isn't technically what I'm using at my house, but it's one I own and have a VPS using in a datacenter, and regardless that shouldn't stop the webserver from starting.
So, sorry about the wall of text, but I wanted to give the most info I could. Does anyone have any ideas?