Upgrades Not Happening (9.3-CURRENT)

[ewhac]

Lately, I've been trying to apply the latest automatic updates to my FreeNAS box, and the update process is hanging. The updates appear to download, and the update process seems to start, but the GUI hangs and never completes.

Last night, I tried something a bit more aggressive, and downloaded the 9.3-CURRENT GUI Upgrade image. The image uploaded to the box, and once again the update process appeared to launch but, again, seemed to hang and never completed. The GUI prompt never left the "Uploading Step 2/2" phase.

I pulled out the clown hammer and rebooted the system, and tried the GUI Upgrade path again. This time the update failed, as the dataset the previous attempt had unpacked into already existed. I SSHed into the machine, `zfs destroy`ed that dataset (which I was deeply uncomfortable doing), and started the upgrade again. The dataset was re-created, re-populated, and mounted, but then the update script seemed to just sit there. CPU and I/O activity were minimal, and I could not figure out what the update script was hanging on. The logs did not contain any clues as to what it was up to or what difficulties it was having.

Everything else about the system appears to be working normally -- CIFS file sharing, SSH logins, miniDLNA in a jail, etc. It just seems aggressively disinterested in completing an update.

Just now, as I am writing this up, I recall a couple of weird incidents that make me wonder if it's an IPv6 issue. To destroy the upgrade image dataset, I had to 'su' to root. The `zfs destroy` command took about 60 seconds to complete, during which there was no disk activity (the blinky light didn't blink). When I then exited that shell, the exit took about 45 seconds to complete. In my experience, weird delays like that happen when the system tries to do something via IPv6, fails, and finally falls back to IPv4.

This is the first major issue I've had with FreeNAS. I can't begin to imagine what I screwed up. Does this sound repairable, or will I need to reinstall from scratch?

 

[dlavigne]

Just to verify, how big is the boot device and how many BEs are displayed in System -> Boot?

 

[ewhac]

Just to verify, how big is the boot device

A SanDisk USB key, just a hair more than 4GB:

Code:

[root@NASthing] ~# camcontrol devlist
[ ... ]
<SanDisk Cruzer Fit 1.26>  at scbus7 target 0 lun 0 (da0,pass4)
[root@NASthing] ~# camcontrol readcap da0
Last Block: 7821311, Block Length: 512 bytes

and how many BEs are displayed in System -> Boot?

Unless there's a way to work that out from the command line, you'll have to wait until I get home before I can tell you that...

 

[Ericloewe]

Sounds like the boot device might be too full.

 

[ewhac]

...how many BEs are displayed in System -> Boot?

And the answer is: Four.

 

[cyberjock]

OK, but the manual says you should have an 8GB boot device minimum. You have about 1/2 of that, and during the upgrade process you need free space above and beyond what you normally need.

So I think the answer to the problem is "user configuration issue" due to inappropriate hardware.

 

[ewhac]

OK, but the manual says you should have an 8GB boot device minimum.

It does? (*checks manual*) Oh, it does.

When was that requirement established? When I first installed FreeNAS (late version 8-dot-something), I recall the manual saying you needed 2GB of storage -- 1GB for each of the images, and the upgrade process would ping-pong between them.

So I think the answer to the problem is "user configuration issue" due to inappropriate hardware.

Well, all righty, then; thank you kindly for pointing that out. Guess I'll pick up a 16GB USB stick this weekend. Can I copy the images off the existing USB stick to the new one (how?), or am I better off starting from a fresh install?

Would it be appropriate to file a bug to the effect of, "System update does not fail informatively when minimum required storage not available?"

 

[Ericloewe]

It's easier to just install to the new device (s).
9.3's boot device snapshots require more storage than the mostly static image from the old versions.
4GB is marginal, but 16GB will be fine for a while (I installed 9.3 in February and despite the frequent updates I'm still at roughly 4GB used.

 

[jlpellet]

I'd suggest a check would be to delete all but 2 boot environments through the gui to see if freeing up space fixes the problem. Not sure why one would need more than 2-3 versions.

 

[ewhac]

Just got through installing 9.3 on to a fresh 16GB USB stick, restored the config DB, and now almost everything appears to be happy again.

Warts I noticed:

Although the FreeNAS server gets its IP address and hostname from a separate DHCP server, I wasn't able to connect to the Web UI using its hostname. I had to use the IP address directly. Not sure what's up with that. (This problem vanished after the config DB was restored.)

After restoring the config, the UPS daemon tools (NUT) is reporting an error in the logs:

Code:

Mar 21 17:03:30 NASthing kernel: <118>chown: /usr/local/etc/nut/ups.conf: No such file or directory
Mar 21 17:03:30 NASthing kernel: <118>chown: /usr/local/etc/nut/upsd.users: No such file or directory
Mar 21 17:03:30 NASthing kernel: <118>chown: /usr/local/etc/nut/upsd.conf: No such file or directory
Mar 21 17:03:30 NASthing kernel: <118>chmod: /usr/local/etc/nut/ups.conf: No such file or directory
Mar 21 17:03:30 NASthing kernel: <118>chmod: /usr/local/etc/nut/upsd.users: No such file or directory
Mar 21 17:03:30 NASthing kernel: <118>chmod: /usr/local/etc/nut/upsd.conf: No such file or directory
Mar 21 17:03:30 NASthing kernel: <118>Starting nut_upslog.
Mar 21 17:03:30 NASthing root: /etc/rc: WARNING: failed to start nut_upsmon
Mar 21 17:03:30 NASthing kernel: <118>Network UPS Tools upslog 2.7.2
Mar 21 17:03:30 NASthing kernel: <118>logging status of apc-es-750@gateway:3493 to /var/log/ups.log (300s intervals)
Mar 21 17:03:30 NASthing kernel: <118>Starting nut_upsmon.
Mar 21 17:03:30 NASthing kernel: <118>Network UPS Tools upsmon 2.7.2
Mar 21 17:03:30 NASthing kernel: <118>fopen /var/db/nut/upsmon.pid: No such file or directory
Mar 21 17:03:30 NASthing kernel: <118>Unable to use old-style MONITOR line without a username
Mar 21 17:03:30 NASthing kernel: <118>Convert it and add a username to upsd.users - see the documentation
Mar 21 17:03:30 NASthing kernel: <118>Fatal error: unusable configuration
Mar 21 17:03:30 NASthing kernel: <118>/etc/rc: WARNING: failed to start nut_upsmon

The UPS GUI panel has a username set, but no password.

 

[ewhac]

After restoring the config, the UPS daemon tools (NUT) is reporting an error in the logs:
[ ... ]
The UPS GUI panel has a username set, but no password.

I stopped the UPS service, opened the GUI panel, and re-entered the password. I re-started the service, and now it seems happy. Dunno where that password had got to...