Upgrade Failing via CLI

[chris0583]

Hey gents I hope someone can help me on this one.

I am trying to upgrade from 9.2.1.5 to 9.2.1.9 via CLI as my Web GUI is down and I don't have access to the pysical machine right now. I am getting "An unhandled exception was thrown by the application." via the web gui. This issue just showed up. The system is configured to boot on Monday at 1AM files are rsynced to the machine and then performs a automated shutdown at Monday 6AM. System has been running on this hardware for 4 years and works perfectly. I was checking systems for shell shock issues and got to this one and I could not get to the web interface. I figured the best way to fix it is to upgrade to the latest 9.2.x version.

This is what I am getting when I run an upgrade via CLI. Can anyone help me

[root@archive] ~# mount -uw /dev/ufs/FreeNASs1a
[root@archive] ~# xz -cd /mnt/Archive_1/Media/temp/FreeNAS-9.2.1.9-RELEASE-x64.GUI_Upgrade.txz | sh /root/update && touch /data/need-update
+ PARTNUM=2
+ OTHER_PARTNUM=1
+ set -e
+ . /etc/nanobsd.conf
+ NANO_DRIVE=ufs/FreeNAS
+ mount
+ grep ufs/FreeNASs2
+ echo ufs/FreeNAS
+ cut -f2 -d/
+ LABELNAME=FreeNAS
+ glabel status
+ awk '/ufs\/FreeNASs1a/ { print $3; }'
+ sed -e s,s1a,,
+ TARGET_DRIVE=da0
+ dd if=/dev/zero of=/dev/da0s2 bs=1m count=1
+ echo 5470
+ dd if= of=/dev/da0s2 obs=128k conv=sparse
+ wait
+ fsck_ffs -n /dev/da0s2a
Can't stat /dev/da0s2a: No such file or directory Can't stat /dev/da0s2a: No such file or directory + tunefs -L FreeNASs2a /dev/da0s2a
Can't stat /dev/da0s2a: No such file or directory
+ tunefs -L FreeNASs2a /dev/da0s2a
tunefs: /dev/da0s2a: could not find special device
[root@archive]

Current System:
[root@archive] ~# freenas-debug -h
+--------------------------------------------------------------------------------+
+ FreeNAS-9.2.1.5-RELEASE-x64 (80c1d35) + +--------------------------------------------------------------------------------+
Operating system type: FreeBSD
Operating system release: 9.2-RELEASE-p4
Operating system revision: 199506
Kernel version: FreeBSD 9.2-RELEASE-p4 #0 r262572+17a4d3d: Wed Apr 23 10:09:38 PDT 2014 root@build3.ixsystems.com:/tank/home/jkh/build/9.2.1/freenas/os-base/amd64/fusion/jkh/9.2.1/freenas/FreeBSD/src/sys/FREENAS.amd64
Hostname: archive.xyx.com
Name of kernel file booted: /boot/kernel/kernel

+--------------------------------------------------------------------------------+
+ Hardware +
+--------------------------------------------------------------------------------+
Machine class: amd64
System architecture: amd64
Machine model: Intel(R) Core(TM)2 Quad CPU Q6700 @ 2.66GHz
Number of active CPUs: 4
Number of CPUs online: 4
Current CPU frequency: 2.28 Ghz
Physical Memory: 3.98 GiB

 

[cyberjock]

Well, I hate to say it, but doing updates via the CLI is really completely unsupported. That being said, I couldn't tell you if the way you are updating is even how the devs would recommend it as a "hacky solution". I've never tried to do that via the CLI.

You do have only 4GB of RAM, and if you are running ZFS there's a *very* high probability that the problem is because of insufficient RAM.

I'd recommend you do one of the following:

1. Choose not to update it until you can be local to the machine (an update that renders the system unusable won't be useful if you can't even SSH into it).
2. If you have port forwarding enabled in the SSH settings of the WebGUI you should be able to do an SSH tunnel to get to the WebGUI. Note that you still have the risk of #1 happening, and if the SSH service can't/doesn't/won't start then you'll be sunk
3. Try to get someone to touch the box locally and do the upgrade via the WebGUI.

Personally, I don't upgrade my boxes remotely unless I have full IPMI and WebGUI access so that I have total control if things go sideways. We know Murphy loves all of us very, very much.

 

[chris0583]

Well, I hate to say it, but doing updates via the CLI is really completely unsupported. That being said, I couldn't tell you if the way you are updating is even how the devs would recommend it as a "hacky solution". I've never tried to do that via the CLI.

You do have only 4GB of RAM, and if you are running ZFS there's a *very* high probability that the problem is because of insufficient RAM.

I'd recommend you do one of the following:

1. Choose not to update it until you can be local to the machine (an update that renders the system unusable won't be useful if you can't even SSH into it).
2. If you have port forwarding enabled in the SSH settings of the WebGUI you should be able to do an SSH tunnel to get to the WebGUI. Note that you still have the risk of #1 happening, and if the SSH service can't/doesn't/won't start then you'll be sunk
3. Try to get someone to touch the box locally and do the upgrade via the WebGUI.

Personally, I don't upgrade my boxes remotely unless I have full IPMI and WebGUI access so that I have total control if things go sideways. We know Murphy loves all of us very, very much.

CyberJock,

Thanks for the quick reply.

Port forwarding is open and working fine. I had it checked locally and the issue is the same. "An unhandled exception was thrown by the application" via the web GUI. I don't think RAM is an issue as this same setup has been running for years this way and the GUI worked fine a few weeks ago last time I check the system. This being said I am having another 4 GB put in the machine to bring her up to 8 GB as I see that is recommended.

If that does not work I will have the system updated locally to the latest 9.2.x branch or should I go directly to 9.3?

I will keep the board informed of the progress and see if I can isolate the problem.

Chris

 

[cyberjock]

Port forwarding is open and working fine. I had it checked locally and the issue is the same. "An unhandled exception was thrown by the application" via the web GUI. I don't think RAM is an issue as this same setup has been running for years this way and the GUI worked fine a few weeks ago last time I check the system. This being said I am having another 4 GB put in the machine to bring her up to 8 GB as I see that is recommended.

If I got $5 every time someone claimed it worked fine for years, then found out more RAM fixed the problem, I'd be retired. I've even done Teamviewer sessions with people and watched them learn the lesson first-hand.

That being said, I'd upgrade to 9.2.1.9, then 9.3. If possible, upgrade to 9.2.1.9 for a few days, then go to 9.3 if everything is fine.

 

[chris0583]

Cyberjock

Sorry but your not gonna be $5 buck richer :). Could not find mem in the data center for the unit. I walked them thru a 9.2.1.9 upgrade and the web interface came right up. Just updated to 9.3 as well and everything is running sweet. Ran an full rsync from all sites (25 Streams at once) mem did not go above 2GB Thanks for all the help and advice. (Have the extra 4 GB mem on order just in case)

Chris