Can't start crashplan plugin after upgrade to FreeNAS 9.10

[Tom_]

Hi,

I upgraded FreeNAS from 9.3.1 to 9.10 STABLE without deleting my jails dataset. Now I can't start the crashplan service in my crashplan jail.

After a fresh reboot of FreeNAS the jail starts (and receives an IP) but the service does not start. Trying to start the service from the FreeNAS GUI returns "Some error occurred". Nothing is added to /var/log/messages.

When I go to the crashplan jail on the CLI and run "service crashplan onestart" I get:

Code:

/usr/local/etc/rc.d/crashplan: WARNING: no shebang line in /usr/bin/cpuset
Starting CrashPlan Engine ... Using standard startup
OK

I googled that error and found this bug report, but that doesn't seem to be my issue because the linux kernel module is loaded:

Code:

$ kldload linux
kldload: can't load linux: module already loaded or in kernel

When I restart the jail from the FreeNAS GUI, the jail restarts but does not get an IP, and I have the following in /var/log/messages:

Code:

Mar 30 22:43:28 freenas kernel: epair0a: link state changed to DOWN
Mar 30 22:43:28 freenas kernel: epair0a: link state changed to DOWN
Mar 30 22:43:28 freenas kernel: epair0b: link state changed to DOWN
Mar 30 22:43:28 freenas kernel: epair0b: link state changed to DOWN
Mar 30 22:43:32 freenas kernel: ifa_del_loopback_route: deletion failed: 48
Mar 30 22:43:32 freenas Freed UMA keg (udp_inpcb) was not empty (240 items).  Lost 24 pages of memory.
Mar 30 22:43:32 freenas Freed UMA keg (udpcb) was not empty (2171 items).  Lost 13 pages of memory.
Mar 30 22:43:32 freenas Freed UMA keg (tcp_inpcb) was not empty (90 items).  Lost 9 pages of memory.
Mar 30 22:43:32 freenas Freed UMA keg (tcpcb) was not empty (27 items).  Lost 9 pages of memory.
Mar 30 22:43:32 freenas Freed UMA keg (ripcb) was not empty (30 items).  Lost 3 pages of memory.
Mar 30 22:43:32 freenas hhook_vnet_uninit: hhook_head type=1, id=1 cleanup required
Mar 30 22:43:32 freenas hhook_vnet_uninit: hhook_head type=1, id=0 cleanup required
Mar 30 20:43:32 freenas devd: Executing '/etc/pccard_ether epair0a start'
Mar 30 20:43:32 freenas devd: Executing '/etc/pccard_ether epair0b start'
Mar 30 20:43:32 freenas devd: Executing '/etc/rc.d/dhclient quietstart epair0a'
Mar 30 22:43:32 freenas epair0a: Ethernet address: 02:ff:60:00:07:0a
Mar 30 22:43:32 freenas epair0b: Ethernet address: 02:ff:b0:00:0a:0b
Mar 30 22:43:32 freenas kernel: epair0a: link state changed to UP
Mar 30 22:43:32 freenas kernel: epair0a: link state changed to UP
Mar 30 22:43:32 freenas kernel: epair0b: link state changed to UP
Mar 30 22:43:32 freenas kernel: epair0b: link state changed to UP
Mar 30 22:43:32 freenas kernel: epair0a: promiscuous mode enabled
Mar 30 22:43:32 freenas kernel: ng_ether_ifnet_arrival_event: can't re-name node epair0b
Mar 30 22:43:32 freenas kernel: ng_ether_ifnet_arrival_event: can't re-name node epair0b
Mar 30 22:43:43 freenas manage.py: [freeadmin.navtree:621] An error occurred while unserializing from http://192.168.1.206/plugins/crashplan/1/_s/treemenu: No JSON object could be decoded

However I can enter the jail and run "dhclient epair0b", which successfully assigns the correct IP to the jail.

Code:

DHCPREQUEST on epair0b to 255.255.255.255 port 67
DHCPACK from 192.168.1.100
bound to 192.168.1.208 -- renewal in 21600 seconds.

I'd appreciate any troubleshooting suggestions, because right now my backup isn't working, which is less than ideal...

Thanks for reading!

Tom

 

[wtfR6a]

try hard coding the jails IP address.

 

[SweetAndLow]

Set the default gateway, that should fix your problem.

 

[Tom_]

Thanks for the replies.

After disabling DHCP, setting the IP and gateway, the jail keeps the correct IP when it restarts, but the crashplan service still won't start. I get the same error message running "service crashplan onestart" as in the OP.

Any other suggestions?

Tom

 

[wtfR6a]

thats worth looking into the recent threads, this one specifically https://forums.freenas.org/index.php?threads/tested-crashplan-4-5-setup.41418/page-2#post-274872
Basically - its not trivial to troubleshoot individual problems and you should take those issues to the existing threads. Good luck!

 

[SweetAndLow]

Can you ping the gateway? Can you ping Google?

 

[Tom_]

Thanks for the suggestion. I was able to ping my router but not google.com from within the crashplan jail. The jail's resolv.conf was blank, so I added the nameserver details to that. Now I can ping google.com, but crashplan still won't start (same error as in OP).

I don't know why I have to configure the DNS manually for this jail after upgrading. I suppose it's worth mentioning that I didn't have any of these networking problems before upgrading to FreeNAS 9.10—everything was working just fine with DHCP.

I've also noticed that when I restart this jail, the checkbox for IPv4 DHCP becomes checked even if it was unchecked and networking was configured manually before the restart. Something is definitely not right here.

Any other ideas, short of trashing all my jails and starting again, would be appreciated. If that's what I have to do I will do it, but it'll take a lot of time to restore the configuration across 4 or 5 plugins. Better than not having any backups, I guess.

 

[Tom_]

crashplan starts and runs backups correctly after I added the following line to /usr/pbi/crashplan-amd64/share/crashplan/bin/CrashPlanEngine:

Code:

export LD_LIBRARY_PATH=/usr/pbi/crashplan-amd64/share/crashplan/jre/lib/i386/jli

No idea if this is a robust fix or not.

The networking issue seems to be separate. After a fresh reboot of FreeNAS, all of my jails get their correct IPs etc. from DHCP, but if I restart any jail it gives the JSON error described in the first post and doesn't get an IP. Should I report this as a bug?

 

[icsy7867]

I am actually having both of your issues!

Autostart'ed Jails seem to pull IP's just fine. But if I manually restart any Jail it fails to get an IP from DHCP.

The quick and dirty solution I found is:

Code:

jls
sudo jexec (jail number) tcsh
ifconfig (Take note of the proper network adapter)
dhclient epairxx

After this, the jail grabs it's correct IP address as per normal.

As far as the Crashplan issue, I can't get that one resolved currently. While trying to start the crashplan service from the webgui I get:

Code:

Some Error Occured

But If I access the Jail and run
service crashplan onestart I get:

Code:

service crashplan onestart
Starting CrashPlan Engine ... Using standard startup
OK

But my client application wont connect to the server like it did before. I am still getting the "Failed to connect to the backup engine, Retry?" Does it matter where you put:

Code:

export LD_LIBRARY_PATH=/usr/pbi/crashplan-amd64/share/crashplan/jre/lib/i386/jli

?

 

[Tom_]

I guess you want that near the start of the script. It's the first line after the shebang in mine.

Have you checked the log files in /var/log/crashplan/? That's how I spotted the Java issue in my jail.

See this thread for more about the dhcp problem: https://forums.freenas.org/index.php?threads/42310/

 

[ICANSEEYOU7687]

Thanks for your replies and links to the info!

I am having a similar issue but it seems that my installation is still communicating with crashplan. I can log into crashplan's website here:
https://www.crashplan.com/account/

And when I do, I can see that the last communication date was a few minutes ago. I am noticing a few strange things though...
If I try to enable the crashplan plugin from the installed "plugins" menu I see

Code:

Some error occurred

But if I jexec into the jail I can do a :

Code:

service crashplan onestart

without any issues...

My engine_error.log says:

/etc/host.conf: line 2: bad command `hosts'
/etc/host.conf: line 3: bad command `dns'

and my engine_output.log has a Java warning:

Code:

Java HotSpot(TM) Client VM warning: Can't detect initial thread stack location - find_vma failed

I also decided to go ahead and update everything to 4.6.0 and download the 4.6.0 client to see if that would make a difference, but I am seeing the same effect.