FN11 Jailed Crashplan PRO with VNC (iocage) 2017-12-26

[kjake]

kjake submitted a new resource:

Jailed Crashplan PRO with VNC (iocage) - Crashplan Pro in a iocage jail for FreeNAS - with VNC!

As iocage is becoming the preferred Jail manager in FN11.2, I migated all of my Jails to it. There is no working Plugin for FN/iocage, so I concocted my own, but not using the iocage plugin system.

I liked the Crashplan Docker container that I used to use because it mounted all of the Crashplan config paths for persistent storage and included a VNC server with a noVNC webserver for viewing the desktop. I've done my best to replicate this in a Jail using iocage.

Instructions on how to...

Read more about this resource...

 

[kjake]

Currently installs Crashplan PRO 4.9.0 Build 33 as recommended.

From Code42 Support:

We are pushing out our 6.6 client for CrashPlan PRO soon but need to take care of a few small bugs before we push it out to everyone so those support docs will be applicable soon but at this point you should still reference the version 4 docs and use the installer available from your App Downloads tab in your account.

 

[scrappy]

Thanks for this. I plan on switching over to Crashplan Pro soon. While I have had more luck running Crashplan in a Linux VM, I might give jails another go.

 

[kjake]

Wanted to add that my setup has been running now normally and I've been updating the Github README as needed with tips and clarifications. I have needed to bump my Max Java memory to 2048. Using the default 1024 wouldn't crash anything, but the backup would just freeze and the java process usage would spike. This is a well documented issue with lots of files to backup though and nothing special to the Jail setup.

 

[turtlepants]

Setup Info:

• FreeNAS 11.1
• 32GB Ram
• 12TB dataset (approx 8TB used).

I finally took the plunge and migrated from Corral (Docker container of Crashplan) to FreeNAS 11.1 (Trying IOCAGE). I was able to get a the jail up and running using this method but had to make a few changes for this to work. I had to tweak the configuration since I have a lot of data to backup (~8TB) and using more than 4GB for the java heap size would cause crashplan to fail to start. I need at least 8GB according to Crashplan so, I did the following;

I changed the bin/CrashPlanPro_FreeBSD_install.csh...

Code:

# Use 64-bit java instead...
#set JAVA_PKG=`cat $CRASHPLAN_LOC/crashplan-install/install.defaults | grep i586 | cut -d'=' -f2 | cut -d/ -f7`
#fetch --no-verify-peer `cat $CRASHPLAN_LOC/crashplan-install/install.defaults | grep i586 | cut -d'=' -f2` -o $CRASHPLAN_LOC/$JAVA_PKG
set JAVA_PKG=`cat $CRASHPLAN_LOC/crashplan-install/install.defaults | grep X64 | cut -d'=' -f2 | cut -d/ -f7`
fetch --no-verify-peer `cat $CRASHPLAN_LOC/crashplan-install/install.defaults | grep X64 | cut -d'=' -f2` -o $CRASHPLAN_LOC/$JAVA_PKG

# Stick with Crashplan Home for now...
#sed -i .backup 's/<orgType>CONSUMER<\/orgType>/<orgType>BUSINESS<\/orgType>/g; s/central.crashplan.com/central.crashplanpro.com/g' $CRASHPLAN_LOC/conf/default.service.xml $CRASHPLAN_LOC/conf/my.service.xml
sed -i .backup 's/<orgType>BUSINESS<\/orgType>/<orgType>CONSUMER<\/orgType>/g; s/central.crashplan.com/central.crashplanpro.com/g' $CRASHPLAN_LOC/conf/default.service.xml $CRASHPLAN_LOC/conf/my.service.xml

and bin/patch-scripts_CrashPlanEngine...

Code:

# Use 64-bit java instead...
#- export LD_LIBRARY_PATH="/usr/local/share/crashplan/linux-sun-jre1.8.0/jre/lib/i386/jli"
- export LD_LIBRARY_PATH="/usr/local/share/crashplan/linux-sun-jre1.8.0/jre/lib/amd64/jli"

I also changed bin/run.conf after the installation for the large java heap size to -Xmx12G.

This all seems to work. I can login my home account, adopt the old pc, and start syncing everything. It starts to fall apart during the "Synchronizing block information". It goes for a while, then the Crashplan engine stops. I can keep on restarting it until it finally finishes. Then, after the block syncing is done, the engine keeps on stopping when trying to start the backup.

Below is the log after the engine stops;

/usr/local/share/crashplan/engine_output.log...

[01.15.18 07:30:01.700 INFO main com.backup42.service.CPService] Locale set to English
[01.15.18 07:30:01.701 INFO main com.backup42.service.CPService] *************************************************************
[01.15.18 07:30:01.701 INFO main com.backup42.service.CPService] *************************************************************
[01.15.18 07:30:01.701 INFO main com.backup42.service.CPService] STARTED CrashPlanService
[01.15.18 07:30:01.702 INFO main com.backup42.service.CPService] CPVERSION = 4.9.0 - 1436674888490 (2015-07-12T04:21:28:490+0000) - Build: 33
[01.15.18 07:30:01.702 INFO main com.backup42.service.CPService] LOCALE = English (United States)
[01.15.18 07:30:01.702 INFO main com.backup42.service.CPService] ARGS = []
[01.15.18 07:30:01.702 INFO main com.backup42.service.CPService] *************************************************************
[01.15.18 07:30:02.225 INFO main com.backup42.service.CPService] Adding shutdown hook.
[01.15.18 07:30:02.257 INFO main com.backup42.service.CPService] loadConfig, BEGIN Loading Configuration
[01.15.18 07:30:02.298 INFO main com.backup42.service.CpsFolders] CpsFolders chmod successful. confPath=conf/
[01.15.18 07:30:02.298 INFO main com.backup42.service.CPService] BEGIN Copy Custom
[01.15.18 07:30:02.298 INFO main com.backup42.service.CPService] Directories: [.Custom, custom, /repository/.Custom, /repository/custom]
[01.15.18 07:30:02.298 INFO main com.backup42.service.CPService] NOT waiting for custom skin to appear
[01.15.18 07:30:02.298 INFO main com.backup42.service.CPService] NO customizations found.
[01.15.18 07:30:02.300 INFO main com.backup42.service.CPService] END Copy Custom
[01.15.18 07:30:02.301 INFO main backup42.common.config.AppConfig] Loading from default: , /usr/local/share/crashplan/conf/default.service.xml
[01.15.18 07:30:03.020 INFO main backup42.common.config.AppConfig] Loading from my xml file:, conf/my.service.xml
[01.15.18 07:30:03.122 WARN main up42.common.config.ServiceConfig] ServiceConfig:: ServiceConfig version higher than it should be. uniqueId=830928200280013934
[01.15.18 07:30:03.145 INFO main up42.common.config.ServiceConfig] ServiceConfig:: Loading ServiceConfig, newInstall=false, configVersion=11, configDateMs=1515981473266, installVersion=1436674888490, currentVersion=1436674888490
[01.15.18 07:30:03.145 INFO main up42.common.config.ServiceConfig] ServiceConfig:: OS = Linux
[01.15.18 07:30:03.270 INFO main com.backup42.service.CPService] AuthorityLocation@-1379701855[ location=central.crashplanpro.com:443, hideAddress=false, isLocked=false ]
[01.15.18 07:30:03.271 INFO main com.backup42.service.CPService] Checking Java memory heap max.
[01.15.18 07:30:03.273 INFO main com.backup42.service.CPService] END Loading Configuration
jtux64 Loaded.
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00000008085e2347, pid=10483, tid=37002233600
#
# JRE version: Java(TM) SE Runtime Environment (8.0_72-b15) (build 1.8.0_72-b15)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.72-b15 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# J 6927 C2 sun.nio.ch.AbstractPollSelectorImpl.updateSelectedKeys()I (132 bytes) @ 0x00000008085e2347 [0x00000008085e2300+0x47]
#
# Core dump written. Default location: /usr/local/share/crashplan/core or core.10483 (max size 9007199254740991 kB). To ensure a full core dump, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /usr/local/share/crashplan/hs_err_pid10483.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#

The java error sun.nio.ch.AbstractPollSelectorImpl.updateSelectedKeys that occurs might have something to do with the bin/crashplan file. I did notice that this is different from the Crashplan version and I am not sure if that is because it is a jail config instead of a standard linux config?


Original Crashplan version...

Code:

#!/bin/sh
# Linux Standard Base comments
### BEGIN INIT INFO
# Provides:          CrashPlanEngine
# Required-Start:    $local_fs $network $remote_fs
# Required-Stop:     $local_fs $network $remote_fs
# Should-Start:
# Should-Stop:
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: CrashPlan Engine
# Description:       CrashPlan backup service engine
### END INIT INFO

#############################################################
# Init script for CrashPlanEngine
#############################################################

# Defaults
SCRIPTNAME=<INITSCRIPT>

case "$1" in
start)
    $SCRIPTNAME start   
    ;;
stop)
    $SCRIPTNAME stop
    ;;
restart)
    $SCRIPTNAME restart
    ;;
force-reload)
    $SCRIPTNAME force-reload
    ;;
status)
    $SCRIPTNAME status
    ;;
*)  
    echo "Usage: $0 <start|stop|restart|force-reload|status>" >&2
    exit 3
    ;;
esac
exit 0

New version from script...

Code:

#!/bin/sh
#
# $FreeBSD: head/sysutils/linux-crashplan/files/crashplan.in 371372 2014-10-23 00:36:47Z wg $

# PROVIDE: crashplan
# REQUIRE: DAEMON

. /etc/rc.subr

name="crashplan"
rcvar=${name}_enable

crashplan_bin="/usr/local/share/crashplan/bin"
command="${crashplan_bin}/CrashPlanEngine"
pidfile="/var/run/CrashPlanEngine.pid"

crashplan_start() {
    local crashplan_poll_selector
    crashplan_poll_selector="-Djava.nio.channels.spi.SelectorProvider=sun.nio.ch.PollSelectorProvider"

    /usr/bin/install -d /var/log/crashplan || exit 1

    sed -E -e "s/-Djava.nio.channels.spi.SelectorProvider=[^[:blank:]]+//"      \
           -e "s/SRV_JAVA_OPTS=\"/SRV_JAVA_OPTS=\"${crashplan_poll_selector} /" \
           -e "s/GUI_JAVA_OPTS=\"/GUI_JAVA_OPTS=\"${crashplan_poll_selector} /" \
        -i bak "${crashplan_bin}/run.conf" || exit 1

    /usr/bin/cpuset -l 0 ${command} start
}

start_cmd="${name}_start"
stop_cmd="${command} stop"
status_cmd="${command} status"

load_rc_config ${name}
run_rc_command "$1"

Please help! Thanks!

 

[turtlepants]

Ok, I thought this might be a hardware problem. I've ran MemTest on the RAM and it passed. One of my HDDs is showing Current_Pending_Sectors = 26. However it has passed a long smart test & I ran a scrub just in case. So I am not too sure that this is an issue. Crashplan is up and running again, but it started "Synchronizing block information" again. We shall see if I get Java Fatal errors...

EDIT:

The engine is still stopping after everytime it tries to sync but now it is different but consistent...

[01.18.18 15:37:02.841 INFO main com.backup42.service.CPService] Locale set to English
[01.18.18 15:37:02.842 INFO main com.backup42.service.CPService] *************************************************************
[01.18.18 15:37:02.842 INFO main com.backup42.service.CPService] *************************************************************
[01.18.18 15:37:02.842 INFO main com.backup42.service.CPService] STARTED CrashPlanService
[01.18.18 15:37:02.843 INFO main com.backup42.service.CPService] CPVERSION = 4.9.0 - 1436674888490 (2015-07-12T04:21:28:490+0000) - Build: 33
[01.18.18 15:37:02.843 INFO main com.backup42.service.CPService] LOCALE = English (United States)
[01.18.18 15:37:02.843 INFO main com.backup42.service.CPService] ARGS = []
[01.18.18 15:37:02.843 INFO main com.backup42.service.CPService] *************************************************************
[01.18.18 15:37:03.300 INFO main com.backup42.service.CPService] Adding shutdown hook.
[01.18.18 15:37:03.381 INFO main com.backup42.service.CPService] loadConfig, BEGIN Loading Configuration
[01.18.18 15:37:03.405 INFO main com.backup42.service.CpsFolders] CpsFolders chmod successful. confPath=conf/
[01.18.18 15:37:03.405 INFO main com.backup42.service.CPService] BEGIN Copy Custom
[01.18.18 15:37:03.405 INFO main com.backup42.service.CPService] Directories: [.Custom, custom, /repository/.Custom, /repository/custom]
[01.18.18 15:37:03.405 INFO main com.backup42.service.CPService] NOT waiting for custom skin to appear
[01.18.18 15:37:03.405 INFO main com.backup42.service.CPService] NO customizations found.
[01.18.18 15:37:03.407 INFO main com.backup42.service.CPService] END Copy Custom
[01.18.18 15:37:03.408 INFO main backup42.common.config.AppConfig] Loading from default: , /usr/local/share/crashplan/conf/default.service.xml
[01.18.18 15:37:04.524 INFO main backup42.common.config.AppConfig] Loading from my xml file:, conf/my.service.xml
[01.18.18 15:37:04.643 WARN main up42.common.config.ServiceConfig] ServiceConfig:: ServiceConfig version higher than it should be. uniqueId=831412090643106950
[01.18.18 15:37:04.664 INFO main up42.common.config.ServiceConfig] ServiceConfig:: Loading ServiceConfig, newInstall=false, configVersion=11, configDateMs=1516137207703, installVersion=1436674888490, currentVersion=1436674888490
[01.18.18 15:37:04.665 INFO main up42.common.config.ServiceConfig] ServiceConfig:: OS = Linux
[01.18.18 15:37:04.892 INFO main com.backup42.service.CPService] AuthorityLocation@-1379701855[ location=central.crashplanpro.com:443, hideAddress=false, isLocked=false ]
[01.18.18 15:37:04.893 INFO main com.backup42.service.CPService] Checking Java memory heap max.
[01.18.18 15:37:04.897 INFO main com.backup42.service.CPService] END Loading Configuration
jtux64 Loaded.
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x0000000808d070d5, pid=40646, tid=35700950784
#
# JRE version: Java(TM) SE Runtime Environment (8.0_72-b15) (build 1.8.0_72-b15)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.72-b15 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# J 4272 C1 java.util.concurrent.ConcurrentHashMap.replaceNode(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (416 bytes) @ 0x0000000808d070d5 [0x0000000808d07060+0x75]
#
# Core dump written. Default location: /usr/local/share/crashplan/core or core.40646 (max size 9007199254740991 kB). To ensure a full core dump, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /usr/local/share/crashplan/hs_err_pid40646.log
Compiled method (c1) 28821494 4272 ! 3 java.util.concurrent.ConcurrentHashMap::replaceNode (416 bytes)
total in heap [0x0000000808d06d90,0x0000000808d096d0] = 10560
relocation [0x0000000808d06eb8,0x0000000808d07050] = 408
main code [0x0000000808d07060,0x0000000808d089e0] = 6528
stub code [0x0000000808d089e0,0x0000000808d08b08] = 296
metadata [0x0000000808d08b08,0x0000000808d08b30] = 40
scopes data [0x0000000808d08b30,0x0000000808d092d8] = 1960
scopes pcs [0x0000000808d092d8,0x0000000808d09508] = 560
dependencies [0x0000000808d09508,0x0000000808d09510] = 8
handler table [0x0000000808d09510,0x0000000808d09678] = 360
nul chk table [0x0000000808d09678,0x0000000808d096d0] = 88
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#

I can manually restart and it seems to be fine again but I cannot be checking this everyday wondering if it is working or not. I guess I give up, I am going to try either running Ubuntu VM or Rancher with NFS....

 

[danb35]

I altered the script slightly to use variables, as you did with the Unifi script, but have no idea how to submit that change on github, so the updated version is attached.

 

[kjake]

@danb35 - thanks for that. I actually had a version with variables going already, but forgot to commit to Github. I was waiting for Code42 to publish the next version of Crashplan.

Well, they have - 6.7 is the latest version now and it's very different. 64-bit Java is now required, but it actually works best to use BSD-native Java now. Also, the Desktop client is an Electron app. I've gotten both to run beyond basic errors and missing libraries, but neither wants to finish running due to further internal errors. I'll keep working at it, but I'm afraid that we will need to finally have to setup a Linux VM to do this proper.

@turtlepants - the Java is really finicky for that version of CrashPlan. I could only get it to work reliably with the 32-bit Java as I published it.

 

[kjake]

So yeah, I've decided that I'm abandoning CrashPlan. Setup Backblaze B2 in minutes and it is baked into FN11 now. I don't see value in spending hours troubleshooting Crashplan each time it updates. ¯\_(ツ)_/¯

 

[rclarkin]

Well, that is just sad. I have been using CrashPlan Home for quite a while and now that they are discontinuing that product line I had hoped to switch to CrashPlan Pro. I also do not see value in spending hours and hours troubleshooting so I think that I'm also off to a different solution.

Thank you for all of the hard work and detailed notes! They definitely have helped me feel good about switching backup solutions.

 

[kjake]

On B2, don't forget to set caps on your transactions - and they're daily caps. If you use the default FN setup, and have a lot of files, you're gonna burn through those free Class C transactions quickly. I helped chill my setup out by having multiple tasks for different areas that I care about to be backed up, and then only run the task every 12 hours right now - did rate limiting at my firewall since that is missing from the FN setup AFAIK.