Memory allocation for crashplan plugin

[berd1921]

I'm attempting a very large backup to crashplan from freenas.

I've been running out of memory for deduplication, and the backup would get stuck analyzing but never send the files. I was able to get my backup running again by changing the memory allowed for the crashplan by editing the file:

usr/pbi/crashplan-amd64/share/crashplan/bin/run.conf

You can also make the memory allocation change through the crashplan UI

However, I can't allocate more than 3072 mb without crashing the plugin. If I set 4096mb for example I can't restart the plugin on freenas. To send the entire 10TB i'm trying to backup to crashplan I probably need to allocate 8-10GB of ram to crashplan. I have 16gb installed.

Is there a way to limit ram usage by ZFS/Freenas system so that I can allocate more to crashplan, or do I need to go up to 32GB of ram?

 

[Ericloewe]

You have two options:

• Stop using deduplication
• Add more RAM and be prepared to keep adding RAM with no upper bound

You shouldn't even think of deduplication on 16GB of RAM, unless you have a tiny amount of data that is copied thousands of times!

 

[SweetAndLow]

I think they are referring to crashplan dedup which uses memory in the jail. One suggestion is to turn that off in the crashplan setting. The other is get More memory.

 

[andoy31]

I have 32gb menory and i still cant get it to go above 3gb.
I dint have the luxury of time to figure it out, hence i just went with "whatever works" :)

 

[cyberjock]

The problem is the linux jail that crashplan runs in is 32-bit. So no, the limit can't be fixed except to stop using dedup. :(

 

[andoy31]

thanks cyberjock for confirming my suspicion :)

managed to update the JAVA to openjdk7 x64-- and can now increase the memory to more than 3GB.

- it somehow invalidated my server "crashplan_1" and will have to adopt from the previous backup
- it's currently synchronizing block information

requirements:
- jail datastore has ~60MB free space for openjdk installation
- internet connection

the following are the steps:
- shutdown your crashplan plugin service thru the freenas plugin page
- navigate to your jail - thru jls then jexec comands
- run the command - pkg_add -r openjdk7
- backup your current 32bit java (mv /usr/pbi/crashplan-amd64/linux-sun-jre1.7.0 /usr/pbi/crashplan-amd64/jre7i32 )
- create softlink to your new java x64 ( ln -s /usr/local/openjdk7 /usr/pbi/crashplan-amd64/linux-sun-jre1.7.0)
- modify the file /usr/pbi/crashplan-amd64/share/crashplan/bin/run.conf and change the memory from 3GB to whatever you like (you can use your friendly vi)
- start the crashplan service thru the freenas plugin page.
- connect to crashplan and adopt from the previous backup.


I'm just testing this out- so far it's currently synchronizing block information.
will update for any findings/

 

[andoy31]

just for some update and results:
- crashplan was able to start using the new 64bit java however it has some problems seeing the disk shares
- somehow the changes are non-persistent


will try to troubleshoot further if i have the time again tomorrow but most probably it'll be on the holiday break.

 

[berd1921]

just for some update and results:
- crashplan was able to start using the new 64bit java however it has some problems seeing the disk shares
- somehow the changes are non-persistent


will try to troubleshoot further if i have the time again tomorrow but most probably it'll be on the holiday break.

Were you able to get it working with 64bit java? This seems like it could be a big fix.

 

[andoy31]

i do apologise, it hasn't move a thing and i doubt it'll be moving any time soon as I'm relocating next month.

 

[OttRob]

Has anyone managed to figure this one out? Being limited to 3GB kind of sucks.

 

[Chris123]

Did anyone have any success with this?

I'm having the same issue. Just moved my 16TB archive from a Windows pc to FreeNAS. The 16TB is already on Crashplan Central, but due to the 3Gb memory allocation limit on the Crashplan plugin/jail I'm not even able to adopt the archive from my previous location.

My FreeNAS server have 32Gb ecc ram and a Xeon E3-1230v3 cpu. I'm not using de-duplication on any FreeNAS volume/dataset, and I've turned off the de-duplication for the data transfer to Crashplan using this setting in the my.service.xml file: <dataDeDupAutoMaxFileSizeForWan>1</dataDeDupAutoMaxFileSizeForWan>

I did try the other solution that have been suggested by some here; to create a VirtualBox VM running Linux and Crashplan. It seems to work good, but if possible I would rather have liked to have Crashplan in a jail.

 

[Chris123]

Ok, did some googling on whether a 64-bit linux emulator will be available or not (currently only 32-bit linux emulator as per Cyberjocks comment above). And it seems like it is just around the corner for FreeBSD, most likely be available in the 10.3-RELEASE at the end of this month according to this FreeBSD forum thread: https://forums.freebsd.org/threads/54713/

What would that mean in terms of FreeNAS release? How long does FreeNAS releases usually lag behind the FreeBSD releases?

Or maybe Linux emulation in jails will become redundant with the new 'bhyve' in version 10?

 

[Robert Trevellyan]

What would that mean in terms of FreeNAS release?

https://forums.freenas.org/index.php?threads/9-10-is-now-in-nightlies-testing.41806/

 

[Chris123]

Thanks. Make sense.

 

[Chris123]

Reading the Release Announcement for the FreeBSD 10.3-RELEASE, it now seems like, if I understand it correctly, that Linux 64-bit emulation has been implemented:

The Linux(R) compatibility layer has been substantially improved and
now is capable of running 64-bit applications on amd64 (x86_64), 1:1
threading, VDSO and subset of the epoll(7) family sufficient for the
majority of programs.

 

[MauricioU]

just for some update and results:
- crashplan was able to start using the new 64bit java however it has some problems seeing the disk shares
- somehow the changes are non-persistent


will try to troubleshoot further if i have the time again tomorrow but most probably it'll be on the holiday break.

Do you have an update on this? Can you get crashplan installed in a jail as a 64 bit environment? Really trying to figure this out as I am trying to back up 8.8 TB and my speeds are atrocious due to the 3 GB Ram limit of 32 bit.

 

[SweetAndLow]

Do you have an update on this? Can you get crashplan installed in a jail as a 64 bit environment? Really trying to figure this out as I am trying to back up 8.8 TB and my speeds are atrocious due to the 3 GB Ram limit of 32 bit.

Crash plan is dropping support for home users. Are you sure you want to choose them for your backup right now?

Sent from my XT1096 using Tapatalk

 

[MauricioU]

Crash plan is dropping support for home users. Are you sure you want to choose them for your backup right now?

Sent from my XT1096 using Tapatalk

Yeah, I'm using Crashplan Pro. I'm a bit weary but I don't know what other option I could go with that isn't overly expensive to back up 8.8 TB and growing. With that being said, do you have any alternative solutions? Or an answer for my question regarding a 64 bit environment? Would love the help!

 

[danb35]

With that being said, do you have any alternative solutions?

A Google Apps account is $10/mo and gives unlimited storage. rclone works well to copy data, and encrypts if desired.

 

[MauricioU]

A Google Apps account is $10/mo and gives unlimited storage. rclone works well to copy data, and encrypts if desired.

Is this true? Last I checked you get 30 GB for $5/user and then you can get unlimited for an extra $5. But this "unlimited" is only unlimited if you have at least 5 users. Otherwise its 1 TB. Can you confirm?

https://gsuite.google.com/pricing.html?tab_activeEl=tabset-companies

^if you look under business "unlimited cloud storage (or 1TB per user if fewer than 5 users)"