SpiderOak installation into a Jail (FreeNAS 9.2)

Status
Not open for further replies.

thewizard

Dabbler
Joined
Apr 1, 2014
Messages
30
SpiderOak on FreeNAS (alternative to Dropbox)

I’ve been a fan of SpiderOak for years; it’s a great alternative to DropBox for me. They recently had some sales on storage so I bit the bullet and dived into paid storage, up until now I’d been on the free plan with just a few GB. I'm not affiliated with SpiderOak in anyway, the attached PDF has images, this is a straight copy/paste from the Word file I originally wrote for this post. I am coming at this from a fairly new to FreeBSD - Windows user. Devilspawn basically.

My first challenge was getting it to run on FreeNAS. It ended up not being that hard, so I thought I’d post a short “How To” here for other people to follow.

I’m running FreeNAS 9.2, first up you need to be familiar with the concept a “Jail” and the difference between running a Shell command on the FreeNAS box itself compared to running a Shell command on the Jail itself.

Shell on the FreeNAS box, note the command line says “root@freenas”:

Shell launched from the Jail, note it says “root@Jail”:

Basically we are going to install the FreeBSD Port of SpiderOak onto our FreeNAS Jail. I used this guide here to help me understand FreeBSD ports, http://doc.freenas.org/index.php/Installing_non-PBI_Software#Compiling_FreeBSD_Ports_with_make

Ok step by step,
1. Create a Jail from the GUI, of type “portjail”.
2. Add some storage to your Jail so it can read the files you want to send to SpiderOak, I’d recommend Read Only:

3. From the FreeNAS Shell run the command “kldload linux”, this allows Linux emulation to be installed into the jail.

4. Open up the link below in any browser and have a read, someone else has kindly done the work of porting SpiderOak to FreeBSD (which FreeNAS is based on), https://www.freshports.org/net/spideroak/

5. From the Jail Shell run the commands
a. portsnap fetch extract
b. portsnap fetch update
c. This will take a while!

6. Now we need to download the source code for SpiderOak, this requires a little bit of manual trickery to get working:
a. https://spideroak.com/getbuild?platform=slackware&arch=i386&dummy=/spideroak-5.1.1-i686-1.tar.gz
b. Clicking that link will actually redirect you to version 5.1.3, which is both smart and stupid for reasons we’ll get to.
c. You need to put that file into /usr/ports/distfiles in the Jail and rename it (I did this just via a SAMBA share on my windows box). It needs to be called “spideroak-5.1.3-i686-1.tar.gz”. Note the filename very precisely.

7. Now because the filename is different the make file will fail!
a. Under /usr/ports/net/spideroak there is a “Makefile”, open this in your favourite text editor, I choose notepad J
b. Change the PORT VERSION from 5.1.1 to 5.1.3

8. Almost there, these files use checksums to prevent tampering. Luckily for us no dependencies changed from 5.1.1 to 5.1.3 so all we have to do is run the command:
a. “make makesum” from the spideroak port directory.

9. The above two steps will not be necessary when some nice FreeBSD developer finds the time to update the Fresh Ports website with a newer batch of SpiderOak source code.
10
Now you can run the commands:
a. cd /usr/ports/net/spideroak/ && make install clean
11. This will install SpiderOak and all of its prerequisites.
Now a quick tutorial on actually running SpiderOak from the Jail Shell,
· This is the first command to run, “SpiderOak --setup=-“
o This will prompt you for your login, and ask you to provide a new machine name.
· Next you want to go, “SpiderOak --include-dir=/mnt/mystuff”, this says hey I want that folder and all of its subfolders backed up.
· Finally “SpiderOak --headless” this starts SpiderOak and tells it to do what it does and start processing files and uploading them!
 

Attachments

  • SpiderOak on FreeNAS.pdf
    1.5 MB · Views: 1,190

fullmetaljester

Dabbler
Joined
Dec 24, 2013
Messages
15
I just posted the other day looking for this exact setup, THANKS!

Also we may see an official plugin this year according to SpiderOAK.
 

fullmetaljester

Dabbler
Joined
Dec 24, 2013
Messages
15
I was able to set this up without issue, but I'm noticing the following error in my logs, after which the spideroak headless instance shuts down. It was able to create the 1st directory i added on the SpiderOak servers, but its not uploading anything. Thoughts?

Apr 2 15:49:24 freenas kernel: linux: pid 94046 (inotify_dir_watcher): syscall inotify_init not implemented
Apr 2 15:49:26 freenas kernel: linux: pid 94048 (inotify_dir_watcher): syscall inotify_init not implemented
Apr 2 15:53:37 freenas kernel: linux: pid 94287 (inotify_dir_watcher): syscall inotify_init not implemented
Apr 2 15:53:39 freenas kernel: linux: pid 94289 (inotify_dir_watcher): syscall inotify_init not implemented
 

fullmetaljester

Dabbler
Joined
Dec 24, 2013
Messages
15
I was able to set this up without issue, but I'm noticing the following error in my logs, after which the spideroak headless instance shuts down. It was able to create the 1st directory i added on the SpiderOak servers, but its not uploading anything. Thoughts?

Apr 2 15:49:24 freenas kernel: linux: pid 94046 (inotify_dir_watcher): syscall inotify_init not implemented
Apr 2 15:49:26 freenas kernel: linux: pid 94048 (inotify_dir_watcher): syscall inotify_init not implemented
Apr 2 15:53:37 freenas kernel: linux: pid 94287 (inotify_dir_watcher): syscall inotify_init not implemented
Apr 2 15:53:39 freenas kernel: linux: pid 94289 (inotify_dir_watcher): syscall inotify_init not implemented


Turns out you can ignore this msg, as BSD uses Kqueue and not inotify.

The shutdown issue was because I was forgetting to tell spideroak to run in the background and closing the shell window was killing the process.

Thanks again!
 

thewizard

Dabbler
Joined
Apr 1, 2014
Messages
30
Hey mate, I've found a similar issue the GUI interface times out after awhile and even if you leave the Jail Terminal window open it eventually stops the process and SpiderOak ceases to upload. I fixed it by adding a Cron job to my FreeNAS. My "Jail ID" is "2", you can get it apparently by running the cmd jls.

The Cron job itself is simple:
jexec 2 spideroak --headless

I set the job to run once a day just before midnight. Perfect for my needs.
 

fullmetaljester

Dabbler
Joined
Dec 24, 2013
Messages
15
I pushed it into the background this way: "spideroak ---headless &" from the jail shell.

been running nonstop for days ;)
 

fullmetaljester

Dabbler
Joined
Dec 24, 2013
Messages
15
I can confirm this process still works with the 5.1.8 port and the 5.2.0 install file. Just rename it to 5.1.8.

Also you will need to run the following before starting the Spideroak install.

pkg install linux_base-c6-6.6_4

sysctl compat.linux.osrelease=2.6.18 (set this one in your Jail config to make permanent)
 

maikthulhu

Cadet
Joined
Jul 14, 2015
Messages
2
I'm using FreeNAS 9.3 and was having trouble getting this going (current version in ports tree as of this writing is 5.1.8). I tried following the instructions here but kept having trouble, so here is my attempt to fill in the gaps by providing step-by-step what worked for me. Hope this helps.
  1. Create a standard jail
  2. Add storage (I concur that anything you attach should be in Read-Only mode)
  3. On your FreeNAS host, get a shell and type: kldload linux
    • Make this load at boot by modifying /boot/loader.conf and adding: linux_load="YES"
      • I haven't tested the above because I don't want to reboot, but I'm pretty sure that's how you do it.
  4. (The rest of these commands are performed in the jail) Update the ports tree: portsnap fetch extract && portsnap fetch update
  5. Create the distfiles folder: mkdir /usr/ports/distfiles
  6. Go to the distfiles folder: cd /usr/ports/distfiles
  7. Get SpiderOak 5.2.0: fetch "https://spideroak.com/getbuild?platform=slackware&arch=i386&dummy=/SpiderOak-5.2.0-i686-1.tar.gz"
  8. Rename the package to reflect a 5.1.8 version number and expected extension: mv SpiderOak-5.2.0-i686-1.tar.gz SpiderOak-5.1.8-i686-1.tgz
  9. Go to the SpiderOak ports folder: cd /usr/ports/net/spideroak
  10. Generate new checksums so the Makefile can build using the 5.2.0 package: make makesum
  11. Attempt the install: make install
  12. The install fails at this point stating it can't find PIL._imaging.so. I'm assuming some files were renamed because I found _imaging.so which seems to fit the bill, so: cd work/stage/usr/local/share/spideroak/ && cp _imaging.so PIL._imaging.so
  13. Go back to the port's folder: cd -
  14. Attempt the install again: make install clean
  15. Attempting to run SpiderOak now results in complaints about missing libsodium.so. I wasn't sure where to look but since it's a Linux program with Linux libraries, I decided to grab the 32-bit debian package to source the missing library. First, go to tmp: cd /tmp
  16. Next grab the debian package: fetch "http://http.us.debian.org/debian/pool/main/libs/libsodium/libsodium13_1.0.3-1_i386.deb"
  17. Extract the debian package: tar zxvf libsodium13_1.0.3-1_i386.deb
  18. Extract the data archive: tar zxvf data.tar.xz
  19. Go to the folder containing library files: cd usr/lib/i386-linux-gnu/
  20. Copy files to SpiderOak's share folder: cp libsodium.so.13* /usr/local/share/spideroak/
  21. Go to SpiderOak's share folder: cd /usr/local/share/spideroak/
  22. Create a symlink for libsodium.so: ln -s libsodium.so.13 libsodium.so
  23. Clean up: cd /tmp && rm -r libsodium* usr control.tar.gz data.tar.gz debian-binary
Now just follow the instructions above for setting up SpiderOak and you're good to go.
 

dreamerns

Dabbler
Joined
May 1, 2015
Messages
48
I've been using spideroak for more then a year. I recently built a FreeNAS box and started switching all my stuff there. Making SpiderOak run on it is the last piece of the puzzle I haven't solved. I was able to run 5.1.8 in jail (it required linux_base-c6-6.6_4), however I noticed some bad issues. Spideroak installs SpiderOak Hive folder by default. On first run it copied all mu stuff there and everything was fine. As soon as I run sync or headless, it zeroes out all the files. Since that folder is in sync, that means that files on my other computers also get zero in size. Now, the files are not deleted, it's just that their size is changed. In order to get them back you need to go file by file and download historic version and copy it back (even that doesn't always work). This happened to me on next trial. Then I tried to build 5.2.0 but I had some issues and wasn't able to do it (I think I had same issues as maikthulhu). Then I tried running SpiderOak on VirtualBox jail. I needed to put 2Gb of memory, because Spideroak was crashing on less (couldn't find anywhere what is minimum configuration for spideroak). I was able to make it work at the end, I've mounted my dataset through NFS, but then I hit the same problem of files being zeroed out. After that I gave up and couldn't find time to get back to it.
maikthulhu, thanks for the update, I'll try your approach this week. This time I'll mount dataset as read-only. Just one tip be sure not to have that folder in sync, because if you write something to it on another device and spideroak tries to write to it on FreeNAS, I don't think it will be smart enough to figure out there is something wrong with permissions and it will destroy that file.
 

maikthulhu

Cadet
Joined
Jul 14, 2015
Messages
2
That's interesting. I wonder if it had something to do with migrating to a different system/location so SpiderOak is getting confused. All of the stuff I am syncing was never synced before and is managed by the other jails.

I had no use for the Hive on my SpiderOak jail so I used --exclude-dir=/root/Spideroak\ Hive
 

dreamerns

Dabbler
Joined
May 1, 2015
Messages
48
I was able to install 5.2.0. It was exactly as you described. Thanks :)
I still had issue on first run (SpiderOak --setup=-). It created SpiderOak Hive and zeroed out all the files inside :-( First exclusion didn't work out, since once I ran headless it sync and then all my SpiderOak Hives on other devices got zeroed out. Luckily, I knew this will happen, so I made proper backup before. I excluded it one more time....and it doesn't seems to make any more problems. I added read-only datasets and they seemed to be backed up fine :) I think I won't use their sync feature anymore (at least not with linux/bsd client). Just wanted to know if anybody else had this problem or there is something wrong only with my account?

It seems SpiderOak 6.0 is out and it is called SpiderOakOne.
 
Last edited:

dreamerns

Dabbler
Joined
May 1, 2015
Messages
48
I hit another issue. If SpiderOak runs in headless mode (as background process), it only checks and uploads changes when process start. Afterwords all the changes to included dirs are just ignored. I've tried this 5x today and it is consistent. I would add new files to read-only dataset that is shared with jail and wait for couple of hours. After that I would check if process is still running (true) and use SpiderOak GUI client on another computer to check if files have been uploaded (they have not). I guess I'll just have to switch to batch mode and put it in cron.
 

pdx_beagle

Cadet
Joined
Nov 26, 2015
Messages
3
Thanks for all of this input. I'm very interested in getting SO working on my FreeNAS Mini. However, I'm getting stuck early on in the install. After creating a brand-new jail and assigning storage, when I try to enable the linux compatibility layer: "kldload linux" I get the following error:

kldload: can't load linux: Operation not permitted

I have been unable to find any relevant information about this error, and can't successfully proceed with the install without linux emulation. I am on the latest version of FreeNAS as of today: FreeNAS-9.3-STABLE-201511040813 Any tips or solutions from the gurus out there would be appreciated!
 

Teeps

Dabbler
Joined
Sep 13, 2015
Messages
37
Anyone tried this lately with a 9.10 jail? I am about to upgrade from 9.3 so I can get crashplan running in a jail instead of as a plugin. SO has an unlimited data offer going for the next few days.
 

dreamerns

Dabbler
Joined
May 1, 2015
Messages
48
Just to add to topic, I'm running SpiderOak now in VM in FreeNAS Corral, works like charm :smile:
 

dreamerns

Dabbler
Joined
May 1, 2015
Messages
48
No, I didn't use docker for this. I looked into docker options, but didn't find any good one. Problem is that SpiderOak headless client is not really good. I just created Ubuntu Server 16.04 VM and installed it inside.
 
Status
Not open for further replies.
Top