rsync over ssh with Android phone

[justusiv]

Seems like this has gotten off the topic of the OP, Everyone is talking about alternatives to rsync
However those alternatives all choke and die on my sdcard with 60k+ files. and rsync handles them in less then a minute
The program i use is the same the the OP i believe
https://play.google.com/store/apps/details?id=eu.kowalczuk.rsync4android&hl=en
I have had this setup and working very nicely for over a month however i have not set it up on a freenas box yet.....
but that is my ultimate goal.
however i just got my freenas box setup about 5 minutes ago....

 

[cyberjock]

Seems like this has gotten off the topic of the OP, Everyone is talking about alternatives to rsync
However those alternatives all choke and die on my sdcard with 60k+ files. and rsync handles them in less then a minute
The program i use is the same the the OP i believe
https://play.google.com/store/apps/details?id=eu.kowalczuk.rsync4android&hl=en
I have had this setup and working very nicely for over a month however i have not set it up on a freenas box yet.....
but that is my ultimate goal.
however i just got my freenas box setup about 5 minutes ago....

And it won't work with FreeNAS' rsync. That's why we got off topic. There doesn't appear to be any rsync program that is compatible with the latest and greatest stuff.

The person that made FolderSync does have on his website that he intends to implement rsync in the future. Considering it supports so darn many options I fully expect him to implement it someday. When he does you can be assured that I plan to update this thread.

Edit: FolderSync WILL do what you want despite the number of files. You have control of syncing with your FreeNAS server based on file dates/times or contents(which I believe is via checksum). MANY programs update files on your SD card but the file date/times do NOT get updated, so I do file contents. I found that about 1/2 the programs I use that save files to your SD card(game saves, config files, etc) don't update the file date/time and some don't even change file size. Rsync can't be going by file contents if you are doing a scan of 60k files in less than a minute.

Your 3 options:

1. Go by file/date/file size(not very thorough from my experience)
2. Go by file contents(CPU heavy on at least one end but minimal network utilization).
3. Copy the files every time(low CPU usage but very heavy network utilization).

Rsync was designed so you didn't have to do #3.

 

[justusiv]

I am an owner and user of foldersync myself however it flat out can not handle 60k+ files. It takes hours and hours to presync and that was when i have had 10-20k files and it would finally fail and never even transfer a file.

Since i have been a freenas owner for about 5 minutes i dont know all the details..... can you not rsync via the ssh service? i see there is an rsync service also. do you have to do it through that?
My other thought is a workaround. I can rsync to a box i have setup in my shop from android just fine. works great in fact.... maybe a little to great :P but try it for yourself. from there i could rsync from the box to the freenas.

 

[cyberjock]

The problem is that program uses a very lame SSH program called dropbear. It supports only the oldest and least secure encryption keys. Those key types are not compatible with FreeNAS' rsync. So you can't setup FreeNAS to use the ssh key. I emailed the person that wrote the rsync backup program for Android and he said "sorry, it won't work without those keys supported". From a couple of chats in IRC I was told that those keys are obsolete and not supported in FreeNAS. So I abandoned the idea of using rsync(for now) in favor of FolderSync.

I'm not sure what phone you have, but I have about 52k files on my phone and have zero problems syncing my phone to my server. The only problem I have is if my phone is in a place with poor wifi reception sometimes the connection will terminate mid-sync. However since I have extremely powerful wifi I don't have that problem anymore :)

 

[justusiv]

I have had both a Galaxy Nexus and a Galaxy Note II for me foldersync starts running super slow after the initial sync when syncing a ton of files. I use it for a quick dropbox and box.net sync. but i can not backup my whole sd card with it.
Back to getting data from your android to your freenas, if you have a quick pitstop in a middle man server it seems to me you can pull this off. Ok dropbear might not be the latest greatest but it works and it is not clear text. That pitstop will be behind my network so internal speed will be blazing and i am more then satisfied with the speeds i get on my 4g connection to my home setup currently.
Like i said i have yet to play with moving data to my freenas box via rsync but there are options.
And as i write this i just remembered that Terminal IDE has a working rysnc built in.... you can write a shell script that can do the dirty work or just do it by hand. I would think you could pick the encyrption type with that one dss rsa etc.... someone give it a whirl.

 

[cyberjock]

I don't know what to say. I just deliberately touched every file on my sd card, and they all resynced(of course, it took a while) but no issues. My phone is a Droid Bionic...

I doubt you're going to find anyone to test it out. AFAIK all of the experimenting done I did myself. Everyone else provided some assistance when I got stuck and other options.

 

[justusiv]

ummmm so i just rsyned from my phone(android) to my freenas box. I have not setup an rsa key yet but it worked when prompting me with a password. I dont imaging getting my key going will be that crazy.

 

[cyberjock]

ummmm so i just rsyned from my phone(android) to my freenas box. I have not setup an rsa key yet but it worked when prompting me with a password. I dont imaging getting my key going will be that crazy.

That's the issue. If I have to enter a password that means the backups aren't automated. What happens when I'm in a hurry and accidentally close the prompt for the password. Or what if the program only gives you 60 seconds to enter the password. Do you plan to set a timer to make sure you can enter your password as soon as its prompted? You either have to go passwordless(which is stupid on so many levels) or get the keypair working. Not to mention that the dropbear included isn't the latest version, but the latest version(if you compiled yourself for Android I presume) still wouldn't work.

The key is your "password", but the keytype used isn't compatible with FreeNAS.

 

[justusiv]

Ok maybe i have not read this thread thoroughly enough but i thought the issue was that people couldn't get it done, not that that couldn't "automate" it. The program i used to rsync over isn't that playstore rsync one that only supports dropbear it was a terminal that had rsa support. So in my humble opinion i can set this up to use "keys" it still isn't 100% automated at that point but a simple shell script to get it done. I have about 2 days worth of linux/terminal/freenas/rsync/rsa/dropbear/etc experience so if i am a little off on this forgive me. If i get a shell script using an rsa key have we made progress or am i just catching up?

 

[cyberjock]

No clue. I tried to use the rsync program from the Android store. When I figured out it wasn't compatible at all I gave up. I wasn't about to try to mess with creating some bastardized custom backup setup. I was hoping for something that could be useful to the forum as a whole with little or no setup. Never know what won't work when you start customizing stuff in FreeNAS from the command line. I try to stick to the rule if yo can't do it in the GUI don't do it unless you ahve no choice. In my case I had a choice.. FolderSync.

There's just too many people that aren't FreeBSD experts and want a simple solution that works. I thought rsync would work best because that's built into FreeNAS and is designed especially well for backups. That's the only reason I tried rsync at all.

 

[danividalg]

Hi,

I think I have solved the problem with dropbear ssh to FreeNAS.

The trick is on manually creating the key with dropbear command (from a command terminal):
/data/data/eu.kowalczuk.rsync4android/files/dropbearkey -t rsa -f /sdcard/id_rsa -s 4096 > /sdcard/id_rsa.pub

This will create a private (/sdcard/id_rsa) and public (/sdcard/id_rsa.pub) keys of 4096 length (very strong) in RSA format.

Then you only need to use /sdcard/id_rsa as private key on RSync for Android app and from the /sdcard/id_rsa.pub file only copy and paste the line "ssh-rsa.........root@localhost" on your ssh-public-key field of the user you want to use for rsync to FreeNAS.

In addition, I want to comment that I use the following rsync options "-vHrltDpczhs --progress --exclude .android_secure"

Hope this help.
If anyone have a question ask here.


Best regards :)

 

[ShutterBC]

Hi,

I think I have solved the problem with dropbear ssh to FreeNAS.

The trick is on manually creating the key with dropbear command (from a command terminal):
/data/data/eu.kowalczuk.rsync4android/files/dropbearkey -t rsa -f /sdcard/id_rsa -s 4096 > /sdcard/id_rsa.pub

This will create a private (/sdcard/id_rsa) and public (/sdcard/id_rsa.pub) keys of 4096 length (very strong) in RSA format.

Then you only need to use /sdcard/id_rsa as private key on RSync for Android app and from the /sdcard/id_rsa.pub file only copy and paste the line "ssh-rsa.........root@localhost" on your ssh-public-key field of the user you want to use for rsync to FreeNAS.

In addition, I want to comment that I use the following rsync options "-vHrltDpczhs --progress --exclude .android_secure"

Hope this help.
If anyone have a question ask here.


Best regards :)

Just writing to report that this works beautifully for me, and makes a lot of sense. The author of the app could pretty easily add an option within the app to choose key types to generate, at the slight expense of usability -- but in this case it's rather worth it.

 

[JJT211]

My Droid Bionic
Bionic

Wow youre still on the Droid Bionic?!? That was like 5 phones ago for me....lol

 

[cyberjock]

Feb 2013 was when that post was made.. I've had 2 phones since then. :P