Rsync over SSH issue

[LIGISTX]

I am having yet another issue with 11.1U4, I swear my luck has just been horrible.

I can't get an old script I had that used lftp to transfer files from a remote host to my freenas box to work, so I am trying to resprt to rsync. Less elegant, but at this point I will take anything.

I can get it to work in CLI no problem. I SSH into my box, use

Code:

rsync -avz -e ssh user@remotehost:remote/directory /local/directory

Works totally fine, I have tested this multiple times.

Now, first issue, I try that code broken down into the WebUI for easy management, and it doesn't work. I get this error:

The Remote Path you specified does not exist or is not a directory.
Either create one yourself on the remote machine or uncheck the
'rsync_validate_rpath' field.
*Note*: This could also happen if the remote path entered
exceeded 255 characters and was truncated, please restrict it to
255. Or it could also be that your SSH credentials (remote host,etc) are wrong

Ok, well, I found a way around that webUI error, I tried user@remotehost'sACTUALIPaddress instead of its hostname (which works in the CLI, I can ssh with the hostname credentials...), and that got rid of that error, but still no dice. I get this in my log file while using the WebUI Rsync setup:

rsync error: error in rsync protocol data stream (code 12) at io.c(226)

So I said screw it, at this point I just want it to work, so I put the exact code from the CLI that worked into a cron job in freenas WebUI and it doesn't work. I am sorta running out of patients for 11.1, and I have no idea why everything I try just refuses to work. This is a little ranty, but I am starting to wonder if 11.1 just wasn't for me and I should have stayed on 11.0 where I had no issues and everything worked...

My old script no longer works, after exhausting all possible things I could possibly think of, including trying to run it via a cron job in the jail it was in (this script worked last month on my 11.0 box, and works right now on my friends 9.x box), and I can't seem to get rsync working, albeit this is my first attempt at using rsync.

Any help would be amazingly appreciated.

Also, all of this has been in the "old" webUI, is there a new webui in anticipation of iocage ect that I should be using instead?

 

[c32767a]

I can't diagnose why you're having problems, but if it helps, I use rsync with 11.1 all the time.

I have an rsync module configured under Services > Rsync. It specifies the local path to where I want target data to be synced.

On the remote host, which is BSD and not FreeNAS, I have a cron job that runs:

/usr/local/bin/rsync -av --bwlimit=2500 --ignore-existing /data/* rsync://192.168.1.5/datasync

so I push the data to the nas.

IF you're going FreeNAS to FreeNAS, that should just be an rsync task on one side and a rsync module on the other. it's really pretty basic. Maybe you're overcomplicating something? Are both ends FreeNAS? if not, what's at each end?

 

[LIGISTX]

I can't diagnose why you're having problems, but if it helps, I use rsync with 11.1 all the time.

I have an rsync module configured under Services > Rsync. It specifies the local path to where I want target data to be synced.

On the remote host, which is BSD and not FreeNAS, I have a cron job that runs:

/usr/local/bin/rsync -av --bwlimit=2500 --ignore-existing /data/* rsync://192.168.1.5/datasync

so I push the data to the nas.

IF you're going FreeNAS to FreeNAS, that should just be an rsync task on one side and a rsync module on the other. it's really pretty basic. Maybe you're overcomplicating something? Are both ends FreeNAS? if not, what's at each end?

Thanks for the reply.

The remote end is Ubuntu. It’s not a server I own or have admin rights on, so I am not sure what extent I can go to on that end up run a cron for example, but it does have rsync installed, which I believe means theoretically I could have my job set up as a normal rsync task and not rsync over ssh.

That being said, I don’t own the server so I don’t want to give it any info about my FreeNAS box I don’t have to give it which is why I attempted to set it up as a pull over ssh, leave it all up to FreeNAS.

And like I said, it works fine in the CLI, I tested multiple dummy paths I created just for testing and it worked flawlessly, but won’t work if I try it through webUI.

Any idea why that would be? Or am I missing a step and need to do something on the remote host side?

Thanks!


Sent from my iPhone using Tapatalk

 

[c32767a]

And like I said, it works fine in the CLI, I tested multiple dummy paths I created just for testing and it worked flawlessly, but won’t work if I try it through webUI.

Any idea why that would be? Or am I missing a step and need to do something on the remote host side?

If the remote side isn't running an rsync service, then the "rsync" labeled functions in the UI won't work. I see the config options for rsync over SSH, but I've never gotten them to work in our environment.

Since you just want to connect to the remote host and do a pull, IMHO, the simplest thing to do would be to make sure the command does not require any user interaction and then add it as a cron job in the UI. Go to Tasks > Cron Jobs. Just paste your entire command line into the "command" box, then set the schedule for what you want.. Make sure you use full paths in the rsync command. Relative paths will end up based on wherever FreeNAS runs cron jobs, which is probably your home dir, but not guaranteed.

 

[LIGISTX]

If the remote side isn't running an rsync service, then the "rsync" labeled functions in the UI won't work. I see the config options for rsync over SSH, but I've never gotten them to work in our environment.

Since you just want to connect to the remote host and do a pull, IMHO, the simplest thing to do would be to make sure the command does not require any user interaction and then add it as a cron job in the UI. Go to Tasks > Cron Jobs. Just paste your entire command line into the "command" box, then set the schedule for what you want.. Make sure you use full paths in the rsync command. Relative paths will end up based on wherever FreeNAS runs cron jobs, which is probably your home dir, but not guaranteed.

Thanks for the info. I believe I did try this as I copied my exact CLI code into the cron job webUI and it didn’t work. I was using full paths, and it had just worked in the CLI. This is very reminiscent of the issue I was having previously of not being able to execute a script via the webUI cronjob.

When I get home later today I can try this approach again and try and track down a log of what is going wrong.

Just for background, previously I was attempting to do this same file transfer via a script I have, which used to work on 11.0 and is still working on a deployed 9.x box. The script works fine if I run it in a jail manually, but a cronjob fails to execute it correctly regardless of how I try and tell it to execute it with flags and such. It’s just starting to feel like something is broken in 11.1 in regard to cronjobs.

I will attempt again later and try and report back what I can find. Unfortunate I work for a defense contractor and beyond not having internet for most of my day, I definitely wouldn’t be able to ssh into my box even if I did :(.


Sent from my iPhone using Tapatalk

 

[LIGISTX]

In case you are at all curious about my script issue, this is the thread I had and still haven’t been able to get it working, thus the rsync route. The script was much more elegant as I was able to keep a list of folders I wanted to sync in a .txt that I referenced, rsync will require a separate rsync job for each folder (I only have about 15 folders I want to sync, so it’s not the end of the world).

Just sorta feels like this is a cronjob issue. Since in both instances things that work in the CLI I can’t get to work via a cronjob.

https://forums.freenas.org/index.ph...-script-within-jail-from-freenas-shell.63370/

Thanks again for the help. This has been very frustrating.

Sent from my iPhone using Tapatalk

 

[LIGISTX]

Ok, I think in this instance I did find an error on my part, I did use shh-add, but I didn't use something like keychain to keep the key available until the next server reboot for instance. This would be limiting the cron and rsync tasks from being able to be successful.

Thankfully I realized this today as soon as I started playing with what could possibly be wrong, but I also have never set this up myself, so that will be today's endeavor. To bad I have a softball game here pretty soon....

 

[LIGISTX]

Ok, edited my .bash_profile and it now remembers ssh-add through sessions. So that is a start, but it didn't fix any issues.

I still can't seem to get rsync to work via webUI, nor does the cronjob.

The cronjob is an exact copy paste of my CLI code.

It is

Code:

rsync -avz -e ssh user@hostname:/directory/to/test/file/on/remote /mnt/freenas/test/dir

Works great in CLI, doesn't work in a cronjob. I can't figure out why I can't get code to execute from a cronjob correctly. I tried EVERYTHING with my old script, but now I am trying something "as simple" as rsync and it won't work. I am pretty confused, does no one else have these issues with 11.1?

@c32767a

 

[LIGISTX]

Any ideas?


Sent from my iPhone using Tapatalk