rsync: Directories written, files not?

[kooper2015]

Hi there all!

Not really new to FreeNAS, but now having an issue with rsync I can't
currently solve, without any hint.

1. Source: QNAP, Windows share.
2. Target: FreeNAS 9.3 latest as of now. Windows/SMB as well.

Set module to rsync from the QNAP to my FreeNAS. All is done with the GUIs on
both stations, no command options are set. The built-in QNAP test says: OK
and calulates some MB/s. Starting the task on the QNAP writes the FIRST level
of directory-structure perfectly, but copying the files looks like this on
FreeNAS:

Feb 20 20:47:51 freenas rsyncd[61466]: rsync: mkstemp
"/2002-05-some-path/.IMG_0329.JPG.z8UKDM" (in rsyncQNAPFotos) failed:
Operation not permitted (1)
Feb 20 20:47:51 freenas rsyncd[61466]: rsync: mkstemp
"/2002-05-some-path/.IMG_0330.JPG.bFdHmJ" (in rsyncQNAPFotos) failed:
Operation not permitted (1)

I do wonder about /.IMG..., usually this would mean files are hidden. But
they aren't written.

The rights should be OK on both (root, group wheel because it is all local;
admin on QNAP), even if I change those it'd give the same issue. Changing
filesystem of the target to UNIX does not help neither. Owner of the
directories *IS* root/wheel on FreeNAS:

drwxrwxr-x+ 2 root wheel 2 Feb 20 20:47 2002-05-some-path

On the QNAP owner is guest/user or admin (tried both). If admin, directory is
not written.

I do wonder about /.IMG..., usually this would mean files are hidden. But
they aren't written. But that's probably the way rsync reserve the files.

So, I don not know on which end is the problem: on QNAP or on FreeNAS? Any help appreciated!

 

[dlavigne]

Were you able to figure this out?

 

[kooper2015]

Hey,
thanks for feedback.

No, not yet. Currently I'm having another issue (can't export this backup- dataset to my Windows box any more, although the settings of the main, other dataset on the same pool is fine, with same parameters and permissions). Probably this comes from a downgrade to 201501241715, because 1 post (https://forums.freenas.org/index.php?threads/cifs-fails-after-9-3-201501301837-update.27858/) mentioned a similar issue after that version. Did not have the time to go back to actual.

CIFS/Samba seems to be a mess. Millions of different hints in this forum, but nothing seems to work or is just a bloody workaround. I can understand people leaving FreeNAS because of this! AND now I myself have doubts that my main dataset is safe and stable, with all that mess of permissions.

Everything else seems to be a breeze (I roughly installed my FreeNAS in 1h with required zpool, dataset, etc. as home user), but if FreeNAS can't get CIFS/Samba access safe and stable, FreeNAS will be or become a no-go.

BR,
kooper2015.

 

[SweetAndLow]

Samba cifs access is perfectly safe and stable, people just don't understand it and don't know how it works. Because people who know how it works don't post as much as people with issues things look skewed. That post you linked was actually user error and not an actual issue.

 

[Spearfoot]

@kooper2015 I am having exactly the same problem. In my case I am trying to use an rsync task to pull a share from my Synology NAS. Have you found a solution yet?

FWIW, I found a bug report on this very issue and have added a post there:

https://bugs.freenas.org/issues/7713#change-52758

 

[Spearfoot]

The problem seems to be related to the difficulty in translating permissions settings between FreeNAS/FreeBSD and Linux.

I've discovered a work-around: turn off the 'Archive' and 'Preserve permissions' checkboxes when you edit/create the rsync 'pull' task.

 

[kooper2015]

@Spearfoot!
Thanks for the link to the bug and your link there to my post here! So it seems developers are investigating... finally. I was not able to find this report, although I was looking for it. Just too many.

I have no solution; and I stopped trying. But I desperately need a backup.

As a side note, I was not using SSH.
I tried push and pull (from/to FreeNAS/QNAP, respectively), with all the results given in the bug-report:
rsync: connection unexpectedly closed
rsync error: error in rsync protocol data stream
rsync: failed to set permissions on...
rsync: mkstemp "..." failed: Operation not permitted (1)

BR, kooper2015.

 

[Spearfoot]

@kooper2015: if feeble memory serves - the mkstemp and other failures you describe are all permissions related. I'm a novice at *nix-style permissions, so I can't really explain what I did to make this work; but I suggest that tinkering with the permissions settings on your source and target systems may lead to a working solution. At least, it did for me...

 

[kooper2015]

Yes, Spearfoot, I know this all is related to permissions.
Permissions can be really difficult on plain Windows systems as well. And so traversing between Linux/FreeBSD/Windows/CIFS is even more complex...
I'm reluctant to change the permissions or ownerships of 1000s of files/directories on one or the other end. That's what the OS should do. Something seems quite wrong if directories can be written, but files fail in a single-user setup....

 

[kooper2015]

Found another one, which seems to be related:
https://bugs.freenas.org/issues/5854
This is flagged as 'nice to have', though. :(

This issue is 1 year old, no progress.

I can't run rsync -rltgoD (for example, as stated in last comment of #5854) on the source, as the QNAP doesn't allow any free options in the GUI (and I must use the GUI).

Cheers.

 

[kooper2015]

See also:
https://bugs.freenas.org/issues/12127

This seems to be very similar. That bug appears to happen FreeNAS to FreeNAS, so the QNAP NAS doesn't seem to be the problem?
Hope this is fixed asap ('screened' right now)...

Edit: typo.

 

[desertrider]

My Solution: I was looking at Bug #1217, for this rsync error, and the programmer mentioned this was by design and you have to use "-A --no-perms" included in your command and when I used it everything worked.

Include "-A --no-perms" in your rsync command and it should now work (hope this helps someone as I was pulling my hair out over this –no-perms )

I know this is an old thread but I had this same issue with ZFS datasets, share type=Windows, and using SMB shares. In SSH (I use Putty), using rsync to copy files from one dataset to another would copy directories but NOT the files and you would get an error regarding the mkstemp failed - also it continues but if you have a lot of files it will stop

I had --progress with --stats and it just quit in my Putty SSH window - I was able to hit Ctrl-C to cancel and then I could see it trying to write all the mkstemp files into the directories but got mkstemp failed operation not permitted.

Here is my total WORKING command, used in Putty SSH, I use to copy files from one directory to another along with getting progress & stats:
rsync --partial --stats --progress –A –a –r –v --no-perms /mnt/BEAST22T/Videos/ /mnt/MINI7T/Backups_PERMANENT/Videos-New-Location

** You can add the “-n” for test run - it will run and let you know the results without actually copying the files ***

Here is what each of the commands I use is performing - I include for those who may want to use this and customize it:

--no-perms (**Must use –A with this) (MUST HAVE THIS FOR SMB, Windows ACL Shares or you will get directories but not files copied) – this tells it to not preserve permissions.
--perms = -p = preserve permissions (This is the original perms command that we are adding --no-perms

--partial
By default, rsync will delete any partially transferred file if the transfer
is interrupted. In some circumstances it is more desirable to keep partially
transferred files. Using the --partial option tells rsync to keep the partial
file which should make a subsequent transfer of the rest of the file much faster.

--stats
This tells rsync to print a verbose set of statistics on the file transfer,
allowing you to tell how effective the rsync algorithm is for your data.

--progress
This option tells rsync to print information showing the progress of the transfer.
This gives a bored user something to watch.
This option is normally combined with -v. Using this option without the -v option
will produce weird results on your display.

-r, --recursive
This tells rsync to copy directories recursively.
If you don't specify this then rsync won't copy directories at all.

-A --acls
Preserve ACLs (implies –p = --perms which is permissions when using –A you do not need -p)

-a, --archive
This is equivalent to -rlptgoD. It is a quick way of saying you want recursion and
want to preserve almost everything.

-v, --verbose
This option increases the amount of information you are given during the transfer.
By default, rsync works silently.
A single -v will give you information about what files are being transferred
and a brief summary at the end.
Two -v flags will give you information on what files are being skipped and
slightly more information at the end.
More than two -v flags should only be used if you are debugging rsync.

--delete
This tells rsync to delete any files on the receiving side that aren't on - You can add this if you want to us this to keep exact copies on the target as the files change on the source.

 

[guermantes]

Just mentioning my experience in case it can help someone.

I was struggling with the very problem the thread describes, and could not get it to work when using -a flag (rsyncing from linux desktop to freenas over ssh). I tried verious variants of the 'expanded -a flag' and found out it was -p and -t that were causing me problems, so I ended up using -rlgoDPv. At the moment, losing local permissions was acceptable to me.

I was happy to find this thread and started trial and erroring with @desertrider's proposed solution to include -A --no-perms. So I tried -av -A --no-perms. Alas, that broke the connection:

Code:

opening connection using: ssh -l ......
sending incremental file list
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [sender=3.1.1]

After some further exploration I tested simply -av --no-perms, now it works. I don't now if I have achieved something or if that is just equivalent to -rlgoDPv. Anyway, it seems the ACLS could be the culprit. Don't know if it is relevant, but the files I am transferring have no permissions set explicitly in windows. So maybe there 'is no ACL' to begin with?

 

[TrumanHW]

Hey y'all. I've started doing a copy -- and it seems that it [only] copies the folders ... and not the files with the actual data.

Is there an actual solution to this within this thread? I can't identify it if so.

I'm trying to extract my QNAP to a FreeNAS.. Thanks.

 

[rcd1]

Why is this such a problem with rsync when it is no problem with scp?

 

[rcd1]

Anyway, just to confirm - adding --no-perms to the rsync command seems to allow the copy to proceed as normal.

 

[shaulliv]

Has the same problem.
The solution seems to be to change the ACL mode of the dataset to passthrough(It's in the advanced options).

DISCALIMER: I have not tried this on windows yet, from other posts this could cause issues.
https://www.truenas.com/community/threads/file-rights-acl-please-kill-me-now.91439/