Replication failed emails, the sequel

[indivision]

That is definitely worrying! Is the file system on the PULL system mounted? It should look exactly the same at dataset and below level as the dataset you are replicating does on the PUSH side.

I did some further searching around since we have a nice, google-friendly symptom ("replication blank permissions").

According to this link, the blank folder on PULL is an expected security feature after all: https://bugs.freenas.org/issues/15837

So, I'm confused. Do you guys see your files on PULL when you set it up? It sounds like you are not supposed to...

And finally, that still leaves us with the "Replication Failed" email. It looks like it is the email notification that is faulty and the replication is working. But, I still want some way to verify that...

 

[artlessknave]

Well. I tried this. But, just got the same results. Blank permissions/owner.

I also tried changing those permissions/owner via the GUI on PULL. But, they remain blank after the change attempt.

wait, the pool on PULL has blank permissions? how is that even possible?
you specified that you "also" tried through the GUI, this kind of implies you did something NOT in the GUI. this should be ALL done in the GUI otherwise things can get confused b/w what you changed and what the GUI knows.

make pool vanguard on PULL
should be root/wheel - if this is the same as PUSH should be able to leave the same, otherwise change to whatever user/group you are logging in with.

 

[indivision]

Sorry. Probably the wrong terminology. It is the recursive datasets that have blank permissions. Same as is described in the bug report I linked to above. But, sounds like this is a normal behavior. I believe it's what you should see in your replicated datasets as well.

 

[rogerh]

That's all very interesting. This problem does not occur on my replication to zfsonlinux (on Centos 7). I have 3 levels of dataset on PULL, the dataset in which all the replication is done, which has become readonly following replication, the datasets which are actually replicated (which do not include my main pool/volume on PUSH) and the daughter datasets below 'jails'. All three levels have the zfs property of readonly but are all mounted below the automounted pool on PULL on /mnt/, all have permissions listable with ls, and the files within each level of dataset can be opened and read. So on zfsonlinux at least the readonly property is consistent with normal mounting and at least read access to the filesystem.


I would be very interested in the findings of anyone else who is using FreeNAS as a replication destination.

 

[rogerh]

That's all very interesting. This problem does not occur on my replication to zfsonlinux (on Centos 7). I have 3 levels of dataset on PULL, the dataset in which all the replication is done, which has become readonly following replication, the datasets which are actually replicated (which do not include my main pool/volume on PUSH) and the daughter datasets below 'jails'. All three levels have the zfs property of readonly but are all mounted below the automounted pool on PULL on /mnt/, all have permissions listable with ls, and the files within each level of dataset can be opened and read. So on zfsonlinux at least the readonly property is consistent with normal mounting and at least read access to the filesystem.


I would be very interested in the findings of anyone else who is using FreeNAS as a replication destination.

 

[indivision]

Hm. Interesting. I would also like to hear from others to double-confirm that it's normal.

Only thing I can think of is that there is some kind of setting that turns the ability to mount readonly drives off or on.

 

[rogerh]

Hm. Interesting. I would also like to hear from others to double-confirm that it's normal.

Only thing I can think of is that there is some kind of setting that turns the ability to mount readonly drives off or on.

It would seem quite important to be able to do so if the backup is to be of any use when needed. What happens if one does turn of the readonly property on a lowest level dataset? Does it become easily readable? Is it automounted?

 

[artlessknave]

I see totally normal appearing unix permissions on PULL. I had blank folders at an earlier time, when i was overwriting replications due to setting the destination pool wrong.
I have no idea how having inaccesable files on a backup server could possibly be a security feature; if you cant get to the files how is that a backup?
(I couldn't paste this before cuz my backup freenas was...under maintenance)

$ ls -la /mnt; ls -la /mnt/bkps ; ls -la /mnt/bkps/prod/data; ls -la /mnt/bkps/misc
total 6
drwxr-xr-x 3 root wheel 128 Jul 31 21:00 .
drwxr-xr-x 19 root wheel 26 Jul 31 20:57 ..
drwxrwxr-x 6 shdt1s wheel 6 Jul 14 01:58 bkps
-rw-r--r-- 1 root wheel 5 Jul 22 22:08 md_size
total 59
drwxrwxr-x 6 shdt1s wheel 6 Jul 14 01:58 .
drwxr-xr-x 3 root wheel 128 Jul 31 21:00 ..
drwxrwxrwx 3 root wheel 3 Jun 14 06:39 .home
drwxrwxr-x 15 shdt1s wheel 15 Jul 31 20:57 misc
drwxrwxr-- 3 shdt1s wheel 3 May 16 12:46 prod
drwxrwxr-x 9 shdt1s wheel 9 Jul 31 20:57 tyg
total 21862
drwxrwxr-- 22 shdt1s wheel 43 Jul 17 19:31 .
drwxrwxr-- 3 shdt1s wheel 3 May 16 12:46 ..
drwxrwxrwx 3 shdt1s wheel 3 Jul 7 23:07 .recycle
-rwxrwxr-- 1 shdt1s wheel 3792959 Jan 13 2017 2960_hg.pdf
-rwxrwxr-- 1 shdt1s wheel 934781 Jan 13 2017 9368.pdf
-rwxrwxr-- 1 shdt1s wheel 52990 Jul 7 2012 assessment.pdf
drwxrwxr-- 18 shdt1s wheel 28 Jul 19 18:46 bkp
-rwxrwxr-- 1 shdt1s wheel 2717040 Dec 4 2012 Books-of-Skyrim.mobi
drwxrwxr-- 4 shdt1s wheel 17 Mar 18 10:41 boot
drwxrwxr-- 10 shdt1s wheel 10 Mar 17 02:52 code
drwxrwxr-- 5 shdt1s wheel 5 Mar 27 04:41 converts
drwxrwxr-- 3 shdt1s wheel 751 Mar 18 12:56 cursors
-rwxrwxr-- 1 shdt1s wheel 718294 Apr 14 2012 cursors.7z
-rwxrwxr-- 1 shdt1s wheel 424 Aug 24 2016 dhparam.pem
-rwxrwxr-- 1 shdt1s wheel 245 Aug 24 2016 dhparam1024.pem
drwxrwxr-- 24 shdt1s wheel 116 Jul 22 05:43 downloads
-rwxrwxr-- 1 shdt1s wheel 6068 Dec 25 2014 drivesmartstatus0.txt
drwxrwxrwx 2 shdt1s wheel 4 Jul 9 22:01 firmware
drwxrwxr-- 14 shdt1s wheel 16 Mar 18 15:37 game-saves
drwxrwxr-- 38 shdt1s wheel 39 Mar 17 07:12 games
-rw-r--r-- 1 shdt1s wheel 8462780 Jul 17 19:29 hp-smx-provider-500.03.02.00.23-434156.vib
-rw-r--r-- 1 shdt1s wheel 4561026 Jul 17 19:25 hpacucli-9.40-12.0.vib
-rwxrwxr-- 1 shdt1s wheel 481680 Jul 14 2009 imagex.exe
drwxrwxr-- 14 shdt1s wheel 15 Jul 15 19:23 iso
drwxrwxr-- 10 shdt1s wheel 11 Mar 18 15:47 mdev
drwxrwxr-- 18 shdt1s wheel 18 May 14 07:22 media
-rwxrwxr-- 1 shdt1s wheel 1928 Jul 17 2012 MediaMonkey.exe.lnk
-rwxrwxr-- 1 shdt1s wheel 4851 Aug 26 2012 Menu Settings.xml
drwxrwxr-- 8 shdt1s wheel 38 Mar 18 15:48 mess
drwxrwxr-- 3 shdt1s wheel 37 Mar 18 15:44 n64roms
-rwxrwxr-- 1 shdt1s wheel 940 Sep 28 2013 naruto bijuu-jinchurichi song.txt
-rwxrwxr-- 1 shdt1s wheel 2996 Jun 12 2014 path masks (OLD).txt
-rwxrwxr-- 1 shdt1s wheel 0 Aug 12 2012 rsync
drwxrwxr-- 2 shdt1s wheel 9 Apr 9 19:33 scans
drwxrwxr-- 2 shdt1s wheel 15 Jun 22 2015 screenshots
drwxrwxr-- 9 shdt1s wheel 96 Jul 12 07:24 scripts
-rwxrwxr-- 1 shdt1s wheel 571 Aug 27 2016 size.txt
-rwxrwxr-- 1 shdt1s wheel 1105 Apr 23 2012 skyrim - Shortcut.lnk
drwxrwxr-- 9 shdt1s wheel 15 Mar 18 15:33 sync
-rwxrwxr-- 1 shdt1s wheel 1028102 Oct 3 2015 this is nine
-rwxrwxr-- 1 shdt1s wheel 22528 Sep 3 2015 Thumbs.db
drwxr-xr-x 4 shdt1s wheel 4 Jul 16 18:27 vm2
-rwxrwxr-- 1 shdt1s wheel 139916 Nov 6 2012 win8 recipt.xps
total 80
drwxrwxr-x 15 shdt1s wheel 15 Jul 31 20:57 .
drwxrwxr-x 6 shdt1s wheel 6 Jul 14 01:58 ..
drwxrwxr-x 3 shdt1s wheel 3 Mar 3 11:50 .home
drwxrwxrwx 3 shdt1s wheel 3 Apr 13 23:21 .recycle
drwxrwxrwx 5 shdt1s wheel 6 Apr 13 23:24 Blizzard
drwxrwxrwx 9 shdt1s wheel 13 Jul 16 01:44 downloads
drwxrwxr-x 3 shdt1s wheel 3 Mar 27 14:31 ext2fs
drwxr-xr-x 4 shdt1s wheel 4 Jun 6 23:23 home
drwxr-xr-x 11 root wheel 12 Jul 11 19:47 jaildata
drwxr-xr-x 12 root wheel 12 May 25 10:42 jails
drwxr-xr-x 2 root wheel 2 Apr 23 18:55 misc
drwxrwxr-x 4 shdt1s wheel 4 Feb 17 02:00 ntfs
drwxrwxrwx 7 shdt1s wheel 10 Jul 6 19:04 portable
drwxrwxrwx 2 nobody nobody 2 Apr 9 18:48 scans
drwxrwxr-x 2 shdt1s wheel 2 May 15 16:40 vm



 

[indivision]

I have no idea how having inaccesable files on a backup server could possibly be a security feature; if you cant get to the files how is that a backup?

An explanation was given from a FreeNAS team-member in the link above.

It's a security feature because it prevents your back-up from being altered by tinkering/some-other-process. It's no longer a back-up if it doesn't reflect the data you backed up.

You can get the files if/when you need them. It just requires the additional step of turning the read-only flag off first.

 

[artlessknave]

except mine doesn't do that....

 

[indivision]

except mine doesn't do that....

It sounds like you did initially. Maybe the way you have it now is something you altered that is not actually the intended functionality.

 

[artlessknave]

initially, yes, until I realized I was literally overwriting one pools replica with another pools replica. when I gave each pool a dataset to replicate to, it looked like a normal filesystem. that is the output I pasted.
when I had it with mangled permissions as you describe, it was successfully replicating and then cheerfully erasing it. everything you describe sounds like that, except you don't HAVE multiple pools, so you cant have done the same thing I did.

 

[indivision]

initially, yes, until I realized I was literally overwriting one pools replica with another pools replica. when I gave each pool a dataset to replicate to, it looked like a normal filesystem. that is the output I pasted.
when I had it with mangled permissions as you describe, it was successfully replicating and then cheerfully erasing it. everything you describe sounds like that, except you don't HAVE multiple pools, so you cant have done the same thing I did.

Yeah. I think that my backups are ok, despite the email errors (that I still get!). It would be nice to be able to look at the files for an additional re-assurance... Seems like read-only shouldn't necessarily mean not being able to even see that the files exist.

 

[artlessknave]

if you clone a snapshot, you should be able to see if it shows files.
if the clone has no files and the filesystem shows no files, then I don't see how its a backup, since it has no files
I honestly suspect its all broken, but if cloning a snap works then I guess its functionally a backup

 

[PhilipS]

An explanation was given from a FreeNAS team-member in the link above.

It's a security feature because it prevents your back-up from being altered by tinkering/some-other-process. It's no longer a back-up if it doesn't reflect the data you backed up.

You can get the files if/when you need them. It just requires the additional step of turning the read-only flag off first.

The security feature is that the volume is read only - not that you can't view the files. You can't view the files, because the system can't mount the child datasets when the parent dataset is readonly since the mount point can't be created. You should be able to zfs set readonly=off on the parent dataset and then create the mount point and then mount the child datasets.

When the replication script creates datasets it creates them with readonly=on. So even though vanguard/replica is not being replicated, if the replication task created it, it will be readonly when it doesn't need to be.

If vanguard/replica is readonly you can run

zfs inherit readonly vanguard/replica

Then mkdir for each of your child dataset mount points if they don't exist already on mnt/vanguard/replica and then run mount -a

The mounts for the grandchildren are part of the child replica (vitae), so they should show up and work once vitae is mounted.

This is what I have in my notes for getting this to work for my replicas back when I created them. I can see all my files in the replica through the CLI and they are read only volumes. HTH.

 

[indivision]

Thank you for the thorough breakdown.

To clarify, the data is already there, right? I have no need to see the files other than for peace of mind. I only need to see them if some issue comes up where I need to restore from backup. Then, I can go through the routine to mount, set readonly flag, etc.

 

[Derek Humes]

I am seeing this same issue on a pair of FreeNAS 9.10.2-U4 boxes. One is my primary, and it pushes snapshots to a backup device at another location, both of them running same version of FreeNAS. Every night, one of my recursive datasets (not the top level one, and only one out of about 80) send an email failure notice that
"Hello,
The replication failed for the local ZFS {dataset location} while attempting to
apply incremental send of snapshot auto-20171018.1800-2w -> auto-20171019.1800-2w to 192.168.2.244"

I check the status on replication and it reads "up-to-date". Also the dates change in the emails to match the previous days snapshots.

 

[Apollo]

Hi Indivision,

I haven't migrated my server yet to Freenas 11 on the Push side, and I am using my own replication script.
I am replicating to Freenas 11 on my backup PC and swap backup volumes.
Both my backup volumes are Write protected, so that Freenas doesn't mess with the data, except that replication is not being prevented. This has the benefit of guarenteeing datasets and snapshot will not be modified causing next snapshots to fail.

The issue with Read only volumes is that if you start replicating from scratch, or you created a new dataset that doesn't exist on the PULL side, having Read only volume will not prevent replication, however, when your reboot your PULL, or try to mount the dataset, Freenas will give you an error such as "Cannot mount dataset...".
It is scary at first, and you doubt your replication process and existance of your data as well as its integrity.
The good thing about ZFS is that your data is present and safe if the snapshot actually exist on the PULL.
If the dataset is not mountable, you will not be able to go into the folder location as it doesn't exist in memory.
If in doubt, you can do a diff of the snapshots or list the file content of the snapshots on the PULL but is pointless.

If the dataset cannot be mounted, because the volume is read only, you just need to go to CLI and change the flag as follow:

sudo zfs set readonly=off tank

Reboot your PULL server and Freenas will be able to mounted the remaining dataset.

You will be able to CD into the folders and list your files.

you can set volume as reado only after that:
sudo zfs set readonly=on tank

The key point to remember:
After replication, if the snapshot appears on the PULL, it means the files that exist on the PUSH that are pointed by this snapshot are also on the PULL.

There are also two types of replications, one which replicate the entire dataset permissions ( using the "I" option) or the one that will remove all permissions ( using the "i" option).