Replication seems to work, but files are not visible.

[Malakym]

We have some replication already setup and it is working as we expect.

After setting up a new dataset to replicate, we find that it copies the data (bandwidth use, takes as long as we'd expect etc), the destination shows the correct dataset size using zfs list and in the GUI it shows the dataset has been created. The snapshots from the source also show as snapshots on the destination. But if we ls on the command line to any of these folders they are empty.

At first we were on 9.10.2, so updated to -U1 (86c7ef5) and started the dataset/replication from scratch with no luck. Also setup a new machine to try this and the target is also "empty", and tried without encryption as well.

This may be related, but when we export this volume and remove it, when we reimport it cannot find the dataset/directory to mount after reimporting the zfs volume.

 

[m0nkey_]

You don't specify what the target machine is, what OS it's running, etc.

It could be the ZFS dataset on the target machine is not mounted.

 

[Malakym]

You don't specify what the target machine is, what OS it's running, etc.

It could be the ZFS dataset on the target machine is not mounted.

Sorry, assumed the assumption would be what I have... FreeNAS -> FreeNAS

ZFS dataset is mounted (zfs mount), its a dataset which was created just for this job and has nothing else in it. Does the replication not mount it anymore on the destination?

 

[m0nkey_]

If both machines are FreeNAS, verify your replication configuration based on the documentation: http://doc.freenas.org/9.10/storage.html#examples-common-configuration

 

[Malakym]

If both machines are FreeNAS, verify your replication configuration based on the documentation: http://doc.freenas.org/9.10/storage.html#examples-common-configuration

This is all fine, the key is okay since I can watch the log on both machines and see the replication going on (that and data being transfered), and can manually connect from one to another with the
/data/ssh/replication key - also noted that the first problem we had was with a machine that has already got replication items setup from months ago.

I can list the snapshots on the destination and they match the source.

 

[PhilipS]

This happened to me when I setup replication to a target dataset that does not already exist on PULL. When the replication creates the target, it doesn't seem to get mounted properly or something. This may only happen on nested datasets. Not sure what the fix is without deleting the dataset on PULL and creating it before starting replication - which is what I did. This is on 9.10.2-U1.

 

[PhilipS]

Okay, did some testing on the scenario that happened to me, and I am able to reproduce this, maybe your scenario is similar:

Create parent dataset and child dataset on PUSH (I created tank/test and tank/test/subtest and placed a file in subtest called testfile)
Do not create parent dataset on PULL
Create a periodic snapshot on the child dataset
Create replication task on the child dataset
After replication, the child dataset will not be mounted on PULL
This will be sent to stderr: failed: cannot mount '/mnt/tank/test/subtest': failed to create mountpoint

On PULL:

Code:

[root@FreeNASSD2] /mnt/tank/test# ll
total 9
drwxr-xr-x  2 root  wheel  2 Feb  7 09:37 ./
drwxr-xr-x  8 root  wheel  8 Feb  7 09:37 ../

[root@FreeNASSD2] /mnt/tank# zfs get mountpoint tank/test/subtest
NAME               PROPERTY    VALUE                   SOURCE
tank/test/subtest  mountpoint  /mnt/tank/test/subtest  default

[root@FreeNASSD2] /mnt/tank/test# mkdir subtest
mkdir: subtest: Read-only file system

[root@FreeNASSD2] /mnt/tank# zfs inherit readonly tank/test
[root@FreeNASSD2] /mnt/tank/test# mkdir subtest
[root@FreeNASSD2] /mnt/tank/test# mount -a
[root@FreeNASSD2] /mnt/tank/test# cd subtest
[root@FreeNASSD2] /mnt/tank/test/subtest# ll
total 2
drwxr-xr-x  2 root  wheel  3 Feb  7 09:45 ./
drwxr-xr-x  3 root  wheel  3 Feb  7 09:46 ../
-rw-r--r--  1 root  wheel  0 Feb  7 09:45 testfile
[root@FreeNASSD2] /mnt/tank/test/subtest# zfs set readonly=on tank/test

There is a bug report on this: https://bugs.pcbsd.org/issues/15050

Status is "Behaves correctly" I would think the preferred behavior would be for the datasets should be set to readonly AFTER all the child mount points are created on first replication.

 

[PhilipS]

I made a feature request to change the behavior of the autorepl.py script to set the readonly property on the parent after the child datasets are created: https://bugs.pcbsd.org/issues/20971

 

[Malakym]

Took us a while, but our testing narrowed it down to the read-only flag as well.

 

[ccdanieldb]

I have the same problem. How did you view the files on the destination freeness pool. It looks like the GUI has the readonly flag set to false. Is there somewhere else that I can look for the flag.

OS Version:
FreeNAS-11.2-U7
(Build Date: Nov 19, 2019 0:4)
Processor:
Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz (32 cores)
Memory:
128 GiB

 

[PhilipS]

zfs get readonly {dataset} in the shell will allow you to check for the read only flag. If the dataset isn't mounted you won't be able to see the files.

 

[ccdanieldb]

Thanks, the above command

Code:

zfs get readonly

in shell, returned that the readonly flag was set to off. I gave a reboot a try and after the reboot I was able to see the files in the data set. I am now, not sure if it is a bug or I just failed to mount the data set. But it looks good now. I wonder if it could be as simple as the permissions had not had a chance to recursively set across the entire pool.

 

[Dariusz1989]

Yo

I guess I have the same problem? I made new replication task using wizard. The job copied 4tb of data but when I open the folder its empty. Nothing inside. How do I see the data?