replication dataset set a read-only

[nojohnny101]

My system specs for both my main server and my backup server are in my sig.

I have found multiple threads on this issue but most are from around Feb/Mar of 2016.
Here, here and here. I thought I would ask since most of those threads say it was something to be fixed by a software update. Both system are running the latest stable train: FreeNAS-9.10.1-U2 (f045a8b).

If I am remembering correctly, these issues started sometime after I updated to either U1 or U2 on both the main server and the backup server. (I can't pinpoint which one). The replicated dataset was definitely not read-only before updating to U1.

I noticed because I started getting these error messages in the console of the PULL machine:

Code:

Oct 18 23:02:30 tank-backup collectd[3258]: statvfs(/mnt/tank-backup/replicate/tank) failed: No such file or directory

These occur at 10 second intervals consistently.

I have a periodic snapshot task setup on the PUSH machine to replicate mnt/tank (recursive) to the destination mnt/tank-backup/replicate.
What is strange is that all other replicated datasets on PULL are not read-only. The only that seemed to have "flipped" to read-only after the updates was the top level one (mnt/tank). Strange.

I should note replication is still operating as scheduled and is successful.

Any ideas?

 

[depasseg]

Do you only receive error messages for read only datasets?

Are they mounted at that location? I was seeing this error a while ago, and it was do to the fact that the datasets weren't being mounted, and so the process to report the stats (statsvfs) couldn't pull the info and threw that error.

 

[nojohnny101]

Do you only receive error messages for read only datasets?

Yes.

Are they mounted at that location? I was seeing this error a while ago, and it was do to the fact that the datasets weren't being mounted, and so the process to report the stats (statsvfs) couldn't pull the info and threw that error.

I'm not sure what you mean here. Why would all other replicated child datasets be mounted and not read-only? Is it a setting? Is it true (mentioned in the other threads) about it being dangerous to mount a replicated dataset as read-write?

 

[depasseg]

I'm not sure what you mean here.

Back in one of the 9.3.1 updates, the replicated datasets wouldn't mount (https://bugs.freenas.org/issues/12455). So not only was the dataset read-only, it wasn't mounted. So the stats gathering scripts (dataset usage, etc) would fail (hence that statsvfs error message). I found that the "read-only" was a red herring. I'm not sure if that's the same thing that's happening here, but it's something to consider.

You can try setting the datasets as read write to see if that fixes the issue. People have been doing that since this change was found and I haven't heard of any problems.

Is it a setting?

I think it's a bug. Assuming it's actually happening as I'm understanding.

 

[nojohnny101]

What you are saying makes sense. Following that train of thought, is it true that replication doesn't require the dataset to be mounted? Replication is successful but the I think you are correct in saying the stat gathering scripts are throwing errors because it is not mounted.

I'd prefer to leave it read-only (although as I said, strange that only the parent dataset is read-only, all other replicated datasets are not). I think I'll just try and mount it and see if the messages go away. I hope the mount persists though through the replications that I have setup to run every night.

 

[depasseg]

is it true that replication doesn't require the dataset to be mounted?

yes, zfs replication happens at a layer below the filesystem mounting.

Can you post the output of

Code:

zfs mount

? If they are unmounted, and you manually mount them (zfs mount -a), and then the error message disappear, my guess is that after the next replication the error will return. If that is the case, I would suggest opening a bug report, since I've been hearing about this recently.

 

[depasseg]

Looks like someone else has it reopened. https://bugs.freenas.org/issues/15355

 

[nojohnny101]

here is the output of zfs mount

Code:

[root@tank-backup] ~% zfs mount

freenas-boot/ROOT/9.10.1-U2	 /

freenas-boot/grub			   /boot/grub

tank-backup					 /mnt/tank-backup

tank-backup/.warden-template-standard--x64  /mnt/tank-backup/.warden-template-standard--x64

tank-backup/replicate		   /mnt/tank-backup/replicate

tank-backup/replicate/.warden-template-standard--x64  /mnt/tank-backup/replicate/.warden-template-standard--x64

tank-backup/replicate/adam	  /mnt/tank-backup/replicate/adam

tank-backup/replicate/tank	  /mnt/tank-backup/replicate/tank

tank-backup/replicate/family	/mnt/tank-backup/replicate/family

tank-backup/replicate/jail	  /mnt/tank-backup/replicate/jail

tank-backup/replicate/jail/plex  /mnt/tank-backup/replicate/jail/plex

tank-backup/replicate/jail/plexpy  /mnt/tank-backup/replicate/jail/plexpy

tank-backup/replicate/master	/mnt/tank-backup/replicate/master

tank-backup/replicate/nader	 /mnt/tank-backup/replicate/nader

tank-backup/replicate/photolibrary  /mnt/tank-backup/replicate/photolibrary

tank-backup/tunnel			  /mnt/tank-backup/tunnel

tank-backup/replicate/jail/.warden-template-VirtualBox-4.3.12  /mnt/tank/jail/.warden-template-VirtualBox-4.3.12

tank-backup/replicate/jail/.warden-template-standard  /mnt/tank/jail/.warden-template-standard

tank-backup/replicate/jail/.warden-template-standard-9.3-x64  /mnt/tank/jail/.warden-template-standard-9.3-x64

freenas-boot/.system			/var/db/system

freenas-boot/.system/cores	  /var/db/system/cores

freenas-boot/.system/samba4	 /var/db/system/samba4

freenas-boot/.system/syslog-f1ae6c68bbe041c7bb38cadeec088781  /var/db/system/syslog-f1ae6c68bbe041c7bb38cadeec088781

freenas-boot/.system/rrd-f1ae6c68bbe041c7bb38cadeec088781  /var/db/system/rrd-f1ae6c68bbe041c7bb38cadeec088781

freenas-boot/.system/configs-f1ae6c68bbe041c7bb38cadeec088781  /var/db/system/configs-f1ae6c68bbe041c7bb38cadeec088781

I have tried both "zfs mount -a" and "zfs mount mnt/tank-backup/replicate/tank" and both fail to stop the statvfs messages. Strangely (I hope this isn't something stupid I'm missing), the second command gives a "dataset does not exist" error.

Glad to see a bug has been opened up.

 

[depasseg]

the zfs mount command needs the dataset (tank-backup/replicate/tank), not the filesystem (/mnt/blah).
But it appears that it's mounted, so I'm confused. Is tank your primary pool name? IOW, what is the output of zfs mount on your primary machine?

 

[nojohnny101]

hmm, the reason I ran the "mount -a" with the filesystem at the beginning is because that is what is documented: Oracle ZFS Mount Documentation (maybe i'm reading it wrong).

Sorry it has taken me long, I wanted to compile all the correct information. Thanks for the help:
Here are screenshots of the layout of the datasets and volumes on primary and secondary machines:

I just have one periodic snapshot task setup that is at the top level of primary (tank, recursive) that replicates to the dataset "replicate" on the backup.


ZFS Mount on primary machine:

Code:

[root@tank] ~# zfs mount
freenas-boot/ROOT/9.10.1-U2	 /
freenas-boot/grub			   /boot/grub
tank							/mnt/tank
tank/.warden-template-standard--x64  /mnt/tank/.warden-template-standard--x64
tank/adam					   /mnt/tank/adam
tank/family					 /mnt/tank/family
tank/jail					   /mnt/tank/jail
tank/jail/.warden-template-VirtualBox-4.3.12  /mnt/tank/jail/.warden-template-VirtualBox-4.3.12
tank/jail/.warden-template-standard  /mnt/tank/jail/.warden-template-standard
tank/jail/.warden-template-standard-9.3-x64  /mnt/tank/jail/.warden-template-standard-9.3-x64
tank/jail/plex				  /mnt/tank/jail/plex
tank/jail/plexpy				/mnt/tank/jail/plexpy
tank/master					 /mnt/tank/master
tank/nader					  /mnt/tank/nader
tank/photolibrary			   /mnt/tank/photolibrary
tank/.system					/var/db/system
tank/.system/cores			  /var/db/system/cores
tank/.system/samba4			 /var/db/system/samba4
tank/.system/syslog-f1ae6c68bbe041c7bb38cadeec088781  /var/db/system/syslog-f1ae6c68bbe041c7bb38cadeec088781
tank/.system/rrd-f1ae6c68bbe041c7bb38cadeec088781  /var/db/system/rrd-f1ae6c68bbe041c7bb38cadeec088781
tank/.system/configs-f1ae6c68bbe041c7bb38cadeec088781  /var/db/system/configs-f1ae6c68bbe041c7bb38cadeec088781

 

[depasseg]

OK - So keep in mind that "Tank" on your primary machine equates to "Tank-backup/replicate" on the backup machine. You don't have any datasets called tank/tank on primary, so that one one backup must be something old, which is why you could be getting errors. I would detach the backup-tank pool (but don't destroy it) and then import the pool.

Second, you were misinterpreting the mount -a command. It needs the dataset path, not the filesystem path. (/mnt/whatever is the filesystem path, you need to use tank/whatever and backup-tank/whatever as your dataset paths). But it appears that things are normal.

 

[nojohnny101]

thanks guys. I went ahead and detached the volume and then imported again via GUI. It made things worse. I am not getting 6 statvfs errors every 10 seconds that includes in the original one. Of course restart makes these go away until the next replication occurs.

I know it is not hurting anything, just annoying. Bug report?

 

[depasseg]

Bug report?

Yep. Check out the one I linked to above. Sounds very similar.

 

[lopr]

OK - So keep in mind that "Tank" on your primary machine equates to "Tank-backup/replicate" on the backup machine. You don't have any datasets called tank/tank on primary, so that one one backup must be something old, which is why you could be getting errors. I would detach the backup-tank pool (but don't destroy it) and then import the pool.

yeah, that's exactly what I am experiencing too. Replication of my system/jail pool results in an extra dataset inside the destination dataset that is read-only, mounted but not actually there where it should be which results in the statvfs error. ">umount /mnt/pool/backup/SDD-jails/SSD-jails" makes it stop, which is what I am doing via a cronjob after the replication timeslot which makes that logspam bearable.

@nojohnny101 did you find a solution to that problem?

 

[nojohnny101]

I did not. I ended up changing my snapshot and replication scheme and I have not had problems since:

1) I made individual snapshot schedules and replication tasks for each of my datasets. Previously I setup a snapshot for the top level dataset and set recursive (to cover all child datasets). I now have individual ones.

2) Second change I made was to create a dataset on the receiving machine (ex: tank/replicate) and have all my snapshot get replicated into tank/replicate. Previously I had replication going to the top on the receiving end (ex: tank).

This setup has been running much better with no errors.

 

[lopr]

I did not. I ended up changing my snapshot and replication scheme and I have not had problems since:

1) I made individual snapshot schedules and replication tasks for each of my datasets. Previously I setup a snapshot for the top level dataset and set recursive (to cover all child datasets). I now have individual ones.

interesting. was this pool also your system dataset pool?

point 2 seems irrelevant since I am doing that already.

 

[nojohnny101]

My system dataset is on both PUSH and PULL are located on the pool that houses my data.