Child datasets of readonly datasets are not mounted

[mpfusion]

Hi,

I have the problem that child datasets of readonly datasets are not mounted and cannot be mounted:

cannot mount '/mnt/tank/foo/bar': failed to create mountpoint

The freenas box receives replication streams from another freenas box, which works fine. I also understand why it's set readonly - to prevent any deviation of the backup from the source.

But the data is inaccessible. This box hosts backups and every week an rsync script kicks off to copy all data off the freenas box for offsite storage. This works only if the datasets are mounted. The readonly datasets received via zfs replication are not backed up for offsite storage because they can't be mounted.

Is there a way to mount readonly datasets into readonly datasets? I'd like to keep the readonly flag, since I'm not intending to write the data, it's just being picked up by rsync. What's the best way to solve this?

Related bug report (behaves correctly): #15837

FreeNAS-11.1-U4
Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz
163693MB

 

[Johnny Fartpants]

reboot the server on which the datasets are not mounted and boom!

 

[mpfusion]

reboot the server on which the datasets are not mounted and boom!

I can't reboot right now, it's in use. Next week I could do that. However, it has been rebooted yesterday. So the problem doesn't seem to be resolved after a reboot. Why would a reboot help if zfs mount -a can't mount the datasets (see OP)?

I can replicate the issue on two different boxes.

 

[Johnny Fartpants]

Why would a reboot help if zfs mount -a can't mount the datasets

I often find that datasets are not mountable on my replication targets and a reboot of the target always sorts it.

I don't think it has anything to do with the datasets being readonly.

 

[Johnny Fartpants]

I can't reboot right now, it's in use

When you say in use do you mean its receiving a replication stream?

 

[mpfusion]

It is or might be sending or receiving replication streams, yes. But I meant, the filer is in use. Other people use it, I can't turn it off right now. I'll report back on Monday. Monday I can reboot it. Maybe I can reboot the 2nd one (I have two filers with the same issue) over the weekend. In any case I'll report if that fixed/didn't fix the issue.

 

[Johnny Fartpants]

ok. Just to be clear you want to reboot the replica target not the primary.

 

[mpfusion]

ok. Just to be clear you want to reboot the replica target not the primary.

I just rebootet the replica target, same result. All datasets are loaded except the children of readonly datasets.

 

[Johnny Fartpants]

When you say 'loaded' except the children of read-only datasets do you mean you can't cd to them?

 

[mpfusion]

When you say 'loaded' except the children of read-only datasets do you mean you can't cd to them?

Sorry, typo. I wanted to say “mounted” not loaded.

Exaxtly, I can't “cd” into because they're empty (well, not mounted).

 

[Johnny Fartpants]

Im really not sure then buddy.

I've got a very similar setup on many boxes my end running versions from 9.10 to 11.1-U6 and non of them have this issue.

Sorry I couldn't help.

 

[mpfusion]

No problem. I have two boxes showing the same problem. This reduces the chance of having one “weird” configuration. I might check if I can replicate this in a VM with a minimal setup.

Since you have a similar setup, my questions: On the receiving side (assuming you're using the built-in ZFS replication), do the received datasets have the read-only property set? If they do, are they all mounted?

 

[Johnny Fartpants]

No problem. I have two boxes showing the same problem. This reduces the chance of having one “weird” configuration. I might check if I can replicate this in a VM with a minimal setup.

Since you have a similar setup, my questions: On the receiving side (assuming you're using the built-in ZFS replication), do the received datasets have the read-only property set? If they do, are they all mounted?

Yes Im using built-in ZFS replication and yes all received datasets are read-only and should be by default and they are all mountable and traversable and ls able.

 

[Johnny Fartpants]

What does your zfs get on the dataset say about it being mounted?

 

[mpfusion]

Property “mounted” is no, which is correct. I looked through all datasets and found an inconsistency. *Some* can indeed be mounted and they're also read-only. So in general it means that read-only datasets can indeed be mounted. But not always. I then compared the zfs properties of one directory which could be mounted with one that couldn't. Result (apart from the usual differences e.g. creation date, used…):

PROPERTY VALUE SOURCE_good SOURCE_bad
quota none received local
readonly on inherited from … local

I don't know why the readonly propery has a different source on the different datasets. And I also don't know if that even makes a difference, since the value is both the same (on).

Then I compared the “quota” property and the source for all datasets that have been received via zfs receive and found that the source is “local”, “default” or “received”. This I also don't understand. Shouldn't the value always be “received” for datasets that have been “zfs receive”'ed?

I'll run some more experiments the next days. Maybe I'll delete and re-receive some of the datasets and check the values ones more.

 

[mpfusion]

Okay, I destroyed the “misbehaving” datasets, waited for the replication to kick in again. After replication the “quota” property was indeed set to “received”, the “readonly” property to “inherited” and the “canmount” to “default”.

33 datasets are sucessfully replicated and mounted but four of them cannot be mounted. I then rebooted, but same story:

cannot mount '/mnt/tank/foo/bar': failed to create mountpoint

What is special about those datasets that they can't be mounted? Is there a way to find out *why* zfs can't create the mountpoint?