Stale file handle on NFS child datasets

[Elegant]

Hello,

We currently have a dataset structure similar to the below:
vol1/parent/child1 vol1/parent/child2

As each dataset is its own filesystem, we must export each of them individually similar to the below:
/mnt/vol1/parent/child1 -maproot="root":"wheel" -sec=sys /mnt/vol1/parent/child2 -maproot="root":"wheel" -sec=sys

The issue is when we mount these child datasets, we constantly receive ls: cannot open directory '.': Stale file handle when inspecting the directory. Strangely, we do not have any issues if the root user accesses the share or any non-root user when --mapall="root":"wheel" is specified.

Only when a non-root user accesses the share do we see the stale file handle. Reads and writes still proceed as normal (confirmed in FreeNAS) but it is not longer possible to view the contents of the share using ls on the client. This persists across all linux clients in our network.

When we move these childsets to the same level of the parent (i.e. vol1/child1) the issue disappears but we are seeking to have a better structure for our datasets.

What is the proper process to ensure we do not receive a Stale file handle when mounting child datasets over NFS? Please note that remounting the shares has had no effect.

Thanks!

 

[Samuel Tai]

You should add -alldirs to the top-level volume export.

 

[Patrick M. Hausen]

Doesn't -alldirs work with a single export? Or is that still limited to a single dataset? Maybe you need -alldirs on the parent even with seperate exports to traverse the mountpoints?

 

[Samuel Tai]

Only the parent export needs to be listed with -alldirs. The child exports would be moot, and can be removed.

 

[Elegant]

Same issue if -alldirs is specified. I've tried both of the combinations suggested:

Option 1:
/mnt/vol1/parent -alldirs -maproot="root":"wheel" -sec=sys

Option 2:
/mnt/vol1/parent -alldirs -maproot="root":"wheel" -sec=sys /mnt/vol1/parent/child1 -maproot="root":"wheel" -sec=sys /mnt/vol1/parent/child2 -maproot="root":"wheel" -sec=sys

Proceeding with option 1 still produces a stale file handle but the root can no longer view the contents receiving a ls: reading directory '.': Input/output error. This is due to NFS' inability to traverse nested datasets (similar topic here). Subdirectories work fine though.

Option 2 ends with the same result, stale file handle except when accessed with the root. Unfortunately, -alldirs seems to have no impact.

Does FreeNAS not provide the right handle for non-root users?

 

[Samuel Tai]

You'll have to change the nested datasets to a single dataset with subdirectories.

 

[Elegant]

Hoping to avoid that in order to maintain hierachy and to allow for quotas on each child dataset. Is there an underlying reason NFS does not support this configuration?

 

[Patrick M. Hausen]

I am puzzled myself. I know about the "one export per filesystem" restriction but my NFS knowledge is from the UFS era so I am not of much help here. If there is a fundamental problem with the nested setup, I'd export only "leaf" datasets and not the parent. Refactor the setup to match if necessary. Why do you need to export the parent dataset anyway?

I don't like files in non-leaf folders and services bound to non-leaf datasets in general, but that might be just me ;)

 

[Elegant]

We don't need the parent exported, I removed the parent export from the OP to avoid the confusion. Thanks!

 

[Samuel Tai]

Can you display the permissions for both child directories, and for the parent?

 

[Samuel Tai]

Does the stale file handle message persist when the clients unmount then remount the NFS child folders?

 

[Elegant]

Permissions of parent dataset:

Code:

# file: parent
# owner: administrator
# group: my admins
            owner@:rwxpDdaARWcCos:fd-----:allow
            group@:rwxpDdaARWcCos:fd-----:allow
         everyone@:--------------:fd-----:allow

Permissions of child1 dataset:

Code:

# file: child1
# owner: administrator
# group: my admins
  group:my admins:rwxpDdaARWcCos:fd-----:allow
      user:child1:rwxpDdaARWc--s:fd-----:allow
         everyone@:--------------:fd-----:allow

Permissions of child2 dataset:

Code:

# file: child2
# owner: administrator
# group: my admins
            owner@:rwxpDdaARWcCos:fd-----:allow
            group@:rwxpDdaARWcCos:fd-----:allow
         everyone@:rwxpDdaARWc--s:fd-----:allow
         everyone@:--------------:fd-----:allow

All stale file handles persist for both child datasets regardless of any unmounting/remounting. It should be noted though that the shares are mounted by the root user. Users should then be able to access them based on the ACLs.

 

[Samuel Tai]

What mount options do you use on the client system?

 

[Elegant]

Nothing, in some cases I may specify noatime. Issue appears with both though. Thanks!

 

[Samuel Tai]

What happens if you use noac?

 

[Elegant]

Same results, stale file handle for everyone but root.

 

[KrisBee]

@Elegant To be clear, which version of FreeNAS are you using? Is the FreeNAS the server configured for NFSv3 or NFSv4 (your exports look life NFSv3)? Which linux clients are in use (distro and version)?

 

[Elegant]

Using FreeNAS-11.3-U3.1 (latest at time of posting). FreeNAS is configured to use NFSv4 only. NFS settings can be seen below:

Currently mounting these shares on Alpine 3.11, Debian 10 and Fedora 32. All are receiving a stale file handle when mounted by root and accessed by a non-root user. Again, users with the appropriate ACLs are able to write and delete files from the share (confirmed in FreeNAS) but viewing the share's contents with ls is not possible.

Issue is restricted to child datasets share over NFS. Parent datasets are accessible by non-root users without issue.

Thanks!

 

[KrisBee]

I'd consider updating to FreeNAS 11.3-U3.2 as fixes make reference to problem with "ACL User and Group ownership" etc. But do note the known issues posted at https://www.ixsystems.com/blog/library/freenas-11-3-u3-2/

As you're not using kerberos, is there a particular reason you did not choose to select "NFSv3 ownership model with NFSv4"?

 

[Elegant]

We upgraded FreeNAS to 11.3-U3.2 and enabled "NFSv3 ownership model with NFSv4" but the stale file handle remains. We're beginning to wonder if this is a bug instead of configuration issue.