SOLVED NFS 4.1 datastore mounted as read-only

[egnbnd]

Hello,
We have FreeNAS-11.1-U5 with VMWare vCenter 6.5 / ESXi 6.5.0. Currently we are using NFS 3 but we would like to move to NFS 4.1.
NFSv4 is enabled (Services -> NFS) but when creating new datastore in vSphere it is attached in read-only mode. While looking at vmkernel log:

Code:

2018-07-31T10:50:09.058Z cpu0:34830 opID=f6b6ea7a)NFS41: NFS41_VSIMountSet:402: Mount server: my-server.data, port: 2049, path: /mnt/rz2.1/zzz/sync, label: ds-nfs4, security: 1 user: , options: <none>
2018-07-31T10:50:09.058Z cpu0:34830 opID=f6b6ea7a)StorageApdHandler: 982: APD Handle  Created with lock[StorageApd-0x4305abbc6140]
2018-07-31T10:50:09.059Z cpu2:33436)NFS41: NFS41ProcessClusterProbeResult:3865: Reclaiming state, cluster 0x4305abbc73c0 [0]
2018-07-31T10:50:09.061Z cpu0:34830 opID=f6b6ea7a)NFS41: NFS41FSCompleteMount:3582: Lease time: 120
2018-07-31T10:50:09.061Z cpu0:34830 opID=f6b6ea7a)NFS41: NFS41FSCompleteMount:3583: Max read xfer size: 0x20000
2018-07-31T10:50:09.061Z cpu0:34830 opID=f6b6ea7a)NFS41: NFS41FSCompleteMount:3584: Max write xfer size: 0x20000
2018-07-31T10:50:09.061Z cpu0:34830 opID=f6b6ea7a)NFS41: NFS41FSCompleteMount:3585: Max file size: 0x800000000000
2018-07-31T10:50:09.061Z cpu0:34830 opID=f6b6ea7a)NFS41: NFS41FSCompleteMount:3586: Max file name: 255
2018-07-31T10:50:09.061Z cpu0:34830 opID=f6b6ea7a)WARNING: NFS41: NFS41FSCompleteMount:3591: The max file name size (255) of file system is larger than that of FSS (128)
2018-07-31T10:50:09.061Z cpu0:34830 opID=f6b6ea7a)WARNING: NFS41: NFS41FSCompleteMount:3601: RECLAIM_COMPLETE FS failed: Not supported; forcing read-only operation
2018-07-31T10:50:09.061Z cpu0:34830 opID=f6b6ea7a)NFS41: NFS41FSAPDNotify:5651: Restored connection to the server my-server.data mount point ds-nfs4, mounted as 224813fb-f26022ab-0000-000000000000 ("/mnt/rz2.1/zzz/sync")
2018-07-31T10:50:09.061Z cpu0:34830 opID=f6b6ea7a)NFS41: NFS41_VSIMountSet:414: ds-nfs4 mounted successfully

I’ve found VMWare thread related to the issue: https://communities.vmware.com/thread/517788
And there even appears to be some fixes on FreeBSD: https://www.mail-archive.com/freebsd-stable@freebsd.org/msg135292.html

There is also recent revision related to ReclaimComplete here:

Code:

This patch along with r332790, r334492 and r336357 allow ESXi 6.7 NFSv4.1 mounts
work ok. ESX 6.5 NFSv4.1 mounts do not work well, due to what I believe are
violations of RFC-5661 and should not be used.

Did anyone encounter the same issue? Does anyone has ESXi 6.5/6.7 working fine with NFS 4.1 on FreeNAS?
Thanks!

 

[kdragon75]

Do you still have the same location mounted or shared as NFS3 ANYWHERE? Also why not move to iSCSI?

 

[kdragon75]

I just dug through the links. Its not what I thought it was. I still don't understand why people shy away from iSCSI.

 

[egnbnd]

Do you still have the same location mounted or shared as NFS3 ANYWHERE?

No, we don't have the same location mounted or shared as NFS3.

why not move to iSCSI?

That's kind of architectural decision and it was expected that switching to NFS 4.1 would be much smaller chunk of work.

 

[kdragon75]

Yeah moving to iscsi would involve setting up the new datastore and using storage vMotion to move it all over. Betwean powerCLI, host profiles, and distributed switches there are lots of low touch ways to configure/deploy iSCSI on large clusters.

 

[egnbnd]

We've just tested ESXi 6.7 with NFS 4.1 and it still mounts in read-only mode with the same message:

Code:

WARNING: NFS41: NFS41FSCompleteMount:3916: RECLAIM_COMPLETE FS failed: Not supported; forcing read-only operation

 

[xnaron]

Did you ever figure out how to get it to mount r/w?

 

[drros]

Did you ever figure out how to get it to mount r/w?

As I understand FreeNAS have NFS version 4, and not a 4.1 which is required by ESXI\Vcenter. I believe i read somewhere that 4.1 will be implemented in future releases of FN.

 

[xnaron]

As I understand FreeNAS have NFS version 4, and not a 4.1 which is required by ESXI\Vcenter. I believe i read somewhere that 4.1 will be implemented in future releases of FN.

Good to know. I'll see if 11.2 supports 4.1.

Thanks

 

[egnbnd]

Did you ever figure out how to get it to mount r/w?

Unfortunately, no.
Some time ago I've made tests with 11.2 Beta 2 and it still not works. Not tested with Beta 3 yet, but don't see relevant issue fixes in RN.

 

[JoeAtWork]

FreeNAS 11.2 u1 and it still fails, it is not clear how to setup NFS v4.1 on FreeNAS for VMware ESXi 6.x as far as permissions on the file system and any extra settings. AUTH_SYS means we do not have to have FreeNAS tied to Active directory to start a NFS connection

It sounds like there was some work done by the BSD guys this summer :
https://lists.freebsd.org/pipermail/freebsd-stable/2018-July/089284.html

I am doing testing on ESXi 6.0 and 6.7 and have no issues with FreeNAS NFS v3 mounts

esxcli storage nfs41 add -H 192.24.85.74 -a AUTH_SYS -s /mnt/Share/vmware6/nfs41 -v datastoreNFS4test

 

[egnbnd]

Hi, revision 5fb1d550 which contains Freebsd nfs fixes was merged to main branch and it is included in 11.2 U2.
We updated our test freenas and NFS4 worked just fine with our ESXi 6.5!

 

[Hakkers]

Does not seem to be working on 11.2 U4.1 and ESXi 6.7
Can write to the datastore, but can't delete the data just written.

 

[Poached_Eggs]

Does not seem to be working on 11.2 U4.1 and ESXi 6.7
Can write to the datastore, but can't delete the data just written.

Where I've been pulling my hair out this last week is with esxi 6.7 cluster with freenas 11.2u4.1 - when mounting NFS, instead of a single shared datastore - I get datastore , datastore (1) and even sometimes datastore (2) - 3 total identical servers and can't figure ryme or reason on why. Seems to be random - it's not mounting to ip/fqdn , upper/lower case - mounts are identical.

I did run across something strange - that was a child object under the NFS mount, when also shared reset the parent folder permissions....

 

[Hakkers]

Someone with a working set-up (nfs/esxi6.7) willing to share some details?

 

[usmclvsop]

Define 'working'. I have it working with some caveats.
I am running ESXi 6.7.0 Update 2 without vCenter & client version 1.33.3. FreeNAS is running 11.2-U4.1 and the NFS service has enable NFSv4 checked.
I created a user and group named esxi in FreeNAS which then was set as the owner and group of SSDPool using Unix permissions.

From the Freenas shell

Code:

#cat /etc/exports
V4: / -sec=sys
/mnt/SSDPool/nfs_vmfs1 -alldirs -maproot="root":"wheel" -network 192.168.1.0/24

I apologize for not saving my sources, but there was a post about how if you don't set maproot to root that root will get set to UID -412345678 (or whatever it actually is). I'm not sure if this even matters since I'm mounting the share as the created esxi user.

In ESXi then clicking new datastore, mount nfs datastore, share set to /mnt/SSDPool/nfs_vmfs1 and nfs version 4 selected, then username of exsi with the password I set that user to in FreeNAS. It shows up as datastore of type NFS41 and you can create VMs on it.

The biggest caveat is that the NFS API bug has not been patched for the embedded host client. If I try to use the datastore browser on the nfs share it throws: An error occurred, please try again
So you cannot move VMs to it using the GUI. Secondly, if you boot ESXi while the NFS share is unavailable the datastore gets pretty borked. Even after it was up, refreshing the datasource in the gui did nothing. I had to delete the datasource, unregister the orphaned VMs, and the re-add the NFS datasource. After that I was able to re-register all the existing VMs on it. I did not test if rebooting ESXi after the NFS share was back up would fix the borked datasource and VMs.

TLDR: If you can write to the datastore but not delete that is a unpatched bug only affecting the web client GUI, VMs can be run with some limitations

 

[Hakkers]

Right, for it to be usable (for me) i'll have to wait until this NFS bug is patched.
Meanwhile sadly also found that i need vCenter for NFS multipath.

Thanks for sharing.

 

[PeteKowalsky]

I think I've hit this issue as well. Even though I've mounted as NFS v4.1, RW, on VMware ESXi, 6.5.0, 13635690, I cannot delete any FOLDERS I've created. I CAN move VMs to and from (seemingly without issue storage vMotion works fine - been doing it for a while). I just cannot delete any directories. I CAN delete FILES (including .vmx / .vmdk's / etc from the HTML5 UI with no issue). FWIW, I'm on FreeNAS-11.2-U5, and using VCSA (VMware vCenter Server Appliance) 6.7.0.32000, 14070457.

Does creating an "esxi" user on FreeNAS help anything? Maybe I should just go back to NFS v3 if I don't need multipath?

Just wondering the current status of this issue with others here - Thanks!

 

[usmclvsop]

Pete, there is a bug in the API that VMWare needs to patch. Doubt there is any way to get around this by changing settings. I cannot speak to vCenter as I only have access to the free license.


Has anyone tested if 6.7 update 3 includes the NFS API patch for the embedded host client? (Easiest test: can you delete a file or folder from the datastore browser that has type NFS41?)

Trying to install it failed because of a change to the bootloader and I'd rather avoid attempting the workarounds unless I know it's going to fix my issue. Bootloader issue: https://communities.vmware.com/thread/617099?start=15&tstart=0

 

[Hakkers]

The biggest caveat is that the NFS API bug has not been patched for the embedded host client.

@usmclvsop: do you happen to have a link to this bug-report? Would be nice to be able to monitor it's progress.