(Solved - Partially) NFS Shares timeout on Ubuntu

[Bryan Lee]

Despite several days of searching various forums and trying various fixes, I can't get my Ubuntu desktop to mount a NFS share off my FreeNAS.

FreeNAS-9.10.2-U1
Ubuntu 16.04

Both systems can ping the hostname, FQDN of themselves and the other machine. Both systems can do a reverse DNS lookup of themselves and the other machine.

For the NFS Server
Server UDP NFS-clients
Bind IP: 192.168.76.4 (my freenas box, *not* any of the jails)
Enable NFSv4: checked
Log mountd requests: checked
Log rpc.statd and rpc.lockd: checked

For the NFS Share
/mnt/chamber/PlexMedia
Authorized IP: 192.168.76.2 (my desktop)
All Directories
maproot User: root (was just testing, I know this would be set to something else for security)
Maproot Group: wheel
Security: none

On my ubuntu box:

$ sudo mount -v -t nfs -o tcp 192.168.76.4:/mnt/chamber/PlexMedia plexmedia/
mount.nfs: timeout set for Tue Jan 31 15:14:15 2017
mount.nfs: trying text-based options 'tcp,vers=4,addr=192.168.76.4,clientaddr=192.168.76.2'
mount.nfs: mount(2): Connection timed out
mount.nfs: Connection timed out


Running tcpdump on my ubuntu box, I see NFS packets being sent to the server, but no replies.

$ sudo tcpdump -n port 2049
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:21:23.472202 IP 192.168.76.2.775 > 192.168.76.4.2049: Flags [ S], seq 615082167, win 29200, options [mss 1460,sackOK,TS val 123806653 ecr 0,nop,wscale 10], length 0
15:21:24.469517 IP 192.168.76.2.775 > 192.168.76.4.2049: Flags [ S], seq 615082167, win 29200, options [mss 1460,sackOK,TS val 123806903 ecr 0,nop,wscale 10], length 0
15:21:26.473515 IP 192.168.76.2.775 > 192.168.76.4.2049: Flags [ S], seq 615082167, win 29200, options [mss 1460,sackOK,TS val 123807404 ecr 0,nop,wscale 10], length 0
15:21:30.481504 IP 192.168.76.2.775 > 192.168.76.4.2049: Flags [ S], seq 615082167, win 29200, options [mss 1460,sackOK,TS val 123808406 ecr 0,nop,wscale 10], length 0



When I search /var/log/messages for nfs messages I see none when I try to connect, but I do see this repeated:

Jan 30 23:39:39 hogwarts nfsd: can't register svc name


On my ubuntu box I run rpcinfo:

$ rpcinfo -s 192.168.76.4
program version(s) netid(s) service owner
100000 2,3,4 local,udp6,tcp6,udp,tcp portmapper superuser
100005 3,1 tcp,udp,tcp6,udp6 mountd superuser
100003 3,2 tcp,udp nfs superuser
100024 1 tcp,udp,tcp6,udp6 status superuser
100021 4,3,1,0 tcp,udp,tcp6,udp6 nlockmgr superuser



Any ideas? I'm running out of them.

 

[dlavigne]

Does http://stackoverflow.com/questions/26414957/freenas-cannot-mount-nfs fix it?

 

[Bryan Lee]

Does http://stackoverflow.com/questions/26414957/freenas-cannot-mount-nfs fix it?

I already have that field filled. The client is in that field, and from my understanding that field simply gets put into the FreeNAS /etc/hosts file, which I have verified it is:


192.168.76.2 gallifrey.domain.org gallifrey
127.0.0.1 hogwarts.domain.org hogwarts
::1 hogwarts.domain.org hogwarts



Yes, I may be a bit of a geek, don't judge me for my hostnames! (replaced actual domain name with domain.org)

 

[Bryan Lee]

Although interestingly, it now appears that reverse DNS is not working on the FreeNAS, I could have sword it was working previously....

Code:

$ dig -x 192.168.72.2

; <<>> DiG 9.10.4-P4 <<>> -x 192.168.72.2
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 38094
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;2.72.168.192.in-addr.arpa.	IN	PTR

;; Query time: 19 msec
;; SERVER: 192.168.76.1#53(192.168.76.1)
;; WHEN: Wed Feb 01 10:54:41 EST 2017
;; MSG SIZE  rcvd: 54

I could have sworn this was working before. And from all the reading I do, if reverse DNS isn't working, and you can't get the reverse-DNS entries in your DNS server, that you can add them to the Network Settings-> Global Configuration -> Host Name Database. But I've already verified that the entry is in there.

 

[dlavigne]

Anything in /var/log/messages on the FreeNAS side (or equivalent log on the Ubuntu side) during the attempted mount?

 

[Bryan Lee]

Anything in /var/log/messages on the FreeNAS side (or equivalent log on the Ubuntu side) during the attempted mount?

Absolutely nothing in /var/log/messages on the FreeNAS or in /var/log/syslog on the Ubuntu side.

Only the timeout error listed in my first post when I try to mount with verbose output.

 

[Bryan Lee]

I have corrected the reverse DNS lookup entries. It took some reconfiguration of my DD-WRT router. Reverse DNS lookups of both the desktop and FreeNAS server returns the FQDN, for both the FreeNAS and the desktop.

The problem still persists. Timeouts when mounting.

 

[Bryan Lee]

Okay, I run tcpdump on my Ubuntu desktop, I can see the SYN packets leaving the interface.

But I run tcpdump on the FreeNAS box, and I can't see the SYN packets arrive.

But this is only for NFS. All other protocols (SSH and HTTP most notably) work fine.

Both systems are plugged into the same switch and are on the same subnet. No routers or firewalls between them.

 

[Bryan Lee]

Checked the arp tables on both machines. They both have the proper IP address -> MAC Address resolution.

This one seriously has me confused. :-\

 

[Bryan Lee]

Okay, possibly an issue on my Ubuntu box, although I have no idea what.

I tried this command on my Ubuntu box while running tcpdump on FreeNAS

Code:

$ telnet hogwarts.domain.org 2049
THISISATEST

^]
telnet> QUIT

And then looking at the pcap I see the SYN, SYN ACK, ACK and payload packet containing THISISATEST

 

[Bryan Lee]

AHA... Success, I know the difference, but I don't know why it makes a difference... perhaps someone might be able to explain?

I noticed that when I was mounting the source port was a low numbered reserved port.

Code:

$ sudo mount -t nfs4 -o proto=tcp,port=2049 hogwarts.domain.org:/mnt/chamber/PlexMedia /home/user/plexmedia/

tcpdump would give me this output

Code:

$ sudo tcpdump port 2049
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
21:00:55.605448 IP gallifrey.domain.org.981 > hogwarts.domain.org.nfs: Flags [ S], seq 17195482, win 29200, options [mss 1460,sackOK,TS val 172099686 ecr 0,nop,wscale 10], length 0

But when I would telnet to port 2049, it would use a high numbered un-reserved port.

Code:

$ sudo tcpdump  port 2049
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
21:54:02.445604 IP gallifrey.domain.org.48044 > hogwarts.domain.org.nfs: Flags [ S], seq 2045915505, win 29200, options [mss 1460,sackOK,TS val 172896397 ecr 0,nop,wscale 10], length 0
21:54:02.445763 IP hogwarts.domain.org.nfs > gallifrey.domain.org.48044: Flags [ S.], seq 1067092929, ack 2045915506, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 540981989 ecr 172896397], length 0
21:54:02.445791 IP gallifrey.domain.org.48044 > hogwarts.domain.org.nfs: Flags [.], ack 1, win 29, options [nop,nop,TS val 172896397 ecr 540981989], length 0
21:54:05.873421 IP gallifrey.domain.org.48044 > hogwarts.domain.org.nfs: Flags [P.], seq 1:14, ack 1, win 29, options [nop,nop,TS val 172897253 ecr 540981989], length 13

Then I did some searched and found the noresvport option

Code:

$ sudo mount -t nfs4 -o noresvport,proto=tcp,port=2049 hogwarts.domain.org:/mnt/chamber/PlexMedia /home/user/plexmedia/

It instantly mounted!

So I am unsure if it is Ubuntu that is failing when it sends out low numbered source port NFS packets, or if it is FreeNAS that somehow doesn't like them. I have read some documentation that suggests using high numbered source ports is less secure and that servers need to be configured to allow them. If anyone has some insight, I'd love to hear it.

 

[Bryan Lee]

So I am unsure if it is Ubuntu that is failing when it sends out low numbered source port NFS packets, or if it is FreeNAS that somehow doesn't like them. I have read some documentation that suggests using high numbered source ports is less secure and that servers need to be configured to allow them. If anyone has some insight, I'd love to hear it.

After doing a little more research I've learned a bit more about the security concerns. A NFS server generally wants to see this clients use a source port that is in the reserved range of 1-1024. This is because it is assumed that since only the root user of the client can use those ports, that we can "trust" that connection. But high numbered source ports, any user of the system can initiate that connection. And we "trust" users less than we trust root.

Although apparently this is mitigated in NFSv4 when you use Security of krb5, krb5i, or krb5p. when you don't use security or use Security of sys (Sharing->NFS->Edit Share->Advanced Mode), it uses uid/gid mappings the same was NFSv3 did. But apparenty with NVSv4 and krb5 there is more authentication that goes on. I'm still researching this part of if, so I hope to have more info later.

 

[Bryan Lee]

Plot thickens.

To expound, using NFSv4 (without any security), when I did a directory listing, it was listing files owned by very high UID/GID.

So, as a test, on FreeNAS I turned off the enable NFSv4 and ran the exact same mount command, but I specified nfs instead of nfs4.

When I did that the FreeNAS server complained of:

Code:

mountd[7161]: mount request from 192.168.76.2 from unprivileged port

When I removed the noresvport option, now the mount would not occur and it would timeout.

So now, my options are..

• Try to figure out how to get FreeNAS to allow NFSv3 mounts from unprivileged ports
• Try to figure out why mounts from unprivledged ports are timing out (for both NFSv3 and NFSv4)
• Play around with NFSv4 using security=sys, which I hear mimics the security paradigm of NFSv3

The battle rages on....

 

[Bryan Lee]

The plot thickens... and curdles a little bit.

Okay, when using NFSv4, sec=sys, and noresvport, I was getting weird uid/gid mapping isues. Basically on the client every file looked like it was owned by nobody.

After doing hours of google searches and experimenting, some people seem to be claiming that sec=sys is just a little broken, be it by design, or bugs. But everyone seemed to say that when using Kerberos authentication, that idmapd correctly would map gid/uid.

So I hopped onto FreeNAS -> Directory Service -> Kerberos Realms and setup a brand spanking new realm for my local network.

I then installed the proper Kerberos packages for my Ubuntu desktop. I then tried to mount the share.

The problem is that sec=krb5 appears to be incompatible with my before mentioned noresvport workaround. When mounting it would complain about an invalid option. Upon removing noresvport from the mount command, I would return to the mount timeout issue I found previously.

I think I need to install tcpdump on my FreeNAS to see if the packets from my Ubuntu box when using a reserved port are making it to the FreeNAS, and see if it sends anything back. Because looking at tcpdump on my desktop, it's sending NFS packets out, but receiving none back when using a reserved port.