TrueNAS and DNS for IP address based network operations - high delay - weird issues

[Mauricio Silveira]

This is a definitely weird one.... does TrueNAS ( or FreeBSD ) rely on reverse-dns for NON-DNS network operations?
I've been having a hell of a time trying to debug NFS connection issues: Couldn't add NEW NFSv3 shares to my ESXi host... I have probably stumbled on this before and gave up...

To make things even more interesting, NFSv4 worked fine ... until reboot...

Let's get to my battleground:

Goal: Replace current NFSv41 shares with NFSv3, because I'm using a hyperconverged ESXi host with TrueNAS Core, HBA passthrough, and NFS41 mounts stopped working for some odd reason, followed by UUID madness, etc ( Maybe the DNS issue was biting me already without my awareness ) - enough information about the issue that led me to find the TrueNAS issue...

I have my internal DNS from a VM inside a NFS share, the IP is 192.168.1.33 ...

TrueNAS IP : 10.25.25.26
ESXi IP: 10.25.25.25

---

So, I decided to run basic tests: Try to connect to the ESXi host ( 10.25.25.25 ) ssh port using nc and telnet: ( tried with other hosts, same results)

root@storage01[~]# cat /etc/resolv.conf
# Generated by resolvconf
search ad.mydomain.local
nameserver 192.168.1.33

root@storage01[~]# time sh -c 'nc -zv 10.25.25.25 22'
Connection to 10.25.25.25 22 port [tcp/ssh] succeeded!
sh -c 'nc -zv 10.25.25.25 22' 0.00s user 0.00s system 88% cpu 0.004 total

root@storage01[~]# time sh -c 'echo quit | telnet 10.25.25.25 22'
Trying 10.25.25.25...
Connected to 10.25.25.25.
Escape character is '^]'.
Connection closed by foreign host.
sh -c 'echo quit | telnet 10.25.25.25 22' 0.00s user 0.01s system 0% cpu 15.116 total

On ESXi host:

[root@vmhost01:~] time sh -c 'esxcfg-nas --add --version 3 --host 10.25.25.26 --share /mnt/ALL-SSDS/tank tank'
Connecting to NAS volume: tank
Unable to connect to NAS volume tank: Unable to complete Sysinfo operation. Please see the VMkernel log file for more details.: Sysinfo error: Unable to connect to NFS serverSee VMkernel log for details.
Command exited with non-zero status 255
real 0m 30.27s
user 0m 0.00s
sys 0m 0.00s

---

THEN, i changed DNS to google's DNS: ( Skipped nc test, no need here )

root@storage01[~]# cat /etc/resolv.conf
# Generated by resolvconf
search ad.mydomain.local
nameserver 8.8.8.8

root@storage01[~]# time sh -c 'echo quit | telnet 10.25.25.25 22'
Trying 10.25.25.25...
Connected to 10.25.25.25.
Escape character is '^]'.
Connection closed by foreign host.
sh -c 'echo quit | telnet 10.25.25.25 22' 0.00s user 0.01s system 58% cpu 0.016 total

On ESXi host:

[root@vmhost01:~] time sh -c 'esxcfg-nas --add --version 3 --host 10.25.25.26 --share /mnt/ALL-SSDS/tank tank'
Connecting to NAS volume: tank
tank created and connected.
real 0m 0.12s
user 0m 0.00s
sys 0m 0.00s

---

That led to ESXi errors when trying to add NFSv3 shares...


YES, it is VERY weird:
It interferes with NFSv3 when ADDING new shares to the ESXi host, but is able to recover after the TrueNAS vm is UP
It works with NFSv4 when ADDING new shares to the ESXi host, but is unable to recover after the TrueNAS vm is UP ( missing connection, wrong uuid, invalid VMs, etc )

Now I know I had this very same issue in another test environment before, but I'm not sure about the details... I gave up on that setup

Well... any hints?

Either I'm very stressed out and I am missing some basic stuff, or what!?


Current test:
I've moved all shares to NFSv3, now I'll set my DNS server back to 192.168.1.33 and reboot to check if all Shares will come to life once the TrueNAS VM is up

 

[Mauricio Silveira]

Update:
Adding the ESXi host entry to "Host Name Database" under Network/Global Configuration (/etc/hosts) solves the issue with THAT host
10.254.254.254 vmhost01.ad.livreti.com.br vmhost01

And I still struggle to understand...

Rebooting with this setup to see what will happen

 

[Mauricio Silveira]

This almos led me to believe it was NFS-only related to reverse lookup https://www.truenas.com/community/threads/nfs-mount-times-out.7270/

But my tests with telnet on port 22 shows it's not the case