Great write performance, terrible read performance with 10gbe and NFS

[EncrustedPole]

Hello,
I am running TrueNAS core on ESXI to evaluate it, and creating a NFS mount. I am testing performance with a computer running Debian that is running on the same host.
My ZFS pool consists of 1x Samsung 970 Evo SSD. Running default settings on TrueNAS core, except for adding tso and lro to nic options, and setting NFS to v4

Truenas specs are:
4 dedicated cores of Xeon E5-2667v3
32gb dedicated ram, 2133mhz ddr4
NIC is VMXNET3

Debian is:
6 dedicated cores of Xeon E5-2667v3
12gb dedicated ram, 2133mhz ddr4
NIC is VMXNET3
NFS is mounted with TrueNAS:/mnt/test/media /mnt/disk2 nfs _netdev,nofail,x-systemd.device-timeout=5,rsize=131072,wsize=131072 0 0

Currently, my write speed has been maxing at ~4Gb/s, but my read speed has only been at ~1.25Gb/s. iperf connection between the 2 is ~15Gb/s from debian to TrueNAS, and 7.8Gb/s from TrueNAS to debian.
Any ideas of settings to try? i have been crawling things and changing settings, but the only that have helped are enabling tso and lro on TrueNAS.

 

[EncrustedPole]

Hello,
I am running TrueNAS core on ESXI to evaluate it, and creating a NFS mount. I am testing performance with a computer running Debian that is running on the same host.
My ZFS pool consists of 1x Samsung 970 Evo SSD. Running default settings on TrueNAS core, except for adding tso and lro to nic options, and setting NFS to v4

Truenas specs are:
4 dedicated cores of Xeon E5-2667v3
32gb dedicated ram, 2133mhz ddr4
NIC is VMXNET3

Debian is:
6 dedicated cores of Xeon E5-2667v3
12gb dedicated ram, 2133mhz ddr4
NIC is VMXNET3
NFS is mounted with TrueNAS:/mnt/test/media /mnt/disk2 nfs _netdev,nofail,x-systemd.device-timeout=5,rsize=131072,wsize=131072 0 0

Currently, my write speed has been maxing at ~4Gb/s, but my read speed has only been at ~1.25Gb/s. iperf connection between the 2 is ~15Gb/s from debian to TrueNAS, and 7.8Gb/s from TrueNAS to debian.
Any ideas of settings to try? i have been crawling things and changing settings, but the only that have helped are enabling tso and lro on TrueNAS.

Forgot to mention that my NICs on the host are 10gb, but based on the iperf speeds and that nothing needs to cross I didn't think this was relevant initially.