Poor(?) NFS performance with Proxmox/Truenas

[wille61]

Hello!

I have a three node Proxmox cluster connected to a TrueNAS server where we store all of our VM:s. I've had some delay/lag when managing the VM:s so i decided to do some bench marking.


I thought i had pretty decent hardware for this application, but maybe this is all i can expect?


Storage specs
TrueNAS-12.0-U8
Supermicro 1114S-WN10RT
128GB DDR4 3200MHz
Epyc 7282 16c32t 2,8GHz
4x Samsung PM983 1,92TB NVMe U.2 (Striped mirror)
2x Intel Optane P4801X 100GB NVMe U.2 (Mirror SLOG)
Intel X550-T2 10G Base-T NIC


All PVE-nodes have Solarflare 10GbE SFP+ NIC's, iPerf output between storage and nodes is 10Gbit/s~ in all tests i've performed, AVG 0.8-1ms latency.


Pool/NFS settings is default, sync on.

Performance test using FIO:

Debian VM BS=128k write: IOPS=951, BW=119MiB/s Debian VM BS=64k write: IOPS=1234, BW=77.2MiB/s Debian VM BS=4k write: IOPS=1831, BW=7324KiB/s
PVE Node BS=128k write: IOPS=1471, BW=184MiB/s PVE Node BS=64k write: IOPS=1741, BW=109MiB/s PVE Node BS=4k write: IOPS=2837, BW=11.1MiB/s
TrueNAS Pool BS=128k write: IOPS=6778, BW=847MiB/s TrueNAS Pool BS=64k write: IOPS=15.2k, BW=949MiB/s TrueNAS Pool BS=4k write: IOPS=15.2k, BW=949MiB/s

I'm planning to update Truenas to 13.x in the upcoming weeks, it's a production environment and i can only take it down for maintenance between 12:00AM - 04:00AM.

Other than that I'm out of ideas, except turning of sync which i don't want to do.

Any suggestions?

 

[Nick2253]

Turning off sync temporarily is a useful diagnostic tool to rule out disk/controller/network issues. If you get similar performance with sync off, then you know that your bottle neck is likely the network. If you get much improved performance, then it points to a problem with something in your pool setup.

 

[wille61]

Turning off sync increased the performance 3x~, so the question is where to look now..

When i first built the storage i used regular consumer SSD's as slogs (Samsung PM9A3), of course that didn't work too well. I was hoping for a big increase in performance when i swaped them out for P4801x's but there wasn't really that big of difference unfortunately..

 

[NugentS]

Well the P4801x's are great SLOG's - better than the ones I use which work really well. So that really shouldn't be the issue.
Can you please:
1. Post how the pool and datsets are configured
2. Post the specific FIO test(s) you ran. Those IOPS look very low for mirrored pairs of U.2
3. Out of interest how are you booting the server? USB, SataDOM, NVMe or more U.2 drives or even SATA - mostly cos I am curious
4. Not that this is relavent - but the PM983 is a read intensive drive with a DWPD of 1.3. So not ideally suited for the work load you are using them for. I would have thought this would only effect endurance though and its still DWPD of 1.3
5. Post zpool status in codeblocks please

PM843 Features
PM983 offers tremendous performance for Read-Intensive data centres by applying PCIe Gen 3, achieving 540K IOPS in Random Read and 3000 MB/s in sequential read speed. Using impressively low power in small form factors (2.5”, M.2), PM983 delivers an efficient SSD solution for mixed data workloads. The Samsung SSD PM983 delivers wide bandwidth of up to 3,000MB/s for sequential read speed and up to 1,400MB/s for sequential write speed under up to 8.0W power. With the help of Toggle 3.0 NAND Flash interface, the Samsung SSD PM983 delivers random performance of up to 480KIOPS for random 4KB read and up to 42KIOPS for random 128KB write in the sustained state. By combining the enhanced reliability Samsung NAND Flash memory silicon with NAND Flash management technologies, the Samsung SSD PM983 delivers the extended endurance of up to 1.3 drive writes per day over 3 years, which is suitable for enterprise applications, in M.2 form factor lineups: 960GB, 1.92TB and 3.84TB.

 

[HoneyBadger]

Latency on NVMe devices can be very sensitive to things like power-saving measures. I know the Ryzen series has stability issues with C6 sleep states, but I've also seen significant performance deltas on PCIe devices as they rapidly toggle in and out of a low-power state. Look for both CPU based (C-state, Cool N Quiet) and PCIe power savings (ASPM) and disable them all.

If you're able, kicking an Optane device out of the log vdev temporarily and running the SLOG benchmark using disktest (it's destructive, so you can't use it on an active device) would tell you if you're bottlenecking there somehow:

diskinfo -wS /dev/XXX

I realize that your situation makes it difficult to do this sort of testing during the daylight hours (although I don't know your timezone) - but as disabling sync tripled the results it seems as if that's where we want to start looking.