TrueNAS Core + XCP-NG, poor iSCSI performance

[MisterBaz]

I have TrueNAS Core 13.0-U5.1 running on a Dell R730xd (2 x Xeon(R) CPU E5-2623 v3, 128GB of DDR4 RAM).
Using a PCIe to NVMe expansion card, I configured 4 x 1TB NVMe drives in a striped mirror (RAID10 equiv).
I have TrueNAS setup for management on a 10Gb connection, iSCSI on its own 40Gb connection, and SAMBA/NFS on its own 40Gb connection.
I have an XCP-NG hypervisor host setup on another R730. I have a Mellanox ConnectX-3 card flashed for 40GbE directly connected to TrueNAS via a Mellanox QSFP+ DAC.

I have an XCP-NG hypervisor (8.2 or whatever is latest, I can't remember). It is setup with management on a 10Gb NIC, and two 10Gb NICs setup for multipathing for iSCSI.

I have a separate spin pool. From my desktop with 10Gb NICs, I can nearly max out my connection when pulling or pushing a large ISO file.

The troubleshooting:

• I have a Windows Server 2022 VM with its storage running on the flash-based iSCSI previously mentioned. If I try and pull/push a large ISO file to the same TrueNAS share, it maxes out at ~120MB/s.
• I've run this same test with the VM's storage on an NFS share on the same flash, iSCSI on the spin pool, and NFS on the spin pool. EXACT same performance.
• When migrating the storage BACK to the flash-based iSCSI, I saw the host hit ~325MB/s.
• I created a SAMBA share off the flash storage and was able to hit 600MB/s pushing and 1GB/s pulling.

Is there something I can do to increase iSCSI performance in TrueNAS for hypervisor platforms (specifically XCP-NG)?

UPDATE: See post #5 below. I've installed a 40Gb card to my XCP-NG host and directly connected it to the 40Gb card on my TrueNAS server and somehow get worse performance?

 

[HoneyBadger]

The connection maxing out at the 120-125MB/s number strongly suggests that the data is taking a detour down a 1Gbps path rather than the 10/40Gbps autobahn you've intended for it.

I'm a little puzzled by the network topology here - perhaps I'm not envisioning it correctly and need it diagrammed, but if you run an iperf test from the Windows 2022 VM to the IP that TrueNAS is serving SMB on, does it hit 10Gbps?

 

[smcclos]

I agree and would look at the traffic on the network adapters.

Also with iSCSI, the portals should only listen to IP address that are attached to the 10/40 GB links. The 1GB links should be prevented to attaching to the LUNS.

 

[MisterBaz]

The connection maxing out at the 120-125MB/s number strongly suggests that the data is taking a detour down a 1Gbps path rather than the 10/40Gbps autobahn you've intended for it.

I'm a little puzzled by the network topology here - perhaps I'm not envisioning it correctly and need it diagrammed, but if you run an iperf test from the Windows 2022 VM to the IP that TrueNAS is serving SMB on, does it hit 10Gbps?

xcp-ng:
Intel X520-DA2 10Gb DAC to switch - iSCSI only
Intel X520-DA2 10Gb DAC to switch - xen management, vm trunk

TrueNAS Core:
Mellanox 40Gb DAC to Switch - iSCSI only
Mellanox 40Gb DAC to Switch - NFS, SAMBA
Broadcom 10GbE to Switch - management

iperf testing was a great suggestion, and I just hadn't gotten around to it yet:
iperf3 between a linux VM on xcp-ng and my Windows desktop (I have an Intel X520-DA2 NIC) hits 9Gbps
iperf3 between a linux VM on xcp-ng and TrueNAS management (Intel X520-DA2 NIC) hits 9.2Gbps
iperf3 between TrueNAS 40Gb (SAMBA) interface and Windows desktop hits 9.3Gbps
iperf3 between TrueNAS 40Gb (SAMBA) interface and linux VM on xcp-ng hits 8Gbps
iperf3 between TrueNAS 40Gb (SAMBA) interface and Windows VM on xcp-ng hits 9.8Gbps

So, network-wise, xcp-ng, truenas, and the VMs can support 10Gb networking as far as iperf testing is concerned. That being said, iSCSI performance is absolutely terrible.
An Ookla speed test performed on the Windows VM in xcp-ng could hit 800Mbps down and 350Mbps up. I have symmetric gig, so it is possible I'm seeing half the upload due to the virtual disk not being able to provide enough IOPS to support greater than 350Mbps.

iSCSI is on its own VLAN. This hypervisor and TrueNAS are the only devices on said VLAN. Everything has jumbo packets enabled (can confirm via MTU settings as well).

UPDATE: OK, now all of a sudden when I run a crystaldiskmark I'm maxing the 10Gb connections. So I guess case closed?

 

[MisterBaz]

Major update. I've installed a 40Gb ConnectX-3 flashed for 40Gb Ethernet in my XCP-NG host and connected it directly to TrueNAS's 40Gb Chelsio card using a Mellanox QSFP+ DAC. I'm somehow getting worse iSCSI performance now?

iperf can get me 30Gbps through that link:

Flash iSCSI over 40Gb link:

Spinning disk block iSCSI over 40Gb link:

How is this even possible? My spinning disk pool can match (or slightly exceed) performance of my flash pool? I should be seeing 2GB/s speeds from my flash pool. What gives?

 

[MisterBaz]

Swapped QSFP+ DACs and can now hit 37Gbps, but NVMe IOPS is still worse than my rust RAID10 array.