TrueNAS Core Poor iSCSI Performance

[glloyd-a]

We are in the process of rolling out a new storage server and have started to do some baseline performance testing and are seeing surprisingly poor performance. In our test we have provisioned a VM within Proxmox that is using a Zvol attached to the cluster via iSCSI. The TrueNAS server is connected to a 25GbE switch with 2 links that are aggregated and the Proxmox host is connected to the same switch but with a single 25GbE interface. The TrueNAS server pool that is backing the zvol has 10 - 7.68TB NVMe SSD's and when we run fio on the storage server directly we are seeing the following:

fio --bs=128k --direct=1 --directory=/mnt/n-store/proxmox/nfs-share-01 --gtod_reduce=1 --ioengine=posixaio --iodepth=32 --group_reporting --name=randrw --numjobs=2 --ramp_time=10 --runtime=60 --rw=randrw --size=256M --time_based

READ: bw=8348MiB/s (8754MB/s), 8348MiB/s-8348MiB/s (8754MB/s-8754MB/s), io=489GiB (525GB), run=60001-60001msec
WRITE: bw=8344MiB/s (8749MB/s), 8344MiB/s-8344MiB/s (8749MB/s-8749MB/s), io=489GiB (525GB), run=60001-60001msec

When we run the same command within the VM that is using the iSCSI disk we get the following results:
READ: bw=220MiB/s (231MB/s), 220MiB/s-220MiB/s (231MB/s-231MB/s), io=12.9GiB (13.9GB), run=60036-60036msec
WRITE: bw=220MiB/s (231MB/s), 220MiB/s-220MiB/s (231MB/s-231MB/s), io=12.9GiB (13.9GB), run=60036-60036msec

We have updated the MTU within the network stack to 9000 but to be honest I am not sure that is in affect and will be looking into it further.

Any advice on where to go next to try to figure why it is performing so poorly would be much appreciated.



Storage server specs:
TrueNAS Core 13.0-U5.3
AMD EPYC 9124
RAM 393204MB ECC
Micron 7450 x10 NVMe SSD
MT28800 NIC

Compute server specs:
Proxmox 8.0.4
AMD EPYC 7543P
RAM 500GB DDR4-4300 ECC
MT28800 NIC

Switch:
QSW-M5216-1T

 

[Lix]

You should not use link aggregation with iSCIS. Can you test NFS as well, just to compare the baseline against iSCSI?

 

[glloyd-a]

You should not use link aggregation with iSCIS. Can you test NFS as well, just to compare the baseline against iSCSI?

Thanks for the reply!

We ended up removing link aggregation and in the process found that our problem was actually with bad cables.

 

[Lix]

Great to hear that the issue was solved :)