Hi,
I have problems with SMB multichannel when I use the TCP settings recommended by Mellanox for Linux and 40GbE.
I'm using Manjaro Kernel 6.4.6-1 and TrueNas Scale - 22.12.3.3, the systems are directly connected with 2x56 Gb/s and MTU 9000.
I use two VirtIO vNICs for all Windows VMs, since the Mellanox CX3-Pro cards have a bug with SR-IOV and Windows.
But to rule out that it has something to do with KVM or VirtIO, I also tested with a bare metal installation on Windows10 22H2 and Mellanox driver version WinOF-5_50_54000, the results are completely identical
With the TCP settings recommended by Mellanox for 40GbE, I get an average of 1.3 GB/s for files from Disk and 2 GB/s for files from ARC with e.g. big ISO files copied via file manager.
With SMB Multichannel and Mellanox TCP settings, I get up to 2.4 for reads from ARC and up to 3GB/s writes to Truenas, but everything from Disk is unuseable slow, the transfer speed varies between 30 MB/s and max. 100 MB/s
SMB Multichannel works without Sysctl changes, but with Mellanox TCP settings normal SMB is faster.
TCP Settings for Manjaro and TrueNAS
Nvidia - linux-sysctl-tuning
[manja-02 ~]# cat /etc/sysctl.d/99-sysctl.conf
#net.ipv4.tcp_congestion_control=dctcp
net.ipv4.tcp_timestamps=0
net.ipv4.tcp_sack=1
net.core.netdev_max_backlog=250000
net.core.rmem_max=4194304
net.core.wmem_max=4194304
net.core.rmem_default=4194304
net.core.wmem_default=4194304
net.core.optmem_max=4194304
net.ipv4.tcp_rmem=4096 87380 4194304
net.ipv4.tcp_wmem=4096 65536 4194304
net.ipv4.tcp_low_latency=1
net.ipv4.tcp_adv_win_scale=1
net.ipv4.tcp_mtu_probing=1
I have problems with SMB multichannel when I use the TCP settings recommended by Mellanox for Linux and 40GbE.
I'm using Manjaro Kernel 6.4.6-1 and TrueNas Scale - 22.12.3.3, the systems are directly connected with 2x56 Gb/s and MTU 9000.
I use two VirtIO vNICs for all Windows VMs, since the Mellanox CX3-Pro cards have a bug with SR-IOV and Windows.
But to rule out that it has something to do with KVM or VirtIO, I also tested with a bare metal installation on Windows10 22H2 and Mellanox driver version WinOF-5_50_54000, the results are completely identical
With the TCP settings recommended by Mellanox for 40GbE, I get an average of 1.3 GB/s for files from Disk and 2 GB/s for files from ARC with e.g. big ISO files copied via file manager.
With SMB Multichannel and Mellanox TCP settings, I get up to 2.4 for reads from ARC and up to 3GB/s writes to Truenas, but everything from Disk is unuseable slow, the transfer speed varies between 30 MB/s and max. 100 MB/s
SMB Multichannel works without Sysctl changes, but with Mellanox TCP settings normal SMB is faster.
TCP Settings for Manjaro and TrueNAS
Nvidia - linux-sysctl-tuning
[manja-02 ~]# cat /etc/sysctl.d/99-sysctl.conf
#net.ipv4.tcp_congestion_control=dctcp
net.ipv4.tcp_timestamps=0
net.ipv4.tcp_sack=1
net.core.netdev_max_backlog=250000
net.core.rmem_max=4194304
net.core.wmem_max=4194304
net.core.rmem_default=4194304
net.core.wmem_default=4194304
net.core.optmem_max=4194304
net.ipv4.tcp_rmem=4096 87380 4194304
net.ipv4.tcp_wmem=4096 65536 4194304
net.ipv4.tcp_low_latency=1
net.ipv4.tcp_adv_win_scale=1
net.ipv4.tcp_mtu_probing=1
Attachments
Last edited:
