SMB-Multi-Channel and network settings

[tannebil]

As I understand it, SMB Multi-Channel lets a client with multiple NICs talk to a TrueNAS Scale server with multiple NICs at a higher speed that any one NIC. However, the TrueNAS Scale documentation is quite specific that a server can't have multiple NICs on the same subnet. Do I really need to build out a second network (VPN or physical) with a different subnet if I want to use Multi-Channel on TrueNAS Scale?

I've never heard that come up in discussions about SMB Miulti-channel support on Synology or any of the other commercial NAS systems so maybe I'm just missing something obvious in TrueNAS Scale?

 

[morganL]

As I understand it, SMB Multi-Channel lets a client with multiple NICs talk to a TrueNAS Scale server with multiple NICs at a higher speed that any one NIC. However, the TrueNAS Scale documentation is quite specific that a server can't have multiple NICs on the same subnet. Do I really need to build out a second network (VPN or physical) with a different subnet if I want to use Multi-Channel on TrueNAS Scale?

I've never heard that come up in discussions about SMB Miulti-channel support on Synology or any of the other commercial NAS systems so maybe I'm just missing something obvious in TrueNAS Scale?

Yes, multiple subnets is the configuration that is tested and verified.

 

[tannebil]

Thanks.

Once I realized that two different subnets can run on the same physical network without using a VLAN, it doesn't seem quite as big an issue. I won't need to make any L3 connection and can just add static addresses for a different subnet on all the machines that are going to use multi-channel.

That seems simple enough. Any gotchas I need to worry about? The subnet won't have a gateway or any of the "normal" network services like a gateway, DHCP or DNS but I don't see where that would create an issue. Will Linux, Windows, and Macs be smart enough to automagically build the appropriate routing tables and only send traffic for that subnet to that interface or do I need to do something with static routes (where I know some theory but never actually used)?

 

[morganL]

Thanks.

Once I realized that two different subnets can run on the same physical network without using a VLAN, it doesn't seem quite as big an issue. I won't need to make any L3 connection and can just add static addresses for a different subnet on all the machines that are going to use multi-channel.

That seems simple enough. Any gotchas I need to worry about? The subnet won't have a gateway or any of the "normal" network services like a gateway, DHCP or DNS but I don't see where that would create an issue. Will Linux, Windows, and Macs be smart enough to automagically build the appropriate routing tables and only send traffic for that subnet to that interface or do I need to do something with static routes (where I know some theory but never actually used)?

That's why subnets are needed.. its automates the routing to load balance the interfaces.

 

[tannebil]

Well for one brief moment, it was working and the server did 570MB/s over two 2.5 Gb links between my TrueNAS server and a Windows 11 client. But in the process of testing different settings, I managed to break it. It's in a weird state right now where the server only transmits on the primary interface. Changed all the cables and but both the client and server on the same switch and I tried resetting multi-channel with a disable/reboot/enable/reboot but it didn't help.

The first server interface has IPV4 and IPV6 configured via DHCP reservation. The second server adapter on the server is manually configured for IPV4 on a different subnet but I didn't configure IPV6 (mostly because I don't really understand IPV6 well enough). Win 11 autoconfigured IPV6 on both interfaces on the client. I tried disabling IPV6 on the server with net.ipv6.conf..all.disable_ipv6 in sysctl and on the network adapters on the client so both only had IPV4. It didn't change anything.

One oddity is that the first screenshot shows the network performance before I made the physical network changes while the second shows afterwards. It's subtle but notice that the maximum Rx doubled for the secondary after the change. Some kind of cable, port, or autonegotiation issue that I hadn't noticed?

Any suggestions?

 

[tannebil]

I guess it's not surprising that there is networking magic in making multichannel work. Today I test what happens when I physically disconnect interfaces. The test is copying an 11GB rar file from TrueNAS to my Win 11 client and then the same file back to TrueNAS.

The attached graphic shows two tests. The first is with the secondary network disconnected at the client. All the Tx traffic with the client flows through the primary interface but the Rx traffic with the client shows as a fairly even split between the primary and secondary interfaces despite the secondary interface being physically disconnected and no route existing between the primary and secondary subnets. The second is with the secondary network disconnected at the server and all traffic goes through the primary.

The actual performance is that when both network interfaces are active on both the client and TrueNAS, transfers FROM the server take twice as long to complete (40 seconds) as transfers TO the server (20 seconds). The pool is a mirror of 2x 4TB NVMe PCIe 3.0x1 drives so much faster than the network.

So why is there traffic recorded for the secondary interface even when the client is disconnected from that network?

 

[morganL]

Not clear why - if its physically disconnected.

 

[tannebil]

Figuring it out would take far more knowledge about TrueNAS internals and SMB multi-channel than I possess And those that do have it, likely have more productive things to do.

It's still just a test build so I might just reinstall TrueNAS and see if that clears the asymmetric speed issue.

 

[morganL]

Figuring it out would take far more knowledge about TrueNAS internals and SMB multi-channel than I possess And those that do have it, likely have more productive things to do.

It's still just a test build so I might just reinstall TrueNAS and see if that clears the asymmetric speed issue.

Asymmetric speeds happen because writes get acknowledged faster than reads get responded to. The solution is to create more queue depth.

The only puzzle was the RX traffic split....

 

[tannebil]

Tried another test. This time copying an 8.33 GB MP4 file between a Mac Studio with a single 10 GB connection to the same subnet as the primary interface for the server. Client->server->client->server. When I first tried it, the server showed all the write traffic on the primary interface but all the read traffic on the secondary interface. Really weird. That's at 22:45 on first graph.

I puzzled about that for awhile, rebooted the server and tried the same sequence again. This time, the traffic was all on the primary interface. Weird only in that it changed. That's at 23:25 on the first graph.

I rebooted and tested again but this time went server->client->server->client this time so that the first copy from the server came from the pool rather than the ARC cache. The read times were 29 and write times were 31. That's at 23:55 on the second graph. The third graph is the pool I/O from that run of the test.

It would appear that something weird is happening with the reporting if the server does any multi-channel until it is rebooted (the actual time the copies took never changed). It also appears that the ARC cache (15GB) had no effect on performance in this limited test. Definitely not a generalizable conclusion, but might be relevant to my use case where the server is only going to to be a backup target for PBS, Time Machine, and SMB file copies and a read-cache might not matter much.

 

[tannebil]

I reinstalled (useful practice for the future) and the same problem occurred where writes to TrueNAS used multi-channel but reads from it did not. I went through some debugging on the Windows side and noticed that the client was using IPV6 addresses for the connections. Since TrueNAS didn't have an IPV6 address on the secondary adapter (mostly because I don't know how to create one manually), I disabled V6 on both of the client interfaces and reads immediately started using both channels. So all is good (at least until I try to manually assigned a V6 address as a learning experience).

One oddity was that when I reinstalled, on the first boot, both adapters had been assigned DHCP addresses on the same subnet and DHCP was enabled for both in the GUI. That violated two restrictions that I thought existed in TrueNAS Scale: 1) DHCP can only be active on one interface, and 2) multiple interfaces have to be on different subnets. After I manually configured the secondary, the DHCP restriction came into force in the GUI but I decided I'd had enough fun for for the day and didn't experiment with trying to assign the secondary to the same subnet.

 

[MajorKhanage]

I reinstalled (useful practice for the future) and the same problem occurred where writes to TrueNAS used multi-channel but reads from it did not. I went through some debugging on the Windows side and noticed that the client was using IPV6 addresses for the connections. Since TrueNAS didn't have an IPV6 address on the secondary adapter (mostly because I don't know how to create one manually), I disabled V6 on both of the client interfaces and reads immediately started using both channels. So all is good (at least until I try to manually assigned a V6 address as a learning experience).

One oddity was that when I reinstalled, on the first boot, both adapters had been assigned DHCP addresses on the same subnet and DHCP was enabled for both in the GUI. That violated two restrictions that I thought existed in TrueNAS Scale: 1) DHCP can only be active on one interface, and 2) multiple interfaces have to be on different subnets. After I manually configured the secondary, the DHCP restriction came into force in the GUI but I decided I'd had enough fun for for the day and didn't experiment with trying to assign the secondary to the same subnet.

I have the same issues.

Disabling IPv6 sadly didn't work for me. I just think that SMB Multichannel is not implemented well in Windows.

I also got the defaults of DHCP on multiple NICs out of the box, even though you can't (and shouldn't) configure it that way. I also had the issue that when I changed them to static the changes wouldn't commit properly. It comes up with the testing bit but not the ability to commit the changes. (Even when only changing a NIC that you are not using to connect to the web UI).

I had to use the console menu to configure the static IPs, then it was all okay after that and now works in the web UI correctly too. I had this happen on 2 different installs, so there is definitely a bug lurking somewhere there.

Back to the multi-channel piece...is it still working correctly for you on your Windows client? Just interested to know if it was a permanent fix for you or not. I've had it in Windows before (not using TrueNAS) where SMB multi sometimes worked and sometimes didn't.

 

[tannebil]

Still works last time I checked. My Windows machine has been booted into Proxmox for the last month so I have not checked recently.

V6 is still turned off as I'd need to manually assign V6 addresses and I don't know the right way to do that as 99.9% of my network configuration experience is with V4. I'm sure I could figure it out but, you know, competing priorities (like going on vacation )