Hello all:
I'm presently battling three used Chelsio T580-LP-CR's that I picked up recently and wanted to see if I'm missing anything. I've installed these cards on my small network, which is set up with a Windows 10 box as my primary workstation with conventional 1GbE connections to a few other servers, but for speed, I wish to have two connected as DAC's with these T580 40GbE NICs.
The Windows box specs are as follows:
Specs of the TrueNAS box I'm working in are as follows. This is the one with the DAC connection to Win 10:
Both cards are in PCIe 3.0 x8 slots, or in slots capable of x8 speeds. Cables are 5m-long QSFP+ NetApp 112-00178's.
Here's the problem:
I've set the static IP's in both Windows and TrueNAS, and in Windows and the cards won't see each other, nor will they ping each other. This is with the same, basic configuration I've used for ages with prior cards: Windows NIC has a static IP of x.x.2.11 and a gateway of x.x.2.10, and the server NIC is x.x.2.10.
Historically, I have not had to set a gateway on TrueNAS (Core or Scale) or the similar NAS4Free/XigmaNAS boxes to get them each to recognize each other.
The Chelsio cards though, won't connect at all. They won't ping, no packets are sent, and Windows sees it as unplugged. TrueNAS Core 13 sees their links as down in the GUI; ifconfig shows the following. Note that I was trying different subnets here, so please disregard the x.x.3.20 address on one of the ports:
Mind, I've checked and re-checked those QSFP+ cables. ifconfig may be reporting "status: no carrier," but each end is locked as they should.
This is what gets me: I bought the Chelsio cards to replace a set of Mellanox Connect-X3 CX354A 40/56GbE cards (set in 40GbE Ethernet mode, not IB, of course) which were connecting 100% without issue with this same configuration and cables, on the same exact computers, with the same exact settings. And even prior to that, the same HP workstation and servers were all fitted with 10GbE Connect-X2's, with the same IP configurations, without issue.
The only reason I pulled the Connect-X3's is because they were barely cresting ~21Gbps when tested in iperf3 (with -P 16) and I couldn't find any bottleneck other than the possibility of the card itself. Also, for whatever reason, the T580's are not recognized in TrueNAS SCALE either, and the Mellanox cards only recognized in Core 13, and not in any v12 BSD OSes (such as XigmaNAS).
After pulling my hair out with the Mellanox cards, I decided to follow what I'd read here on the forum about TrueNAS having better support and overall tuning for the Chelsios, but now I'm stumped and regretting it!
Any advice would be gladly appreciated. Thank you!
-Kurt
I'm presently battling three used Chelsio T580-LP-CR's that I picked up recently and wanted to see if I'm missing anything. I've installed these cards on my small network, which is set up with a Windows 10 box as my primary workstation with conventional 1GbE connections to a few other servers, but for speed, I wish to have two connected as DAC's with these T580 40GbE NICs.
The Windows box specs are as follows:
- HP Z8 G4
- x2 Xeon Silver 4110's
- 128 GB DDR4
- 1TB NVMe boot drive
- Current NIC is the T580*
Specs of the TrueNAS box I'm working in are as follows. This is the one with the DAC connection to Win 10:
- TrueNAS Core 13
- Asus Sabertooth X99
- E5-2620 v3
- 4GB Hynix PC4-2400 ECC DDR4 (spare from X10 build, waiting on more)
- No drives or HBAs at the moment
- Chelsio T580-LP-CR
Both cards are in PCIe 3.0 x8 slots, or in slots capable of x8 speeds. Cables are 5m-long QSFP+ NetApp 112-00178's.
Here's the problem:
I've set the static IP's in both Windows and TrueNAS, and in Windows and the cards won't see each other, nor will they ping each other. This is with the same, basic configuration I've used for ages with prior cards: Windows NIC has a static IP of x.x.2.11 and a gateway of x.x.2.10, and the server NIC is x.x.2.10.
Historically, I have not had to set a gateway on TrueNAS (Core or Scale) or the similar NAS4Free/XigmaNAS boxes to get them each to recognize each other.
The Chelsio cards though, won't connect at all. They won't ping, no packets are sent, and Windows sees it as unplugged. TrueNAS Core 13 sees their links as down in the GUI; ifconfig shows the following. Note that I was trying different subnets here, so please disregard the x.x.3.20 address on one of the ports:
Code:
cxl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9216
description: Chelsio #1
options=6ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP,NOMAP>
ether 00:07:43:2e:c0:b0
inet x.x.2.20 netmask 0xffffff00 broadcast x.x.2.255
media: Ethernet none
status: no carrier
nd6 options=9<PERFORMNUD,IFDISABLED>
cxl1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9216
description: Chelsio #2
options=6ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP,NOMAP>
ether 00:07:43:2e:c0:b8
inet x.x.3.20 netmask 0xffffff00 broadcast x.x.3.255
media: Ethernet autoselect
status: no carrier
nd6 options=9<PERFORMNUD,IFDISABLED>Mind, I've checked and re-checked those QSFP+ cables. ifconfig may be reporting "status: no carrier," but each end is locked as they should.
This is what gets me: I bought the Chelsio cards to replace a set of Mellanox Connect-X3 CX354A 40/56GbE cards (set in 40GbE Ethernet mode, not IB, of course) which were connecting 100% without issue with this same configuration and cables, on the same exact computers, with the same exact settings. And even prior to that, the same HP workstation and servers were all fitted with 10GbE Connect-X2's, with the same IP configurations, without issue.
The only reason I pulled the Connect-X3's is because they were barely cresting ~21Gbps when tested in iperf3 (with -P 16) and I couldn't find any bottleneck other than the possibility of the card itself. Also, for whatever reason, the T580's are not recognized in TrueNAS SCALE either, and the Mellanox cards only recognized in Core 13, and not in any v12 BSD OSes (such as XigmaNAS).
After pulling my hair out with the Mellanox cards, I decided to follow what I'd read here on the forum about TrueNAS having better support and overall tuning for the Chelsios, but now I'm stumped and regretting it!
Any advice would be gladly appreciated. Thank you!
-Kurt
Last edited:
