Problems with LACP - TrueNAS-SCALE-22.02.0

[mvipe01]

Hello everyone,

I am trying to setup a LACP (2) 10gbps but am only getting 1 of the nics linked in. Below is my setup:

NIC: Chelsio (2) 10gbps SFP+ card.
Switch: Dell S4810P

I have configured the LACP in the Dashboard GUI:

ifconfig shows:
enp130s0f0: flags=6211<UP,BROADCAST,RUNNING,SLAVE,MULTICAST> mtu 1500
ether 3e:93:41:5e:80:b1 txqueuelen 1000 (Ethernet)
RX packets 84953 bytes 11029745 (10.5 MiB)
RX errors 0 dropped 1 overruns 0 frame 0
TX packets 2529886 bytes 3710105527 (3.4 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp130s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 80:61:5f:04:81:6f txqueuelen 1000 (Ethernet)
RX packets 736 bytes 91264 (89.1 KiB)
RX errors 0 dropped 736 overruns 0 frame 0
TX packets 114944 bytes 4868567 (4.6 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

bond0: flags=5187<UP,BROADCAST,RUNNING,MASTER,MULTICAST> mtu 1500
ether 3e:93:41:5e:80:b1 txqueuelen 1000 (Ethernet)
RX packets 84953 bytes 11029745 (10.5 MiB)
RX errors 0 dropped 946 overruns 0 frame 0
TX packets 2529886 bytes 3710105527 (3.4 GiB)
TX errors 0 dropped 6 overruns 0 carrier 0 collisions 0


Both enp130s0f0/1 show up at 10000mbps, however ethtool shows the bond0 only up at 10000mbps not the expected 20000mbps:

Settings for enp130s0f0:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: 10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: off
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes

Settings for enp130s0f1:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: 10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: off
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes

Settings for bond0:
Supported ports: [ ]
Supported link modes: Not reported
Supported pause frame use: No
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: off
Port: Other
PHYAD: 0
Transceiver: internal
Link detected: yes

Dell S4810P configuration (I have tried both passive and active modes):

interface TenGigabitEthernet 0/8
no ip address
!
port-channel-protocol LACP
port-channel 5 mode passive
no shutdown
!
interface TenGigabitEthernet 0/9
no ip address
!
port-channel-protocol LACP
port-channel 5 mode passive
no shutdown
!

interface Port-channel 5
description "TrueNAS LAN"
no ip address
switchport
spanning-tree 0 portfast
no shutdown
!
(I have tried with and without portfast).

brief shows that the link is up however only interface 0/9 is linked in (0/8 is not up)

ifconfig on truenas as well as interface status in the switch both show ports 0/8 and 0/9 as up at 10000mbps

The bond0 doesnt seem to show enp130s0f1 as a slave even though in the GUI it is shown:

root@black[~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v5.10.93+truenas

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2+3 (2)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 3e:93:41:5e:80:b1
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 1
Actor Key: 15
Partner Key: 5
Partner Mac Address: 00:01:e8:8b:9c:0e

Slave Interface: enp130s0f0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 4
Permanent HW addr: 80:61:5f:04:81:6e
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 1
Partner Churned Count: 1
details actor lacp pdu:
system priority: 65535
system mac address: 3e:93:41:5e:80:b1
port key: 15
port priority: 255
port number: 1
port state: 61
details partner lacp pdu:
system priority: 32768
system mac address: 00:01:e8:8b:9c:0e
oper key: 5
port priority: 32768
port number: 139
port state: 62

When creating the bond0 in the GUI the output of /var/log/messages contains the following which does not even mention or attempt to link in enp130s0f1:

ip link show does not even show enp130s0f1 as part of the bond0 even though it is clearly in it in the GUI's tab:

I am at a loss here, I have multiple other systems running and configured and working using LACP however I just simply cant get TrueNAS Scale to link the second interface into the LACP.

Any help is GREATLY appreciated. Thank you for your time.

 

[mvipe01]

Still hitting my head against the wall trying to get the LACP working, can anyone here please provide a hand?

I think it has to do with the second interface not being marked as a SLAVE properly.

enp130s0f0 shows as a SLAVE but enp130s0f1 isnt showing up as a slave even though it is configured in the scale GUI:

It is also missing from /proc/net/bonding/bond0:

Thanks

 

[bgermain1689]

The linux bonding module automatically creates "bond0" unless you explicitly disable this via the module option max_bonds=0, Ive had issue on multiple systems trying to modify and use this default bond, including scale. I simply changed my config to use the name "bond1" and everything magically started working as expected. If you must keep the bond0 name you'd have to drop the option in /etc/modprobe.d though i don't know if this persist across system updates.

additionally something like this might work but not completely sure.

midclt call system.advanced.update '{"kernel_extra_options": "max_bonds=0"}'

 

[bgermain1689]

The linux bonding module automatically creates "bond0" unless you explicitly disable this via the module option max_bonds=0, Ive had issue on multiple systems trying to modify and use this default bond, including scale. I simply changed my config to use the name "bond1" and everything magically started working as expected. If you must keep the bond0 name you'd have to drop the option in /etc/modprobe.d though i don't know if this persist across system updates.

additionally something like this might work but not completely sure.

midclt call system.advanced.update '{"kernel_extra_options": "max_bonds=0"}'

or

midclt call system.advanced.update '{"kernel_extra_options": "bonding.max_bonds=0"}'

 

[Darrell Patenaude]

Change layer 2 + 3 to 3 + 4 in your lacp setup in truenas. fixed it for me.

 

[morganL]

Recommend updating to SCALE 22.12.3 - there's been a thousand+ bug fixes since 22.2.0
We don't any anyone's time wasted.

 

[Darrell Patenaude]

I did upgrade to 22.12.3.1 today and I couldn't get the lacp to work as in only 1 10gb nic would work. I searched with no luck. By chance I saw a proxmox post that explained the issue and it worked. I also had to log into the ipmi to do this as the web didn't work.

 

[Gustav]

Recommend updating to SCALE 22.12.3 - there's been a thousand+ bug fixes since 22.2.0
We don't any anyone's time wasted.

Hi. I have the same problem, but with Active Backup (FAILOVER)
Bond don´t switch to another port .
Is there a fix planed ?
theme: https://www.truenas.com/community/threads/problems-with-failover-link-aggregation.107673/