SOLVED mlxen0: Interface stopped DISTRIBUTING, possible flapping

[Lars Jensen]

Hi

Using FreeNAS 11.2 U1 with Mellanox ConnectX 3 (2x10 gigabit) and LACP has run for 6 months without issue, recently traffic has increased a bit (from 5-10 megabit to 60-100 megabit avg), and flapping now occurs. In FreeNAS 11.2 U2 there's a bug fix for dhclient concerning flapping but this install doesn't use DHCP.

In another FreeNAS 9.3 install no flapping occurs using the same switches, even with very megabit avg, but the NIC is an Intel card (and of course another version of FN).

Any experience with Mellanox and FreeNAS in regards to flapping in general?

Switches are Dell/Force10 S4048-ON 48 Port SFP+.

 

[Elliot Dierksen]

Are the port settings the same in both cases on the switch? Things like active versus passive LACP, flow control, etc? Are these DAC cables, twisted pair, or optical? Is it always the same port that flaps?

 

[Lars Jensen]

Yes, same configuration on the switch.

It's DAC cables.

I can't for sure say it's the same port, but will check.

dmesg outputs:

...
mlx4_core0: port level mtu is only used for IB ports
mlx4_core0: port level mtu is only used for IB ports
mlx4_core0: port level mtu is only used for IB ports
mlx4_core0: port level mtu is only used for IB ports
mlx4_core0: port level mtu is only used for IB ports
mlx4_core0: port level mtu is only used for IB ports
mlx4_core0: port level mtu is only used for IB ports
mlx4_core0: port level mtu is only used for IB ports
mlx4_core0: port level mtu is only used for IB ports
mlx4_core0: port level mtu is only used for IB ports
mlx4_core0: port level mtu is only used for IB ports
mlx4_core0: port level mtu is only used for IB ports
Limiting closed port RST response from 1358 to 200 packets/sec
mlxen0: Interface stopped DISTRIBUTING, possible flapping

ifconfig outputs:

mlxen0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:02:c9:33:0a:b0
hwaddr 00:02:c9:33:0a:b0
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (10Gbase-CX4 <full-duplex,rxpause,txpause>)
status: active
mlxen1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:02:c9:33:0a:b0
hwaddr 00:02:c9:33:0a:b1
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (10Gbase-CX4 <full-duplex,rxpause,txpause>)
status: active
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:02:c9:33:0a:b0
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: active
groups: lagg
laggproto lacp lagghash l2,l3,l4
laggport: mlxen0 flags=0<>
laggport: mlxen1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>

 

[dlavigne]

Does updating to 11.2-U5 resolve the issue?

 

[Lars Jensen]

It's a switch thing! One of the switches actually freezes up. Running VLT and secondary has primary role now. No indications of error visible on the switch lights, everything green BUT no activity blinks on the ports, only steady green. And servers showed active links to the switches. Had me fooled big time until I inspected the VLT details.
Very sorry for the inconvenience!