lagg on 2 x 10GB (ix0,ix1) not working.

tmueko · May 27, 2014

I have a dual port 10GB with a Cisco SG500X. Without lagg0 everything is working.
when I configure lagg0 (using lacp) I can't access ip of lagg0. Cisco-Switch shows a working lacp and I can even sniff on that device and see broadcasts and multicast pakets. I can also see the arp-requests for that ip but it doesn't answer.

where do i start searching?

dlavigne · May 27, 2014

Post the output of ifconfig (within code tags).

tmueko · May 27, 2014

Code:

ix0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
    ether 90:e2:ba:4c:14:f8
    inet 192.168.0.124 netmask 0xffffff00 broadcast 192.168.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (autoselect <full-duplex>)
    status: active
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO>
    ether 90:e2:ba:4c:14:f9
    inet 192.168.0.108 netmask 0xffffff00 broadcast 192.168.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (autoselect <full-duplex>)
    status: active
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
    ether 10:60:4b:92:c5:5c
    inet 192.168.0.35 netmask 0xffffff00 broadcast 192.168.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex,master>)
    status: active
bge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
    ether 10:60:4b:92:c5:5d
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (none)
    status: no carrier
ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
    nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:e1:7a:f3:a3:00
    nd6 options=9<PERFORMNUD,IFDISABLED>
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 11 priority 128 path cost 2000
    member: ix0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:a3:c3:00:0b:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active

this is without lagg

dlavigne · May 27, 2014

this is without lagg

Sorry, I should have specified with lagg.

tmueko · May 27, 2014

this is with configured lagg

Code:

ix0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
    ether 90:e2:ba:4c:14:f8
    inet 192.168.0.124 netmask 0xffffff00 broadcast 192.168.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (autoselect <full-duplex>)
    status: active
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO>
    ether 90:e2:ba:4c:14:f8
    inet 192.168.0.108 netmask 0xffffff00 broadcast 192.168.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (autoselect <full-duplex>)
    status: active
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
    ether 10:60:4b:92:c5:5c
    inet 192.168.0.35 netmask 0xffffff00 broadcast 192.168.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex,master>)
    status: active
bge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
    ether 10:60:4b:92:c5:5d
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (none)
    status: no carrier
ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
    nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:e1:7a:f3:a3:00
    nd6 options=1<PERFORMNUD>
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 11 priority 128 path cost 2000
    member: ix0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 55
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:ab:65:00:0b:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
    ether 90:e2:ba:4c:14:f8
    inet 192.168.123.1 netmask 0xffffff00 broadcast 192.168.123.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    laggproto lacp lagghash l2,l3,l4
    laggport: ix1 flags=18<COLLECTING,DISTRIBUTING>
    laggport: ix0 flags=18<COLLECTING,DISTRIBUTING>

dlavigne · May 27, 2014

Is the lagg getting its IP by DHCP or is it staticly set?

The interfaces in the lagg should not have IP addresses and atm their IP addresses are in a different network.

tmueko · May 27, 2014

I can remove the ip-adresses from ix0 and ix1 but it makes no difference.

Code:

ifconfig ix0 -alias 192.168.0.124
ifconfig ix1 -alias 192.168.0.108
 
[root@freenas] /var/log# ifconfig -a
ix0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
    ether 90:e2:ba:4c:14:f8
    inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (autoselect <full-duplex>)
    status: active
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO>
    ether 90:e2:ba:4c:14:f8
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (autoselect <full-duplex>)
    status: active
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
    ether 10:60:4b:92:c5:5c
    inet 192.168.0.35 netmask 0xffffff00 broadcast 192.168.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex,master>)
    status: active
bge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
    ether 10:60:4b:92:c5:5d
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (none)
    status: no carrier
ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
    nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:e1:7a:f3:a3:00
    nd6 options=1<PERFORMNUD>
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 11 priority 128 path cost 2000
    member: ix0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 55
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:ab:65:00:0b:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=400b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO>
    ether 90:e2:ba:4c:14:f8
    inet 192.168.0.124 netmask 0xffffff00 broadcast 192.168.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    laggproto lacp lagghash l2,l3,l4
    laggport: ix1 flags=18<COLLECTING,DISTRIBUTING>
    laggport: ix0 flags=18<COLLECTING,DISTRIBUTING>

One funny thing: configured 192.168.123.1 on lagg0 and 192.168.123.2 on my computer. When I ping 192.168.123.1 I can see the ARP-Request on the lagg0-Interface

Code:

# tcpdump -nevfi lagg0 arp | grep 192.168.123
16:31:11.538425 ec:1a:59:3d:8b:4e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.123.1 tell 192.168.123.2, length 46

ser_rhaegar · May 27, 2014

It seems to me you have a switch configuration issue.

Are there multiple vlans configured on your switch?
Is the LACP interface of the switch ports setup to be a trunk instead of an access port?
Is it intentional that the lagg is on a different subnet?
Also just to reiterate dlavigne's question, is the lagg set for DHCP or a static IP?

tmueko · May 27, 2014

I tested both: static ip and dhcp. I used another network so i don't trash access to bge0.

switch config is as easy as i can be:

Code:

switchce8705#show running-config
config-file-header
switchce8705
v1.3.7.18 / R750_NIK_1_35_647_358
CLI v1.0
set system queues-mode 4
!
bonjour interface range vlan 1                       
hostname switchce8705
logging host 192.168.222.253
username cisco password encrypted xxx privilege 15
ip ssh server
clock timezone CET 1
clock summer-time web recurring eu
sntp unicast client enable
sntp unicast client poll
sntp server 0.de.pool.ntp.org poll
sntp server de.poolntp.org poll
!
interface tengigabitethernet1/1/1
channel-group 1 mode on
!
interface tengigabitethernet1/1/2
channel-group 1 mode on
!
exit

c32767a · Jun 3, 2014

I would check and make sure both ends are configured to use LACP.

The Cisco switch is set to channel-group mode on, which effectively disables LACP.
It looks like your lagg is configured to use LACP.

If that is the case, things will behave as you have observed. e.g. won't work.. :)

channel-group mode active might produce better results.

tmueko · Jun 4, 2014

I tested FreeBSD-9.2 and FreeBSD-10 which doesn't even show a connect (status: inactive). As 10GB are enough we decided to use "Failover" which worked perfectly.

tmueko · Jun 5, 2014

We tested a little more and it turns out that the twinaxial-cable was the problem: with a normal 850nm multimode cable and transciever everything is working fine!

Code:

[root@freenas] ~# ifconfig -v ix0
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO>
    ether 90:e2:ba:4c:14:f8
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (10Gbase-SR <full-duplex>)
    status: active
[root@freenas] ~# ifconfig -v ix1
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO>
    ether 90:e2:ba:4c:14:f8
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (10Gbase-SR <full-duplex>)
    status: active
[root@freenas] ~# ifconfig -v lagg0
    lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO>
    ether 90:e2:ba:4c:14:f8
    inet 192.168.0.60 netmask 0xffffff00 broadcast 192.168.0.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    laggproto lacp lagghash l2,l3,l4
    laggport: ix1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: ix0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>

Important Announcement for the TrueNAS Community.

lagg on 2 x 10GB (ix0,ix1) not working.

tmueko

Explorer

dlavigne

Guest

tmueko

Explorer

dlavigne

Guest

tmueko

Explorer

dlavigne

Guest

tmueko

Explorer

ser_rhaegar

Patron

tmueko

Explorer

c32767a

Patron

tmueko

Explorer

tmueko

Explorer

Similar threads