LAG not working correctly...

Status
Not open for further replies.

TheBlueDalek

Cadet
Joined
Jun 28, 2013
Messages
8
...or have I just screwed it up?

Hi all!

I recently purchased a TP-Link TL-SG2424 24-port managed switch, as I was tired of daisy chaining a bunch of smaller 5 and 8 port un-managed switches. It also gave me the opportunity to trunk multiple NIC's / ethernet ports together. As the switch does not come with LACP enabled, I had to install the latest firmware from TP-Link's website.

I have two machines with multi ethernet ports on them:

- My Ubuntu Server running on a Supermicro board has no trouble, and I am getting true load balancing over both ports.

- my FreeNAS box appears to be setup correctly, but does not appear to be operating as I believe it should.

My FreeNAS build is an AMD A4-3400 APU on a Gigabyte motherboard. The Gigabyte board has a single Broadcom gigabit NIC on board. I recently added an (HP branded) Intel PCIe dual gigabit card for the purposes of relieving network congestion.

The onboard Broadcome is RE0 (192.168.1.120), and the two Intel ports are EM0 and EM1 (192.168.1.122). I have the two Intel ports set as a LAG through the GUI, and LACP enabled within the switch. After rebooting the FreeNAS box, my switch automatically detects the LAG automatically. Running ifconfig shows the following:

Code:
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC>
        ether 00:26:55:d8:d5:96
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
        ether 00:26:55:d8:d5:96
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 5000
        options=3898<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
        ether 50:e5:49:52:40:1a
        inet 192.168.1.120 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet6 ::1 prefixlen 128
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=3<PERFORMNUD,ACCEPT_RTADV>                                                                                                                                         
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500                                                                                                             
        options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>                                                                                                             
        ether 00:26:55:d8:d5:96                                                                                                                                                         
        inet 192.168.1.122 netmask 0xffffff00 broadcast 192.168.1.255                                                                                                                   
        media: Ethernet autoselect                                                                                                                                                     
        status: active                                                                                                                                                                 
        laggproto lacp                                                                                                                                                                 
        laggport: em1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>                                                                                                                         
        laggport: em0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>


I have the onboard NIC set to have the management GUI interface, and I have the dual NIC set as the point of access for NFS access.
I have verified the GUI and SSH is not available over the .122 address, and only .120

The problem I am having, is when I am streaming, the media is being sent out over the single NIC (.120), not out over the LAG (.122). I have confirmed this by starting an XBMC media stream and running the following command on the FreeNAS system:

systat -ifstat

Code:
                   /0  /1  /2  /3  /4  /5  /6  /7  /8  /9  /10
    Load Average
 
      Interface          Traffic              Peak                Total
          lagg0  in    10.032 KB/s        10.032 KB/s          14.395 GB
                out    0.000 KB/s          0.008 KB/s          76.658 KB
 
            lo0  in      0.000 KB/s          0.354 KB/s          108.917 KB
                out    0.000 KB/s          0.354 KB/s          108.917 KB
 
            re0  in      0.038 KB/s          8.770 KB/s            1.734 MB
                out  807.233 KB/s        807.239 KB/s            9.610 GB
 
            em1  in      0.000 KB/s          0.140 KB/s            8.810 MB
                out    0.000 KB/s          0.023 KB/s          339.117 KB
 
            em0  in    10.032 KB/s        10.032 KB/s          14.387 GB
                out    0.000 KB/s          0.032 KB/s          413.809 KB


Just to confirm, the fstab on the XBMCbuntu machine is as such:

Code:
# <file system> <mount point>  <type>  <options>      <dump>  <pass>
# / was on /dev/sda1 during installation
UUID=392e04a4-5103-4a57-90d3-7844e89f564e /              ext4    errors=remount-ro 0      1
# swap was on /dev/sda5 during installation
UUID=deb11f89-12a1-46ed-ba5b-75a856647378 none            swap    sw              0      0
192.168.1.122:/mnt/storage/media        /mnt/media      nfs    rsize=131072,wsize=131072,rw,intr,noatime 0 0
tmpfs  /tmp    tmpfs  defaults 0 0


I have even tried configuring the LAG through the startup menu. I tried 'load balancing' (my preference) as well as LACP.

For comparison, here is the /etc/network/interfaces from my Ubuntu machine:

Code:
# This file describes the network interfaces available on your system                                                                                                                   
# and how to activate them. For more information, see interfaces(5).                                                                                                                   
                                                                                                                                                                                       
# The loopback network interface                                                                                                                                                       
auto lo                                                                                                                                                                                 
iface lo inet loopback                                                                                                                                                                 
                                                                                                                                                                                       
# The primary network interface                                                                                                                                                         
#auto eth0                                                                                                                                                                             
#iface eth0 inet dhcp                                                                                                                                                                   
                                                                                                                                                                                       
#eth0 is manually configured, and slave to the "bond0" bonded NIC                                                                                                                       
auto eth0                                                                                                                                                                               
iface eth0 inet manual                                                                                                                                                                 
bond-master bond0                                                                                                                                                                       
                                                                                                                                                                                       
#eth1 ditto, thus creating a 2-link bond.                                                                                                                                               
auto eth1                                                                                                                                                                               
iface eth1 inet manual                                                                                                                                                                 
bond-master bond0                                                                                                                                                                       
                                                                                                                                                                                       
# bond0 is the bonded NIC and can be used like any other normal NIC.                                                                                                                   
# bond0 is configured using static network information.                                                                                                                                 
auto bond0                                                                                                                                                                             
iface bond0 inet dhcp                                                                                                                                                                   
# bond0 uses standard IEEE 802.3ad LACP bonding protocol
bond-mode 4
bond-miimon 100
bond-lacp-rate 1
bond-slaves none


and the output from ifconfig:

Code:
bond0    Link encap:Ethernet  HWaddr 00:30:48:c7:99:6c 
          inet addr:192.168.1.148  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::230:48ff:fec7:996c/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:16962880 errors:0 dropped:3 overruns:0 frame:0
          TX packets:13733017 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:15582600559 (15.5 GB)  TX bytes:15836762907 (15.8 GB)
 
eth0      Link encap:Ethernet  HWaddr 00:30:48:c7:99:6c 
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:16943462 errors:0 dropped:2 overruns:0 frame:0
          TX packets:10398168 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:15580267837 (15.5 GB)  TX bytes:15376140363 (15.3 GB)
          Interrupt:71 Memory:de420000-de440000
 
eth1      Link encap:Ethernet  HWaddr 00:30:48:c7:99:6c 
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:19418 errors:0 dropped:1 overruns:0 frame:0
          TX packets:3334849 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2332722 (2.3 MB)  TX bytes:460622544 (460.6 MB)
          Interrupt:70 Memory:de460000-de480000
 
lo        Link encap:Local Loopback 
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:804 errors:0 dropped:0 overruns:0 frame:0
          TX packets:804 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2015030 (2.0 MB)  TX bytes:2015030 (2.0 MB)


Is there something I have missed or done incorrectly? Any and all help / pointers are appreciated!
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
There is something you've missed. LACP doesn't work the way you think. If you don't have 10+ users there really is no point.

This is a very common mistake for people that have never used LACP and decide to use it. Please search the forums as this has been discussed to death many times over the months.
 

ljw1

Dabbler
Joined
Apr 29, 2012
Messages
16
The simple solution is to ensure that the network cards are not in the same subnet. If you want all the traffic to go over the bonded link for 192.168.1.* then you need to change em0 to something like 192.168.2.2 or 10.1.1.2.
 

TheBlueDalek

Cadet
Joined
Jun 28, 2013
Messages
8
There is something you've missed. LACP doesn't work the way you think. If you don't have 10+ users there really is no point.

This is a very common mistake for people that have never used LACP and decide to use it. Please search the forums as this has been discussed to death many times over the months.

Thanks for the reply. Your right, I did not have a correct understanding, I thought it more like teaming or 'bonding' as Ubuntu calls it.

I have a couple other questions:

The text menu that comes up when a monitor is connected, has an option called 'load balance', is this a functional feature for FreeNAS?

Is there a difference between the 802.3ad protocol through Ubuntu versus FreeNAS? The reason I ask is that I get load balancing through Ubuntu with the config as listed above.

My understanding of network routing is that if a machine has two IP's, and a request is received on IP 1, the response would be sent back over the same IP. Why would the request be sent to IP 1, yet the response/traffic is being sent back via a different IP? The network traffic snap shot I posted above shows the media is being streamed from IP 2, a NIC not associated with the LAG, but the connection between machines is being made over IP 1?

Thanks again!
 
Status
Not open for further replies.
Top