LACP and jumbo frames - settings different in 9.1?

[Whatts]

Hi all,

In 8.3.1 I had set up LACP with an MTU of 9000, and this had to be done separately for each member NIC of the lagg (so in the options field of each of the NICs).

In 9.1 it seems this has to be done in the options of the lagg only?

Can anyone confirm that is correct and I'm not missing something?

(I've just swapped out my temporary AMD board on 8.3.1 to a permanent E3 Xeon / Supermicro combo on 9.1, so I can't just compare both machines)

Greetz,

Tom aka. Whatts

 

[MenacingM]

I'm having issues with this as well.

Apparently you must set the same MTU on the NICs first before creating your lagg and they must all be set to the same MTU.

One setting you cannot set on lagg is MTU, this must be inherited from the NICs.

After setting the MTU on your NICs you can create your lagg and reboot. Doing this I was able to get the lagg to display the jumbo frames MTU but it did not survive the restart and my MTU then went back down to 1500.

 

[dlavigne]

What is the driver name for the NICs (from ifconfig)?

 

[MenacingM]

Manufacturer (Gigabyte EP45-UD3P) states the internal NICs as Realtek 8111C.

The system is correctly using RE(4). NICs are re0 and re1 in ifconfig.

Before anyone starts blasting me on using Realtek NICs, I'm aware of the jumbo frames issue with these models under FreeBSD. For some reason, I can enable jumbo frames up to 9K on Windows & Linux (although admittedly I haven't put those drivers through the paces of testing 9K utilization) but in FreeBSD using the RE(4) driver, the highest I can go is MTU 6122. I have no idea why this is the highest value (defies logic to me as this value is not a multiple of 1024) but I determined this through trial and error. Trying to set the MTU to a value higher than 6122 kicks back the following error:

ifconfig re0 MTU 9000

Code:

ioctl (set mtu): Invalid argument

However, after setting both re0 and re1 to MTU 6122 using IFCONFIG and then creating the LACP/LAGG bonding using the Console Menu method (which kills all network access and seemingly causes the main console to lock up) I was able to see LAGG0 had an MTU of 6122 by logging into another console session locally using Option F2. However, this did not survive the reboot. Upon reboot both Realtek NICs and the LAGG0 bond had an MTU of 1500.

As I have an embedded install, I tried setting the MTU of both NICS using preinit and postinit commands:
ifconfig re0 up mtu 6122
My hope was that the MTU could be set before the LAGG0 bond was loaded but that didn't work. I haven't tried an rc.conf method of setting the MTU yet, so I'm open to trying this if someone can point me in the right direction.

I'll be installing an Intel NIC to see if my mileage varies and this can be chalked up to a Realtek driver <-> LAGG/LACP issue.

More info on the Realtek 8111C NICs below:

ifconfig

Code:

re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
    ether 6c:f0:49:06:ec:ae
    inet6 fe80::6ef0:49ff:fe06:ecae%re0 prefixlen 64 scopeid 0x8
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
re1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
    ether 6c:f0:49:06:ec:ae
    inet6 fe80::224:1dff:fe80:205b%re1 prefixlen 64 scopeid 0x9
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
    ether 6c:f0:49:06:ec:ae
    inet 10.0.0.5 netmask 0xffffff00 broadcast 10.0.0.255
    inet6 fe80::6ef0:49ff:fe06:ecae%lagg0 prefixlen 64 scopeid 0x10
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
    media: Ethernet autoselect
    status: active
    laggproto lacp lagghash l2,l3,l4
    laggport: re1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: re0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>

pciconf -lv | grep -B3 network

Code:

re0@pci0:6:0:0:    class=0x020000 card=0xe0001458 chip=0x816810ec rev=0x02 hdr=0x00
    vendor    = 'Realtek Semiconductor Co., Ltd.'
    device    = 'RTL8111/8168B PCI Express Gigabit Ethernet controller'
    class      = network
--
re1@pci0:7:0:0:    class=0x020000 card=0xe0001458 chip=0x816810ec rev=0x02 hdr=0x00
    vendor    = 'Realtek Semiconductor Co., Ltd.'
    device    = 'RTL8111/8168B PCI Express Gigabit Ethernet controller'
    class      = network

dmesg | grep -i ethernet

Code:

re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F PCIe Gigabit Ethernet> port 0xce00-0xceff mem 0xefeff000-0xefefffff,0xefee0000-0xefeeffff irq 16 at device 0.0 on pci6
re0: Ethernet address: 6c:f0:49:06:ec:ae
re1: <RealTek 8168/8111 B/C/CP/D/DP/E/F PCIe Gigabit Ethernet> port 0xae00-0xaeff mem 0xefcff000-0xefcfffff,0xefce0000-0xefceffff irq 17 at device 0.0 on pci7
re1: Ethernet address: 00:24:1d:80:20:5b

 

[dlavigne]

Just a heads up that FreeBSD usually puts MTU info in the manpage of the driver which is always in section 4. ifconfig gives the driver name, just remove the interface number (e.g. re0 means read the section 4 manpage for re). For example, search for MTU at http://www.freebsd.org/cgi/man.cgi?...=FreeBSD+9.1-RELEASE&arch=default&format=html.

 

[MenacingM]

I'll try to remember that next time I'm working with crappy NICs ;)

 

[MenacingM]

Also, DLavigne, I edited the post to include my attempts at a workaround using preinit commands. Next I was going to try editing rc.conf but wasn't sure where/how to do this properly on an embedded install.

 

[dlavigne]

Making edits to rc.conf is currently not available through the GUI, though a work around might be an init script that appends custom entries to rc.conf. It would be a good idea to request a GUI front-end to rc.conf as a feature request at https://bugs.freenas.org/projects/freenas (hint hint).

 

[MenacingM]

Message received: Feature Request #3051