lagg failover - 10g and 1g ?

SMnasMAN

Contributor
Joined
Dec 2, 2018
Messages
177
I know you cant mix different speeds with true active LACP link agg, but the Lagg->Failover option in freenas seems a bit different than true LACP.

So can we mix 10g and 1g in a lagg failover on freenas? (would add the 10g first so its primary, 2nd 10g second, and the 1g 3rd).

Also, does freenas lagg failover mode only work based on link up/down ? (or does it use arp or pings / other to check if a link is up/down) - ie if a switch "froze up" but its the links were still showing as enabled/link up, would this cause freenas failover to NOT swap to 2nd link?

Thanks
 

SMnasMAN

Contributor
Joined
Dec 2, 2018
Messages
177
I have not. any ideas?

tks
 

Jessep

Patron
Joined
Aug 19, 2018
Messages
379
Normally in business for redundancy at a minimum you would have 2 dual port NICs per server, going to dual switches. One port of each card going to each switch. This protects from port failure, card failure, and switch failure.

In a home lab the it's highly unlikely you are running any workloads that would require anything of that nature.

You are likely fine with a single 10Gb port as it's unlikely your pool can send more than 1GB/s.
 

SMnasMAN

Contributor
Joined
Dec 2, 2018
Messages
177
Normally in business for redundancy at a minimum you would have 2 dual port NICs per server, going to dual switches. One port of each card going to each switch. This protects from port failure, card failure, and switch failure.

In a home lab the it's highly unlikely you are running any workloads that would require anything of that nature.

You are likely fine with a single 10Gb port as it's unlikely your pool can send more than 1GB/s.
thanks for reply jessep, im actually aware of all of that, but my question still is if its possible (mix 10g and 1g in failover). I do have some NFS to vmware shares, so those are important.
forexample i know esxi says you can not mix 10g and 1g for failover (not lagg but failover), but esxi failover is different that FN.

any one tried this with FN? tks
 
Joined
Dec 29, 2014
Messages
1,135
any one tried this with FN?
I haven't tried that, but it seems like it should work. From a Cisco switch perspective, you cannot add mixed speed connections to a port channel (LAGG in FreeNAS speak). Failover doesn't require any different configuration from the perspective of the switch.
 

SMnasMAN

Contributor
Joined
Dec 2, 2018
Messages
177
i forgot to update this, but i went back and tested this recently. and while it did seem to work well when fail'd over to the 1g link (was getting full 100 MB/s + UP and down) i was having unusual speeds only on the upload side of the 10g link (ie download ~ 400 MB/s , upload ~20-30 MB/s). (was testing with SMB and NFS sync=disabled file copy operations)

Also there was no clear logic for the fail over to "prefer" the 10g link. (ie if the 10g link was current primary, and the 10g fails, it would correctly move to the 1g link with no interruption, but once the 10g link was active again, it would stay on the 1g link).

when i tested just the 10g link alone in non failover mode (ie as a control), i did not see the slow upload issue. (ie would see ~400 MB/s on both upload and download)
 

coldnight

Cadet
Joined
Jun 29, 2011
Messages
3
What are the devices you are using? I am doing mostly the same thing as I'd like be able to turn off my Cisco 93128 ( its loud ) when I'm not debugging things. I have a BCE0 device and BXE0 device, and, in the shell: ifconfig lagg0: BCE0 is the master/preferred link. As if the device naming is driving the primary link device.

ja.




i forgot to update this, but i went back and tested this recently. and while it did seem to work well when fail'd over to the 1g link (was getting full 100 MB/s + UP and down) i was having unusual speeds only on the upload side of the 10g link (ie download ~ 400 MB/s , upload ~20-30 MB/s). (was testing with SMB and NFS sync=disabled file copy operations)

Also there was no clear logic for the fail over to "prefer" the 10g link. (ie if the 10g link was current primary, and the 10g fails, it would correctly move to the 1g link with no interruption, but once the 10g link was active again, it would stay on the 1g link).

when i tested just the 10g link alone in non failover mode (ie as a control), i did not see the slow upload issue. (ie would see ~400 MB/s on both upload and download)
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
I always set up my ESXi servers this way, pairing 10G and 1G vswitch uplinks with the 10G active and the 1G as a standby.

I wanted to do the same thing running FreeNAS/TrueNAS 'on the metal', so I tried failover LAGG groups under FreeNAS 11.2-U8, 11.3-U5, and TrueNAS 12.0-U5.1

My interfaces are a SolarFlare SF6122F 10G dual-port (sfxge0, sfxge1) and 2 x motherboard Intel ports (igb0, igb1)

It works under 11.2-U8: my 10G interface comes up as the master and the 1G as the failover.

But it does not work under 11.3 or 12.0: despite ordering the 10G first (which should make it the master) I always end up with the 1G port as the master:
Code:
lagg1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: Storage
        options=e407ab<RXCSUM,TXCSUM,VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether ac:1f:6b:cf:a6:5d
        inet 10.0.58.12 netmask 0xffffff00 broadcast 10.0.58.255
        laggproto failover lagghash l2,l3,l4
        laggport: sfxge1 flags=0<>
        laggport: igb1 flags=5<MASTER,ACTIVE>
        groups: lagg
        media: Ethernet autoselect
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>

Here is the matching setup, where you can see the interface ordering:
lagg-setup.jpg


I also verified the ordering by looking at the network_lagginterfacemembers table in the configuration database.

There is an open issue about this on Jira, NAS-104883 "Unable to prioritize interface member in LAGG", but it's been on the backburner for a long time:


:rolleyes:
 

JoeAtWork

Contributor
Joined
Aug 20, 2018
Messages
165
I want to do the same thing lagg(10gb + 1gb failover)

I know if can do this with VMware with 2 vSwitches

Is there some kind of command we can run as a cron job to try to promote the 10g port to master?

ifconfig : state MASTER|BACKUP
 

JoeAtWork

Contributor
Joined
Aug 20, 2018
Messages
165
I want to do the same thing lagg(10gb + 1gb failover)

I know if can do this with VMware with 2 vSwitches

Is there some kind of command we can run as a cron job to try to promote the 10g port to master?

ifconfig : state MASTER|BACKUP
My bad state master is for CARP VIP's, does not apply to LACP failover groups
 

JoeAtWork

Contributor
Joined
Aug 20, 2018
Messages
165
if the 1gb port is master you have to remove it and add it so the 10gb is master active, from the command line :

ifconfig lagg1 -laggport igb2
ifconfig lagg1 laggport igb2
 

JoeAtWork

Contributor
Joined
Aug 20, 2018
Messages
165
And just like that my effort goes down the pooper!

The setting does not survive a reboot. My master is back to the 1gb NIC igb2! :-(

lagg1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: lagg for me nfs options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether 00:1a:21:a7:6a:a8 laggproto failover lagghash l2,l3,l4 laggport: ix0 flags=0<> laggport: igb2 flags=5<MASTER,ACTIVE> groups: lagg media: Ethernet autoselect status: active nd6 options=9<PERFORMNUD,IFDISABLED>
 

MGYVR

Dabbler
Joined
Mar 14, 2021
Messages
16
And just like that my effort goes down the pooper!

The setting does not survive a reboot. My master is back to the 1gb NIC igb2! :-(

lagg1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: lagg for me nfs options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether 00:1a:21:a7:6a:a8 laggproto failover lagghash l2,l3,l4 laggport: ix0 flags=0<> laggport: igb2 flags=5<MASTER,ACTIVE> groups: lagg media: Ethernet autoselect status: active nd6 options=9<PERFORMNUD,IFDISABLED>
Not sure if this will work with your particular use case (or FreeNAS version), but I also have a single 10Gbe main and single 1Gbe backup and I just setup the lagg0 with only the 10Gbe in lagg via the UI, then use the shell to "ifconfig lagg0 laggport [1Gbe interface]", then setup an init (post-init) command to run the same command on reboot. I have tested this on both a TrueNAS-13.0-U3.1 and a 12.0-U7. And after reboot everything remains correct.

I didn't test this on the 12.0 server, but on the 13.0 server no matter how I setup both the interfaces into the lagg in the UI, the 1Gbe interface always ended up the master.

Sorry I didn't see your post about the exact command I suggested, and I'm not entirely sure OTTOMH when the CRON commands would be fired off during boot, but the post-init commands in the UI have proven reliable for me.
 
Last edited:
Top