Mixed 10GbE 1GbE link aggregation with load balance and failover

[RickH]

I run several FreeNAS servers in a production environment for a mid-size scanning and document management company. We use these servers for both VMware ISCSI datastores and production storage over CIFS. All told we have more than 120 TB of storage spread over 5 FreeNAS servers. I've been extremely happy with the performance and flexibility that FreeNAS has offered us, and despite many of my vendors trying to sell me a new platform, plan on continuing to use FreeNAS for a long time to come!

We're in the process of upgrading our network backbone to 10GbE and I'm trying to figure out the best way to set everything up. In the past I've always added a 4 port 1GbE intel NIC to the onboard ports and created a single LACP link aggregation and split the cable runs to a split lacp link aggregation on at least 2 of my stacked Dell PowerConnect switches. This approach has provided excellent performance and redundancy as it survives a cabling or single switch failure with no issues - the load is simply balanced across the remaining active connections.

Due to some cost constraints, I have only a single 10GbE switch at this point (that unfortunately doesn't support stacking or spreading ling aggregation channels across multiple switches even if I get a 2nd). My newest FreeNAS server has a dual port Chelsio 10GbE NIC as well as a 2 on-board 1GbE ports. What I would like to accomplish is 2 have traffic balanced across the 2 10GbE connections with failover to the 1GbE ports if the 10GbE switch somehow goes down.

I tried setting up a single static Load Balanced aggregation with all 4 ports active and the LAGG priority number set to 1 on the 10GbE ports and 10 on the 1GbE ports - this gave me the stability I was looking for; the connection remained steady through individual cable pulls and when powering down a single switch - but performance was all over the board. The iscsi connections to my VMware hosts seems to randomly switch between the 10GbE and 1GbE connections and running 'systat -ifstat' showed that even though both 10GbE connections were up, the actual amount of data being transferred was almost even across all 4 connections.

Does anyone have any suggestions on how to best set this up? Is there a way to set up a 10GbE LCAP LAGG a second 1GbE LCAP LAGG and then somehow put those into a failover group?

I appreciate any feedback!

 

[jgreco]

Yeah, LACP doesn't support using connections of different speed, and even using same speed interfaces with different drivers is frowned upon. The good news is that you can probably make combining 10G and 1G work if you try hard enough, the bad news is that it'll be very difficult for a very modest return and you probably shouldn't try.

You should try to avoid mixing LACP with iSCSI. Use two separate networks for iSCSI, preferably on separate switches if you want redundancy. It's probably cheaper to go on eBay and shop for 10G gear if you're on a budget, as there are a lot of very reasonable devices such as the Dell 8024f or the Dell x4012. You can absolutely put a fast 10G network for one of the iSCSI networks and a 1G network for the other, and then preference them in ESXi to prefer the faster path. It isn't clear that there's a good option here for CIFS.

 

[RickH]

I know it's been a while since I posted this, but after a lot of testing I came up with a solution that I'm pretty happy with - thought I would share in case anyone else has a similar setup:

My setup contains 4 ESXi servers - 2 FreeNAS servers - 1 10GbE switch - and 2 stacked 1GbE switches (and a bunch of other stuff that doesn't apply to this config)
Each of the 4 ESXi hosts and the FreeNAS servers have 2 10GbE ports and 2 1GbE ports

I created 2 failover link aggs on each of my FreeNAS servers each with 1 of the 10GbE links as primary and one of the 1GbE links as secondary. The 1GbE cables were run to separate switches and no Link Aggregations were configured on the switches themselves. I created 4 separate iscsi VLAN's on my switches (20-23) and assigned 2 of them to each of the failover laggs on the FreeNAS servers. The ESXi config is a little more complicated, but each of the 4 NIC's got it's own VLAN assignment and multipathing was configured to preference the 10GbE connections.

This gave me consistent iscsi performance (when everything is up the data flows only on the 10GbE connections) while also providing the redundancy I needed (as long as 1 switch is up the connections are active)

This setup makes some compromises for CIFS performance as I can only use 1 of the failover laggs on my Primary LAN network so CIFS is only flowing through 1 of the 10GbE interfaces and if I were to somehow loose my 10GbE switch and the 1GbE switch that that failover group is connected to I would loose CIFS access - I'm considering assigning a separate ip to the 2nd failover lagg and then using some DNS load balancing combined with some VLAN routing on my switches (this would be a lot easier if FreeNAS allowed me to assign IP's on the same network).

I'm probably overthinking the CIFS thing - overall I'm very happy with the current setup and feel it offers the best balance of performance and redundancy available with my current hardware. I would welcome any feedback though!

 

[cyberjock]

You should try to avoid mixing LACP with iSCSI. Use two separate networks for iSCSI, preferably on separate switches if you want redundancy. It's probably cheaper to go on eBay and shop for 10G gear if you're on a budget, as there are a lot of very reasonable devices such as the Dell 8024f or the Dell x4012. You can absolutely put a fast 10G network for one of the iSCSI networks and a 1G network for the other, and then preference them in ESXi to prefer the faster path. It isn't clear that there's a good option here for CIFS.

I just want to clarify this statement. You *can* use LACP with iSCSI. It will work properly. BUT, if you are using iSCSI only, you are probably better served with the multipath I/O feature that iSCSI offers, and should setup 2 separate networks for iSCSI. Of course, if you plan to use LACP and do iSCSI and something else (CIFS/NFS/etc) then you may want to do LACP. You just have to decide what is more important for your situation and then design it accordingly.