How to setup link aggregation

[Laser411]

So, How do I properly setup link aggregation? I have an RT-ac68u (Don't know if its link aggregation compatible or not) but more importantly, How should the settings end up looking like? I have IgbO and emO. Also, if they both need to be in the member list, how do I add igbO (currently the main) while staying connected, or is there an easy way to do this through ssh? My system is headless just fyi.

The ultimate goal is load balancing so I can increase the throughput which is bottlenecked by the network.

 

[Mirfster]

Thinking the best you could hope for would be to use "Fail-Over" mode; unless everything supports LACP. Perhaps it may be better to just forget about it...

Check out: LACP ... friend or foe?

 

[Yatti420]

Yep common misconception about LACP..

 

[m0nkey_]

You need a switch that supports 802.3ad for link aggregation. As @Mirfster has already mentioned, the best you can do is failover.

 

[jgreco]

So, How do I properly setup link aggregation? I have an RT-ac68u (Don't know if its link aggregation compatible or not) but more importantly, How should the settings end up looking like? I have IgbO and emO. Also, if they both need to be in the member list, how do I add igbO (currently the main) while staying connected, or is there an easy way to do this through ssh? My system is headless just fyi.

The ultimate goal is load balancing so I can increase the throughput which is bottlenecked by the network.

You shouldn't try to add different network card types to the same lagg. This can be problematic (though I'd also guess it will work for the specific case of igb and em).

 

[ttabbal]

You shouldn't try to add different network card types to the same lagg. This can be problematic (though I'd also guess it will work for the specific case of igb and em).

I'm curious what happens with a setup like this? I recently picked up a compatible switch, more just because I needed more 1G ports, but it supports LACP so I figured I'd try it and see how things work for my uses. I've only had the link up for a couple days, but it seems alright. It's using an onboard Realtek NIC along with a PCIe Intel card. After testing, if I want to keep the LACP I was planning to buy another Intel card, or maybe a dual port unit. So I'm asking to know what to look out for.

I know the Realtek NICs can be iffy, but I wanted to test with what I have. So far, it's working well, and I can see traffic being spread over the cards as expected, when using multiple clients. Obviously, if I see issues this is the first thing to change back.

 

[jgreco]

I'm curious what happens with a setup like this?

Having network cards that have significantly differing performance and behaviour characteristics can have all sorts of unanticipated (and potentially hard-to-track-down) side effects.

I've already blown out my daily quota of creative-but-trite network breakage, and I don't really care to theorize any more today about the creative ways in which networking can break when you do something that's commonly advised against. Basically there are reasons that these are commonly advised against, and it's usually because the learner spent days or weeks beating their head against some sort of weird bogon in a production network and then passed it on to their peers, and I try to be a peer rather than a head-beater where possible (but I've been both). After enough head-beatings you start to develop memory loss and then you don't always remember the exact why of every single thing.

As long as you're willing to accept the risk, and as long as you realize that you could run into something that breaks your network in some strange way and requires you to stop doing it, there's no actual harm in trying to do it.

 

[ttabbal]

Fair enough. I'm fine with having to turn it off, and even having that be the default go-to move if something is acting funny. The LACP stuff is an experiment. And if I decide I like the way it works, I'll get a better NIC for it. I'm not opposed to spending the $30 on a known good Intel NIC. I just wanted to make sure the general idea was workable before I did so. Now that I think about it, I should probably order one for my desktop as well. That's using some onboard Marvel something or other at the moment.... You would think good ethernet would be more commonly available, or that at least someone other than Intel would make a decent NIC. Might as well save on shipping and pick up both at the same time.

I read the other thread you linked, interesting. I loved the "I'm on wifi" comment... lol... I know this stuff is more complex than it looks, and it looks complex enough. I haven't done enterprise networking for a while, but have some experience. Enough to know this sort of thing can get weird fast. I only posted to check on known failure modes to be aware of. As it's the last thing I changed, and will be until I decide if I'm keeping it, it's also the first to get kicked out if there are issues. I solemnly swear I won't post about network issues without disabling it and re-testing with the single Intel NIC. :)

 

[pirateghost]

What do you think you are gaining from your LACP?

 

[depasseg]

What do you think you are gaining from your LACP?

Performance improvements on par with the Turbo Encabulator, of course.

 

[jgreco]

I solemnly swear I won't post about network issues without disabling it and re-testing with the single Intel NIC. :)

You get my blessing then ;-)

 

[ttabbal]

Well yeah! Gotta have the turbo.

I'm not positive I'm gaining much, thus the testing. I have had a few times where the single link saturated, causing minor performance issues for various client machines. Things like media playback stuttering. Issues are always while many machines are hitting the server, up to about 8 machines. The utilization log in the web-UI shows the network interface maxed out when the issues crop up. While 10Gbe would be preferred, it's just not worth the money right now for a home user. I would be able to get most of the desired effect by using a VLAN or separate switch and segmenting some of the clients, but if LACP can solve the problem and provide a failover in case a NIC/cable/etc dies, that's a bonus. If not, I'm out nothing but a little time setting it up.

And I may just turn it off and not bother. It's about 40% me wanting to try various capabilities and see how they work. It it works well enough, I'll get a better card and avoid that whole issue. If not, I'll drop it and avoid the issue that way. I know it's not a magic bullet and has limitations. I suspect it might be useful for an edge case, that's all.

 

[jgreco]

Honestly, if you need it, 10GbE isn't that expensive. There's a Dell 5524 on eBay right now for $219, a Chelsio T420-CR is usually $150, so you can get there for maybe $400.

 

[ttabbal]

Thanks for the info on 10G. That's far better than I had found in some simple looking around a little while back. I'll keep it in mind as a future upgrade path.

Did more research on Realtek.. sheesh... I had no idea they were THAT bad. I figured "kinda suck" was being nice, but it's closer to "just shoot yourself". So, yeah, ordering some more Intel NICs than I had planned.

BTW jgreco, the fail thread is excellent. Now I just have to hope I don't earn my way into it.

 

[jgreco]

Well, the thing about the Realtek stuff is that there's such a wide variance in experiences. This might have to do with any of a number of things, ranging from processor speed to user expectations. *shrug*

The fail thread is kind of sad actually, but it's kind of the end result of me repeating the same things over and over and people just disbelieving and being totally defiant because "they know better" and a "prove it" type of belligerence.

As with this thread, here, I don't even have any issue with it if you want to do something questionable/marginal/bad/dumb/stupid/fail-y/catastrophic for anyone grown up enough to own the results of their decisions - in such a case I merely want them to be aware so that the decision is an informed one.

The community here is pretty awesome. You'll almost always get some useful feedback and sometimes it'll be something totally unexpected.

 

[Black Ninja]

Honestly, if you need it, 10GbE isn't that expensive. There's a Dell 5524 on eBay right now for $219, a Chelsio T420-CR is usually $150, so you can get there for maybe $400.

Speaking of Chelsio where you see Chelsio T420-CR for sale ?
I can't find a single card , not on ebay not anywhere ?! I was going to start a new post "Where you guys getting these Chelsio T420-CR cards ?" because everyone with proper nas hardware has them but they are nowhere to be found ?

 

[jgreco]

As with all things eBay, you have to buy them when they're available.

http://www.ebay.com/sch/i.html?_from=R40&_sacat=0&_nkw=t420-cr&LH_Complete=1&rt=nc

Looks like about a dozen listings sold so far this year in the $80-$180 range.

 

[Black Ninja]

As with all things eBay, you have to buy them when they're available. :)

http://www.ebay.com/sch/i.html?_from=R40&_sacat=0&_nkw=t420-cr&LH_Complete=1&rt=nc

Looks like about a dozen listings sold so far this year in the $80-$180 range.

Are you magician too ?:D
How you were able to do this ? I've been checking for awhile on ebay for "chelsio t420 or chelsio t520" and only found one card for $999 ?!?

Was I so stupid to enable a filter without paying attention to screw my results ?! I am not home and I am on a laptop , everything sucks on laptop in my defense.

Does T420 overheats because of the funny small copper heatsink or I should buy T520 which has reasonable heatsink ?

 

[Black Ninja]

As with all things eBay, you have to buy them when they're available. :)

http://www.ebay.com/sch/i.html?_from=R40&_sacat=0&_nkw=t420-cr&LH_Complete=1&rt=nc

Looks like about a dozen listings sold so far this year in the $80-$180 range.

Wait a minute ! All of the listing you show me are already sold. Not single one is available ?!:(

 

[jgreco]

The point is that you just have to keep your eyes open and buy one when they're available. Hitting a dry spell is just part of the whole eBay experience.