Too many LACP discussions lately. Here's what you need to know.
1) Link aggregation groups are an interesting way to boost network throughput. They're typically managed by a protocol called Link Aggregation Control Protocol (LACP). Link aggregation is designed to bundle several interfaces together to allow networking gear, servers, etc., to benefit from multiple links. Link aggregation can be used as master/standby for network redundancy, or it can be used to increase network bandwidth. Master/standby is easily accomplished through "failover" mode. The remaining discussionin this post primarily focuses on increasing network bandwidth.
2) LACP REQUIRES BOTH SIDES OF A CONNECTION TO SUPPORT LACP. This means you need a managed or smart switch. Typical consumer grade switches do not support LACP.
3) High speed ethernet network performance is substantially impacted by out-of-order packet delivery. While this happens quite a bit on long hauls across the Internet, it causes massive problems for the TCP segment reassembly on the far end, and substantially reduces performance.
3a) As a result, IEEE 802.3ad reads:
This standard does not mandate any particular distribution algorithm(s); however, any distribution algorithm shall ensure that, when frames are received by a Frame Collector as specified in 5.2.3, the algorithm shall not cause:
a) misordering of frames that are part of any given conversation, or
b) duplication of frames.
The above requirement to maintain frame ordering is met by ensuring that all frames that compose a given conversation are transmitted on a single link in the order that they are generated by the MAC Client; hence, this requirement does not involve the addition (or modification) of any information to the MAC frame, nor any buffering or processing on the part of the corresponding Frame Collector in order to reorder frames.
3b) In FreeBSD, a well-designed hash is used to distribute flows between ports. All traffic for a given flow will be transmitted by one particular interface. The hash includes the Ethernet source and destination address and, if available, the VLAN tag, and the IPv4 or IPv6 source and destination address. This is a nice stateless method to manage distribution of flows. The astute reader will note that a stateful mechanism could possibly be used on a server, but FreeBSD fills many networking roles, and for many of those roles, a stateful mechanism would be inappropriate.
3c) The attached switch probably implements a completely different hashing algorithm, so return traffic may well not flow back to the same interface.
3d) It is possible to instruct FreeBSD to use "roundrobin" hashing, to accomplish what most end users seem to expect LACP to do. Unfortunately, this ultimately results in violations of the misordering prohibition noted in 3a), and while it often appears to work swimmingly well on a lightly loaded network, it typically devolves into rapid hell as the network gets busy, because packet queuing delays become a major factor - just the thing you DON'T want to happen.
3e) If you don't understand that point 3) encompasses and requires all of this, please reread until you do.
4) So the basic problem many people experience is that in order to get a good distribution of traffic, you need more than just two clients. If you have two clients, there is a 50% chance that both flows will end up on the same link of a dual-link LACP connection. If you have three clients, at least two will end up on one connection while another probably has a connection all to itself - but! It is still possible for all three to end up on the same link! And what client generates network traffic 100% of the time? In practice, it is usually difficult to get link aggregation to work well until you have at least a dozen clients and a very busy network.
5) LACP adds another layer of complexity to the networking configuration. In practice, we've seen users try to combine two different interface types (such as sis and em) and while this should theoretically work, it is bad to do in practice. Ideally a lagg interface should be created from two neighboring interfaces of the same type, configured identically, without any IP addresses, just options "up".
6) I find that most users would be better served by a quick upgrade to 10G connectivity to their server. Done properly, this allows a server to deliver 1G reliably to or from multiple clients simultaneously.
7) This discussion is also closely related to putting multiple interfaces on a single subnet, a strategy people often believe works until they find out it doesn't work the way they were expecting.
I will be aggressively pruning responses to this thread, especially pointlessly argumentative ones. It is what it is. I didn't write the standard though I agree with it, and also agree with the implementation decisions. I have been operating a LACP based network for many years and find the technology quite agreeable, but I also understand that it is hard for a newcomer to totally grasp. I don't mind expanding this if it turns out my explanation is deficient in some way (which it probably is).
1) Link aggregation groups are an interesting way to boost network throughput. They're typically managed by a protocol called Link Aggregation Control Protocol (LACP). Link aggregation is designed to bundle several interfaces together to allow networking gear, servers, etc., to benefit from multiple links. Link aggregation can be used as master/standby for network redundancy, or it can be used to increase network bandwidth. Master/standby is easily accomplished through "failover" mode. The remaining discussionin this post primarily focuses on increasing network bandwidth.
2) LACP REQUIRES BOTH SIDES OF A CONNECTION TO SUPPORT LACP. This means you need a managed or smart switch. Typical consumer grade switches do not support LACP.
3) High speed ethernet network performance is substantially impacted by out-of-order packet delivery. While this happens quite a bit on long hauls across the Internet, it causes massive problems for the TCP segment reassembly on the far end, and substantially reduces performance.
3a) As a result, IEEE 802.3ad reads:
This standard does not mandate any particular distribution algorithm(s); however, any distribution algorithm shall ensure that, when frames are received by a Frame Collector as specified in 5.2.3, the algorithm shall not cause:
a) misordering of frames that are part of any given conversation, or
b) duplication of frames.
The above requirement to maintain frame ordering is met by ensuring that all frames that compose a given conversation are transmitted on a single link in the order that they are generated by the MAC Client; hence, this requirement does not involve the addition (or modification) of any information to the MAC frame, nor any buffering or processing on the part of the corresponding Frame Collector in order to reorder frames.
3b) In FreeBSD, a well-designed hash is used to distribute flows between ports. All traffic for a given flow will be transmitted by one particular interface. The hash includes the Ethernet source and destination address and, if available, the VLAN tag, and the IPv4 or IPv6 source and destination address. This is a nice stateless method to manage distribution of flows. The astute reader will note that a stateful mechanism could possibly be used on a server, but FreeBSD fills many networking roles, and for many of those roles, a stateful mechanism would be inappropriate.
3c) The attached switch probably implements a completely different hashing algorithm, so return traffic may well not flow back to the same interface.
3d) It is possible to instruct FreeBSD to use "roundrobin" hashing, to accomplish what most end users seem to expect LACP to do. Unfortunately, this ultimately results in violations of the misordering prohibition noted in 3a), and while it often appears to work swimmingly well on a lightly loaded network, it typically devolves into rapid hell as the network gets busy, because packet queuing delays become a major factor - just the thing you DON'T want to happen.
3e) If you don't understand that point 3) encompasses and requires all of this, please reread until you do.
4) So the basic problem many people experience is that in order to get a good distribution of traffic, you need more than just two clients. If you have two clients, there is a 50% chance that both flows will end up on the same link of a dual-link LACP connection. If you have three clients, at least two will end up on one connection while another probably has a connection all to itself - but! It is still possible for all three to end up on the same link! And what client generates network traffic 100% of the time? In practice, it is usually difficult to get link aggregation to work well until you have at least a dozen clients and a very busy network.
5) LACP adds another layer of complexity to the networking configuration. In practice, we've seen users try to combine two different interface types (such as sis and em) and while this should theoretically work, it is bad to do in practice. Ideally a lagg interface should be created from two neighboring interfaces of the same type, configured identically, without any IP addresses, just options "up".
6) I find that most users would be better served by a quick upgrade to 10G connectivity to their server. Done properly, this allows a server to deliver 1G reliably to or from multiple clients simultaneously.
7) This discussion is also closely related to putting multiple interfaces on a single subnet, a strategy people often believe works until they find out it doesn't work the way they were expecting.
I will be aggressively pruning responses to this thread, especially pointlessly argumentative ones. It is what it is. I didn't write the standard though I agree with it, and also agree with the implementation decisions. I have been operating a LACP based network for many years and find the technology quite agreeable, but I also understand that it is hard for a newcomer to totally grasp. I don't mind expanding this if it turns out my explanation is deficient in some way (which it probably is).
