Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.
Resource icon

Resource Jumbo frames notes

We got into a little back-and-forth on jumbo frames on the 10 gig thread. I don't know if anyone cares about this but I thought I'd throw some info out there.

IP network packets have a Maximum Transfer Unit (MTU), which on a conventional Ethernet interface is 1500 bytes. Ethernet is not the only transport available for IP, but it is fairly common. IP endpoints are expected to be able to ascertain the smallest MTU supported by the intermediate networks, most recently using technologies such as PMTUD.

In the early days of Ethernet, we moved fairly rapidly from 10Mbps to 100Mbps and beyond. This was great for throughput but had the horrifying side effect of increasing the number of packets per second gear was expected to cope with. Especially on a hub network, this was ... problematic. Further, CPU and interrupt load on the endpoints were rapidly increasing as speeds increased. Someone recognized that putting more data in a single packet was a fix. Thus was born ... Jumbo frames!

Skipping a bunch of history, though, I want to point out: in moving from 10Mbps to 10Gbps Ethernet, we have increased the network throughput by a factor of 1000. Jumbo typically increases the frame size from 1500 to something probably not larger than 9216, which is only about 6 times larger. Jumbo sizes have not kept pace with throughput increases, and therefore potential packets per second continue to increase with throughput.

Today, jumbo primarily works to slightly increase the data-to-overhead ratio of an Ethernet network, moving network efficiency from about 95% (1500 mtu) to around 99% (9000 mtu). The reduction in CPU and interrupt processing has largely been nullified by the introduction of large segment offload (LSO) and large receive offload (LRO), and hub oriented technologies have virtually disappeared. Modern switch silicon may be capable of handling a gigapacket per second or more, all at wire speed.

Layer 2

Layer 2 refers to the Data Link Layer. For Ethernet, you can think about this as the network which is formed by attaching all the member devices to a switch. All of these devices must agree on certain parameters, and rather importantly all of them must agree on and be able to support the MTU the administrator wishes. If you have two devices on the network, one of which is capable of and configured for a 9000 MTU, but the other device cannot even process 9000, everything will appear to work until a TCP or UDP packet larger than 1500 is sent to that limited device, at which point the packet is lost.

Jumbo frames must be configured identically on all devices attached to a jumbo network, or erratic and unpredictable behaviour may result.

Layer 3

Layer 3 refers to the Network Layer. For Ethernet, these are routed networks. Please note that what many end users refer to as a "router" is actually a NAT gateway and not a router at all. A router is a device that performs routing, the act of selecting the best path for a packet and moving it on to its next hop.

Routers are what make the Internet possible. They connect together discrete networks, both within an organization and externally. Routers are the natural boundary point of a network's MTU, and therefore the point at which a packet might encounter a different MTU. A router may be expected to perform fragmentation or forward packets from a network with one MTU to another. I'll come back to this in a bit.

Modern jumbo and issues

A lot of modern equipment supports jumbo, but when I say that, I really mean "supports." It is a technical feature on a checklist, and sometimes the reality is that it doesn't work as well as you'd expect. Or at all.

One of the problems with jumbo is that on much i386/amd64 hardware, jumbo is treated differently from standard traffic. In some of the FreeBSD drivers, for example, buffers for jumbo traffic are allocated/handled differently (in some cases using fixed parameters) than standard traffic. This can lead to different behaviours for a host's connection to the network depending on whether or not jumbo has been enabled.

Another is that you might decide to design a storage network as jumbo, which is actually a fairly sensible thing to do. However, if you then need to introduce a device that doesn't support jumbo, or supports jumbo but with a smaller MTU than you've selected, you're hosed.

The real problem is dealing with the layer 3 issue, however. If you have a NAS such as FreeNAS, it probably expects to be able to access your DNS server, your mail server, etc. If those are not on the jumbo network, and your router cannot properly cope with being the MTU gatekeeper, you can run into mysterious issues such as the test mail working but crisis notification mails not working. That's very bad.

So in recent years, I've seen a tendency for network administrators to ditch jumbo. It adds a lot of complexity and potential issues to a network for a modest boost in performance.

Here at grinch central, the move towards layer 3 switching revealed some unhappy issues with network MTU handling by layer 3 switches: they assume a chassis-wide MTU. This appears to be an ugly issue with a lot of layer 3 switches.


Well, my conclusion, anyways: Screw jumbo.

Jumbo might be fine for a totally dedicated storage network without any external routing requirements, but even there it is quite possible to run into issues with how FreeBSD handles jumbo traffic differently.

Jumbo is a technology that served a valid purpose at one point, but hardware offload and wire speed routing have significantly eaten away at the advantages it once offered. It will offer you a modest performance boost if you can get it configured properly, but that time might be better spent designing your network topology differently.
Thanks: mccann73
First release
Last update
3.50 star(s) 2 ratings

More resources from jgreco

Latest reviews

Thanks! I think you summed this up quite nicely. It certainly aligns with my personal experiences.
Some of the listed limitations do not apply with current O/S's. Example: MTU of different sizes will cause failures. I use different size jumbo frames all the time (4K, 7K, 9K, etc.) without a problem...
If you attempt to send a 9K jumbo frame to a host whose hardware cannot receive larger than 4K, you're going to have problems. I've debugged too many issues with jumbo frames on networks over the years, and lots of them come down to people thinking what you've just said, but the reality being somewhat different. The damnable thing about it is that problems only show up when a sufficiently large jumbo packet is sent, or sometimes even large numbers of jumbo packets. This results in curious things such as "we just added wifi and it doesn't work" or "the printer randomly doesn't work."

Most of the truly important reasons that jumbo is significant have to do with interrupt load, and most of the current hardware and drivers have various mitigation strategies such as interrupt coalescing and packet processing offload features. Today's modern systems aren't taxed by it in the same way that an old 2005-era PCI based system was. As a result, there's much less value to dealing with all the annoyance of jumbo.