Jumbo frames on 10GbE

webdawg

Contributor
Joined
May 25, 2016
Messages
112
I read the stickies. I understand that jumbo is not recommended anymore, but I have built a back-end 10gbit switching fabric and I have some questions.

Lets say I have 2 vlans. One is Jumbo, and one is not as it gets pushed to standard gbit ethernet network, and who knows if everything is jumbo there (hint, it is not)...

I have 10 gbit interface
vlan 1
vlan 2

Can I have seperate MTU's on the VLANs?

IE vlan 2 is just for the san/fabric part of the network. Pure 10gbit, but vlaned. - If I set this to 9000 mtu does this get respected....

But what if I set vlan 1 to 1500 so everything else on my gbit network likes it....

Does it mess up the jumbo stuff?

You can set MTU's for interfaces AND vlans. So what should I do there. What respects what?
 
Joined
Dec 29, 2014
Messages
1,135
Even if your switch thinks the MTU is 9000, it doesn't matter if the hosts on the other VLAN's are using an MTU of 1500. The big problem is is a host wants an MTU of 9000 across the entire path, but it isn't there. Hopefully that makes sense.
 

webdawg

Contributor
Joined
May 25, 2016
Messages
112
Okay,

So I guess the first question (two issues here), comes down to this:

Can a single network card if VLANed properly operate across different VLANs with different MTU's.

VLAN1 = 1500 MTU
VLAN2 = 9000 MTU

Does this work?

If the answer to this is yes:

What is the proper configuration, and can a MTU JUST be attached to a VLAN?

Do I set NO MTU for the interface0?

Do I set 1500 MTU for vlan1? and then...

Set 9000 MTU for vlan2?

I suppose another question is: does the switch respect different MTU sizes across different VLANS heh. I guess this is where the 'path' part (that you mentioned) comes in? 9000 MTU packets, and 1500 MTU packets mixing across the same physical path making the entire conversation moot?
 
Joined
Dec 29, 2014
Messages
1,135
Can a single network card if VLANed properly operate across different VLANs with different MTU's.
It depends on the OS and the NIC. Most likely, yes.
I suppose another question is: does the switch respect different MTU sizes across different VLANS
Typically, the MTU maximum is set at a global level in the switch for layer 2 frames. MTU on a layer 3 interface (like "interface vlan 51" in Cisco speak) would be a different thing. The biggest thing (IMHO) is the later 2 aspect. Thing about a 3 switch configuration like below.

S1 <-> S2 <-> S3

Assume S1 and S3 have jumbo frames enabled, but S2 does not. Hosts durectly connected to S1 could pass jumbo frames to each other. Hosts directly connected to S3 could pass jumbo frames to each other. Those hosts would not be able to pass jumbo frames to hosts in the other switch because the underlying transport doesn't support it. Even if jumbo frames are enabled, all the switches will happily pass frames with sizes smaller than the MTU max.
 

webdawg

Contributor
Joined
May 25, 2016
Messages
112
The switches that I am using have no configuration settings for Jumbo frames, but I think I understand the switch piece.

All the switches are the same vendor (MikroTik).

What I am not fully educated on right now is how the Layer 3 MTU setting is handled across the 10 gbit switching fabric. If I have the 9000 MTU equipment on one VLAN (layer 2 setting), and all the 1500 MTU equipment on another VLAN...but they both traverse the same Layer 1 (physical 10Gbit fiber) network...then how does the switch handle the 1500 and 9000 different payload sizes on the same phyiscal layer?

Sure they are separated @ layer 2 via vlan, but for jumbo to work doesn't everything have to be 9000 for layer 1, 2, and 3?
 
Joined
Dec 29, 2014
Messages
1,135
There is no such thing as MTU at layer 1. If there isn't an option for jumbo frames, check with the product features to see if it supports jumbo frames. If not, this whole discussion is moot. Any device at layer 2 or layer 3 will accept frames/packets that don't exceed its maximum. There are only issues when you try to use more than what is normal for ethernet (1500), and something along the path doesn't support that.
 

webdawg

Contributor
Joined
May 25, 2016
Messages
112
Thanks for keeping the conversion going. I get that there is no such thing as MTU at layer one. I just stated that to illustrate that both VLANS are on the same physical medium. The devices do support jumbo frames. I get all this.

What I do not understand, is how across the same core backbone that 1500 mtu packets, and 9000 mtu packets mix, while walled via vlan tags, but do not interfere/effect switch/link performance.
 

webdawg

Contributor
Joined
May 25, 2016
Messages
112
I converted everything over to 1500 mtu just to test, and it is slower. Same performance really. Until I get the question answered: If FreeBSD VLAN interfaces respect MTU, and override core MTU settings I think I will keep the same. I may revisit this again in the future.
 
Joined
Feb 22, 2017
Messages
27
Basically, VLANs are different logical networks laid on top of a single physical network.

The basic gist is that when you tag traffic for a VLAN, what the hardware literally does is take the bits, put them in an ethernet frame that is less than the MTU for the VLAN, then puts adds a header and footer (encapsulates) to the frame that says "this frame belongs to VLAN X". The switch then gets it, rips the VLAN encapsulation off and says "does this frame fit the MTU for VLAN X? Yes? Okay, off we go!" It then re-encapsulates the frame and send it out the required port to get the destination frame on its way.

Generally what one does is set the MTU on the main interface to the jumbo size, then if you really want, you set the MTU smaller on your VLAN (aka sub) interface. However, ethernet and the like have mechanisms that make setting the MTU on the sub interface rather unneeded.
 
Top