SOLVED freenas mini XL+ vnet Jail bridged to vlan extremely slow

EHN Admin

Dabbler
Joined
Jan 29, 2016
Messages
12
I have a brand new freenas mini XL+. My network switch is gigabit. I have a couple of vlans configured on the network.
transfers to host system is very near gigabit speeds.
I created a jail with vnet/vimage which is bridged to one of the vlans. Doing installs of packages such as 'pkg install tmux'
have a reported transfer rate of 8.2 kbps, downloads take forever to finish.
1) Has anyone else experienced this?
2) What can I do to diagnose the problem?
3) What information would be helpful?

Thanks in advance
 

EHN Admin

Dabbler
Joined
Jan 29, 2016
Messages
12
Were you able to resolve this?
No, I have not resolved this. A little more information.
I have tried using iperf to measure network throughput, in a couple situations...
local network(server) to the host(client) ~960 mb/s
local network(client) to the host(server) ~940 mb/s
local network(client) to the jail (server) wide range ~80 mb/s - ~400 mb/s
local network(server) to the jail (client) ~960 mb/s

Without the VLAN everything works fine. It's only with the VLAN in place does it all slow down.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,737
Try to turn off all hardware acceleration features for the physical interface the VLAN is on. Like e.g. for igb
Code:
ifconfig igb0 -rxcsum -rxcsum6 -txcsum -txcsum6 -tso -vlanhwtag -vlanhwtso

If that helps you can add these into the interface setup of the GUI.

HTH,
Patrick
 

EHN Admin

Dabbler
Joined
Jan 29, 2016
Messages
12
Patrick, thank you.
That is a huge help, it is many magnitudes faster. What do these options do? I plan on playing with them to determine which are really necessary.
 

momobozo

Cadet
Joined
Aug 16, 2017
Messages
8
It appears the only one that makes a real difference is the property -vlanhwtag.
Thank you for this. Per the FreeBSD man page:
Code:
     vlanmtu, vlanhwtag, vlanhwfilter, vlanhwcsum, vlanhwtso
         If    the driver offers user-configurable VLAN support, enable re-
         ception of    extended frames, tag processing    in hardware, frame
         filtering in hardware, checksum offloading, or TSO    on VLAN, re-
         spectively.  Note that this must be issued    on a physical inter-
         face associated with vlan(4), not on a vlan(4) interface itself.

     -vlanmtu, -vlanhwtag, -vlanhwfilter, -vlanhwtso
         If    the driver offers user-configurable VLAN support, disable re-
         ception of    extended frames, tag processing    in hardware, frame
         filtering in hardware, or TSO on VLAN, respectively.


Seems like this is a bug in FreeBSD that hasn't been resolved yet: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=230996
 

listhor

Contributor
Joined
Mar 2, 2020
Messages
133
Im sorry for resurrecting an old thread but I'm facing exactly same problem: dead slow tcp performance in iocage jail behind vnet + bridge. My configuration, lagg on two physical interfaces, vlan11 with parent lagg and being a member of bridge11:
Code:
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: LAN3
options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d9:d5:60
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: LAN2
options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d9:d5:60
    hwaddr ac:1f:6b:d9:d5:61
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
lagg4095: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Trunk
 options=e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether ac:1f:6b:d9:d5:60
    laggproto lacp lagghash l2,l3,l4
    laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    groups: lagg
    media: Ethernet autoselect
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
vlan11: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Zasoby
    options=200401<RXCSUM,LRO,RXCSUM_IPV6>
    ether ac:1f:6b:d9:d5:60
    inet 172.16.1.62 netmask 0xffffffc0 broadcast 172.16.1.63
    groups: vlan
    vlan: 11 vlanpcp: 2 parent interface: lagg4095
    media: Ethernet autoselect
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
bridge11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: zasoby
    ether 02:11:2a:4b:ab:0b
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.6 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 19 priority 128 path cost 2000
    member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 17 priority 128 path cost 2000
    member: vlan11 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 8 priority 128 path cost 2000000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: nextcloud as nic: epair0b
    options=8<VLAN_MTU>
    ether ae:1f:6b:ba:b5:81
    hwaddr 02:3e:a9:17:2f:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=1<PERFORMNUD>
vnet0.6: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: test2 as nic: epair0b
    options=8<VLAN_MTU>
    ether ae:1f:6b:4b:48:3c
    hwaddr 02:27:12:04:90:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=1<PERFORMNUD>

Jail has interface set as vnet0:bridge11. In general connectivity is fine but for example pkg downloads packages with speed of 8.2 kb/s...
When I disable VLAN_HWTAGGING on igb interfaces pkg speeds up to over 1 Mb/s but host CPU usage goes up quickly.
Is there any other way to assign IP vlan address to jail??
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,737
The please disable hardware offloading and keep the setting that way. This is the official documented solution.
 

listhor

Contributor
Joined
Mar 2, 2020
Messages
133
The please disable hardware offloading and keep the setting that way. This is the official documented solution.
Thanks, but this is rather workaround and not a solution... Heavy traffic may max out CPU. Thus the only solution is to revert to shared IP or move towards dockerized solutions on other platform??
As much as I like ZFS features I'm disappointed with Free/TrueNAS usability. It seems like main purpose is to run it only for data sharing and backup purposes. VMs/bhyve and jails they are still behind other solutions (proxmox, esxi and docker). Maybe I'm wrong - I will gladly admit it if it's proved otherwise :smile:

EDIT:
I went for simple solution, disabled vnet and assigned in jail alias for existing in host vlan (default gateway is also in vlan11, ssh and GUI are in other vlan):
Code:
iocage set ip4_addr="vlan11|172.16.1.2/26" defaultrouter="172.16.1.1" nextcloud


EDIT2:
Above mentioned vlan11 must be a member of bridge interface (all done in GUI), otherwise alias won't be assigned...
 
Last edited:

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,737
HW offloading does not make sense if the interface is only a layer 2 link and there are more layers like lagg above it in the stack. With the VLAN directly on the physical if and no bridge, it should work. In other situations it's known to interfere with the virtual interfaces further up the stack.

You can get all of this annoying layer 2 stuff off your physical link in FreeBSD by creating a bridge without any physical members and using routing. We use that in production and it is perfectly stable. I'd have to try if you can create such a setup in the FreeNAS context. You would definitely have to use some tunable hacking to create the bridge and iocage to configure the jails, but it could work.

Expect dramatically improved bridge performance and less CPU use in FreeBSD 13. Unfortunately the changes cannot be backported, because they need FreBSD 13 kernel changes.

BTW, what do you consider "high cpu", anyway? Just curious. I mean that chip is there to do work, right? :wink:
 

listhor

Contributor
Joined
Mar 2, 2020
Messages
133
HW offloading does not make sense if the interface is only a layer 2 link and there are more layers like lagg above it in the stack. With the VLAN directly on the physical if and no bridge, it should work. In other situations it's known to interfere with the virtual interfaces further up the stack.

You can get all of this annoying layer 2 stuff off your physical link in FreeBSD by creating a bridge without any physical members and using routing. We use that in production and it is perfectly stable. I'd have to try if you can create such a setup in the FreeNAS context. You would definitely have to use some tunable hacking to create the bridge and iocage to configure the jails, but it could work.

Expect dramatically improved bridge performance and less CPU use in FreeBSD 13. Unfortunately the changes cannot be backported, because they need FreBSD 13 kernel changes.

BTW, what do you consider "high cpu", anyway? Just curious. I mean that chip is there to do work, right? :wink:
Thanks again, I was using bridge created in tunables but over a year ago, after one of FreeNAS-10 or 11 updates I had a boot loop, so I stopped using it. That's why I've been using vnet+bridge. And when I disabled hwtagging on igb interfaces (lagg physical ifaces) I noticed immediately in TrueNAS dashboard with nextcloud "idling" that average CPU usage jumped from 7-10% to 20%. I was observing it for half an hour only but decided to revert changes and now I use alias vlan interface (described in previous post). What are disadvantages, promiscuous mode?
 

ChrisRJ

Wizard
Joined
Oct 23, 2020
Messages
1,903
As much as I like ZFS features I'm disappointed with Free/TrueNAS usability. It seems like main purpose is to run it only for data sharing and backup purposes. VMs/bhyve and jails they are still behind other solutions (proxmox, esxi and docker).
Yes, FreeNAS/TrueNAS is a storage appliance. So complaining about it being more complicated to use for virtualization, is like saying that Proxmox could have better usability for being used as a storage appliance. Both statements are true, but so is complaining that a sports car cannot carry as much load as a lorry. TrueNAS Scale, if I understand things correctly, seems to be what you would like to have.
 

listhor

Contributor
Joined
Mar 2, 2020
Messages
133
Yes, FreeNAS/TrueNAS is a storage appliance. So complaining about it being more complicated to use for virtualization, is like saying that Proxmox could have better usability for being used as a storage appliance. Both statements are true, but so is complaining that a sports car cannot carry as much load as a lorry. TrueNAS Scale, if I understand things correctly, seems to be what you would like to have.
I guess the truth is somewhere in the middle. But I think that having efficient vnet stack isn't anything extraordinary?
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,737
What are disadvantages, promiscuous mode?
Disadvantages of alias IP addresses instead of vnet? No isolated network stack, so no ipfw rules inside the jail, no lo0 interface inside the jail ... some software expects that. Our hosting customers definitely do. Being able to bind Elasticsearch server to 127.0.0.1 etc. ...

There nothing inherent in vnet that mandates bridging. You could assign an pair of IP adresses in a /30 or to an epair(4) or probably (not yet tried) even do in "unnumbered" (re-using with /32) on the host side and add a host route to the jail ... whatever sort of networking suits your fancy.

Our plan for our hosting environment is to continue with if_bridge(4), but make all the bridges private (no connection to any physical) and use dynamic routing to get jail mobility. This scales way better than putting all jails "on the wire". Large broadcast domains (layer 2 networks) are bad. This case has long been settled. One should try to avoid them when possible.

A small home network is a different thing altogether, but I definitely would like more flexibility in TrueNAS. Tried to put a jail on a different network. TrueNAS ended up creating a "bridge0" and throwing arbitrary member interfaces in there, anyway. We have way better control over this in our own environment, of course. What we don't have on the other hand is any UI for that. It's all Ansible ...
 

listhor

Contributor
Joined
Mar 2, 2020
Messages
133
Disadvantages of alias IP addresses instead of vnet? No isolated network stack, so no ipfw rules inside the jail, no lo0 interface inside the jail ... some software expects that. Our hosting customers definitely do. Being able to bind Elasticsearch server to 127.0.0.1 etc. ...
(...)
A small home network is a different thing altogether, but I definitely would like more flexibility in TrueNAS. Tried to put a jail on a different network. TrueNAS ended up creating a "bridge0" and throwing arbitrary member interfaces in there, anyway. We have way better control over this in our own environment, of course. What we don't have on the other hand is any UI for that. It's all Ansible ...
Being honest, I don't know much about network logic used in Free/TrueNAS. I wish it would be as simple as macvlan with docker.
An I would like to stay away from "manual" input/overrides (creating extra interfaces in host and/or routes between them) in cli as those might would not survived upgrades. Could you explain further this part:
You could assign an pair of IP adresses in a /30 or to an epair(4) or probably (not yet tried) even do in "unnumbered" (re-using with /32) on the host side and add a host route to the jail
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,737
Could you explain further this part:
This is all outside the FreeNAS/TrueNAS context, of course. I am not quite sure if TN can be forced to configure networking that way, but possibly. Would have to experiment in a VM a bit and I have other things on my "almost-but-not-quite-work-because-fun" list of hacking to do, first.

But generally: vnet relies on the epair(4) interface, which is essentially a pair of virtual interfaces that are connected by a virtual wire. If I look at the end in my Nextcloud jail, it looks like this:
Code:
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
[...]
    ether 02:ff:60:07:4c:d7
    hwaddr 02:cd:ad:3d:16:0b
    inet 192.168.1.53 netmask 0xffffff00 broadcast 192.168.1.255
[...]

On the host there is the other end of that one:
Code:
vnet0.2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: cloud as nic: epair0b
[...]
This interface does not have any IP address!

It's just the other end of that "virtual cable" and it is plugged into a "virtual switch", namely bridge0:
Code:
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
[...]
    member: vnet0.2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
[...]
    member: vlan100 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
[...]

So the "bridge0" connects the jail's interface with my local LAN on VLAN 100 - could be anything like em0, igb0, ... instead. Physical network. So the jail participates in the LAN's broadcast domain. ARP, IPv6 neighbour discovery, has got it's own default gateway(s), can have additional routes and firewall rules etc.

The downside is that with 100s of jails that one "LAN" becomes crowded. So more and more bandwidth is wasted for all these broadcast/multicast protocols: ARP, neighbour discovery, ... the "virtual switch", i.e. the FreeBSD "bridge" interface needs to forward every single ARP request on the wire to every single jail. Because broadcast ...

So why not get rid of this bridging and broadcasting altogether? If not in the TrueNAS context, a jail created with an epair interface get's e.g. epair0b inside the jail while the host gets epair0a as its "end" of the interface.

So picture:

em0/igb0/whatever on the host: 192.168.0.1/24
epair0a on the host: 192.168.1.1/30
epair0b inside the jail: 192.168.1.2/30

That way the host and the jail can talk on layer 3 over that epair "wire" with the two configured IP addresses. For the jail to be able to reach the outside world the host must be configured to be a router (trivial, any Unix can do that) and the jail's default gateway must point to 192.168.1.1.
Then we need to tell the rest of the world that this jail exists on this host, so we need a route known in all other hosts and/or routers on our network that reads:

192.168.1.0/30 (that's .0 .1 .2 .3 of which .1 and .2 can be used) goes to ---> 192.168.0.1 (the IP address of our jail host).

Tadah! Jail with vnet and its own address without bridging. Now the downside is that with bridging if you move a jail from one host to a new one, if all are on the same LAN you have IP/jail mobility. At the cost of all this broadcast/ARP mumbo jumbo. But it's trivial to implement and it "just works" for reasonably small environments.
If you want that mobility with routed connectivity as laid out above, then you need a dynamic routing protocol like OSPF or BGP and you need to trust your jail hosts in participating in that routing protocol without messing up your network.

But essentially that's what all larger cloud infrastructures do. Use dynamic routing to connect containers or VMs. Docker is no different. If you want container mobility you need to publish those IP addresses somehow. Only in the case of a single docker host it looks simpler than our jails but actually isn't. Because you can do the very same with jails. This is what TN does when you run a jail/plugin with NAT.

There is the bridge interface (again) but it's not connected to the outside world. Alle the jails are put on that bridge/switch. And the bridge interface itself has got an IP address and is the default gateway for the jails. And then you NAT on the host to get the traffic flowing.


Phew, in German we ask tounge-in-cheek "alle Klarheiten beseitigt?", i.e. "all clarity successfully eliminated?" after such a lecture :wink: . Hope that shed some light on the topic for you (and possibly others).
 
Last edited:

listhor

Contributor
Joined
Mar 2, 2020
Messages
133
Thanks very much @Patrick M. Hausen ! I need to read it at least twice or more and then try to apply it on test jail... And at least I know general logic behind jails vnet...
 
Top