TrueNAS 12.0-STABLE - Set up VLANs

[Newfoundland.Republic]

What I did last night is create a new vlan25, created bridge25 with an IP of 192.168.25.253. Then I created the jail with vnet0/bridge25:ven0 and assigned 192.168.25.252, etc. to the jail. Everything seems to work.

I was hoping that I could create the bridge on the untagged 10.100.200.x (e.g., 10.100.200.250 - bridge200) and then assign the VLANs (20, 25, 100, etc.) to bridge200. But, it doesn't seem like that will work.

 

[Patrick M. Hausen]

I was hoping that I could create the bridge on the untagged 10.100.200.x (e.g., 10.100.200.250 - bridge200) and then assign the VLANs (20, 25, 100, etc.) to bridge200. But, it doesn't seem like that will work.

Nope. I don't know if the tagged frames will be forwarded by the bridge. Technically there is no reason they would not. But then your jail would need to send and receive tagged frames, too!

That's why it doesn't work. Again, think "switch". Port based VLAN ("VLAN interface") plugs into bridge, everything here is untagged. Jails plug into the same bridge.

Physical --> LAGG (if present) --> VLAN --> Bridge

 

[Newfoundland.Republic]

Thanks Patrick - you've been a great help. Two Thumbs Up!

Now, I'm marking this thread closed.

 

[NinthWave]

@Patrick M. Hausen

Hello,
The thread has not been marked closed or solved so I'll ask my own questions since I am trying to do about the same.

So I have pfSense as a firewall|router and a T610 running TrueNAS bare metal. At the time of first installation, there was only one LAN and everything fell under. Now, I have segmented my network and I'd like some jails/VMs to run under their related vlans. Since my T610 has 2 NICs and that I can use the second one (bce1) to avoid messing everything, I would like to go that way.

bce0 is used to manage TrueNAS and is set as DHCP while pfSense gives it a static IP.

Q0: Why is it that in «/Interfaces», there is no bridge0 but my actual jails refers to bridge0 ?

Q1: Since bce1 will only hosts vlans or VMs; I guess it should be set similar to bce0 (DHCP). But then, each jail will have it's own vtnet IP; so why bce1 should have it's own IP, it won't be managed ?

Q2: Disable Hardware Offloading When one wants to tick that box, it says that it could degrade networking performance and would only be recommended for VMs/VLANs. Well, this is exactly the sole purpose of bce1 in my intended purposes. Should I tick that box ?

Q3: Then, in «/Interfaces», I should create VLAN10 and VLAN20 ?

Q4: Then, in «/Interfaces», I should create bridge10 and bridge20 ?

Q5: For, lets say bridge10, should the member(s) be solely vlan10 or ( vlan10, bce1 ) ?

 

[Patrick M. Hausen]

A0: bridge0 is created automatically. If you run anything but the default setup of one single interface for everything, you need to create your bridges and assign jails manually. Best disable autostart for all jails and VMs and reboot to get a clean state without any bridge IF active.
A1: a member interface of a bridge MUST NOT have an IP address. And of course no DHCP either.
A2: yes of course. You must disable hardware offloading for bce1.
A3: yes
A4: yes
A5: only the VLAN interfaces become members of the bridge interface

A6 which you did not ask: in your VM configuration simply assing bridge10 or bridge20 to the appropriate device. For the jails change vnet_default_interface from "auto" to "none" (important!) and explicitly set bridge10 or bridge20 further down in the "interfaces" setting.

 

[NinthWave]

So if I understand this correctly, as soon as one wants to run any virtualization through VMs or jails, it is definitely recommended to have two NICs: one for TrueNAS management / share and the other one for all under virtualization ?

 

[Patrick M. Hausen]

No, why? You can easily run all on one Gigabit NIC which is the default.

Edit: what exactly are you concerned about? If you fear your file sharing performance will be negatively impacted by "disable hardware offloading" - it won't. Hardware offloading is a micro optimization that is not noticeable at 1 Gbit/s speed and with a modern CPU. The calculations will be done by your main CPU instead of the NIC. That's it. You will be able to saturate that Gbit link, nonetheless.

See for example this study by VMware:

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/whitepaper/vmware-vsphere-pnics-performance-white-paper.pdf

While figure 6 looks impressive we don't use VXLANs in TrueNAS and figure 7 is much more relevant to our use case. So you are going from 11% CPU overhead for received packets to 15%. And most NAS use cases are anything but CPU bound. Meaning there are plenty of CPU cycles left. Hence my claim that you won't notice.

 

[NinthWave]

No, why?

Can I keep software offloading on for bce0 which is for administration and shares
And disable it for bce1 that will handle all VM ?


In my case, I am just happy top have two NICs to lower the risks of me messing access to TrueNAS itself while I configure my VLANs.

 

[Patrick M. Hausen]

You can leave hardware offloading enabled for bce0 and disable it for bce1. Be careful about the change in the jail settings: vnet_default_interface: none. If not you might re-create the "automatic" bridge and bridge your two physical interfaces wich will probably bring down your network.

I went all out to write at length about how it all fits together, here:

VLANs Bridges and LAG Interface best practice questions

I have a working Truenas 12.0U3 installation. The installation has as follows (note I used this reference documentation https://www.truenas.com/community/threads/how-to-setup-vlans-within-freenas-11-3.81633/ ) Switch => LAG (gig24 + gig 25) => Trunk vlan 1-1000 => TRUENAS Server VLAN 1 is the...

www.truenas.com

 

[NinthWave]

@Patrick M. Hausen

Hello,
The thread has not been marked closed or solved so I'll ask my own questions since I am trying to do about the same.

So I have pfSense as a firewall|router and a T610 running TrueNAS bare metal. At the time of first installation, there was only one LAN and everything fell under. Now, I have segmented my network and I'd like some jails/VMs to run under their related vlans. Since my T610 has 2 NICs and that I can use the second one (bce1) to avoid messing everything, I would like to go that way.

View attachment 50382

bce0 is used to manage TrueNAS and is set as DHCP while pfSense gives it a static IP.

Q0: Why is it that in «/Interfaces», there is no bridge0 but my actual jails refers to bridge0 ?

Q1: Since bce1 will only hosts vlans or VMs; I guess it should be set similar to bce0 (DHCP). But then, each jail will have it's own vtnet IP; so why bce1 should have it's own IP, it won't be managed ?

Q2: Disable Hardware Offloading When one wants to tick that box, it says that it could degrade networking performance and would only be recommended for VMs/VLANs. Well, this is exactly the sole purpose of bce1 in my intended purposes. Should I tick that box ?

Q3: Then, in «/Interfaces», I should create VLAN10 and VLAN20 ?

Q4: Then, in «/Interfaces», I should create bridge10 and bridge20 ?

Q5: For, lets say bridge10, should the member(s) be solely vlan10 or ( vlan10, bce1 ) ?

@Patrick M. Hausen

VLANs Bridges and LAG Interface best practice questions

[...] The reason not to use the "native VLAN" is that first pf can get confused (rules on the untagged interface "catch" tagged traffic) and second, as soon as you configure a bridge on the physical interface, you cannot use VLANs, anymore. physical --> LAGG --> VLAN --> bridge [...] What...

www.truenas.com

I just reallized I have about the same conversation in 2 threads I thoughts were the same and that I just simply don't understand this matter of:

• one NIC, one LAN, no 802.1q
• one NIC, one LAN + 802.1q
• two NICs, one LAN + 802.1q

Before
When I first installed TrueNAS, I had only LAN
All jails where installed with: vnet=on and interface was vnet0:bridge0

Now
pfSense has different 802.1q vlans.
TrueNAS is still connected to the network throuh bce0 at subnet1

The "cable" in bce0 comes from a switch as tagged since it is a trunk for the different vlans that will be distributed through different jails or VMs: 10.0.10.x/24 ; 10.0.20.x/24; 10.0.40.x/24


My knot
So what I get from post #26 and #27 is that if I want 802.1q to work on TrueNAS :
the native VLAN or vlan=1 won't work along with the other vlans, hence LAN=10.0.0.x/24 should be in anything but vlan=1 ?

I am honesly not proud of not getting that right.

 

[Patrick M. Hausen]

Move the VLANs to bce1 and use bce0 for untagged only. Don't mix tagged and untagged on the same physical port. You can always run VLAN 1 tagged, too, on any reasonable switch.

The untagged VLAN will work together with other VLANs on the same port but then you cannot put a bridge on the untagged.

There are so many different conditions to consider that I think it is best simply to never mix tagged and untagged. Much less room for problems in the future.

 

[NinthWave]

Move the VLANs to bce1 and use bce0 for untagged only. Don't mix tagged and untagged on the same physical port. You can always run VLAN 1 tagged, too, on any reasonable switch.

The untagged VLAN will work together with other VLANs on the same port but then you cannot put a bridge on the untagged.

There are so many different conditions to consider that I think it is best simply to never mix tagged and untagged. Much less room for problems in the future.

Thanks. This is what I though I understood in post #26 but your answer in post #27 made me believe I was off.

Jail Creation with 802.1q vlans
So according to this:

• bridge0 is created automatically with basic install but does not show in /network/interfaces
• vnet0 is created automatically at TrueNAS installation
• there is no vnet1
• I do not add a bridge1 for bce1 because that would be for an untagged lan
• I do add VLAN20 -->BRIDGE20 and so forth for all other VLANS that will have a jail connected to them
• When a jail will be created:
  • vnet_default_interface=none
  • IPv4 Interface = vnet0
  • IPv4_Address=10.0.x.y in the subnet of the vlan in question
  • IPv4_default_router=10.0.x.1 ; the pfSense interface doing DHCP for this vlan.

Correct this time ?

 

[Patrick M. Hausen]

Correct. Missing for the jail:

• interfaces: vnet0:bridge20 - which maps the vnet0 interface of the jail to your preferred bridge. It's down in the "Network Settings" section of the jail.

If the VLAN 20 is for jails only, don't put an IP address on vlan20 and neither on bridge20. It's all just layer2. The jails have their own stack.
If VLAN 20 is also used for the host, put the IP address on bridge20, not on vlan20.

 

[NinthWave]

If the VLAN 20 is for jails only, don't put an IP address on vlan20 and neither on bridge20. It's all just layer2. The jails have their own stack.
If VLAN 20 is also used for the host, put the IP address on bridge20, not on vlan20.

Again, I need clarification.

Let's say I want to run Emby on TrueNAS to serve medias on a couple of media players.

Actual setup

• Under a single LAN, I created a jail to run emby server : vnet0:bridge0
• The jail has it's own IP so I can SSH or SFTP the jail: 10.0.0.7
• Emby server (the host) got assigned the same IP as the jail : 10.0.0.7 but to connect to the management GUI, I do specify the port

The proper way

• Add interfaces vlan30 and bridge30 (IOT)
• Create a emby jail associated with vnet0:vlan30 but no IP to either vlan30 or bridge30
• Install emby in emby_jail
• Give an IP to emby for management and sharing files
• SSH TrueNAS and access the jail solely with iocage instead of SSHing the jail directly like before

Am I still right?

 

[Patrick M. Hausen]

• Under a single LAN, I created a jail to run emby server : vnet0:bridge0
• The jail has it's own IP so I can SSH or SFTP the jail: 10.0.0.7
• Emby server (the host) got assigned the same IP as the jail : 10.0.0.7 but to connect to the management GUI, I do specify the port

The Emby server is not running on the host. It is running in the jail. A jail is a lightweight virtual machine with its own IP stack (if vnet is used).
I bet your TrueNAS management interface was not at 10.0.0.7. Wherever that was - that is the host.

• Add interfaces vlan30 and bridge30 (IOT)
• Create a emby jail associated with vnet0:vlan30 but no IP to either vlan30 or bridge30
• Install emby in emby_jail
• Give an IP to emby for management and sharing files
• SSH TrueNAS and access the jail solely with iocage instead of SSHing the jail directly like before

Am I still right?

Sort of. Interfaces and jail - yes. But you use the emby application (whatever that is) at the emby address in vlan 30. So your pfsense needs to route between your desktop and the emby jail. And if you want to do filesharing in addition to whatever emby does, this is done from the host because file sharing is builtin into TrueNAS. So you mount a dataset that is outside of the emby jail (create in Storage > Pools) e.g. /mnt/<yourpool>/shares/emby into the jail at some convenient location e.g. /var/emby.
If you want the file sharing address to be in vlan 30, too, then your host does need an IP address there. So give the host an address for the bridge30 interface, not the vlan30 one.

And of course you can continue to ssh to the jail IP address in vlan 30 if your pfsense routes and permits that. Why wouldn't you?

 

[NinthWave]

@Patrick M. Hausen

I had to leave things for a few days but I am back to it. I was tired of messing around with my production TrueNAS (yeah, unadvertently mixing tagged and untagged fuçk3d even the switch) so I am trying in a VMWare environment.

Each time I try to setup a vlan IF, I get a problem:

Code:

Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 138, in call_method
    result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self,
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1213, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/local/lib/python3.9/site-packages/middlewared/service.py", line 484, in create
    rv = await self.middleware._call(
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1213, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 975, in nf
    return await f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/network.py", line 917, in do_create
    await self.middleware.call('interface.disable_capabilities', name)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1256, in call
    return await self._call(
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1224, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
  File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1128, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 979, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/interface/capabilities_freebsd.py", line 54, in disable_capabilities
    iface = netif.get_interface(iface)
  File "netif.pyx", line 2167, in netif.get_interface
KeyError: 'vlan20'

In the end, I can start the jail but I wonder about that "key error"

 

[Patrick M. Hausen]

Don't tick "Disable hardware offloading" for the VLAN interfaces. Only apply this to the parent physical.

 

[NinthWave]

Only apply this to the parent physical.

That's what I did. Only for bce1, not the vlan nor the bridge.

 

[Patrick M. Hausen]

Your screenshot shows it ticked for the VLAN - therefore the error message.

 

[NinthWave]

Your screenshot shows it ticked for the VLAN - therefore the error message.

Duh! Thanks, I was sure I did not tick it.