Putting jails on separate VLANs

[medicineman25]

Having a bit of trouble getting a jail onto a separate VLAN from TrueNAS and I guess just understanding how VLANs work on TrueNAS in general. The documentation seems either a bit sparse or I'm just not finding the right stuff.

What I want is to have the jails get assigned dhcp leases from my dhcp server running on my network, outside of Proxmox or indeed TrueNAS. TrueNAS should be on VLAN1 (net0), my unifi jail on VLAN10 (net1), my transmission client on VLAN20 (net2).

I have TrueNAS running inside Proxmox, here is my hardware tab for that VM:

I try to add my jails to either net2 or net1 vnet_default_interface, but they are still getting IP addresses from the net0 subnet. The same happens in the opposite direction if I move TrueNAS to VLAN20, all of the jails will then receive IP addresses from that subnet.

Jic: on the switch that Proxmox is connected to, all the appropriate VLANs are assigned to the switch port.

******************************************************************************************************************

EDIT:

Found a couple threads on this issue, going to explore further

Thread #1: https://www.truenas.com/community/threads/truenas-12-0-stable-set-up-vlans.89280/
Thread #2: https://www.truenas.com/community/threads/separating-jails-with-vlans.43537/
Thread #3: https://www.truenas.com/community/threads/how-to-set-separate-vlan-for-jail.54019/#post-374176

 

[IOSonic]

Hey Medicine Man,

I don't fully understand your network setup within TrueNAS. Can you post a screenshot of "Network > Interfaces"?

Before deploying the jails, have you verified that your second and third NICS can reach vlans 10 and 20 from within FreeNAS? One easy way you could do this is allow them to get DHCP addresses and verify that they get an address from the correct subnet.

My guess is that you're going to need to deploy a couple of new bridges and attach those NICs as members. At that point, verify that those NICs get DHCP addresses from the right subnet. If everything works at that point, redeploy your jail, but be sure to the jail "Networking" tab and specify the bridge in the format <vtnet#:bridge#>. See my screenshot.

 

[medicineman25]

Hey IOSonic,

Yes indeed that's actually the problem I'm facing: they cannot get a dhcp lease from the correct subnet despite adding different NICs to the jail's network config

I have also tried adding the two vlans you see below which are in fact tagged 10 and 20 respectively, still only getting dhcp from vlan1 (vnet0) i.e. 10.0.0.0/24, both the parent NICs of vlan10/20 are vnet0.

I've now tried to delete and re-add the below vlans with their parent interfaces being set to vnet1 and vnet2 respectively, then add that as vnet_default_interface for each jail in the "Basic Properties" section. That did not work either.

Sure no worries:

So basically those extra "physical" NICs are just virtual NICs added in Proxmox. They're not actually physical ports on the box.

I figured that to add jails to those extra NICs I would just add them to the jail config, but you speak of bridges and perhaps that is a missing ingredient to what I am trying to achieve here. I will try that after I get back from my errands.

Also, I notice there are two sections to set network properties. "Basic Properties" and "Network Properties". What is the difference?

I can see at this link that VNET creates an entire network stack per jail. So does that mean that the way I am currently approaching this, I have essentially created two network stacks per jail by having a Proxmox NIC and a VNET NIC?

I'm guessing that I should instead add a proxmox NIC per subnet and then a bridge inside TrueNAS to that NIC per jail, in the Network Properties?

Also, see earlier configs here:

(vlan20 is identical to vlan10 and I changed the VLAN Parent Interface to vnet1/2 but did not change anything)

 

[medicineman25]

Ok so after messing about with a bunch of settings, I discovered a few important things about PM and TN.

PM:
1. You can add bridges to NICs if you use the <port name>.<vlan> syntax e.g. vmbr0.10
2. When adding said bridges, they are already tagged so don't add a tag to the network interface you create for each VM.

TN:
1. For some reason DHCP doesn't/didn't clear the default route (I was using it to test network connectivity.... yyyyyyyeh: bad idea). Make sure you have set the default route if you can ping it from inside the network from a diff VM, but not from outside.
2. It will not automatically delete interfaces that were removed in PM, even after a reboot. You will need to manually delete these and don't let their presence fool you into thinking there are network interfaces assigned from the PM as some kind of autoconfig thing.

Here is what I ended up doing and what you should do if you want to be crazy like me:

1. Create vlans via bridges using <port>.<vlan> syntax in proxmox:

Notice how only one of these interfaces needs an IP address, this is just my management address for proxmox itself.

2. Add interfaces to the TrueNAS VM (and other VMs if you want), adding them to the desired bridge and leave the "VLAN Tag" field blank. We don't need it because the PM bridge is tagging the port for us.

3. In TN, create a bridge for each vnet in Network -> Interfaces (as previously mentioned, if these already existed inside TN and then you add a network device to the VM in PM, then TN will just pick it up)

3a. Double check you have your interfaces as well (vtnetN in TN = netN in PM e.g. vtnet0 = net0):

4. In the jail's "Basic Properties" set DHCP (if you are using dhcp) and VNET to true and set vnet_default_interface to auto. If you are not using dhcp, just make sure that the allocated IP address matches the VLAN that the PM bridge is connected to:

5. Then, as is suggested by IOSonic, add the interface and bridge using the <interface>:<bridge> syntax in the jail's Network Properties section:

That's it, your jail should be able to sit on a different VLAN to TrueNAS.