TrueNAS 12.0-STABLE - Set up VLANs

[NinthWave]

By curiosity, when "vnet1" or "vnet2" would be invoked? Is there a vnet# per physical interface ?

 

[Patrick M. Hausen]

No. There is one vnet per virtual interface in your jail. The first one being vnet0. To use multiple interfaces in a jail, to connect to e.g. bridge10, bridge20, bridge30, you set: vnet0:bridge10,vnet1:bridge20,vnet2:bridge30 in the "interfaces" config of your jail.

Then you can apply an IPv4/6 address to each interface like in my screenshot.

 

[NinthWave]

Everything was working on VMWare.

Now I am trying, on the production TrueNAS, to move an existing jail from bce0 (untagged) to the new bce1 vlan and the jail refuses to start for a "vnet error"

Code:

root@truenas[~]# iocage start cups
No default gateway found for ipv6.
* Starting cups
  + Started OK
  + Using devfs_ruleset: 1000 (iocage generated default)
  + Configuring VNET FAILED
  route: writing to routing socket: Network is unreachable
add net default: gateway 10.0.0.1 fib 0: Network is unreachable

Stopped cups due to VNET failure

Where should I look to pinpoint the problem?

It's a CUPS jail, no sharing.

 

[Patrick M. Hausen]

iocage get all cups, please.

 

[NinthWave]

iocage get all cups, please.

I hope I won't look dumb again by overlooking something. I believe the switch is configured correctly as all other vlan clients are working correctly. Only the TrueNAS bce1 seems to be non-working. Unfortunately, I don't have a laptop to connect to this port with vlan activated on it's NIC and ping pfSense.

Code:

root@truenas[~]# iocage get all cups
CONFIG_VERSION:28
allow_chflags:0
allow_mlock:0
allow_mount:0
allow_mount_devfs:0
allow_mount_fusefs:0
allow_mount_nullfs:0
allow_mount_procfs:0
allow_mount_tmpfs:0
allow_mount_zfs:0
allow_quotas:0
allow_raw_sockets:1
allow_set_hostname:1
allow_socket_af:0
allow_sysvipc:0
allow_tun:0
allow_vmm:0
assign_localhost:0
available:readonly
basejail:0
boot:0
bpf:1
children_max:0
cloned_release:12.2-RELEASE
comment:none
compression:lz4
compressratio:readonly
coredumpsize:off
count:1
cpuset:off
cputime:off
datasize:off
dedup:off
defaultrouter:auto   # I tried with 10.0.0.1 which is the default router, no more success
defaultrouter6:none
depends:none
devfs_ruleset:4
dhcp:0
enforce_statfs:2
exec_clean:1
exec_created:/usr/bin/true
exec_fib:0
exec_jail_user:root
exec_poststart:/usr/bin/true
exec_poststop:/usr/bin/true
exec_prestart:/usr/bin/true
exec_prestop:/usr/bin/true
exec_start:/bin/sh /etc/rc
exec_stop:/bin/sh /etc/rc.shutdown
exec_system_jail_user:0
exec_system_user:root
exec_timeout:60
host_domainname:none
host_hostname:cups
host_hostuuid:cups
host_time:1
hostid:4c4c4544-004b-5a10-8058-b2c04f4e4e31
hostid_strict_check:0
interfaces:vnet0:bridge20
ip4:new
ip4_addr:10.0.20.5/24
ip4_saddrsel:1
ip6:new
ip6_addr:none
ip6_saddrsel:1
ip_hostname:0
jail_zfs:0
jail_zfs_dataset:iocage/jails/cups/data
jail_zfs_mountpoint:none
last_started:none
localhost_ip:none
login_flags:-f root
mac_prefix:862b2b
maxproc:off
memorylocked:off
memoryuse:off
min_dyn_devfs_ruleset:1000
mount_devfs:1
mount_fdescfs:1
mount_linprocfs:0
mount_procfs:0
mountpoint:readonly
msgqqueued:off
msgqsize:off
nat:0
nat_backend:ipfw
nat_forwards:none
nat_interface:none
nat_prefix:172.16
nmsgq:off
notes:none
nsem:off
nsemop:off
nshm:off
nthr:off
openfiles:off
origin:readonly
owner:root
pcpu:off
plugin_name:none
plugin_repository:none
priority:99
pseudoterminals:off
quota:none
readbps:off
readiops:off
release:12.2-RELEASE
reservation:none
resolver:/etc/resolv.conf
rlimits:off
rtsold:0
securelevel:2
shmsize:off
stacksize:off
state:down
stop_timeout:30
swapuse:off
sync_state:none
sync_target:none
sync_tgt_zpool:none
sysvmsg:new
sysvsem:new
sysvshm:new
template:0
type:jail
used:readonly
vmemoryuse:off
vnet:1
vnet0_mac:862b2bbd67e5 862b2bbd67e6   # There is 2 MACs, strange
vnet0_mtu:auto
vnet1_mac:none
vnet1_mtu:auto
vnet2_mac:none
vnet2_mtu:auto
vnet3_mac:none
vnet3_mtu:auto
vnet_default_interface:none
vnet_default_mtu:1500
vnet_interfaces:none
wallclock:off
writebps:off
writeiops:off

 

[NinthWave]

Code:

root@truenas[~]# pciconf -lv | grep -A1 -B3 network
bce0@pci0:1:0:0:        class=0x020000 card=0x02371028 chip=0x163914e4 rev=0x20 hdr=0x00
    vendor     = 'Broadcom Inc. and subsidiaries'
    device     = 'NetXtreme II BCM5709 Gigabit Ethernet'
    class      = network
    subclass   = ethernet
bce1@pci0:1:0:1:        class=0x020000 card=0x02371028 chip=0x163914e4 rev=0x20 hdr=0x00
    vendor     = 'Broadcom Inc. and subsidiaries'
    device     = 'NetXtreme II BCM5709 Gigabit Ethernet'
    class      = network
    subclass   = ethernet

 

[Patrick M. Hausen]

Code:

defaultrouter:auto   # I tried with 10.0.0.1 which is the default router, no more success
[...]
ip4_addr:10.0.20.5/24

The defaultrouter must be in the 10.0.20.0/24 network - that's why the jail does not start. I assume the jail is supposed to be accessed from hosts in other networks and be able to reach the internet (to install packages and updates, at least)? Then you need something in that VLAN that is a router. Either your layer3 switch (if it is layer3 capable), a separate firewall, or as a last resort your TrueNAS host.

I can help with setting up the TrueNAS host as a router if necessary but I would not recommend it.

 

[NinthWave]

Code:

defaultrouter:auto   # I tried with 10.0.0.1 which is the default router, no more success
[...]
ip4_addr:10.0.20.5/24

The defaultrouter must be in the 10.0.20.0/24 network - that's why the jail does not start. I assume the jail is supposed to be accessed from hosts in other networks and be able to reach the internet (to install packages and updates, at least)? Then you need something in that VLAN that is a router. Either your layer3 switch (if it is layer3 capable), a separate firewall, or as a last resort your TrueNAS host.

I can help with setting up the TrueNAS host as a router if necessary but I would not recommend it.

Is there anything missing in my pfSense rules ?

 

[Patrick M. Hausen]

How should I be able to tell? What is "PRINT net"? 10.0.20.0/24? I can only help you get the jail to start. If the jail is indeed a part of "PRINT net" you need to set "PRINT address" - whatever that is - as the default gateway and DNS server in your jail config.

 

[NinthWave]

I can help with setting up the TrueNAS host as a router if necessary but I would not recommend it.

I'll ask a friend, good with networking to check this and go the TrueNAS=host as a last resort.

 

[Patrick M. Hausen]

No ... your pfSense is the router! All is well. Why don't you just configure your jail correctly? My remark was more "I can't tell you if your rules are correct because I don't know what you want to achieve ..." - the pfSense is definitely not the cause of your jail not starting.

Let's wind back to the last step - your jail does not start because iocage aborts with some error message you cannot make sense of.

This is because the defaultrouter you configured must be an address in the same network as the jail's IPv4 address: 10.0.20.5/24. So whatever your pfSense has as an IP address in that network - that's what you put into the jail's "defaultrouter" field. And in the "resolver" field you put the string "nameserver <same-ip-address-as-defaultrouter>" and I am confident your jail will start.

 

[NinthWave]

And in the "resolver" field you put the string "nameserver <same-ip-address-as-defaultrouter>" and I am confident your jail will start.

That's what was missing. I'd love to buy you a beer

 

[NinthWave]

Sort of. Interfaces and jail - yes. But you use the emby application (whatever that is) at the emby address in vlan 30. So your pfsense needs to route between your desktop and the emby jail. And if you want to do filesharing in addition to whatever emby does, this is done from the host because file sharing is builtin into TrueNAS. So you mount a dataset that is outside of the emby jail (create in Storage > Pools) e.g. /mnt/<yourpool>/shares/emby into the jail at some convenient location e.g. /var/emby.
If you want the file sharing address to be in vlan 30, too, then your host does need an IP address there. So give the host an address for the bridge30 interface, not the vlan30 one.

So I have transfered the Emby server form native lan to vlan10 (sorry, not 30).
jail=emby
IP=10.0.10.20

Since the movies are on the host at /mnt/movies, there was a working share prior to moving emby to the vlan

I have given an IP to the bridge10
IP=10.0.10.2

I can access the Emby Web management from my PC (on native subnet)
Kodi does not return a connexion error to emby

I can see the movies, art....

But I can not start any playback as it returns a "no stream error"

I guess I missed something again.
The host (hence the files) are in another subnet at 10.0.0.6, I guess that's why Emby and Host don't talak together but I don't get where I should connect them together.

 

[Patrick M. Hausen]

The TrueNAS itself does not need an IP address in that VLAN.
What's your defaultrouter setting for the emby jail? Is it the IP address of your pfsense?
The rest is up to your pfsense ruleset. If Kodi and emby use multicast - what I don't know - you probably need a multicast repeater plugin for your pfsense. I only use OPNsense, so no idea what is available.

But foremost: why are you doing this? What are you gaining by ripping a small private LAN apart using technology intended for data centers managed by professionals? What's the user story?

You have a working network. You change things around that break the network. Why?

 

[NinthWave]

you have a working network. You change things around that break the network. Why?

I am glad you asked because I did not want to hijack this thread.

Short story, I have had an accident and seriously broken one ankle. I have been off for 3 months now and still can't walk. With too much time to spare, I am trying to spend my time sitting in front of my computer since that's all I can do apart from reading books. So I am learning to make my way though networking, linux, FreeBSD.... and my days go by.

On shodan.io, I have read that people got hacked from their printer or cell phone. Plus we have a kid at home that does I don't know what behind his door. For one, I know he blindly installs game cheats that freely access windows RAM and registry...

And there is a kind of popular lobby on segmenting everything in order to avoid hacking. And honestly, I am not sure what part is necessary.

I have TrueNAS which is the repository of:

• 1400 CDs I manually ripped
• hundreds of DVDs I manually ripped
• all my documents (banking, résumés...)
• backups

So I segmented like crazy to avoid one thing accessing another.

I now have 8 vlans

• LAN (safe hosts)
• IOT (phone ATA, smart home, TV boxes)
• Wifi (safe hosts)
• Guests (wifi guests)
• Surveil (cameras and NVR)
• LAN2 (kid's PC, really unsafe)
• Print (CUPS and printer)
• Dev (to try things)

So what I am trying to do with all this is prevent unsafe devices that have access to TrueNAS one way or another to be a backdoor to fu¢$ my precious data with a ransomware following lack of FW update on said unsafe devices.

Maybe, without hijacking the purpose of this thread which is setuping vlans in TrueNAS, as a wizened sage, you could share your thoughs on what is usefull and what is not. I am much open to return to a simpler network.

 

[Patrick M. Hausen]

Well for one I would deem it reasonable to keep this emby thing in the same VLAN as the clients accessing it. What I would do in this case is move the web interface of the TrueNAS and possibly the SMB share, too, to a more isolated zone/VLAN. And then mount the meticulously ripped media into the emby jail read-only.

And you will probably need to educate yourself about pfSense. Which definitely is beyond the scope of this forum. The good news is that there is a pfSense one, too.

Kearning the network basics is of course a great idea, but how about reading about the fundamentals - addressing, routing, bridging/switching/VLANs first? There is not much documentation on the iXsystems/TrueNAS side for all of that, because all of this is "data center stuff" managed by pros. In the OPNsense forum we have similar requests like "I added this VLAN here and this one there and now it doesn't work ...". All of this stuff took me years to learn. And I only have so much time for free support ...

Keep going, though!
Patrick

 

[NinthWave]

Well for one I would deem it reasonable to keep this emby thing in the same VLAN as the clients accessing it.

This is what I am trying to achieve. Emby server (serving movies) is now in vlan10 along with the TVboxes taht are pulling said movies.

What I would do in this case is move the web interface of the TrueNAS and possibly the SMB share, too, to a more isolated zone/VLAN.

Is it not the goal of an emby jail in vlan10? I am not sure how it is different from your previous sentence ?

And then mount the meticulously ripped media into the emby jail read-only.

I could do that since the jail itself is just 2 GiB while the movies account for 500 GiB.
But I was under the impression that it is best practice to keep the jails for the "systems" and the datasets (not in iocage) for data...

And you will probably need to educate yourself about pfSense.

I am a member of pfSense forum and ask questions there as well. And I have a friend that can help me with some networking. The thing is, books and FAQs tell you how to make each specific thing work but they rarely tell you which way to not make things you'll regret when needs grow.

 

[NinthWave]

What's your defaultrouter setting for the emby jail? Is it the IP address of your pfsense?

The default router is at 10.0.10.1.

The rest is up to your pfsense ruleset.

I thought that since jails have their own network stack, the embyjail would be able to access the movie dataset though this stack and ACL.

If I am to set rules in pfSense to allow embyjail 10.0.10.20 to access TrueNAS host 10.0.0.6, it seems to me it defeats the purpose of isolating emby in it's own jail in the first place ?

 

[Patrick M. Hausen]

Is it not the goal of an emby jail in vlan10? I am not sure how it is different from your previous sentence ?

I understood your last problem that you moved emby into VLAN 10 and then video playback stopped working? So there is at least one client not in VLAN 10 that needs to access emby?

But I was under the impression that it is best practice to keep the jails for the "systems" and the datasets (not in iocage) for data...

Yes, correct. Your data goes in regular TrueNAS datasets outside of iocage. These can then be mounted into the jails, read-write or read-only as desired.

I thought that since jails have their own network stack, the embyjail would be able to access the movie dataset though this stack and ACL.

The movie dataset resides on the TrueNAS that also hosts the emby jail, right? In that case the mounts do not use the network at all. They are configured at "Jails > jailname > Mount Points" in the UI. These are strict local Unix mounts, no sharing protocol involved.

HTH,
Patrick

 

[Phill23]

Hi Patrick,

really nice explanation!
One question I still have open:

If I want to use a LAN IP for TrueNAS and VLANs for the jails etc. but use all on one LAGG I would have to create one Bridge directly on the LAGG for the TrueNAS IP?!

So Physical --> LAGG--> LAN Bridge | VLANs --> VLAN Bridges

Physical --> LAGG (if present) --> VLAN --> Bridge