How to setup VLANs within FreeNAS 11.3

[HolyK]

Phew -- nice writeup -- sometimes I get to the point I've figure I've wasted so much time on this topic that once its running I don't want to modify -- BUT

Hah yea i feel you. I've spent almost whole day figuring out why the F$%# the host NIC (igb0) is being added to the bridge and screwing the jail subnet. I had new bugreport prepared with all the details but went to 400-check again before submitting. I went through the whole jail config again and "just" tried the "none" instead of auto. The moment i saw the proper IP in the GUI i was like "Riiiiight, makes sense ... (FU$^#^ "auto" piece of CR#$)..."

Just some clarification points for me so I can try when I have time
1. Your running untagged VLAN1 and tagged VLAN(s) into FreeNAS?

Nope, I've dropped whole VLAN1 because of my paranoia level over 9000. VLAN-1 is default. I don't want any defaults :] So i have several non-default VLANs on my local network. They're marked as Un-Tagged on the switch ports for devices which are VLAN NOT aware (like Workstation, HTPC, TV, ....). Then i have two TRUNK ports with all of my VLAN TAGs included.

First one is for my "UniFi AC Lite" AP (as i have multiple SSIDs, each with separate VLAN for isolation between personal devices/network and "Guests").
Second one is the NAS so i can isolate the Jails as described above.

Note that both of these switch ports have one VLAN marked as UnTagged. This ensures that both devices will be on the internal/secured subnet. FreeNAS itself could be on tagged VLAN as well (your setup) but the UnifiAP can NOT be on tagged VLAN otherwise the UniFi controller wont connect to AP. (Maybe Ubiquiti already fixed that maybe not, no idea. I had this issue like 4 years ago as well as bunch of other ppl as i saw on their support forums. Or maybe there was the VLAN-1 hard-coded in AP firmware so that was the only one "working"? Not sure but I've figured it out by untagged port and no complains since then.)

2. With your setup -- are you able to create a jail/VM that runs on this untagged network? I'm thinking for example of a bridge0, with untagged traffic on it? How would you set this up?

Yes, first approach is plain bridge between Jail and host NIC. Anyway i would NOT go that way with exposed Jails to outside world because NAS and Jail (or more jails withing the same bridge) would share the network stack. The iface gets into promiscuous mode so if any of the Jails gets compromised one could easily tcpdump and sniff whatever goes through not only the jail iface itself but the other jails AND the NAS itself as well. (I am not 100% sure here so correct me if i am wrong but this is how i understand the "bridge")

So the second and "safe" approach is to drop VNET completely and go with NAT instead. In that case the network stack is separated from HOST system. Packets gets tagged by the untagged VLAN ID when they arrive on switch port.

Or there is a third way if you have multiple NICs on the HOST system. In that case you could just pass the whole interface to Jail and have it physically separated by different cable/switchport (where you can TAG it by different ID)

3. The remainder of your setup for the tagged networks - bridge/vlan creation and vnet configuration looks great. Thanks for that tip to set to none rather than auto. Mine are all set at auto.

I saw your ifconfig output and i was scratching my head why you don't have the parent NIC as a member of bridge and i had. I would say it is because even your parent NIC is on VLAN. No idea how the "auto" decides what is the "default iface" though.

Hope this helps. Good luck if you ever going to touch your working network. :D

 

[KevDog]

Hah yea i feel you. I've spent almost whole day figuring out why the F$%# the host NIC (igb0) is being added to the bridge and screwing the jail subnet. I had new bugreport prepared with all the details but went to 400-check again before submitting. I went through the whole jail config again and "just" tried the "none" instead of auto. The moment i saw the proper IP in the GUI i was like "Riiiiight, makes sense ... (FU$^#^ "auto" piece of CR#$)..."


Nope, I've dropped whole VLAN1 because of my paranoia level over 9000. VLAN-1 is default. I don't want any defaults :] So i have several non-default VLANs on my local network. They're marked as Un-Tagged on the switch ports for devices which are VLAN NOT aware (like Workstation, HTPC, TV, ....). Then i have two TRUNK ports with all of my VLAN TAGs included.

First one is for my "UniFi AC Lite" AP (as i have multiple SSIDs, each with separate VLAN for isolation between personal devices/network and "Guests").
Second one is the NAS so i can isolate the Jails as described above.

Note that both of these switch ports have one VLAN marked as UnTagged. This ensures that both devices will be on the internal/secured subnet. FreeNAS itself could be on tagged VLAN as well (your setup) but the UnifiAP can NOT be on tagged VLAN otherwise the UniFi controller won't connect to AP. (Maybe Ubiquiti already fixed that maybe not, no idea. I had this issue like 4 years ago as well as bunch of other ppl as i saw on their support forums. Or maybe there was the VLAN-1 hard-coded in AP firmware so that was the only one "working"? Not sure but I've figured it out by untagged port and no complains since then.)

Yes, first approach is plain bridge between Jail and host NIC. Anyway i would NOT go that way with exposed Jails to outside world because NAS and Jail (or more jails withing the same bridge) would share the network stack. The iface gets into promiscuous mode so if any of the Jails gets compromised one could easily tcpdump and sniff whatever goes through not only the jail iface itself but the other jails AND the NAS itself as well. (I am not 100% sure here so correct me if i am wrong but this is how i understand the "bridge")

So the second and "safe" approach is to drop VNET completely and go with NAT instead. In that case the network stack is separated from HOST system. Packets gets tagged by the untagged VLAN ID when they arrive on switch port.

Or there is a third way if you have multiple NICs on the HOST system. In that case you could just pass the whole interface to Jail and have it physically separated by different cable/switchport (where you can TAG it by different ID)

I saw your ifconfig output and i was scratching my head why you don't have the parent NIC as a member of bridge and i had. I would say it is because even your parent NIC is on VLAN. No idea how the "auto" decides what is the "default iface" though.

Hope this helps. Good luck if you ever going to touch your working network. :D

Ok --- so to answer my questions
1. Ok I kind of get what you are referring to. But a couple of things
Your setup, might or might not worked if the packets coming from the switch are a combination of tagged and untagged. I set mine up so that every single packet coming into FreeNAS is tagged with a VLANID -- hence my need for a VLAN1/bridge0 combination. (Short story -- so I kind of blew apart my FreeNAS system -- supposedly I installed some packages into the 11.3RC2 base with the pkg manager and when I rebooted -- system was hosed. So I basically reverted back to 11.2U7 and then by this time I just upgraded to 11.3 Final. I might need to change the instructions above since it was possible with 11.3 to create all the bridges(0,20,30,40) and vlans(1,20,30,40) via the GUI. In fact I found out if I was trying to implement the setup of bridges and vlans through system tunables they wouldn't work (as they had worked in 11.3RC2 and 11.2U7).

In terms of your Unifi stuff -- I have 3 unifi switches and two AC-Pros. I'm running 3 wireless guest networks (VLAN1, 30,40) and I have a few more wired networks in addition that aren't wireless. I changed the management VLAN of all my Unifi Equipment to VLAN40 which for me is the management VLAN. So in essence I believe its possible the UnfiAP to be on the tagged VLAN.

2. In terms of VNET and packet sniffing -- I understand your security concerns, however I thought the use of VNETs actually made it so YOU COULD NOT sniff the traffic on the bridge. Yes you can set VLANs up without the use of VNET, however I thought in these circumstances in the absence of VNET this was the problem where packets could be sniffed. Perhaps I have this backwards.

3. I'll mess around with the auto/none features this weekend to see if they change anything.

I think in both are cases we are doing "about the same thing", but the difference is the tagging and what networks we need exposed to FreeNAS. You clearly do not want the "default" network exposed where as I am trying to have it exposed.

I'm glad things are up and running for you as they are for me. Documentation in this area would be difficult b/c of the number of variables, however the FreeNAS documentation in general in this area doesn't really seem to make an effort.

 

[HolyK]

Ok --- so to answer my questions
1. Ok I kind of get what you are referring to. But a couple of things
Your setup, might or might not worked if the packets coming from the switch are a combination of tagged and untagged.

Actually it already is "mixed" (If I understand you correctly)

See my PC in connected to untagged (VID7) port. So when i am connecting to my NAS via SSH it gets like this ... untagged packets gets VID 7, travels through switch as tagged, then TAG is stripped out when leaving the other switch port into which FreeNAS is connected (untagged port with VID7 as well) then untagged packets enters FreeNAS NIC.

Frame 202: 60 bytes on wire (480 bits), 60 bytes captured (480 bits)
Ethernet II, Src: Micro-St_AA:AA:AA (AA:AA:AA:AA:AA:AA), Dst: SuperMic_BB:BB:BB (BB:BB:BB:BB:BB:BB)
Destination: SuperMic_BB:BB:BB (BB:BB:BB:BB:BB:BB)
Source: Micro-St_AA:AA:AA (AA:AA:AA:AA:AA:AA)
Type: IPv4 (0x0800)
Padding: 000000000000
Internet Protocol Version 4, Src: 10.2.7.22, Dst: 10.2.7.33
Transmission Control Protocol, Src Port: 56713, Dst Port: 4141, Seq: 1, Ack: 31, Len: 0

Another connection i can do is SSH from my PC directly to one of my Jails. But these are on different VLAN (lets say "5"). So again packet leaves untagged, then gets VID7 upon entering switch, travels through router this time due to different VID/subnet. Then it goes back to switch and leaves trunk port with re-marked VID to 5. After that it enters NAS NIC igb0 (still with VID5), then VLAN5 interface, <VID gets stripped out here>, bridge5, vnet and ends up inside Jail without VLAN TAG (because Jails are NOT VLAN aware)

Sniff from within NAS (physical NIC):

Frame 428: 86 bytes on wire (688 bits), 86 bytes captured (688 bits)
Ethernet II, Src: EacAutom_AA:AA:AA (AA:AA:AA:AA:AA:AA), Dst: SuperMic_BB:BB:BB (BB:BB:BB:BB:BB:BB)
Destination: SuperMic_BB:BB:BB (BB:BB:BB:BB:BB:BB)
Source: EacAutom_AA:AA:AA (AA:AA:AA:AA:AA:AA)
Type: 802.1Q Virtual LAN (0x8100)
802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 5
000. .... .... .... = Priority: Best Effort (default) (0)
...0 .... .... .... = CFI: Canonical (0)
.... 0000 0000 0101 = ID: 5
Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: 10.2.7.22, Dst: 10.2.5.71
Transmission Control Protocol, Src Port: 56456, Dst Port: 1717, Seq: 1, Ack: 1, Len: 28
SSH Protocol

And local sniff from within the jail itself (not 802.11Q VLAN markups)

Frame 258: 82 bytes on wire (656 bits), 82 bytes captured (656 bits)
Ethernet II, Src: EacAutom_AA:AA:AA (AA:AA:AA:AA:AA:AA), Dst: SuperMic_BB:BB:BB (BB:BB:BB:BB:BB:BB)
Internet Protocol Version 4, Src: 10.2.7.22, Dst: 10.2.5.71
Transmission Control Protocol, Src Port: 56456, Dst Port: 1717, Seq: 1, Ack: 1, Len: 28
SSH Protocol

Basically it will never be end2end "VLAN only traffic".

In terms of your Unifi stuff ... I changed the management VLAN of all my Unifi Equipment to VLAN40 which for me is the management VLAN. So in essence I believe its possible the UnfiAP to be on the tagged VLAN.

Yes i did the same back in the day and ended up hard-resetting my stuff. I don't have Unifi switch or something. Just the AP with the controller as application on my desktop. I don't even bother installing the Unifi controller into jail. It is just overkill for my needs. The AP is "set and forget" anyway. All things (DHCP, client/ip management, etc...) is happening on my pfSense anyway.

2. In terms of VNET and packet sniffing -- I understand your security concerns, however I thought the use of VNETs actually made it so YOU COULD NOT sniff the traffic on the bridge. Yes you can set VLANs up without the use of VNET, however I thought in these circumstances in the absence of VNET this was the problem where packets could be sniffed. Perhaps I have this backwards..

Well ... i would say this:
A] No VNET, No NAT ... just a plain bridge -> This is "sniffable" across Jails (if more jails are within the same bridge)
B] Plain NAT, no VNET -> Not sniffable. NAT separates the network stack. But it has its own limits (no 802.1Q i believe and other stuff)
C] VNET + bridge + VLAN but without parent NIC as member of bridge (VLAN has NIC and bridge has VLAN) -> No way to "cross-sniff" between jails.
D] VNET + bridge (while bridge has host NIC as a parent) -> Safe as well due to VNET
E] VNET + bridge + parentNIC within bridge. Here i am not sure and yes i might be wrong with my previous statement. You're right that VNET should separate the network stack (epairb -> vnet -> bridge -> NIC) but how does that works in case you have multiple "vnets" within the same bridge together with parent NIC? Cross-jail sniffing is most probably not possible due to VNET but (!) in theory you might be able to sniff out the traffic from parent NIC? Or does the VNET effectively cuts the line on HOST side? Now i am really curious how does that works and once i have time i'll fire up few test Jails and play with it a bit :]

I think in both are cases we are doing "about the same thing", but the difference is the tagging and what networks we need exposed to FreeNAS. You clearly do not want the "default" network exposed where as I am trying to have it exposed.

Yes i agree. Yet yours is a bit more hardened by its complexity (NAS management on VLAN as well). On the other hand my network goes always through some VLAN within the switch/router. There is nothing outside (no excluded ports on switch/router). And in general no device can reach other one on different VLAN by default. This is blocked on pfSense by generic rules. Also cross-talk within the same subnet is blocked. Then i have explicit rules allowing cross-vlan connections (Only one Workstation can actually directly connect to FreeNAS over ssh or web interface). So if i omit the scenario where my workstation gets compromised i see two potential attack vectors. Either compromising one of the NET exposed Jails AND successfully escaping jail OR exploiting some SMB vulnerability. I don't have other services enabled (not counting things like SMART or UPS ...)

I'm glad things are up and running for you as they are for me. Documentation in this area would be difficult b/c of the number of variables, however the FreeNAS documentation in general in this area doesn't really seem to make an effort.

Well there is no official documentation except "this button enables that" without any further details. All of the "howtos" are community-based and as you said there is so many of the variants. But i am glad we now have proper VNET and NAT implementation. Back in the day the VLAN and advanced networking was actively "rejected" by iX. Luckily they changed their mind as the community was crying loud for this for a long time.

Anyway please share outputs from your tests for sure. More details on forum helps others to get the necessary pieces of their own puzzle. Thanks!

 

[rcdevils]

I followed the guide but when starting the jails I get the following error:

Code:

Error: [EFAULT] Stopped downloader due to VNET failure

Any suggestions on getting around it?

 

[KevDog]

Hey I've never seen that error, but could you maybe post or type your configs/setup and how your VLAN tagging is constructed?

 

[rcdevils]

Thanks for the quick reply.
Hopefully this is sufficient. If you need anything in particular let me know!

Code:

root@cells[~]# ifconfig
igb0: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:25:90:e0:65:d4
        hwaddr 00:25:90:e0:65:d4
        nd6 options=1<PERFORMNUD>
        media: Ethernet autoselect
        status: no carrier
igb1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:25:90:e0:65:d5
        hwaddr 00:25:90:e0:65:d5
        nd6 options=1<PERFORMNUD>
        media: Ethernet autoselect
        status: no carrier
cxgb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: cxgb0
        options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:07:43:06:7d:59
        hwaddr 00:07:43:06:7d:59
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet 10Gbase-SR <full-duplex>
        status: active
cxgb1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:07:43:06:7d:5a
        hwaddr 00:07:43:06:7d:5a
        nd6 options=1<PERFORMNUD>
        media: Ethernet none
        status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
vlan10: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: trust
        options=680703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:07:43:06:7d:59
        inet 192.168.10.11 netmask 0xffffff00 broadcast 192.168.10.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet 10Gbase-SR <full-duplex>
        status: active
        vlan: 10 vlanpcp: 0 parent interface: cxgb0
        groups: vlan
vlan20: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: media-IOT
        options=680703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:07:43:06:7d:59
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet 10Gbase-SR <full-duplex>
        status: active
        vlan: 20 vlanpcp: 0 parent interface: cxgb0
        groups: vlan
bridge10: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:70:bf:49:05:0a
        nd6 options=9<PERFORMNUD,IFDISABLED>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vlan10 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000000
bridge20: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:70:bf:49:05:14
        nd6 options=9<PERFORMNUD,IFDISABLED>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vlan20 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 7 priority 128 path cost 2000000

Here is the jail config

Code:

{
    "allow_chflags": 0,
    "allow_mlock": 0,
    "allow_mount": 0,
    "allow_mount_devfs": 0,
    "allow_mount_fusefs": 0,
    "allow_mount_nullfs": 0,
    "allow_mount_procfs": 0,
    "allow_mount_tmpfs": 0,
    "allow_mount_zfs": 0,
    "allow_quotas": 0,
    "allow_raw_sockets": 0,
    "allow_set_hostname": 1,
    "allow_socket_af": 0,
    "allow_sysvipc": 0,
    "allow_tun": 0,
    "allow_vmm": 0,
    "assign_localhost": 0,
    "boot": 0,
    "bpf": 1,
    "children_max": "0",
    "cloned_release": "11.3-RELEASE",
    "comment": "none",
    "defaultrouter": "192.168.10.1",
    "defaultrouter6": "auto",
    "depends": "none",
    "devfs_ruleset": "4",
    "dhcp": 0,
    "enforce_statfs": "2",
    "exec_clean": 1,
    "exec_fib": "0",
    "exec_jail_user": "root",
    "exec_poststart": "/usr/bin/true",
    "exec_poststop": "/usr/bin/true",
    "exec_prestart": "/usr/bin/true",
    "exec_prestop": "/usr/bin/true",
    "exec_start": "/bin/sh /etc/rc",
    "exec_stop": "/bin/sh /etc/rc.shutdown",
    "exec_system_jail_user": "0",
    "exec_system_user": "root",
    "exec_timeout": "60",
    "host_domainname": "none",
    "host_hostname": "test-new",
    "host_hostuuid": "test_new",
    "host_time": 1,
    "hostid": "00000000-0000-0000-0000-002590E065D4",
    "hostid_strict_check": 0,
    "interfaces": "vnet0:bridge10",
    "ip4": "new",
    "ip4_addr": "vnet0:bridge10|192.168.10.201/24",
    "ip4_saddrsel": 1,
    "ip6": "new",
    "ip6_saddrsel": 1,
    "ip_hostname": 0,
    "jail_zfs": 0,
    "jail_zfs_dataset": "iocage/jails/test_new/data",
    "jail_zfs_mountpoint": "none",
    "last_started": "2020-02-10 04:48:08",
    "login_flags": "-f root",
    "mac_prefix": "000743",
    "mount_devfs": 1,
    "mount_fdescfs": 1,
    "mount_linprocfs": 0,
    "mount_procfs": 0,
    "nat": 0,
    "nat_forwards": "none",
    "notes": "none",
    "owner": "root",
    "priority": "99",
    "release": "11.3-RELEASE-p6",
    "resolver": "search hive;nameserver 192.168.10.1",
    "rtsold": 0,
    "securelevel": "2",
    "stop_timeout": "30",
    "sysvmsg": "new",
    "sysvsem": "new",
    "sysvshm": "new",
    "template": 0,
    "vnet": 1,
    "vnet0_mac": "00074331b61d 00074331b61e",
    "vnet1_mac": "none",
    "vnet2_mac": "none",
    "vnet3_mac": "none",
    "vnet_default_interface": "none",
    "vnet_interfaces": "none"
}

 

[KevDog]

Can you confirm that other devices on VLAN 10 are working? Just curious -- your main freenas installation -- what VLAN or network is associated with this?

 

[rcdevils]

Can you confirm that other devices on VLAN 10 are working? Just curious -- your main freenas installation -- what VLAN or network is associated with this?

The main installation of FreeNAS is using vlan10 and all the shares/services work besides jails.

For the jail config I changed IPV4 Interface from vnet0:bridge10 to just "---" option and now the jail starts.

It is in the vlan and I can ping it, though not sure why it works.

 

[KevDog]

Looking at my own config I see this:

Code:

ip4_addr:10.0.1.156/24

Yours was listed like:

Code:

ip4_addr:vnet0:bridge10|192.168.10.201/24

Try taking out the vnet0:bridge10 of the ip4_addr

 

[rcdevils]

Looking at my own config I see this:

Code:

ip4_addr:10.0.1.156/24

Yours was listed like:

Code:

ip4_addr:vnet0:bridge10|192.168.10.201/24

Try taking out the vnet0:bridge10 of the ip4_addr

Thanks for all the help!
I did that and everything connects :)

Now I'm facing another weird issue which I'm not sure belongs here, if not I can make a new post about it.

I am running into some speed issues . I have about a 200Mb/s download but when downloading something in the Jail it is extremely slow.
I ran this command in the jail and outside in just the FreeNAS root shell as an example.

Code:

curl -o /dev/null http://speedtest.sea01.softlayer.com/downloads/test100.zip

In the Jail I get about 10-20KB/s download
In the FreeNAS root shell I get about 20MB/s download. What would cause such a huge slow down?

 

[KevDog]

@rcdevils

That might be an issue for a different post. Is it all the vnet jails or just the one? What if you don't put the jail in the VLAN? Does it matter then?

 

[rcdevils]

@KevDog

It happens in all the vnet jails. If I create a NAT jail , such as when installing a plugin, I get the full download speed in the jail.
I then took the NAT jail and change the jail settings to a vnet which then caused it to slow down. So my guess it has something to do with the vnet as the base freenas is also using vlans to route its traffic.

It can also be the vlan-vnet interaction so tomorrow I can remove vlans and try just running vnets.

Edit:
So after adding some tunables and changing the MTU I get my full internet download speed! Thanks for all the help.

Tunables I have used:

Code:

kern.ipc.maxsockbuf=16777216
net.inet.ip.intr_queue_maxlen=2048
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.recvspace=4194304
net.inet.tcp.recvbuf_inc=524288
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.sendspace=2097152
net.inet.tcp.sendbuf_inc=32768
net.route.netisr_maxqlen=2048

 

[HolyK]

Edit: So after adding some tunables and changing the MTU I get my full internet download speed! Thanks for all the help.

Hi, please don't use "some tunables" and share the knowledge with the rest of the community. You never know who will face similar issue like you. . Thanks! :]

 

[ferremontagud]

Hi, I followed the steps and jails start without problem but has not network output, can not see the gateway of jail.
Please, can you help me? I am testing in a virtual environment but now I am very obfuscated :(

Physical adapter: hn0

I created from webui a vlan1 (parent iface hn0) and bridge0 (members vlan1) -- bridge0 created by webui because system tunable can not create interface at system startup. Vlan has ip network config and host has network output.. I can access via webui too.

Ok, now go to jail. From webui create a vlan11 (parent iface hn0) and a bridge11 (members vlan11) and I create a jail with this config:
vnet: on, bpf: on, ipv4 address:192.168.50.6, ipv4 netmask:/30, ipv4 defaultrouter:192.168.50.5, ipv4 interface: ------, vnet_interfaces:none, interfaces:vnet0:bridge11, vnet_default_interface:none

Jail start ok but has not network connection.

I have reviewed multiple times the networks adapters from host but seems to be fine.

Code:

root@freenas[~]# ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
hn0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: hn0
        options=8051b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,TSO4,LRO,LINKSTATE>
        ether 00:15:5d:88:d4:59
        hwaddr 00:15:5d:88:d4:59
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
vlan1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vlan1
        options=80000<LINKSTATE>
        ether 00:15:5d:88:d4:59
        inet 192.168.50.2 netmask 0xfffffffc broadcast 192.168.50.3
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        vlan: 1 vlanpcp: 0 parent interface: hn0
        groups: vlan
vlan11: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu1500
        description: vlan11
        options=80000<LINKSTATE>
        ether 00:15:5d:88:d4:59
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        vlan: 11 vlanpcp: 0 parent interface: hn0
        groups: vlan
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: bridge0
        ether 02:a9:2e:77:89:00
        nd6 options=9<PERFORMNUD,IFDISABLED>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vlan1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 2000
bridge11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: bridge11
        ether 02:a9:2e:77:89:0b
        nd6 options=9<PERFORMNUD,IFDISABLED>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0.3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 7 priority 128 path cost 2000
        member: vlan11 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000
vnet0.3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: VLAN11 as nic: epair0b
        options=8<VLAN_MTU>
        ether 00:15:5d:37:6f:cf
        hwaddr 02:df:f0:00:07:0a
        nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    groups: epair

Thanks! ;)

 

[KevDog]

Is your switch configured to tag and pass VLAN11 traffic?

 

[ferremontagud]

Yes, checked. I am using a mikrotik chr on a hyper-v environment.
When I use my previous config on this jail, network connection works fine but dashboard of freenas is empty. It seems that network jail is wrong configured, that's why I was using your guide :)

 

[ferremontagud]

@KevDog Previous config is: vnet:on, vnet_interfaces: vlan11, interfaces: vlan11:hn0, vnet_default_interfaces: none and put this on rc.conf inside of jail (ifconfig_vlan11="inet 192.168.50.6 netmask 255.255.255.252", defaultrouter="192.168.50.5"). This config broke FreeNAS dashboard, but jail has network connection.
Any ideas why your steps is not working for me?

Thanks!

 

[KevDog]

@ferremontagud

Your posts are a little difficult for me to follow. I assume you are configuring all your bridges and vlans within the FreeNAS gui. One thing you might want to try is to assign for vlan11 an ip address similar to how how assigned an ip address to vlan1. Why I suggest this is that you could then reach the main FreeNAS web interface at either the vlan1 IP address of vlan11 IP address to ensure that vlan11 tags are being passed through the switch appropriately.

I don't think you need to put anything inside your jail's rc.conf file to get this to work.
How are you configuring your jails in the web gui?

 

[ferremontagud]

I know.. sorry but my English is not good.
Yes, I am configuring all bridges and vlans from FreeNAS gui.
Vlans are being passed correctly, I can access to FreeNAS gui from two VLANS without problems.
I will try to explain it better :)

Pyshical adapter: hn0
Vlans: vlan1(member of bridge0) and vlan11(member of bridge11).

Vlan1 is for access to FreeNAS gui.
Vlan11 is for use from a jail.

Config of jail is:

Code:

{
    "allow_chflags": 0,
    "allow_mlock": 0,
    "allow_mount": 0,
    "allow_mount_devfs": 0,
    "allow_mount_fusefs": 0,
    "allow_mount_nullfs": 0,
    "allow_mount_procfs": 0,
    "allow_mount_tmpfs": 0,
    "allow_mount_zfs": 0,
    "allow_quotas": 0,
    "allow_raw_sockets": 1,
    "allow_set_hostname": 1,
    "allow_socket_af": 0,
    "allow_sysvipc": 0,
    "allow_tun": 0,
    "allow_vmm": 0,
    "assign_localhost": 0,
    "boot": 0,
    "bpf": 1,
    "children_max": "0",
    "cloned_release": "11.3-RELEASE",
    "comment": "none",
    "defaultrouter": "192.168.50.5",
    "defaultrouter6": "auto",
    "depends": "none",
    "devfs_ruleset": "4",
    "dhcp": 0,
    "enforce_statfs": "2",
    "exec_clean": 1,
    "exec_fib": "0",
    "exec_jail_user": "root",
    "exec_poststart": "/usr/bin/true",
    "exec_poststop": "/usr/bin/true",
    "exec_prestart": "/usr/bin/true",
    "exec_prestop": "/usr/bin/true",
    "exec_start": "/bin/sh /etc/rc",
    "exec_stop": "/bin/sh /etc/rc.shutdown",
    "exec_system_jail_user": "0",
    "exec_system_user": "root",
    "exec_timeout": "60",
    "host_domainname": "none",
    "host_hostname": "VLAN11",
    "host_hostuuid": "VLAN11",
    "host_time": 1,
    "hostid": "0dbb722b-20aa-8840-a20d-264c5b81b30e",
    "hostid_strict_check": 0,
    "interfaces": "vnet0:bridge11",
    "ip4": "new",
    "ip4_addr": "192.168.50.6/30",
    "ip4_saddrsel": 1,
    "ip6": "new",
    "ip6_saddrsel": 1,
    "ip_hostname": 0,
    "jail_zfs": 0,
    "jail_zfs_dataset": "iocage/jails/VLAN11/data",
    "jail_zfs_mountpoint": "none",
    "last_started": "2020-02-20 16:19:21",
    "login_flags": "-f root",
    "mac_prefix": "00155d",
    "mount_devfs": 1,
    "mount_fdescfs": 1,
    "mount_linprocfs": 0,
    "mount_procfs": 0,
    "nat": 0,
    "nat_forwards": "none",
    "notes": "none",
    "owner": "root",
    "priority": "99",
    "release": "11.3-RELEASE-p6",
    "resolver": "/etc/resolv.conf",
    "rtsold": 0,
    "securelevel": "2",
    "stop_timeout": "30",
    "sysvmsg": "new",
    "sysvsem": "new",
    "sysvshm": "new",
    "template": 0,
    "vnet": 1,
    "vnet0_mac": "00155d376fcf 00155d376fd0",
    "vnet1_mac": "none",
    "vnet2_mac": "none",
    "vnet3_mac": "none",
    "vnet_default_interface": "none",
    "vnet_interfaces": "none"
}

Ifconfig command from FreeNAS host with Jail started:

Code:

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
hn0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: hn0
        options=8051b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,TSO4,LRO,LINKSTATE>
        ether 00:15:5d:88:d4:59
        hwaddr 00:15:5d:88:d4:59
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
vlan1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vlan1
        options=80000<LINKSTATE>
        ether 00:15:5d:88:d4:59
        inet 192.168.50.2 netmask 0xfffffffc broadcast 192.168.50.3
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        vlan: 1 vlanpcp: 0 parent interface: hn0
        groups: vlan
vlan11: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu1500
        description: vlan11
        options=80000<LINKSTATE>
        ether 00:15:5d:88:d4:59
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        vlan: 11 vlanpcp: 0 parent interface: hn0
        groups: vlan
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: bridge0
        ether 02:a9:2e:77:89:00
        nd6 options=9<PERFORMNUD,IFDISABLED>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vlan1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 2000
bridge11: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: bridge11
        ether 02:a9:2e:77:89:0b
        nd6 options=9<PERFORMNUD,IFDISABLED>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0.4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 7 priority 128 path cost 2000
        member: vlan11 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000
vnet0.4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: VLAN11 as nic: epair0b
        options=8<VLAN_MTU>
        ether 00:15:5d:37:6f:cf
        hwaddr 02:5f:d0:00:07:0a
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair

Ifconfig command from jail:

Code:

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 00:15:5d:37:6f:d0
        hwaddr 02:5f:d0:00:08:0b
        inet 192.168.50.6 netmask 0xfffffffc broadcast 192.168.50.7
        inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
    groups: epair

Well, when i try to ping a host I see that jail not have connection.

Code:

ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
^C
--- 8.8.8.8 ping statistics ---
8 packets transmitted, 0 packets received, 100.0% packet loss

 

[KevDog]

Why do you have two inets in your jail?

inet 192.168.50.6 netmask 0xfffffffc broadcast 192.168.50.7
inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255

What about your routing table in the jail? netstat -rn

Are you sure your 255.255.255.252 netmask is correct?