VLAN - explain please on how to do it.

HolyK

Ninja Turtle
Moderator
Joined
May 26, 2011
Messages
653
Ah yea, sorry for the OT :/ And for not being to help you with the issue.
 

ezra

Contributor
Joined
Jan 15, 2015
Messages
124
@chipgoon Thanks for all those write ups! Im about to setup 2x NIC as LAGG and create a few VLANs with the lagg0 as parent interface. Id like to use VNET on all the jails and use DHCP in every VLAN also ill have.a DNS server listning on every subnet.

Any pointers here? How to specify to use dhcp for example?

I have OPNsense (pfsense) for running so the vlan setup is a breeze.


Ive tried to setup a vlan on em0 last week but that totally broke the connection, is this because the lack of a bridge?

Thanks
 

ezra

Contributor
Joined
Jan 15, 2015
Messages
124
@chipgoon Could you provide the steps stripped down perhaps? I think i messed up something... Can't get it right, been banging my head for over 4 hours...

I have the following:
_1x bridge0 (normal vm's and jails and freenas 192.168.3.0/24)
| - 1x NIC ibg0
| - 1x tap0
| - 1x tap1
| - 7x vnet0:x

1x bridge 1 (not created by me, vlan666 is not a member)
1x vlan666 (Specific jails need to be in here 192.168.7.0/24)
1x vnet1:0

1x OPNsense setup with vlan666 setup with DNS server at default gateway 192.168.7.1

1x test jail:
VNET=on
DHCP=off
BPF=off
IPV4IFACE=(tried vlan666, bridge 1, igb0)
IPV4ADD=192.168.7.2/24
DEFAULTROUTER=192.168.7.1
IFACE=vnet1:vlan666 (again, tried bridge1 and others, no avail)
RESOLVER=192.168.7.1

Also i cant add anything to -fib 1 only -fib 0, do i need to manually create one? I added them to fib 0 and used that one, also tried without...

Tunables:

Code:
cloned_interfaces
bridge0
rc
yes

ifconfig_bridge0
addm igb0 up
rc
yes

net.add_addr_allfibs
0
sysctl
yes

net.fibs
4
loader
yes


Ifconfig:
Code:
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=2400b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,RXCSUM_IPV6>
    ether 70:85:c2:62:06:62
    hwaddr 70:85:c2:62:06:62
    inet 192.168.3.2 netmask 0xffffff00 broadcast 192.168.3.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:c1:92:84:a4:00
    nd6 options=9<PERFORMNUD,IFDISABLED>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 6 priority 128 path cost 2000000
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 2000000
    member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=80000<LINKSTATE>
    ether 00:bd:16:0b:f8:00
    hwaddr 00:bd:16:0b:f8:00
    nd6 options=1<PERFORMNUD>
    media: Ethernet autoselect
    status: active
    groups: tap
    Opened by PID 3989
tap1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=80000<LINKSTATE>
    ether 00:bd:13:17:f8:01
    hwaddr 00:bd:13:17:f8:01
    nd6 options=1<PERFORMNUD>
    media: Ethernet autoselect
    status: active
    groups: tap
    Opened by PID 4365
vlan666: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=200001<RXCSUM,RXCSUM_IPV6>
    ether 70:85:c2:62:06:62
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    vlan: 666 vlanpcp: 0 parent interface: igb0
    groups: vlan
bridge1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:c1:92:84:a4:01
    nd6 options=1<PERFORMNUD>
    groups: bridge
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 0 ifcost 0 port 0


Code:
root@freenas:~ # netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.7.1        UGS     vlan666
127.0.0.1          link#2             UH          lo0
192.168.3.0/24     link#1             U          igb0
192.168.3.2        link#1             UHS         lo0
192.168.7.0/24     70:85:c2:62:06:62  US      vlan666


Please... anyone ^^ documentation on this one is very scarce... I dont know what i'm doing here. I'll stop before I mess up anything.

edit: https://forums.freenas.org/threads/...other-vlan-jail-topic-tried-everything.73671/
 
Last edited:

frv

Cadet
Joined
Feb 23, 2015
Messages
2
Struggling with the same ... got it almost solved ...

Noone happens to know if it is possible to assign a vlan interface a different mac than the parent nic ?
 

Dan0maN

Cadet
Joined
Dec 15, 2014
Messages
1
@Kennyvb8: I think I recently ran into and fixed the problem that you were experiencing, locking yourself out. The issue shows up when you use a non-physical interface, such as a lagg (in fact, I believe that the lagg interfaces are the only type of enumerated interfaces that FN supports in their rewrite of the rc.conf variables). Here's what's going on:

During boot, sysrc sources /etc/defaults/rc.conf, which includes the /etc/rc.conf and /etc/rc.conf.local files. The /etc/rc.conf.local file has been rewritten by FN to query the database for its persistent configuration and echo them into rc.conf-style formats. In that file, in the "interface_config" function, at line 251 in v11-2U5, there is the only mention of "cloned_interfaces":
Code:
    251                 if [ -n "${lagg_id}" ]; then
    252                         cloned_interfaces="${cloned_interfaces} ${interface}"
    253                         echo "cloned_interfaces=\"${cloned_interfaces}\""
    254                 fi

Here, is where it creates the first (only) instance of the "cloned_interfaces" keypair.

The problem comes in when you attempt to create a tunable with the same name as something important, as the code doesn't check if there is already a keypair existing (really, it would be very hard to do this sort of check). It happens at line 873 of the same file:
Code:
    873         local tun_var tun_value
    874         ${FREENAS_SQLITE_CMD} ${FREENAS_CONFIG} "SELECT tun_var, tun_value FROM system_tunable WHERE tun_enabled = 1 AND tun_type = 'rc' ORDER BY id" | \
    875         while read -r tun_var tun_value; do
    876                 echo "${tun_var}=\"${tun_value}\""
    877         done

If you create an rc-type tunable that creates another instance of the "cloned_interfaces" keypair, then as sysrc sources this file, it overwrites the first instance of it, which inhibits the enumeration of the lagg interface, and ultimately breaks the networking on your system during boot.

To work around this, you can set the "cloned_interfaces" keypair equal to itself before adding the non-physical interfaces you wish to enumerate:
Code:
Name:  cloned_interfaces
Value:  ${cloned_interfaces} bridge41 bridge99


I see that some are using the ifconfig_* options for the bridges to manually add interfaces during boot. While searching for how to get what I was trying to do working, I ran into the autobridge_* options. Some may want to look into them as well. You specify the bridge interfaces that you want it to look for, and when it is enumerated, it will automatically put the interfaces you need into it:
Code:
autobridge_interfaces="bridge41 bridge99"
autobridge_bridge41="vlan41"
autobridge_bridge99="vlan99"

I used these tunables to help build bridges that I know will always be connected to the VLANs that I want. I've had mixes results leaving it up to warden and iocage to set the correct VLANs, and when I upgraded from 11-1U6 to 11-2U5, the bridges got swapped. Now with those manually set bridges above, I set my jails, both warden and iocage, to use those specified bridges:
Code:
iocage set interfaces="vnet0:bridge99" cage1
warden set bridge-ipv4 wardenjail1 bridge41


Hope some or all of this helps someone.
xDan0maNx
 

StarkJohan

Explorer
Joined
Mar 27, 2015
Messages
62
Since my last post I started using a VM on the freenas box (pihole DNS) which did the vnet setup I had with bridges and post-init scripts even messier. I never really had the time to get things working automatically from boot until I finally had a proper look at things again today.

I want to give a huge thank you to chipgoon for chiming in on this issue. The FIB was the missing link for me and once I understood this the rest was a breeze to setup. Your comment on using whatever NIC I wanted for the VLAN parent also worked as you suspected. I want to point out that proper naming of the tunables is needed for them to work and the static routes needs to be added as well and all in the correct order (e.g. if needs to exist to set a default gw on the if). Se attached image for my setup.

Also, to clarify berkeley packet filter cannot be used without vnet and the "interfaces" of the jail cannot be just "vlanX", it has to be in the form "interface:bridge" so in my case "vlan4:eth1" works just fine.
 

Attachments

  • Screenshot 2019-09-13 at 17.27.13.png
    Screenshot 2019-09-13 at 17.27.13.png
    122.6 KB · Views: 363
Top