Jail in separate network - routing table

miro.cz · Jan 15, 2018

Hi,
I would like to separate few Jail accessible from the internet but I am stucked with routing in DMZ jails.

My FreeNAS ( FreeNAS-11.1-RELEASE) is running HP Microserver Gen8 so there are two physical NICs (bge0, bge1) in my case configured in LACP as lagg0. For FreeNAS and regular Jails I created interface vlan11 and configured IP address 192.168.11.20 (network 192.168.11.0/24, GW 192.168.11.254). For DMZ Jails I created interface vlan14 . There is no IP address for FreeNAS configured. All the routing is provided with ipfire firewall connected with trunk port to L2 switch (vlan11 192.168.11.254 vlan14 192.168.14.254). Freenas switch connection is also port trunk – tagged vlan11, vlan14.

There is output of ifconfig and netstat -rn

Code:

root@freenas:~ # ifconfig
bge0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
		ether 00:fd:45:fd:72:34
		hwaddr 00:fd:45:fd:72:34
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect (1000baseT <full-duplex>)
		status: active
bge1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
		ether 00:fd:45:fd:72:34
		hwaddr 00:fd:45:fd:72:35
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect (1000baseT <full-duplex>)
		status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
		options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
		inet6 ::1 prefixlen 128
		inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
		inet 127.0.0.1 netmask 0xff000000
		nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
		groups: lo
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
		ether 00:fd:45:fd:72:34
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect
		status: active
		groups: lagg
		laggproto lacp lagghash l2,l3,l4
		laggport: bge0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
		laggport: bge1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
vlan11: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=80001<RXCSUM,LINKSTATE>
		ether 00:fd:45:fd:72:34
		inet 192.168.11.20 netmask 0xffffff00 broadcast 192.168.11.255
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect
		status: active
		vlan: 11 vlanpcp: 0 parent interface: lagg0
		groups: vlan
vlan14: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
		options=80103<RXCSUM,TXCSUM,TSO4,LINKSTATE>
		ether 00:fd:45:fd:72:34
		nd6 options=9<PERFORMNUD,IFDISABLED>
		media: Ethernet autoselect
		status: active
		vlan: 14 vlanpcp: 0 parent interface: lagg0
		groups: vlan


root@freenas:~ # netstat -rn
Routing tables

Internet:
Destination		Gateway			Flags	 Netif Expire
default			192.168.11.254	 UGS	  vlan11
127.0.0.1		  link#3			 UH		  lo0
192.168.11.0/24	link#5			 U		vlan11
192.168.11.20	  link#5			 UHS		 lo0

I created standard Jail named DMZ_jail_1 with these configuration – IPv4 address: 192.168.14.21, IPv4 netmask: 24, IPv4 default gateway: 192.168.14.254, VIMAGE of and assigned to NIC: vlan14. I can ping IP addresses in same network but there is missiong GW – configured but not propagated.

Code:

root@DMZ_jail_1:/ # netstat -rn												 
Routing tables																 
																			   
Internet:																	   
Destination		Gateway			Flags	 Netif Expire					
192.168.14.21	  link#6			 UHS		 lo0						   
root@DMZ_jail_1:/ #

I tried to find out information how to do routing in FreeBSD, FreeNAS for Jails in differrent subnets. I found interesting information there:

https://forums.freenas.org/index.php?threads/freenas-jails-in-different-multiple-subnets.41539/
https://forums.freenas.org/index.php?threads/how-to-set-separate-vlan-for-jail.54019/

I think for me there is way to create second routing table for the DMZ network and assign it to DMZ Jails.

For that I configured few wariables in System – Tunables in GUI

I checked result how it was applied to main system:

Code:

root@freenas:~ # sysctl net.fibs
net.fibs: 4
root@freenas:~ # sysctl net.add_addr_allfibs
net.add_addr_allfibs: 0
root@freenas:~ # setfib 1 netstat -rn
Routing tables (fib: 1)

Internet:
Destination 	 Gateway			Flags	 Netif Expire
default 		 192.168.14.254	 UGS	 vlan14
127.0.0.1 		 lo0				UHS		 lo0
192.168.14.0/24 00:fd:45:fd:72:34 US	 vlan14

root@freenas:~ #
root@freenas:~ # netstat -rn
Routing tables

Internet:
Destination 	 Gateway			Flags	 Netif Expire
default 		 192.168.11.254	 UGS	 vlan11
127.0.0.1 		 link#3			 UH		 lo0
192.168.11.0/24 link#5			 U		vlan11
192.168.11.20 	 link#5			 UHS		 lo0
192.168.14.0/24 link#6			 U		vlan14
192.168.14.21 	 link#6			 UHS		 lo0
192.168.14.22 	 link#6			 UHS		 lo0

When I connect jail from main system with command “setfib 1 jexec 8 /bin/tcsh”

Code:

root@freenas:~ # setfib 1 jexec 8 /bin/tcsh
root@DMZ_jail_1:/ #
root@DMZ_jail_1:/ #
root@DMZ_jail_1:/ # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=12.962 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=7.301 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 7.301/10.131/12.962/2.831 ms
root@DMZ_jail_1:/ #

I need to find out how to assign routing table fib 1 to DMZ_jail_1

Thx Mirek

dlavigne · Jan 22, 2018

Were you able to resolve this? If not, did you try to see if it works using iocage jails?

miro.cz · Jan 27, 2018

dlavigne said:
Were you able to resolve this? If not, did you try to see if it works using iocage jails?

Still no success in my way.

I tried iocage jail as you recommended. In my case named DMZiocage-1 but got same problem with routes. Jail has configured defaul GW but it wasn't propagated to jail. Below there is part of jail config and routing table. In the next post there is solution. It would be helpful for other FreeNAS users.

Thank you so much Mirek

Code:

root@freenas:/mnt # iocage get all DMZiocage-1 | less
.
.
defaultrouter:192.168.14.254
ip4_addr:vlan14|192.168.14.22/24
.
.

root@DMZiocage-1:~ # netstat -rn
netstat: kvm not available: /dev/mem: No such file or directory
Routing tables

Internet:
Destination		Gateway			Flags	 Netif Expire
192.168.14.22	  link#6			 UHS		 lo0
root@DMZiocage-1:~ #

miro.cz · Jan 27, 2018

There is solution I have used

Created second routing table for DMZ in System – Tunables in GUI like in my init post.

Then created jail

Code:

activate iocage zpool different from system (datastore2 in my case)
root@freenas:~ # iocage activate datastore2

fetched realese for new jails
root@freenas:~ # iocage fetch
Creating datastore2/iocage
Creating datastore2/iocage/download
Creating datastore2/iocage/images
Creating datastore2/iocage/jails
Creating datastore2/iocage/log
Creating datastore2/iocage/releases
Creating datastore2/iocage/templates
[0] 9.3-RELEASE (EOL)
[1] 10.1-RELEASE (EOL)
[2] 10.2-RELEASE (EOL)
[3] 10.3-RELEASE
[4] 10.4-RELEASE
[5] 11.0-RELEASE (EOL)
[6] 11.1-RELEASE

Type the number of the desired RELEASE
Press [Enter] to fetch the default selection: (11.1-RELEASE)
Type EXIT to quit:
Fetching: 11.1-RELEASE

Downloading : MANIFEST [####################] 100% 0Mbit/s
Downloading : base.txz [####################] 100%  4.96Mbit/s
Downloading : lib32.txz [####################] 100%  4.96Mbit/s
Downloading : doc.txz [####################] 100%  4.94Mbit/s
Downloading : src.txz [####################] 100%  4.94Mbit/s
 4.94Mbit/sExtracting: base.txz...
Extracting: lib32.txz...
Extracting: doc.txz...
Extracting: src.txz...

* Updating 11.1-RELEASE to the latest patch level...

created jail named DMZiocage-1 with IP 192.168.14.22 that use vlan14 interface (only for DMZ)
root@freenas:~ # iocage create -n DMZiocage-1 ip4_addr="vlan14|192.168.14.22/24" defaultrouter=192.168.14.254 -r 11.1-RELEASE

enabled ping from jail with allowing raw sockets
root@freenas:~ # iocage set allow_raw_sockets=1 DMZiocage-1

assigned second routing table to jail
root@freenas:~ #iocage set exec_fib=1 DMZiocage-1

set DNS server for jail
iocage set resolver="nameserver 8.8.8.8" DMZiocage-1

list iocage jail
root@freenas:~ # iocage list
+-----+-------------+-------+--------------+---------------+
| JID |	NAME	 | STATE |   RELEASE	|	  IP4	  |
+=====+=============+=======+==============+===============+
| 16  | DMZiocage-1 | down	| 11.1-RELEASE | 192.168.14.22 |
+-----+-------------+-------+--------------+---------------+

start jail I have created
root@freenas:~ # iocage start DMZiocage-1
* Starting DMZiocage-1
  + Started OK
  + Starting services OK
root@freenas:~ #

open jail console
root@freenas:~ # iocage console DMZiocage-1

now I am in Jail shell
root@DMZiocage-1:~ #

ping to Google DNS
root@DMZiocage-1:~ # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=25.316 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=59.641 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=57 time=13.037 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 13.037/32.665/59.641/19.723 ms
root@DMZiocage-1:~ #

I hope that it will be helpfull.

Bye Mirek

ezra · Feb 23, 2019

Sorry to revive this but, you seem like the only one that tackled this problem? Don't you use a bridge for this? Could you show me your hosts ifconfig?
https://forums.freenas.org/threads/...other-vlan-jail-topic-tried-everything.73671/

sgargel · Nov 28, 2019

ezra said:
Sorry to revive this but, you seem like the only one that tackled this problem? Don't you use a bridge for this? Could you show me your hosts ifconfig?
https://forums.freenas.org/threads/...other-vlan-jail-topic-tried-everything.73671/

Worked for me too. After some headache found that I was wrong to set "static_routes". Value is DMZnet DMZ_gw without routes_ prefix!

No need to use a bridge interface.

Code:

igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether d0:50:99:c3:9d:3c
    hwaddr d0:50:99:c3:9d:3c
    inet 192.168.10.190 netmask 0xffffff00 broadcast 192.168.10.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
    ether d0:50:99:c3:9d:3d
    hwaddr d0:50:99:c3:9d:3d
    inet 192.168.11.104 netmask 0xffffff00 broadcast 192.168.11.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
    inet 127.0.0.1 netmask 0xff000000
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: lo

Code:

my-nas01# setfib 1 netstat -rn
Routing tables (fib: 1)

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.11.1       UGS        igb1
127.0.0.1          lo0                UHS         lo0
192.168.11.0/24    d0:50:99:c3:9d:3d  US         igb1

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#3                        UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#3                        U           lo0
ff02::/16                         ::1                           UGRS        lo0

Brad Ackerman · Jan 5, 2020

Works for me; I just need to go through the FreeNAS UI and give everything on the main LAN a specific interface to bind to.

Important Announcement for the TrueNAS Community.

Jail in separate network - routing table

miro.cz

Cadet

dlavigne

Guest

miro.cz

Cadet

miro.cz

Cadet

ezra

Contributor

sgargel

Cadet

Brad Ackerman

Cadet

Similar threads