FreeNAS Jails in different(multiple) subnets

jkmuk

Dabbler
Joined
Mar 11, 2015
Messages
24
I am pulling my hair out trying to resolve this. can some Freenas Networking experts please help ?

I plan to create multiple jails in my Freenas box and plan to run these jails in different subnets. However there isnt any easy to follow "How to" that I could find. The problem seems to be that the epairXb interfaces used by the (VIMAGE) jails seems to be attached the default route which is on it own VLAN. There by breaking the vlan setup of all the jails as they need to be on different vlans. I have attached a picture of what is happening now.

What I want to be able to do is have the jails assigned to vlan112 or vlan111. The freenas box itself needs to be on vlan110. Is this possible in Freenas 9.3.1? Any pointers will be well recieved.
 

Attachments

  • Network Diagram.png
    Network Diagram.png
    11.2 KB · Views: 709

jkmuk

Dabbler
Joined
Mar 11, 2015
Messages
24
in the picture 'ig0' and 'ig1' are the only two physical NICs . They are teamed (link aggregation using LACP) as 'lagg0' from that point forward everything is a virtual interface.
 
D

dlavigne

Guest
AFAIK, you cannot do this without a physical NIC for each subnet. Someone else might know more.
 
Joined
Feb 22, 2016
Messages
6
I achieved what you are describing by following these steps:
1- Create VLAN interface
2- Define tunables in loader
3- Define routes in rc (in seperate route table with fibs)
4- Apply route fib to your jail permanently: echo "/usr/sbin/setfib 1 /bin/sh /etc/rc" > /mnt/backups/jails/.plex_1.meta/jail-start
5- Reboot

You can use these commands to debug as root in freenas host:
sysctl net.fibs
sysctl net.add_addr_allfibs
ifconfig vlan40
ifconfig vlan70
setfib 1 netstat -rn
setfib 1 jexec 7 /bin/tcsh

Or from within the jail:
netstat -rn

Note:
If you use the webgui and open the terminal from there you will not get the correct route table in the jail. You should enable ssh server inside the jail and log in to it that way instead.
In these commands I used my vlan tags, internal IP addressing and jail number. You should of course substitute with your own.

Sources:
http://savagedlight.me/2014/03/07/freebsd-jail-host-with-multiple-local-networks/
http://www.mattzone.com/freenas/freenas_20150901.html
 

Attachments

  • tunables-routes.png
    tunables-routes.png
    107.2 KB · Views: 692
  • jail-config.png
    jail-config.png
    141.6 KB · Views: 742
  • vlans.png
    vlans.png
    91.7 KB · Views: 683
  • interfaces.png
    interfaces.png
    133.2 KB · Views: 717
Last edited:

jkmuk

Dabbler
Joined
Mar 11, 2015
Messages
24
Ludovic Lamarre many thanks, I will try this and post my result. But before I proceed a qucik question, do these configuration changes persist over re-boots? Given that Freenas is architected to be managed from the GUI, I do not want to be doing this on every reboot as the configuration changes are overwritten in many cases unless i start customising Freenas which I do not want to.
 
Joined
Feb 22, 2016
Messages
6
Yes. As you can see from the screenshots I just attached, most of the configs are made through the GUI. The only config made outside of GUI is step number 4 which is a command written inside your jails dataset on a physical hard drive.
 

jkmuk

Dabbler
Joined
Mar 11, 2015
Messages
24
ON step 4 - I do not have a jail-start file in the *.meta folder. I am using Freenas 9.3.1 . The files there are
Code:
./                  host                jail-post-delete*   jailtype
../                 id                  jail-post-start*    mac
autostart           iface               jail-post-stop*     vnet
defaultrouter-ipv4  ipv4                jail-pre-delete*
defaultrouter-ipv6  ipv6                jail-pre-start*
fstab               jail-flags          jail-pre-stop*
 
Joined
Feb 22, 2016
Messages
6
Yes, neither did I. You can just create the file yourself. The command I gave you should take care of creating the file for you.
 

jkmuk

Dabbler
Joined
Mar 11, 2015
Messages
24
You sir (Ludovic Lamarre) are a star. This worked like a charm. I have successfully setup two VLANs and they are working fine, I plan to go up to 4 which I believe should not be a problem give that the fibs count is already set to 4 , seems to have read my mind. One last questions, are there any side effects that I should worry about? Memory usage, stability issues etc?
 
Joined
Feb 22, 2016
Messages
6
Thanks, glad this could help you out! Here are some release notes of freebsd regarding fibs:
https://www.freebsd.org/releases/7.1R/errata.html
[20090105] The Release Notes for 7.1-RELEASE should have mentioned that FreeBSD now supports multiple routing tables. To enable this, the following steps are needed:
https://www.freebsd.org/releases/8.4R/relnotes-detailed.html
A loader tunable net.fibs now supports specifying the number of routing tables. The ROUTETABLES kernel option can still be used to set the default number of routing tables.[r235104]

So fibs were introduced in FreeBSD 7.1 with a special kernel option and since 8.4 with a loader tunable. Max 16 route tables. I don't think you need to worry about making the system unstable provided that you keep using the default routing table for your base system.

Also you should make sure you are not making your system vulnerable by exposing it on an unsecure subnet.

You should bind your webgui to only your management subnet IP because otherwise people will be able to connect to your freenas through all your subnets.

You should also note that the rest of your services directly provided by the base OS (FreeNAS) may be accessible through your additional IPs (assigned to each individual jail). SSH, FTP, CIFS, etc.
 

Attachments

  • 2016-02-29 18_28_08-freenas - FreeNAS-9.3-STABLE-201602031011.png
    2016-02-29 18_28_08-freenas - FreeNAS-9.3-STABLE-201602031011.png
    140.7 KB · Views: 543

stranger

Dabbler
Joined
Apr 11, 2014
Messages
31
Does this still work in 9.10? I've tried it but without any success. in my jail, "delete" (a test jail) the warden output is like:
Code:
Mounting user-supplied file-systems
jail -c path=/mnt/important/gaol/delete ip4.addr=10.254.0.30/16 host.hostname=delete  allow.raw_sockets=true persist
Starting jail with: /usr/sbin/setfib 1 /bin/sh /etc/rc
Setting hostname: delete.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
32-bit compatibility ldconfig path: /usr/lib32
Creating and/or trimming log files.
Starting syslogd.
Clearing /tmp (X related).
Updating motd:.
Starting cron.

Within the jail I get the following output:
Code:
sysctl net.my_fibnum
net.my_fibnum: 0

I have a workaround:
Code:
setfib 4 jexec delete sysctl net.my_fibnum					  
net.my_fibnum: 4

but it's not that nice. I'm reluctant to submit a bug report as my system is an upgrade from 9.2 and I kept my jails (it's a lot of work to update them); maybe I'll go thru all the work for FreeNAS 10.
I've tested by creating a jail with the images:
http://download.freenas.org/jails/9.3/x64/freenas-standard-9.3-RELEASE.tgz
and
http://download.freenas.org/jails/10/x64/freenas-standard-10.3-RELEASE.tgz

I've also tried setting an rc variable: jail_delete_fib=4
and a jail flag:
Code:
jail -c path=/mnt/important/gaol/delete ip4.addr=10.254.0.30/16 host.hostname=delete  exec.fib=1 allow.raw_sockets=true persist



I can't really test all that much as this is my home and hence production box. Maybe someone with a clean instead could try to reproduce the issue.


Given that my small home installation had 4 network ports, it's pretty clear that production installs need better support for multi-NIC systems. That is better ways to split jails up into routing tables, bind NICs etc.
 

Krautmaster

Explorer
Joined
Apr 10, 2017
Messages
81
Dear all.

Years later i sturggle almost the same. My goal is:

LAN (Nic vmx0)
192.168.2.10 (gateway, router) -> WAN
vmx0 192.168.2.25 (Plex Jail) -> Problem is here (no internet from Jail)
vmx0 192.168.2.95 (Freenas itself)
...

DMZ: (Nic vmx1)
192.168.1.1 (Pfsense Firewall) -> DHCP from 192.168.2.10 -> WAN
vmx0 192.168.1.102 (Freenas DMZ Nic via DHCP or static I don't care. not required at all)
vmx1 192.168.1.200 (Nextcloud Jail on Freenas, static)


Problem: I can only access the web from Nextcloud Jail if I enter the DMZ router to default router settings in my general freenas system settings.

FreeNas device.

VMX0 LAN Interface, DHCP by router 192.168.2.10 , /24 subnet
VMX1 DMZ interface, DHCP by 2nd router, 192.168.1.* /24 Subnet with other webservers
VMX2 (10G Point to Point with workstation) -> this one is wayne at this point

I sucessfully configured a nextcloud jail to access VMX1, static IP. But with the false default router in Freenas. Config:
Code:
root@freenas[~]# iocage get all nextcloud
CONFIG_VERSION:14.1
allow_chflags:0
allow_mlock:0
allow_mount:0
allow_mount_devfs:0
allow_mount_nullfs:0
allow_mount_procfs:0
allow_mount_tmpfs:0
allow_mount_zfs:0
allow_quotas:0
allow_raw_sockets:1
allow_set_hostname:1
allow_socket_af:0
allow_sysvipc:0
allow_tun:0
available:readonly
basejail:no
boot:on
bpf:no
children_max:0
cloned_release:11.2-RELEASE
comment:none
compression:lz4
compressratio:readonly
coredumpsize:off
count:1
cpuset:off
cputime:off
datasize:off
dedup:off
defaultrouter:192.168.1.1
defaultrouter6:none
depends:none
devfs_ruleset:4
dhcp:off
enforce_statfs:2
exec_clean:1
exec_fib:0
exec_jail_user:root
exec_poststart:/usr/bin/true
exec_poststop:/usr/bin/true
exec_prestart:/usr/bin/true
exec_prestop:/usr/bin/true
exec_start:/bin/sh /etc/rc
exec_stop:/bin/sh /etc/rc.shutdown
exec_system_jail_user:0
exec_system_user:root
exec_timeout:60
host_domainname:none
host_hostname:nextcloud
host_hostuuid:nextcloud
host_time:yes
hostid:fc7ded00-a73b-11e9-ba35-00155d026400
hostid_strict_check:off
interfaces:vnet0:bridge0
ip4:new
ip4_addr:vmx1|192.168.1.200/24
ip4_saddrsel:1
ip6:new
ip6_addr:none
ip6_saddrsel:1
jail_zfs:off
jail_zfs_dataset:iocage/jails/nextcloud/data
jail_zfs_mountpoint:none
last_started:2019-09-16 15:33:17
login_flags:-f root
mac_prefix:00155d
maxproc:off
memorylocked:off
memoryuse:off
mount_devfs:1
mount_fdescfs:1
mount_linprocfs:0
mount_procfs:0
mountpoint:readonly
msgqqueued:off
msgqsize:off
nmsgq:off
notes:none
nsemop:off
nshm:off
nthr:off
openfiles:off
origin:readonly
owner:root
pcpu:off
priority:99
pseudoterminals:off
quota:none
release:11.2-RELEASE-p11
reservation:none
resolver:nameserver 192.168.1.1
rlimits:off
securelevel:2
shmsize:off
stacksize:off
state:up
stop_timeout:30
swapuse:off
sync_state:none
sync_target:none
sync_tgt_zpool:none
sysvmsg:new
sysvsem:new
sysvshm:new
template:no
type:jail
used:readonly
vmemoryuse:off
vnet:off
vnet0_mac:00155dbab581 00155dbab582
vnet1_mac:none
vnet2_mac:none
vnet3_mac:none
vnet_default_interface:vmx1
vnet_interfaces:none
wallclock:off


I wanted to do the same for a plex jail. In LAN, VMX0, static, eg 192..168.2.25. Any how, I cant get it to work.

Config:
Code:
root@freenas[~]# iocage get all plex
CONFIG_VERSION:14.1
allow_chflags:0
allow_mlock:0
allow_mount:0
allow_mount_devfs:0
allow_mount_nullfs:0
allow_mount_procfs:0
allow_mount_tmpfs:0
allow_mount_zfs:0
allow_quotas:0
allow_raw_sockets:1
allow_set_hostname:1
allow_socket_af:0
allow_sysvipc:0
allow_tun:0
available:readonly
basejail:no
boot:on
bpf:no
children_max:0
cloned_release:11.2-RELEASE
comment:none
compression:lz4
compressratio:readonly
coredumpsize:off
count:1
cpuset:off
cputime:off
datasize:off
dedup:off
defaultrouter:192.168.2.10
defaultrouter6:none
depends:none
devfs_ruleset:4
dhcp:off
enforce_statfs:2
exec_clean:1
exec_fib:0
exec_jail_user:root
exec_poststart:/usr/bin/true
exec_poststop:/usr/bin/true
exec_prestart:/usr/bin/true
exec_prestop:/usr/bin/true
exec_start:/bin/sh /etc/rc
exec_stop:/bin/sh /etc/rc.shutdown
exec_system_jail_user:0
exec_system_user:root
exec_timeout:60
host_domainname:none
host_hostname:plex
host_hostuuid:plex
host_time:yes
hostid:fc7ded00-a73b-11e9-ba35-00155d026400
hostid_strict_check:off
interfaces:vnet0:bridge0
ip4:new
ip4_addr:vmx0|192.168.2.25/24
ip4_saddrsel:1
ip6:new
ip6_addr:none
ip6_saddrsel:1
jail_zfs:off
jail_zfs_dataset:iocage/jails/plex/data
jail_zfs_mountpoint:none
last_started:2019-09-16 16:34:31
login_flags:-f root
mac_prefix:00155d
maxproc:off
memorylocked:off
memoryuse:off
mount_devfs:1
mount_fdescfs:1
mount_linprocfs:0
mount_procfs:0
mountpoint:readonly
msgqqueued:off
msgqsize:off
nmsgq:off
notes:none
nsemop:off
nshm:off
nthr:off
openfiles:off
origin:readonly
owner:root
pcpu:off
priority:99
pseudoterminals:off
quota:none
release:11.2-RELEASE-p11
reservation:none
resolver:nameserver 192.168.2.10
rlimits:off
securelevel:2
shmsize:off
stacksize:off
state:up
stop_timeout:30
swapuse:off
sync_state:none
sync_target:none
sync_tgt_zpool:none
sysvmsg:new
sysvsem:new
sysvshm:new
template:no
type:jail
used:readonly
vmemoryuse:off
vnet:off
vnet0_mac:none
vnet1_mac:none
vnet2_mac:none
vnet3_mac:none
vnet_default_interface:vmx0
vnet_interfaces:none
wallclock:off


ifconfig in freenas says:

Code:
root@freenas[~]# ifconfig
vmx0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0c:29:44:55:04
        hwaddr 00:0c:29:44:55:04
        inet 192.168.2.95 netmask 0xffffff00 broadcast 192.168.2.255
        inet 192.168.2.25 netmask 0xffffff00 broadcast 192.168.2.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
vmx1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0c:29:44:55:0e
        hwaddr 00:0c:29:44:55:0e
        inet 192.168.1.102 netmask 0xffffff00 broadcast 192.168.1.255
        inet 192.168.1.200 netmask 0xffffff00 broadcast 192.168.1.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
vmx2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=60039b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0c:29:44:55:18
        hwaddr 00:0c:29:44:55:18
        inet 192.168.3.1 netmask 0xffffff00 broadcast 192.168.3.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:eb:1f:d8:f6:00
        nd6 options=1<PERFORMNUD>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vmx1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 2000


how would it be configured correctly?
 
Top