jail private net - route to internet

[stranger]

I'm trying out a few configs to get something similar to what I require (for reasons of security and network segregation).

I have my normal network on 10.0.0.0/16 and I've set up a few jails on 10.50.0.0/24. The idea is that there should be free communication between these jails without interference or listening in from other jails.
I've set these jails' gateway to be 10.50.0.1 and added this as an alias to the physical interface via the GUI.

The jails unfortunately still don't have connectivity to the internet. Other jails that are within 10.0.0.0/16 do have access to the internet.
So what as I doing wrong?

Is there some security setting that doesn't permit spanning of subnets? Any suggestions to get this working would be gratefully received.

thanks

 

[cyberjock]

Unless I'm mistaken your jails have to be on the same subnet as your "normal network" to function like you want (aka, properly).

 

[johnjaylward]

You should be able to configure a static route on your network to bridge the traffic across subnets. I usually do this either on the box itself that has the dual subnets, or on a dedicated router that will receive traffic from both subnets.

http://doc.freenas.org/9.3/freenas.html#static-routes

That is for the 9.3 docs, but it looks very similar in 9.2

If you need further control over what traffic is allowed to cross the subnet boundary, you will need to configure firewall rules as well.

Edit --
PS. Static routes are a pain to use and maintain if your network topology changes. If you are truly sharing traffic from your jails to your LAN or WAN, CyberJock's suggestion to leave them on the same subnet is better, then you only need firewall rules to limit traffic and not static routes and firewall rules.

 

[stranger]

When you add the gateway IP to your physical interface, the route is set up so no need for a static route there.
Here's the IPv4 routing table:

Internet:
Destination Gateway Flags Refs Use Netif Expire
default 10.0.0.1 UGS 0 505 igb2
10.0.0.0/16 link#4 U 0 875 igb2
10.0.0.70 link#4 UHS 0 42 lo0
10.10.0.0/16 link#5 U 0 5892 igb3
10.10.0.74 link#5 UHS 0 63 lo0
10.50.0.0/24 link#4 U 0 165 igb2
10.50.0.1 link#4 UHS 0 0 lo0
127.0.0.1 link#8 UH 0 3322 lo0


Please note that I'm just trying to get this working with igb2 and that the default route is on igb2.

So from the jail I can ping 10.50.0.1 and 10.0.0.70 but not 10.0.0.1.
My guess is that FreeNAS has a policy that prevents the spanning of subnets/acting as a gateway for the physical interface.
I think that I can do what I want with FreeBSD as it doesn't blast away my configurations but I'm very reluctant to give up the convenience of FreeNAS.

I think I basically need the detailed technical notes for how FreeNAS differs from FreeBSD (scripts etc) but that's not available from the docs. Anyone able to point me to such info or is it closed source??

Thanks for your replies.

 

[cyberjock]

I think I basically need the detailed technical notes for how FreeNAS differs from FreeBSD (scripts etc) but that's not available from the docs. Anyone able to point me to such info or is it closed source??

There is no such "technical notes". The source code is freely available at github though. ;)

 

[stranger]

I've considered going thru the git (once I find out where it is) but from experience I find that looking for what you are interested in, especially with GUIs, usually requires expert knowledge.
For instance, do the freeNAS python script implement the bridging for the interfaces or is that carried out else where, e.g. in warden?
Can anyone here point me in the right direction.

Thanks

btw a bugbear, one bit of advice I saw suggested not setting the defaultroute but then jails insist on it.
I can see why now - there's a bit of a hack using the default route to determine which physical interface to bridge to.

 

[spion]

Here are my notes on what I did to get things to work.

First thing is the default jail that freenas 9.10 is installing is for freebsd version 10

Code:

[root@openvpn910 /]# freebsd-version
10.3-RC2

I wasn't able to install any packages with the default pkg configuration as /usr/local/etc/pkg/repos/FreeBSD.conf pointed to a 9 repo. I updated this to the following

Code:

FreeBSD: {
  url: "pkg+http://pkg.FreeBSD.org/freebsd:10:x86:64/latest",
  mirror_type: "srv",
  ip_version: 4,
  enabled: yes
}

You'll also note i changed it so pkg preferred to use ipv4. Without this change pkg ran very very slow.

After this I ran

Code:

pkg update

and I was able to install packages.

After I followed the guide the first time everything worked great! then i restarted the jail a second time and everything broke :(

After a bit of investigating I found out the tun device number changed after every jail reboot. After this I was no longer able to use the VPN to connect to my internal network or the vpn network. The ifconfig line in the first post to rename the tun interface did not appear to work as expected. I did find https://forums.freebsd.org/threads/22143/ which outlined a way to get a consistent tun device name however I did not want to modify anything outside of the jail.

What I ended up doing was modifying the openvpn configuration to load the tunnel device from a separate file and make it so the ipfw script generates this file.

First thing I did was modify rc.conf

Code:

openvpn_enable="YES"
openvpn_if="tun"
openvpn_configfile="/mnt/openvpn/openvpn.conf"
openvpn_dir="/mnt/openvpn"

gateway_enable="YES"
firewall_enable="YES"
firewall_script="/usr/local/etc/ipfw.rules"

natd_enable="YES"
natd_interface=`/sbin/ifconfig -l | tr " " "\n" | /usr/bin/grep epair`
natd_flags=""

ip6addrctl_policy="ipv4_prefer"  # Use IPv4 instead of IPv6

I opted for using natd so I needed fewer firewall rules. I also remove the cloned_interfaces section since I create them later on in the ipfw script.

Next, the modifications to the ipfw.rules script

Code:

#!/bin/sh

EPAIR=$(/sbin/ifconfig -l | tr " " "\n" | /usr/bin/grep epair)
ipfw -q -f flush
ipfw -q add 50 divert 8668 ip4 from any to any via ${EPAIR}

OVPN_CONF_DIR="/mnt/openvpn"

TUN=$(ifconfig tun create)
echo "dev $TUN" > $OVPN_CONF_DIR/openvpn_tun.conf

Finally the modification of the openvpn configuration

Code:

port 10011
proto udp
#dev tun0
config /mnt/openvpn/openvpn_tun.conf
ca /mnt/openvpn/keys/ca.crt
cert /mnt/openvpn/keys/openvpn-server.crt #Server key
key /mnt/openvpn/keys/openvpn-server.key
dh /mnt/openvpn/keys/dh2048.pem #Diffie-Hellman parameters are now 2048 bits long
topology subnet
server 172.16.199.0 255.255.255.0 #Purple network
ifconfig-pool-persist ipp.txt
push "route 192.168.10.0 255.255.255.0" #Yellow network
route 192.168.10.3 255.255.255.0 172.16.199.1 #Routes traffic from the Yellow network side (192.168.10.0/24)
  #to the Purple network side (172.16.199.0/24)
keepalive 10 120
group nobody
user nobody
comp-lzo
persist-key
persist-tun
verb 3

After that I have a fully functioning openvpn configuration. The only other change (aside from the networks) is using the 'topology subnet' configuration. I made this change while troubleshooting so the routes on the client side were easier to reason about.

I also have some configuration I excluded that allows ldap auth. Didn't post it here since the main tutorial isn't for that.

Hopefully this helps someone else!