Jail Vimage Multiple Network Configuration

[WillD]

Good Evening, hope you are all doing well and thanks for reading this note.

I just installed FreeNAS after reading about it and doing some homework before diving in.

I'm trying to install a Jail Transmission using VIMAGE for multiple networks. I'm migrating my TransmissionBT from Vmware (OpenSuse, With vSphere switch) to FreeNAS. This setup worked jsut fine on Vmware. (Not comparing, just saying...lol)

PBI version - 2.77_1
FreeNAS version - FreeNAS-9.1.1-RELEASE-x64 (a752d35)
Private network interface - em0
Public network interface - em1

I'm trying to "dual" home this jail on two network, lets say 10.0.0.0/24 is the private interface and 20.0.0.0/24 is the public interface. With a default router as 20.0.0.1/24.

I applied 10.0.0.2/24 on the physical em0 that's the main private interface for the FreeNAS server.
I applied 20.0.0.2/24 on the physical em1 that's the main public interface for the FreeNAS server.

I built the jail successfully and configure it as following...

Jail setup
IPv4 Address - 10.0.0.3/24
IPv4 Aliase - 20.0.0.3/24
IPv4 default - 20.0.0.1
VIMAGE - Enabled

When i apply the virtual machine...

bridge0 is created with no address with interfaces epair0a and em0 added as member. In the jail epair0b is assigned the ip address 10.0.0.3/24 and 20.0.0.3/24 with the correct default gateway (20.0.0.1).

I'm not able to ping the 20.0.0.0/24 network from jail until i add em1 to bridge0. I think this were my problem starts.

Let me continue. Once i add the em1 to the bridge0 i can ping the physical interface 20.0.0.2/24 however not able to ping the gateway 20.0.0.1/24... However if i ping from the gateway to 20.0.0.3 i get a response as long as i'm pinging from the gateway?? Weird.

Other observation... i have full access to the 10.0.0.0/24 private address with you problems. Also, they are both connected to the same cisco router (gateway)...

So my problem in a nut is, i'm not able to keep the pass traffic to the public network... The traffic just fails... The private network seems to work just fine.

Please let me know if got any suggestions... i'm new to FreeNAS but have played with FreeBSD years ago. I took a chance to read the documentation so i'm not sure if this is a bug or i'm not doing something correct.

Thanks in advance.

 

[WillD]

**** SOLVED ****

I have had better days... This one just needed a little bit of forget about the GUI and work on the FreeBSD.

Basically, i miss-understood what i was doing (documentation). When i added the second em0, em1 and apair0b to bridge0 i created a "slit horizon" kind of layer two routing mess.

With a little help from this site...

http://wiki.polymorf.fr/index.php/Howto:FreeBSD_jail_vnet

And a little more troubleshooting. Solved. I created a separate bridge (bridge1) moved the em1 inteface to that bridge and created a new apair1a/b interface. Works like a hotdamn... well until i reboot.

I need to do some reading and figure how to configure the jail outside the gui setup, to put this option together. Now, off to moving over my media server... hmm.. this ZFS this looks interesting too. Maybe that first!

Salud!

 

[cyberjock]

My guess is you have done something wrong. Keep in mind that the USB stick is read only unless you mark it writable and then make your changes. Even at that, MANY configuration files are auto-generated on bootup, so you could make the file writable and save changes, then on reboot see them go away magically.

Generally speaking, if you are doing things outside the GUI, you are doing things that are probably not going to work out for you in the long run.

 

[cyberjock]

I'm just warning you that this is likely to turn into a much bigger can of worms than you have probably realized. Trying to circumvent the GUI is not a good idea. What you should focus on is what the GUI either isn't doing correctly or won't let you do(because fields don't exist or doesn't apply them properly), then put in tickets at bugs.freenas.org to get those changes implemented.

You may have better luck just using the full fledged FreeBSD and doing everything manually.

 

[WillD]

Thanks, fair comment... i was more focused on getting it to work. I'm assuming the location that the Jail GUI writes the network configuration is writable... no? If not you wouldn't be able to make changes to the jail config. I will try to chase down where the network config is stored... It has to be somewhere in the Jails area. I have that on a separate file system than the USB boot device.

I submitted a ticket. I wouldn't consider it a bug, more a lack of that feature. Basically, i want to have multiple broadcast domains/bridges/network segments presented to a Jail. Not an alias.

https://bugs.freenas.org/issues/3321

bug report submitted. Hope I contributed.

 

[cyberjock]

Well, some of the jails settings are in your config file(which is stored on your USB and partially used to generate some settings files on bootup like I explained). Some will be in the jail itself. You'd have to examine the code to see where the lines are drawn between the two.

To be honest, I cannot see why you'd ever have a want/need for 2 interfaces in a jail. You already get 10Gb/sec from just one. I can't imagine a jail ever being "limited" by those speeds, and I can't imagine any networking configuration where 2 network interfaces (that are completely artificial anyway since they are emulated) is useful at all. Not to sound like a total jerk, but it sounds like you failed "Networking-101" somewhere. I'd expect that your ticket will probably sit there until cancelled. You should probably go back and figure out what configuration you are trying to achieve and how you can achieve it with just 1 interface. Money says there is a way to do it, you just don't know how. :)

 

[WillD]

This has nothing to do with bandwidth/capacity or LAG interfaces.. Logical segmentation with epair/bridges is sufficient for what I'm doing. I will figure out a way to make this stable, i know it's a custom solution. But when you can separate layer 2 traffic it opens up a big window for flowing services. I respect that the GUI is kept simple/scalable, but I like to push the limits of possible. So anytime i can get under the hood, it's a good thing.

I have also noticed some other interesting "functions" that FreeNAS does.

Example this rather persistent interesting address that pops up randomly on a networking reset.

inet 67.215.65.132 netmask 0xff000000 broadcast 67.215.65.132

I would submit a ticket but probably get cancelled. Meh..

Need to do my homework on why that is happening since i failed my "Networking-101 somewhere".

Any who, your reputations precedes you CYBERJOCK... Thanks for keeping the forum interesting... with your ISM!

white flag waving... i don't need a flame way just trying to make this platform happen for my needs and share my experience with others. I'm new to FreeNAS but this is not my first rodeo.

 

[cyberjock]

I wasn't trying to flame you. I was just explaining that I don't see how it would matter. How would having 2(or even 10) virtual network interfaces affect latency? You realize that the 10Gb link is fast enough that the link is as latency free as you are going to get, unless you think you are going to do 10Gb/sec to/from your pools consistently. If so, you are my hero! I'm one of a few people that have seen pools hit 1GB/sec, and even then its not sustained.

Is that IP address your local address or is it something that appears to be statically called from FreeNAS somewhere? I'm just wondering if that's the IP address for a test bed at iXsystems. If so, that might explain why some people complain that their FN server isn't being "broadcasted". I've never had the problem, but /shrug. Who knows!?

 

[WillD]

That address seems to be registered with openDNS.

http://whois.net/ip-address-lookup/67.215.65.132

I do use OpenDNS Services for my local lan. But that address doesn't match any of their anycast DNS addresses. Plus it seems that there are some reported bugs with that address.

https://bugs.freenas.org/issues/1736

This is only resolved with a fresh reboot. It's a full /32 address so it could possibly be used for a sort of "null" interface address by FreeNAS. I need to figure out how to replicate this. Currently it's attached to my bridge0 interface but i have scene it on several other during my testing yesterday.

 

[titan_rw]

I've assigned a second NIC to a jail before. As at the time (not sure about now), it wasn't possible to get vlan support in a jail. So I simply used another NIC on an untagged vlan, and assigned it directly to the jail. It would be great if you could bridge the vlanX NIC's from freenas itself to the jails.

 

[xic044]

Anyone knows how to assign a socond NIC to a Jail?

 

[hsaff]

...need this as well...running FreeNAS device on 2 VLANs and Jail should contain Squid as a kind of filtering between VLANs. So jail should act as a gateway with routing between VLAN. Anyone done before? Need to step to FreeBSD?

 

[cyberjock]

So install something like Squid in the jail and set it up? I'm not really sure what the limitation is aside from perhaps experience and knowledge to actually do this yourself. This is very much a "FreeBSD" thing as this kind of thing is definitely NOT something a file server would typically handle. But anyone with sufficient knowledge of FreeBSD should be able to get it working in a jail.

 

[hsaff]

So install something like Squid in the jail and set it up? I'm not really sure what the limitation is aside from perhaps experience and knowledge to actually do this yourself. This is very much a "FreeBSD" thing as this kind of thing is definitely NOT something a file server would typically handle. But anyone with sufficient knowledge of FreeBSD should be able to get it working in a jail.

...Maybe. I already googled around and it seems that even FreeBSD is limited in its jails network config. Ideally I could choose to assign jails to whatever network segment/ VLAN I want and even assign them to more than one.
Maybe the only way out is to completely virtualise all services (FreeNAS, Squid, ...) on bare metal/VMware or maybe bhyve?

Would be nice if the thread is kept open and updated in case the feature becomes available

 

[michaeleino]

Dears,
It's difficulty, is the easiness :)
first create the jail with VIMAGE & unchecked NAT & vanilla...
with FreeNAS termial go to JAILS directory for me /mnt/strg/jails/
for each jail you create there is a .jail.meta folder (Ex. Jail name Vbox43 there is a .Vbox43.meta ) the .Vbox43.meta is the configuration folder for this jail,
cd to this folder, edit & add to the bottom of files as follow:
jail-pre-start

Code:

#added by mic
ifconfig bridge172 create ## set bridgeN as you need
ifconfig bridge172 up
ifconfig epair172 create ## set epairN as you need
ifconfig bridge172 addm epair172a addm em1 ## Change em1 to your NIC name from ifconfig
ifconfig epair172a up

jail-post-start

Code:

#added by mic
ifconfig epair172b vnet "${JAILNAME}"
#jexec "${JAILNAME}" ifconfig epair172b up
jexec "${JAILNAME}" ifconfig epair172b inet 192.168.10.11 netmask 255.255.255.0 up #set ifconfig as your needs & don't forget to add UP :) 

jail-pre-stop

Code:

#added by mic
ifconfig bridge172 destroy
ifconfig epair172a destroy

also files is attached ;) simple ?

 

[hsaff]

Great!!!...you are the man!!!!...And you are right: it is easy if you know how to do :D

Will try to integrate this into my configuration at weekend and let you know.

MANY THANKS!!!

 

[braynshock]

I know this thread is old, but wanted to just say Michaeleino's config worked like a champ...
I have a pfSense VM in a virtualbox jail, with 2 bridged NICs, one on the internet, one on the Intranet.

 

[hopey]

...and again, michaeleino you had a perfect idea - thanks a lot!

also worked for me with freenas 9.3

 

[noobnas]

Worked awesome for me too. I was able to do this in a virtual box jail, and I bridged a second nic so that I could use SMB multipathing in Windows.

 

[airflow]

Hey, thanks for this interesting thread. I'd also like to configure a jail in my FreeNAS installation to have two logical interfaces. You guys seem to have done it with the config-snippets given above. My question to you: Has anybody done this by using VLAN-interfaces as the source-interface which is to be connected to the jail?

In my installation, I use a lagg0-interface with two physical interfaces. One logical network is untagged (it's the network the main FreeNAS is on), the other is tagged. I added the VLAN-id via the GUI, and for tests I configured an IP in the range of the VLAN (also via the GUI). This works perfectly, and I can connect to other systems in the VLAN back and forth my using the new IP. So the basic connectivity, dot1q-tagging etc from FreeNAS works fine.

The next logical step is to connect this working VLAN (tag=50) to the jail.

Code:

# jail-pre-start
ifconfig bridge50 create ## set bridgeN as you need
ifconfig bridge50 up
ifconfig epair50 create ## set epairN as you need
ifconfig bridge50 addm epair50a addm vlan50 ## Change em1 to your NIC name from ifconfig
ifconfig epair50a up

# jail-post-start
ifconfig epair50b vnet "${JAILNAME}"
#jexec "${JAILNAME}" ifconfig epair50b up
jexec "${JAILNAME}" ifconfig epair50b inet 192.168.50.11 netmask 255.255.255.0 up

In the host I see now these interfaces (selection):

Code:

vlan50: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=303<RXCSUM,TXCSUM,TSO4,TSO6>
        ether d0:50:99:2d:fc:aa
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
        vlan: 50 parent interface: lagg0

bridge50: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:ac:b5:e2:f4:32
        nd6 options=1<PERFORMNUD>
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vlan50 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 55
        member: epair50a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 9 priority 128 path cost 2000

epair50a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:ff:20:00:09:0a
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active

In the jail, I see these interfaces:

Code:

epair1b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:ff:70:00:0c:0b
        inet 172.22.2.32 netmask 0xffffff00 broadcast 172.22.2.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
epair50b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:ff:70:00:0a:0b
        inet 192.168.50.11 netmask 0xffffff00 broadcast 192.168.50.255
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active

The upper interface epair1b is the one configured via the FreeNAS-GUI, and connects to lagg0 directly (untagged). epair50b is configured by the config-snippets above, and connects to vlan50 (which physically also uses lagg0, tagged). For a reason I don't understand, the upper interface works, while the lower does not. When I ping from the jail, I don't even see tagged ARP-packets going out lagg0. If I do it from the host (FreeNAS), I see the packets, and everything works. So it is the stitching between the vlan-interface and jail via the bridge, which doesn't work. Any ideas?

Thanks,
airflow