VLan, PFsense, and Jails

FooisOP

Cadet
Joined
Oct 21, 2019
Messages
7
I have an issue to set up properly with vlans. I spend entire week battling this annoying issue.

I got WAN, LAN, and OPT1 port on my PFsense Box. My Freenas is connected to the OPT1 port. Wifi, and other lans are connected to Lan Port. To simplify things.

Pfsense: 192.168.1.1
Freenas: 192.168.2.11
Wifi Vlan 2: 192.168.3.1

I have my jails set as a static within the 2.x subnet. I have other jails listed but for the sake of it, I want focus one specific jail. If I get this jail working on vlan, then I can do the same for others.

MineOS: 192.168.2.100.

I am able to connect mineos, and other jails. However, I want to build a stronger granulated control in my firewalls. Since all of my jails resides in opt1(server). I feel like opening up too many ports to my freenas is not the best way to go. See my images.

The mineOS is fine granulated, but it messes up the freenas communications if I want to download new jail apps, or update freenas. Thus, I want this to be moved over to VLAN port 9.

I decided to build a VLAN 9(MineOS). I then set them in 10.0.1.1 range. I want all of my jails in 10.0.x.x range to simplify things.

MineOS (VLAN9): 10.0.1.2

I cannot access to mineOS nor any other jails with this setup. Pfsense and my machine can't ping to 10.0.1.2.

In my DHCP Leases tab, it says that 10.0.1.2 is online.

In my firewall rules, I set all ports to open (for testing purposes). One in Wifi Vlan 2, and Server. I can communicate to the server and mineOS no problems. BUT with mineOS VLAN rules, I cannot.

I tried setting up in freenas vlans 9 to match the same number as pfsense vlans. It still won't detect. I have tried many different approach to this problem with various of settings, overlaying rules, and other means. Nothing works other than having jails and freenas in the same interface assignments.

What gives? What am I missing here?
 

Attachments

  • Screenshot_2019-10-21_17-47-20.png
    Screenshot_2019-10-21_17-47-20.png
    97.1 KB · Views: 622
  • Screenshot_2019-10-21_17-49-32.png
    Screenshot_2019-10-21_17-49-32.png
    106.5 KB · Views: 646
  • Screenshot_2019-10-21_17-50-14.png
    Screenshot_2019-10-21_17-50-14.png
    19.6 KB · Views: 544

Heracles

Wizard
Joined
Feb 2, 2018
Messages
1,401
Hey FooisOP,

All your screenshots are about packet filtering rules. There rules are layer 3 (IP). What you need to fix first is layer 2.

What switch are you using ? Are you sure that switch supports vlans ? What VLAN ID did you use ? Are you doing a sinle .1Q vlan uplink or are you doing port base vlan on some ports then merged in a .1Q trunk ?

Please, tell us more about layer 1 (cabling) and layer 2 (switches, vlan ID, ...)

Only from there we will be able to help you.
 

FooisOP

Cadet
Joined
Oct 21, 2019
Messages
7
Thanks for the reply.

What switch are you using ? Are you sure that switch supports vlans ?

I thought I made it clear about my physical layer.

I got WAN, LAN, and OPT1 port on my PFsense Box. My Freenas is connected to the OPT1 port. Wifi, and other lans are connected to Lan Port.

I only got one physical cable from freenas to OPT1 port. I do have unmanaged switch but it is not active nor plugged in. It is not used. I don't need a switch since I only need two ports which my pfsense box has. One for Wifi, and one for server. As I said above. If I do need a switch, I will buy managed switch that supports vlan.


What VLAN ID did you use ? Are you doing a sinle .1Q vlan uplink or are you doing port base vlan on some ports then merged in a .1Q trunk ?

Again

MineOS (VLAN9): 10.0.1.2

VLAN 9. That is what I am using. I don't see any options about "trunk" in pfsense box. So I have no clue what that is nor what it does.

I do not know how to edit my post. There is no "edit" button anywhere. I forgot to include the interface. I'll include the VLAN images too.
2nd image is from PFsense VLAN tab.
3rd image is from FreeNAS VLAN tab.
 

Attachments

  • Screenshot_2019-10-21_19-59-22.png
    Screenshot_2019-10-21_19-59-22.png
    26.2 KB · Views: 488
  • Screenshot_2019-10-22_08-21-52.png
    Screenshot_2019-10-22_08-21-52.png
    2.8 KB · Views: 501
  • Screenshot_2019-10-22_08-22-32.png
    Screenshot_2019-10-22_08-22-32.png
    11.9 KB · Views: 478

Heracles

Wizard
Joined
Feb 2, 2018
Messages
1,401
Hi again,

No, it was not clear at all. You used the same wording for the connection of a single FreeNAS server to a single pfSense port, and also for the connection of multiple network devices to another single pfSense port.

Another thing that was not mentioned and is now visible is your hardware. Your FreeNAS is using a Realtek NIC. These ones are known to be problematic with FreeNAS. Any chance to get yourself an good NIC like an Intel one ?

What you need is to design your network first. Only once done can you start to implement it.

How many vlans are you looking for between OPT1 and FreeNAS ?
What ID will you give to each ?
What IP Addressing will you use on each one ?
Any DHCP on these or all static IP ?

Once done, you need to create all your vlans in both FreeNAS and pfSense.
Once the virtual network interfaces are created for these vlans, you need to configure the IP address on each virtual NIC.
Once every NIC is properly configured, you need to configure your routIng for FreeNAS.
Only after that will you be able to create your jails with virtual IP stacks that will allow you to what you need.

Without complete virtual IP stacks, your routing will end up asymetric and pfSense will not let you go through.

Have fun designing your system,
 

FooisOP

Cadet
Joined
Oct 21, 2019
Messages
7
Hi again,
No, it was not clear at all. You used the same wording for the connection of a single FreeNAS server to a single pfSense port, and also for the connection of multiple network devices to another single pfSense port.

Oh okay. I'll make sure I'll do it better next time.

Another thing that was not mentioned and is now visible is your hardware. Your FreeNAS is using a Realtek NIC. These ones are known to be problematic with FreeNAS. Any chance to get yourself an good NIC like an Intel one ?

I did not realize that Realtek is a real pita with FreeNAS, and BSD systems. I should've researched better.
My PFSense box is AMD + Intel luckily. Here are the specs.

CPU: AMD Embedded G series GX-412TC, 1.0Ghz/1.4Ghz quad Jaguar core with 64 bit and AES-NI support, 32K data + 32K instruction cache per core, shared 2MB L2 cache.
NIC: 3 x Gbit (Intel i210AT)
Memory: 4 GB ECC DDR3-1333 DRAM

1) How many vlans are you looking for between OPT1 and FreeNAS ?
2) What ID will you give to each ?
3) What IP Addressing will you use on each one ?
4) Any DHCP on these or all static IP ?

1) I plan to have 3 VLANs for 3 different Jails.
2) Are you talking about VLAN ID? If so, VLAN 3, VLAN4, VLAN 5.
3/4) I want each Jail has it's own IP address with NO DCHP. Static only.

Each Jail should be in 10.0.x.x range. These will be public facing. I am aware of the risk for exposing my services to the world.
Minecraft 10.0.0.2
CloudStorage: 10.0.0.3 --Or should it be 10.0.1.1?
Matrix: 10.0.0.4 --Or should it be 10.0.2.1?

Users who joins takes IP slot. It increments as more people join. Therefore, it would be better to have 10.0.x.1/24 for each jails. Am I right?

I am debating if I should get VPN for these services.

Thanks for helping me out.
 

Heracles

Wizard
Joined
Feb 2, 2018
Messages
1,401
HI again,

So for your config, you only need 3 networks with /30 netmasks.

So Lets go this way :
VLAN 3 : Minecraft
pfSense = 10.0.0.1 / 30 (or 255.255.255.252) and Jail = 10.0.0.2, same netmask. Default GW = 10.0.0.1
VLAN 4 : Cloud
pfSense = 10.0.0.5 / 30 ; Jail = 10.0.0.6, same netmask, Default GW = 10.0.0.5
VLAN 5 : Matrix
pFSense = 10.0.0.9 / 30 ; Jail = 10.0.0.10, same netmask, Default GW = 10.0.0.9

You will have to go in pfSense first and create all these interfaces, with these VLAN IDs, these IPs and netmask.
Once done, you do in FreeNAS and create the corresponding VLAN interfaces and associate each of them with the proper Jail.

That of course is if your Realtek card does support VLAN tagging and it works under FreeNAS....

Good luck with your setup,
 

FooisOP

Cadet
Joined
Oct 21, 2019
Messages
7
I have done exactly like that before and it didn't work. I tried again to the exact details with your suggestion. It still doesn't work. When I open up mineos shell, and ping to 10.0.0.1, it would get a ping then it hangs. I tried many different options and it's still not working. I pinged 10.0.0.2 from pfsense, it hangs with 100 percent packet loss.

Realtek card

Except it is not RealTek Card. I just gave you the specs. It says Intel I210AT. I'll provide the image what it says from the pfsense terminal. The box supports VLAN. I have wireless AP that runs VLAN2 and it works. Pfsense is not the issue but FreeNAS is the issue.
 

Attachments

  • Screenshot_2019-10-24_10-35-33.png
    Screenshot_2019-10-24_10-35-33.png
    2.2 KB · Views: 482
  • Screenshot_2019-10-24_10-35-15.png
    Screenshot_2019-10-24_10-35-15.png
    7.6 KB · Views: 449
  • Screenshot_2019-10-24_10-34-44.png
    Screenshot_2019-10-24_10-34-44.png
    15.2 KB · Views: 427
  • Screenshot_2019-10-24_10-33-58.png
    Screenshot_2019-10-24_10-33-58.png
    15.5 KB · Views: 439
  • Screenshot_2019-10-24_11-35-08.png
    Screenshot_2019-10-24_11-35-08.png
    3.9 KB · Views: 471

Heracles

Wizard
Joined
Feb 2, 2018
Messages
1,401
Except it is not RealTek Card.

The maker of the card does not mean much... At the end, for the card to be named RExxx, it really looks like it is driven by the Realtek driver. Here, my Intel NIC ports are igbX and my broadcom are bgeX. If whoever made your NIC used a Realtek chipset, your NIC ends up being a Realtek.

For a single packet to go through and not the other, I suspect the NIC and the driver first. Unless you can find yourself a real and good NIC, I doubt I can help you more on this one...

Good luck,
 

FooisOP

Cadet
Joined
Oct 21, 2019
Messages
7
The maker of the card does not mean much... At the end, for the card to be named RExxx, it really looks like it is driven by the Realtek driver. Here, my Intel NIC ports are igbX and my broadcom are bgeX. If whoever made your NIC used a Realtek chipset, your NIC ends up being a Realtek.

I can't find anywhere about NIC protocol numbering systems. Nothing says about it.

All realtek cards shows as "RExxx? "

If that is the case, you're mistaken. Look again, it is igbX. That's intel. I think you're looking at wrong pictures in wrong thread.
 

Heracles

Wizard
Joined
Feb 2, 2018
Messages
1,401
The name of the card is associated with the name of its driver.

I am looking at the 3rd screenshot in your second post. That one is clearly from FreeNAS' WebUI and clearly says RExxx as the name of the NIC.

That is what makes me think that your NIC is a Realtek.

You may go read this about naming Realtek and have an idea of how problematic they are...

You can also read this...

So Yes, I am pretty sure that card is a Realtek and as such, not reliable and not something I can help you troubleshoot...
 

FooisOP

Cadet
Joined
Oct 21, 2019
Messages
7
Oh now I see what you're saying. This whole time, I was assuming you were talking about PFSense box that uses RealTEK NIC.

Yeah, it checks out. My PC mobo uses realtek NIC. Good spotting.

I'll figure something else then. Perhaps, buy a intel NIC pci card and use that instead. Hopefully that would fix the VLAN / Jails issue. I'll come back and report to you once I get Intel NIC PCI card.
 
Top