Quick question about SFP+ modules.

[nightshade00013]

@Chris Moore Shoestring and as much as possible, LOL. Right now I am tapped out for a month or two. I have to find something with at least two 10G ports and would prefer to have a couple extra for additions. That is why the Quanta would work out well. Right now I have an 8 port gigabit switch sitting next to the router which has to daisy a couple other switches. One line off the main switch goes to my living room where a five port switch distributes to the TV, my dekstop and my wife's laptop. The other switch that goes from the first 8 port heads into where the FreeNAS sits and has a wireless router set as an access point and the FreeNAS.

My main goal was to get the FreeNAS and my desktop into as fast and direct a route as possible hence using a couple extra cards for now in my router and bridging them with the LAN. Then later on I can just grab a switch and slowly add whatever I choose. I know that I can just drop the line straight from the FreeNAS to my Desktop and I have enough OM3 to do that but having it ran to where the router is just makes dropping in a single switch easier since my plan is to replace the 8 port with one larger switch.

 

[nightshade00013]

Well I finally got the drivers to load build and load after fighting it for a while. I got frustrated in trying to do it here and there multiple times so I just did it on the router. Had to install git and then just pull the FreeBSD source. Worked and loaded without a hitch so if someone wants to do something like this I have no issue with sharing the drivers. Now to get them to bridge and see what all I need to do.

 

[nightshade00013]

Annnnnnddddd..... SUCCESS!!!

Credits go to a post on the pfSense boards for the instructions to setup the bridge. https://forum.pfsense.org/index.php?topic=48947.0

1. Assign your additional interfaces and enable them with type 'none'. They will come up as OPT1, OPT2 etc.
2. Switch bridge filtering from the bridge members onto the bridge itself (assuming you don't need to apply firewall rules between devices on the bridge/switch). Go to System: Advanced: System Tunables: and edit the two values.
Change net.link.bridge.pfil_member to 0.
Change net.link.bridge.pfil_bridge to 1.
Apply these changes.
3. Now create a bridge in Interfaces: (assign): Bridges: and add to it the additional interfaces you just created, you can select multiple interfaces by holding Ctrl. I named the bridge 'Switch Configuration' to remind me how I have configured it.
4. Now go to Interfaces: (assign) and change the LAN assignment to bridge0. Save and reconnect your ethernet cable to one of the bridge interfaces. It should come back up, however you will want to make sure you have console access before you do this as if you've done something different you could end up locked out!

5. Assign the interface that was originally assigned to LAN and enable it with type 'none'. Add it to bridge0 to include it in the 'switch'.

Just have to reboot after the changes are applied in OpnSense but it is working just fine. This post is coming from a bridged network off of a Mellanox ConnectX-2 10G connection. I now have one #3!!!!! of a headache but it is working. Just have to make a minor change in FreeNAS interfaces and I am good to go.

 

[nightshade00013]

So turned off the Intel Gigabit adapter in FreeNAS and set the Mellanox to the same IP static. Did a reboot and just copied a couple things to and from my computer. It's a lot faster than it was for sure. Right now I am hitting speed bumps as my drive is a little on the slow side but going from 100 to 300 MB/s is still a major cut in transfer times. Did come across one issue last night though I think I got it figured out. I think that the Mellanox adapters were defaulting to an odd MTU and since they are bridged with a 1Gbit connection that is likely defaulting to a 1500MTU I think they were causing a crash. I set all of the interfaces in the bridge to an MTU of 1500 and did a restart and it has been stable through multiple transfers. The memory usage also dropped from 20% to 7% on the router.

In Windows the MTU defaulted to 1524 or 1514 but it was odd and I was tired so I changed it right away.

While it's not exactly ideal this will work until I can purchase a switch and just switch to a single interface on the router.