Quick question about SFP+ modules.

[nightshade00013]

So I am going to see if I can't get rid of the network bottleneck for my FreeNAS and desktop.

Here is what I am looking at doing:

Grab 4 Mellanox ConnectX-2 NIC's

Put one in my desktop, put one in my FreeNAS and put two into my OPNsense box and bridge my LAN connection with the two Mellanox cards so I do not need to pick up a switch for now. I know this can theoretically be slower than an actual switch but later on my plan is to get a switch and still have a spare Mellanox NIC for something else and the bottleneck will be shifted from my 1GB NIC's to the pool in my FreeNAS. And my router Should have more than enough power to handle this with a dual Xeon L5630 set of CPU's and 8GB of ECC ram which I can double if needed as I have two more sticks of ECC ram sitting around.


Anyway I found a place with cables and transceivers and want to know if they will work correctly (or theoretically so) before I pull the trigger on it.

https://www.fs.com/products/11589.html Transceiver

https://www.fs.com/products/41735.html Cable

My main question is the transceiver compatibility, from what I read the Mellanox cards will work with pretty much anything but they do have transceivers that are compatible with particular brands.

Anyway comments on the setup would also be good. I do know that I could just buy two and run a link between my FreeNAS and my desktop but I do plan to expand my 10G hardware over time so having the card in my router will just be another learning experience and I can always run a single line direct if I can't get something to work.

 

[acp]

I used bought a CX312A (dual port) and two CX311A ConnectX-3 boards. There dual port is suppose to go into the FreeNAS box but the card came as full profile. Waiting for the low profile bracket. The 311's came with low profile bracket so I used that to test. The card was seen by FreeNAS with no issues. I'm running 11.1. the other system is running xenserver 7.3 and it had no issues with either. I'm using this:

2m (7ft) Mellanox MCP2101-X002B Compatible 10G SFP+ Passive Direct Attach Copper Twinax Cable

To connect the two. Bought from the fiberstore.

My only gripe is that sometimes the FreeNAS box wouldn't always see the connection being up. Being impatient I would stop/start the driver for the mlxen interface. A switch would most likely solve this and I do have two ports available, but I need three, so that is why I bought the dual port card.

This is my first entrance into 10gig. I have read the transceiver can be picky but mellonax device are less so. I got the Gen 3 cards off eBay pretty cheap. The single ports only require a 4x slot but they are gen3 pcie devices. Don't know how well they would work if put into a gen2 slot. I had an open slot so no worries there. The speed so far is awesome. It seems possible to have my VMs be diskless and run strictly over a network.



Sent from my Nexus 5X using Tapatalk

 

[nightshade00013]

Everything I am looking at is full height brackets. It's all PCIe gen 2 but not worried as I have a ton of open slots and the ConnectX-2 should work perfect. Basically the same board in my router as well as server except on has built in SAS and the other does not. The board I am using in my router does not have video nor does the IPMI work but honestly it has not been a problem. If I need a screen I just toss in a graphics card and then pull it when I am done. I figure that by dropping two cards into the router I can easily bridge everything and give my entire network a decent access speed to the FreeNAS while allowing my desktop to transfer large files a lot faster.

In total what all I am looking at including cable will be less than 150.00 and later on when I grab a switch I can just grab some optics and cables as needed.

 

[BigDave]

Anyway I found a place with cables and transceivers and want to know if they will work correctly (or theoretically so) before I pull the trigger on it.

https://www.fs.com/products/11589.html Transceiver

https://www.fs.com/products/41735.html Cable

My main question is the transceiver compatibility, from what I read the Mellanox cards will work with pretty much anything but they do have transceivers that are compatible with particular brands.

Contact the FiberStore, I've found their customer service to be both knowledgeable and timely. They will let you know if the transceivers you've chosen will work with those NICs.

 

[nightshade00013]

Thanks, I did last night but since it's the weekend I figure I will not get a response till Monday or Tuesday and I am chomping at the bit to get going.

 

[BigDave]

Thanks, I did last night but since it's the weekend I figure I will not get a response till Monday or Tuesday and I am chomping at the bit to get going.

iirc FS customer service is US based and will be closed for NY DAY, I would anticipate a response sometime Tuesday. If they treat you, half as good as I was treated, you are in for a delightful surprise.
10GB networking was a lot of fun for me, just FYI, when starting out I made the mistake of going with some older hardware (chelsio T310 cards) and they are collecting dust now. I now run Chelsio T420 & Intel X520 (dual port) cards in my system and if I could wind back time, I would never have bought single port cards or 8 year old tech.
(of course I'm refering to the T310 cards).

 

[nightshade00013]

I hope they do good. Have heard a lot of good things so far.

But being on a budget limits what I can do a lot of times hence the reason why I am going with some older stuff and not getting a switch right away.

I am making money with my crypto mining and that is what is buying the parts actually but I still have to expand it out as well so I can't throw a few hundred into a switch just yet.

 

[BigDave]

I'm very familiar with budget limits, but staying under the limitation is not one if my strong points:p:D:(

 

[nightshade00013]

Same here. I can make a small fortune buying computer stuff. Just have to start with a large fortune first and go from there.

I also love tinkering with cars, built a few rifles and shoot some pistol competitions when I can get the chance. And they all take even more money. However my wife has a huge advantage as most of the time everything just works and she doesn't even have to think about it. And if it doesn't work she doesn't even worry as I can usually fix it.

I should also mention that my offer was accepted on the Mellanox cards. 14.50 each isn't too bad IMHO I remember buying 1GB NIC's for that ten years ago.

 

[jgreco]

I'm very familiar with budget limits, but staying under the limitation is not one if my strong points:p:D:(

Isn't the point to always exceed the number?

 

[BigDave]

Isn't the point to always exceed the number?

lol
I know I can always find a rationalization for doing so...

 

[nightshade00013]

Ok, so I got a message back last night about the Mellanox ConnectX-2 I am looking at and they assured me that their "Generic" transceiver should work just fine. I put in the order about 4:00 AM so hopefully everything should show up soon.

I put an offer in on Ebay the other day cutting the price of each one down to 14.50 each and I bought four. Hopefully I can get the bridge setup easy in OpnSense and make everything happy with as little difficulty as possible.

And in case someone is looking for a couple cheap 10G nic's I will post the link here https://www.ebay.com/itm/351416547732

They are also selling a lot of 25 for 12.50 each but if you buy a couple they are sure to accept 14.50

 

[nightshade00013]

Ok, FreeNAS picked up the Mellanox card without a hitch. With Windows 7 the latest driver would not install and extracting the MSI and CAB with the driver on the latest version gave a driver that just didn't work. I was however able to get the 4.8 version of the Mellanox driver to work without a hitch, just had to force windows to use the older version and after a reboot I was good to go.

What I have to do now it get OpnSense to load the drivers. Seems they strip the driver out so I will have to build and package it back in which is what I am looking at doing in a few. I believe pfSense does the same thing because "there are no internet connections that fast" but to have a 10G connection backbone isn't out of the question IMHO. Plus leaving the drivers in place for NIC's isn't going to be a huge bloat to the OS really. Anyway was going to build the drivers on FreeNAS in a jail but then something hit me. FreeNAS is based on the same version as my OpnSense box. I think I may just be able to pull the mlx4ib.ko from the one and drop it in the other.

Anyway, will report back and see what happens.

 

[nightshade00013]

Hmmm well that didn't work so well. Must be something built different in how FreeNAS uses the driver.

Guess I have to do it the hard way then.

https://www.reddit.com/r/homelab/comments/6ga3sq/using_10gb_sfp_mellanox_connectx2_as_your_lan/

 

[Chris Moore]

Anyway I found a place with cables and transceivers and want to know if they will work correctly (or theoretically so) before I pull the trigger on it.

https://www.fs.com/products/11589.html Transceiver

https://www.fs.com/products/41735.html Cable

My main question is the transceiver compatibility, from what I read the Mellanox cards will work with pretty much anything but they do have transceivers that are compatible with particular brands.

I have not priced that hardware in several years, but I had no idea the prices had gotten that low. Last time I bought one of those transceivers, I recall that it was over $300. These prices are much more reasonable.
I don't know if you have looked at this ever, but you might want to look into it as I found it interesting when I was setting up my 10GB network at home.
https://youtu.be/MgNpI6VAAhI

 

[jgreco]

Yeah, new optics, especially vendor-locked ones, are outrageously priced. Lots of us just buy from secondhand sources. You can build a fully redundant network at a very low cost if you go about it the right way.

 

[BigDave]

I have not priced that hardware in several years, but I had no idea the prices had gotten that low. Last time I bought one of those transceivers, I recall that it was over $300. These prices are much more reasonable.
I don't know if you have looked at this ever, but you might want to look into it as I found it interesting when I was setting up my 10GB network at home.
https://youtu.be/MgNpI6VAAhI

Hey Chris, check out the 10GB Networking Primer that jgreco authored about 3 years ago.
It was turned into a resource and has been updated, it was my blue print for my home network. IMHO you should link to that, instead of an offsite video:p

 

[Chris Moore]

Hey Chris, check out the 10GB Networking Primer that jgreco authored about 3 years ago.
It was turned into a resource and has been updated, it was my blue print for my home network. IMHO you should link to that, instead of an offsite video:p

I guess I'll have to read his write-up up but the video I linked to is actually part of a series and it's very good you should watch it

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[nightshade00013]

Yep a lot of stuff is out there. The Mellanox cards are not vendor locked which is great for me. I just bought generic ones for now. Looking at a Quanta LB6M when I get to the point of picking up a switch. With that and a little work I should be able to run my whole house at up to 10Gbit speeds.

Only problem I am having right now is getting the driver to compile correctly. I can generally follow instructions but at every turn something fails. Seems there are just a bunch of prerequisites that the average person doesn't know a ton about but isn't mentioned.... I just need the stuff to load the drivers in the router for a little while till I can grab the switch.

 

[Chris Moore]

Yep a lot of stuff is out there. The Mellanox cards are not vendor locked which is great for me. I just bought generic ones for now. Looking at a Quanta LB6M when I get to the point of picking up a switch. With that and a little work I should be able to run my whole house at up to 10Gbit speeds.

Only problem I am having right now is getting the driver to compile correctly. I can generally follow instructions but at every turn something fails. Seems there are just a bunch of prerequisites that the average person doesn't know a ton about but isn't mentioned.... I just need the stuff to load the drivers in the router for a little while till I can grab the switch.

What kind of budget are you looking at and how many ports?


Sent from my SAMSUNG-SGH-I537 using Tapatalk