10 Gig Networking Primer

[Ericloewe]

Don't forget the optics. Depending on the vendor, they may be a very significant cost.

 

[JustinClift]

As a data point, cable pricing (on Ebay) fluctuates wildly. I bought a 30M QSFP+ fibre cable a few months ago for £18.95. They seem to be selling for several times that price now.

Hmmm... maybe I should have stocked up. :D

 

[JustinClift]

Don't forget the optics. Depending on the vendor, they may be a very significant cost.

Yeah. The cable + optics looks like it'll be more expensive than the cards.

 

[JustinClift]

@Ericloewe Fibre cables and optics aren't something I personally have much experience with yet, so it's hard to advise correctly.

For a 10GbE SFP+ connection, do these seem like the correct bits?

• http://www.ebay.com/itm/152361680500 ← Seems to be a Finistar clone of the Mellanox MFM1T02A-SR transceivers
• http://www.ebay.co.uk/itm/331643153000 ← 25 metre cable. OM2 (seems to mean 850nm), and using LC-LC connectors

Hmmm, this 30m cable might be a better bet? http://www.ebay.com/itm/252720988213

 

[Ericloewe]

@Ericloewe Fibre cables and optics aren't something I personally have much experience with yet, so it's hard to advise correctly.

For a 10GbE SFP+ connection, do these seem like the correct bits?

• http://www.ebay.com/itm/152361680500 ← Seems to be Finistar clone of the Mellanox MFM1T02A-SR transceivers
• http://www.ebay.co.uk/itm/331643153000 ← 25 metre cable. OM2 (seems to mean 850nm), and using LC-LC connectors

Hmmm, this 30m cable might be a better bet? http://www.ebay.com/itm/252720988213

It's a good idea to stick to OM3 and OM4 fiber. The transceivers should be ok, but research is definitely in order.

 

[Stryf]

Just bought the Ubiquiti ES-16-XG for around $551 but need to wait a week for it to arrive.

 

[JustinClift]

Thats amazing! 40 gigabit connection! Thats crazy cheap. I will definitly look into this

Heh Heh Heh, not exactly. ;)

Those cards can operate in one of two modes (changeable via software setting):

• 10Gb Ethernet, or
• 40Gb Infiniband

At present, FreeNAS only supports the Ethernet mode, and doesn't natively support the Infiniband mode. So, its probably best to look upon those particular cards as 10GbE ones. :)

As a data point, I'm putting some time into adding the Infiniband bits. But it's an "in my spare time" thing so could take a few months (or more).

 

[c32767a]

There are some alternatives. One easy target is if gigabit is acceptable for your endpoints (PC's and other clients), it is not that hard to find a gigabit switch with several 10G uplinks. The cheapest decent one I've seen in (2014) times is probably the Dell Networking 5500 series, such as the 5524, often available for around $400 on eBay. That model comes with two 10G SFP+ slots, which could be used for a FreeNAS box and a workstation at 10G, while also allowing all remaining stations to share in the 10G goodness. Now that it's 2016 we're also seeing the Dell Networking N2024, which is an entry-level Force10 based switch. If you don't mind eBay for all purchases, you can get a basic 10G setup for your NAS and one workstation for less than $1000.

Minor Nit, the N2024 is sourced from the same place the older Powerconnect 5xxx switches were. It does not come from the Force10 side of the house.
It doesn't (yet, if ever) run a derivative of the Force10 OS and it's a Helix4 based chipset, which is different from all the other Broadcom based switches that are coming from the Force10 side of the house.
The N2000 platform also has a few weird issues with 10BaseT and 100BaseT. If you still have slower devices on your network, you'll want to check that they work OK on the switch while you can still return it.. :)

 

[RickH]

Really fascinating. Something I might have to look into.

What kind of power draw to they pull?

Sorry it took me so long to get an answer to this - I don't have a meter so I had to wait until I installed another one in my CoLo and look at the power draw in the portal.

Adding the Quanta LB6M switch bumped my 120V circut by about .95 amps on average - so at 120V with dual power supplies it draws right around 100w. This was with 12 of the 24 ports connected via copper cables and no other optics installed...

 

[Drk]

Going to ask here too, anyone else has problems with the Intel 520 (rebranded from Dell) and FreeNAS 10. In short, when I enable the interface it crashes the system. I do no have DHCP setup on that vlan cause it is all static based. They worked great in 9.10 but not with 10. I ordered spare card that I am going to test out when it comes in.

Am going to conform that the Dell 0CYK8V is the same as the Intel 520, was told it was, and it shows to be but making sure.

 

[Drk]

No. Stay away from all the -X crap. It's all legacy crap. CX4 is basically four Infiniband channels at 3Gbps or something like that. LX4 is basically four wavelengths over fiber. LX4 is absolutely not compatible with SR, LR, or anything else for that matter. This is not the stuff you want to be using.

Use SFP+ with 10GBase-SR/LR or 10Gbase-T unless you like delving into hardware compatibility issues and then like to be disappointed by how your first generation 10G hardware burns watts like a flamethrower while simultaneously is incapable of actually maxxing out 10G.

The reason you may see deals on this gear on eBay is that no one wants it, and they're hoping some sucker (you) who doesn't know any better picks it up and gives them beer money.

This include like the Cisco Onex 10g (CVR-X2-SFP10G)? I know it use's the spf+ Sr's but could never find any answers on this.

 

[Stryf]

Going to ask here too, anyone else has problems with the Intel 520 (rebranded from Dell) and FreeNAS 10. In short, when I enable the interface it crashes the system. I do no have DHCP setup on that vlan cause it is all static based. They worked great in 9.10 but not with 10. I ordered spare card that I am going to test out when it comes in.

Am going to conform that the Dell 0CYK8V is the same as the Intel 520, was told it was, and it shows to be but making sure.

I didn't notice any issues with FreeNAS Corral when I was testing it on ESXi with my Intel x520 (non-Dell rebrand) nic. Ifconfig was using vmx0 instead of the ix0 though.

On another topic, the Ubiquiti ES-16-XG switch I bought worked out fine. It has an overwhelming amount of customization options and had no issues with Unix and Intel short range transceivers.

 

[Drk]

I didn't notice any issues with FreeNAS Corral when I was testing it on ESXi with my Intel x520 (non-Dell rebrand) nic. Ifconfig was using vmx0 instead of the ix0 though.

On another topic, the Ubiquiti ES-16-XG switch I bought worked out fine. It has an overwhelming amount of customization options and had no issues with Unix and Intel short range transceivers.

Thank you alot, I will do more testing on my end.

 

[jgreco]

This include like the Cisco Onex 10g (CVR-X2-SFP10G)? I know it use's the spf+ Sr's but could never find any answers on this.

As a module for your switch? Yeah, it'll probably *work*, but a switch old enough to need it is going to be a space heater. If you're going to go that route, you might just get yourself a X2-10GB-SR module and an appropriate SC-to-LC cable.

If you look at the Cisco X2 modules, notice that they all have extensive heatsinking. Also, there were some compatibility issues back in "the day", so it is difficult to say what sort of compatibility issues you might encounter. I haven't had particularly bad feedback from most people following the advice to buy modern-ish SFP+ gear.

Honestly I've gone back and forth several times in my head now as to whether it'd be better to cram an SFP+ SR optic into that module you suggested (additional complexity/layers, LC on each end of the fiber, probably more-compatible if it works). I hate people asking me questions like this because I want to tell you things I know should work.

 

[Drk]

As a module for your switch? Yeah, it'll probably *work*, but a switch old enough to need it is going to be a space heater. If you're going to go that route, you might just get yourself a X2-10GB-SR module and an appropriate SC-to-LC cable.

Yeah, I have the that X2-10GB-SR and a switch it works with both seem to be great and the 3560e does not seem to do bad for heat and power. Time will tell :); I got it for nothing out of a datacenter pull I did so not a loss. This is what I was running freenas 9.* with on 10g and it was solid until freenas 10. At some point, I will move to a newer switch but this is just for my home so not too worried.

I was just asking for your option to; I do not claim to know it all or ever will just more input is great, and you seem to be a person with may more years in the networking game than me.

Yeah, I have been thinking about going LC to LC, right now it goes from my FreeNAS box to the 10g with a Cisco SFP-H10GB-CU1M. I may change that and test it with the LC to LC and fiber I have.

Maybe when I get time, I will have a chance to set down and find out why the 10g intel crashes my box and see if the new Chelsio card I got does or does not do the same. If it crash too, i will test the LC to LC and see...yeah lots of fun.

I have some fiber module's I can test with at work too, so if that comes down to that I will just try stuff at work and see what happens, the one thing I do not have is an intel 10g nic since intel only likes intel really. Other than FS.com

Thank you a lot for your time and input, you have been really nice and helpful thank you.

 

[Drk]

I didn't notice any issues with FreeNAS Corral when I was testing it on ESXi with my Intel x520 (non-Dell rebrand) nic. Ifconfig was using vmx0 instead of the ix0 though.

On another topic, the Ubiquiti ES-16-XG switch I bought worked out fine. It has an overwhelming amount of customization options and had no issues with Unix and Intel short range transceivers.

Thank you, for the info.

Yeah, I have some of them 10g 16 port switch and they work well, there are some tweaks with them sometimes, but overall, they work well.

 

[c32767a]

Maybe when I get time, I will have a chance to set down and find out why the 10g intel crashes my box and see if the new Chelsio card I got does or does not do the same. If it crash too, i will test the LC to LC and see...yeah lots of fun.

For what it's worth, you may want to see about whether your X520 might need a firmware update. Dell tends to distribute pointlessly customized "certified" drivers for their expansion cards. It may be you're hitting a bug that's been fixed in newer firmware.

 

[Drk]

For what it's worth, you may want to see about whether your X520 might need a firmware update. Dell tends to distribute pointlessly customized "certified" drivers for their expansion cards. It may be you're hitting a bug that's been fixed in newer firmware.

You know what...you are 100% right on that. I am not sure why the hell I did not even think about that. Damn brain. I will do that maybe this weekend and see; I should be able to flash it with straight intel firmware too as you normally can with dells.

It may have something to do with the card does not have the right efi driver and needs it flashed to support it too.

 

[Ericloewe]

Good news everyone, this document is now integrated with the Resources section. This thread is now a discussion thread - one would argue that it's always been a discussion thread, but now XenForo knows what to do with it.

Use the tabs above to navigate to the document itself. Or follow the link below if you're lazy:
https://forums.freenas.org/index.php?resources/10-gig-networking-primer.42/

 

[Joshua Parker Ruehlig]

was wondering, would these be a decent choice?
http://m.ebay.com/itm/Chelsio-10GB-...t-PCI-Express-x8-110-1047-20-d0-/162243769842

and what else would i need, the listing is very bare? im really confused by all the cabling options, i think direct-attach might be the easiest soltuion but not sure if these even use them. i cant even tell if these would accept a SFP module?

Thanks