10 Gig Networking Primer

[JustinClift]

Just to point out... I have some DDR Infiniband gear (copper, with CX4 connectors) that does 20Gb/s per port pretty flawlessly.

Short range only (15m max?) and my cables are only ~2 metres. So... rackable is about it. ;)

 

[jgreco]

Just to point out... I have some DDR Infiniband gear (copper, with CX4 connectors) that does 20Gb/s per port pretty flawlessly.

Short range only (15m max?) and my cables are only ~2 metres. So... rackable is about it. ;)

FreeNAS doesn't support Infiniband. You'll find that the CX4 style Ethernet stuff all sucks, because it's first generation hardware, because sanity prevailed and the world went to better technology.

 

[Black Ninja]

FreeNAS doesn't support Infiniband. You'll find that the CX4 style Ethernet stuff all sucks, because it's first generation hardware, because sanity prevailed and the world went to better technology.

You are a dreamcrusher !

P.S. People are digging anything 10Gb they can find from any corner of the universe and then jgreco destroys their dreams, but in a mean time saves them a lot of money they would of waste on stuff they will not need. I include myselft in this statement.

 

[Borja Marcos]

Speaking of 10 GbE cabling, ours is running flawlessly with SFP+ DA cables (passive twinax). It's a simple solution suitable for top of the rack switches, it saves power and reduces heat.

However, ixgbe cards didn't identify them properly due to a bug in the media detection code. The problem is fixed now on -CURRENT and will soon be MFCd to -STABLE.

The cards worked but failed to identify the media and flag it as full duplex.

 

[CookiesLikeWhoa]

I tried to search for this but didn't find much. Are the Chelsio T540/440-CR nics supported? My gut says yes, and it appears to be based off the same chip as the t520 but I haven't found anything solid yet.

Thanks in advance!

 

[Mlovelace]

I tried to search for this but didn't find much. Are the Chelsio T540/440-CR nics supported? My gut says yes, and it appears to be based off the same chip as the t520 but I haven't found anything solid yet.

Thanks in advance!

Yes the T540 has copper connections whereas the T520 has sfp+

 

[CookiesLikeWhoa]

Yes the T540 has copper connections whereas the T520 has sfp+

Hmm according to Chelsio's site the t540 uses sfp+ it's just a quad port version.


Sent from my iPhone using Tapatalk

 

[Mlovelace]

Hmm according to Chelsio's site the t540 uses sfp+ it's just a quad port version.


Sent from my iPhone using Tapatalk

Right, apologies, the T540-CR has two 10Gbe and two 1Gbe. The Intel X540 is the Gbase-T copper, I confused the part numbers. All of the T5 asic Chelsio adapters are supported as they all use the same driver.

Edit: If you're looking to buy one for freeNAS look for the T520-SO-CR version. It's what iX uses in the trueNAS.

 

[wtfR6a]

I think its the CR model they use in TrueNAS which has the on-board hardware that off-loads TCP/IP, iSCSI, FCoE and iWARP RDMA processing from its host system. The SO-CR only has TCP/IP offload.

 

[Mlovelace]

I think its the CR model they use in TrueNAS which has the on-board hardware that off-loads TCP/IP, iSCSI, FCoE and iWARP RDMA processing from its host system. The SO-CR only has TCP/IP offload.

It's the SO-CR I asked the engineer when I was getting quotes for a trueNAS

 

[CookiesLikeWhoa]

Well I was primarily drawn by the quad port aspect. I need to have some segmented networks


Sent from my iPhone using Tapatalk

 

[Mlovelace]

Well I was primarily drawn by the quad port aspect. I need to have some segmented networks


Sent from my iPhone using Tapatalk

That's what vlans are for, but to each his own. :)

 

[Stryf]

Would anyone recommend the Quanta LB6M as a decent 10gb switch for home use? Seems to be around $300 on the bay and there aren't any reviews on the youtube.

 

[CookiesLikeWhoa]

From what I read on ServeTheHome it's a CLI only switch. Has decent Layer 3 functions, but getting a manual to help use the CLI can be troublesome. For $300 though, that's a pretty screaming deal for a 10GB switch.

 

[Mirfster]

Would anyone recommend the Quanta LB6M as a decent 10gb switch for home use? Seems to be around $300 on the bay and there aren't any reviews on the youtube.

I asked the same thing on Page 12 (at the bottom) in this very same thread and jgreco provided a response. Have not pulled the trigger and bought one yet though. Still holding out for a 8024F. Also I heard that the LB6M is pretty loud and my need some modding to quiet it down so that may be something to consider.

 

[cookiesowns]

I wouldn't go anywhere near the Dell stuff, unless it's the rebranded force10. LB6M.. is... heard some issues with VLANs and intervlan routing.. seems to fall over sometimes.

If you're using these switches as pure SAN/Storage network, they'll probably be fine, but if you're doing hyper-converged I would look else where. ( Juniper, Edge-Core w/ Cumulus, or Arista )

There's some good deals on Arista's currently. 32x40Gbe, Trident II+ based, so you can have 104 ports of 10Gbe breakout if you want. Just more expensive on optics / DACs, but essentially future proof.

http://www.ebay.com/itm/Arista-DCS-...3-Switch-96-10G-SFP-R-F-Airflow-/161972688237

No direct experience but I know they're kinda common in the data center. Noisy and hot, probably. There's some backstory you should probably research, I think maybe Quanta got bought out or something odd happened to them, I just don't remember offhand and don't have time to Google-fu it. It's nice to be able to get firmware updates.

Quanta is an OEM/ODM, they didn't get "bought" However those LB6M's are pre Trident or in a different family/derived from it.... hence why they are so cheap. Most likely a "cloud-scale" company was using these for very simple switching needs, or used at IXPs..

https://www.broadcom.com/products/ethernet-communication-and-switching/switching/bcm56820

Personally I run Edge-Core with Cumulus Linux on it. Super awesome, being able to do networking in "Linux" style and use Linux automation tools to manage your network is really, frigging great.

I may pick up a 7050QX in the near future for more 40GbE ports.. prices on 40GbE is really dropping fast, plus I need more density on 10GbE.

 

[CookiesLikeWhoa]

Well since that brings up 40Gbe, what are good 40Gbe switches?

I've seen the Mellanox SX series (1024/1036, L3 capabilities), Dell S6000 (L3 Capabilities), and now that Arista 7050QX dropped around here. Any consensus?

 

[cookiesowns]

Well since that brings up 40Gbe, what are good 40Gbe switches?

I've seen the Mellanox SX series (1024/1036, L3 capabilities), Dell S6000 (L3 Capabilities), and now that Arista 7050QX dropped around here. Any consensus?

Really, anything that's Trident 2 based will be similar. Has some fairly shallow buffers, but should be okay unless your workloads have microbursts. Mellanox is great, but again, my personal opinion is that unless you are doing IB, or running Mellanox end-end, I would just grab something from a typical networking vendor.

I like Arista though, they aren't picky with optics, great Network OS, LANZ is super awesome for checking for microbursts or buffer issues. I don't think any other platform has this on Trident 2 ASICs, so I'll definitely miss that on the AS5712 with Cumulus Linux..

If only those Arista's hit ebay 6 months earlier, could have gotten two for some serious density along with ECMP.

Nice name username btw ;)

 

[CookiesLikeWhoa]

Hmm that's good to know. Are the CLI only or do they have a fancy GUI.

You as well!

 

[cookiesowns]

Hmm that's good to know. Are the CLI only or do they have a fancy GUI.

You as well!

Most of the enterprise / datacenter switches are CLI only, but I love cumulus because you interact with the switch as if it was a server.

Being able to edit /etc/network/interfaces config as if they were regular ports on a server is amazing.

eg:

Code:

swp52     Link encap:Ethernet  HWaddr SNIP
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:713663105 errors:0 dropped:69592 overruns:0 frame:0
          TX packets:721297889 errors:0 dropped:1 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:696893192761 (649.0 GiB)  TX bytes:765934317965 (713.3 GiB)

cumulus@cumulus:mgmt-vrf:~$ ethtool swp52
Settings for swp52:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseT/Full
                                10000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Advertised link modes:  Not reported
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Speed: 10000Mb/s
        Duplex: Full
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
Cannot get wake-on-lan settings: Operation not permitted
        Link detected: yes

/etc/network/interfaces


# Uplink Bond ( VLAN TRUNKED TO core-10g-01)
auto bond0
iface bond0
        bond-slaves glob swp49-52
        bridge-vids 10 20 30 40 50 60 100 200 300
        bridge-allow-untagged no