10 Gig Networking Primer

[indy]

I am about to take the plunge into 10GbE with 2x Intel x520-DA1 and 1x Twinax cable.

Two questions:

1) Does the x520 get hot?
My FreeNAS box has only moderate airflow and my PC has almost zero airflow.
The Intel specs specify a maximum of 4.6W power consumption and require 100 fpm air flow... its a bit hard to put into perspective though.
Maybe the Intel x710 series would be a better choice in that regard, but they are about 3.5 times as expensive since there are few used ones about.

2) Would FreeNAS get 'annoyed' at the link going up and down whenever i start/shutdown my PC?
I imagine at least daily emails...
Any other reasons I should only consider 10GbE with a suitable switch?

 

[MatthewSteinhoff]

1) Does the x520 get hot?

My X520-DA2 is 105-110 °F at the fins. Ambient room temperature is 65. Rated operating temperature is 32 °F to 131 °F (0 °C to 55 °C). The overall case interior is much lower, high 70s, low 80s. For comparison, the LSI SAS9207-4i4e sees 102 °F at the fins.



The case is a Supermicro SC826E16-R1200LPB and we've overstuffed in a way that was never intended by the manufacturer. Here is a photo with the plastic airflow director removed...



Totally not recommended but incredibly stable. (This is also the reason I always recommend far more disk bays than you expect to need. Otherwise, you end up doing stupid stuff later.)

Cheers,
Matt

 

[CookiesLikeWhoa]

Most of the enterprise / datacenter switches are CLI only, but I love cumulus because you interact with the switch as if it was a server.

Being able to edit /etc/network/interfaces config as if they were regular ports on a server is amazing.

eg:

Code:

swp52     Link encap:Ethernet  HWaddr SNIP
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:713663105 errors:0 dropped:69592 overruns:0 frame:0
          TX packets:721297889 errors:0 dropped:1 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:696893192761 (649.0 GiB)  TX bytes:765934317965 (713.3 GiB)

cumulus@cumulus:mgmt-vrf:~$ ethtool swp52
Settings for swp52:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseT/Full
                                10000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Advertised link modes:  Not reported
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Speed: 10000Mb/s
        Duplex: Full
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
Cannot get wake-on-lan settings: Operation not permitted
        Link detected: yes

/etc/network/interfaces


# Uplink Bond ( VLAN TRUNKED TO core-10g-01)
auto bond0
iface bond0
        bond-slaves glob swp49-52
        bridge-vids 10 20 30 40 50 60 100 200 300
        bridge-allow-untagged no

Thanks for the info! I should really man up and learn how to configure a switch from the CLI. I ended up getting a Mellanox SX1024 for, at least what I think is, a really good deal.

Initial set up was done via console and CLI, but that was stupid simple. Once that was done I headed over to the web GUI. While it looks great, is pretty clunky. So I might spend some time learning the CLI.

I looked up the Arista manual...2750 something pages. Made my head want to explode, at least it's thorough.

 

[indy]

It runs hotter then the LSI? That is not what I was hoping for.
Thank you very much for your detailed reply though!
I might have to get one of those infrared thermometers myself :)

 

[Stilez]

I'm taking the plunge with Chelsio T420s connecting a FreeNAS box, workstation and ESXi 6.0 homelab. The cards are bought, all I need now are transceiver modules and cable. I'll be picking those off EBay or Fibrestore. Because of compatibility issues could someone check I've got it all right?

From what I understand it's safest to buy Chelsio OEM or Finisar transceivers. I've seen Finisar FTLX8571D3BCL-CL described online as "compatible", but there's no "compatibility page" on Finisar's website for Chelsio adapters, and I can't even find the web page or product guide on their site for the 8571. Finisar's website points to the 8574 for SFP+ SR instead but no info if it's compatible. So I'm a bit lost which transceivers are compatible, how "fussy" the T420 is about matching components, and is there a vendor lock-in or anything else I should be aware of? Are any other transceivers recommended for the Chelsio T4's?

The cable only needs to run under 30m, and perhaps if I move the boxes around even under 15m. If I understand the datasheet, the T420 + SM10G-SR have LC-LC terminations. Is looking for OM2 (or higher) LC-LC cable, all I need to check, for it to be compatible, or is there anything else to check? What should I look for in photos to recognise a well-made optical cable/termination from a crappily-made one?

Last, a technical question - as the T420-CR is dual-port and offloads a load of host processing, if a packet incoming on one port would be routed or bridged by the host and sent out on the other port, does the card handle it internally (ie near-wire-speed) or does it need to be passed across PCI-E to the host for OS network stack handling and then sent back to the card for the next hop?

Thanks for the pointers!

 

[indy]

So I got my hands on 2x XL710-QDA1 and an Fujitsu/Intel QSFP+ Fiber DAC... for very little money :)

Initial observations
Installation was just as simple as any other network card.
I expected something disappointing or magical but the basic connectivity just plainly worked.
The chip and module stay incredibly cool, even under heavy load very little airflow should suffice to keep the components at low temperatures.
Great stuff if one is aiming for a quiet home-server.

Performance
Out-of-the-box ~1,5 Gbit/s, single thread, half-duplex
That kind of put a damper on my mood.
Tests with an Ubuntu live cd were a lot more promising though.
After some fiddling with the driver and network options on the Windows-side (mainly disabling interrupt moderation) the connection is now cpu-bound.
Windows system:Intel 4690K @4.1Ghz, v7 x64, iperf v2.0.9
FreeNAS system: Intel 1220v3, 9.10.1-U1, iperf v2.0.5
~15 Gbit/s, single-thread, half-duplex (100% cpu-usage on one core, both sides)
~25 Gbit/s, dual-thread, half-duplex (100% cpu-usage on all cores, both sides)
~32 Gbit/s quad-thread, half-duplex (100% cpu-usage on all cores, both sides)
~32 Gbit/s octa-thread, half-duplex (100% cpu-usage on all cores, both sides)

 

[CookiesLikeWhoa]

I'm taking the plunge with Chelsio T420s connecting a FreeNAS box, workstation and ESXi 6.0 homelab. The cards are bought, all I need now are transceiver modules and cable. I'll be picking those off EBay or Fibrestore. Because of compatibility issues could someone check I've got it all right?

From what I understand it's safest to buy Chelsio OEM or Finisar transceivers. I've seen Finisar FTLX8571D3BCL-CL described online as "compatible", but there's no "compatibility page" on Finisar's website for Chelsio adapters, and I can't even find the web page or product guide on their site for the 8571. Finisar's website points to the 8574 for SFP+ SR instead but no info if it's compatible. So I'm a bit lost which transceivers are compatible, how "fussy" the T420 is about matching components, and is there a vendor lock-in or anything else I should be aware of? Are any other transceivers recommended for the Chelsio T4's?

The cable only needs to run under 30m, and perhaps if I move the boxes around even under 15m. If I understand the datasheet, the T420 + SM10G-SR have LC-LC terminations. Is looking for OM2 (or higher) LC-LC cable, all I need to check, for it to be compatible, or is there anything else to check? What should I look for in photos to recognise a well-made optical cable/termination from a crappily-made one?

Last, a technical question - as the T420-CR is dual-port and offloads a load of host processing, if a packet incoming on one port would be routed or bridged by the host and sent out on the other port, does the card handle it internally (ie near-wire-speed) or does it need to be passed across PCI-E to the host for OS network stack handling and then sent back to the card for the next hop?

Thanks for the pointers!

You can shoot fibrestore an email and they'll get give you a transceiver that will work. They're really good about it.

 

[Stilez]

You can shoot fibrestore an email and they'll get give you a transceiver that will work. They're really good about it.

Thanks - doing that now. But I'd still like a broader reply to these, because I'd like to learn for future, not just be told a model that works. (I think people here will understand that).

 

[CookiesLikeWhoa]

Thanks - doing that now. But I'd still like a broader reply to these, because I'd like to learn for future, not just be told a model that works. (I think people here will understand that).

Unfortunately you won't find a "compatibility" page for aftermarket parts like this. It's really going to be a check what you have and see if it works. For my Chelsio t-420's I use Finisar FTLX8571D3BCL Compatible 10GBASE-SR/SW SFP+ 850nm 300m DOM Transceiver from the Fiber store and it works perfectly. Some brands are more open than others. My Mellanox SX1024 switch doesn't seem to give a rip what module I put in it, as long as it fits it will work. My Netgear XSM7024 is a little more picky than that, but still pretty accepting to other modules. My Chelsio t420 is not very open at all.

Buying from a reputable source is your best way to get quality products. You'll want OM3 cable with LC-LC terminations.

 

[Stilez]

Unfortunately you won't find a "compatibility" page for aftermarket parts like this. It's really going to be a check what you have and see if it works. For my Chelsio t-420's I use Finisar FTLX8571D3BCL Compatible 10GBASE-SR/SW SFP+ 850nm 300m DOM Transceiver from the Fiber store and it works perfectly. Some brands are more open than others. My Mellanox SX1024 switch doesn't seem to give a rip what module I put in it, as long as it fits it will work. My Netgear XSM7024 is a little more picky than that, but still pretty accepting to other modules. My Chelsio t420 is not very open at all.

Buying from a reputable source is your best way to get quality products. You'll want OM3 cable with LC-LC terminations.

This was what I needed to know - that they were used by others and found to be compatible. I bought the Finisars, but went with OM2 LC-LC because I don't need the extra length and preferrred the orange jacket for visibility (if it's being worked on or used "in the open" as a patch cable)

 

[Mlovelace]

Thanks - doing that now. But I'd still like a broader reply to these, because I'd like to learn for future, not just be told a model that works. (I think people here will understand that).

10 Gig Networking Primer

As I stated earlier in this thread I have 68 Fiber Store transceivers in production and I've had no issues to date.

 

[RickH]

Would anyone recommend the Quanta LB6M as a decent 10gb switch for home use? Seems to be around $300 on the bay and there aren't any reviews on the youtube.

I know I'm a little late to the game on this reply, but I actually have 5 of the Quanta LB6M switches and have helped a client set up another. There simply isn't anything else out there that offers a comparable value - the last 2 I bought I got for $225/ea.

To address your direct question, I would NOT recommend this switch for home use unless you have a soundproofed server/network closet. These things run hot and are extremely loud! (almost the same volume level as a typical 1U server on boot-up) If you get the dual power supply model there are a total of 7 40mm fans (2 in each PS and another 3 in a hot-swappable back panel) and judging by the amount of heat coming out, I wouldn't recommend pulling or disabling any of those. These are definitely much better suited to a rackmounted business/datacenter application.

If your use application can get around the noise problem, the switches themselves are very capable and have been absolutely rock solid in my experience. There are a couple of considerations:

• There is absolutely no GUI or web interface - initial configuration has to be done with a console cable (RJ45 Cisco style serial cables work with these units), and from there it's all going to be via console over SSH (they do support telnet as well, but you're going to want to disable that for security reasons) - IMO this isn't really a downside as I find the GUI in most switches to be 'clunky' at best and I tend to do most of my config through the CLI anyways.
• There isn't a true 'manual' available for these - the firmware identifies as 'Fastpath QOS' and the availalbe Fastpath configuration guides are close enough to get you going, but not everything matches exactly so be prepared to spend a few hours 'fumbling around'
• As best I can tell, the latest firmware for these is 1.2.0.14 - it would seem that most of the current eBay units are shipping with this version, but you can find the FW image online if not. Upgrading is straightforward, but you'll need the console cable and a TFTP server to host the FW image.
• As all of my switches are set up a 'top of rack' config, I use copper direct attach cables, so I can't comment on optic compatibility, but I haven't seen any compatibility issues using nic's from Intell, Mellanox, and Chelsio in my servers.
• These do not provide any type of 'stacking' functionality so if your setup has multiple switches, you'll have to configure each of them individually
• Link aggregation (including LCAP) is supported on both the 10GbE and 1GbE ports.
• These aren't full Layer3 switches - There are some references to QoS and vlan routing in the firmware, but it's nothing I've been able to get running.
• Regular vlan functions are present and work with no issues (once you figure out how to configure them)
• Airflow on these switches is front-to-back, I always install my switches on the back of my racks so I opened them up and reversed all the fans to change the airflow - I didn't notice any difference on the reported running temps by doing this, but I'm sure I 'voided my warranty'.

I don't actually own any other 10GbE switches so I can't to a side-by-side comparison on performance, but I can tell you that in my testing I've seen transfer rates as high as 940 MiB/sec over a single 10GbE link through these switches to my FreeNAS server and 1,400 MiB/sec using ISCSI multipathing. (both tests were from a Windows 2012 R2 server with an Intel NIC using the Windows ISCSI initiator to a FreeNAS server with a Chelsio NIC) - these numbers didn't significantly change when the switch was removed from the test and the servers were directly attached. All of my Quanta switches have been rock solid and I've never experienced any type of lock-up that required a reboot - they've all been continuously powered on since the initial configuration was completed.

In summary; if you're OK with the noise and lack of Layer3 functionality, I would highly recommend these switches!

(NOTE: I included a couple pics of my LB6M installs)

 

[drwoodcomb]

Does anyone know if this card would be a good choice for connecting my FreeNAS box directly to a windows 10 PC?

Mellanox ConnectX-2 Single-Port SFP+ 10GBE Network Card MNPA19-XTR

I found that these cards can be found on ebay for anywhere between $15-$40 so my idea was to buy on for the FreeNAS box and another for the windows 10 PC and then connect them together with a fiber cable. I still need to figure out what kind of cable I would need since SPF+ is confusing as hell. If anyone knows where this card is supported by FreeNAS 9.10 and Windows 10 as well as what kind of optical cable I need for this set up I would greatly appreciate it.

If this works I could have a 10 gigabit connection for a little over $100!

An additional question I have and hope someone could answer for me is:

For both the FreeNAS box and the Windows 10 PC I currently have 1 gigabit wired connection via my wireless router that allows all my devices in the house connect to the FreeNAS box. When I install the 2 SFP+ cards to have a direct connection to each other is there a way for me to configure the FreeNAS server and the Windows 10 PC to only use the SFP+ connection when transferring data between each other and then to use the 1 gigabit wired connection when it comes to everything else. For example they both use the 1 gigabit connection to connect to the internet and in the case of the FreeNAS box it would use the 1 gigabit connection to transfer data to all the other devices in the house.

 

[apple4ever]

I know I'm a little late to the game on this reply, but I actually have 5 of the Quanta LB6M switches and have helped a client set up another. There simply isn't anything else out there that offers a comparable value - the last 2 I bought I got for $225/ea.

<snip>

Really fascinating. Something I might have to look into.

What kind of power draw to they pull?

 

[JustinClift]

Does anyone know if this card would be a good choice for connecting my FreeNAS box directly to a windows 10 PC?

Mellanox ConnectX-2 Single-Port SFP+ 10GBE Network Card MNPA19-XTR

In theory (!), it should work fine. That being said, the Mellanox drivers are fairly new to FreeNAS so haven't received as much testing and use as the Chelsio and other 10GbE drivers. (Hardware wise, the adapters are pretty good. EMC, Isilon, etc use Mellanox adapters in their products)

Cabling wise, how far apart are your FreeNAS box and Win 10 PC?

Asking because SFP+ is just the connector type on the end of a cable. The cable itself can be either fibre or copper, so knowing how far apart they are will give a hint as to the right type you need. :)

 

[JustinClift]

@drwoodcomb This might be helpful as a guide to the correct cable type: https://www.mellanox.com/products/interconnect/ethernet-direct-attach-copper-cables.php

If the two computers are more than 7 meters apart, it looks like you'd need to use fibre after all:

https://www.mellanox.com/products/interconnect/ethernet-optical-transceivers.php

 

[JustinClift]

And thinking about it even more, you might be better off going with a card which has a QSFP+ port, as that gives more options for cables. These seem fairly cheap:

http://www.ebay.com/itm/112277404229

 

[drwoodcomb]

Thats an excellent question, I cant believe I left that part out!

Its about 20-30 meters between the server and the computer.

In theory (!), it should work fine. That being said, the Mellanox drivers are fairly new to FreeNAS so haven't received as much testing and use as the Chelsio and other 10GbE drivers. (Hardware wise, the adapters are pretty good. EMC, Isilon, etc use Mellanox adapters in their products)

Cabling wise, how far apart are your FreeNAS box and Win 10 PC?

Asking because SFP+ is just the connector type on the end of a cable. The cable itself can be either fibre or copper, so knowing how far apart they are will give a hint as to the right type you need. :)

t

 

[JustinClift]

Ok, that's longer than a direct attach copper cable will reach. A fibre cable would be ok though.

 

[drwoodcomb]

And thinking about it even more, you might be better off going with a card which has a QSFP port, as that gives more options for cables. These seem fairly cheap:

http://www.ebay.com/itm/112277404229

Thats amazing! 40 gigabit connection! Thats crazy cheap. I will definitly look into this