10 Gig Networking Primer

[Ericloewe]

Assuming the card is good, you'd need either a twinax cable with DACs (simple, cheaper and somewhat less compatible) or an SFP+ in the variety you need plus matching fiber (probably 10GBase-SR with OM3 or OM4 fiber).

 

[Joshua Parker Ruehlig]

Ok, I think I'm starting to get it. DACs you buy a sinlge cable, SFP+ you buy the module for each side, plus the cable.
From the picture I don't see anywhere an SFP+ module would plug into. Do these always slide into a slot or sometimes stick out?

In the first post, it mentions latency. Is a DAC on par with SFP+?
Also, is a DAC low power consumption like SFP+?

Thanks!

EDIT
Ohh, and for my application I'm not too worried about compatibility. I was going to buy 2 cards, and whatever cables needed to direct connect them.
I was hoping to get a 10G link between my FreeNAS and a FreeBSD server. Currently I have these connected with 2*1G connections using round-robin, but performance isn't great in one of directions and I think LAGG is to blame.

 

[Mlovelace]

was wondering, would these be a decent choice?
http://m.ebay.com/itm/Chelsio-10GB-...t-PCI-Express-x8-110-1047-20-d0-/162243769842

and what else would i need, the listing is very bare? im really confused by all the cabling options, i think direct-attach might be the easiest soltuion but not sure if these even use them. i cant even tell if these would accept a SFP module?

Thanks

That looks like a single SR port on the card with standard LC connector. So you'd have to get OM3 or OM4 fiber patch cable. I would guess that card is based on the legacy T3 Chelsio ASIC. If it uses the T3 ASIC then freeNAS should have drivers for it.

 

[Joshua Parker Ruehlig]

That looks like a single SR port on the card with standard LC connector. So you'd have to get OM3 or OM4 fiber patch cable. I would guess that card is based on the legacy T3 Chelsio ASIC. If it uses the T3 ASIC then freeNAS should have drivers for it.

sounds good. I'll probably just order 2 at some point, then once I get the card and verify, get the cable.

 

[Ericloewe]

That looks like a single SR port on the card with standard LC connector. So you'd have to get OM3 or OM4 fiber patch cable. I would guess that card is based on the legacy T3 Chelsio ASIC. If it uses the T3 ASIC then freeNAS should have drivers for it.

Oh yeah, how weird. The transceiver is built-in. It's an awfully non-standard color, but SR would be my first bet - definitely double-check the specs, since the ebay seller was so useless in that regard.

 

[Joshua Parker Ruehlig]

I'm thinking it might be better to go with 2 of these. since the available slot in my FreeBSD server is PCIe 2.0 x4, at least this will give it a bit more bandwidth.
https://www.madepc.com/Chelsio-10GB-2-Ports-PCI-e-110-1088-30-Opt-Adapter-p/701005995-12.htm

This card seems too cheap, but then I gotta buy 2 x SFP+ adapters and a cable right?

Would is be as simple as picking up this as well?
https://www.amazon.com/10Gtek-SFP-H10GB-CU1M-10GBASE-CU-Passive-1-Meter/dp/B00WHS3NCA/

 

[Ericloewe]

Would is be as simple as picking up this as well?

Depends on how evil Chelsio is with SFP+ lockouts. You might need a Chelsio-specific model that's flashed with the correct keys.

This card seems too cheap

It might also be an older model. They're more power-hungry and they don't quite do 10Gb/s, IIRC (more like 6-7, like Intel with old drivers).

 

[Mlovelace]

I'm thinking it might be better to go with 2 of these. since the available slot in my FreeBSD server is PCIe 2.0 x4, at least this will give it a bit more bandwidth.
https://www.madepc.com/Chelsio-10GB-2-Ports-PCI-e-110-1088-30-Opt-Adapter-p/701005995-12.htm

This card seems too cheap, but then I gotta buy 2 x SFP+ adapters and a cable right?

Would is be as simple as picking up this as well?
https://www.amazon.com/10Gtek-SFP-H10GB-CU1M-10GBASE-CU-Passive-1-Meter/dp/B00WHS3NCA/

The card linked in the madepc.com is the S320E-LP-CR and you could use either DAC or optics with fiber patch cable. The bracket in the picture is non-standard so you might have to find a replacement bracket to get it into your case. I have run the Extreme Networks DAC and the Intel DAC in our T520-CR without problem. You might want to look at the prices on the fiber store website if you're going the optics route.

 

[Joshua Parker Ruehlig]

Thanks

Yeah, I was getting "N320E-SR" from a serverthehome thread for that model number. Which conflicts with the listing that say PCIe 3.0 x8. Only reason I preferred that listing was the extra bandwidth PCIe 3.0 (running at 2.0) would give me, but I assume that listing is wrong about PCIe 3.0.

I guess I really need to be looking for something T4/T5 generation in order to get PCIe 2.0.

 

[Mlovelace]

Thanks

Yeah, I was getting "N320E-SR" from a serverthehome thread for that model number. Which conflicts with the listing that say PCIe 3.0 x8. Only reason I preferred that listing was the extra bandwidth PCIe 3.0 (running at 2.0) would give me, but I assume that listing is wrong about PCIe 3.0.

I guess I really need to be looking for something T4/T5 generation in order to get PCIe 2.0.

Look for the SO-CR versions of the T420/T520 adapters, it should be cheaper than the CR versions and freeNAS doesn't leverage the additional features of the CR cards anyway.

Edit: The T3 ASIC cards from Chelsio are all going to be PCIe gen 1. They also use more power than the T5 and teh T4 ASIC cards.

 

[Joshua Parker Ruehlig]

sounds good. ok I think I know what to look for now. I'm in no rush, so gonna take me time and look for a decent deal.

 

[Lars Day]

This document is now available in the Resources section! Use the tabs above to navigate to the document itself.

The last version follows inside the spoiler tag:

Show : Old version

I've been talking 10gigE networking with a number of people in the last year or two. Time for a resource for people to refer to.

History

In the early 1990's, we had 10Mbps ethernet. In 1996, 100Mbps ethernet. In 1999, gigabit ethernet. In 2002, 10 gigE. About every three or four years, an order of magnitude increase in network speeds was introduced. In all cases but the last, within about 5 years of introduction, reasonably priced commodity gear became available for that technology. We stalled out with 10G because the technology became more difficult. Copper based 10G wasn't practical at first. Further, and perhaps unexpectedly, it seemed that gigabit was actually finally sufficient for many or even most needs.

LACP for gigabit

A lot of people have worked around the lack of 10gigE with Link Aggregation, that is, using multiple gigabit connections from a server to a managed switch. Unfortunately, the load balancing protocols used do not provide a standard way to balance traffic to a small number of destinations, such as what often happens with fileservers. LACP kind of sucks for NAS.

10 gigabit Ethernet technologies

There's a bunch, but let's stick to the practical stuff you might actually want to use, and leave out stuff like XFP or CX4. The modern stuff you want to use boils down to just two: SFP+ and 10GBASE-T.

SFP+ is a technology that allows an engineer to put one of several different modules into a switch or network interface card. An evolution of older SFP technology, it is usually backwards-compatible with SFP. An SFP+ module is essentially a transceiver that talks to the switch and requires a "device driver" of sorts, so you need an SFP+ module that is compatible with the switch or ethernet adapter (some vendors also engage in vendor lock-in requiring you to use their own branded modules). This kind of sucks, but once you get over that hurdle, you have lots of options and it is incredibly flexible. SFP+ is available in various flavors. The ones you're likely to use:

SR optics (the word we usually use to refer to optical SFP modules) are short range, and when used with OM3 (aqua colored) fiber, can run for up to 300 meters. This is usable for any distance shorter than 300 meters.

LR optics are longer range, and when used with the proper singlemode fiber, can run up to 10 kilometers.

These are laser based products and you should not look into the optics or ends of the fiber. ;-)

Also available are direct-attach SFP+'s, where two SFP+ modules have been permanently connected together via twinax cable. These are essentially patch cables for SFP+. The downside is that sometimes you run into compatibility issues, especially when you have two SFP+ endpoints from different manufacturers who both engage in vendor lock-in. The upside is that they're cheaper and (maybe?) more durable than optics and fiber, where you need to not be totally stupid and careless with kinking the fiber.

Sadly, no SFP+ modules are available for 10GBASE-T. There are SFP modules for gigabit however.

SFP+ has somewhat lower latency and reduced power consumption compared to 10GBASE-T.

One of the biggest caveats here, though, is that once you go down the SFP+ path, you probably want to stick with it. There's no easy switching away from it except to do a forklift upgrade. (But don't feel bad, SFP+ marks you as a diehard networker.)

The Intel X520 card is an example of an SFP+ card, which is available in one and two port configurations, and -DA (direct attach) and -SR (short range) optic variants. The difference between -DA and -SR is simply that the -SR will include Intel SR optics.

10GBASE-T is the copper 10G Ethernet standard. Much more familiar to most end users, this uses RJ45 modular connectors on Category 6 or better cable. Category 6 will typically reach up to around 50 meters. This was basically a worthless standard up until recently, when several manufacturers have started to create less-expensive switches that support 10GBASE-T. Probably the most notable of these is the Netgear ProSafe XS708E, available for $800-$900.

I believe that 10GBASE-T will ultimately be the prevailing technology, but it is very much the VHS (ref VHS-vs-Betamax) of the networking world. It is an inferior technology that burns more power and makes more compromises. Most of the deployed 10G networking out there today is still NOT copper, so for the next several years, at least, the best deals on eBay are likely to be for SFP+ or other non-copper technologies.

What Do You Need?

While it is tempting to think of your entire network as 10 gigabit, in most cases this is at least a several thousand dollar exercise to make happen, factoring in the cost of a switch, ethernet cards, and wiring.

There are some alternatives. One easy target is if gigabit is acceptable for your endpoints (PC's and other clients), it is not that hard to find a gigabit switch with several 10G uplinks. The cheapest decent one I've seen in (2014) times is probably the Dell Networking 5500 series, such as the 5524, often available for around $400 on eBay. That model comes with two 10G SFP+ slots, which could be used for a FreeNAS box and a workstation at 10G, while also allowing all remaining stations to share in the 10G goodness. Now that it's 2016 we're also seeing the Dell Networking N2024, which is an entry-level Force10 based switch. If you don't mind eBay for all purchases, you can get a basic 10G setup for your NAS and one workstation for less than $1000.

We recently debated another alternative, which is to abuse the FreeNAS box itself as a bridge using FreeBSD's excellent bridging facility. This is very cost-effective but has some caveats ... primarily that you need to be more aware that you've got a slightly hacked-up configuration. Since modern ethernet technologies are fully capable of point-to-point operation, without a switch, clients can be hooked up directly to the server (via a 10Gbase-T crossover cable, or SFP+). The simple case of a single workstation hooked up to the server via a direct cable is fairly easy. Multiple workstations might involve bridging. If you wish your clients to receive Internet connectivity, that's more complicated as well.

In 2013, Netgear introduced a few new 10GBASE-T switch options including the ProSafe XS708E which offers 8 ports for a cost around $100 per port.

The Dell PowerConnect 8024F is often available on eBay for around $1800, offering a mix of SFP+ ports along with four 10GBASE-T. This is probably the cheapest option to get 10gigE for a NAS or two, some ESXi boxes, and then a few runs of 10GBASE-T for workstations.

What Card Do I Pick?

This forum has been very pro-Intel for gigabit cards, because historically they've "just worked." However, for 10gigE, there have been some driver issues in versions of FreeNAS prior to 9.3 that lead to intermittent operation. Additionally, the Intel adapters tend to be rather more expensive than some of the other options. 10gigE is not in high demand, so often some of the niche contenders have products that may, counterintuitively, be very inexpensive on the used market. These cards may be just as good a choice - if not better - than the Intel offerings. We're Intel X520 here but the following notes are gathered from forum users.

@depasseg and I note: Intel X520 and X540 are supported via the ixgbe driver. Intel periodically suffers from knockoff cards in the new and used markets. There should be a Yottamark sticker on it that'll help authenticate the card as genuine. Check the country, datecode, and MAC address Yottamark gives you, don't just blindly trust it. Not a good choice if you wish to run versions prior to 9.3. https://bugs.freenas.org/issues/4560#change-23492 Also note that there's been a variety of problem reports with the X540 and TSO.

@Mlovelace, @depasseg, and @c32767a note: Chelsio is iXsystems' card of choice. @Norleif notes that the S320E-SR-XFP can sometimes be found for less than $100 on eBay. The Chelsio T3, T4 and T5 ASICs are fully supported by the current version of FreeNAS and are the cards shipped for 10gigE if you buy a TrueNAS system. iXsystems: "FreeNAS 9.2.1.5 supports the Chelsio T3 with the cxgb driver. It supports the T4/T5 with the cxgbe driver. We keep these drivers current with the latest code from the vendor. By far and away the Chelsio cards are the best FreeBSD option for 10Gbe." Also note that the S310E only supports PCIe 1, so speeds may be limited especially in an x4 slot. @Mlovelace also has found a great vendor for generic Chelsio SFP+ optics.

@depasseg and @c32767a note: SolarFlare: Some users recommend the SFN 5162F.

@Norleif reports: IBM NetXTREME II 10GBit (Broadcom BCM 57710) Works in FreeNAS 9.3, can sometimes be found for less than $100 on eBay.

@Borja Marcos notes: Beware the Emulex "oce" cards - serious issues with them, panics when moving some traffic. There is a patch (see relevant discussions on the freebsd-net mailing list) but the stock driver crashes badly.

As usual, this post isn't necessarily "complete" and I reserve the right to amend it and/or delete, integrate, and mutilate the reply thread as I see fit in order to make this as useful as possible.

Also, if you'd like to repost this elsewhere, I'd appreciate it if you would have the courtesy to ask. And to credit the source.

Since we are talking about 10GB; Will Qlogic QLE8142 adapters work with FreeNAS?
Thank you.
Regards...

 

[Chuck Remes]

Every 6 months or so I look for any new SOHO 10Gbe routers. Just found a *switch* (not router) from Asus. It's the Asus XG-U2008 and it offers 2 10GbaseT ports and 8 1GbaseT ports.

https://www.asus.com/us/Business-Networking/XG-U2008/

All this for $250 (or $230 at Amazon). Not bad. I might grab one to connect my Freenas box to my main desktop.

 

[Mlovelace]

Every 6 months or so I look for any new SOHO 10Gbe routers. Just found a new one from Asus. It's the Asus XG-U2008 and it offers 2 10GbaseT ports and 8 1GbaseT ports.

https://www.asus.com/us/Business-Networking/XG-U2008/

All this for $250 (or $230 at Amazon). Not bad. I might grab one to connect my Freenas box to my main desktop.

A point of correction, that is a unmanaged switch not a router.

 

[jgreco]

A point of correction, that is a unmanaged switch not a router.

I'm sitting here wondering why it took from 2014...2017 for a reasonably priced 10G switch to appear.

 

[Mlovelace]

I'm sitting here wondering why it took from 2014...2017 for a reasonably priced 10G switch to appear.

It's not a bad price at all for a 10 port (dual 10Gbe) switch. I have to say that I've been very pleased with my asus wireless router as well. So, it might not be a bad option for people to get their feet wet with 10Gbe.

 

[Chuck Remes]

A point of correction, that is a unmanaged switch not a router.

Just clarified the original post. I *did* mean to say router because ideally I could find a nice little Asus wireless router with a 10Gbe WAN port and a few 10Gbe and 1Gbe LAN ports. For now I'll need to be satisfied with a switch.

 

[Stilez]

This resource needs updating:

• Prices have come down a lot. For example, 10G dual port Chelsio T420s (which I'm using) are $50-100 on EBay in 2017. Buy 3 and you can directly connect your FreeNAS server, backup, and ESXi box/workstation for under $400, and get free full offload (including enterprise iSCSI/SMB related offload) to trial into the bargain.
  
• More explanation needed of the variants within optical technology: SFP/SFP+, LC/SR/CR/CX4, OM2/OM3 cable, reliable brands and outlets, transceiver compatibility, and all that. For homebrew and small offices, SFP+ cards and SR10 transceivers (Finisar is reliable and seem to be universally compatible), OM2 cable (or OM3 for longer distances, Wikipedia has an article with distances for these), and LC connectors. CR and SFP+ is the one to look at if new, don't get CX4 or SFP, they work but no real benefit, go for the ones I've stated if you're new to it.
  
• If you only want to connect 2 or 3 boxes, you don't need a switch. Just directly connect the cards and allocate an IP manually. If you want more, then look for cheap switches from Dell, HP, Ubiquiti Edgswitch, etc, that either have 4 or so 10G SFP+ ports already, or take 2 x dual 10G SFP+ modules. They aren't that expensive on EBay and are reliable as they're enterprise builds.
  You could also use the BSD Routing Project and build your own, or - at a pinch - even pfSense configured as a switch/bridge if you need a GUI and are so inclined (you'll get about 4gbps out of it with good hardware and no rules but not the full line speed bandwidth, general purpose firewalls can't push data that fast without a lot of custom tuning and good hardware; it's not many nanoseconds per packet! So this is *not* recommended.)
• Config/tuneables?

 

[diedrichg]

• Prices have come down a lot. For example, 10G dual port Chelsio T420s (which I'm using) are $50-100 on EBay in 2017.

It always gives me pause when I see a high-end product being photographed ON TOP of an anti-static bag.

 

[jgreco]

It always gives me pause when I see a high-end product being photographed ON TOP of an anti-static bag.

Really? I was cruising fleabay looking for something obscure and came across stuff like

https://www.ebay.com/sch/greensouptv-5/m.html

all the time. My personal fav is circuit board on carpet. Haven't yet seen a nice Van de Graaff generator used as a backdrop.