I've been talking 10gigE networking with a number of people in the last year or two. Time for a resource for people to refer to.
History
In the early 1990's, we had 10Mbps ethernet. In 1996, 100Mbps ethernet. In 1999, gigabit ethernet. In 2002, 10 gigE. About every three or four years, an order of magnitude increase in network speeds was introduced. In all cases but the last, within about 5 years of introduction, reasonably priced commodity gear became available for that technology. We stalled out with 10G because the technology became more difficult. Copper based 10G wasn't practical at first. Further, and perhaps unexpectedly, it seemed that gigabit was actually finally sufficient for many or even most needs.
LACP for gigabit
A lot of people have worked around the lack of 10gigE with Link Aggregation, that is, using multiple gigabit connections from a server to a managed switch. Unfortunately, the load balancing protocols used do not provide a standard way to balance traffic to a small number of destinations, such as what often happens with fileservers. LACP kind of sucks for NAS.
10 gigabit Ethernet technologies
There's a bunch, but let's stick to the practical stuff you might actually want to use, and leave out stuff like XFP or CX4. The modern stuff you want to use boils down to just two:
SFP+ and
10GBASE-T.
SFP+ is a technology that allows an engineer to put one of several different modules into a switch or network interface card. An evolution of older SFP technology, it is usually backwards-compatible with SFP. An SFP+ module is essentially a transceiver that talks to the switch and requires a "device driver" of sorts, so you need an SFP+ module that is compatible with the switch or ethernet adapter (some vendors also engage in vendor lock-in requiring you to use their own branded modules). This kind of sucks, but once you get over that hurdle, you have lots of options and it is incredibly flexible. SFP+ is available in various flavors. The ones you're likely to use:
SR optics (the word we usually use to refer to optical SFP modules) are short range, and when used with
OM3 (
aqua colored) fiber, can run for up to 300 meters. This is usable for any distance shorter than 300 meters.
LR optics are longer range, and when used with the proper singlemode fiber, can run up to 10 kilometers.
These are laser based products and you should not look into the optics or ends of the fiber. ;-)
Also available are direct-attach SFP+'s, where two SFP+ modules have been permanently connected together via twinax cable. These are essentially patch cables for SFP+. The downside is that sometimes you run into compatibility issues, especially when you have two SFP+ endpoints from different manufacturers who both engage in vendor lock-in. The upside is that they're cheaper and (maybe?) more durable than optics and fiber, where you need to not be totally stupid and careless with kinking the fiber.
Sadly, no SFP+ modules are available for 10GBASE-T. There are SFP modules for gigabit however.
SFP+ has somewhat lower latency and reduced power consumption compared to 10GBASE-T.
One of the biggest caveats here, though, is that once you go down the SFP+ path, you probably want to stick with it. There's no easy switching away from it except to do a
forklift upgrade. (But don't feel bad, SFP+ marks you as a diehard networker.)
The
Intel X520 card is an example of an SFP+ card, which is available in one and two port configurations, and -DA (direct attach) and -SR (short range) optic variants.
The difference between -DA and -SR is simply that the -SR will include Intel SR optics.
10GBASE-T is the copper 10G Ethernet standard. Much more familiar to most end users, this uses RJ45 modular connectors on Category 6 or better cable. Category 6 will typically reach up to around 50 meters. This was basically a worthless standard up until recently, when several manufacturers have started to create less-expensive switches that support 10GBASE-T. Probably the most
notable of these is the Netgear ProSafe XS708E, available for $800-$900.
I believe that 10GBASE-T will ultimately be the prevailing technology, but it is very much the VHS (ref VHS-vs-Betamax) of the networking world. It is an inferior technology that burns more power and makes more compromises. Most of the deployed 10G networking out there today is still NOT copper, so for the next several years, at least, the best deals on eBay are likely to be for SFP+ or other non-copper technologies.
What Do You Need?
While it is tempting to think of your entire network as 10 gigabit, in most cases this is at least a several thousand dollar exercise to make happen, factoring in the cost of a switch, ethernet cards, and wiring.
There are some alternatives. One easy target is if gigabit is acceptable for your endpoints (PC's and other clients), it is not that hard to find a gigabit switch with several 10G uplinks. The cheapest decent one I've seen in (2014) times is probably the Dell Networking 5500 series, such as the 5524,
often available for around $400 on eBay. That model comes with two 10G SFP+ slots, which could be used for a FreeNAS box and a workstation at 10G, while also allowing all remaining stations to share in the 10G goodness. Now that it's 2016 we're also seeing the Dell Networking N2024, which is an entry-level Force10 based switch. If you don't mind eBay for all purchases, you can get a basic 10G setup for your NAS and one workstation for less than $1000.
We recently debated another alternative, which is to abuse the FreeNAS box itself as a bridge using FreeBSD's excellent bridging facility. This is very cost-effective but has some caveats ... primarily that you need to be more aware that you've got a slightly hacked-up configuration. Since modern ethernet technologies are fully capable of point-to-point operation, without a switch, clients can be hooked up directly to the server (via a 10Gbase-T crossover cable, or SFP+). The simple case of a single workstation hooked up to the server via a direct cable is fairly easy. Multiple workstations might involve bridging. If you wish your clients to receive Internet connectivity, that's more complicated as well.
In 2013, Netgear introduced a
few new 10GBASE-T switch options including the
ProSafe XS708E which offers 8 ports for a cost around $100 per port.
The
Dell PowerConnect 8024F is often available on
eBay for around $1800, offering a mix of SFP+ ports along with four 10GBASE-T. This is probably the cheapest option to get 10gigE for a NAS or two, some ESXi boxes, and then a few runs of 10GBASE-T for workstations.
What Card Do I Pick?
This forum has been very pro-Intel for gigabit cards, because historically they've "just worked." However, for 10gigE, there have been some driver issues in versions of FreeNAS prior to 9.3 that lead to intermittent operation. Additionally, the Intel adapters tend to be rather more expensive than some of the other options. 10gigE is not in high demand, so often some of the niche contenders have products that may, counterintuitively, be very inexpensive on the used market. These cards may be just as good a choice - if not better - than the Intel offerings. We're Intel X520 here but the following notes are gathered from forum users.
@depasseg and I note:
Intel X520 and X540 are supported via the ixgbe driver.
Intel periodically suffers from knockoff cards in the new and used markets. There should be a Yottamark sticker on it that'll help authenticate the card as genuine. Check the country, datecode, and MAC address Yottamark gives you, don't just blindly trust it. Not a good choice if you wish to run versions prior to 9.3.
https://bugs.freenas.org/issues/4560#change-23492 Also note that there's been a variety of
problem reports with the X540 and TSO.
@Mlovelace,
@depasseg, and
@c32767a note:
Chelsio is iXsystems' card of choice.
@Norleif notes that the S320E-SR-XFP can sometimes be found for less than $100 on eBay. The Chelsio T3, T4 and T5 ASICs are fully supported by the current version of FreeNAS and are the cards shipped for 10gigE if you buy a TrueNAS system. iXsystems: "FreeNAS 9.2.1.5 supports the Chelsio T3 with the cxgb driver. It supports the T4/T5 with the cxgbe driver. We keep these drivers current with the latest code from the vendor. By far and away the Chelsio cards are the best FreeBSD option for 10Gbe." Also note that the S310E only supports PCIe 1, so speeds may be limited especially in an x4 slot.
@Mlovelace also has found a
great vendor for generic Chelsio SFP+ optics.
@depasseg and
@c32767a note:
SolarFlare: Some users recommend the SFN 5162F.
@Norleif reports:
IBM NetXTREME II 10GBit (Broadcom BCM 57710) Works in FreeNAS 9.3, can sometimes be found for less than $100 on eBay.
@Borja Marcos notes: Beware the Emulex "oce" cards - serious issues with them, panics when moving some traffic. There is a patch (see relevant discussions on the freebsd-net mailing list) but the stock driver crashes badly.
As usual, this post isn't necessarily "complete" and I reserve the right to amend it and/or delete, integrate, and mutilate the reply thread as I see fit in order to make this as useful as possible.
