Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.
10 Gig Networking Primer

10 Gig Networking Primer

nightshade00013

FreeNAS Expert
Joined
Apr 9, 2015
Messages
1,259
Thanks
293
It tends to depend on if the hardware is locked down to a brand or not. That is where it becomes more limited with DAC's since if both products are hardware locked and are different you can not use a DAC to link the two.

The optics are basically the same but since the cable is removable and the transceivers will be able to inter-operate you just use whatever is required or a "generic" one. Some devices will work just fine with pretty much whatever but you will not be able to get the same information unless you use ones that are flashed to show they are their brand. The way I ended up going was to just hit FiberStore www.fs.com and let them look it up.

Also want to add in here that the Quanta LB6M became a little bit more viable as it seems they can be flashed to a Brocade TurboIron

If you are looking for one they also have a deal going on till the end of march https://forums.servethehome.com/index.php?threads/the-quanta-lb6m-deal-thread.17824/
 

loch_nas

FreeNAS Aware
Joined
Jun 13, 2015
Messages
79
Thanks
23
Thanks a lot! The fog clears more and more.

I've found the information that generic DACs work with the D-Link switch. And there are some people (in other forums) that claim that the Intel 10Gbit controller is more tolerant than other Intel NICs. So it should be possible to go the DAC-way if I cannot get fully compatible transceivers.

And yes, I have discovered fs.com already today afternoon and looked around there a lot. The good news is that they also deliver to my place and the stuff there is really affordable. The price difference to the original transceivers is amazing. I guess that all the things there come from china, but hey ... I'm not Scrooge McDuck swimming in a pool of money.
So if I understand right I have to get one transceiver branded for the switch (like D-Link) and one transceiver branded for the NIC (Intel).
I will ask fs.com directly which seems to be the safest way.

The Quanta LB6M switches look interesting (wow! 24-Port SFP+), but firstly I don't have enough space (depth) and secondly they are difficult to find at a decent price (for Austria).
In Europe it's generally a lot more difficult to get decent used stuff from ebay.

So I will give the D-Link switch a try as it's decently priced (~300 €), is energy-saving (=>not so loud) and has the desired 4 SFP+ interfaces. I will inform you guys if and how things work.
 

c32767a

FreeNAS Experienced
Joined
Dec 13, 2012
Messages
360
Thanks
57
Thanks a lot! The fog clears more and more.

I've found the information that generic DACs work with the D-Link switch. And there are some people (in other forums) that claim that the Intel 10Gbit controller is more tolerant than other Intel NICs. So it should be possible to go the DAC-way if I cannot get fully compatible transceivers.

And yes, I have discovered fs.com already today afternoon and looked around there a lot. The good news is that they also deliver to my place and the stuff there is really affordable. The price difference to the original transceivers is amazing. I guess that all the things there come from china, but hey ... I'm not Scrooge McDuck swimming in a pool of money.
So if I understand right I have to get one transceiver branded for the switch (like D-Link) and one transceiver branded for the NIC (Intel).
I will ask fs.com directly which seems to be the safest way.

The Quanta LB6M switches look interesting (wow! 24-Port SFP+), but firstly I don't have enough space (depth) and secondly they are difficult to find at a decent price (for Austria).
In Europe it's generally a lot more difficult to get decent used stuff from ebay.

So I will give the D-Link switch a try as it's decently priced (~300 €), is energy-saving (=>not so loud) and has the desired 4 SFP+ interfaces. I will inform you guys if and how things work.
I posted this a while ago #5. Basically, Twinax compatibility can be measured in 2 dimensions..

1) Active or Passive cables.. Some interfaces (NIC or Switch) require active, some require passive. some support both.
2) "Certified" hardware.. Some vendors only will allow "blessed" cables/optics. Sometimes there's an override, sometimes there isn't.

Optics are slightly different.. Specific media types may or may not be supported by a switch (ZR, LR, SR, etc) and specific vendors may only allow their coded optics. LR and SR optics are nearly always supported, so if you get 2 SR SFP+ modules, you should be fine. If your device doesn't have an override (eg cisco's "service unsupported-transciever") You can get the generic ones coded for your devices from fs.com.
 

loch_nas

FreeNAS Aware
Joined
Jun 13, 2015
Messages
79
Thanks
23
Again thnx a lot.
The compatibility issue is not something I'm really used to. But the good thing is that there's a lot to learn.

I had a chat with a support staff from fs.com (in Germany) and they assured me that they have compatible transceivers for the hardware I want to get.

So I ordered:
2 transceivers for the Intel NICs -> Intel E10GSFPSRX Compatible 10GBASE-SR SFP+ 850nm 300m DOM Transceiver Module
2 transceivers for the D-Link Switch -> D-Link DEM-431XT-DD Compatible 10GBASE-SR SFP+ 850nm 300m DOM Transceiver Module
2 optical fiber cables -> 2m (7ft) LC UPC to LC UPC Duplex 2.0mm PVC(OFNR) OM3 Multimode Bend Insensitive Fiber Patch Cable

Now I'm waiting for the hardware and the network equipment.
I really like that shop, seems to be quite competent and friendly.
Let's see how things work.
 
Last edited:

Stilez

FreeNAS Experienced
Joined
Apr 8, 2016
Messages
251
Thanks
68
I'm using very common 10G kit (Chelsio T420-CR's + Finisar SG10's + SR LC-LC cable) on a small 10G subnet that contains my NAS, a 2nd server (currently disconnected) and my Windows workstation/data crunching monster. All the machines use the same cards and transceivers, and they all have 9k jumbo configured. The connecting glue is a Netgear SFP+ managed switch with the same transceivers. Apart from that I've got no special config in use, and haven't enabled TOE - it's pretty much all at default.

Using iperf I ought to be able to get near line speed between any two of these, but I'm nowhere near it (bandwidth roughly 360 MB/sec both ways). I know in the past I was getting 7 gbps (~ 980 MB/sec) on Samba, after overheads, so I know the hardware's up to it. But I lost that config.

Reading the primer and thread it's clear I need to do a fair bit of 10G tuning, but as I'm troubleshooting other stuff right now, I'm really short on time to work on this as well for a bit.

To help me at least get "most of the way there", and leave the rest for another time, I'd really appreciate if anyone can offer a shortlist of the major tunables or ifconfig settings to start with for this combination, or to look at "down the line", to get me at least a bit closer to 10G as a starting point. Even if its not perfect. I'm hopeful this isn't difficult, since T420-CR's are about the single most commonly recommended setup for 10G on the forums :)

Thanks for helping!
 

c32767a

FreeNAS Experienced
Joined
Dec 13, 2012
Messages
360
Thanks
57
Now I'm waiting for the hardware and the network equipment.
I really like that shop, seems to be quite competent and friendly.
Let's see how things work.
Fiberstore made their business by serving the carrier and ISP market with good service and parts. They're larger now, but they seem to be holding on to their core principles. I think you'll be fine.. enjoy the faster network.
 

Mlovelace

FreeNAS Expert
Joined
Aug 19, 2014
Messages
1,040
Thanks
260
I'm using very common 10G kit (Chelsio T420-CR's + Finisar SG10's + SR LC-LC cable) on a small 10G subnet that contains my NAS, a 2nd server (currently disconnected) and my Windows workstation/data crunching monster. All the machines use the same cards and transceivers, and they all have 9k jumbo configured. The connecting glue is a Netgear SFP+ managed switch with the same transceivers. Apart from that I've got no special config in use, and haven't enabled TOE - it's pretty much all at default.

Using iperf I ought to be able to get near line speed between any two of these, but I'm nowhere near it (bandwidth roughly 360 MB/sec both ways). I know in the past I was getting 7 gbps (~ 980 MB/sec) on Samba, after overheads, so I know the hardware's up to it. But I lost that config.

Reading the primer and thread it's clear I need to do a fair bit of 10G tuning, but as I'm troubleshooting other stuff right now, I'm really short on time to work on this as well for a bit.

To help me at least get "most of the way there", and leave the rest for another time, I'd really appreciate if anyone can offer a shortlist of the major tunables or ifconfig settings to start with for this combination, or to look at "down the line", to get me at least a bit closer to 10G as a starting point. Even if its not perfect. I'm hopeful this isn't difficult, since T420-CR's are about the single most commonly recommended setup for 10G on the forums :)

Thanks for helping!
These should get you started, but when you get time, do some experimenting with the buffer sizes.
Tunables.JPG
 

Brezlord

FreeNAS Experienced
Joined
Jan 7, 2017
Messages
110
Thanks
15
Hi all,
I'm currently running FreeNAS as a VM in ESXi 6.5 with Intel X520 NIC with no issues. I've recently bought a Ubiquity EdgeSwitch 16-Port 10G (ES-16-XG) switch. It is super easy to setup with the web GUI or industry standard CLI. I'm waiting on a NMVe to PCI adapter so I can upgrade mt SLOG to a Samsung PM953 480GB before retesting write speed. Is there any performance loss to be expected when running FreeNAS on ESXi? I currently max out NFS sync writes at 140MB/s which is the speed of my current Inappropriate (ok for 1 GBe) Intel S3500 SLOG ssd. I am contemplating installing FreeNAS on bare metal just to test the difference and satisfy my curiosity. What are you're thoughts on 10GBe and running FreeNAS virtualized?

Thanks,
Simon
 

JustinClift

FreeNAS Experienced
Joined
Apr 24, 2016
Messages
274
Thanks
54
Can you recomend which 10 Gbit card buy?
This is probably what you want:

https://forums.freenas.org/index.php?resources/hardware-recommendations-guide.12/

Look for the "Download Now" button near the top right, as that has the link to the PDF.

If you're on a tight budget though, then second hand Mellanox cards seem to work ok in at least point-to-point configurations. Saying that from personal experience, though most people are using the more expensive network cards around (Chelsio, Intel). :)
 
Joined
Sep 19, 2018
Messages
22
Thanks
0
hey all! amazing topic with loads of material. i am form UK so limited amount of 2nd hand kit here.

my requirements are - want to connect my pc to nas on 10gbe. want to make it a staged upgrade:

phase 1 -

Install on PC and NAs: Dell X520-DA2
then connect them up with DAC. (would love recommendation on cable) ideally longest possible as not sure where it all will go just yet.

phase 2 - buy:
1. switch(need advice for small home network)
2. transceivers - again need advice.
3. buy cable - also need advice
4. did i miss anything else?

some advice would be really appreciated!
 

Stilez

FreeNAS Experienced
Joined
Apr 8, 2016
Messages
251
Thanks
68
hey all! amazing topic with loads of material. i am form UK so limited amount of 2nd hand kit here.

my requirements are - want to connect my pc to nas on 10gbe. want to make it a staged upgrade:

phase 1 -

Install on PC and NAs: Dell X520-DA2
then connect them up with DAC. (would love recommendation on cable) ideally longest possible as not sure where it all will go just yet.

phase 2 - buy:
1. switch(need advice for small home network)
2. transceivers - again need advice.
3. buy cable - also need advice
4. did i miss anything else?

some advice would be really appreciated!
I'm in the UK, and my LAN is on 10G, you'll have no difficulty. I can see you've already made some plans. Although I had a very similar aim to you, I didn't use the components you're using. Perhaps what I've done will be useful, so I'll run you through what I'm using here.

The first thing is, you almost don't need "staged". You need NIC cards, transceivers and cables - that's all. As soon as you plug the cards in, you'll find if they work (on FreeNAS/FreeBSD) or at worst, as soon as you install the drivers (on your PC if it's Windows). If your PC is running Linux or something, I haven't a clue but it should work quickly enough. As soon as the cards are recognised, you'll want to plug in the transceivers and cables. At which point you have a 10G link. (Staging? What staging :) )

When I was buying, the recommended advice was to go for Chelsio rather than Intel based 10G NICs. Apparently Intel are top notch at 1G but for 10G Chelsio NICs were the #1 recommendation by both pfSense and FreeNAS projects, both of which run on FreeBSD. That's a pretty weighty recommendation, so if you are working on the basis that it's automatically Intel for NICs, you might want to consider them. Even if Intel are now equally recommended, it's likely Chelsio could be cheaper as their previous generation cards are easily available 2nd hand and were also rock solid, whereas for Intel that's not so much the case.

  • NIC Cards - Get a card that takes SFP+ form factor transceivers. The Dell X520-DA2 you mentioned is SFP+ anyway, and if you do go for Chelsio, pick any of their "CR" cards. The cards you want will be named "T4xx" or "T5xx", and have the letters "CR" in their name. I'm using several T420-CR cards here, and never had an issue with them. I'd recommend the T420-CR or T520-CR as good dual-port cards for Windows and FreeNAS. (There's a few variations on the models, like "LL" for low latency, etc, but if it has the right Txxx and also "CR", it'll work for you). Like the Dell card you mention, the dual port cards are easiest to find online.

  • Transceivers/cable - Some cards have manufacturer lock-in/lock-out (or used to, anyhow). I can't say if Dell's cards are locked to Dell transceivers (probably not!), but Chelsio's definitely aren't. Some models also have compatibility issues. Finisar make very good transceivers and very little in the way of compatibility issues with Intel or Chelsio - get those. They aren't expensive. It can be worth it to buy on EBay from the US, for these, if you can't find them in the UK. The transceivers needs to match the cable obviously. There are numerous kinds of cable, but the one I use, after a lot of reading up to get my head round it all, is called "OM2" - it's got enough range for any usual house. If you need more, then OM3 will work but you'll need to check transceiver compatibility. Don't go for "CX" or "direct attach", get a standard optical cable and transceivers :) The other compatibility issue is the cable connector form factor. So for example, you might get "LC-LC" cable, which will link between two transceivers that both take LC connectors. That's what I use. The transceivers have numerous model numbers, but are often coded "SG10". The model number I use is "FTLX8571D3BCL" and they handle 850nm optical fibre such as OM2 with an LC connector.

  • Switch - Many companies do decent SFP+ 10G switches of course. The problem is, that most of them use 10G as an uplink port so you don't get that many of them, which is limiting. You'll probably want >2 ports, and finding a budget switch that has >2 ports, * and * isn't stupid price, is tricky. I can tell you what I did, and what I found, but a lot depends on your longer term goals.

    First off, you can directly link your devices. You get 2 ports on each card, so you can directly wire your NAS to a PC and one other device, without need for a switch, for data. It's crude, but free.

    If you want just 2 ports (and some 1G ports), there are numerous switches - even Cisco models aren't that expensive for this simple requirement.

    If you want more than that, then you need to think and look round 2nd hand gear. There are brands like Ubiquiti that do many SFP+ port switches, but they seem to have quality issues, or did at one time not long ago. Arista are very good, but CI only, not GUI. HP have some very good ones, but you need a module to get >2 ports, modules are available second hand and this is one very good option, but check the module will work with the switch you get if you do that!

    I ended up with Netgear - they have a good reputation for build quality and software, and they have a very full range of switch control, so you can set ACLs (permissions) on each port, more fully than on many budget switches. They weren't that expensive either, and have 4 SFP+ ports. For a small desktop switch I'd go with Netgear or Cisco.

  • Where to buy - All this stuff is easily available on EBay - Chelsio cards at £99, Transceivers at £9 (many of these, enter as a search on EBay without the quotes: (FTLX8571D3BCL,sg10) (transceiver,transceivers) ), and OM2 cable with LC-LC connectors at £5 for 5m (you can get 10m/20m/30m OM2 if you want it longer, I picked 5m as an example). As of today, these are available current prices with UK sellers

Hope this helps - if you need more info, ask!
 
Last edited:
Joined
Sep 19, 2018
Messages
22
Thanks
0
I'm in the UK, and my LAN is on 10G, you'll have no difficulty. I can see you've already made some plans. Although I had a very similar aim to you, I didn't use the components you're using. Perhaps what I've done will be useful, so I'll run you through what I'm using here.

The first thing is, you almost don't need "staged". You need NIC cards, transceivers and cables - that's all. As soon as you plug the cards in, you'll find if they work (on FreeNAS/FreeBSD) or at worst, as soon as you install the drivers (on your PC if it's Windows). If your PC is running Linux or something, I haven't a clue but it should work quickly enough. As soon as the cards are recognised, you'll want to plug in the transceivers and cables. At which point you have a 10G link. (Staging? What staging :) )

When I was buying, the recommended advice was to go for Chelsio rather than Intel based 10G NICs. Apparently Intel are top notch at 1G but for 10G Chelsio NICs were the #1 recommendation by both pfSense and FreeNAS projects, both of which run on FreeBSD. That's a pretty weighty recommendation, so if you are working on the basis that it's automatically Intel for NICs, you might want to consider them. Even if Intel are now equally recommended, it's likely Chelsio could be cheaper as their previous generation cards are easily available 2nd hand and were also rock solid, whereas for Intel that's not so much the case.

  • NIC Cards - Get a card that takes SFP+ form factor transceivers. The Dell X520-DA2 you mentioned is SFP+ anyway, and if you do go for Chelsio, pick any of their "CR" cards. The cards you want will be named "T4xx" or "T5xx", and have the letters "CR" in their name. I'm using several T420-CR cards here, and never had an issue with them. I'd recommend the T420-CR or T520-CR as good dual-port cards for Windows and FreeNAS. (There's a few variations on the models, like "LL" for low latency, etc, but if it has the right Txxx and also "CR", it'll work for you). Like the Dell card you mention, the dual port cards are easiest to find online.

  • Transceivers/cable - Some cards have manufacturer lock-in/lock-out (or used to, anyhow). I can't say if Dell's cards are locked to Dell transceivers (probably not!), but Chelsio's definitely aren't. Some models also have compatibility issues. Finisar make very good transceivers and very little in the way of compatibility issues with Intel or Chelsio - get those. They aren't expensive. It can be worth it to buy on EBay from the US, for these, if you can't find them in the UK. The transceivers needs to match the cable obviously. There are numerous kinds of cable, but the one I use, after a lot of reading up to get my head round it all, is called "OM2" - it's got enough range for any usual house. If you need more, then OM3 will work but you'll need to check transceiver compatibility. Don't go for "CX" or "direct attach", get a standard optical cable and transceivers :) The other compatibility issue is the cable connector form factor. So for example, you might get "LC-LC" cable, which will link between two transceivers that both take LC connectors. That's what I use. The transceivers have numerous model numbers, but are often coded "SG10". The model number I use is "FTLX8571D3BCL" and they handle 850nm optical fibre such as OM2 with an LC connector.

  • Switch - Many companies do decent SFP+ 10G switches of course. The problem is, that most of them use 10G as an uplink port so you don't get that many of them, which is limiting. You'll probably want >2 ports, and finding a budget switch that has >2 ports, * and * isn't stupid price, is tricky. I can tell you what I did, and what I found, but a lot depends on your longer term goals.

    First off, you can directly link your devices. You get 2 ports on each card, so you can directly wire your NAS to a PC and one other device, without need for a switch, for data. It's crude, but free.

    If you want just 2 ports (and some 1G ports), there are numerous switches - even Cisco models aren't that expensive for this simple requirement.

    If you want more than that, then you need to think and look round 2nd hand gear. There are brands like Ubiquiti that do many SFP+ port switches, but they seem to have quality issues, or did at one time not long ago. Arista are very good, but CI only, not GUI. HP have some very good ones, but you need a module to get >2 ports, modules are available second hand and this is one very good option, but check the module will work with the switch you get if you do that!

    I ended up with Netgear - they have a good reputation for build quality and software, and they have a very full range of switch control, so you can set ACLs (permissions) on each port, more fully than on many budget switches. They weren't that expensive either, and have 4 SFP+ ports. For a small desktop switch I'd go with Netgear or Cisco.

  • Where to buy - All this stuff is easily available on EBay - Chelsio cards at £99, Transceivers at £9 (many of these, enter as a search on EBay without the quotes: (FTLX8571D3BCL,sg10) (transceiver,transceivers) ), and OM2 cable with LC-LC connectors at £5 for 5m (you can get 10m/20m/30m OM2 if you want it longer, I picked 5m as an example). As of today, these are available current prices with UK sellers

Hope this helps - if you need more info, ask!
Thanks mate! really appreciate your advice.

I am going to bite the bullet on intel, i have read the issues resolved with firmware and latest bug fixes. fingers crossed.

Have gone for :
2no Dellx520-Da2 (each include 2 tranceivers) for £65 each.
2no extra transceivers which FS.com advised that are compatible
OM3 cabling. short for NAS to switch and 30M for switch to PC
and a bit of a risk on FS.com own branded switch:
S3900-24T4S Fanless 24-Port 10/100/1000BASE-T Gigabit Stackable Managed Switch with 4 10GE SFP+ Uplinks - £210 which is a bargain, but will see how it goes. It was just released so may be complete garbage :)

I think thats all i need. I will try to bond 2 ports from NAS to switch and 1 to 1 for PC to switch for now.

fingers crossed.

Am i missing anything?
 
Last edited by a moderator:

marcevan

FreeNAS Experienced
Joined
Dec 15, 2013
Messages
365
Thanks
13
Just ordered an Intel X540Ti NIC in lieu of SFP+ as I have 10G switch with each port 10G capable + only 2 SFP ports. Since FreeNAS is the bottleneck of activity and my house is wired with Cat-6a, seems like RJ was the way to go. Arrives today with a few 6a cables so I can power down the NAS, open it, swap the network card out, put back in and see how things go.
 

rmccullough

FreeNAS Experienced
Joined
May 17, 2018
Messages
144
Thanks
16
What is the most cost-effective way to implement 10G on my home network? Earlier in this thread it seems to discourage Intel chipsets in favor of Chelsio. Is this still the case? If I were to look for this hardware on eBay, what is a reputable brand for NIC and Switch that is known to work well with FreeNAS/FreeBSD, as well as providing a great value?
 

danb35

FreeNAS Wizard
Joined
Aug 16, 2011
Messages
10,417
Thanks
3,681
Chelsio continues to work well with pretty much everything, and is pretty reasonably-priced. I believe Intel works well also, but I think Chelsio is still more popular. As to switches, I just picked up a Dell 5524P for $125 shipped (I wanted PoE), but it only has two SFP+ ports. I think @Chris Moore ran across a ~$150 switch with 48x GbE and 4x SFP+ not too long ago, which is an outstanding value.

You can get optics on eBay, but I don't know that you'll save much (if any) compared to just buying them from fs.com. And their fiber patch cables are great.
 

rmccullough

FreeNAS Experienced
Joined
May 17, 2018
Messages
144
Thanks
16
Chelsio continues to work well with pretty much everything, and is pretty reasonably-priced. I believe Intel works well also, but I think Chelsio is still more popular. As to switches, I just picked up a Dell 5524P for $125 shipped (I wanted PoE), but it only has two SFP+ ports. I think @Chris Moore ran across a ~$150 switch with 48x GbE and 4x SFP+ not too long ago, which is an outstanding value.

You can get optics on eBay, but I don't know that you'll save much (if any) compared to just buying them from fs.com. And their fiber patch cables are great.
Thanks, this is exactly what I was looking for.

I want to confirm something. Would it be possible to use 1 SFP+ port to connect to my FreeNAS server and the other to run a cable to a 2nd switch (with SFP+) in my office, would it provide 10G connectivity between the switches? Thus allowing greater throughput from the clients in my office (and WiFi AP) to the switch in my crawlspace (where FreeNAS is)?
 

danb35

FreeNAS Wizard
Joined
Aug 16, 2011
Messages
10,417
Thanks
3,681
Would it be possible to use 1 SFP+ port to connect to my FreeNAS server and the other to run a cable to a 2nd switch (with SFP+) in my office, would it provide 10G connectivity between the switches?
Sure; I'm doing something very much like this since a week ago. Server rack is out in my workshop, with FreeNAS, Xen (xcp-ng) server, and Proxmox plugged into 10G ports on the switch, and the fourth 10G port is fiber into the house, to the 5524 (which I'll be replacing, probably, with the 5524P).
 

Chris Moore

Super Moderator
Moderator
Joined
May 2, 2015
Messages
9,362
Thanks
2,997
This is the switch I bought:
https://www.ebay.com/itm/Aruba-S250...ASE-T-4x1000BASE-X-10GBASE-X-KMJ/232504836333
I did need to update the firmware and modify the default settings, but it is working great.
I ended up with a card like this in each of my FreeNAS servers:
https://www.ebay.com/itm/Chelsio-10...8-30-2x-10G-SFP-Tranceivers-HIGH/113233543113
I just use a single port, but I have a couple of other computers connected to the additional two 10Gb ports on the switch and most of the devices in my network are fine on a 1Gb connection.
 
Top