10 Gig Networking Primer

[helloha]

I am after some chelsio 10gbe base-t cards. Does anyone have a list of the part numbers for them? Its hard to work out from the card descriptions if they are sfp or base-t.
I know that 10gbe copper SFP modules are hilarously expensive. I just want to put together a small 10gbe network without using fibre.

Just get this:

Lot 5 Chelsio S310E-CR 10Gbps SFP+ Adapter PCI-e x8 10GB 110-1107-30 10 Gigabit
http://www.benl.ebay.be/itm/361426967654?_trksid=p2060353.m1438.l2649&ssPageName=STRK:MEBIDX:IT

Add twinax cable and you're done!

 

[helloha]

Unless there is an SG300 model I don't know about that includes a 10Gbps port, I'm confused about why you would expect a 10Gbps SFP+ transceiver to work in a 1 Gbps switch?

As for your second question, just choose any subnet that you aren't already using in your network (I used 192.168.250.x, but it's totally arbitrary) and give the 10Gbps NIC on your PC one IP (e.g. 192.168.250.1) and the FreeNAS end another (192.168.250.2). When you map a drive on the PC, make sure you use the 10Gbps FreeNAS IP to do it, or you'll be using your 1G link instead. Do not try to configure any default gateway on your 10G IP. The default gateways stays with the 1G NIC; that's how the system knows how to get to the Internet.

Thanks for the tip. As for the SG-300 situation, I'm on a hackintosh and nearly all NIC drivers are a flaky hack. Was looking for an affordable and reliable solution to connect to my switch without having to fork out €€€ for a smalltree card.

 

[Rand]

Just for (my own as much as others) reference, the following settings provided max 10GBe speed on an Intel x550 onboard nic with Freenas 9.10 (almost) release version (ie FreeBSD 10 based)
(based on https://pleiades.ucsc.edu/hyades/FreeBSD_Network_Tuning, v10 values)

Nic options: "mtu 9014 rxcsum txcsum tso4 lro"

sysctl kern.ipc.somaxconn=2048
sysctl kern.ipc.maxsockbuf=16777216
sysctl net.inet.tcp.recvspace=4194304
sysctl net.inet.tcp.sendspace=2097152
sysctl net.inet.tcp.sendbuf_max=16777216
sysctl net.inet.tcp.recvbuf_max=16777216
sysctl net.inet.tcp.sendbuf_auto=1
sysctl net.inet.tcp.recvbuf_auto=1
sysctl net.inet.tcp.sendbuf_inc=16384
sysctl net.inet.tcp.recvbuf_inc=524288

O/C the appropriate tunables must be set for keeping these values

I have reached > 1GB/s with this config (RamDisk on Windows -> 950 nvme single disk datastore via cifs, E3-1245v5, 16GB ram, 16,2 GB testfile )

Edit: clarified used version

 

[Mlovelace]

Nic options: "mtu 9014 rxcsum txcsum tso4 lro"

Do you normally run jumbo frames on your network or did you find you needed jumbos to get the throughput?

 

[Rand]

I run them throughout the environment.
I dont think they are absolutely necessary, in some earlier experiments the NIC settings improved performance by only 10% (from abysmal 58MB to 65MB right after installation).
Haven't had the time to deactivate the various values to see which one is the most beneficial yet.

Based on yet earlier experiments (Intel AT2 adapter) I'd say the Nic options are relevant for offloading data/freeing the cpu if that is limited, but the sys values are the ones really improving performance

 

[bartnl]

Just for (my own as much as others) reference, the following settings provided max 10GBe speed on an Intel x550 onboard nic with Freenas 10.3 Beta:
(based on https://pleiades.ucsc.edu/hyades/FreeBSD_Network_Tuning, v10 values)

thanks for posting, I've used similar settings for 9.10 and am getting 300-400MB/s which already is more than what I reached with the Freenas defaults. Will compare my settings when at home.

 

[Rand]

It was on 9.10 (Release -1 day) actually, fixed in OP

 

[bartnl]

yeah, i had upgraded and see a big improvement. Went from 185MB/s to 400MB/s when copying between my Windows 10 PC with a 12 disk raid 6 volume and my FreeNAS with 12 disk raidz2 volume. Would love to get closer to your figures, but I already use the same parameters

 

[Rand]

Whats your win config?
On my old box I switched Win and FreeNas CPUs in the end ( E3-1230v3/i3-4130 ) b/c Windows just couldnt cope with the 10G on i3 with higher speeds.
Looks way better on the E3 - FreeNas on the other hand did have little issues maxing my back then disk speed (600 MB/s) over 10GBE
O/C there are some optimizations to be done in Win as well (in nic settings)

 

[bartnl]

I actually have two Windows 10 64-bit machines, one is my workstation and has a X99 motherboard with an i7-5820K CPU and 32 GB of RAM. Other PC is currently acting as a fileserver and has a E3-1275V2 CPU and also 32 GB of RAM. This Windows fileserver is equipped with an Areca 1882ix-24 Raid card. Both windows PC's have an Intel X540 10Gb NIC in them and are connected to the same 10Gb switch as my FreeNAS box is. Between the Windows PC's I can get close to line speed (~1,15 GB/s) using a RAM disk on one end and a 12 disk Raid6 array on the other. I'm confused as to why performance on my FreeNAS box is no better.
I suspect FreeBSD support for my FreeNAS hardware is not optimal. With the speeds I'm now getting FreeNAS could be limited by the storage pools but then again I get identical performance both to a single SSD and a 12 disk raidz2 pool.

 

[Rand]

Hm that indeed sounds a little fishy.
You could try one of the 540s in the server maybe ... I now the Chelsios are supposed to be good but it might be worth a shot.
Else no idea except playing around a bit more - there where some other settings that i didnt use that might help ... https://calomel.org/freebsd_network_tuning.html
And some older bugs having to do with TSO (https://bugs.freenas.org/issues/4560)

 

[guc32]

So with the release of Freenas 9.10, do Mellanox ConnectX-2 cards work now?

 

[guc32]

So got my Mellanox today and popped into Freenas box. No go. Thought Freenas 9.10 is on FreeBSD 10? Shouldnt the card work then?

 

[jgreco]

So got my Mellanox today and popped into Freenas box. No go. Thought Freenas 9.10 is on FreeBSD 10? Shouldnt the card work then?

Why would you expect to be able to pop some random crap card into your box and have it work?

Mellanox has their own driver and you might be able to run a ConnectX-3 or ConnectX-4 card if you hack in support for their driver, but the ConnectX-2 is specifically called out by name as unsupported even by Mellanox.

 

[TXAG26]

Love it, "some random crap card"!

 

[guc32]

Lol. Doh, Didnt know the Mellanox is that bad. Can you tell im a total Noob. :(

So I should get a Chelsio card then? Ive see on ebay S320 and N320. Will they both work? Are those the card I should be getting?

 

[jgreco]

Love it, "some random crap card"!

Well? Really. What would you call it, when a card isn't listed in the FreeBSD HCL, and you go to the manufacturer's site and they talk about their own driver, and then you pull up the driver's documentation and it explicitly calls out that card as not being supported any longer? If its own manufacturer can't even be arsed to support it with a driver, it's a pretty crappy card, eh?

And to do this in a thread where we've spent so much time and effort discussing the things that actually do work, well, I think it's fair that I get to be a little annoyed.

 

[jgreco]

Lol. Doh, Didnt know the Mellanox is that bad. Can you tell im a total Noob. :(

So I should get a Chelsio card then? Ive see on ebay S320 and N320. Will they both work? Are those the card I should be getting?

There's a reason they're blowing those out on eBay for five bucks or whatever ridiculous price they are today.

The older Chelsio S320E-CR cards may be okay but they're limited to PCIe 1.1 at 8 lanes (~16Gbps), and there've been some complaints that they're not real good at higher speeds. If you're just a home user who wants to be able to blast gigE to several clients simultaneously, but maybe not more than 4 or 5Gbps, my *guess* is that'd be a reasonable pick. These are also cheap for that same reason - older, slower, hotter tech. Google extensively, this is not an endorsement.

If you can justify it, the Chelsio T420-CR is probably a better choice. That's been working fine here and is what we've seen many other users using. The Chelsio Terminator 5 stuff is of course even nicer yet, but sometimes you have to settle for "good enough."

The Intel X520-SR2 is also a fine choice, but may require a little tweaking for optimal performance.

 

[guc32]

There's a reason they're blowing those out on eBay for five bucks or whatever ridiculous price they are today.

The older Chelsio S320E-CR cards may be okay but they're limited to PCIe 1.1 at 8 lanes (~16Gbps), and there've been some complaints that they're not real good at higher speeds. If you're just a home user who wants to be able to blast gigE to several clients simultaneously, but maybe not more than 4 or 5Gbps, my *guess* is that'd be a reasonable pick. These are also cheap for that same reason - older, slower, hotter tech. Google extensively, this is not an endorsement.

If you can justify it, the Chelsio T420-CR is probably a better choice. That's been working fine here and is what we've seen many other users using. The Chelsio Terminator 5 stuff is of course even nicer yet, but sometimes you have to settle for "good enough." :)

The Intel X520-SR2 is also a fine choice, but may require a little tweaking for optimal performance.

Excellent!!! Thats what I needed to go in the right direction. Thank You

 

[Lighti]

The Mellanox Connectx-2 may be the crappy card from hell, but she already has PCIe 2.0 and is said to only need around 7 watt of power, while the Chelsio S320E has only PCIe 1.1 and needs more than 16 watt of power.

mod note: Deleted discussion of possible Mellanox ConnectX fudge. That thing's been kicking around in an awful state for years now. We don't encourage modifying the FreeNAS firmware. We don't suggest using virtually untested drivers on a FreeNAS system. We don't encourage doing things that place data at risk. And I don't want this in my 10 Gig Networking Primer thread since inevitably someone will roll around and consider it to be tacit approval. - JG