Will this gear do 10 gigabits?

[Spearfoot]

Yikes I just installed 10Gbe CX-4 to my switch and X2 10Gbe transceiver to connect my desktop and get half of that on iperf. I had been thinking about replacing the processors on my X7 board for a long time now but maybe I should just be looking for a reasonable X10 board. system specs in sig.

Yeah, I bet you're running into bus limitations like I did on my X8SIE -- only worse!

 

[chris crude]

Sorry to revive an old thread, but just got around to installing and iperf testing the X520-DA1 cards. only getting 1.4GB transfer rates. All stock settings. Any ideas where to start troubleshooting my issues? Any help is appreciated. Systems are:

Freenas=9.10.2-U1
Mobo=Supermicro X11SSM-F-O
CPU= Xeon E3-1240 v5
RAM= 2xSamsung M391A2K43BB1-CPB 16GB DDR4-2133 ECC Un-Buffer LP Server Memory
1G nic=192.168.50.107/24 -to router
10G nic=10.10.10.1/16 – direct to client

Windows 10 Pro 64bit
Mobo= Gigabyte Technology Co., Ltd. X58A-UD3R
CPU= i7 950
RAM =18GB Corsair
1G nic=192.168.50.5/24 -to router
10G nic=10.10.10.2/16 – direct to server

 

[Spearfoot]

Sorry to revive an old thread, but just got around to installing and iperf testing the X520-DA1 cards. only getting 1.4GB transfer rates. All stock settings. Any ideas where to start troubleshooting my issues? Any help is appreciated. Systems are:

Freenas=9.10.2-U1
Mobo=Supermicro X11SSM-F-O
CPU= Xeon E3-1240 v5
RAM= 2xSamsung M391A2K43BB1-CPB 16GB DDR4-2133 ECC Un-Buffer LP Server Memory
1G nic=192.168.50.107/24 -to router
10G nic=10.10.10.1/16 – direct to client

Windows 10 Pro 64bit
Mobo= Gigabyte Technology Co., Ltd. X58A-UD3R
CPU= i7 950
RAM =18GB Corsair
1G nic=192.168.50.5/24 -to router
10G nic=10.10.10.2/16 – direct to server

Yikes! Network troubleshooting is always a tar pit!

The X11SSM-F board is fairly new, so I'm looking at your somewhat-elderly Gigabyte X58A-UD3R motherboard:

Right off the bad, I see that Intel specifies a bus speed of 4.8 GT/s for the Core i7 950 while the X520 is PCIe v2.0 @ 5.0GT/s (as @MatthewSteinhoff pointed out). So you're losing a little bit of bandwidth there...

Page 12 of the X58A manual (available here) has a couple of caveats about PCIe slot usage:

(Note 2)
For optimum performance, if only one PCI Express graphics card is to be installed, be sure to install
it in the PCIEX16_1 slot; if you are installing two PCI Express graphics cards, it is recommended
that you install them in the PCIEX16_1 and PCIEX16_2 slots.
(Note 3)
The PCIEX8_1 and PCIEX8_2 slots share bandwidth with the PCIEX16_1 and PCIEX16_2 slots
respectively. When PCIEX8_1 is populated with an expansion card, the PCIEX16_1 slot will operate
at up to x8 mode; when PCIEX8_2 is populated with an expansion card, the PCIEX16_2 slot will
operate at up to x8 mode.

So... what PCIe cards have you installed in the Gigabyte system? And where did you install them? Which slot did you use for the X520? You may be losing bandwidth due to sharing between the slots, depending on what you have installed, and where you installed it.

Lastly... Intel has some tuning tips for their ethernet adapters that may be handy.

Good luck!

 

[chris crude]

so I'm looking at your somewhat-elderly Gigabyte X58A-UD3R motherboard:

Hey hey hey thats my baby. She's not elderly she's ahead of her time! Triple channel memory is still going to be a thing YOU WATCH! haha

I did look at my PCIe card management before i installed this. I am using the PCIEx8_2 slot, but the card i have in the 16_2 slot is a RAID5 disk controller of a brand whose name cannot be spoken around here. Its only a PCIEx4 card so cutting it to 8x mode isnt an issue. Almost embarrassed to say i think my issue was being new to iperf. I was only running a single stream test. Once i upped the streams the bandwidth increased.

 

[chris crude]

OK taught myself how to use tunables and loaded these recommended https://forums.freenas.org/index.php?threads/10-gig-networking-primer.25749/page-7#post-272868
and i realize i still wasnt using iperf properly by just adding streams. Also, I know i wont get much speed out of the RAID5 controller, but i think the fact my disks are on PCIE16_2 and my nic is on PCIE8_2 that it is choking that lane. a file transfer from my client C: SSD is about 400MB and transfer from the RAID5 partition is 100-150MB

 

[Spearfoot]

TL;DR: In answer to my original question: Yes! This gear will do 10 gigabits!

I've replaced the elderly Dell 2816 switch in my lab with a Dell PowerConnect 5524P unit purchased on eBay. I installed Intel X520-DA1 NICs (also purchased on eBay) in my primary and secondary All-in-One (AIO) servers and connected them to the 5524's two 10G SFP+ ports using 'custom made' TwinAx cables from The Fiberstore (www.fs.com). By 'custom made', I mean that I added a note to my order indicating I would be connecting X520-DA1 cards to a Dell 5524, and the good folks at the Fiber Store made up cables with Intel- and Dell-compatible transceivers on either end - at no upcharge!

Ten Gigabit Goodness Ensued! :)

Well, not quite ten... I get a maximum transfer rate of ~8.2 Gbit/s on the primary server (an X10SL7-F), but only ~3.7 Gbit/s on the secondary (an X8SIE-4LNF). The primary server's results are well within what I'd been led to expect by the experiences of others, but I'm a little surprised at the secondary server's poor showing. In its defense, it is an older machine. I've tried tweaking PCI-related BIOS settings and using an alternative ESXi NIC driver, so far to no avail: 3.7Gbit/s seems to be the limit for this box. But I welcome suggestions from the experts here on the forum for improving its performance and proving me wrong about this.

Both AIO servers run VMware ESXi 6.0U3 with FreeNAS 9.10.2-U1 installed as a VM (see 'my systems' below for details).

I tested transfer rates by running iperf in server mode on the appropriate FreeNAS instance, and connecting to it from four client iperf instances running on a pair of desktops, a laptop, and the FreeNAS VM on the other AIO server. See the screenshots below for results.

Overall I'm quite happy with this setup. I'm getting the desired performance where it counts, on my primary AIO server, and the secondary server, while not quite as fast as hoped, nevertheless has gained a four- or five-fold increase in transfer rates vs plain old gigabit Ethernet. And I'm especially pleased with the relatively low cost!

Other notes:
The Dell 5524P is loud! I expected it to be, because it's a PoE switch. It has a beefier 600W power supply containing two integral 40x40x28mm fans - Delta FFB0412SHN units rated at 54.5 dBA, the kind of 'screamers' we've all seen in 1U gear - plus two additional chassis fans which are temperature-controlled and never seem to come on. I've ordered a pair of Sunon GM1204PQV1-8A fans rated at 36.5 dBA to replace the screamers. This gear is all located in my shop, so I don't mind things being a little noisy, but not screaming Banshee levels of noisy!

I purchased my 5524P for $195. The same seller has since raised the price to $215... better get 'em while they're hot! But if you're willing to forego PoE support, you can still get the common, garden variety 5500-series switches for under $200.

Transfer rates, primary FreeNAS server:
View attachment 16287
Transfer rates, secondary FreeNAS server:
View attachment 16288

Not sure how I managed to do it... but now I'm only getting ~4Gbps max where before I was getting twice that! The only change I can think of is the FreeNAS upgrade to 9.10.2-U2.

I'm stumped...

 

[Rick Arman]

Not sure how I managed to do it... but now I'm only getting ~4Gbps max where before I was getting twice that! The only change I can think of is the FreeNAS upgrade to 9.10.2-U2.

I'm stumped...

is it fixed? if so how did you do it?

 

[Spearfoot]

is it fixed? if so how did you do it?

Yes, it worked as expected once I enabled jumbo frames:

https://forums.freenas.org/index.ph...hen-receiving-network-data.53722/#post-371578

 

[Rick Arman]

Thanks so much! now I know you're running FreeNAS virtualized but did you implement tunables or did you revert back to stock?

 

[Spearfoot]

Thanks so much! now I know you're running FreeNAS virtualized but did you implement tunables or did you revert back to stock?

I've got several tunables configured. Note that only about half of these are enabled:

 

[Rick Arman]

I've got several tunables configured. Note that only about half of these are enabled:
View attachment 21550

thanks again!!!!

 

[Rick Arman]

Last question: are you running 9 or 11? I am running 9.10.2 now and building a new FreeNAS box.

 

[Spearfoot]

Last question: are you running 9 or 11? I am running 9.10.2 now and building a new FreeNAS box.

I'm running 11.0-U3