PCIe X8 vs. X4 slot speeds for 10Gb fiber

[BigDave]

I'm just playing around while gathering up the hardware to add 10Gb connections for two
desktops and my FreeNAS machine. So far I have two Chelsio S310E-CR cards, two Chelsio
SM10G-SR transceivers and a 2M length of OM3 cable. I installed one card in my desktop rig
and one card in my FreeNAS test rig (X9SCM).
My test rig's mobo has 4 PCI express ports, all four of them are X8 in physical size but only
slots #7 & #6 are electrically X8. The remaining slots #4 & #5 are shown on the Supermicro
Reference Guide as electrically being X4.
I wanted to test my purchases I've made so far, so that I could return anything found to be
defective while waiting on a Dell SFP+ switch bargain to fall in my lap. ;)
I dropped the cards in both machines and noticed the FreeNAS test rig did not pick up the
card during bootup, so I thought I had a lemon somewhere. I first switched transceivers = no
diff, FreeNAS rig still a no go. Switched cards (other was working in my Windows 7 rig) and
it turns out my motherboard (X9SCM) has a bad PCIe slot. The good news is that after
switching the card to one of the X4 slots, it works! here's the output of ifconfig cxgb0...

Code:

[root@freenas2 ~]# ifconfig cxgb0
cxgb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TX
CSUM_IPV6>
ether 00:07:43:06:08:c2
inet 192.168.250.2 netmask 0xffffff00 broadcast 192.168.250.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet 10Gbase-SR <full-duplex>
status: active
[root@freenas2 ~]#

My question is this...
Will this card perform @ 10Gb in an X4 PCIe slot being that the card is X8???

 

[jgreco]

Generally. A PCIe 2.0 lane is capable of about 500MBytes/sec (4Gbits/sec). So you've got 16Gbps of PCIe to the card. You can't ACTUALLY support running 10Gbps bidirectionally. However, your FreeNAS box is also probably incapable of putting that much traffic out unless you have an SSD based pool, so, my suggestion, don't get too OCD about it. The x4's fine.

 

[Ericloewe]

Generally. A PCIe 2.0 lane is capable of about 500MBytes/sec (4Gbits/sec). So you've got 16Gbps of PCIe to the card. You can't ACTUALLY support running 10Gbps bidirectionally. However, your FreeNAS box is also probably incapable of putting that much traffic out unless you have an SSD based pool, so, my suggestion, don't get too OCD about it. The x4's fine.

That's actually 500MB/s in each direction, per lane, so 16Gb/s in each direction. So it should be plenty for a single 10GbE connection.

 

[jgreco]

Is it? Am I having a senior moment? I guess that could be, I've had no coffee so far today.

 

[BDMcGrew]

From purely a hardware perspective and in a Linux world totally unrelated to FreeNAS ... we ship our hardware and software with high end Dell Precision workstations and regularly use Intel i540-T2 (2x10GB Copper) cards in the x4 slots. We then deploy with a stand-alone cheap Netgear 10GB switch but have had a few customers trunk the 2 10GB connections with LACP on the fancy Cisco switches.

With winchester drives in RAID0 we were able to "keep up, good enough" but once we moved to SSD's, we were easily able to saturate a 20GB trunked connection and sustain those kinds of transfer rates.

Moral of the story, OS's aside, yes, the hardware will do it in a 4 rail slot but I'm jealous you've got fiber and I don't; it just sounds cooler :)

Good luck!

 

[Ericloewe]

Is it? Am I having a senior moment? I guess that could be, I've had no coffee so far today.

Well, PCI-e is a full-duplex interface, unlike conventional PCI or PCI-X and Wikipedia says you've got the per-lane speed right (I never remember the bandwidth of the various PCI-e versions...).

 

[jgreco]

Well, PCI-e is a full-duplex interface, unlike conventional PCI or PCI-X and Wikipedia says you've got the per-lane speed right (I never remember the bandwidth of the various PCI-e versions...).

I need to start my morning coffee. You're right, of course, I'm just still half asleep.

 

[jgreco]

but I'm jealous you've got fiber and I don't; it just sounds cooler :)

Looks cooler too.



All the purty aqua fiber.

 

[BigDave]

Is it? Am I having a senior moment? I guess that could be, I've had no coffee so far today.

Did your network capable coffee pot get confused with the time change?

Thanks for the replies. Enjoy your extra hour of Sunday...

 

[Ericloewe]

Did your network capable coffee pot get confused with the time change?
View attachment 9165
Thanks for the replies. Enjoy your extra hour of Sunday...

Our extra hour was last week, you insensitive clod!

 

[BigDave]

Our extra hour was last week, you insensitive clod

LMAO!

 

[BigDave]

Ok, I need some advice here on an SFP+ 10GbE switch purchase.

I was all excited to see Dell 6224 units going for cheap on ebay
until digging deeper and finding out the front facing fiber ports are 1GbE,
AND the SFP+ add-on moduals are $300 a piece

So back to researching I went and found this about Dell model N2024 through Wiki.

The N2000 is an Ethernet switch with limited IP capabilities. There are two 'sizes': 24 x 1GbaseT or 48 x 1GBaseT ports and each of them available as POE+ or standard switch. All models in the N20xx series can be stacked with other models in the same series. Although they do use same stacking-cables as the N3000 series it is NOT possible to stack N2000 with N3000 switches. All models come with 2 x 10Gbase SFP+ uplink ports and two 'twentigig' stacking ports at the back. Management can be done by assigning IP address to switch or one of the vlan-interfaces.[20] The N2000 is marketed as the follow-up for the legacy PowerConnect 5500 models as well as the Force10 S25 and S50 models.

So, I turned to eBay and found this one.
Looks nice with a one year warranty.
Am I barking up the right tree here, what do y'all think???

 

[anodos]

Our extra hour was last week, you insensitive clod!

It seems like Europe's ahead of us in everything these days. At least we still have our obesity rates.

 

[BigDave]

I'm gonna pick this switch after much research and thought. Dell Networking X1052
It's a newer generation product, better at saving power, lower wattage overall, has the four ports I
need and is new with warranty @ $607.05 direct from Dell's website.
It's gonna be a while before I can save up the funds, so feel free to voice your approval/disapproval,
I really would like some opinions on this. Thanks
Dave

 

[BigDave]

@jgreco I'm trying to buy a switch here, are you on vacation? ;)
I would like your feedback on my previous posts above ^^^^^^^^^^^^^^^^^

 

[mav@]

While discussing PCIe speeds, people forgot that Chelsio S310E-CR is quite old and supports only PCIe 1.x, that is 250MB/s per lane. So for that card 8x slot will provide about 16Gbit/s, that is enough for one 10GigE port, but from my real experience is not enough for two (gives about 12Gbit/s total). Putting this card into 4x slot will probably limit it to about 6-7Gbit/s.

 

[jgreco]

@jgreco I'm trying to buy a switch here, are you on vacation? ;)
I would like your feedback on my previous posts above ^^^^^^^^^^^^^^^^^

Yeah, the forumware/sysadmin/Dru/Jordan/whoever decided to blacklist me for a few weeks; I get more productive when I'm not skiving off answering posts here on this darn forum anyways.

 

[jgreco]

Ok, I need some advice here on an SFP+ 10GbE switch purchase.

I was all excited to see Dell 6224 units going for cheap on ebay
until digging deeper and finding out the front facing fiber ports are 1GbE,
AND the SFP+ add-on moduals are $300 a piece

Yeah, n00bs (sorry) seem to get a little confused/excited/etc about SFP ports, which of course look just like SFP+. I just pulled a pair of Dell 6024F's from the data center but it honestly isn't 48 ports of 10GbE

So back to researching I went and found this about Dell model N2024 through Wiki.

The N2000 is an Ethernet switch with limited IP capabilities. There are two 'sizes': 24 x 1GbaseT or 48 x 1GBaseT ports and each of them available as POE+ or standard switch. All models in the N20xx series can be stacked with other models in the same series. Although they do use same stacking-cables as the N3000 series it is NOT possible to stack N2000 with N3000 switches. All models come with 2 x 10Gbase SFP+ uplink ports and two 'twentigig' stacking ports at the back. Management can be done by assigning IP address to switch or one of the vlan-interfaces.[20] The N2000 is marketed as the follow-up for the legacy PowerConnect 5500 models as well as the Force10 S25 and S50 models.

So, I turned to eBay and found this one.
Looks nice with a one year warranty.
Am I barking up the right tree here, what do y'all think???

Yeah, that looks great. I don't think those were selling for anywhere near the ~$400 price point that the 5524 was at the time I wrote the 10G primer, but the basic idea's the same, and the use of a non-stupid stacking technology is very attractive.

Bear in mind that I don't have any direct experience with that switch or other members of its family, but I can say that over the years we've had good-to-great luck with Dell networking gear.

 

[jgreco]

I'm gonna pick this switch after much research and thought. Dell Networking X1052
It's a newer generation product, better at saving power, lower wattage overall, has the four ports I
need and is new with warranty @ $607.05 direct from Dell's website.
It's gonna be a while before I can save up the funds, so feel free to voice your approval/disapproval,
I really would like some opinions on this. Thanks
Dave

That may be fine as well, but if you look at expansion, you're somewhat limited. One of the reasons I really liked the 5524 (and by extension the N2024) is that you have 24 ports of 1G to two ports of 10G *PLUS* the ability to expand arbitrarily. The X1052 is 48 ports to four ports of 10G; if you want to expand, you need to use one or more of those four 10G ports. If you're very certain this will never be s problem, then I have no particular opinion beyond that.

 

[jgreco]

While discussing PCIe speeds, people forgot that Chelsio S310E-CR is quite old and supports only PCIe 1.x, that is 250MB/s per lane. So for that card 8x slot will provide about 16Gbit/s, that is enough for one 10GigE port, but from my real experience is not enough for two (gives about 12Gbit/s total). Putting this card into 4x slot will probably limit it to about 6-7Gbit/s.

That's a good point, though on the flip side, most of the people around here won't be exceeding those speeds