Want to hop up on the 10 GbE train

[Elliot Dierksen]

I would say it is most likely the card, but it could be the optics too. Do you have a different slot for the Mellanox card? Is it possible that airflow isn't good and the card is overheating?

 

[IceBoosteR]

I would say it is most likely the card, but it could be the optics too. Do you have a different slot for the Mellanox card? Is it possible that airflow isn't good and the card is overheating?

Hi,
thanks for your answer.

The first freeze happens after ~4 minutes in operation, I would assume its not the heat. After several hours I would think so, but not under 5 Minutes without transfering huge amount of data...
Why would you guess is is maybe the optics? I would thinks it is like a normal Ethernet-cable: There is a link available - or not. Ic there is no link, no computer-system woulod freeze or crash...
-Ice

(Edit: The cable is ok I guess, as it is showing up as 10GBe and transferring data was also possible, but who knows :) )

 

[Elliot Dierksen]

You didn't say if it was an SFP+ card, or 10G Base-T. That was just a guess. I would say optics are less likely. Either a bad card, or something about the PCIe slot/BIOS options that don't play well together are the mostly likely causes. Again, the former seems the most likely.

 

[IceBoosteR]

You didn't say if it was an SFP+ card, or 10G Base-T. That was just a guess. I would say optics are less likely. Either a bad card, or something about the PCIe slot/BIOS options that don't play well together are the mostly likely causes. Again, the former seems the most likely.

Hi,
it is SFP+ using a DAC cable.

 

[Elliot Dierksen]

Which slot are you using for the Mellanox card? I see the slots for that motherboard are listed like this:

4 Slots,
1x16 (x16, G3)
1x16 (x4, G2)
1x1 (x1, G2)
1 x PCI Slot

The Mellanox card is PCIe 2.0 and x8 (product description says compatibility is "Fits x8 or x16 slots"), so that would means that the (x16, G3) which could be a problem if that is where you have your SAS controller. Are you using the built in Intel Rapid Storage Controller?

 

[IceBoosteR]

Which slot are you using for the Mellanox card? I see the slots for that motherboard are listed like this:

4 Slots,
1x16 (x16, G3)
1x16 (x4, G2)
1x1 (x1, G2)
1 x PCI Slot

The Mellanox card is PCIe 2.0 and x8 (product description says compatibility is "Fits x8 or x16 slots"), so that would means that the (x16, G3) which could be a problem if that is where you have your SAS controller. Are you using the built in Intel Rapid Storage Controller?

Hi,

thanks for the answer. Before I put something in my running "prod" system, I do tests first. Therefore I have an AMD system just for software and hardware testing. It's an APU (CPU with integreated graphics) and a Full ATX board. I have taken one of the PCIe 2.0 x16 ports, which are only x8 electrical (so it is identical as the card and the port in my server).
So I am not using the Rapid storage controller. But to get back to your question, yes I have the SAS controller in the x16 slot and would like o put the Mellanox card in the second x16 slot. I have contacted the seller, maybe I can send it back. I think about buying a Mellanox ConnectX-3 instead.
-Ice

 

[Elliot Dierksen]

You should keep in mind when you select a 10G card that your 2nd slot in the production may be x16 physically, but is x4 electrically. Sometimes it makes a difference what kind of 10G card your system/BIOS likes. That is why I am using older Cisco 10G cards in some of my servers. I had Chelsio's there, but the BIOS always put the fans on high which made them obnoxiously loud. What kind of 10G cards does Dell sell with that server? That would probably be some good guidance.

 

[Chris Moore]

1x16 (x16, G3)
1x16 (x4, G2)

The x4 slot is probably not capable enough to drive that 10Gb Mellanox card at full speed. Because, as you pointed out, it is PCIe 2.0 and x8, which means it was likely designed to use the full complement of 8 PCIe lanes to move it's data and will, most likely, only operate at half speed with just half the lanes connected. I use the Mellanox cards myself, but I put them in x8 slots when I do.

 

[Johnnie Black]

An x4 PCIe 2.0 slot works and has enough bandwidth for a single port Connectx-2, it can bottleneck a dual port NIC during simultaneous use.

 

[IceBoosteR]

I have now received the ConnectX-3. I will try to get it to work in my test system. Bandwidth should be ok, 10Gbit are around 1,25GB/s max! where PCIe 2.0x4 provided 2GB/s max.

 

[IceBoosteR]

Card is now stable for hours, but I have a problem on the switch maybe. There are two SFP+ ports. Both works with the ConnectX-2 and it worked at first also with the new card. But at the moment, I do not get a lin on port 28. On port 27 which is identical (10Gbit SFP+) is it working. Do I have a faulty port now? ://

Edit: Okey thats strange. Port 27 works fine with Windows and Freenas. Port 28 is only working with Windows. So its not faulty, but weird!

 

[Chris Moore]

Hi Chris,
thanks for your input.
That particular switch is not available where I live

Sorry that you can't get it, but I thought I would post my results here so anyone that was involved in the previous conversation would get the update.
I purchased one of these switches:
https://www.ebay.com/itm/Aruba-S250...ASE-T-4x1000BASE-X-10GBASE-X-KMJ/232504836333
I could not get it to connect to computers by means of the SFP+ ports, only to other switches, using the two models of copper DAC cables I had. I updated the firmware to the latest version that is available, and that appears to have corrected he problem with the switch being unwilling to communicate with a computer. I needed to remove the 10Gb ports from the stacking group and update the firmware and I couldn't use the Cisco DAC cables I have. It may work with other brands, but it will still not work with the Cisco cables. It does work with the Arista Netwoks branded cables I have and I will be trying some other configurations soon with fiber transceivers from Fiberstore and Intel as well as some DAC cables from another vendor or two. I will let you know how it goes. Time invested was the biggest cost here. The switch was only $125 and free shipping but I had to dig through a lot of documentation and even join their forum and ask some questions before I got what I needed.
I will be doing some speed tests later to see if there is an improvement over the home-brew switch I have been using. One thing I noticed immediately, it has a lot less heat output.