Chelsio T580-CR QSFP+ interface for FreeNAS as a choice...

[TrumanHW]

Is this considered a good NIC..? In the same way the T520-SR is..? Also -- is it an 'efficient' card (wattage & heat I guess) ?

Or is that only a question for Switches..? Or would I be better off looking at an SFP28 // 25GbE SFP+..?
I'm thinking of creating an NVMe array and looking for ways to use it..
Also (germane to the issue) ... does Thunderbolt (3) work in FreeNAS (my MB has a TB3 port) ...

Thanks

 

[TrumanHW]

MORE QUESTIONS:
Would I be better off getting 28SFP instead of QSFP+ ...? They're certainly more expensive. Is there a link between being more expensive and being more desirable in some way..?

A lot of these 28GbE (or are they 25??) have less heatsink material or even none ... than my 10GbE ... is that essentially why? Are they that much more efficient that they can still stay cool (and thus, is why they're more than an ostensibly faster NIC..?)

Is there a difference between these two items..? Does ACAT mean something or is that a generic name..? Is there a difference between CX4121A and an MCX4121A..? Or "Lx EN" ..? It's not easy to find the naming convention // product decoder...

• MELLANOX CX4121A ConnectX-4 Lx EN
• MELLANOX MCX4121A-ACAT ConnectX-4

25GBE / SFP28 1 PORT

• MELLANOX ConnectX-4 Lx EN
• MELLANOX MCX4111A-ACAT EN

25GBE / SFP28 2 PORT

• Intel XXV710 DA2
• Broadcom BCM957404 57404
• MELLANOX CX4121A ConnectX-4 Lx EN
• MELLANOX MCX4121A-ACAT ConnectX-4

 

[Elliot Dierksen]

It depends on what kind of switch you are using. I am using T580's to connect to a Cisco Nexus 3064 which has 4 QSFP ports. That works great for me. I can get over 39G throughput using iperf between the two hosts. I am using optical cables for two reason. The fiber patch cords are easier to route through the server cable management arms, and it is easier to make sure each side of the link has transceivers that it likes. Anecdotal evidence only, but so far the 40G cards appear to be less particular about optics than 10G cards. YMMV.

 

[TrumanHW]

Yeah, I'd heard that about QSFP+ (perhaps from you in another thread, which, thank you for the consistent help)... when asking about that as a potential cause for sllooow as SHEEIT performance of my FreeNAS system; which I found out was me being duped by dedupe!!! :-D

But yea, the drives in other tests in RAID-5 (cldn't do RAIDzX as it was in Windows) got 7.5GB/s sustained R/W ... not bits, GBytes. So def would prefer if not 28 or 40 ... QSFP100 ... but also want quiet where possible. IS that a more likely result of increased efficiency on that generation..?

 

[Elliot Dierksen]

So def would prefer if not 28 or 40 ... QSFP100 ... but also want quiet where possible. IS that a more likely result of increased efficiency on that generation..?

I haven't touched anything in excess of 40G so far, so I don't have anything useful to contribute there. This is somewhat anecdotal, but look at taller devices if noise matters. The fans in a 1U device have to turn faster than the larger ones in a 2U device to move the same amount of air. That isn't a 100% thing, but I switched to 2U servers instead of 1U for that reason. It does seem to be mostly holding true so far. My experience with the Nexus line in general is their fans tend to be loud, high pitched, and obnoxious. The 1U 3064's I am using are louder than the 2U HP's I used to use, but they are tolerable. It would be impossible to have anything in the 5K line in your office and work effectively without hearing protection. All IMHO, of course.

 

[TrumanHW]

haha, hearing protection. yeah.

Makes sense -- it's what annoys me about the thinner and thinner laptops MO of apple, used to excuse integrating soldered components that fail more often -- all to give us something that only they say we want. I personally LOVED the Lenovo W series, which were beefy, had thick ass heatsinks, and never once broke a sweat. But, how could apple charge $2700 for an 8TB SSD if they did that??? Or $1200 for 128GB of RAM that on the 366th day would result in a VERY expensive repair were something to fail.

Anyway, thanks for all of the help. I guess the last question left is ...

why SFP28 is MORE expensive than QSFP40 .... ?
Unless that's relates to my prev. question re: temperatures (efficiency) which you're (understandably) unsure of.

Thanks

 

[TrumanHW]

Oh! one more Q ... what's open ethernet..? Worth caring about..?

Given how much more pricy they are, unless an SFP28 comes around again, I'll either stick with QSFP+ or peer-to-peer connection ...

 

[Elliot Dierksen]

why SFP28 is MORE expensive than QSFP40 .... ?

I don't know. I suspect because there isn't as much gear/demand for it. SFP+ goes to 10G, SFP28 goes to 25G, but QSFP goes to 40G. It sounds to me like SFP28 has been superseded by QSFP, but the is just my guess.