Cheap 10Gb?

[David Dyer-Bennet]

Just noticed some very cheap refurb. cards on Newegg.com. $40 each.

They're Mellanox ConnectX-2 10GbE PCIe 2.0 x8 Standard Height Network Interface Card, MNPA19-XTR with Cisco 3 Meter 10Gb Copper Twinax Cable SFP-H10GB-CU3M 37-0961-03.

So -- PCIe x8 whereas the simple 10Gb copper cards I've seen (in the $300 range) are x4, but on every board I've seen there are x1 slots and x16 slots, some of which only perform at x8, which ends up meaning that x8 doesn't seem to burn any more scarce resources in slot allocation than x4 does.

If read that card right, it's SFP+ socket -- so you can put in a range of adapters to do either optical or copper ethernet out of them.

But what's really interesting -- that "copper twinax cable" that's included. From the photos, it looks like that cable goes directly into the SFP+ maybe? I'm kind of hoping that two of those cards and one cable would let one connect two systems (only) at 10Gb very cheaply.

There's also the question of whether FreeBSD would support that card.

I haven't worked with this level of real server hardware, so I don't understand it. What do people think, am I just confused, or is there some chance this might work? I'm currently waiting for too much data to copy at 1Gb, and it's distressing (plus the network connection is failing every hour or two for unknown reasons, without leaving any log entries, which is also distressing, but that's another, ongoing, story).

 

[depasseg]

I'm not sure if freenas has the mellanox drivers (check the 10GbE networking primer). But yes the copper twinax cables are, for simplicity, the equivalent of a fiber pair with an SFP+ on each end.

 

[Mirfster]

Older article, but it says that the Mellanox ConnectX-2 Cards are NOT supported (http://www.servethehome.com/cheap-10gb-sfp-19-mellanox-connectx-2-cards/):

Practically speaking, there are still a few examples of OSes without ConnectX-2 support. Prime examples are FreeBSD 9.3 based FreeNAS and NAS4Free versions which did not have built-in support for the cards.

Don't take it as gospel though I only did a little looking...

 

[Mlovelace]

I don't see the mellanox included in the freenas kernel but you could always download the freenas source, install the drivers and build the updated mellanox freenas. It's not as hard as it sounds. You can do it inside virtualbox.

 

[Mirfster]

I have the Chelsio 110-1088-30 10gb 2 port cards. Got two of them for $100.00 on ebay a while back. Still can find some out there for around $40 - $50 each. Just make sure the one you order has the correct bracket (half-height or full-size depending on your case).

Examples:
http://www.ebay.com/itm/Chelsio-10G...751046?hash=item5d5ef015c6:g:GcsAAOSw7hRWQQkL
http://www.ebay.com/itm/OEM-Chelsio...813086?hash=item280ffb349e:g:ISQAAOSwKtlWkz8j

 

[David Dyer-Bennet]

Older article,

Is it an older article? When I view it, it says Sept. 21, 2015, which is only a few months ago, sadly.

 

[Mirfster]

Okay, maybe its not that old. But, technically it is "older".. ;)

 

[danb35]

OTOH, the Chelsio S310E-CR, while an older card, is well-supported on FreeNAS, and dirt cheap ($18 each in this listing).

 

[David Dyer-Bennet]

well-supported on FreeNAS

I suppose a modern card might be supporting 45Gb and such. The 3x (successor to the card I started asking about) does I think, or maybe that's the 4x. It's so nice to have technology move on and have things like that wash up on the beach.

Okay, okay, I just bought two of the Chelsio from that link.

You wouldn't happen to know what kind of cabling is necessary for a card-to-card connection between them would you?

 

[danb35]

Something like this would be simplest. Or you could get SFP+ optics for each card and a fiber patch cable.

 

[David Dyer-Bennet]

Good, I thought that was the case, but realized I hadn't checked before committing. At least short-term, this is primarily for speeding up moving data from old to new servers, and I have no existing 10Gb infrastructure and don't expect to upgrade too soon. (Okay, one version of reality does have me putting 10Gb in my desktop and the server that sits next to it, to give me better access to files there. But that's, again, essentially a standalone link with no other 10G in sight.)

 

[depasseg]

Keep in mind that old/underpowered machines might not be able to handle 10gbe sustained transfer speeds.

 

[David Dyer-Bennet]

Yep. SSH with ARCFOUR is running one CPU maxed out, and best wire traffic is 600Mb/sec. So one thing I know is I'll need to ditch encryption to get a significant improvement (or I could have parallel connections on separate parts of the filesystems; in fact I could benefit from a second one of those right now) (or build ssh with the "none" encryption option).

 

[David Dyer-Bennet]

you could always download the freenas source, install the drivers and build the updated mellanox freenas. It's not as hard as it sounds. You can do it inside virtualbox.

I should probably work towards having that option, if not specifically for Mellanox drivers, then in general. I used to do a lot of building of various products on Linux, Solaris, xBSD, and even OFS/1 or whatever that things was called, but that was in the late 80s / early 90s. I moved to Linux binary distributions and did a lot less with the other OSes, and have gotten lazy.

I'm a software developer by day, and often the one who ends up having to dig into the build process on our products, so it's not unfamiliar territory entirely.

 

[drwoodcomb]

Did anyone ever figure out if Mellanox Connectx-2 cards are supported in FreeNAS? More specifically model MNPA19-XTR?

I bought a pair of these thinking they were supported for some reason but I'm not 100% sure.

I will say that I put one into my FreeNAS machine running FreeNAS 9.10-STABLE and noticed something that was a little confusing


[root@freenas] ~# dmesg | grep -i 'Mellanox'
mlx4_core: Initializing mlx4_core: Mellanox ConnectX VPI driver v2.1.6 (Jan 11 2017)


It looks like FreeNAS initialized the driver automatically but I am concerned because it says VPI driver.

The card I have is Ethernet only and not Ethernet/Infiniband. I am not experienced at all with FreeNAS or all that advanced of a computer user so I am unsure what to think. Is this driver ok? I guess I'd feel much better if this driver was called 'Mellanox ConnextX2 EN'

Even better still would be if my computer magically outputed a message that said 'Don't panic, its all good, enjoy your 10 gigabit speeds'

 

[formgen99]

I put one in my 9.10 box today and it appears to be functioning correctly within freenas.


Sent from my iPhone using Tapatalk

 

[drwoodcomb]

Im glad to hear it. I put one in yesterday and it looks like it might work but Im still waiting on my sfp+ modules and my optical cable.

What sfp+ transeivers are you using if you dont mind me asking. And are they working well?

Also, are you connecting this to a windows 10 PC? I installed the windows 8.1 Driver because Mrllanox doesnt have a windows 10 driver.

In order to update the firmware on both cards I had to do so in my freenas server because the windows version of the firmware tool (mlxup) kept saying "no device detected" even though windows 10 could detect the card and device manager said the card was working fine. I was curious to know if you had this problem also by any chance.

 

[formgen99]

Im glad to hear it. I put one in yesterday and it looks like it might work but Im still waiting on my sfp+ modules and my optical cable.

What sfp+ transeivers are you using if you dont mind me asking. And are they working well?

Also, are you connecting this to a windows 10 PC? I installed the windows 8.1 Driver because Mrllanox doesnt have a windows 10 driver.

In order to update the firmware on both cards I had to do so in my freenas server because the windows version of the firmware tool (mlxup) kept saying "no device detected" even though windows 10 could detect the card and device manager said the card was working fine. I was curious to know if you had this problem also by any chance.

I didn't update the cards, they worked out of the wrapper. My cards are in freenas, esxi 6.5, and xenserver 7.1. No Windows boxes. I'm using them for storage network and vmotion. Getting 820MB/s on my iscsi network.


Sent from my iPhone using Tapatalk

 

[TechLobo]

formgen99 - just to be clear you have the MNPA19-XTR working fine with esxi6.5, as well as freenas 9.10? Native support - no fiddling needed?

Just want to check as I am thinking of getting a couple of these to put in my (whitebox) ESXi 6.5 boxes, to provide 10Gb connectivity to my Freenas box - 10Gb newbie :)

 

[TechLobo]

Was thinking of using the Mellanox MHQH29C-XTR 10GbE dual port in the Freenas box (I understand that 9.10 and 10 have native support for the Mellanox cards). Anyone got any feedback on this card - or am I better of sticking with a Chelsio 110-1099-30?