40Gb Mellanox card setup

[JustinClift]

This sounded relevant, and yep I was changing around cables while the Win7 box booted up which may have caused it to go into IPoIB mode instead of proper Ethernet.

So I turned off the Win7 box (disconnected from the wall, just to make sure), then made sure the cable to the FreeNAS box was connected ok and powered things back on.

This time, the card came up in proper Ethernet mode. Strangely, it's not seeing the FreeNAS box on the other side but that sounds like a problem for investigating later (getting very sleepy here atm).

@yurelle... it's probably worth giving that a shot too. eg power off the box completely (eg from the wall), + make the sure cable is in place before turning things back on. That way it should autodetect ethernet mode when it starts up.

 

[JustinClift]

Ugh. Just tried to update the Mellanox driver to version 5.35, just to see if there's any difference.

The driver update doesn't work (for me) as it complains that MS KB3033929 is missing and refuses to proceed. When trying to manually apply that KB though (downloaded direct from MS KB website)... the KB says it's already installed. WTF? Gah...

I'll try some other different versions too, as the driver quality for these Mellanox Windows drivers doesn't seem inspiring. :(

 

[JustinClift]

Progress made. It looks like the autodetect-wrongly thing really does come from not having the Win7 10GbE card fully cabled up before booting. Even so, it seems unreliable (eg occasional misdetection), so I'm super surprised that Mellanox has removed the option from the Device Manager dialog.

With the amount of other options available for fine tuning, it seems bizarre to take away the most important option. Maybe that's just me being too tired though. ;)

@yurelle, when you get a chance try the above approach. eg make sure your Windows box is connected to the FreeNAS box / switch / whatever-goes-on-the-other-end-of-your-cable before turning on the power. Hopefully that fixes the problem. :)

 

[yurelle]

Thank you, I'll try that. Right now, I'm waiting on an rsync to finish on the ZFS pool (in Fedora), before I can reboot into FreeNAS. It's been running for 3 days syncing many TB of data. It looks like it should finish sometime tonight. Depending upon how late it finishes, I might have to wait until tomorrow after work, to run your test. Thank you for your potential solution. I'll check it ASAP.

-Yurelle

 

[JustinClift]

No worries. Found a better solution anyway. :)

The "mlxconfig" utility that comes with the Mellanox Firmware Tools (WinMFT) can be used to set the port type.

Tested here using an older version of WinMFT (v2.7.2) downloaded from the Mellanox website, as some of the newer WinMFT versions only work with newer cards.

First, generate the port configuration settings text file:


> mlxconfig --gen_cfg port_config.txt



It's a simple 3 line thing that could be created using a standard text editor instead:

[PortProtocolConfig]
Port1Protocol = VPI;
Port2Protocol = VPI;

Change those VPI entries to "ETH" instead, then use mlxconfig to update the card with the new values:


> mst status
MST devices:
-----------
mt26428_pciconf0
mt26428_pci_cr0

> mlxconfig --device mt26428_pciconf0 --cfg port_config.txt
...
-I- Configuration was written to flash successfully



Then reboot and you should be good. :)

 

[yurelle]

I apologize, I got help up yesterday. I'm just trying this now.

I followed your instructions, and verified installation of v2.7.2, and everything appears to have completed successfully; mlxconfig claimed that the changes were written to flash successfully, but when I rebooted, nothing changed. It still doesn't work. I have the FreeNAS box running, and it was running when I rebooted the windows 10 box. But I'm not getting any link lights on the cards, and The Windows Device Manager & Network Adapter Settings pages, both still report the card as being "IPoIB", not ethernet. I also tried spinning up OpenSM on the windows box, but that didn't help. It just reports the port is down.

-Yurelle

 

[JustinClift]

Yeah, I've been finding the same problem here too over the last few days. It doesn't matter which version of the Mellanox drivers are installed - I've tried every release available from the Mellanox site - it's still pretty random which mode it starts in.

As you mention, mlxconfig indeed claims the ports are set to Eth mode, so that setting is definitely "on". It's just completely friggin ignored by the idiotic drivers when they start up.

* I've also tried disabling the IPoIB versions of the adapter and forcibly updating them to use the Ethernet drivers. That _generally_ results in a BSOD. (but not always)

* Also tried creating custom (aka OEM) firmware to force ethernet mode or at least make it the default. Creating custom firmware isn't hard, but I have no idea what setting needs to be changed. All of the one's I've tried (thus far, mostly copied from ConnectX2 EN adapters) either don't make any useful difference... or screw up the card in general (thus needing flashing back to something working).

It's really frustrating, as these cards work well in RHEL/CentOS 7 (my primary desktop OS atm). There, it's just a matter of changing the port type in a config file and voila... it all just works.

 

[yurelle]

Also tried creating custom (aka OEM) firmware to force ethernet mode or at least make it the default. Creating custom firmware isn't hard

What?! Writing custom firmware isn't hard? Teach me your ways! lol I'm actually a programmer myself, and I've wanted to dabble in firmware & drivers, but I never knew where to start. I always assumed it would be really difficult without some kind of reference document explaining all the specific hardware's internal stuff, like interrupts, custom instruction set, etc. I never thought I'd hear someone say off-hand that it was easy-squeeze.

It's really frustrating, as these cards work well in RHEL/CentOS 7 (my primary desktop OS atm). There, it's just a matter of changing the port type in a config file and voila... it all just works.

Yea, after struggling with this for a couple weeks, I was surprised how easy Fedora picked it up & worked no problem. Even the UEFI installer had the cards up & running, before the OS was even installed.

I appreciate your help, I'm not sure what else there is to try. I guess I'm going to stick with Fedora with a non-native ZFS pool for now. It's not perfect, but it works, for now. The more we fiddle with things, I'm getting more uneasy that any solution we do find may be unstable or hard to maintain/replicate.

-Yurelle

 

[JustinClift]

What?! Writing custom firmware isn't hard?

Well... for Mellanox cards - to the level I was meaning :) - it's not difficult. Mellanox provides a binary blob to OEM's like HP, Dell, etc (and can be found online) which just needs merging with a card specific settings (.ini) file. The result is the binary (.bin) firmware file people burn to their cards.

I was just meaning generating the firmware file by trying to figure out appropriate settings to merge with it for forcing Eth mode. Wasn't meaning generating the *actual* binary code that runs on the Mellanox chips. ;)

If it's something you're interested in having a go at as well, I can point you at the right bits and give you the steps. Really isn't hard.

If you do manage to figure out how to force Eth mode, that'd be really useful. :D

I appreciate your help, I'm not sure what else there is to try.

Maybe look for an Intel or Chelsio card to go in the Windows box? Mellanox cards are ok for *nix based systems, but obviously have driver-sanity issues in Windows. :( I'd kind of expect Intel to have their shit together for their 10/40/etc GbE cards. :)

 

[yurelle]

Well... for Mellanox cards - to the level I was meaning :) - it's not difficult. Mellanox provides a binary blob to OEM's like HP, Dell, etc (and can be found online) which just needs merging with a card specific settings (.ini) file. The result is the binary (.bin) firmware file people burn to their cards.

Oh, that makes sense. That's not as complicated as I was thinking.

If it's something you're interested in having a go at as well, I can point you at the right bits and give you the steps. Really isn't hard.

Sure, that's be cool. I probably won't do much with it now, but it seems like it might be a good tool to have in the belt for the future. Do most manufacturers release binaries like that, or is that something unique to Mellanox?

I've wanted to do super low level stuff like that for a while, but never gotten around to it. I've looked into writing a bootloader from scratch, and maybe a simple toy OS, which seem like they'd be fun; but I haven't done it yet. Maybe that should be my belated New Year's resolution. :D

If you do manage to figure out how to force Eth mode, that'd be really useful. :D

Definitely. If I ultimately get it working, I'll post the steps here.

Maybe look for an Intel or Chelsio card to go in the Windows box? Mellanox cards are ok for *nix based systems, but obviously have driver-sanity issues in Windows. :( I'd kind of expect Intel to have their crap together for their 10/40/etc GbE cards. :)

I have a couple QLogic cards, which I bought before knowing that QLogic was acquired by Intel, and the drivers are virtually non-existent; the only one I could find was for Solaris. So much for it being impossible to delete something from the internet.

Thanks again to everyone,
-Yurelle

 

[JustinClift]

Oh, this looks like it might work instead:

https://forums.servethehome.com/ind...nnectx-2-how-to-disable-port-autosense.13437/

I don't have the Windows box around any more for testing it with though. :(

With the custom Mellanox firmware, I started writing up a full guide but that began to take serious time (which is better spent elsewhere). Instead, I'll point at some of the existing info that started me on custom mellanox firmware burning:

• Dell C6100 and Windows Server 2012 no RDMA
  
• https://www.servethehome.com/custom-firmware-mellanox-oem-infiniband-rdma-windows-server-2012/

With the .mlx (XML, with the firmware bytecode dump near the end) and .ini files (settings for each card), take a look through them with a text editor. There's some pretty interesting fields in there, and it's useful to know that our cards have a jumper on them that can be closed in order to reflash from a "I screwed it up" state. Note - I've never managed to screw up a card that bad though, and I've applied some *really* out of left field settings to my cards at times "just to see where the limits are". So, shouldn't be too dangerous. :)

 

[JustinClift]

I have a couple QLogic cards, which I bought before knowing that QLogic was acquired by Intel, and the drivers are virtually non-existent; the only one I could find was for Solaris. So much for it being impossible to delete something from the internet.

Yeah, Mellanox does the same thing with their older drivers too, on purpose, to force people to buy newer cards. It's why I'm happy to recommend Mellanox gear off of Ebay for people doing (cheap) dev/test/lab builds, but for any real build needing a production quality solution I'd personally recommend non-Mellanox.

Just because it's never a good idea to reward a company for treating their users poorly.

 

[yurelle]

It's why I'm happy to recommend Mellanox gear off of Ebay for people doing (cheap) dev/test/lab builds, but for any real build needing a production quality solution I'd personally recommend non-Mellanox.

What are some good non-Mellanox brands? I'm pretty new to this, so I don't know "Good off-brand" from "Cheap-Knockoff", and I'm not really sure where to find out. Do the non-Mellanox brands have a better price too, or is is just a principle thing (which I'm not knocking, by the way. I totally get sticking it to the man ). I get the impression that Mellanox is sort of the IBM/Intel of Infiniband, but I'm not sure who the equivalent AMD would be.

 

[JustinClift]

Oh, this looks like it might work instead:

https://forums.servethehome.com/ind...nnectx-2-how-to-disable-port-autosense.13437/

I don't have the Windows box around any more for testing it with though. :(

I had time to try this out with a different Win 7 box today. And yep, it worked. There's an entry for the adapter in the "System Devices" group in Device Manager too (wasn't obvious at first). That's the one with the port selection setting in it, and bootup was consistent in Windows when that was set (for both IB and Ethernet).

What are some good non-Mellanox brands?

That's covered (briefly) in the Hardware Recommendations PDF, and in more depth in the 10 Gig Networking Primer. :)

 

[JustinClift]

Doing a quick search on Ebay.com (as you're in the US) just now for "chelsio 10Gb/s" shows up a few Chelsio 320E-SR dual port 10GbE adapters for (roughly) $US50 including shipping. That doesn't seem too bad. :)

Oh, here's one for ~$US30. Note that I have no relationship to the seller (etc). Just saw the card on Ebay, and that's it. ;)

 

[yurelle]

That's covered (briefly) in the Hardware Recommendations PDF, and in more depth in the 10 Gig Networking Primer. :)

None of those mention Infiniband. I didn't know the manufacturer pool would be the same between Infiniband & regular ethernet. I figured infiniband would be a more specialized thing with different companies being "good quality" in those spaces.

I had time to try this out with a different Win 7 box today. And yep, it worked. There's an entry for the adapter in the "System Devices" group in Device Manager too (wasn't obvious at first). That's the one with the port selection setting in it, and bootup was consistent in Windows when that was set (for both IB and Ethernet).

Looks like that did it. Thanks!
-Yurelle

--------
Steps:
--------

Device Manager > System Devices > Mellanox ConnectX VPI ... > [Right Click] > Properties

Mine was set like this by default:

Uncheck "HW Defaults", and then choose "ETH" to switch to ethernet mode. Rebooting the machine probably required.

 

[JustinClift]

None of those mention Infiniband. I didn't know the manufacturer pool would be the same between Infiniband & regular ethernet. I figured infiniband would be a more specialized thing with different companies being "good quality" in those spaces.

Ahhh sorry. My misunderstanding. :)

FreeNAS (without customising) doesn't support Infiniband at all. It's just that many modern Mellanox cards can operate natively in either Infiniband or Ethernet mode. So, having the drivers for the Ethernet side of things for Mellanox cards in FreeNAS lets that bit work.

There are Infiniband drivers in FreeBSD, that with a bit of mucking around can be included in FreeNAS if you're willing to go through the compile process. Compiling FreeNAS isn't difficult.

Unless you're wanting to run software that speaks Infiniband natively though (rather than going over IPoIB), it's not worth the effort. The FreeBSD IPoIB drivers apparently aren't well optimised, unlike on Linux where they are.

So, running stuff over IPoIB via Infiniband on FreeBSD/FreeNAS apparently (not personally tested) doesn't get throughput much greater than 1GB/s. eg close to 10GbE speeds. So, "not worth the effort" for most situations. :)

 

[kdragon75]

I've got two Mellanox 40Gb cards working, with FreeNAS 10. One in server, one in a Windows 10 PC. Getting between 400 MB/s to 700 MB/s transfer rates.

Hardware:
2 x MHQH19B-XTR Mellanox InfiniBand QSFP Single Port 40Gbps PCI-E, from eBay for $70.
1 x Mellanox MC2210130-001 Passive Copper Cable ETH 40GbE 40Gb/s QSFP 1m for $52

I had to set both to Ethernet mode, as I couldn't get Infiniband working; FreeNAS 10 has a system Tunable for sys.device.mlx4_core0.mlx4_port0 = eth.

That's it! Thought it might be useful for anybody else investigating these cards.

I have never played with Infiniband. I should give it a shot. That said I have 8gb fiber channel and get about the same speeds on my ESXi hosted VMs.

 

[yurelle]

I have 8gb fiber channel and get about the same speeds on my ESXi hosted VMs.

I'm pretty sure he was bottlenecked by drive IO, and just showed those transfer speeds to indicate that the network was not acting as a bottleneck. That said, I'm running those cards , and I just did a RAM disk to RAM disk transfer from one machine to another across the line (in IPoIB mode from Windows to Linux; running through a Samba network share), and was getting 1.5 GB/s (i.e. 12 Gb/s) of actual throughput.

Running

Code:

dd if=/dev/zero <to local RAM disk>

tops out at 2.56 GB/s, so there does seem to be a network bottleneck there, but not sure if that's samba or the cards.

-Yurelle

 

[JustinClift]

> ... so there does seem to be a network bottleneck there, but not sure if that's samba or the cards.

It might also be driver overhead, and/or other stuff that can be adjusted with tuning.

That's one of those "probably time consuming to find out" things though. ;)