Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

Enabling Mellanox network drivers

copeland3300

Newbie
Joined
Aug 4, 2014
Messages
13
Hey all.
I hope this is the right place to put this, if it's not, mods, please move where appropriate.

I've been trying to add some network drivers for a Mellanox ConnectX2 card I have. I've been successful in adding them to the FreeBSD 9.3 kernel and making everything work, but I can't seem to get it working in FreeNAS. They seem to be in the /usr/src for FreeBSD 9.3, so I'm not sure where the issue comes up.

I've followed the directions on the github page, and after running a "make checkout" I go into "build/nanobsd-cfg/FREENAS.amd64" and add the lines that work in FreeBSD. Then run the "make release" and then "make update". The lines added are here:

makeoptions WITH_MK_OFED=yes
options OFED
device mlx4ib
device mthca
device mlxen

It builds the update file and the iso without throwing an error. The problem is when I either apply the update file through the UI or when I try to boot from the iso, I get a Trap 12 error and I'm left at a dgb> prompt. Adding any of the lines that I've listed above seem to cause it not to boot. This happens both on a VM and on my physical NAS, so I've ruled out hardware being the likely issue.

BTW, if I don't make any of the kernel config changes from above, and build everything stock, the update file and the iso work just fine.

Any help would be greatly appreciated!!!

Thanks!
 

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,780
If the network card you have is not supported natively in FreeNAS, then you are using the wrong network card (or the wrong OS). Installing anything in the base FreeNAS system is a mistake.
 

copeland3300

Newbie
Joined
Aug 4, 2014
Messages
13
If the network card you have is not supported natively in FreeNAS, then you are using the wrong network card (or the wrong OS). Installing anything in the base FreeNAS system is a mistake.
I understand what you're saying and realize this is outside of the supported scope of FreeNAS, but given that with a little tweaking the card works with FreeBSD, I'd think I could get it working I FreeNAS as well.
 

Robert Trevellyan

Pony Wrangler
Joined
May 16, 2014
Messages
3,780
I'd think I could get it working I FreeNAS as well.
You probably can, but that doesn't mean it's a good idea. The refrain you'll hear over and over in these forums is "FreeNAS is an appliance, do not attempt to modify it."

Either way, what you're attempting is beyond my expertise.
 
Joined
Oct 2, 2015
Messages
4
@copeland3300, have you made a feature request at bugs.freenas.org for adding Mellanox ConnectX2 drivers in the freenas 9.3.x kernel? If not, I will have to do it, since I really need them.

In fact there is one - https://bugs.pcbsd.org/issues/2014 - for OFED support, but it is not yet clear if it will happen.
 
Last edited:

copeland3300

Newbie
Joined
Aug 4, 2014
Messages
13
@copeland3300, have you made a feature request at bugs.freenas.org for adding Mellanox ConnectX2 drivers in the freenas 9.3.x kernel? If not, I will have to do it, since I really need them.

In fact there is one - https://bugs.pcbsd.org/issues/2014 - for OFED support, but it is not yet clear if it will happen.
Hey. I didn't make the feature request, as I didn't really want to wait around for months (or years apparently) for support to get added.

I've actually just moved over to FreeBSD 10.2 and manually do everything I need to do or use Webmin for the rest. The cards work without an issue, and regularly see over 1GB/s (yes a gigabyte per second). I also didn't find any kernel tweaks to get me better speed than the default values, but to each their own.

I find that I'm having to do the same amount of work this way as I was with FreeNAS, which is to say, once it's set up, it's done, and I don't frequently have to go back and mess with settings. I've also learned a TON about how this stuff actually works, which has made troubleshooting a little easier.

Sorry this probably isn't the answer you wanted, but I figured it might help you make your decisions.
 
Last edited:
Joined
Oct 2, 2015
Messages
4
Last edited:

JustinClift

FreeNAS Experienced
Joined
Apr 24, 2016
Messages
282
Thanks @Emil Stoyanov, that's good info. I haven't yet gotten around to updating the instructions for 11.0... too many other higher priority things to do. So, it's very good news to see it's working. :D
 
Joined
Aug 26, 2014
Messages
7
Last edited:

JustinClift

FreeNAS Experienced
Joined
Apr 24, 2016
Messages
282
@Daemonic Do you have time to give this .iso a go?


It's standard FreeNAS 9.10 using latest source code from a few hours ago, with this (new, trimmed) patch applied:


What it does (compared to stock FreeNAS 9.10 ISO):
  • Adds the Mellanox IB drivers
  • Adds the IB commands to the install
  • For ConnectX (series 1→4) cards, it hard codes port 1 to be Infiniband, and port 2 to be Ethernet mode (as per your email ;)).
There's another ISO in the same release built from the same source code, but with both ports set to IB mode.

Hoping you have time to give at least the mixed port one a go, and report back what does/doesn't work for you (etc). :)
 
Last edited:
Top