FreeNAS 11.2: Mellanox ConnectX-5 and mlx5en driver support?

[phudson1442]

Hi all,

I'm looking at putting a Mellanox ConnectX-5 EN MCX516A-CCAT in a NVMe-based storage server.

The FreeBSD manual pages indicate that the mlx5en(4) driver has been included since 11.0
https://www.freebsd.org/cgi/man.cgi?query=mlx5en&sektion=4&manpath=freebsd-release-ports

There are instructions to compile the driver into the kernel and load it at boot time. Is FreeNAS already configured this way? or should I expect to have to do these steps myself?

I don't have a lot of FreeBSD experience, some Linux, but not any modification/compiling kernel. So just curious if the ConnectX-5 should work out-of-the-box or not?

Thank you all for your time!

-Paul

 

[SweetAndLow]

Should just work out of the box. If not you can't really do anything about it except file a bug to get it included in the next freenas update.

 

[jgreco]

Hi all,

I'm looking at putting a Mellanox ConnectX-5 EN MCX516A-CCAT in a NVMe-based storage server.

The FreeBSD manual pages indicate that the mlx5en(4) driver has been included since 11.0
https://www.freebsd.org/cgi/man.cgi?query=mlx5en&sektion=4&manpath=freebsd-release-ports

There are instructions to compile the driver into the kernel and load it at boot time. Is FreeNAS already configured this way? or should I expect to have to do these steps myself?

I don't have a lot of FreeBSD experience, some Linux, but not any modification/compiling kernel. So just curious if the ConnectX-5 should work out-of-the-box or not?

Thank you all for your time!

-Paul

It's possible that it'll work, but even if it does, it's quite likely to be a burr under your saddle, especially when doing updates, possibly more often than that.

Check out the 10G Networking Primer and it will give you some idea of the ins and outs of some of the various cards. The real problem that you're fighting is that a NAS is something that really needs 100.000+% problem-free operation, in a stressy environment. Intel, Chelsio, some others, have really good support, and enough people are using them that you can be relatively certain they'll work fine. And only specific cards from those vendors.

You (and I) might not like it, but the reality is that it's really hard to support cards unless the manufacturer does so directly. A number of devices, such as LSI HBA's and Intel ethernets, use vendor-provided drivers written by vendor employees with an intimate cross-platform knowledge of the ins and outs of the chips, including errata that mere mortals may not have access to. With many millions of driver-hours logged by thousands of FreeNAS users for these devices, we also know that they are rock stable.

When you decide to go off the golden path, you are essentially becoming your own guinea pig, and you really just never know exactly what's going to happen. You don't have the luxury of the experience of thousands of other users. If and when something random happens, you won't find a lot of help here in the forums, because no one has that hardware.

I am sure this is not what you want to hear, but I strongly disagree with @SweetAndLow 's assessment, and I want to make sure you understand that there is nontrivial risk of adverse adventures and significant hair loss. On the other hand, it's also possible it'll work just fine. It's just hard to know offhand.

 

[JustinClift]

> I am sure this is not what you want to hear, but I strongly disagree with @SweetAndLow 's assessment,

As a data point, the Mellanox FreeBSD drivers are generally written by Mellanox people. Either their direct staff, or experienced FreeBSD developers hired by them. ;)

The Mellanox ethernet drivers seem pretty stable, as that seems to receive the majority of their attention.

Their Infiniband ones though (for the same cards!), would better be described as "flaky". eg not really for production use, unless you're actually ok with debugging kernel crash dumps, reporting bugs upstream, testing out patches, (etc)

The ConnectX-5 drivers themselves look to be included in FreeNAS:

https://github.com/freenas/build/bl...ofiles/freenas/kernel/FREENAS.amd64#L196-L199

If you have the cards already then give them a go. They should be fine.

But if you don't have the cards already, and you need a sure thing, pick one of the "known good" options as @jgreco advised.

 

[Elliott]

I have a ConnectX-4 card that I want to try with 40GbE. This card is supposed to use mlx5 driver, but I am not sure what these results mean. It says "no carrier" even though the cable is connected.

Code:

# ifconfig mce0
mce0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
   options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 98:03:9b:91:bf:36
    hwaddr 98:03:9b:91:bf:36
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect <full-duplex,rxpause,txpause> (autoselect <full-duplex,rxpause>)
    status: no carrier

Here it appears no have loaded the wrong driver:

Code:

# kldstat -v | grep mlx
        378 mlx4en
        377 mlx4

But here it shows that the right driver is loaded:

Code:

# dmesg | grep mlx
mlx5_core0: <mlx5_core> mem 0xf6000000-0xf7ffffff at device 0.0 numa-domain 0 on pci10
mlx5_core0: INFO: firmware version: 12.23.1020
mlx5_core0: INFO: Module 0, status: unplugged
mlx5_core1: <mlx5_core> mem 0xf4000000-0xf5ffffff at device 0.1 numa-domain 0 on pci10
mlx5_core1: INFO: firmware version: 12.23.1020
mlx5_core1: INFO: Module 1, status: unplugged
mlx5_core1: INFO: Module 1, status: plugged and enabled

I'm pretty good with Linux but new to BSD. How can I find out what version of driver is installed, like modinfo?

 

[jgreco]

Driver versioning isn't really a big thing with FreeBSD. You don't really get to pick and choose, and are usually constrained to what was distributed with the system, unless you're willing to rebuild the kernel.

What you're seeing in the MLX driver output there is the card firmware version, not a driver version.

There may be some tweak that you need to apply to the card to get it into pure ethernet mode. I don't have any insight into that.

 

[Elliott]

Alright I discovered sysctl -a which shows I have mlx4_core version 3.4.1. That should be fine. I believe this model MCX414A-BCAT only does ethernet, no Infiniband option. I am not using a switch here, I just have it connected straight to another computer which also has a Mellanox card. Maybe 40G does not support this topology? I have to look into this. With copper, I know there is a difference between patch and straight cables, but I thought fiber made no distinction.