SOLVED IBM NetXTREME II 10GBit (Fixed in FreeNAS 9.3)

[Norleif]

Here I was thinking I could get into the 10Gbit goodnes for cheap, getting myself a pair of these IBM NetXTREME II cards from eBay, including optics and everything...
I think they're simply re-badged Broadcom NetXtreme II BCM57710 cards, and should be supported by the bxe(4) driver.
I plugged one into each of my FreeNAS boxes (as I got about 9 TB I'd like to replicate from one to the other) and fired them up.
I didn't notice anything out of the ordinary when booting, and in the Network/Interfaces menu I had the option to add a new interface on bxe0 which I did, giving them static IPs in their own subnet.

Problem is, they don't respond to pings ether direction. Or anything else I've tried.
I can see the activity led on both cards blinking away in synch with the ping packets, but no response is ever received.

I've tested the cards in a pair of Windows7 machines and they work just fine. Throughput was limited to about 530MBytes/s by the Samsung 850PRO SSDs and not the NICs.

Also tried one card in Freenas and the other in Win7 but no joy (or ping reply).
One weird thing is the FreeNAS (bxe0 configured for DHCP) got an IP address from the Win7 box, but still no ping replies or anything else.
I can see the DHCP packets going back and forth in Microsoft Network Monitor (packet sniffer) and a bunch of ARP packets going back and forth, but they don't seem to be parsed...?

I'm not really big on UNIX/Linux but I can copy-and-paste shell commands as good as anyone, so I tried "sysctl -a | grep dev.bxe" as per the bxe(4) docs and got a screen full of stuff I don't understand, so I piped it into the attached text file hoping someone might get more out of it then me...
If you have more shell commands you'd want me to copy-and-paste just let me know.

Both FreeNAS boxes are running FreeNAS-9.2.1.8-RELEASE-x64 (e625626), one on a Intel(R) Pentium(R) CPU G620 @ 2.60GHz with 12GB RAM and the other on Intel(R) Xeon(R) CPU E3-1225 V2 @ 3.20GHz with 32GB RAM.

 

[cyberjock]

Hate to break it to you, but Broadcom is one of those that works fine... when it works. The problem is the proper Broadcom chips that *do* work are in very short supply and hard to find on ebay and such. Odds were definitely against you that the Broadcom card would work, which is why nobody in this forum has ever recommended it.

 

[Norleif]

Considering I could get 10 of these for the price of a decent Intel NIC, I just had to try... Right?
It's just so weird it works perfectly on Windows7 and not FreeNAS.
Would it be worth the effort to try them in FreeBSD 10 or 11, or are they using the same non-functional driver as FreeNAS?

This is slightly off topic but considering I got one half of a FreeNAS <-> Win7 10GBit multimode fiber link already, what kind of NICs should I put in the FreeNAS boxes to complete the link?
I haven't found any "recommended" or "known good" list of 10Gb NICs for FreeNAS. Did I miss something?
I want to use multimode fiber, so those Twinax DirectAttach-only or 10GBASE-T cards are out of the question.

 

[cyberjock]

The driver in FreeNAS 9.2.1.x is the same as the driver that was done for FreeBSD 9.2. You can try FreeBSD 10 and see if it works. If it does that's at least a glimmer of hope it will work in FreeNAS 10, but that's not expected until late summer 2015 at the earliest. You can definitely try FreeBSD 11, but since FreeNAS won't be based on it for 2+ years (they haven't even discussed it) I wouldn't bother trying it. :P

 

[Norleif]

Did some more testing on this one, and it seems this is a FreeBSD 9.2 issue, or at least FreeNAS and PC-BSD 9.2 issue.
Worked fine in PC-BSD 10.0-RELEASE, so I grabbed the latest FreeNAS 9.3 nightly (FreeNAS-9.3-M4-586004f-x64 at the time of writing), and it works reasonably well.
No tweaking and I can push about 430MByte/s from Windows7 to FreeNAS. Copied the file twice so I'd get it from Win7 read cache.
dd on the FreeNAS gets me 780MByte/s write speed so it might be possible to squeeze out a bit more, but I'm still happy with 4x increase over 1GbE

I'm not cool enough to run pre-releases of FreeNAS on both my primary and secondary storage boxes, so I bought a pair of what is advertised as Chelsio S320E-SR-XFP for 80 bucks each from eBay. For that price I'm not expecting any optics even if it's shown in the pics, but a bare S320E-LP-CR would hopefully work fine in 8.2.x using a passive twinax SPF+ cable.

 

[Norleif]

Small update regarding BCM57710 and FreeNAS 9.3:
Moved NICs from PCIe-4x to PCIe-8x slots + changed MTU to 9000 and got 9.92Gbit/s in iperf.
Writing to FTP and CIFS at about 570MByte/s.

 

[mjws00]

Very nice to know. Thanks.

 

[jgreco]

I'm not cool enough to run pre-releases of FreeNAS on both my primary and secondary storage boxes

I like this guy's attitude ;-)