The Good NICs.

[praecorloth]

Hey guys. A couple of friends and I are getting ready to test the ever loving shit out of FreeNAS to see if we can tie down some specific dos and don'ts for Samba. One of the things that usually come up in slow Samba transfer threads is "are you using a decent NIC?"

Frankly, I'm skeptical on this one. I understand that many NICs are like the Win Modems of old. But modern processors should be more than powerful enough to handle gigabit traffic, I think. But who knows. I've been wrong before.

What I want to know is what is a good NIC? I'd like to nab two PCI NICs and two PCI express NICs for testing, and since I'm doing this for shits and giggles, I need them to be as inexpensive as possible. Obviously we're trying to break away from the $15 Realteks. But if I could avoid spending $100 per NIC, that would be awesome.

 

[noee]

Fwiw, I have three wired machines to my FreeNAS box (all Win7Ult hosting Vbox w/Mint and PC-BSD) and I ended up putting one of these in each after I got the first one for testing:

Intel Pro 1000/GT

RUnning 8.3 and using the other CIFS "tweaks" from the other threads, GbE is just about saturated. Solid. Usage is backups, media playback (audio/video), general doc storage (dev projects, etc.).

 

[jgreco]

A good NIC is one that:

1) supports polling
2) supports vlans
3) supports jumbo frames well
4) supports large numbers of rx/tx descriptors (as in, the em-based 82542/82543 only support 256)
5) is configured for large numbers of rx/tx descriptors (not sure if em has been fixed to handle this)
6) supports checksum offloading
7) supports coalescing

mmm... well anyways cards supporting these things generally have the potential to be high-performance cards. Many of the Intel desktop cards are better than some of the competition's server cards (well, at least that's the way it used to be). You have to remember that a gigabit is still, relatively speaking, a LOT of data to be throwing around, and the difference between cards, and also different configurations of the SAME card, will have a great impact. If your machine is busy getting slammed by unreal levels of interrupts, it's going to go slower, for example. Yes, machines are faster today, and that can make one hell of a difference as well, but a fast machine with a boat anchor network card is going to drag, at least somewhat, compared to a fast machine with a server grade card that's intelligently configured. Samba is *extremely* sensitive to hardware speed because of its architecture.

For PCI-X, maybe grab some PWLA8492MT's off eBay. They're older generation but solid gigabit cards. DUAL gigabit cards. IBM makes a version that's selling for $15 on eBay. Pretty sure that's the MT, only so much work I'll do for you. ;-) And for PCI-e, IBM makes that too, cheap on eBay too but I always end up having to pull up the Intel docs to decipher the more recent ethernet cards as to which supports what and why, so this is in no way an endorsement of either of these two cards, except to say that both would work far better than a Realtek.

 

[paleoN]

For the PCIe I've had seen good performance with an Intel 9301 CT based NIC. That does use the em driver however.

Whatever you decide I would go with Intel based card for the superior cross-OS driver support.

 

[cyberjock]

Hey guys. A couple of friends and I are getting ready to test the ever loving shit out of FreeNAS to see if we can tie down some specific dos and don'ts for Samba. One of the things that usually come up in slow Samba transfer threads is "are you using a decent NIC?"

Frankly, I'm skeptical on this one. I understand that many NICs are like the Win Modems of old. But modern processors should be more than powerful enough to handle gigabit traffic, I think. But who knows. I've been wrong before.

What I want to know is what is a good NIC? I'd like to nab two PCI NICs and two PCI express NICs for testing, and since I'm doing this for shits and giggles, I need them to be as inexpensive as possible. Obviously we're trying to break away from the $15 Realteks. But if I could avoid spending $100 per NIC, that would be awesome.

I understand all of your skepticism regarding a NIC being a NIC. I shared it.. until about 2 years ago. I had seen some people over the years swear by Intel NICs. I always lol'd at them because I always used the onboard NICs and was always "happy" with the performance. Then after getting all i7s in the house, all Gb cabling and hardware, upgraded all the machines to SSDs, I was still getting only 60-80MB/sec to and from my server. I thought that surely my computers should be plenty powerful enough to do more. I tried jumbo frames, tweaking all sorts of settings in the network settings. I spent many hours reading forums looking for people that had registry tweaks. I could never get more than about 80MB/sec, even when copying to and from ramdrives across my network. I dismissed anyone that mentioned Intel NICs because it couldn't matter, right?

Well, I finally decided that I had wasted enough of my time and I'd try Intel NICs before I just give up and accepted the fact that I was getting all the speed I would ever get. I spent something like $125 on Intel NICs for all of my machines. Installed them all and.... then I saw triple digit speeds. I have no clue what makes Intel NICs so good. Could be the drivers.. could be the hardware.. could be both. The only thing I know is that if I want outstanding performance and great cross-OS support I always use an Intel NIC. If I'm building a computer for someone, I stick with the built-in. But for my machines I always go with Intel now.

I wouldn't have believed it if I hadn't seen it myself. Heck, I didn't believe it until I saw it myself.

 

[jgreco]

Actually you're probably fine with Broadcom (bge) too, popular on AMD/Opteron stuff. We were pretty much an all-AMD shop for years here but have seen some rather distressing failures (Broadcom/Tyan's fault, not AMD directly) and examples of Broadcom chips that worked a bit subpar. We ditched Tyan for HP. The HP gear has been rock solid, but it's lots of watts, and with the advent of Sandy Bridge, we started swinging back to Intel stuff built in our shop because it seemed like the best way to get low power, high reliability gear that also managed to meet compatibility requirements for both VMware and FreeBSD. So we're also heavy on Intel ethernet now.

Anyways, Intel cards are NOT a magic bullet. Over the years, there have been some good gotchas with them. The primary difference being that Intel actually maintains a team developing a FreeBSD driver, and since they come with lots of gear, they're also one of THE most deployed ethernet chipsets - at least from a FreeBSD/PC point of view.

noobsauce80's observation is pretty basic: a cheap network card has dodgy silicon and, often, dodgy drivers as well. The drivers in particular can suffer from "it-works-itis", which is the mentality of working on a product until it works. Working is not always the same thing as working correctly or working robustly or working efficiently, and Intel has put substantial effort in on all those fronts. They've paid attention to how their cards impact the processor, because an efficient network card makes their CPU's look better and everything runs faster for benchmarks (or, for that matter, for production use).

But the flip part of that is that you should match cards with the task, if possible. An Intel desktop card is best suited to a desktop, and a server card is best suited to a server. The server card has silicon that a desktop won't find useful and might even slow it down. The desktop card may lack the silicon for offloading and heavy concurrent loads that the server cards have. You want fast? Select your parts intelligently. You're more likely to get fast if you build with stuff that's designed to go fast.

And with everyone ditching their servers for "cloud computing", there's no damn excuse not to get some absolutely awesome cards and take advantage of it all.

 

[toddos]

For the PCIe I've had seen good performance with an Intel 9301 CT based NIC. That does use the em driver however.

Whatever you decide I would go with Intel based card for the superior cross-OS driver support.

Seconded. Intel 9310 PCIe NICs are the only ones I buy. I have 4 on my network right now across different machines and different OSes (Windows, Linux, FreeNAS)

 

[jgreco]

Useful related link: Intel Ethernet Desktop And Server Adapter Quick Reference (pdf)

The 9301 is the Intel CT Desktop, 82574L, which is about four years old at this point and still a pretty awesome choice. Lots of server motherboards use this for the onboard ethernets. However, it is aimed at the desktop market. If I was going to be adding a card to a server, it'd probably be the Intel Ethernet Server Adapter I340-T4. We use the T4's in some of our ESXi hosts with great success, and I'm pretty sure we determined that they worked fine with FreeBSD as well. Bonus, you can get them cheap on eBay. Looks like a two-port IBM version going for a hundred bucks...

 

[praecorloth]

Good stuff! Thank you everyone. I will make sure to report back with the test setups, everything we tried, and what the results were.

 

[jgreco]

Oh, and if you want your eyes to bleed and your head to split:

Totally Awesome Realtek-Based StarTech ST1000SPEX4 Why-The-Heck-Would-Anyone-Make-A-Realtek-4-Port Gigabit Ethernet Network Adapter Card

For the bargain price of $199!

 

[cyberjock]

Oh, and if you want your eyes to bleed and your head to split:

I got a good laugh at that comment right there!

 

[jgreco]

You know that scene in Indiana Jones where the Nazi melts? Now imagine that guy, screaming, "Why, why would you make this?"

And you'll have the perfect image to go with that thought.