slow network throughput on intel 82574L

[virtual.ste]

Goodevening
Sorry for may bad English!
I'm new to forum but old freenas user.
I have a supermicro atom board X7SPA-HF with D525 processor with 2 embedded Intel nic 82574L. My problem is that i can't get more than 500-600 MBit/sec with iperf test on gigabit connection.
I tried many things read on internet; disable autoselect and forcing to 1000 mbp/s full duplex, disable advanced power management on bios board, direct cable connection to bypass switch and modify following parameters:

net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.sendbuf_max=16777216
hw.em.rxd=4096
hw.em.txd=4096

I think it's software or driver issue because i tried to install ubuntu 12.04 on usb disk and, with iperf test, i can get around 900 MBit/sec without any changes.
I'm using FREENAS 9.2 x64 but i tried 9.1.1 release and NAS4FREE 9.1 with same dropped bandwidth.

ifconfig on related nic
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
ether 00:25:90:38:2e:6c
inet 192.168.1.11 netmask 0xffffff00 broadcast 192.168.1.255
inet6 fe80::225:90ff:fe38:2e6c%em0 prefixlen 64 scopeid 0x5
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active

Anyone of you has the same nic controller?
If so, how can i get more throughput?

Thanks

 

[cyberjock]

I have that network card on my server. They work great out of the box. To be honest, there's 3 things that make me cringe...

1. You have an Atom. Those are not and never will be performance kings. They are meant to be low power. They are almost a joke today as you can buy Haswell Xeons that idle at around the same wattage as the old Atoms. You're also limited to a max of 4GB or 8GB of RAM. Neither of which is "very much RAM" for ZFS. Even at 8GB, you are at the minimum called for in the FreeNAS manual. At 4GB you are at 1/2 of the minimum, and you should not and cannot expect miracles if you are so far below the minimum recommended RAM.
2. Your buf_max is... HUGE! I doubt you are doing yourself any favors with a size like that. Sure, it did some amazing things back in the day, but you aren't dealing with an archaic 7.0 or 8.0 FreeBSD build. I think both of those are bad ideas. Not sure about the hw.em.Xxd setting. But to be honest, if you look at #3, I'd wager its probably not doing you any favors.
3. FreeNAS should work out of the box with very good performance. If you are having to tweak it significantly you are probably doing something wrong.

Intel NICs are the cream of the crop for network performance. What you are seeing is likely the result of your hardware not being able to handle the load. We've been telling people for years not to use Atoms as they don't use ECC RAM(very dangerous with ZFS) and they aren't performance kings. They generally are very slow, very laggy, and have serious limitations to what you can do with them.

 

[virtual.ste]

Thanks for your quick response!
I understand that using Atoms board don't give best performance but i'm trying to have good compromise between performance and power consumption for home use nas...
What do you think about Avoton motherboard? I know that is however an Atom but support ECC ram and has more powerful processor than D525
I will consider a Hasweel system as you suggested

 

[eraser]

My problem is that i can't get more than 500-600 MBit/sec with iperf test on gigabit connection.

Rerun your iperf tests but add "-w 256K" to the iperf command line on both your iperf client and iperf server. I'll bet you see better iperf results.

 

[virtual.ste]

I tried with 256k, same result!

 

[eraser]

I tried with 256k, same result!

• What brand of NIC card does your other system have? Verify that all "offload" settings are enabled in your NIC. (they are automatically enabled in FreeNAS, but I don't know what Operating System you are running on the other computer).
• Bypass the switch by connecting a cable directly between your iperf server and client.
• Please remove any hard coded speed/duplex settings you set. Gigabit links should always* be configured to autodetect speed/duplex.

Then please run your test again and post the iperf output from both the client and the server.
Thanks!

*98% of the time this is the case, and the 2% where it might need to be hard coded is not the case here.

 

[eraser]

Actually I re-read your previous post and see that you ARE able to get the good iperf speeds when running a different OS on the same hardware. That is an excellent data point and something I skipped over the first time I read this post.

I mainly wanted to see the output from the iperf test to confirm that the -w 256k parameter was set on both sides. I'm sure that you did do that properly.

So this hints that the problem might have something to do with either the underlying FreeBSD operating system or with the customizations that FreeNAS puts on top of it. If you have time, can you try installing FreeBSD 9.2 (instead of FreeNAS) onto your supermicro X7SPA-HF system and test with iperf again (with the larger -w size)? That would tell us if the problem is with FreeBSD or with FreeNAS.

 

[eraser]

Goodevening
I have a supermicro atom board X7SPA-HF with D525 processor

On a side note, I read that the X7SPA-HF board supports a maximum of 4GB RAM. If that is still true then you may not be able to take full advantage of the ZFS filesystem (since prefetching is disabled by default on systems with less than 4 GB available RAM).

Many people on this forum will refuse to help troubleshoot ZFS performance on systems with less than 6-8 GB of RAM. Something to keep in mind.

But maybe you are planning to use the UFS filesystem instead of ZFS... if that is the case then you can ignore this reply.

 

[virtual.ste]

Thanks eraser.
I was thinking to do an iperf test with freebsd 9.2..
I will try asap as well as iperf test with your advice.

 

[virtual.ste]

On a side note, I read that the X7SPA-HF board supports a maximum of 4GB RAM. If that is still true then you may not be able to take full advantage of the ZFS filesystem (since prefetching is disabled by default on systems with less than 4 GB available RAM).

Many people on this forum will refuse to help troubleshoot ZFS performance on systems with less than 6-8 GB of RAM. Something to keep in mind.

But maybe you are planning to use the UFS filesystem instead of ZFS... if that is the case then you can ignore this reply.

I found an old thread
http://forums.freenas.org/threads/my-new-nas-box-running-on-8-0-2.1310/
that say X7SPA-HF-D525 can support 8 gb ram but operating at 800Mhz...
My system can use 6 GB ram so theoretically i can have a decent performance!

 

[eraser]

One more thought. When you tested speeds with the ubuntu installed did you use the same ethernet port on your motherboard as you did when doing iperf testing with FreeNAS?

I ask because my older desktop motherboard (while not a SuperMicro) also has two built in NICs. Turns out that one NIC is wired through an internal PCIe connection while the other is wired through a slower PCI connection. I get better performance from the built-in NIC that is wired through the PCIe connection.

Not sure if that is still done these days, but something that just popped into my head.

 

[virtual.ste]

One more thought. When you tested speeds with the ubuntu installed did you use the same ethernet port on your motherboard as you did when doing iperf testing with FreeNAS.

Yes, same interface