What is my limiting factor for network reads / writes?

[Joshua Parker Ruehlig]

Hey guys! I've been benchmarking my FreeNAS device and everything seems fantastic
I was wondering though what are my limiting factors for network performance?

SPECS____________
FreeNAS 8.0.2 64 bit
CPU - e350
NIC - Realtek 8111E
RAM - 8GB
ZFS VOLUME - (5xHD204UI - RAIDZ1) + (40GB X25V - L2ARC)

(Remote System) Ubuntu Server 11.10 64bit
CPU - e350
NIC - Realtek 8111E
RAM - 8GB

(Switch)
TRENDnet 8-Port Unmanaged Gigabit GREENnet Standard Switch
_______________________________
Local Benchmark (Ram > ZFS Volume)
write - 260MB/s
read - 300MB/s

Networked Benchmark (Ram > Switch > ZFS Volume) - Ubuntu Webserver mounts FreeNAS NFS share
write - 65MB/s = 520Mb/s
read - 80MB/s = 640Mb/s
_______________________________

Why am I not gettng full Gigabit speeds ( 125MB/s=1Gb/s ) when I have gigabit nics and a gigabit switch?

My possiblilites are..
*Realtek 8111E NIC under FreeBSD 8.0.2
*Realtek 8111E NIC under Ubuntu 11.10
*Gigabit Switch
*NFS Mount Options

Things I might try with your recommendations..
*Better Gigabit Switch
*NFS mount options
*Intel PCI NIC for my FreeNAS box ( PWLA8391GT? )
*Intel PCI-e x4 NIC for my Ubuntu box (don't want to have to use this slot though...)

Thanks!

 

[tropic]

Hola. I actually think you're doing well with the equipment you're using.

I ran FreeNAS 8 Release on a similar setup for a while: Asus E35M1-I / AMD E-350 APU / 8GB RAM / Intel 82572EI / 5 x HD204UI in RAID-z. Cheap Netgear GS108Tv2 gigabit switch and 9K jumbo frames enabled everywhere on the LAN. I hit the same ceiling you did.

I tried tweaking from the UI, tuning the transmit buffers on a few different models of Intel nics, setting up an LACP LAG and running concurrent transfers, etc. I was never able to saturate a single gigabit pipe. I'm not a fan of Realtek nics in production environments, but the Intel nics didn't do any better than the onboard RTL8111E. Concurrent transfers to/from separate LAN machines wouldn't even add up to a single maxed-out gigabit connection.

I threw in the towel, pulled the motherboard and replaced it with an Intel DH67BL, Core i7-2600K and 16GB RAM. Speed problems solved with zero tweaking. Out of curiosity I replaced the motherboard again with a Gigabyte GA-Z68M-D2H so I could see if the RTL8111E was the culprit. Nope. I downgraded to a Core i5-2500 and 8GB RAM... still nice and fast.

I settled on the machine in my sig as my everyday home NAS. I think the E-350 was the bottleneck in the original box, and the 8GB/single channel memory setup probably didn't help.

The only thing scarier than a hardware guy firing up putty is a software guy with a screwdriver, so take this with a grain of salt. I think you'd be better served being happy with your moderately zippy energy-saving rig than by upgrading your nics and network hardware to find that you're hitting the same old bottleneck. If you really need those extra mbps, you might consider putting more horses under the hood or reducing overhead by using UFS instead of ZFS.

 

[survive]

Hi Joshua,

Try using "iperf" to measure your network throughput. It's installed by default in 8.0.2.

It's going to be pretty rare\only under ideal circumstances that you will ever be able to run your gig-e connection full bore, so don't be to upset if you can't reach that speed.

-Will

 

[Joshua Parker Ruehlig]

Thanks for the info, so cpu is likely the limiting factor, didn't think of that. Well I'm more than happy with my setup and just know if I were to have this NAS in a more mission critical place I would need a xeon board.

Guess I'll pass on the intel NIC.

I'm gonna try to get 2x8GB sticks of RAM in this and see if it works. I read on the ASRock e350 mobo specs that it can take 16GB of ram, but I'm skeptical. (Though I have the Asus e350 board) If it does work I'll post on here so others can upgrade.

 

[Joshua Parker Ruehlig]

Thanks for the info, so cpu is likely the limiting factor, didn't think of that. Well I'm more than happy with my setup and just know if I were to have this NAS in a more mission critical place I would need a xeon board.

Guess I'll pass on the intel NIC.

I'm gonna try to get 2x8GB sticks of RAM in this and see if it works. I read on the ASRock e350 mobo specs that it can take 16GB of ram, but I'm skeptical. (Though I have the Asus e350 board) If it does work I'll post on here so others can upgrade.

2x8GB of RAM worked in both my e-350 motherboards. These make great ZFS boxes, wish I bought 2 sets of oit when the rebate was still valid.
Here's the ram I got http://www.newegg.com/Product/Product.aspx?Item=N82E16820220619