Poor Throughput - Help me Narrow Down Cause

[craftycraft]

Hi everyone,

I've spent the better part of the day reading the forum and gaining ideas for how to solve this (lots of great suggestions from cyberjock). I just built a new FreeNAS server and have been playing around with different configurations before migrating. I am having much trouble with transfer speeds on my new setup- average 4-7 MB/s transfer rate for CIFS shares over the network read and write.

Interestingly, I installed Windows 7 on this box earlier today just for fun and created a share- transfer speeds over the network were around 55 MB/s.

I am using a MSI 890GXM-G65 board, AMD FX-6100, 8GB memory (more on the way, had issues with TigerDirect order) and the onboard Realtek 8111DL.

This is my ifconfig output:

Code:

[root@freenas2 ~]# ifconfig                                                   
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500     
        options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC
>                                                                             
        ether 6c:62:6d:38:6d:76                                               
        inet 10.10.10.114 netmask 0xffffff00 broadcast 10.10.10.255           
        media: Ethernet autoselect (1000baseT <full-duplex,master>)           
        status: active                                                         
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384             
        options=3<RXCSUM,TXCSUM>                                               
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9                             
        inet6 ::1 prefixlen 128                                               
        inet 127.0.0.1 netmask 0xff000000                                     
        nd6 options=3<PERFORMNUD,ACCEPT_RTADV>  

Results of dd tests:

Code:

[root@freenas2 /mnt/test]# dd if=/dev/zero of=./testfile2 bs=1m count=10k     
10240+0 records in                                                             
10240+0 records out                                                           
10737418240 bytes transferred in 63.796683 secs (168306842 bytes/sec)         
[root@freenas2 /mnt/test]# dd if=./testfile2 of=/dev/null bs=1m               
10240+0 records in                                                             
10240+0 records out                                                           
10737418240 bytes transferred in 42.458355 secs (252892941 bytes/sec) 

And finally, results of iperf test from a wired Windows client:

Code:

Client connecting to 10.10.10.114, TCP port 5001
TCP window size: 0.01 MByte (default)
------------------------------------------------------------
[108] local 10.10.10.118 port 57670 connected with 10.10.10.114 port 5001
[ ID] Interval      Transfer    Bandwidth
[108]  0.0- 1.0 sec  6.27 MBytes  52.6 Mbits/sec
[108]  1.0- 2.0 sec  6.10 MBytes  51.2 Mbits/sec
[108]  2.0- 3.0 sec  6.42 MBytes  53.9 Mbits/sec
[108]  3.0- 4.0 sec  6.42 MBytes  53.9 Mbits/sec
[108]  4.0- 5.0 sec  6.45 MBytes  54.1 Mbits/sec
Done.

I've tested with several cat5e and cat6 cables and haven't seen any improvement. I've also tested with different clients... I've ensured that the clients are also connected at gigabit speeds.

I am hoping someone may be able to suggest more things I can try to narrow down the source of the problem. I'm at a loss currently and am ready to begin pulling my hair out.

Thanks guys!

 

[cyberjock]

Your dd benchmarks look fine.

But, your iperf is total crap. If you are running gigabit, you should be seeing about 900Mbits/sec. If you are running 100Mb, your speed should be in the 80-90Mbits/sec.

Are you using wifi? Those numbers look suspiciously like wifi speeds or powerline adapters, which are terrible choices for servers.

 

[craftycraft]

I was hoping to hear from you!

No sir. I was testing over the network drops in my home, but these tests were run with client and server joined directly to the router. I have no clue where to go from here.

 

[cyberjock]

Hmm.. I don't have any good recommendations then. Realtek is known as "Realsh*t" around here. They are know to just flat out suck major donkey balls. I'd definitely get an Intel NIC for your server as a good starting point. A second good point would be to put an Intel NIC in your primary desktop if it doesn't already have one.

Aside from that, I would check out your link speeds and see if they are 1Gb or not. Something is horribly wrong, and it is *definitely* network related. The question is whether the problem is a network card, network cable, or a switch, router, etc. I'd start looking at the network speeds reported on your machines and try to find one that sucks. Typically this stuff comes down to one of four things:

1. Some 1Gb link is 100Mb because of a bad cable or similar.
2. Someone has wifi or powerline adapters and they forgot they were in the data path.
3. Someone has a 100Mb device in their home and they totally forgot it's 100Mb.
4. Jumbo Frames and/or LACP was setup on hardware that it should not be setup on.

Even the best wifi and/or powerline adapters are total suicide for data speeds. That's why I mentioned it first. Your speeds are also close to the theoretical 54Mb that 802.11g is limited to(which makes me very suspicious). You might have one of the worst iperf results I've seen in a long time.

In any case, that's about all the advice I can give. Something is very wrong with your network configuration. You're kind of on your own to go and experiment and figure out where the bad link in the chain is.

Good luck!

 

[xcom]

On your windows client... Pay atention to your TCP Window Size:

Code:

TCP window size: 0.01 MByte (default)

That's a problem...

 

[craftycraft]

Thank you for all the suggestions. I will get to work on this again tomorrow and see what I can figure out. I'll also try another switch with only the server and client to see what that does.

I'll also take your suggestion of picking up an Intel NIC. What do you think of this one:
http://www.ebay.com/itm/INTEL-PRO-1...US_Internal_Network_Cards&hash=item2ed05e930b

Thanks again for your thoughtful reply. I hope I can get this thing figured out!

 

[cyberjock]

That NIC is exactly what I use in my desktops. ;)

 

[xcom]

Thank you for all the suggestions. I will get to work on this again tomorrow and see what I can figure out. I'll also try another switch with only the server and client to see what that does.

I'll also take your suggestion of picking up an Intel NIC. What do you think of this one:
http://www.ebay.com/itm/INTEL-PRO-1...US_Internal_Network_Cards&hash=item2ed05e930b

Thanks again for your thoughtful reply. I hope I can get this thing figured out!

Before you go out and buy hardware try to change your TCP Window Size on your iperf client using the -w switch... To something like 1024K or at least try to match it to your server TCP Window Size...
Post your results.

Good luck.

 

[gpsguy]

Assuming both the client and server are gigabit ethernet, run a network cable between them and take the router out of the equation. Try your tests again.

 

[c32767a]

On your windows client... Pay atention to your TCP Window Size:

Code:

TCP window size: 0.01 MByte (default)

That's a problem...

+1

This smells like a "Network performance Optimizer" visited his TCP stack settings.