slow network throughput on HP microserver

[knightwolf]

hey everyone I need a little help here. I will try to be as detailed as possible here.

current setup:
2x windows 7 pc's

connecting to:
Freenas 8.1 release
HP microserver
8gig ram
5x 2Tb hdds running ZFS raid5 (or raidz)

network:
billion 7800n router
all computers wired (no wireless in this issue) using CAT5e cables.

I am having issues with slow copy speeds either using FTP or CIFS from my windows 7 to the NAS. I get a strange bursting where utilisation will have short peaks of 30mb/sec and then drop to 0 ....back to 30 then drop to 0, this happens for the entire duration of the copy using CIFS. with FTP the rate is slightly higher but still get the same pattern. so the average transfer rate = 3-6mb/s which on a gig network is unbelievably slow. I can provide images if you like.

Some things I have tried.

1)Eliminating the router completely by direct cables and static ip addresses. - Same issue

2)replaced all cables - Same Issue

3)copied from windows 7 -> windows 7 computers - sustained transfer rate at 40mb/s so all was working ok and not issues at all

4)tried copy programs like teracopy to see if it was windows 7 - Same Issue

my Thoughts:
could possibly be the buffering size on the HP server is set too low. but the only instructions I can find on changing this is for freebsd and for ver 7. it seems (I may be wrong) that this is now automatically set. I did find instructions on how to change this for freenas 7 but not 8, am I right to assume this is now set to auto?

Possibly an issue with my network card in the Server. I did pick up an intel one today to try but it seems I got a PCI one and not a PCIe, Duh!

I have images, and even wireshark traces if that helps. it has been a very very long time since I have done any hands on with linux, will try anything but may need a little extra information as I am quite rusty.

any help would be greatly appreciated.

 

[ProtoSD]

Knightwolf,

I think @Milhouses post below might give you some insight to the problem. You might try using another NIC and disabling the onboard NIC.

http://forums.freenas.org/showthrea...up-a-FreeNAS-box&p=12673&viewfull=1#post12673

 

[Milhouse]

Knightwolf,

I think @Milhouses post below might give you some insight to the problem. You might try using another NIC and disabling the onboard NIC.

http://forums.freenas.org/showthrea...up-a-FreeNAS-box&p=12673&viewfull=1#post12673

Yes - the simplest way to test is with iperf on the FreeNAS (it's pre-installed in 8.0.1-RELEASE) and another iperf instance running on one of the Windows PCs (Windows binary here, instructions here).

An iperf test will allow you to determine the raw performance level of your network, and also ensure it is stable.

While running the iperf test, keep an eye on any messages being logged to /var/log/syslog on the FreeNAS - if you see the NIC resetting, your only option is to treat yourself to a new non-Broadcom NIC (I'd recommend Intel CT NICs, HP also come recommended).

 

[knightwolf]

Try `iperf --help' for more information.
[root@nas] ~# iperf -c 192.168.1.4
------------------------------------------------------------
Client connecting to 192.168.1.4, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
I couldn't find the syslog

but I was checking via the gui console.

Oct 13 20:10:41 nas avahi-daemon[1719]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Oct 13 20:10:50 nas ntpd[1512]: time reset +0.887392 s

is this still pointing to me needing a new network card?
Oct 13 20:26:10 nas ntpd[1512]: kernel time sync status change 2001


[ 3] local 192.168.1.3 port 42183 connected with 192.168.1.4 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 1.07 GBytes 919 Mbits/sec

 

[Milhouse]

I couldn't find the syslog

It's a file, in /var/log. To view it, try "cat /var/log/syslog". The entries you saw in the console are normal and nothing to worry about.

Your 10-second test looks OK, good performance, but I'd run it for longer - try the following on the client (runs the test for 60 seconds, reporting stats every 10 seconds, results shown in Megabytes):

Code:

iperf -c 192.168.1.4 -t 60 -i 10 -f M

 

[knightwolf]

when I do "cat /var/log/syslog" I get no such file or directory.



------------------------------------------------------------
Client connecting to 192.168.1.4, TCP port 5001
TCP window size: 0.03 MByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.3 port 31839 connected with 192.168.1.4 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 1072 MBytes 107 MBytes/sec
[ 3] 10.0-20.0 sec 984 MBytes 98.4 MBytes/sec
[ 3] 20.0-30.0 sec 1058 MBytes 106 MBytes/sec
[ 3] 30.0-40.0 sec 1051 MBytes 105 MBytes/sec
[ 3] 40.0-50.0 sec 1066 MBytes 107 MBytes/sec
[ 3] 50.0-60.0 sec 1012 MBytes 101 MBytes/sec
[ 3] 0.0-60.0 sec 6244 MBytes 104 MBytes/sec
[root@nas] /var/log# iperf -c 192.168.1.4 -t 120 -i 10 -f M
------------------------------------------------------------
Client connecting to 192.168.1.4, TCP port 5001
TCP window size: 0.03 MByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.3 port 27225 connected with 192.168.1.4 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 1081 MBytes 108 MBytes/sec
[ 3] 10.0-20.0 sec 1084 MBytes 108 MBytes/sec
[ 3] 20.0-30.0 sec 1104 MBytes 110 MBytes/sec
[ 3] 30.0-40.0 sec 1079 MBytes 108 MBytes/sec
[ 3] 40.0-50.0 sec 1035 MBytes 104 MBytes/sec
[ 3] 50.0-60.0 sec 1084 MBytes 108 MBytes/sec
[ 3] 60.0-70.0 sec 1060 MBytes 106 MBytes/sec
[ 3] 70.0-80.0 sec 1078 MBytes 108 MBytes/sec
[ 3] 80.0-90.0 sec 1060 MBytes 106 MBytes/sec
[ 3] 90.0-100.0 sec 1058 MBytes 106 MBytes/sec
[ 3] 100.0-110.0 sec 1102 MBytes 110 MBytes/sec
[ 3] 110.0-120.0 sec 1064 MBytes 106 MBytes/sec
[ 3] 0.0-120.0 sec 12891 MBytes 107 MBytes/sec

 

[Milhouse]

Maybe I'm going mad... is it syslog or messages? Unfortunately my FreeNAS is in the middle of running a diagnostic on a 2TB disk so will be busy for the rest of the day... :(

In terms of your raw network performance, it all looks very good and stable so not sure what else to suggest right now (when I tested two separate N36L's with their onboard NICs, iperf'ing from one to the other, both NICs would repeatedly cr@p out - can't remember if it was the client or the server though, I have a vague recollection that it only showed up in one direction).

 

[knightwolf]

ok now I am confused.

when you said it was the driver I was like yes that makes total sense. now after doing the iperf test it's telling the opposite. very strange

ok so I switched the serv/client and while the throughput is a lot less it is still sustained at a constant 30% network utilisation and does not drop to 0.


C:\>iperf -c 192.168.1.3 -t 500 -i 10 -f M
-----------------------------------------------------
Client connecting to 192.168.1.3, TCP port 5001
TCP window size: 0.01 MByte (default)
-----------------------------------------------------
[168] local 192.168.1.4 port 60541 connected with 192
[ ID] Interval Transfer Bandwidth
[168] 0.0-10.0 sec 317 MBytes 31.7 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 10.0-20.0 sec 325 MBytes 32.5 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 20.0-30.0 sec 323 MBytes 32.3 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 30.0-40.0 sec 326 MBytes 32.5 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 40.0-50.0 sec 325 MBytes 32.5 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 50.0-60.0 sec 325 MBytes 32.5 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 60.0-70.0 sec 328 MBytes 32.8 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 70.0-80.0 sec 311 MBytes 31.1 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 80.0-90.0 sec 318 MBytes 31.8 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 90.0-100.0 sec 325 MBytes 32.5 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 100.0-110.0 sec 326 MBytes 32.6 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 110.0-120.0 sec 326 MBytes 32.6 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 120.0-130.0 sec 327 MBytes 32.7 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 130.0-140.0 sec 326 MBytes 32.6 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 140.0-150.0 sec 325 MBytes 32.5 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 150.0-160.0 sec 328 MBytes 32.8 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 160.0-170.0 sec 330 MBytes 33.0 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 170.0-180.0 sec 330 MBytes 33.0 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 180.0-190.0 sec 316 MBytes 31.6 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 190.0-200.0 sec 325 MBytes 32.5 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 200.0-210.0 sec 329 MBytes 32.9 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 210.0-220.0 sec 336 MBytes 33.6 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 220.0-230.0 sec 337 MBytes 33.7 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 230.0-240.0 sec 337 MBytes 33.7 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 240.0-250.0 sec 338 MBytes 33.8 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 250.0-260.0 sec 338 MBytes 33.8 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 260.0-270.0 sec 337 MBytes 33.7 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 270.0-280.0 sec 337 MBytes 33.7 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 280.0-290.0 sec 337 MBytes 33.7 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 290.0-300.0 sec 337 MBytes 33.7 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 300.0-310.0 sec 340 MBytes 34.0 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 310.0-320.0 sec 336 MBytes 33.6 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 320.0-330.0 sec 335 MBytes 33.5 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 330.0-340.0 sec 338 MBytes 33.8 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 340.0-350.0 sec 337 MBytes 33.7 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 350.0-360.0 sec 337 MBytes 33.7 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 360.0-370.0 sec 338 MBytes 33.8 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 370.0-380.0 sec 338 MBytes 33.8 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 380.0-390.0 sec 338 MBytes 33.8 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 390.0-400.0 sec 338 MBytes 33.8 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 400.0-410.0 sec 338 MBytes 33.8 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 410.0-420.0 sec 339 MBytes 33.9 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 420.0-430.0 sec 339 MBytes 33.9 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 430.0-440.0 sec 338 MBytes 33.8 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 440.0-450.0 sec 338 MBytes 33.8 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 450.0-460.0 sec 338 MBytes 33.8 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 460.0-470.0 sec 333 MBytes 33.3 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 470.0-480.0 sec 337 MBytes 33.7 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 480.0-490.0 sec 323 MBytes 32.3 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 490.0-500.0 sec 329 MBytes 32.9 MBytes/sec
[ ID] Interval Transfer Bandwidth
[168] 0.0-500.0 sec 16571 MBytes 33.1 MBytes/sec

C:\>

 

[knightwolf]

yeh its messages.


Oct 13 20:10:34 nas kernel: ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
Oct 13 20:10:34 nas kernel: ada1: <ST2000DL003-9VT166 CC3C> ATA-8 SATA 3.x device
Oct 13 20:10:34 nas kernel: ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
Oct 13 20:10:34 nas kernel: ada1: Command Queueing enabled
Oct 13 20:10:34 nas kernel: ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
Oct 13 20:10:34 nas kernel: ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
Oct 13 20:10:34 nas kernel: ada2: <ST32000542AS CC34> ATA-8 SATA 2.x device
Oct 13 20:10:34 nas kernel: ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
Oct 13 20:10:34 nas kernel: ada2: Command Queueing enabled
Oct 13 20:10:34 nas kernel: ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
Oct 13 20:10:34 nas kernel: ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
Oct 13 20:10:34 nas kernel: ada3: <SAMSUNG HD204UI 1AQ10001> ATA-8 SATA 2.x device
Oct 13 20:10:34 nas kernel: ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
Oct 13 20:10:34 nas kernel: ada3: Command Queueing enabled
Oct 13 20:10:34 nas kernel: ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
Oct 13 20:10:34 nas kernel: ada4 at ata0 bus 0 scbus4 target 1 lun 0
Oct 13 20:10:34 nas kernel: ada4: <WDC WD20EARS-00MVWB0 51.0AB51> ATA-8 SATA 2.x device
Oct 13 20:10:34 nas kernel: ada4: 150.000MB/s transfers (SATA, UDMA6, PIO 8192bytes)
Oct 13 20:10:34 nas kernel: ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
Oct 13 20:10:34 nas kernel: SMP: AP CPU #1 Launched!
Oct 13 20:10:34 nas kernel: GEOM: da0s1: geometry does not match label (16h,63s != 255h,63s).
Oct 13 20:10:34 nas kernel: Trying to mount root from ufs:/dev/ufs/FreeNASs1a
Oct 13 20:10:34 nas kernel: ZFS filesystem version 4
Oct 13 20:10:34 nas kernel: ZFS storage pool version 15
Oct 13 20:10:34 nas root: /etc/rc: WARNING: failed precmd routine for vmware_guestd
Oct 13 20:10:40 nas ntpd[1511]: ntpd 4.2.4p5-a (1)
Oct 13 20:10:40 nas proftpd[1674]: 127.0.0.1 - ProFTPD 1.3.3e (maint) (built Mon Sep 19 2011 19:44:01 UTC) standalone mode STARTUP
Oct 13 20:10:41 nas avahi-daemon[1719]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Oct 13 20:10:50 nas ntpd[1512]: time reset +0.887392 s
Oct 13 20:26:10 nas ntpd[1512]: kernel time sync status change 2001
Oct 14 06:17:20 nas ntpd[1512]: kernel time sync status change 6001
Oct 14 06:25:53 nas ntpd[1512]: kernel time sync status change 2001

 

[roberto]

yeh its messages.

Oct 13 20:10:41 nas avahi-daemon[1719]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns!
Oct 13 20:10:50 nas ntpd[1512]: time reset +0.887392 s
Oct 13 20:26:10 nas ntpd[1512]: kernel time sync status change 2001
Oct 14 06:17:20 nas ntpd[1512]: kernel time sync status change 6001
Oct 14 06:25:53 nas ntpd[1512]: kernel time sync status change 2001

Hi knightwolf,

how did you solve the NIC trouble? I'm facing the same problem, and I'm thinking right now to change the onboad nic....

Thank you
roberto

 

[knightwolf]

Almost!!

I have made a lot of improvements and would be happy to help you out.

so far I have a new network card on order and should be here in a few days. until then I have made quite a lot of improvements.

first if you are using the 5th sata port (which I am) and I did also do the Bios update to allow full speed on the port, the thing I missed was right at the end of the instructions there is a Bios setting which needs to be set (don't have the link on me at the moment but I can get it for you tonight if needed) this then allows it to run at 300mb/s so for me 4 drives were running on 300mb/s and 1 on 150mb/s. with that fixed it made a big improvement.

second I did a ZFS 4k check and it was set to 512k. I rebuilt the pool using 4k and that made a major improvement.

now I can sustain 300-400mb/s transfer rate which is very acceptable. that is using either CIFS or FTP (FTP is slightly better) I found with FTP 2-4 simultaneous connections were best as once you go above this the transfer rate drops.

I have a bunch of tcp-ip tuning setting I plan on doing tonight to see if I can improve my through put before the new network card arrives. I will keep you posted.

so the network card is an intel CT gigabit PCIe card and I can post my results when it arrives.

 

[roberto]

Almost!!

I have made a lot of improvements and would be happy to help you out.

WoooW !!!!! Thank you so much for sharing! Tonight I'll post my smb.conf && loader.conf, in order to share what I'm using. I don't know how check the I/O speed of the disk (I think all are @300). Anyway, thank you again for make me happy :)

I'll wait your suggestions and then I'll buy the new disk plus the network card!!!!!
Grazie!!
Roberto

ps: did you try also Wake On Lan ?

 

[knightwolf]

No i haven't tried any wake on Lan settings. was trying to iron out my throughput setting first.

I would be very interested in your .conf settings..

the I/O disk speed appears in your message logs when you first boot up. you turn these on in the GUI via a tick box. also you can get this by SSHing in and running "cat /var/logs/messages" at the prompt

 

[knightwolf]

oh also here is the link with the Bios update and a bunch of really handy information

http://forums.overclockers.com.au/showthread.php?t=958208

 

[knightwolf]

ok so here is the last and important part of the Bios update which I missed


10. Press F10 to get into the bios and go to "chipset" then "SB Sata configuration" and set "SATA IDE combined mode" to disabled. Save and reboot.



found at http://www.avforums.com/forums/15113684-post355.html
http://www.avforums.com/forums/networking-nas/1429720-tims-7-24tb-tiddler-unraid-nas.html

 

[roberto]

WOW! I'll try everything, for sure!

As promised, I copied all my conf files, but unfortunately I copied it into msdos filesystem (usb-win-key) and now I've a lot of strange characters: "=«Írr¨¬e$íhžmâ{ëO<\é" I think something went wrong during copy. This evening I'll fix the issue (Italian time, GMT+1). Sorry for 1-day-delay, but thank you for all the links&tips. I really appreciate it!

Greetings from Milano!
Roberto

 

[roberto]

Here we're, in attach all my config&log files!

 

[golemcito]

Almost!!

I have made a lot of improvements and would be happy to help you out.

so far I have a new network card on order and should be here in a few days. until then I have made quite a lot of improvements.

first if you are using the 5th sata port (which I am) and I did also do the Bios update to allow full speed on the port, the thing I missed was right at the end of the instructions there is a Bios setting which needs to be set (don't have the link on me at the moment but I can get it for you tonight if needed) this then allows it to run at 300mb/s so for me 4 drives were running on 300mb/s and 1 on 150mb/s. with that fixed it made a big improvement.

second I did a ZFS 4k check and it was set to 512k. I rebuilt the pool using 4k and that made a major improvement.

now I can sustain 300-400mb/s transfer rate which is very acceptable. that is using either CIFS or FTP (FTP is slightly better) I found with FTP 2-4 simultaneous connections were best as once you go above this the transfer rate drops.

I have a bunch of tcp-ip tuning setting I plan on doing tonight to see if I can improve my through put before the new network card arrives. I will keep you posted.

so the network card is an intel CT gigabit PCIe card and I can post my results when it arrives.

What speeds do you get under cifs? I mean, reading and writing a big file from/to a win7 machine to the freenas.

Thanx

 

[roberto]

With my config file & with my hardware specs (1 microserver, 4 HD western digital green, 8GB ram) I reach only 110Mbit/s :(

 

[golemcito]

Did you change the network card?