Freenas 8.02 - network pausing for a few seconds on file transfers

[5teve]

Hi Guys

Complete newbie here, never touched linux or any kind of operating system other than windows so please go easy on me!

I have just bought an HP n40l and installed freeNAS 8.02 on it.. very pleasantly surprised how easy it was to get running. A good start...

I currently have only 1 disk (the 250gb provided) but have increased the ram to 8gb, no bios hack yet but its coming. I have 5 x 2tb disks on their way but for now have just been testing with the one disk

I had read that there were issues with the onboard NIC and while testing copying several ISO files or about 4gig a piece i noticed the transfer pause for several seconds then continue. It starts off at approx 100MB/s (according to windows 7) but pauses several times along the way. Just the broadcom issue i thought (as i have no such issues between windows machines)

so like a good newbie i went and bought an Intel CT 1/2 height card. Installed that and deleted the integrated NIC from freeNAS.

Tested again ... and same issue... almost same profile.. long pauses, then continues..

So what i'm asking is what can cause this? is it just the single drive? do i need to 'tune' zfs for the 8gb ram i have? do i need to disable the integrated NIC in the bios?

If you have got any suggestions i would greatly appreciate them, and the detail of how to do them also (and make them permanent). I have puTTY installed so can hook in via ssh (do i sound like i know what i am talking about?!)

Thanks

Steve

 

[jfr2006]

Hi:

If it makes you feel better, i'm also experiencing a similar situation. However, mine only occurs when writing to freenas...When reading there are no problems. Here is a picture from a widget network meter showing that:





I think the problem in my case is lack of power from the cpu, so i'm switching to an Athlon II 270u. I will post the new results.

 

[jgreco]

I'm not exactly sure what the underlying issues are, but this seems to be a ZFS performance thing. It gets kind of cruddy especially on slower hardware. I see approximately what you're talking about on an N36L. I have some older single-core storage servers that perform worse; it seems like ZFS will eat your CPU while doing its thing, so my guess is multicore++.

I've actually just got some nice Supermicro Xeon E3's in shop and have been playing with them, haven't gotten to playing with FreeNAS on them yet though.

 

[5teve]

Thanks Guys.. good to know that i'm not alone... but bad that we are experiencing it..

be good if there was a nice guide for tuning zfs....

i'll see how it goes when it has 5x drives in there..

Looks like i am chasing another issue in that windows 7 machine to windows 7 machine / server (8 disk hardware raid 5 via perc5i) is abysmal.. much slower than the windows 7 to NAS so maybe it isnt the nas box after all? it peaks to start with then slows to about 25-40% of the gige network - realtek cards running same driver in either machine think they are 8111 or 8168 chipsets...

Again... anyone had issues using realtek to the n40l?

Thanks again

Steve

 

[ProtoSD]

be good if there was a nice guide for tuning zfs....

http://wiki.freebsd.org/ZFSTuningGuide

 

[5teve]

Thanks for the link. I nearly understood a few words there!

Will study it and see if I can make head or tail of it!

Steve

 

[Digidoc]

I've experienced the same issue as well. When I was still running my server on my old hardware (AMD Opteron 185 (socket 939), Asus A8N32-SLI Deluxe motherboard, 4GB DDR RAM, seven 2TB HDD's in RAIDZ2, Intel gigabit server NIC, and an extra 2-port SATA card (the board only had 5 SATA ports)), it would hiccup whenever I backed up my files to the server. I use MS SyncToy 2.0 to sync the files on my workstation to the server over a windows share.

When I was initially using the nForce gigabit NIC on the motherboard, transfers were abysmally slow. Installing the Intel NIC helped in my case, but it still hiccuped when I would transfer multiple large files to the server (usually camera RAW files from my Canon 1DsMk3 - around 21MB each).

After upgrading the hardware in the server though there's a marked improvement in it's responsiveness. It still does hiccup but it's nowhere near as bad as it was before. Where before it was unmistakable, now you have to be paying attention to see any hiccuping/slowdown in transfer.

What's in my server now:

AMD Phenom II X6 1090T CPU, MSI NF980 motherboard, 16GB DDR3 RAM.

I transferred over the Intel NIC, the 2-port SATA card, and the seven 2TB HDD's.

I have a 30GB SSD laying around that I'm thinking about adding as a cache drive to see if it makes another improvement in the transfer.

 

[survive]

Hi 5teve,

My guess is that ZFS is causing the network to pause when it flushes the cache to disk. Is there any "rhythm" to the pauses? Try running "zpool iostat <poolname> <interval>" (e.g. zpool iostat zpool0 -2) and see if the disk is writing the whole time.

One other thing, be sure you are running the 64 bit version of FreeNAS so you can use all your RAM.

-Will

 

[5teve]

Thanks for the info guys...

Survive:-

release of 8.02 and the amd64 bit build running on a 4gb usb stick on the inside of the micro server

Yes there is a rhythm to the troughs... and i was thinking the same.. i am still waiting for disks.. so currently have 1 x 250gb that came with the unit.. and 2 very old 400gb sata1 hitachis that have been used and abused over the years.. the 5 x samsungs i have arriving next week (once i strip them out of the enclosures as it was cheaper to buy externals currently) may make a difference.. i hope

below is the details from a 15gb folder transfer with generally large iso files in it.. each low value (lower than normal) arrives at the end of a pause, which incinuate that it is a cache flush.. and faster drives hopefully will keep up.. update interval is 2 secs

stripe 310K 740G 0 202 8.45K 24.8M
stripe 310K 740G 0 436 0 53.2M
stripe 310K 740G 0 733 0 91.4M
stripe 301M 740G 0 768 0 92.0M
stripe 301M 740G 0 840 0 105M
stripe 301M 740G 0 604 0 75.5M
stripe 301M 740G 0 834 0 104M
stripe 301M 740G 0 666 0 82.9M
stripe 715M 739G 0 776 0 88.9M
stripe 715M 739G 0 811 0 101M
stripe 1.12G 739G 0 620 0 73.6M
stripe 1.12G 739G 0 844 0 106M
stripe 1.53G 738G 0 665 0 78.5M
stripe 1.53G 738G 0 850 0 106M
stripe 1.53G 738G 0 689 0 85.3M
stripe 1.96G 738G 0 746 0 88.2M
stripe 1.96G 738G 0 752 0 93.9M
stripe 2.40G 738G 0 611 0 72.4M
stripe 2.40G 738G 0 815 0 102M
stripe 2.40G 738G 0 743 0 92.7M
stripe 2.84G 737G 0 708 0 84.5M
stripe 2.84G 737G 0 784 0 97.8M
stripe 2.84G 737G 0 620 0 76.4M
stripe 2.84G 737G 0 817 0 102M
stripe 2.84G 737G 0 810 0 101M
stripe 2.84G 737G 0 820 0 103M
stripe 2.84G 737G 0 771 0 96.3M
stripe 2.84G 737G 0 844 0 106M
stripe 2.84G 737G 0 548 0 68.3M
stripe 3.28G 737G 0 521 0 61.7M
stripe 3.28G 737G 0 851 0 105M
stripe 3.28G 737G 0 686 0 84.3M
stripe 3.73G 736G 0 813 0 96.4M
stripe 3.73G 736G 0 767 0 94.3M
stripe 4.11G 736G 0 764 0 91.5M
stripe 4.11G 736G 0 819 0 102M
stripe 4.51G 735G 0 649 0 77.4M
stripe 4.51G 735G 0 837 0 105M
stripe 4.93G 735G 0 618 0 73.3M
stripe 4.93G 735G 0 846 0 106M
stripe 4.93G 735G 0 678 0 83.9M
stripe 5.36G 735G 0 741 32.0K 88.4M
stripe 5.36G 735G 0 848 0 106M
stripe 5.79G 734G 0 622 0 73.3M
stripe 5.79G 734G 0 815 0 102M
stripe 5.79G 734G 0 817 0 102M
stripe 5.79G 734G 0 844 0 106M
stripe 5.79G 734G 0 844 0 106M
stripe 5.79G 734G 0 758 0 94.7M
stripe 5.79G 734G 0 816 0 102M
stripe 5.79G 734G 0 809 0 101M
stripe 5.79G 734G 0 832 0 104M
stripe 5.79G 734G 0 428 0 53.5M
stripe 6.23G 734G 1 365 95.9K 42.2M
stripe 6.23G 734G 0 865 0 107M
stripe 6.23G 734G 0 822 0 101M
stripe 6.68G 733G 0 602 32.0K 70.7M
stripe 6.68G 733G 0 840 0 104M
stripe 7.13G 733G 0 661 0 78.1M
stripe 7.13G 733G 0 832 0 104M
stripe 7.13G 733G 0 698 0 87.2M
stripe 7.52G 732G 0 791 0 95.6M
stripe 7.52G 732G 0 826 0 103M
stripe 7.93G 732G 0 637 0 75.7M
stripe 7.93G 732G 0 829 0 104M
stripe 8.36G 732G 0 646 0 74.6M
stripe 8.36G 732G 0 842 0 105M
stripe 8.36G 732G 0 742 0 92.5M
stripe 8.80G 731G 0 681 0 81.3M
stripe 8.80G 731G 0 845 0 106M
stripe 8.80G 731G 0 816 0 102M
stripe 8.80G 731G 0 841 0 105M
stripe 8.80G 731G 0 827 0 103M
stripe 8.80G 731G 0 820 0 103M
stripe 8.80G 731G 0 797 0 99.6M
stripe 8.80G 731G 0 848 0 106M
stripe 8.80G 731G 0 502 0 62.8M
stripe 8.80G 731G 0 380 0 47.6M
stripe 9.24G 731G 0 807 0 95.6M
stripe 9.24G 731G 0 868 0 107M
stripe 9.69G 730G 0 618 32.0K 72.3M
stripe 9.69G 730G 0 852 0 105M
stripe 9.69G 730G 0 665 0 78.6M
stripe 10.1G 730G 0 783 0 97.4M
stripe 10.1G 730G 0 762 0 94.8M
stripe 10.5G 729G 0 689 0 82.2M
stripe 10.5G 729G 0 816 0 102M
stripe 10.9G 729G 0 606 0 71.9M
stripe 10.9G 729G 0 831 0 104M
stripe 10.9G 729G 0 643 0 79.6M
stripe 11.4G 729G 0 805 0 94.0M
stripe 11.4G 729G 0 826 0 103M
stripe 11.8G 728G 0 625 0 73.8M
stripe 11.8G 728G 0 805 0 99.5M
stripe 11.8G 728G 0 839 0 104M
stripe 11.8G 728G 0 832 0 104M
stripe 11.8G 728G 0 849 0 106M
stripe 11.8G 728G 0 844 0 106M
stripe 11.8G 728G 0 796 0 99.4M
stripe 11.8G 728G 0 832 0 104M
stripe 11.8G 728G 0 446 0 55.5M
stripe 11.8G 728G 0 378 0 47.3M
stripe 12.3G 728G 0 703 0 83.9M
stripe 12.3G 728G 0 844 0 106M
stripe 12.7G 727G 0 534 0 62.4M
stripe 12.7G 727G 0 844 0 105M
stripe 12.7G 727G 0 750 0 92.1M
stripe 13.1G 727G 0 709 0 84.2M
stripe 13.1G 727G 0 817 0 102M
stripe 13.5G 726G 0 659 0 78.2M
stripe 13.5G 726G 0 839 0 105M
stripe 13.9G 726G 0 627 0 74.1M
stripe 13.9G 726G 0 843 0 105M
stripe 14.3G 726G 0 651 0 74.2M
stripe 14.3G 726G 0 831 0 104M
stripe 14.3G 726G 0 767 0 95.7M
stripe 14.8G 725G 4 614 2.50K 72.8M
stripe 14.8G 725G 0 819 0 102M
stripe 14.8G 725G 0 835 0 104M
stripe 14.8G 725G 0 830 0 104M
stripe 14.8G 725G 0 573 0 71.7M
stripe 15.6G 724G 0 164 0 13.4M
stripe 15.6G 724G 0 0 0 0
stripe 15.6G 724G 0 0 0 0


On another note....

I have had a look at the tuning guide.. and things are slowly starting to creep in my head..

One confusion i have is loader.conf and sysctl.conf and what goes where...

mainly the vfs tunable parameters.. i have seen them referenced to loader.conf and also to sysctl.conf.. i can edit the files and add lines... so i'm a little on my way..

As i said i am as noob as you can get... so any info & definitive hints would be great...

Thanks again

Steve

 

[ProtoSD]

One confusion i have is loader.conf and sysctl.conf and what goes where...

Someone may have a better explanation, but basically anything that is in sysctl.conf can go in loader.conf

I put most of the stuff from my sysctl.conf in my loader.conf because it's easier to have all my settings in one place and easier to remember to backup loader.conf when doing an upgrade.

There's a setting that I just saw someone post that helped with the 'troughs, I'll see if I can find it and update this post with it.

So now the system is stable. My issues seemed similar to praecorloth's in a sense that it would panic after a certain volume of data copied to it. It doesn't happen any more - it's stable, but the performance is bursty.
I added this:

vfs.zfs.txg.timeout="5"

The bursts became more even - around 10 seconds each with 10 seconds of no activity, doing around 30MB/sec when active (or 15MB/s on average).
I used to get better performance before this last upgrade though.
Network interface has mtu 9000 set.

 

[survive]

Hi 5teve,

I wouldn't stress to much right now....wait till you get your new drives and if you are still having problems with the slowdowns we can explore them then.

-Will

 

[5teve]

Someone may have a better explanation, but basically anything that is in sysctl.conf can go in loader.conf

I put most of the stuff from my sysctl.conf in my loader.conf because it's easier to have all my settings in one place and easier to remember to backup loader.conf when doing an upgrade.

There's a setting that I just saw someone post that helped with the 'troughs, I'll see if I can find it and update this post with it.

Thanks for that... i feel better now as i had found that and added it into loader.. i may mess around with other vfs settings as i have the 8gb ram.. i think i can be generous with them.. however....

Hi 5teve,

I wouldn't stress to much right now....wait till you get your new drives and if you are still having problems with the slowdowns we can explore them then.

-Will

Thanks Will... its what i was thinking... i am hoping they will make the difference.. will begin exploring once i find out if its consistent.

now i just need to find out if i can flash the samsungs (HD204UI/VP4) with an alternative firmware to stop them sleeping lots..

Thanks again guys.. will update soon

Steve

 

[ProtoSD]

I just found another great post in the FreeNAS 7 forums pertaining to this topic that I thought would be good to add here too. Even thought it's in the FN7 forums, it's about FreeNAS 8:

ZFS peaks and valleys

There are some nice graphs and comparisons between settings.

 

[survive]

A little off topic, but I really do miss those real-time graphs from 0.7!

-Will

 

[jfr2006]

I replace my system with a new processor, new board and new controller. On volume1, i still get this pauses. On volume 2, some pauses, but not so severe...

I think this is really an issue with freenas....

 

[frank3523]

I have got the same problem all new hardware except the disks. The disks are 4 maxtor 5400 rpm disks that worked fine in my older freenas 7 server.
Speeds go to about 100Mbs and then suddenly all stops for a wile. Very strange.

 

[5teve]

Well i have got my new 5 drives... and not any improvement.. still very peaky... i then tweaked the loader and sysctl (finally worked out how to get sysctl persistant!) with settings found on the net recommended for 8gb ram. Tweaks have improved things.. i still get a peaky transfer between 25% and 85% network bandwidth and windows copy averages out about 75MB/s so not too bad.. no major pauses... so it must be a zfs 'character' be nice if i could get it smoother.. but as i'm only using it for backup from my server (8x500gb hitachis hanging off a perc5i) it not causing me any major pain..

settings i used (N40L micro server, freenas 8.02 release, 5 x samsung 204ui (taken from samsung story external boxes - they were cheaper!) intel NIC and 8gb non ecc ram..)

loader.conf

vfs.zfs.txg.timeout="5"

sysctl.conf

kern.maxvnodes=250000
vfs.zfs.txg.write_limit_override=1073741824


changed NIC to mtu4000

I'm sure there are settings that will help more so feel free to post your recommendations!

Steve

 

[frank3523]

When I looked at the problem a little better I see it is only doing writes to the freenas. Reads don't give any problems.
And when doing writes it every time stops after a large file most of the time 1 GB is copied. Changing from one file to another then it hangs for a while.
When you have to copy a lot of large files its really annoying.

my Mobo is an asus E35M1-I and I have 8GB of memory installed.
Disk are 4 maxtors 250GB 5400rpm. Config is raid-Z

 

[jgreco]

Don't change the NIC MTU unless the rest of your network is also at the same MTU. And if you do go that route, use 9000. 4000 is a little crazy, unless you have defective-by-design hardware.

 

[5teve]

Don't change the NIC MTU unless the rest of your network is also at the same MTU. And if you do go that route, use 9000. 4000 is a little crazy, unless you have defective-by-design hardware.

i have also changed to 4000 on all other NICs on the network..

I beleive my router (tp link cheapy 1005d v3) only supports to 4kb frames... 9kb didnt really make any difference over 4.... and i think the router is the reason. V4 of the router goes up to 15kb.. NICS elsewhere only go up to 9kb also. as you quite rightly said... defective by design!

Frank

If you have 4 x maxtors at 250gb i would be guessing they are around 5-6 years old? they will be SATA1 and probably quite slow by todays standard. try modifying your loader and sysctl (at your own risk!) to see if you can smooth it out.

also have a look for a way of getting the nas box to write to itself (im a linux newbie) as then that will take your network out of the equation.