Performance issues.

[Andy Lee]

So I've dug up an old computer with a Athlon 64 3000+ processor, with some DD2 which freenas is showing 5103MB, using onboard NIC and everything connected to an unmanaged gigabit switch. The NAS works okay so far, but I kind of expecting a little bit more, for example when doing a large batch of files copying, should not cause lag streaming video on another computer - both using SMB.

What I've noticed or tested so far is...

• from freenas reporting charts, cpu usage always hits over 100% when I copy files into the server (not sure how it can go over 100% especially its a single core)
• I am not saturating the gigabit network yet. I did an iperf but its fluctuates greatly. So I did 100 counts and took the average which is 482 megabit (max 663, min 114) when no one is accessing the NAS
• There are two pools one 3x640mb and another one 3x3tb
• file compression = lzjb
• dd if=/dev/zero of=ddfile bs=2048k count=10000 10000+0 records in
  10000+0 records out
  20971520000 bytes transferred in 61.257771 secs (342348728 bytes/sec)
• dd if=ddfile of=/dev/zero bs=2048k count=10000
  10000+0 records in
  10000+0 records out
  20971520000 bytes transferred in 38.797635 secs (540536043 bytes/sec)

I am open to hardware upgrades. Because this is an old machine - a lot of components that are upgradable like RAM and CPU - is definitely cheaper to get the newer gens - AMD FM2 with DDR3. So I need to weight in whether it will give me a substantial increase in performance (20%-40%) increase. I wish there's some kind of comparison chart for me to see what is acceptable speeds.

Any thoughts anyone?

 

[cyberjock]

Few things:

-Expect compression to tank performance. Your CPU is very old and you aren't helping it at all with compression on.
-The FreeNAS manual says 6GB+1GB for each TB of hard drive space.. you don't even hit the minimum to use FreeNAS with ZFS. I think you've figured this one out already though.
-Your dd tests are invalidated because compressing /dev/zero is the absolute best thing you could do for a compression algorithm. Fastest to compress with almost no CPU loading required, fastest to decompress, and extremely high compression ratios. There's no way you would REALLY get those speeds with compression off.

My guess is you need more of.. everything. More CPU power and more RAM. If you disable compression(and of course recreate the zpool because disabling compression won't decompress the files already on the pools) and max out the motherboards RAM you might be able to get 20-40MB/sec and might be able to stream movies while doing large batch copying. Those are big mights though so don't hold me to them. Adding an Intel NIC may help a little to(you didn't say what manufacturer your NIC is). You shouldn't expect anything above 50MB/sec under any circumstances though. That CPU is 10 freakin' years old! Without at least 2 cores you're really stuck in the ditch performance wise. 4 cores is when you start getting really great performance.

 

[Andy Lee]

A bit embarrassed at the old gear haha, but ya its a freaking old CPU. But that's what everyone say, dig up your own computer and use it as an NAS. I guess its just not the speeds that I was hoping lol.

I know compression is definitely not helping, but I have smaller pool uses no compression yet it still hits 100%+ whenever access it. I guess I'm SERIOUSLY low on processing power.

Anyways, can you give me a ballpark say I replace it with 2/4 core AMD, with 16gig... will I likely see 20-40% increase in performance?

edit: i wanted to add more ram to this existing setup, but... the best I can get is 4 dimm fit with 2gbs... and they are hella expensive now since DDR2 arent mainstreams anymore. Thats why I was weight in replacing rather than upgrading.

edit: mobo is m2n-e, i think the NIC is realtek

 

[cyberjock]

It's really hard to say. I'm not an AMD fan so I can't really give you much of a comparison. I know Intel Atoms are allegedly equal to P4s and you are lucky if you can hit 40MB/sec on them(they have up to 4 cores!).

In your case I'd either accept the performance you have or look at building a new machine. Even a used first gen i3 with 4 cores can saturate Gb. My i3 FreeNAS machine has 16GB of RAM and I regularly get 125MB/sec copying through Windows Explorer! Generally if you can find a machine that can use the amount of RAM you need then the only CPUs you'll be able to use will be overkill.

Hey, there's no shame in at least trying to use a 10 year old system. It would be great as a backup server.

 

[Andy Lee]

oo thats a good reference then. i3 has only dual core too iirc... I'm just gonna find whatever cpu+mobo thats the cheapest and can let me use do 2~3 concurrent activities on it then I'm happy. I am also not looking into do any link aggregation or such so saturating the gigabit is pretty much the max for me (for now a year or so).

edit: if i can get 125MB/sec speeds on Windows Explorer copy, then i'm getting waaaaaaaay more than 40% already. I think I'm under 50MB/sec avg.

 

[cyberjock]

Don't confuse Mb(Megabit) and MB(Megabyte). Gigabit LAN is theoretically limited to 133MB/sec. The fact that I get 125MB/sec is just freakin' awesome. iperf lists its speeds in Mb/sec while file copying is normally MB/sec.

Most people get 70-80MB/sec because their local machine storage only goes that fast. I'm typically limited by whatever local drive I'm copying to or from. In my case 3 drives in RAID0 for max performance as temporary storage for stuff while zipping, reencoding videos, video editing, etc.

 

[jgreco]

using onboard NIC

• from freenas reporting charts, cpu usage always hits over 100% when I copy files into the server (not sure how it can go over 100% especially its a single core)
• I am not saturating the gigabit network yet. I did an iperf but its fluctuates greatly. So I did 100 counts and took the average which is 482 megabit (max 663, min 114) when no one is accessing the NAS

Any thoughts anyone?

What sort of gimpy network card did your motherboard manufacturer bless you with? I am suspicious because you report such variation. Look to see if your CPU is getting hit heavily during iperf tests. If so, possibly the cheapest easy thing to try would be putting a good network card in there. Intel desktop cards are cheap and readily available. Intel server cards will get you a few more percent but may not be worth the "extra" dollars.

If you have a network card that is eating lots of cycles (especially on a single core CPU) just to be serviced by the driver, then you can effectively "get a faster CPU" by swapping in a card that eats less cycles.

Compression on a single core CPU is kind of a loser, mostly during writes. It also means your dd numbers are likely completely artificial, you're testing how fast your CPU is compressing and decompressing a small handful of disk blocks. So turn compression off on the pool and then try those two again.

 

[Andy Lee]

One of these pools does not have compression. But due to lower density, the performance will definitely be slower than the 3x3TBs. Anyways, the results are as follows.
dd if=/dev/zero of=ddfile bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 184.170257 secs (113870287 bytes/sec)
dd if=ddfile of=/dev/zero bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 108.122502 secs (193960735 bytes/sec)

And resource chart looks like this when I did the dd tests. So it appears even without compression my cpu is suffering already.

The activites on the very right are dd processes. The ones before are just streaming video through wifi.

I also found the controller responsible for my for my NIC - its a Marvell 88E1116 Gigabit LAN controller. Haven't really researched it but I have also set my target on an Intel NIC since they are cheap enough to just buy and forget.

So I guess two things is happening? While the drives should be able to write at somewhat decent speeds (writing slowly over 100MB/s on itself), the transfer rate or such is affecting the transfer rate? As shown by iPerf that my network transfer rate graph is ALWAYS a huge zigzag...?

@noobsauce, ya I'm aware of mb and MB. I guess this just means my transfers speeds are unbelievably low side since you had a hint that I might mixed it up lol.

At the end of the day, I know this gear seems to be hopeless and I have set my mind to replace it (sometime not now but soon), but I just see if I can learn something about freenas performance through this diagnosis. Thanks again for your input guys.

 

[cyberjock]

On good hardware most people get 60-90MB/sec. You said your iperf fluctuates between 114 and 663(average of 482) which translates to a min/max/avg of 14/82/60 MB/sec. Considering that test had no CPU loading for ZFS nor any hard drive reading or writing that's decent. The low is really low though and an Intel might fix the low. But overall your CPU is going to cripple your performance(along with not having much RAM).

Considering that streaming a movie is typically 2-5MB/sec that means your system was so taxed during your large batch copying that it couldn't handle 2-5MB/sec for a second stream. When you copy files to/from the server what kind of speeds do you get? I'm really not sure there's any way to get your system to stream a movie during a second read or write operation without a new system. :(

Big picture, your poor baby has less RAM than is recommended, quite old CPU, and a less than ideal NIC. I'm not sure fixing the cheap stuff will make it usable. That CPU will definitely be holding you back. How far is hard to guess.

 

[praecorloth]

Question. Your drives, are they RAIDed? If so, are you RAIDing via a controller on the motherboard? That would explain high CPU usage during dd.

I really don't think the Athlon 64 is lacking for power. I get ~25MB/s transfer speeds via samba from a P4 3.2GHz with 1.5GB DDR2. I moved to an Athlon 64 with 5GB of DDR2 recently, and the transfer rates bumped up to ~33MB/s. Granted, I'm not using compression. The next thing I'm replacing is the NIC, which is a Realtek.

On the topic of NICs, I would say buy an Intel NIC for this bad boy. You'll likely use it even if you don't stick with this motherboard and processor, so you won't have lost anything.

 

[Andy Lee]

Question. Your drives, are they RAIDed? If so, are you RAIDing via a controller on the motherboard? That would explain high CPU usage during dd.

I really don't think the Athlon 64 is lacking for power. I get ~25MB/s transfer speeds via samba from a P4 3.2GHz with 1.5GB DDR2. I moved to an Athlon 64 with 5GB of DDR2 recently, and the transfer rates bumped up to ~33MB/s. Granted, I'm not using compression. The next thing I'm replacing is the NIC, which is a Realtek.

On the topic of NICs, I would say buy an Intel NIC for this bad boy. You'll likely use it even if you don't stick with this motherboard and processor, so you won't have lost anything.

No I'm not raiding via controller on motherboard. Its just plain drives individual drives to mobo and I let freenas raidz them.

The transfer rates varies between 10-40MB/s through Windows transfer. The only thing that frustrates me (which leads me to investigate) is it seems I cannot stream a video file without showing obvious lag while doing a large file copying.

I thoughts on NIC is same too. I can always reuse it even with a new system.... nothing to lose. :)

 

[praecorloth]

The transfer rates varies between 10-40MB/s through Windows transfer. The only thing that frustrates me (which leads me to investigate) is it seems I cannot stream a video file without showing obvious lag while doing a large file copying.

That seems a very high fluctuation in speed. Currently I am running an Athlon 64 x2 5200+ with 5GB DDR2. My storage is 3x 500GB drives in RAIDZ. The fluctuation I get in speed is between 30-34MB/s. iperf shows FreeNAS as a server transferring at 620ish Mbit/s, while as a client it transfers at 420ish Mbit/s. What about pings? If you ssh in to FreeNAS and have it ping your desktop for a minute or so, do you get many dropped packets, or high response times? I'm seeing no dropped packets over 100 pings, with low response times around 0.085ms, occasional 0.1ms, and the rare 0.2ms.

This is all very vexing.

 

[cyberjock]

All iperf does is prove that you have a Gb LAN connection. That in no way proves that you can transfer 620Mb/sec. Your CPU is quite old and you don't even have the minimum RAM the manual calls for (6GB + 1GB for each TB of disk space). Plus, you added in compression. Using compression you have literally crippled your system. Someone a few months ago setup compression and they couldn't get 10MB/sec with an 8 core processor and 24GB of RAM because of the type of compression they chose. Practically nobody uses the compression because its quite CPU intensive and hard disk space is SO cheap. Not to mention that media files generally don't compress at all (and sometimes get bigger!) so you are adding CPU workload for no benefit.

Your setup has a whole bunch of bad things against it. You shouldn't be expecting much of anything from it beyond it being capable of storing files(notice I didn't say stream files).

 

[praecorloth]

Ah, I forgot about the compression. That is something I am not doing.