Plummeting performance

[sunnyinphilly]

My FreeNAS experience was going swimmingly until last night and I'm struggling to figure out what's happening. Some quick background. After a fair amount of reading/research, I tested the waters with an older PC I had laying around (Celeron E3200, 4 WD Green HD's, 2GB non-ECC Crucial memory, etc.) and wanted to see FreeNAS first hand before making any more investments.

I configured a ZFS volume consisting of 2 mirrored 4TB drives and 2 mirrored 2TB drives. I'm not using any plug-ins and figured my limited RAM would at least me allow to do some testing. Performance was actually amazing – I was getting pretty consistent 100+ MB/s writes and equal reads. I transferred well over 2TBs of data on my wired network and figured, all right, I'll get 8GB of ECC RAM, fix the big flaw in the system and put this box into daily use.

That's when things started to go downhill. I installed the RAM last night and everything booted perfectly and performance was right in line with what I had previously seen. Except that after any moderate amount of activity (let's say copying 2-3GB of files), performance falls through the floor. I'm talking 2-3MB/s writes. Maybe it spikes to 6 if I'm lucky. It's atrocious.

Since the RAM swap was the only variable, I'm sorta at a loss. Could this a case of bad memory chips? Or did I commit a FreeNAS sin by initially ignoring the ZFS requirements and thinking I could remedy it after the fact?

Any help would be appreciated.

 

[gpsguy]

Perhaps something else happened when you swapped the RAM.

Re-check all your connections, especially your network cable. Consider replacing the cable. Run Memtest 86+ on your RAM. Run at least 3 complete passes - it'll take some time. Speaking of RAM, does your motherboard and CPU support ECC?

Run "zpool status" and ensure that your pool is functioning correctly. Additionally, run SMART tests on your hard disks. If you want to post any of this data on the forum, please use "code tags".

 

[ewhac]

My first completely random thought is the network interface. What kind of NIC is it? (Spoiler: RealTek is crud.) Are both reads and writes slow, or just writes?

If the NIC checks out, do you see the same slowdown when performing file copies locally, i.e. using the FreeNAS command line?

 

[sunnyinphilly]

@gpsguy: Everything seems to be checking out fine. I ran Memtest 4 times and no errors were reported. Same with zpool status. Switching network cables didn't help either. I'm currently running a SMART test on the drives, but it doesn't seem like this is a drive issue. To test that, I created a new standalone volume with another disk that's in the server and that too is seeing some crappy performance.

@ewhac: It is indeed a Realtek NIC (model 8111c). I've heard they're crummy, but it would seem odd that I was previously seeing solid speeds. Speaking of, copying locally doesn't seem to have the same drop-off. I just tried copying a directory of ten 500MB files, speeds ranged from 34-59MB/s and I didn't see the same drop-off as I do over the network.

 

[gpsguy]

As @ewhac said, Realtek's are known to be iffy under FreeNAS. The easy fix is to buy an Intel Pro/1000. Cost for an OEM version is about $30 USD.

 

[DrKK]

Almost certainly it is the Realtek NIC. I know, it didn't screw up before. Believe me, we've seen it all with these Realteks. Everything about your problems says it's NIC-related, and you're running a Realtek.

 

[sunnyinphilly]

Appreciate the input on this guys. I ordered an Intel Pro/1000 GT card today and will give that a shot.

 

[sunnyinphilly]

Well done guys. I installed my Intel NIC last night (gotta love Amazon same-day shipping) and it totally cured my performance issues. I'm now getting steady read/writes in the 70MB/s range. Still a little slower than what I had previously been seeing, but it's at least consistent. Thanks for the help!

 

[Knowltey]

Well done guys. I installed my Intel NIC last night (gotta love Amazon same-day shipping) and it totally cured my performance issues. I'm now getting steady read/writes in the 70MB/s range. Still a little slower than what I had previously been seeing, but it's at least consistent. Thanks for the help!

Yeah, that would seem to be pretty good performance. I'm not sure what kinds of files you are seeing that performance on since the type of data being transferred can definitely effect the performance to an extent. If you're transferring a bunch of small files the performance will be a bit lower than if you're transferring a single large file.

 

[cyberjock]

Also, having an Intel NIC on your desktop affects throughput too. ;)

 

[Knowltey]

Also, having an Intel NIC on your desktop affects throughput too. ;)

Hehe, yeah. currently dealing with a double Realtek setup until I can get my upgrade here shortly (almost just bought it today, but decided to wait till the Friday paycheck) and it tanks on many small files. It has also been vastly different performances between FreeNAS version. (9.2.1.6 and 7 have been really good though, 9.2.0 was pretty bad though)

 

[epac]

I would like to inform everybody that you DO NOT need an Intel NIC in order to take advantage of your Gbps connection, even though other moderators/admins on this forum would like you to believe that Intel NICs are the only way to go. I have applied two sysctls on multiple FreeNAS boxes, in addition to sharing this information with colleagues, so I do know that these settings are effective.

I know that my FreeNAS box can easily push close to 10Gbps (see iperf output below), so after some experimentation and research (bandwidth throughput calculator), the only thing I had to do to take advantage of the 1Gbps LAN connection was to update my sysctls for the send and receive window size.

[root@freenas] ~# iperf -c localhost
------------------------------------------------------------
Client connecting to localhost, TCP port 5001
TCP window size: 175 KByte (default)
------------------------------------------------------------
[ 3] local 127.0.0.1 port 38306 connected with 127.0.0.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 11.6 GBytes 9.92 Gbits/sec

For those that have Gigabit realtek NICs, try these two sysctls you need to take advantage of your 1Gbps connection is as follows:

net.inet.tcp.recvspace 163840 (default is 65536)
net.inet.tcp.sendspace 163840 (default is 32768)

After applying these two sysctls, you will see that your system can push as close to 1Gbps speed transfers in Windows. Here are two sample iperf output after applying these settings on my freenas box:

linuxhost1:~ $ iperf -c freenas
------------------------------------------------------------
Client connecting to freenas, TCP port 5001
TCP window size: 601 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.0.25 port 48756 connected with 192.168.0.3 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 1007 MBytes 845 Mbits/sec <-- close enough to the 1Gbps connection max.

linuxhost2:~$ iperf -c freenas.
------------------------------------------------------------
Client connecting to freenas., TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.0.26 port 39483 connected with 192.168.0.3 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 940 MBytes 789 Mbits/sec <-- again, as close to the 1Gbps connection max.

As you can see, just updating two simple sysctls can maximize your investment by utilizing your existing Realtek onboard network controller and not have to buy an Intel NIC just to get the same speed as most motherboards that come with Realtek.

 

[DrKK]

LOL.

I can't decide if we should leave this post up here for amusement value, or take it down before other people actually think you have any idea what you're talking about.

What do you think @cyberjock?

 

[cyberjock]

Sorry, but 85% and 78% of theoretical is NOT, I repeat NOT, saturating Gigabit. I almost always get over 960Mb/sec when I do iperf with 2 Intel NICs. So no, I'm sorry but you're tunables may improve things, but they do NOT substitute for the throughput of an Intel.

Let's also not forget the "real" problem people have with Realteks.. they are unreliable pieces of crap. No tunables will fix that unfortunately.

 

[Ericloewe]

The really funny part is his signature. It's a Smörgåsbord of what hardware not to use with FreeNAS.

Notice how he claims to run 9.2.1.5 - convenient, given that 9.2.1.6 seems to have broken a few Realteks...

Non-ECC RAM throughout, the crummiest motherboards money can buy, plugins galore on 4GB of RAM... Hell the "8GB SSD ZILs" sound like they're those crummy SSDs that are slower than many USB flash drives.

 

[cyberjock]

I'm thinking he may be a troller...

 

[epac]

You are entitled to your opinion. I was hoping the sysctls would help other users that choose not to buy an Intel NIC. Take it for what it is, it's getting performance for relatively nothing.

By the way, I have used the FreeNAS to recover Macrium images to image a drive directly from the FreeNAS that indicated 1Gbps reads with those sysctls in place. Lastly, transfers to/from windows shares consistently indicates I am saturating the Gigabit link to/from freenas.

 

[Ericloewe]

I'm thinking he may be a troller...

That's my guess. All the totally-legitimate-video-collection plugins on 4GB of RAM, with RAIDZ2 on a dusty old Athlon 64 X2? Doesn't sound particularly credible to me.

 

[epac]

The really funny part is his signature. It's a Smörgåsbord of what hardware not to use with FreeNAS.

Notice how he claims to run 9.2.1.5 - convenient, given that 9.2.1.6 seems to have broken a few Realteks...

Non-ECC RAM throughout, the crummiest motherboards money can buy, plugins galore on 4GB of RAM... Hell the "8GB SSD ZILs" sound like they're those crummy SSDs that are slower than many USB flash drives.

This is why updating to latest and greatest is not always preferred. In that case, 9.2.1.7 was available right after 9.2.1.6.

Here's some proof of the "old" Athlon 64 X2 hardware running all those plugins:

[root@nasbox] /data# ./hddtemps
Sat Sep 13 15:04:09 PDT 2014
Installed Processor AMD Athlon(tm) 64 X2 Dual Core Processor 5000+
MOTHERBOARD Temperature at 104?F
CPU0 Temperature at 107?F
CPU1 Temperature at 113?F

Drive /dev/ada3 at 102?F - HUA722020ALA330 JKAOA3GA
Drive /dev/ada2 at 96?F - HUA722020ALA330 JKAOA3MA
Drive /dev/ada1 at 96?F - HUA722020ALA330 JKAOA3MA
Drive /dev/ada0 at 100?F - HUA722020ALA330 JKAOA3GA

[root@nasbox] /data# jls
JID IP Address Hostname Path
2 - plexmediaserver_2 /mnt/rz4tb/jails/plexmediaserver_2
6 - couchpotato_1 /mnt/rz4tb/jails/couchpotato_1
8 - sabnzbd_1 /mnt/rz4tb/jails/sabnzbd_1
9 - sickbeard_1 /mnt/rz4tb/jails/sickbeard_1
11 - transmission_1 /mnt/rz4tb/jails/transmission_1
[root@nasbox] ~# warden list
ID AUTOSTART STATUS TYPE
---------------------------------------------------------------------------
couchpotato_1 Enabled Running pluginjail
plexmediaserver_2 Enabled Running pluginjail
sabnzbd_1 Enabled Running pluginjail
sickbeard_1 Enabled Running pluginjail
transmission_1 Enabled Running pluginjail

last pid: 64731; load averages: 0.02, 0.07, 0.07 up 3+07:59:12 15:08:09
108 processes: 1 running, 107 sleeping
CPU 0: 0.0% user, 0.0% nice, 0.4% system, 0.8% interrupt, 98.8% idle
CPU 1: 0.0% user, 0.4% nice, 2.7% system, 0.0% interrupt, 96.9% idle
Mem: 355M Active, 274M Inact, 3077M Wired, 62M Cache, 127M Buf, 149M Free
ARC: 2219M Total, 126M MFU, 1777M MRU, 2704K Anon, 42M Header, 272M Other
Swap: 8192M Total, 196M Used, 7996M Free, 2% Inuse

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
53595 root 1 20 0 326M 10652K select 1 0:36 0.29% /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/
31895 816 55 23 0 336M 64944K usem 0 55:23 0.00% /usr/pbi/sabnzbd-amd64/bin/python2.7 /usr/pbi/sabnzbd-amd6
2487 root 6 20 0 308M 75588K usem 1 16:33 0.00% /usr/local/bin/python /usr/local/www/freenasUI/manage.py r
75737 816 3 20 0 352M 75832K select 1 15:02 0.00% [python2.7]
78840 816 20 20 0 306M 28236K select 1 14:12 0.00% [python2.7]
3545 root 12 20 0 138M 6656K uwait 0 10:45 0.00% /usr/local/sbin/collectd
7429 972 23 20 0 259M 24588K uwait 0 7:56 0.00% /usr/pbi/plexmediaserver-amd64/share/plexmediaserver/Plex
6240 972 16 20 0 203M 17520K uwait 1 6:51 0.00% /usr/pbi/plexmediaserver-amd64/share/plexmediaserver/Plex
80535 921 3 20 0 95124K 6764K select 1 2:22 0.00% /usr/local/bin/transmission-daemon -g /usr/pbi/transmissio
2049 root 1 20 0 12036K 1464K select 1 0:45 0.00% /usr/sbin/powerd
6452 root 6 20 0 151M 11468K usem 0 0:16 0.00% /usr/pbi/plexmediaserver-amd64/bin/python2.7 /usr/pbi/plex
1785 root 1 -52 r0 12040K 8020K nanslp 1 0:15 0.00% /usr/sbin/watchdogd
75967 root 6 20 0 160M 11172K usem 0 0:13 0.00% /usr/pbi/couchpotato-amd64/bin/python /usr/pbi/couchpotato
78079 root 6 20 0 160M 12084K usem 1 0:12 0.00% /usr/pbi/sabnzbd-amd64/bin/python2.7 /usr/pbi/sabnzbd-amd6
79070 root 6 20 0 160M 10952K usem 1 0:12 0.00% /usr/pbi/sickbeard-amd64/bin/python /usr/pbi/sickbeard-amd
80772 root 6 20 0 163M 18976K usem 1 0:12 0.00% /usr/pbi/transmission-amd64/bin/python /usr/pbi/transmission

 

[cyberjock]

Sorry but I'm going to have to lock this thread. I know where this is going and I'm not particular keen on the idea of moderating a fight that is about to break out.