Storage Performance Degradation

[rob78]

First off I would like to say that I love FreeNAS and thank you to everyone involved! I'm having a problem that I can't seem to figure out on my own. I created my FreeNAS server 4/23/2014. After setup I was able to saturate my Gigabit network at 110MB/sec read & write. After copying 6TB of data to the server my writes are still 110MB/sec but my reads are down to 80MB/sec.

Hardware:
CPU: Xeon E3-1230V3 Haswell 3.3GHz LGA 1150 Quad-Core
Motherboard: ASRock H87WS-DL LGA 1150 Intel H87 DDR3 1600/1333
RAM: 16GB Kingston DDR3 SDRAM 1600 ECC Unbuffered Server Memory KVR16E11S8/4
Storage: 6X TOSHIBA DT01ACA300 3TB 7200 RPM 64MB Cache SATA 6.0Gb/s

Setup:
Version: FreeNAS-9.2.1.5-RELEASE-x64
Storage: RAIDZ2 6 Disks, SATA, AHCI Enabled
Compression: lz4 1.00x Ratio
Usage: 6.7TB / 10.6 TB
Scrub: Completed Errors:0 Repaired: 0

I've also attached a boot log for any additional information I forgot: boot.txt

Network:
JPerf 2.0.2: 108 MBtypes/sec

Write:
[root@nas] /mnt/MAINSTORE# dd bs=1024 if=/dev/zero of=testfile count=1M
1048576+0 records in
1048576+0 records out
1073741824 bytes transferred in 4.668831 secs (229980866 bytes/sec)

Read:
[root@nas] /mnt/MAINSTORE# dd bs=1024 if=testfile of=/dev/null count=1M
1048576+0 records in
1048576+0 records out
1073741824 bytes transferred in 10.946070 secs (98093821 bytes/sec)

Observations:
When I perform write tests or actually copy a file to the server gstat reports 50% to 100% %busy on all drives in the RAIDZ2. When I perform a read or copy a file from the server, gstat reports barely 10% %busy. I've attached screenshots for write and read above as a real-world example of my issue.

Summary:
Is 70-80MB/sec all I should expect to get from this hardware/setup? I've cancelled my plans to trunk another network line to the server since I can't even saturate 1 line anymore. I'm hoping you can help me fix this and if there is any additional information or benchmarks you would like me to run please feel free. Thank you in advance for your assistance and keep up the great work!!!

 

[cyberjock]

Well, as the pool gets more and more data it will get slower. I don't seen anything wrong with 80MB/sec. There's so many things that can cause pool slowdown you're going to have to do significant diagnosis to determine where the slowdown is(and if you even have control of it). I can tell you that your pool can certainly do more than 200MB/sec and almost certainly more than 300MB/sec. So I'd take a look at your network hardware and setup and your local machine before you start looking at a problem on FreeNAS. ;)

 

[rob78]

I appreciate your response. I have already verified everything on my network and my local machine. Also if you see the benchmark on the read speed that I provided, the text is from the actual FreeNAS server via SSH:

[root@nas] /mnt/MAINSTORE# dd bs=1024 if=testfile of=/dev/null count=1M
1048576+0 records in
1048576+0 records out
1073741824 bytes transferred in 10.946070 secs (98093821 bytes/sec)

This is showing only 98MB/sec on the server having nothing to do with the network or local machine. I have verified copying/writing files from several of my local machines at 110MB/sec as they are all very fast machines with SSDs. I am interested in advice as to how I should proceed with determining why the read speeds are so slow. Thanks again for your time.

 

[cyberjock]

Your bs is too small.

Try bs=1M

 

[rob78]

Okay so holy crap is this fast:
[root@nas] /mnt/MAINSTORE# dd bs=1M if=testfile of=/dev/null count=50000
50000+0 records in
50000+0 records out
52428800000 bytes transferred in 5.507539 secs (9519460448 bytes/sec)

Am I reading this correctly that it's 9500MB/sec? or 9.5GB/sec? That can't be correct and I must be doing something wrong. Anyway if the problem isn't the read speed and I've already tested the network using jperf (screenshot was attached) and I've verified that I can copy/paste files to the computer I'm testing with to a different computer on the network at 110MB/sec, what else could it be?

 

[joeschmuck]

What about enabling the tunables? If your RAM is not optimized (allocated properly), this should help out some. Also CIFS isn't the best protocol to use. Try NFS, I hear that it is much better if you can use it.

 

[cyberjock]

You have compression enabled and the file is all zeros. ;)

 

[rob78]

Okay so I tried enabling autotune and restarted. It added several memory limits and some other things I don't know about. I didn't notice a performance increase so I turned it back off and have rebooted since so that I'm back to the original install configuration again.

I ran the read test using a video file:
[root@nas] /mnt/MAINSTORE# dd bs=1M if=11GBVIDEO of=/dev/null count=12000
10825+1 records in
10825+1 records out
11351123434 bytes transferred in 1.129449 secs (10050143506 bytes/sec)

I'm attempting to switch to NFS but apparently you need Windows 8 enterprise for native support so I'm looking for a 3rd party software for this. As a side note, I had attempted reading using FTP with similar results at 85MB/sec. Slightly faster than CIFS.

 

[rob78]

Realizing that the server must have cached that file, I ran the test again with a new file. It appears that this test worked properly:
[root@nas] /mnt/MAINSTORE# dd bs=1M if=11GBFILE of=/dev/null count=12000
10825+1 records in
10825+1 records out
11351123434 bytes transferred in 14.163821 secs (801416754 bytes/sec)

So 800MB/sec read sounds right to me.

 

[diehard]

NFS can be enabled on enterprise and ultimate versions of Win 7 & 8 . Or you can try using "Nekodrive".

 

[solarisguy]

764MiB/s ! That is a very fast hard drive (191MiB/s) ! I looked this model up, and yes that is the speed reviewers are seeing.

Can you try on something larger? For example, concatenate some large video files to have your test file > 32GiB ?

 

[rob78]

The NFS with NekoDrive didn't work out well at all. It ran at 25MB/sec then kept stopping completely. I don't know if this gives you guys more info on what might be wrong but I'm stumped...

Ran that test with a 42GB file:

[root@nas] /mnt/MAINSTORE# dd bs=1M if=42GBVIDEO of=/dev/null count=50000
40440+1 records in
40440+1 records out
42404732928 bytes transferred in 84.423422 secs (502286355 bytes/sec)

I'm really stumped:
1. Network tests are 108MB/sec to/from server.
2. Read/Write on server is very fast.
3. Verified that test PC can send/receive files at 110MB/sec on same network.

I guess I'm back to the drawing board. It seems like some sort of setting that's limiting usage or something but I know I didn't set anything like that.

 

[rob78]

To eliminate NFS or CIFS from the equation, I tested using FTP and got the same results:

 

[rob78]

Lastly, here is a screen shot of a network report on the FreeNAS server of a file download vs an upload. It seems like the server can Receive 110MB/sec but only send 85MB/sec. Since I am 100% positive that it's not the test machine, what could be preventing FreeNAS from sending at the full 110MB/sec? The tests show that the pool is more than fast enough.

 

[solarisguy]

Thank you for a large file test.

FTP is good for testing since it is quite simple.

Do the read results come back after you power cycle your FreeNAS server?

Please perform a full power-cycle, that is: shutdown, turn the power off on the power supply, wait at least three minutes, power on the power supply, start the server.

You did gstat. Did you watch top?

P.S.
Can your receiving end machine read that fast (not only fast disks, but also low utilization of the filesystem)?

 

[rob78]

The test PC can read/write to another Windows 8 machine on the same local network at 110MB/sec. I've tested this right before I tested the read from the FreeNAS server.

I shutdown the FreeNAS for 5 minutes, started it up, and tried copying the file to my test PC again. This time I ran both gstat and top:

To eliminate any concerns about my internal network or my test PC (ROB-PC), here's a screenshot of me copying a file from another PC to it at 111MB/sec:

Thanks again for your time.

 

[solarisguy]

So the read speed did not come back...

We can always blame ;) some weird interaction with your not-recommended network card. Can you disable in BIOS your built-in network cards and install an add-on Intel Gigabit Ethernet card?

 

[rob78]

Eureka!!! You're right solarisguy it's the NIC! My first test was the server RECEIVING. This latest test is the server SENDING. Only 85MB/sec.

RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet:

CONFIRMED NETWORK CARD ISSUE: RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet

Can I find different drivers for FreeBSD? Is there a patch for FreeNAS? Or should I just buy a an Intel card?
So mad at myself for missing this, I did a lot of research before building this machine. Oh well.

Thanks again for your help solarisguy & cyberjock!

 

[cyberjock]

Just buy an Intel card.

 

[solarisguy]

As an alternative, perform a budgeting exercise whether to go for a motherboard that takes your Xeon, but has ECC RAM and Intel network.