Freenas 8.3.1 fast transfers, slow directories, cannot stream movies

[froggie]

FreeNAS 8.3.1, 64bit on an 8GB thumbdrive
ASUS P8B75-M LX PLUS, Intel Pentium G2020 Ivy Bridge 2.90GHz, 16GB Crucial Ballistix RAM, 6x WD 3TB Red NAS SATA

The system is quick and my file transfers average 75MB/s. I am very pleased with this.

I have 2 major problems though. I am not able to stream movies from the server. It could be small 1.5GB show or large 14GB movie. Once I try to open the file, my share access to the server basically locks up while the web interface works fine. If I try to open one of my picture backup directories that has about 10,000 RAW files, then the share will hang for about 100-120 seconds before listing the directory. When the system hangs, no other computers can use a share.

I started with a 6 drive ZFS RAID-Z1, but have also tried a 2 drive UFS mirror and stripe. For the slow directory issue, I can use the web command line to list a large directory in about 2-3 seconds. I can use SSH to list a directory in under 1 second. I have tried all 3 sharing protocols (afp, nfs, smb/cifs) on two Macs using OS 10.6.8 and 10.7.5 and both of them are slow, taking over 100 seconds using my stopwatch. I also tried the nfs and smb/cifs shares on a fresh install of Windows 7, 64bit with no luck (a tad over 120 seconds to load).

I cannot see this as being a hardware issue. CPU usage is under 30%, RAM might jump up to about 11GB of my 16GB. I have done a lot of tinkering and searching. I even tried a bunch of tunables from a post - http://forums.freenas.org/archive/index.php/t-5338.html

My friend has a much weaker Solaris server with 2GB RAM, 10x 1TB Seagates in a software ZFS RAID-Z1. He only has a smb/cifs share and it can load my picture backup directory in about 3-4 seconds, file transfer speed is about 35MB/s, and there are no issues streaming movies from it. I really wish I could get this FreeNAS thing working, any help will be appreciated. I am probably going to tinker for about another week before giving up. Thanks.

 

[cyberjock]

I had a similar problem with my server. It would seem to "lock up" for long periods of time and top showed that the system wasn't heavily loaded and zpool iostat didn't show high disk usage. After 2 months of frustration I upgraded from 12GB of RAM to 20GB and the problem went away. If you have some spare RAM to try that would be my first suggestion. Alternately, if you've tried to tweak CIFS, ZFS(tunables), or the sysctls you should get rid of them. All of them. I'd bet big big money you got those tweaks wrong and they are making your problems worse. :P

 

[froggie]

Unfortunately, 16GB of RAM is the max for the motherboard I chose. As another test, I pulled some drives and did a new 3 drive ZFS RAID-Z1. I was not able to see any improvement. I currently do not have any system tunables loaded... I did end up removing all of them once I saw that they did nothing for my particular system.

 

[cyberjock]

Ok.. few other things as last ditch effort...

1. Try enabling autotune. You will have to reboot for the changes to take effect. This might help.
2. That board allegedly can use 8GB DIMMs, so you might be able to use 32GB of RAM. I'm really not sure if more RAM will help your exact issue. Its hard to say without a boatload more information from various CLI commands and such.
3. Try disabling your antivirus and accessing the server. For reasons I cannot explain some A/V programs will freak out and cause major delays(measured in 10+ seconds) to open a folder with just a few files. I was able to verify that the issue is for certain with the A/V and not FreeNAS. I was(keyword: was) using Symantec Antivirus at the time. I figured this out out of desperation. My guess is that the Antivirus/Firewall triggers and prevents your desktop from replying(or prevents the incoming data from being processed immediately causing a long delay) to the FreeNAS server and Samba locks up waiting for your response.
4. If you are familiar with Linux you will find that Linux can access the shares much faster than Windows can. I have no idea specifically why(except that Windows has so much more overhead than linux now its killing itself), I just know that when I need to copy very large amounts of data I use my linux machine.

I'm not a proponent of the Pentium class of CPU. I ignore them completely and I have no idea if that CPU is limited in its abilities at times. I know those CPUs have very little L1 and little(if any) L2 cache. As a result you may have very low CPU usage and the cache misses are killing performance. I know my i3-530 machine has no performance issues with 16GB of RAM and 10x3TB drives in a RAIDZ2 with no tweaks at all except autotune.

 

[froggie]

1. I tried autotune before manually adding other suggestions in the forums, no luck.
2. 16GB RAM is definitely the max. I agree that it probably doesn't matter since I tried 3x 1TB drives in a ZFS RAID-Z1
3. I mostly use my Mac. The Windows 7 computer was a fresh install - no antivirus, no Windows firewall, no UAC, no other software.
4. I agree that Linux is faster for stuff, but my goal is to use this in a real-world scenario where I have 2 Macs. Maybe a Windows computer.

I was able to find someone who used the same processor and similar build as me with great success. http://forums.freenas.org/archive/index.php/t-10855.html I noticed that they did ZFS RAID-Z2.

I took 5x of the WD Red drives and did a ZFS RAID-Z2. My transfer speed jumped from 75MB/s to 110MB/s. It appears that I am now maxing out my network, lol. Processor usage spikes at 50% utilization, RAM usage went down from 11GB to 8GB. I still have the slow directory listing issue. The Mac (afp) is now able to stream video from the server without an issue...Windows (smb) still fails at it. I can even do a file transfer while streaming a video on the Mac without any issues - 90MB/s transfer.

 

[cyberjock]

Holykiller is one of our more knowledgable FreeNAS guys. As such, his build may run great, but he may have added tweaks to get his to work well. Not saying yours shouldn't or that we should tweak, just making the observation.

What I would do is reinstall all of your disks and run a long smart test on all of them. One of them may be doing weird things or is failing. You can run smart tests with the command:

smartctl -t long /dev/adaXX

You will need to adjust the above for each drive. Start the test and let them run for a couple of hours. The drives will tell you how long to wait for the test to finish, which is anywhere from 1-6 hours depending on the drive. Then do the command:

smartctl -a -q noserial /dev/adaXX

And look at the log that is printed out for any error messages. If you get completely lost and want some feedback on the results you can post them here and someone will let you know if any worthwhile info can be gathered. If you want you can post the output for each of your drives before the test in case something obvious is already shown.

Another observation. If you are hitting 50% CPU utilization you are likely hitting the limits of what that CPU can do. If a 5 drive RAIDZ2 can hit 50% then a 6 drive RAIDZ1 may be held back. It's tough to say since you are adding 1 more disk of complexity but removing 1 set of redundancy calculations. Some FreeNAS services are single threaded and single core speed is more important than having 8 cores. More than likely that 50% was Samba hitting its limit(it is for sure single threaded).

 

[gpsguy]

Is the slow directory browsing issue, limited to the large directories like the photo backup directory with 10,000 files?

There are some message threads on the forum that discuss the issue, though I don't recall their being a silver bullet fix.

Do a Google search: site:forums.freenas.org large directories

 

[cyberjock]

In my experience, large directory listings were almost guaranteed to be slow. Even if they had been listed just seconds before! However sometimes directories with 50 files or so would take more than 30 seconds. There would be no disk activity or network activity on either end. Looking at top as well as the CPU usage on the desktop it was almost like neither machine was doing anything useful and was just sitting there. But eventually when the "lockup" ended and I looked at the sniffed network traffic the desktop sent some data to the server which then responded immediately after and the directory listing appeared.

The issue appears to be with the desktop on the surface. But if something is timing out on the server and the desktop's traffic is a different timeout where its requesting the directory listing again then it could be the server. But since I have a mix of Linux Mint 13 and Windows 7 Home Premium machines and the issue has only ever occurred on the Windows 7 machines with Symantec Endpoint Protection installed I dismissed it as a Symantec issue. As soon as I disabled the Symantec firewall from the Windows machines then all of the issues with slow directory listings went away. I tend to think that the firewall is monitoring the traffic and something is triggering it to think the system is compromised and the delay is while it does a more thorough analysis of the traffic or something(sending data to Symantec to see if it matches certain fingerprints?). That is how the newest 12.x version works.. unfortunately. I assume that the home version of Symantec Antivirus has the same technology since Symantec has made many press releases that this new fingerprinting via their servers is the future of antivirus detection.

From my experience and reading the forums there's 2 potential issues for unusually long delays with directory listings.

Server-side from things such as:

• Zpool is very busy and loading a very large directory listing
• The directory is fragmented on the zpool and it takes some time to accumulate all of the pieces
• The system is short on RAM and the ZFS cache is having major issues with discarding data it needs for the generation of the entire directory listing(like a dog chasing its tail)
• Options in Samba are setup to sort the directory listing before sending it to the client. (This is a big CPU hog according to the Samba manual and is not recommended because the client will typically reorganize the entries to match those of the file manager you are using anyway.)

Client-side such as:

• Client is dealing with heavy network traffic or is otherwise unreliably connected to the network(wifi)
• Client firewall is causing issues(as I explained above)
• Client is heavily loaded and the directory listing is a low priority task.

I was never able to positively identify the exact issue with Symantec. I tried various settings but since the issue is very intermittent I gave up. I hadn't made any changes from the default settings when I used Symantec and other people have complained about various other issues with Symantec's firewall causing latency and whatnot I dismissed it as really poor defaults or design on Symantec's part. Surprise.. I got rid of Symantec and have had no issues since.

I do know that Symantec's firewall is very CPU intensive. On my laptop (i7 920 extreme mobile) I can't get above about 25-30MB/sec to or from my FreeNAS with the firewall on. If I disable it(even midway through a very large transfer that was going slow) I can reach over 100MB/sec from my SSD. Looking at my CPU usage 1 core approaches 100% and with the firewall disabled loading drops to single digit percentages on all cores. I saw the same regarding a desktop that was plenty powerful enough to hit high speeds but couldn't hit more than 50-60MB/sec unless I disabled the firewall. Just like the laptop, disabling the firewall would cause an instant increase in network utilization to over 90%. I assume the firewall driver adds just enough latency to cause the network performance to drop significantly.

 

[froggie]

I have a few directories with about 30-40 files that have no problem. The picture directories with 8k-10k pictures are the problem. Once that specific large directory loads, it will be quick and snappy for at least the next 10 minutes.

I think this weekend I am going to try and load up Fedora with zfsonlinux.org and see if the problems still exist.

 

[cyberjock]

I can tell you right now that the issue isn't with ZFS. The fact that you can do an ls and get the results returned in a second or so means its not a file system problem. That kind of narrows the issue down to Samba, networking, or the client machine. Other than that, the exact cause for my situation is unknown.

Good luck!

 

[froggie]

I tried to do Fedora, but was not able to get the zfs stuff working with it. I ended up using openindiana.org and found that it pretty much fixed all of my issues. I can now stream videos with no issues, and large directories load in under 8 seconds compared to the 100+ seconds in freenas. I will miss the freenas web gui though :(