Freenas 8.2 as VM under ESXi 4.1u2. Very high read i/o and latency.

[moshker]

Help sought: I'm seeking tips to help me troubleshoot this problem. Alternatively advice on how to back out (change config to something stable).

Symptoms: High reads (28-34 MB/sec) to datastore containing zfs volumes. High read latency on datastores containing zfs volumes (50-100 ms).

Configuration: Optiplex 760 (3 Ghz cpu, 6 GB RAM, 2 TB Sata disk) running ESXi 4.1u2. Freenas 8.2 running as vm - 1.5 GB RAM, 1 vcpu, and 3 zfs volumes - each lives as a vmdk on the 2 TB sata disk. ZFS volumes aren't being accessed and no file share protocols are enabled (cifs, nfs, etc).

History: Originally was ESXi 4.1u1. Originally was running an 8.0-RELEASE-i386.iso. Was stable and though there was some latency it wasn't nearly as unusable as it is now. Upgraded to 8.03-RELEASE-p1-x64. My recollection is that I installed from iso, intending to overwrite the previous 32 bit install, but somehow it kept the settings. At some point after this the i/o problems became obvious and the Freenas volumes became unusable.

Steps taken: Removed vmdk holding 8.03 configuration. Installed 8.2 from iso. Auto-imported the volumes. Problems re-occurred. Restarts. Turned of Smart. Checked for scrub schedule, default. Checked system processes. Python is at 0:02 time with 6 threads, state looks like it is alternating between uwait and piperd. Vmtoolsd is the third service listed but has a time of 0:00 and a state of select. Originally had 1 GB of RAM. Upped it to 1.5 GB without any apparent affect on performance. Searched forums at Freenas.org and vmware.com without finding anything that seemed to speak to this issue directly.

Any help is appreciated.

Kindest regards,

Oscar

 

[moshker]

Not sure if it helps...

Here are the bottom lines of the messages log:
Mar 13 21:24:22 freenas middleware[2019]: Popen()ing: /sbin/zfs list -Hr -t file
system tv
Mar 13 21:24:23 freenas middleware[2019]: Popen()ing: /sbin/zfs list -H -o name,
volsize -t volume -r tv
Mar 13 21:24:23 freenas middleware[2019]: Popen()ing: /sbin/zfs list -Hr -t file
system movies
Mar 13 21:24:23 freenas middleware[2019]: Popen()ing: /sbin/zfs list -H -o name,
volsize -t volume -r movies
Mar 13 21:24:23 freenas middleware[2019]: Popen()ing: /sbin/zfs list -Hr -t file
system movies
Mar 13 21:24:23 freenas middleware[2019]: Popen()ing: /sbin/zfs list -Hr -t file
system tv
Mar 13 21:43:33 freenas notifier[2019]: Executing: /usr/local/bin/python /usr/lo
cal/www/freenasUI/tools/webshell.py
Mar 13 21:43:40 freenas notifier[2019]: Executing: /bin/pgrep -F /var/run/webshe
ll.pid

 

[moshker]

Another data point... I unmounted the 3 zfs volumes. /main (734 GB - 200 GB free), /tv (780 GB - 150 GB free), and /movies (524 GB - 254 GB free). When /tv is not mounted the i/o problems go away... weird?

 

[louisk]

To make sure I understand, you're using ZFS and you've allocated 1.5G of RAM for the VM? Last I checked, the minimum requirements for ZFS were 4G of RAM. It might help to add more memory.

 

[survive]

Hi moshker,

You are starving your system of RAM.

Read this: http://www.freenas.org/images/resources/freenas8.0.3/freenas8.0.3_guide.html

Section 1.1.2

Here's what I think is happening....you are running your system so low on RAM that it is actually swapping out actively in-use memory to disk to pick up the slack. When you unmount one of the volumes you might be freeing up enough ram that the system doesn't need to swap just to stay up.

Why did you set up 3 pools? You would be better off if you made 1 pool & sliced it up using datasets to segment your data.

The best thing you could do would be to just dedicate the box to FreeNAS...barring that see if you can't score 2 more 2GB DDR2 sticks (I assume you have 2X2GB & 2X1GB DIMMs) to get yourself to a point you can give FreeNAS at least 4GB of RAM.

-Will

 

[moshker]

Thanks for both of your replies. I thought memory might be an issue, and had upped it from 1 GB to 1.5 GB. I didn't see a change in the symptoms. I'm confused that this behavior doesn't affect the other two zfs volumes. When I unmount both of them and mount just the /tv volume I see the symptoms return. When I look at memory utilization I see 99M free, 472M active, 33M cache, 73M inactive, and 762M wired. What is "Wired" memory? I suspect that there is something wrong or different about the third zfs volume, but I don't know how to figure out what it is. I don't mind using ufs for the /tv volume, but I have no idea how to go about converting it! I can give it another 2.5 GB of RAM temporarily to get it up to 4 GB. Maybe move the /tv volume to its own spindle. I'll post the results. Do either of you have any tips about I might be able to look at in the shell to sort what is going on under the hood?

Kindest regards,

Oscar

I made 3 vmdk's and kept them separate to keep it simple in case I had a problem. I could dedicate a box to Freenas, but the extra time, power, heat, noise seemed like a waste. Even a large compromise in performance (< 70%) would be acceptable. The /main and /movie volumes perform very well considering the degree of abstraction (~45 MB/sec). It is only the /TV volume that suffers from extreme latency and performance problems (< 2MB/sec) and shows constant read i/o (30 MB/sec). As if there is some kind of scrub or something happening with the drive, but I can't see what it is that is reading the drive or what purpose it has.

 

[moshker]

Ok, some new scary data! I increased memory to 4 GB. I made a backup of the volume 2 months ago. I mounted it into the Freenas vm. No performance problems. No weird i/o behavior. Then I mounted the one I'd been having problems with and the problems returned just as before. So basically I have a problem with that volume! I have a backup and it works and isn't missing any data, but I want to keep this from happening again and if it does want to be able to fix it or catch it before it turns into a big problem. Advice?

 

[louisk]

So, you have multiple zfs volumes (vmdk) on the same physical 2T SATA disk?

 

[moshker]

I do. There are 3 vmdk's that range in size from 500 GB to 750 GB. Currently they are all working very well. In a recent file transfer I was seeing sustained transfers nearing 80 MB/sec. This is after reducing RAM to 2 GB. Pre-fetching for ZFS is disabled, but I'm not seeing a bit performance hit from it.

 

[louisk]

Are you still using 3 pools or did you consolidate down to 1 pool and use volumes?

 

[moshker]

I don't think they are pools. I just put them in as disks, and then created volumes on those disks. I'm from a Windows background and managing storage as pools isn't intuitive to me (yet).

 

[louisk]

You may want to take a look at the ZFS administration guide
http://docs.oracle.com/cd/E19963-01/pdf/821-1448.pdf

It explains how ZFS works. Probably a good idea if you're going to be storing any important data on it.

 

[moshker]

Huh. I'm not sure how reasonable it is for me to spend the time to read a document that is nearly 300 pages long to store the data I've got. Especially when many of the concepts and abilities of ZFS are entirely out of reach in a virtualized environment that is limited on resources, and aren't exposed in the GUI. I think I looked through the document briefly when I was trying to resolve the problem I was having, but wasn't sure how applicable the OpenSolaris ZFS guide was to the Freenas/BSD implementation. I suppose the file system isn't mature enough, or is so feature rich and complex that I shouldn't expect it to "just work" like NTFS, UFS, xt3/4, etc. Thanks for reminding me about the document though!

Oscar