Inconsistent performance with ESXi and Freenas 9

[estienne]

Hi,

Current setup
Home-made PC with Intel 4440 & 16 GB of RAM, 2X 3TB SATA, 2X 2TB sata, 1Gb interface
Running ESXi
I created a VM with 4 disks, 1 for FreeNas (32G, read later 2GB was enough) and 3X1861GB virtual disk on 2 2TB disk and 1 3tB disk.
Made Freenas working with VM guest 1 cpu (tried 2 also) and 9GB of RAM.

My question/problem:
When writing data to disk using CIFS/Windows, I got ~80MB/s for ~10-30 seconds, then it goes down to 5-10 MB, then up 80MB, and down. CPU (as seen in vmware), is going down too as speed goes down.
Instead of using vmware, I put freenas directly on a usb stick and use 3X 2TB disks (I changed a 3TB for a 2TB disk), and got consistent ~80MB/s write speed.
Anybody have any ideas why the VM is varying so much in performance? It looks like it "sleeps" for some time, then wake up, sleep, wake up... No idea why.

My goal: have 1 PC for NAS (using freenas) and for other Linux & windows virtual guest. It worked, but I was expecting >50MB sustained write and read speed.

 

[cyberjock]

Nope. We don't provide "troubleshooting and support" for ESXi since that's not our product. Our product performs fine on bare metal and that's all we worry ourselves with.

If you read the forums there's some "do's and don'ts" for FreeNAS, of which you've obviously broken at least 1 very serious one.

 

[estienne]

Hi cyberjock, thanks (not sure...) for the honnest answer.
I read about raid5 and forgot about it. It is home data after all. Still hesitate about raid6 (is my data worth it? that is the question).
1 serious one... just out of curiosity, which one? Raid5 or something else (from your point of view)?
--> This is my HOME stuff, won't invest 10 000$ (or more than I already had) in a great setup. Just want to listen movies, and a backup device.

 

[gpsguy]

Search for and read the "absolutely must virtualize" thread.


Sent from my phone

 

[cyberjock]

There are problems with virtualizing FreeNAS that can result in the immediate and unexpected loss of your pool without recovery. Search for what gpsguy says and see that thread. That thread links to another. Both of them are written because so many people have lost data.

Then, check out this thread: http://forums.freenas.org/threads/reboot-produces-gpt-table-corrupt-or-invalid.17260/#post-91239. His pool works fine with ESXi... until he reboots. I just posted 5 mins ago and he looks like he's gotten bit by an ESXi bug too.

This is why we warn people not to virtualize FreeNAS. We don't test for it and we don't really care to. If you find a bug and know what is broken, the developers have fixed it. But the developers have much bigger beasts to slay than bugs that may not even be our own fault.

Big picture, we see far more people lose their data with ESXi than we do without. In short, there's obvious and documented risk with running it in ESXi. It's not well understood what is wrong(see the part about developers having bigger beasts to slay) and the fix isn't apparent. It could be a virtualization bug, it might not. But we've got people that have weird problems that make it obvious that virtualization was used before they lost their data. Due to the consistency with FreeNAS on ESXi we know there is a link. Those same problems have never occurred on real bare-metal setups which only makes troubleshooting even harder. At some point you have to draw the line and say "this is our lawn, this is not" and accept that your product doesn't play well with ESXi because their lawn isn't being cared for.

So the bottom line, if your data is valuable, don't virtualize FreeNAS. If your data is going to always be safely backed up elsewhere Virtualizing is an option. Just don't expect miracles and at any point you might reboot your server and your data will be gone.

 

[estienne]

I read the other thread... and this one... and now I'm convinced that I will pay extra electricity bill :(. I will have a dedicated freenas box and a vmware server, which will be a bit overpower now.
BTW I read the other thread but it was not THAT convincing. Maybe if you add some cut&paste from this one and the other, less people will try it!

 

[gpsguy]

Good choice, grasshopper. ;)

I too, run 2 separate boxes for FreeNAS and ESXi.

 

[eraser]

Still hesitate about raid6 (is my data worth it? that is the question). 1 serious one... just out of curiosity, which one? Raid5 or something else (from your point of view)?
--> This is my HOME stuff, won't invest 10 000$ (or more than I already had) in a great setup.

I have four data disks in my home FreeNAS server. I decided to go with Striped Mirrors. This post helped convince me: http://constantin.glez.de/blog/2010/01/home-server-raid-greed-and-why-mirroring-still-best

 

[cyberjock]

@eraser:

With 4 disks for a home server RAIDZ2 is safer statistically because you can survive any 2 disk failure versus 2 disks failing only if they aren't the same vdev. But to each their own. The alleged increase in writes will never be seen as your Gb LAN will always bottleneck you. One great thing about FreeNAS is that it is flexible and since its your data you get to choose what you do. Good luck to you.

 

[ZFS Noob]

The alleged increase in writes will never be seen as your Gb LAN will always bottleneck you

Unless you're measuring IOPS instead of throughput...

 

[cyberjock]

Except home servers don't care about IOPS. If you want to run an ESXi NFS server you are not making a "home server". Terminology is important. ;)

 

[AiRLAC]

There are problems with virtualizing FreeNAS that can result in the immediate and unexpected loss of your pool without recovery. Search for what gpsguy says and see that thread. That thread links to another. Both of them are written because so many people have lost data.

Then, check out this thread: http://forums.freenas.org/threads/reboot-produces-gpt-table-corrupt-or-invalid.17260/#post-91239. His pool works fine with ESXi... until he reboots. I just posted 5 mins ago and he looks like he's gotten bit by an ESXi bug too.

This is why we warn people not to virtualize FreeNAS. We don't test for it and we don't really care to. If you find a bug and know what is broken, the developers have fixed it. But the developers have much bigger beasts to slay than bugs that may not even be our own fault.

Actually, Todd had the same problem with his setup after switching to bare metal so I don't think his problem has anything to do with ESXi.

For what it is worth, I'm running FreeNAS baremetal-- ESXi is not in the equation.

 

[cyberjock]

You do realize that I've been posting in that thread, so I do actually know what is going on in that thread. Why you feel the need to tell me what I clearly already know is.. beyond me.

 

[AiRLAC]

You do realize that I've been posting in that thread, so I do actually know what is going on in that thread. Why you feel the need to tell me what I clearly already know is.. beyond me.

Not you, but anybody else reading this topic. Just to make it clear, no offense :)

 

[cyberjock]

Beat me to my added comment.. which was going to be:

Todd most likely has a hardware issue or incompatibility. I've got that thread on my bookmarks because I'm very curious to see the final diagnosis of the problem.

 

[Dusan]

Not you, but anybody else reading this topic. Just to make it clear, no offense :)

Heh, it seems I have a twin! :D

 

[eraser]

Except home servers don't care about IOPS. If you want to run an ESXi NFS server you are not making a "home server". Terminology is important. ;)

Home users should definitely care about IOPS performance if they plan to use their FreeNAS for anything other than a simple movie file storage appliance.

 

[jgreco]

Lots of professional IT guys have home labs to experiment and learn on. Some of are wired for dozens of ports of 10GbE at home. Preconceived notions about "home" are bad. Actual use cases are more important. For ESXi, for example, we always see people come in and give a VM count as though that has anything to do with anything. We have lots of VM's in inventory that generate minimal IOPS. Also some heavy hitters. I worry more about the one than the two dozen... because it's the IOPS, not the VM count, that matters. It is perfectly possible for a home user to be using ZoneMinder or other apps generating lots of traffic...

 

[gpsguy]

I surely wouldn't give a VM count for my home lab. Mine are all turned off. They typically sit dormat, until I have a need.

For ESXi, for example, we always see people come in and give a VM count as though that has anything to do with anything.

 

[jgreco]

Active VM count, then. But the point is valid; 50 powered-off VM's is only slightly less stressful than 50 mostly-idle VM's.

We see people come in all the time trying to describe their "requirements" in those terms.