Resource icon

"Absolutely must virtualize TrueNAS!" ... a guide to not completely losing your data.

KMR

Contributor
Joined
Dec 3, 2012
Messages
199
I'm perfectly happy running the FreeNAS VM at 12GB as my other VMs will primarily be linux based so I don't mind running low RAM for most of them. There are a couple of VMs which perform best with 4GB of RAM so it would be nice to be able to have the extra capacity given that VMs are the nerd version of drugs.
 

no_connection

Patron
Joined
Dec 15, 2013
Messages
480
Re: "Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your d

Edit: Here's a tidbit for those virtualizing. ESXi5(not sure about any other version) has a very peculiar use of vCPUs to physical cores. This killed my server performance until I lowered the number of vCPUs in my machine.

I had assigned 4 cores to FreeNAS and 4 cores to a Linux VM on a 4core(8 thread) system. The thought was both could work hard without competing for resources. After all, both will pretty much never be heavily loaded at the same time as the other, so this should be "smart". FreeNAS performed absolutely horribly even with the other VMs idle.

I have:
one quad core VM
two two CPU VM
one single CPU VM
Sharing a quad core CPU (no HT) without too much trouble.

I read somewhere that the bug/problem is mainly for multi-CPU setups and does not have much impact to multi-core VMs. (was a while ago though)

Can you elaborate on the problem?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Re: "Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your d


Edit: Here's a tidbit for those virtualizing. ESXi5(not sure about any other version) has a very peculiar use of vCPUs to physical cores. This killed my server performance until I lowered the number of vCPUs in my machine.

I had assigned 4 cores to FreeNAS and 4 cores to a Linux VM on a 4core(8 thread) system. The thought was both could work hard without competing for resources. After all, both will pretty much never be heavily loaded at the same time as the other, so this should be "smart". FreeNAS performed absolutely horribly even with the other VMs idle.

It turns out that the VM will get processing resources only when the total number of VCPUs is available(regardless of actual loading the VM needs). So even if FreeNAS is completely idle with about 0% CPU usage the VM will NOT be given any timeslices for computing unless I had 4 cores free. I thought it would adjust dynamically, but it does not. So if you give one VM 5 cores and another 4 cores on the same 8 thread system those 2 VMs cannot under any circumstances use the CPU at the same time. Big waste.

Hey, wolflike beast, if you are going to post interesting comments, TACK THEM ON AS A REPLY so people see them. I would discuss these things if I would see them in a timely fashion! Thank you! :smile:
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I have:
one quad core VM
two two CPU VM
one single CPU VM
Sharing a quad core CPU (no HT) without too much trouble.

I read somewhere that the bug/problem is mainly for multi-CPU setups and does not have much impact to multi-core VMs. (was a while ago though)

Can you elaborate on the problem?

In short, if you assign 4 vCPUs to a VM, if that VM needs to process anything(even 1 core's worth), ESXi will allocate 4 cores to the VM. If I only used 1 core for a particular period of time that means the other 3 cores are idle by my VM's OS perspective, but they are in use and unavailable to be used for any other VM or ESXi itself.

Now, let's say I had an 8 core system, 4 vCPUs for FreeNAS VM and 5 vCPUs for a linux VM. That means that at any given moment either the Linux VM or the FreeNAS VM was using the CPU. This is stupid because you've instantly forced those 2 VMs to be available to use no more than 50% of the available time. And if you have more VMs running that problem can get exponentially worse. But, if you set your FreeNAS and Linux VM to 3 vCPUs each, then both can run at the same time and still have 2 vCPUs for anything else you might need.

The vCPU setting is like an exclusive lock for that many cores for a VM. Either the VM gets no processing time at all, or it gets your vCPU's number of cores. There is no in between.

In short, you need to assign it enough vCPUs to get the job done, but don't overdo it as you can hurt the performance of all of the VMs.

This is an ESXi problem and not a FreeNAS problem. The same issue would occur between any 2 VMs with a high number of vCPUs. I'm thinking this is considered normal and expected since it seems that VMWare seems to target their marketing propaganda around optimizing older systems(which will normally be very low in CPU usage and number of cores).
 

titan_rw

Guru
Joined
Sep 1, 2012
Messages
586
In short, if you assign 4 vCPUs to a VM, if that VM needs to process anything(even 1 core's worth), ESXi will allocate 4 cores to the VM. If I only used 1 core for a particular period of time that means the other 3 cores are idle by my VM's OS perspective, but they are in use and unavailable to be used for any other VM or ESXi itself.

I'm going to have to disagree with this. At least with modern virtualization. I know this is how a lot of the early virtualization implementations worked.

For example, my home esxi box is an i3 with 2 cores / 4 threads. All the vm's on it have 4 vcpu's assigned. That means that any vm has full access to the hosts cpu. Obviously if all the vm's were trying to use all 4 vcpu's at the same time there'd be cpu contention, but in my home environment that doesn't happen. But it does give the vm access to lots of cpu power if none of the other vm's need it.

Also, vmware's 'best practices' for desktop virtualization (VDI), is to have 1 physical host cpu core for every 5 virtual desktop's you plan to run. And you'll normally assign 2 vcpu's per vdi instance.

Anyway, my understanding of how current virtulization works, is that's it's pretty much purely time share based. I know there are reasons not to go assigning crazy amounts of vcpu's per vm when there's large amounts of vm's. Even if the host has that many cores available. But cores are not 'exclusive' to a vm when the vm is 'using' them. CPU contention can still occur however if the hosts cpu is not powerfull enough the cpu requirements of the vm's it's running.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Ok, well, that's take straight from their VMWare documents. They actually mention in their docs not to overprovision your CPUs because of exactly what I've explained. I'll see if I can find the original document on the topic.

And you and I can disagree with it, and it doesn't change the reality of how it works. I'll see if I can find the original article. I will say that as soon as I "fixed" my vCPU assignments and nothing else my performance problems disappeared.
 

titan_rw

Guru
Joined
Sep 1, 2012
Messages
586
Like I mentioned, I know there's reasons not to assign 16 vcpu's to every virtual machine, even if the host has dual 8 core processors.

All I'm saying is that a 4 vcpu vm that's using 10% overall cpu doesn't exclusively tie up 4 of the physical cpu cores.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
No, each VM is exclusively using up its number of vCPUs to get work done. In my case, my idle linux box and idle FreeNAS box had some small background tasks running but it would kill performance of both VMs.

Once I found the issue and did searching this is a serious misunderstanding among many VM users.

Here's some links I found in just a few minutes:

http://www.gabesvirtualworld.com/how-too-many-vcpus-can-negatively-affect-your-performance/

http://public.madeinengland.co.nz/how-many-cpus-should-a-vm-have/

Still trying to find the vmware doc that basically explains what those 2 links explain.
 

titan_rw

Guru
Joined
Sep 1, 2012
Messages
586
From what I'm reading, it's still about time sharing. But yes, in my environment, where all 4 cores are assigned to all vm's, and if one vm is using 10% overall cpu, then 10% of the time, no other vm's can execute.

If I had a single vcpu vm using 50% of it's singe core, then 50% of the time there wouldn't be enough cores to time share in the 4 vcpu vm. But the 4 vcpu vm *could* still execute 50% of the time.

I thought you were saying that the single core vm using 50% of it's single core locked use of that single core to that vm 100% of the time. Which it doesn't.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Nope. It looks like we're talking about the same thing.

The issue is that if the VM needs any processing time at all, it will get none unless the VM is assigned the total number of vCPUs you've assigned. If your VM has 4 vCPUs assigned to it and 1 process in that VM needs 1 core, the other 3 vCPUs will be idle(since there is no actual work to be performed), but ESXi cannot assign those "idle" vCPUs to anything else.

In short, don't throw more vCPUs at a VM "because you can". Rightsize your VMs and you'll be pleasantly surprised at how much work you can do. I'm really not sure why "right size" hasn't been one of those keyword/tricky phrases for years. It's been true for so long and yet everyone things the solution to everything is to throw more hardware or software at a problem. That's not true at all and people that do that look like amateurs.

Here's the VMWare doc if anyone wants to read it. It's 157 pages so its some "light" reading:

http://pubs.vmware.com/vsphere-51/t...enter-server-51-resource-management-guide.pdf
 

KMR

Contributor
Joined
Dec 3, 2012
Messages
199
This has made me curious about my pending ESXi configuration (why does life seem to keep getting in the way?) and I'm sorry this is specifically FreeNAS related. My configuration will be using an intel 1230v2 which is a quad core with HT (so 8 logical cores).
Say I have 6 VMs with varying performance needs:
FreeNAS for basic media streaming - 2 cores
Domain controller & DNS (AD) - 1 core
Linux reverse proxy server - 1 core
Win 7 VM for video transcoding duties - 4 cores (although the more the better for this type of work)
Linux email server - 1 core (maybe 2?)
Linux web server - 1 core
This means that I would want at a minimum 10 cores assigned at any given time and possibly more if I wanted to play with other VMs or if the work load of certain VMs increased. Most of these VMs wouldn't be using many cycles very often (web, email, DC, etc) but other VMs could be running full tilt (Win 7).
Are you just saying that I should assign the minimum that the VM should require and re-evaluate if performance suffers or is the CPU capacity undersized for this type of load? Furthermore, if I wanted to dedicate 6 or 8 cores to the Win 7 VM to speed up the transcoding process would this adversely affect performance of the other VMs?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
ESXi is perfectly capable of timesharing the CPU if overcommitted, but you make the problem more complex for the VMware scheduler by assigning multiple cores. It is definitely wise not to just assign massive resources that are not required. Do not forget that a VM can easily have its cores changed with a shutdown/edit/reboot so the smart money says start small.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
This has made me curious about my pending ESXi configuration (why does life seem to keep getting in the way?) and I'm sorry this is specifically FreeNAS related. My configuration will be using an intel 1230v2 which is a quad core with HT (so 8 logical cores).
Say I have 6 VMs with varying performance needs:
FreeNAS for basic media streaming - 2 cores
Domain controller & DNS (AD) - 1 core
Linux reverse proxy server - 1 core
Win 7 VM for video transcoding duties - 4 cores (although the more the better for this type of work)
Linux email server - 1 core (maybe 2?)
Linux web server - 1 core
This means that I would want at a minimum 10 cores assigned at any given time and possibly more if I wanted to play with other VMs or if the work load of certain VMs increased. Most of these VMs wouldn't be using many cycles very often (web, email, DC, etc) but other VMs could be running full tilt (Win 7).
Are you just saying that I should assign the minimum that the VM should require and re-evaluate if performance suffers or is the CPU capacity undersized for this type of load? Furthermore, if I wanted to dedicate 6 or 8 cores to the Win 7 VM to speed up the transcoding process would this adversely affect performance of the other VMs?

Yes, and no. It will adversely affect the performance of the other VMs relative to the timesharing, loading of all of the VMs, and your choice of vCPUs per VM. In short, read that 150+ page document and you should understand it. For VMs I never do reencoding with more than 2 cores because normally reencoding video isn't a "high priority", but killing the performance of some other tasks that need to run near real-time can have problems.
 

titan_rw

Guru
Joined
Sep 1, 2012
Messages
586
Nope. It looks like we're talking about the same thing.

The issue is that if the VM needs any processing time at all, it will get none unless the VM is assigned the total number of vCPUs you've assigned. If your VM has 4 vCPUs assigned to it and 1 process in that VM needs 1 core, the other 3 vCPUs will be idle(since there is no actual work to be performed), but ESXi cannot assign those "idle" vCPUs to anything else.

Yes, but I think this is further complicated by the fact that the OS may load balance the single threaded process to different logical cores. I see it all the time in windows at least. I'll have a 4 core processor, 4 threads (no HT). A process will be consuming a solid 25% of the CPU (one thread / core). But all 4 cores are loaded at exactly 25%. I'd expect one core to be 100%, and the others to be 0%.

Anyway, this is only really an issue on the vm host with relatively few cores that has a relatively large amount of vm's with relatively large assigned numbers of vcpu's, that are all using a constant amount of cpu.

In my case, I have only 5 or so vm's in total, so it's never an issue, even though the host only has 4 threads (only 2 physical cores). Being a home environment, I never have lots of the vm's all running 'single core limited' stuff. Whereas I do end up having times where a vm uses all four vcpu's completely for more than a short amount of time.

Any enterprise vmware environment is going to have far greater than 4 threads on each host. Probably more like 16-32.

Freenas can, and will use lots of cores though. I've got a 4 core / 8 thread cpu for freenas that I've seen only 20-30% idle. So if you limit vcpu's, you 'cut off the top' of the max freenas can use, but it's a bit more friendly for scheduling if the vmware host is very busy with other vm's. One more point that makes me glad I didn't virtualize freenas. I'll stick with a dedicated machine for esxi, and the two dedicated machines I have for freenas.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Yes, but I think this is further complicated by the fact that the OS may load balance the single threaded process to different logical cores. I see it all the time in windows at least. I'll have a 4 core processor, 4 threads (no HT). A process will be consuming a solid 25% of the CPU (one thread / core). But all 4 cores are loaded at exactly 25%. I'd expect one core to be 100%, and the others to be 0%.

Actually, what you see can(and probably is)a lie. Intel does cool tricks like turning off different cores and simulating multiple cores with just one core. This allows Windows(and every other OS out there) to try to smartly take 10 threads of small background tasks and evenly divvy them up among cores without waking up the other cores. There are situations where its better to run one core at 3.6Ghz with all of the background tasks running on that then everycore being awake at 3.0Ghz just for background tasks. So those CPU numbers you see aren't necessarily the truth. From what I've read in Intel presentations and tech guides if you are using a CPU% that appears to be near a core boundary you are probably running with just that number of cores "awake". Especially in Windows. There are ways to determine which CPU is awake and for how much, but it gets hard to track in real-time because it potentially changes 1000s of times a second. Try outputting that info to your video and keep up with it. ;)

Edit: This is one area where I think Intel is showing clear advantages over AMD. AMDs ability to shutdown and wakeup cores has been found to often be either too conservative or too liberal with real-world. There's a very thin line between keeping cores asleep too much and keeping them awake too often. Intel's designs seem to be better at hitting that very small line than AMD. Keep in mind there is a performance penalty anytime you wakeup or put a core to sleep. You're either having to either flush things out of the core in preparation for shutdown or incur the penalty of waking them up and bringing them back to life with things to process.

Anyway, this is only really an issue on the vm host with relatively few cores that has a relatively large amount of vm's with relatively large assigned numbers of vcpu's, that are all using a constant amount of cpu.

Or if you have a few VMs with too many vCPUs assigned. In my case, with my 1230v2 and my 5vCPU FreeNAS VM and 5vCPU Linux Mint VM performance was stupid. Since I have only 8 threads available with HT only 1 of my 2 VMs could be running at any given time. This killed the performance of both VMs(especially noticable if I did a zpool scrub!). Also was a problem if doing iperf testing between the 2 VMs since I had 10Gb between them. In fact, zpool scrubs and iperf testing between the VMs is exactly what lead me to the problem.

I thought I was doing myself a favor by going with 5vCPUs each as that would never bottleneck my scrubs in FreeNAS and I'd never bottleneck Plex in my Linux VM. I thought ESXi would smartly give more processing power to whichever VM actually needed it. Instead I bottlenecked both of them all of the time as neither VM could get more than 50% of the CPU time. Right now I run FreeNAS with 3 vCPUs, my Linux VM with 3vCPUs, and my newly installed UPS Appliance VM with 1vCPU. All performance problems are gone, and my VMs are actually faster now with fewer vCPUs.

Cool huh? Flies in the face of the "amateurs" that think throwing more hardware at the problem will make it go away. That's why I called them "amateurs" above. It's not about having the biggest, fastest, or most expensive hardware. It's about knowing how to use what you have.
 

titan_rw

Guru
Joined
Sep 1, 2012
Messages
586
Yes, I see what you're saying. And I agree. It's just a complicated subject that easily gets confused over a text medium.

I think I misunderstood your initial post on the subject. I thought you had stated that (for example) a 4 vcpu vm with a single core loaded at 1% would completely tie all four cores to that vm and nothing else, during the entire (time) duration of the 1% load.

I was simply stating that in the above example, during the 99% time the single core is not loaded, the 4 cores are free to handle other vm's. I understand that during the 1% time the single core is loaded, the remaining 3 cores are 'unavailable' for any other vm, even though the 3 remaining cores are idle.

So yes, in your example of two 5 vcpu vm's on a 8 thread host, if each vm is needing more than 50% of any of the 5 cores, there will be cpu contention. With the single threaded nature of certain parts of freenas (cifs), I can see this happening.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Yeah. In my case, when a scrub was in progress things got messy because of the virtualization. The 2 VMs would compete and because 50% of the time the "other" VM was running FreeNAS performance was crap. top would show very high CPU usage which made me think it needed more resources for FreeNAS to perform adequately. What bothed me was that my old system was 5 years old, had a SIGNIFICANTLY inferior amount of processing power compared to even 3 cores of a 1230v2, and I couldn't figure this out. My problems looked like this:

1. 5 vCPUs and CPU loading is very high per top on FreeNAS.
2. Scrubs are slow.
3. Couldn't even stream a movie while scrubbing.
4. The old POS system was rarely above 25% CPU usage despite being VERY old.

Put those together and I thought the solution was "moar CPU!!!!!" Turns out I needed less. After jgreco and I had a discussion over PMs I turned down the vCPUs on both the linux and FreeNAS VM and voila, problem solved.

Sure, I can't do screaming fast 1GB/sec scrubs like I used to(I think this is because of the PCI passthrough to be honest as scrubs barely hit 25% CPU loading). But, both the VMs perform more than adequately for my needs as they currently are. Overall, that's all that matters.
 

H2O_Goalie

Dabbler
Joined
Jan 16, 2014
Messages
12
FWIW virtualization is what I do. Cyberjock has got a pretty decent handle on what's known as the "scheduler". The whitepaper located here (for 5.1, not sure it's been rev'd for 5.5 yet but the principles hold true):

https://www.vmware.com/files/pdf/techpaper/VMware-vSphere-CPU-Sched-Perf.pdf

Explains a lot of this stuff. Long story short when your VM wants to execute it must have access to all of the cores (think: VCPUs) assigned to it. Typically it's for such a short period of time that you can get away with it transparently and you won't even notice even when overcommitted a bit. On occasion, particularly for workloads that are latency/contention sensitive on a box that's been overcommitted you'll see problems. This gets into the idea of "right-sizing" your VMs (more often isn't better as Cyberjock found) and learning to "tune" resource allocations. It's also not all about CPU...you can get into trouble on the storage, memory or network fronts as well. This is a fascinating subject but is definitely a rabbit hole...you can disappear down there and never find your way back out.

My day job is doing virtualization for a company that does medical 3D advanced visualization...we're about as latency/contention sensitive as you can get. I pretty much sleep with a stack of VMware whitepapers as my pillow.

PS - yeah, before anyone gets going on "relaxed" co-scheduling...let's keep this on a fairly easy level. There's some leeway now but it's not like co-scheduling for vSMP machines just disappeared and isn't a consideration any longer.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Someone was trying to do that in IRC earlier this month. It didn't go well. Not sure if he was up to the task or just trying because he could.
 
Top