FreeNAS CPUs & ESXi

[EponymousOne]

Hi,

FreeNAS newbie here.

I have ESXi Version 6.7-U2.

I've got a new VM with Freenas 11.2-U5 as the Guest OS.

With the system powered OFF, is it safe to increase the number of CPUs (to say 2) and then maybe at a later date decrease the number back to 1?

(Note: I'm *not* wanting to do hot-adding which isn't supported on FreeBSD 11 anyway).

I've been reading as much documentation on vCPUs and FreeNAS as I could find and I still can't find a conclusive answer to this.

I know that earlier versions of ESXi used to have a warning that changing the number of CPUs, after Guest OS install, can cause things to become unstable; that concerned me.

Would I be better off re-creating the VM from scratch and re-installing FreeNAS with the CPU count already set to 2 or would this just be wasted time? It would take a lot of effort to do this so I'd rather avoid it if possible.

Regards.

 

[jgreco]

It should be fine, but you may find your FreeNAS performs terribly under load.

 

[EponymousOne]

Thanks for the quick reply!

I should have mentioned I have a quad-core Xeon as well - but you probably guessed that :)

So to expand on your answer, do you mean my FreeNAS will perform terribly under load due to increasing from 1 to 2 CPUs - does it work better with just 1 CPU normally? Or did you mean leaving it at 1 will make it perform worse in the long run?

Or did you mean just generally it will perform poorly in ESXi compared to a bare-metal install? :)

Regards.

 

[jgreco]

Sorry, I wasn't clear. A single-core FreeNAS will suffer from contention between the kernel ZFS and userland protocols. Don't forget that ZFS performs the tasks of a conventional hardware RAID inside the kernel, and usually does tasks such as compression as well. And then has to run protocols and all that. Don't starve your NAS for CPU.

Also please note that if you're doing a virtualized FreeNAS, you have the additional opportunity for things to not work right. There have been issues with MSI/MSIX behaviour changing with different numbers of cores, so if you are looking for a stable FreeNAS, your better bet is to just set it up and then leave it alone if you can.

Best practice for ESXi is to generally keep the core count down to the minimum that is acceptable. With some operating systems, this has been problematic over the years because some operating systems install one kernel if you only have a single core, and a different kernel for multiple cores. That's much of the reasoning behind the ESXi core change warning. In the modern era, since most modern CPU's in the last decade have two or more cores, this isn't as big a deal.

Bare metal FreeNAS should always perform better than a virtualized instance.

 

[EponymousOne]

Thanks again.

So based on what you've said I will use a CPU count of 2 and never change it.

However I still am unsure if I should create a brand new VM with CPU=2 and install FreeNAS from scratch or whether I should just increase the CPU count on my existing VM install. Based on what you've said, FreeNAS 11.2/FreeBSD 11.2 should be able to handle the change dynamically I would've thought but I don't want to risk anything.

What do you think is the best/safest approach?

Regards.

 

[melloa]

However I still am unsure if I should create a brand new VM with CPU=2 and install FreeNAS from scratch

I run FreeNAS as a VM on ESXi for years and change setting all the time, not only on FreeNAS VM but on all VMs, to fine tune resources utilization. You will be fine.

As always, save config ;)

 

[EponymousOne]

Thanks for all the help jgreco and melloa. :)

 

[melloa]

Thanks for all the help jgreco and melloa. :)

By the way ... the same rules for memory applies to a VM.... ZFS likes memory :)

 

[jgreco]

Thanks again.

So based on what you've said I will use a CPU count of 2 and never change it.

However I still am unsure if I should create a brand new VM with CPU=2 and install FreeNAS from scratch or whether I should just increase the CPU count on my existing VM install. Based on what you've said, FreeNAS 11.2/FreeBSD 11.2 should be able to handle the change dynamically I would've thought but I don't want to risk anything.

What do you think is the best/safest approach?

Regards.

The VM you wind up with installing FreeNAS from scratch with CPU=N will be exactly the same as the VM you wind up with if you have a FreeNAS VM and you change the value to N. If it's going to fail, it will fail in both cases.

I run FreeNAS as a VM on ESXi for years and change setting all the time, not only on FreeNAS VM but on all VMs, to fine tune resources utilization.

Yeah, but in most VM's you are not doing PCIe passthru, which I will state from experiences with FreeNAS can be twitchy about changing the number of CPU's.

 

[melloa]

Yeah, but in most VM's you are not doing PCIe passthru, which I will state from experiences with FreeNAS can be twitchy about changing the number of CPU's.

If that happens, never did with me. In fact, just because we were talking about that, I decided to check my settings and noticed that I was using 8 CPUs and 92GB RAM. Changed to 4 CPUs and 60GB RAM to save my resources and still going to reduce CPUs:

Note that I only use FreeNAS as a NAS.

 

[Pitfrr]

I've never thought that the CPU count could have an impact and I haven't been playing much with changing vCPU on my FreeNAS VM either... so I never experienced that.

Yeah, but in most VM's you are not doing PCIe passthru, which I will state from experiences with FreeNAS can be twitchy about changing the number of CPU's.

I'd be curious on why a CPU change could have an impact (and why PCIe passthru makes it more sensitive)? Wouldn't a CPU change be completely passive for the OS? Like on bare metal, if you change your CPU it shouldn't have any impact on the OS (or am I wrong?), I'd expect that in a virtualized environment it would be the same?

Or is this not really of concern anymore:

some operating systems install one kernel if you only have a single core, and a different kernel for multiple cores. [...] In the modern era, since most modern CPU's in the last decade have two or more cores, this isn't as big a deal.

Typically memory and CPU would be parameters I'd easily play with (when the VM is down) to balance resources between VMs (never really had to do it so far ) and I wouldn't have expected it to have any impact...