FreeNAS 11.0 VM Virtual CPU assignment issue

[Windows7ge]

Of every forum and guide I've searched, everything says my hardware is compatible and that my setting are correct yet the VMs that I create regardless of how many Virtual CPUs I assign report 2. One per physical processor. I learned that this type of issue is due to a number of features that the hardware must support and settings that need to be enabled in the BIOS including VT-x, VT-d & feature support for Unrestricted Guest (UG) and EPT.

I have checked and verified that VT-x & VT-d are supported and enabled, Going into terminal I believe I verified that I have Unrestricted Guest and EPT and I don't see any of the hardware being incompatible with virtualization yet despite this the virtual machines continue to display 1 core per CPU regardless of what I set it to so I'm at a loss.

Hardware:
CPU: 2x Intel E5 2670
Motherboard: ASRock EP2C602-4L/D16
RAM: Kingston 128GB 1600MHz ECC Unbuffered

Software:
FreeNAS-11.0-U3 (c5dcf4416)
VM software is pre-included and is refereed to as "bhyve" or BeeHive.

 

[dlavigne]

I wonder if this fixes it: https://github.com/pr1ntf/iohyve/issues/259.

If so 11.1-BETA1 (slated for Oct 23) should have that fix.

 

[Windows7ge]

I wonder if this fixes it: https://github.com/pr1ntf/iohyve/issues/259.

If so 11.1-BETA1 (slated for Oct 23) should have that fix.

Are you talking about upgrading to the 5600 series of CPU or the editing of code the poster mentioned underneath?

 

[dlavigne]

Neither. That fix (hyperlinked to the word disabled in the second comment) is already in the FreeNAS nightly train and will be released in BETA1.

 

[Windows7ge]

Neither. That fix (hyperlinked to the word disabled in the second comment) is already in the FreeNAS nightly train and will be released in BETA1.

Ah, the code being omitted seems to be the error messages the would normally be displayed when a component isn't compatible with the user request (I'm not a script user, feel free to correct me at any time.) I don't receive any error messages (unless I attempt to use more than 50% of my system resources which I expected) it's just I can assign 4, 6, 8, 10, 12, etc to the VM and it will only show 2. 1 per physical socket. Interestingly enough if I assign 1 virtual CPU to the VM it WILL show that the VM only has 1 core from 1 CPU. It seems to me like dual socket boards can complicate things if this is indeed a software bug. In theory if I was using a quad socket motherboard I could assign 4 cores to the VM 1 per physical processor.

If the potential fix is as simple as omitting those lines of code in a single file and I'm to assume restarting the server after saving the edit then I know how to use vi & vim. I could test if this is the fix before waiting for the release of the BETA. I just need to know the location of the file.

If the file to edit is dmesg then I have the same problem as the commentor who posted on June 16 I have dmesg.today & dmesg.yesterday but no dmesg in the /var/log/ directory.

If I should just wait for the BETA then I will. I don't need the VMs for anything performance heavy but I'd like them to at least be quad cores for certain multi-core or multi-threaded loads.

 

[rs225]

How did you verify Unrestricted Guest?

 

[Windows7ge]

How did you verify Unrestricted Guest?

I don't know 100% if this verifies it but I typed this in terminal: grep VT-x /var/run/dmesg.boot
The output was: VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID

 

[rs225]

That looks good. It is plausible that the bug cited above is related to your problem. But it also might not hurt to open a bug report on FreeNAS, asking if it would cause what you're observing. From what I see, the original problem was a spurious warning, no mention of actual altered behavior.

What guest OS are you selecting? Have you tried setting up a VM for 64-bit FreeBSD and see if it has the same problem?

 

[Windows7ge]

That looks good. It is plausible that the bug cited above is related to your problem. But it also might not hurt to open a bug report on FreeNAS, asking if it would cause what you're observing. From what I see, the original problem was a spurious warning, no mention of actual altered behavior.

What guest OS are you selecting? Have you tried setting up a VM for 64-bit FreeBSD and see if it has the same problem?

For me I don't get any error messages like others seem to depict. I can set as many Virtual CPUs as I like (up to 16) to the VM but every time it reports 2 cores. More specifically it says

"Sockets: 2
Virtual processors: 2
Virtual Machine: Yes"
(I forgot to mention this is Windows 10 Professional 64-bit on the VM)

What guest OS?...The OS running on the VM? That would be stated above. Setting up a VM for 64-bit FreeBSD. You mean creating a VM running FreeBSD 64-bit? I have not attempted that however I did try installing Linux Ubuntu 16.04 at first the installer didn't want to cooperate, eventually I got it to install however a key bootloader kind of file didn't install. Some research taught me that Ubuntu doesn't like installing properly on VM's so after restarting the VM it kept booting into SHELL> and no forum online gave instructions for my exact copy of Linux to get it going again. I SHOULD run the Linux installer and at least run the "Try Ubuntu" option if I setup that VM with more than 2 cores and it does show me that it has them then I know the problem is that Windows 10 doesn't like being visualized on FreeNAS the problem not being FreeNAS itself. It's an old version of windows 10 Build 1604. I can try updating it to the latest version and see if its behavior changes for the better.

If you like I can conduct both of these tests and get back to you with the results. It would verify if the issue is either of the VM OS's or FreeNAS itself. Since it's a test of both Linux & Windows it should be conclusive in its result (to note both versions are 64-bit).

 

[rs225]

I would do whatever quick tests you can, since I'm sure you'll end up needing more info for a bug report.

 

[Windows7ge]

I would do whatever quick tests you can, since I'm sure you'll end up needing more info for a bug report.

I do believe I have come to a conclusion at least to some degree.
Running Ubuntu 16.04 and checking the CPU via terminal with the commands:
top
1
It showed me that whatever core count I requested was being used. HOWEVER, Ubuntu wasn't registering the cores as 1 CPU with x# of cores. It was registering it as xCPUs with 1 core. For example, 4 CPU's each with one core, or 16CPU's each with one core.

This may explain why I cannot get windows 10 to display more than 2 cores. It tells me I have Two Sockets with Two Processors. This means Windows 10 thinks it has Two physical CPU's each with one core. The reason no higher count of Virtual Cores works is because Windows does not support more than Two physical processors. If I was running a server version of windows it may have displayed more.

So it seems the issue isn't as it first appeared. Virtual Cores does not appear to count how many cores you want assigned to each VM. It counts how many "physical" processors you want assigned to each VM. In this case Windows 10's limit is two. Unless an update is released that corrects this then Windows 10 will never be able to use more than two.

 

[rs225]

That makes sense. Perhaps it will be a future option in bhyve.

 

[Windows7ge]

Neither. That fix (hyperlinked to the word disabled in the second comment) is already in the FreeNAS nightly train and will be released in BETA1.

As I mentioned in my discovery above the issue isn't quite as it first appeared. Virtual CPU's isn't counting cores it's counting physical processors. Windows 10's limit is 2 physical CPU's. So i cannot make a VM with more than two cores (On windows, Linux doesn't seem to care how many CPU sockets you want to use). Perhaps in the future the software will be modified in such a way to distinguish Virtal CPU's from Virtual Cores with separate options for each so users who want to virtualize windows with more than 2 cores can.

In the mean time this means I can create up to 8 dual core virtual machines. Any chance you have some fun ideas of what I could do with them?

 

[chris crude]

Any chance you have some fun ideas of what I could do with them?

I run a Linux VM with pi-hole setup. Made a noticible difference on my home network.

 

[Windows7ge]

I run a Linux VM with pi-hole setup. Made a noticible difference on my home network.

I decided to take a look at pi-hole but it didn't do much for my network, left random blocks on web pages where adblocker would remove the blocks completely and actually missed a lot of advertisements. I'd like to try running a DNS or DDNS server though. It'd allow for faster address resolution and prevent DNS spoofing (I think, don't quote me on that). Do you know of any basic software for that I could run on Ubuntu? If it uses a WebUI that's fine.

 

[Sean Coston]

This issue was discussed back when people were playing around with FreeNAS Corral... sorry to bring THAT up.

anyway, the issue was fixed by setting a system tunable:
System --> tunables --> add tunable and then add tunable variable as hw.vmm.topology.cores_per_package
I set value to 4
I set type to "loader"

This allowed me to build and start a Win 10 Pro VM that reports 4 processors when I allocate 4 in the VMs Virtual CPUs setting in FreeNAS GUI for it.
For my processor, I have 4 cores that are hyperthreaded, so the FreeNAS 11 system "sees" 8 threads as CPUs

I've been told that with bhyve, when one allocates virtual CPUs to a VM this makes those threads potentially available to the VM, but doesn't allocate them to the exclusion of their use by the rest of the system or other VMs. For example, I am running 2 VMs - a Windows 10 VM with 4 virtual cpus and a linux 16.04 vm with 4 virtual cpus. They both run perfectly fine, even when I load the Linux box down with a ton of encoding jobs. Also, if the cpus were allocated exclusively - there would be none left for FreeNAS itself or any of my jails, but I do not seem to have any issues.
As for MEMORY allocation for VMs - I think that bhyve actually locks up any memory you allocate to a VM so it is NOT vailable for use by other VMs or processes.

Hope this info helps. I don't know if I am completely accurate about how bhyve allocates cpus and memory, but the above are my observations regarding how my system functions. If you are reading this and I am totally FOS, please educate me and the masses!

sdc

 

[Windows7ge]

This issue was discussed back when people were playing around with FreeNAS Corral... sorry to bring THAT up.

anyway, the issue was fixed by setting a system tunable:
System --> tunables --> add tunable and then add tunable variable as hw.vmm.topology.cores_per_package
I set value to 4
I set type to "loader"

This allowed me to build and start a Win 10 Pro VM that reports 4 processors when I allocate 4 in the VMs Virtual CPUs setting in FreeNAS GUI for it.
For my processor, I have 4 cores that are hyperthreaded, so the FreeNAS 11 system "sees" 8 threads as CPUs

I've been told that with bhyve, when one allocates virtual CPUs to a VM this makes those threads potentially available to the VM, but doesn't allocate them to the exclusion of their use by the rest of the system or other VMs. For example, I am running 2 VMs - a Windows 10 VM with 4 virtual cpus and a linux 16.04 vm with 4 virtual cpus. They both run perfectly fine, even when I load the Linux box down with a ton of encoding jobs. Also, if the cpus were allocated exclusively - there would be none left for FreeNAS itself or any of my jails, but I do not seem to have any issues.
As for MEMORY allocation for VMs - I think that bhyve actually locks up any memory you allocate to a VM so it is NOT vailable for use by other VMs or processes.

Hope this info helps. I don't know if I am completely accurate about how bhyve allocates cpus and memory, but the above are my observations regarding how my system functions. If you are reading this and I am totally FOS, please educate me and the masses!

sdc

This might be a useful solution but I've long since abandoned the need for a VM with more than two processing cores. Plus more dual core VM's vs a quad core adds redundancy depending on the task. From my experience bhyve like Oracle VM Virtual-box will force the resources you assign to the VM to be used for strictly the VM and no other external task but provided you have enough cores high enough core clock and RAM this isn't an issue. I'm currently running 8 dual-core VMs. Each VM is maxing out the CPU at 100% load and each VM has 8GB of RAM. I'm streaming a video from a SMB file share I have setup and everything is operating seamlessly. Now if you're trying this with more restricted resources it may not run as smoothly.

Oh, and also the 8 VM's are running on a Intel 750 series PCI_e SSD so they have a huge amount of IOPS which in theory lowered the load on the remaining 8 cores 16 threads that the file server has to work with. I'm not entirely sure I'm not a genius with this either.

I've been informed the feature of virtualization has only been supported as of version FreeNAS 11 so it is to be expected that some bugs still need to be worked out. Either way I'll keep your work around in mind.

 

[Windows7ge]

System --> tunables --> add tunable and then add tunable variable as hw.vmm.topology.cores_per_package
I set value to 4
I set type to "loader"

Now I'm glad you informed me of this. I started a new VM project that benefits from having as many CPU cores & as much RAM that are/is available. I referenced what you told me. Added the tunable. Restarted the server and now the VM is running 2 virtual 8 core CPU's. It seems both the tunable and byhve interpret the number as threads not physical cores so if I set the cores_per_package to 8 then when I go into byhve if I set the cores to 16 it will create two 8 core CPUs labeled as "Sockets: 2 Virtual Processors: 16" so If I wanted a single 8 core 16 thread virtual CPU I'd have to set the tunable to 16.

Regardless I've happy with the results and I'm glad you revived the thread. This may be patched in a later release of FreeNAS but this is a good fix.