VirtualBox guests lock-up

[petr]

Hi,

I am in a process of migrating to another machine and during my testing, I've found a few issues with VirtualBox jails template.

On the new machine, I've installed FreeNAS from scratch and also added VirtualBox from jail template, no changes in settings.

I then went on and installed Ubuntu 12.04 as a guest. The installation seemed to run fine, but the process froze close to the end, using all available CPU resources. I could still pause/resume/suspend the guest via the admin panel, but the guest itself was frozen. It booted up fine after reboot and I was able to use it for a moment but it keeps freezing in random places.

I've tried the process twice, installing on different physical pools with exactly the same result. Furthermore, I've booted up one of the guests and selected Memtest86 as a boot option to see if even a low-level software would cause a freeze - and after 20m, the memtest locked up. To be precise, it's not a full lock up, the WallTime in memtest does update itself every 10-20m, but for all intents and purposes it's a lockup - as there is any visible progress in the test percentage.

I would like to add that FreeNAS is running fine on its own and I've tested the machine by installing Ubuntu 12.04 on it directly and running the Memtest86 and stress process for days without any issues, so I doubt that there is an issue there.

Also, when I set up the VM, I've noticed that nested paging was enabled, even when my CPU does not support it. I've disabled it in VM settings

Lastly, I am running very similar setup in terms of FreeNAS + VirtualBox + Ubuntu on another machine without any trouble, the only difference is that the other machine has newer CPU with more capabilities.

"Working fine" CPU: i5 2300
Lockup-CPU : X5355

I've double-checked if the VT-x is enabled on the problematic machine and all options seem to check out. Are there any more settings to check in BIOS?

Here is a screenshot of the Membest86+ showing some CPU info. I've taken the second screenshot when the UI updated itself. We can see that the view updated the walltime but the tests did not move.


When I look at the output of the top command, I can see VBoxHeadless take 100% CPU:
5585 1001 29 20 0 4395M 4167M uwait 5 52:06 100.00% VBoxHeadless

I am also seeking help on VirtualBox forums: link

 

[petr]

First finding - found out that this is somehow memory related. As you can see on the picture, the memtest is trying to access memory outside what was allocated for the host. No idea why that would be happening. Now running a test with 2048 allocated and seems to be running without trouble..

.. and that had finished without trouble. Stepping up to 3072MB

 

[rs225]

Do you have the latest BIOS for the system? You may be running into a CPU errata. If there is no BIOS more recent than 2009, then the question is whether FreeNAS supports the FreeBSD microcode update package.

 

[petr]

Do you have the latest BIOS for the system? You may be running into a CPU errata. If there is no BIOS more recent than 2009, then the question is whether FreeNAS supports the FreeBSD microcode update package.

I think so - I will flash the latest one just in case again.. forgive my ignorance though - what is the FreeBSD microcode update package?

EDIT: I am running the latest version, dated 28/08/2012.

 

[rs225]

Have you tried running with VT-x off?

With a 2012 BIOS, it is pretty likely any important microcode update is already there. The FreeBSD package is the other way to get microcode updates loaded. It works with update bundles that can be downloaded from Intel and AMD.

 

[rs225]

I notice you are running Memtest86+ v4.20. I think that version has at least one serious bug. Can you try 5.01?

 

[cyberjock]

I thought I read somewhere a while back that you should never do a memory test of a VM because it not only doesn't test what people think it testing but the virtualization layer just fubars the heck out of the program.

In essence, don't do this.

 

[rs225]

Yeah, running a memory test in a VM is not a true memory test. But, it shouldn't test memory that isn't there, and figuring out that it works fine with 2GB in the guest might also end up being a clue to the Ubuntu problem.

 

[petr]

I thought I read somewhere a while back that you should never do a memory test of a VM because it not only doesn't test what people think it testing but the virtualization layer just fubars the heck out of the program.

In essence, don't do this.

I agree that it does not have any bearing on actual memory results - but either way, it should not lock up or freeze the VM!

I've got an update what happens with my VMs when I assign them more than 3gb of RAM (could probably handle a little more but 3GB worked and 4 did not in my testing).

The Memtester seems to incorrectly detect the RAM - as the test reads "Testing: 4096M - 4555M 4043M" - meaning that it does see the 4043M limit I set but for some reason continues anyway and locks up / grinds to a halt. The same is happening in normal Ubuntu VM - although harder to reproduce, similar net effect appears after use.

I now have all the VMs set to 3072MB of RAM and they all work absolutely fine - should be enough for my purposes if I clean the VMs up a bit anyway - curious - any of you guys are hitting similar limits?

 

[cyberjock]

Yeah, running a memory test in a VM is not a true memory test. But, it shouldn't test memory that isn't there, and figuring out that it works fine with 2GB in the guest might also end up being a clue to the Ubuntu problem.

Sure. But if they already know its a bug because of the virtual hardware and they include that in their justification for posts that say "don't test RAM in a VM" is that not good enough? Your argument doesn't seem to be valid to me at all if they have said "don't use it in a VM, we aren't responsible if you use it in a VM".

 

[Ericloewe]

RAM tests are very low-level. The kind of stuff that the hypervisor (or any OS more advanced than DOS!) will look at and say "Nope." and terminate the process.

 

[petr]

Sure. But if they already know its a bug because of the virtual hardware and they include that in their justification for posts that say "don't test RAM in a VM" is that not good enough? Your argument doesn't seem to be valid to me at all if they have said "don't use it in a VM, we aren't responsible if you use it in a VM".

Yes - I get your point but in my opinion, if a virtualisation software is actually providing proper abstraction, the worst-case scenario would be the VM lockup/crash (what I am seeing).

I am not arguing that there is a problem only based on the memtest alone - I was able to reproduce the same behaviour in a fully-up-to-date Ubuntu 12.04, it was just much quicker to test in the memtest to get to the correct memory to use. Then, once I re-applied the same memory limits that worked on my Ubuntu install, it stopped randomly freezing and has been solid for a few days.

 

[petr]

RAM tests are very low-level. The kind of stuff that the hypervisor (or any OS more advanced than DOS!) will look at and say "Nope." and terminate the process.

I understand - I am not arguing about this, I only used memtest to work-out the memory limit - which I then verified using a real OS. When I keep the allocated RAM below 3072M, I have been actually looping the memtest without any trouble / crashes.

 

[cyberjock]

Yes - I get your point but in my opinion, if a virtualisation software is actually providing proper abstraction, the worst-case scenario would be the VM lockup/crash (what I am seeing).

But that's just it... virtualization has never provided a true 100% abstraction. There's tons of shortcuts and flat-out omissions. It's very possible that RAM tests (along with a lot of other things) are never going to work as a result. The virtualization layer is swiss cheese. ;)

There's lots of situations where virtualization is nice (like for testing a new copy of some software) and there's places where virtualization is terrible (like testing some new OS that hasn't been even been validated to be compatible with virtualization.. look at the 9.3 beta testers, many of them cannot do a "reboot" if they are using 9.3 in a VM).

 

[petr]

But that's just it... virtualization has never provided a true 100% abstraction. There's tons of shortcuts and flat-out omissions. It's very possible that RAM tests (along with a lot of other things) are never going to work as a result. The virtualization layer is swiss cheese. ;)

There's lots of situations where virtualization is nice (like for testing a new copy of some software) and there's places where virtualization is terrible (like testing some new OS that hasn't been even been validated to be compatible with virtualization.. look at the 9.3 beta testers, many of them cannot do a "reboot" if they are using 9.3 in a VM).

Regarding the reboot - do you mean that the ability of the host machine to reboot got compromised? If you mean the guest, then of course - that is more than possible! I did not mean that virtualisation means that everything works 100% fine - I am sure there is a large number of not implemented features across the board. What I am saying is that whatever a VM guest does, the worst case scenario is a crash of that VM - not anything that would compromise the host. Otherwise I would feel very uncomfortable running jails / VMs on my FreeNAS machine!

But all of that is beside the point - I am willing to accept that using memtest to determine the RAM limits may have been flawed but I got lucky - the working limits for memtest directly translated to "safe" memory amounts for the full Ubuntu 12.04 guest.

Back to my main question then - any of you experiencing trouble if you allocate 4GB or more to a VirtualBox guest? My main symptoms are occasional stall - mainly bound to memory access (CPU-only stress can run pretty much forever, any more memory-involved operations lead to a lockup within a day).

 

[cyberjock]

The guest won't reboot. You have to do a shutdown, then power-on.

There have been a few bugs that have gotten outside of the VM and affected the host. No doubt these can often be turned into security vulnerabilities (it's bad when a host can be pwned by a guest). If memory serves me right the last one that VMWare had was in VMWare 8.x.

I personally have had no problems with guest behavior with as much as 16GB of RAM devoted to a Linux Mint 17 VM.

 

[rs225]

Sure. But if they already know its a bug because of the virtual hardware and they include that in their justification for posts that say "don't test RAM in a VM" is that not good enough? Your argument doesn't seem to be valid to me at all if they have said "don't use it in a VM, we aren't responsible if you use it in a VM".

I don't think they have said don't use it in a VM. v4.20 has some known bugs, and certain distributions of v4.20 don't work right because of a compiler change. However, it is obvious (to us) that it isn't going to be very conclusive in terms of whether your memory chips are actually good or bad. :)

I just think it is an interesting problem, and if it turns out his VMs are doing something odd only on certain processors with >3GB of RAM, trying to figure that out might be worth the effort. Maybe not for FreeNAS itself, but VirtualBox has seen some very obscure problems found and fixed, and I'm sure many of them start out seeming like user error.

OP(petr):This should probably be reported back to VirtualBox with what you know so far and see if they want more information or already know the cause. I would include output from VBoxManage list hostcpuids and VBoxManage list hostinfo, and VBoxManage -v

 

[cyberjock]

I don't want to belittle virtualbox, but it seems to have very peculiar problems with virtualizing. I prefer VMWare Workstation because it seems to work better. Not unexpected since the virtualbox software isn't a commercial product. I use virtualbox when I have to (such as when in a jail), but I gladly use VMWare Workstation if I can. ;)

 

[petr]

I don't want to belittle virtualbox, but it seems to have very peculiar problems with virtualizing. I prefer VMWare Workstation because it seems to work better. Not unexpected since the virtualbox software isn't a commercial product. I use virtualbox when I have to (such as when in a jail), but I gladly use VMWare Workstation if I can. ;)

I agree - I am quite happy VMWare anything user but to have my FreeNAS box do just a little more is a temptation I cannot resist!

Looks like the max memory I am able to run with is 3520M - is it ringing any bells? I am using 12M of video memory on that host as well.

 

[petr]

I just think it is an interesting problem, and if it turns out his VMs are doing something odd only on certain processors with >3GB of RAM, trying to figure that out might be worth the effort. Maybe not for FreeNAS itself, but VirtualBox has seen some very obscure problems found and fixed, and I'm sure many of them start out seeming like user error.

OP(petr):This should probably be reported back to VirtualBox with what you know so far and see if they want more information or already know the cause. I would include output from VBoxManage list hostcpuids and VBoxManage list hostinfo, and VBoxManage -v

I will try again, when I have more specific information. I've tried to get help initially but as BSD is not officially supported OS they seemed reluctant to help. Best would be to try out Vbox on another OS on the same machine but I do not think I've got time to do that :/ .. .especially when the 3.5GB limit I've found is sufficient for my purposes.