I agree - I am quite happy VMWare anything user but to have my FreeNAS box do just a little more is a temptation I cannot resist!
Looks like the max memory I am able to run with is 3520M - is it ringing any bells? I am using 12M of video memory on that host as well.
If memory serves me right with Virtualbox, they are emulating a chipset from years ago and, at that time, was limited to 4GB minus 512MB for reserved space. Of course, this information really won't help at all because, even *if* it explained the problem, there is no fix for it unless you are a programmer and are going to try to change chipsets on your own (which I'd imagine is a HORRENDOUSLY large task).