FreeNAS VM / Research

[nateynate]

First post, been researching a long time.

Home server, small storage. Running ProxMox on a Dell t3600. 64GB ECC RAM, 500GB dual SSD / ZFS mirror for OS drive.

HP LSI 9212 4i PCI-e card connected to two WD Red 4TB NAS drives in a ZFS mirror. The LSI controller passed through directly to the FreeNAS VM and set up as ZFS mirror. Host does not see the drives or controller. FreeNAS VM is dedicated 4 cores and 24 GB of RAM. Planning on backing the pool up via Rsync offsite.

This is a small home / backup NAS and seems to work well.

Advice? Things to watch out for? I have read all the posts and guides about doing this kind of setup. Tried my best to make sure all bases are covered. Just looking for "hey watch out for this", or "make sure you do this".

I will set up periodic snapshots etc.

Any advice / discussion appreciated.

N

 

[nateynate]

I should also mention the WD drives are passed direct with no RAID into the FreeNAS host and show up correctly in SMART O( smartctl -i) and via the FreeNAS camcontrol commands.

 

[kdragon75]

I'm guessing you are running this on ESXi? What's your physical CPU?

For the FreeNAS VM be sure to use VMXNET3 vNICs.

 

[Chris Moore]

I understand your confusion. Difficult post to read.

I'm guessing you are running this on ESXi? What's your physical CPU?

No. ProxMox.

Running ProxMox on a Dell t3600. 64GB ECC RAM, 500GB dual SSD / ZFS mirror for OS drive.

OP expects us to look up the model computer and guess about the hardware that is in it.

Being from Mississippi myself, I know that you should be able to speak in complete sentences instead of what ever that was.
If you have a question, please try to articulate that question rather than expecting us to just read the gibberish you wrote and create commentary.

You should take a moment to review the Forum Rules: https://forums.freenas.org/index.php?threads/updated-forum-rules-4-11-17.45124/

When you post, you are supposed to tell us explicitly what hardware you are using and you should have a specific question.

 

[HoneyBadger]

Advice? Things to watch out for? I have read all the posts and guides about doing this kind of setup. Tried my best to make sure all bases are covered. Just looking for "hey watch out for this", or "make sure you do this".

I will set up periodic snapshots etc.

Any advice / discussion appreciated.

N

My obvious first question is "why not just ZFS directly from ProxMox?" I realize ZFSonLinux isn't the same thing, but if it will fill your needs (home backup and media sharing) then why not save the resources?

Next, make sure you flash that HP card to IT mode since it probably comes with IR from the factory. Neat card though for those who want just four more SATA ports.

Final thought; ProxMox/KVM isn't as well tested as ESXi here. Might not find a lot of reference material. But no reason it shouldn't work if the same guidelines are followed.

 

[nateynate]

I understand your confusion. Difficult post to read.

No. ProxMox.

OP expects us to look up the model computer and guess about the hardware that is in it.

No. See below.

Being from Mississippi myself, I know that you should be able to speak in complete sentences instead of what ever that was.
If you have a question, please try to articulate that question rather than expecting us to just read the gibberish you wrote and create commentary.

No need for a personal bash. I was short and to the point. I see many other posts on these forums that fall in the category you describe, but this one does not. I also do not see you personally attacking all those posters, and don't know why you chose to do it here.

You should take a moment to review the Forum Rules: https://forums.freenas.org/index.php?threads/updated-forum-rules-4-11-17.45124/

When you post, you are supposed to tell us explicitly what hardware you are using and you should have a specific question.

I did have a specific question: It was "Are there any other things I am overlooking, or gotchas that I haven't thought about." I also mentioned advice, for which this forum is known. The intent was create a healthy discussion. Yes, I know there are other posts about this subject. I have read hundreds of them. Believe me, I have done my research.

I also posted the specific hardware : Dell T3600 workstation with 64 GB ECC RAM. HP LSI 92124i PCIe HBA card. 4TB WD Red NAS Drives. Dual WD 500 SSD for the OS and VM storage.

True, I did leave out the exact CPU specs, but that shouldn't create the need to bash me. It also is not entirely relevant to the discussion, as any processor available in that line of machines has the necessary capabilities to do what I am describing.
Its an Intel(R) Xeon(R) CPU E5-2660 @ 2.20GHz .

I didn't leave much to guess. I explained about all that was needed to answer my question, which was intended to induce some discussion and possibly bring up something I might have missed in my many days of research on the project.

It's disappointing that in a forum of experienced folks that personal attitude and attacks can take away from the great help that can be found here. If you were interested in answering my question, or having an informed discussion about the process I described you could have done it with a simple question. Like asking for the needed information without making disparaging comments. I would have gladly replied with any answers necessary.

 

[nateynate]

My obvious first question is "why not just ZFS directly from ProxMox?" I realize ZFSonLinux isn't the same thing, but if it will fill your needs (home backup and media sharing) then why not save the resources?

Next, make sure you flash that HP card to IT mode since it probably comes with IR from the factory. Neat card though for those who want just four more SATA ports.

Final thought; ProxMox/KVM isn't as well tested as ESXi here. Might not find a lot of reference material. But no reason it shouldn't work if the same guidelines are followed.

I do know about ZFS on Linux and have used it successfully in the past, and that's a good point. I run 5 or so FreeNAS boxes, some built to specs found in discussion here (cyberjock, etc) and some from iXsystems. I just like the interface and features, and its less stuff for me to have to set up manually. I use a couple jails, and other odd things on the FreeNAS systems too.

I did flash the card, and it works great. The SATA ports are great for not having to use a breakout cable.

I have a lot of experience with some enterprise installations of vSphere / ESXi as well. I'm using ProxMox as an alternative since a friend turned me on to it. I like the interface and features and simplicity of it as well. I also like that its built on Debian. I do like ESXi and have an installation of it here as well, just going a different route with this install.

Thanks a ton for the help and advice! I appreciate it!

N

 

[HoneyBadger]

I do know about ZFS on Linux and have used it successfully in the past, and that's a good point. I run 5 or so FreeNAS boxes, some built to specs found in discussion here (cyberjock, etc) and some from iXsystems. I just like the interface and features, and its less stuff for me to have to set up manually. I use a couple jails, and other odd things on the FreeNAS systems too.

Fair enough, if nothing else then "consistency" is a point in its favor. You'll just need to mind the RAM usage and probably limit the arc_max on your ProxMox install, as by default it's going to be set to 1/2 the system RAM and I can't see you needing quite that much. Maybe 16GB on the ZoL setup, give 24GB to the FreeNAS VM, and then you still have 24GB left for actual VM use.

I did flash the card, and it works great. The SATA ports are great for not having to use a breakout cable.

Awesome. It does sound like you've done your homework, dotted the T's and crossed the I's. (Hey, wait a minute.)

I have a lot of experience with some enterprise installations of vSphere / ESXi as well. I'm using ProxMox as an alternative since a friend turned me on to it. I like the interface and features and simplicity of it as well. I also like that its built on Debian. I do like ESXi and have an installation of it here as well, just going a different route with this install.

Thanks a ton for the help and advice! I appreciate it!

N

I'm a big fan of ProxMox/KVM myself, I'm just noting that it isn't nearly as popular as VMware and so you'll be a little more on your own if things go sideways, and you'll have to do a little more work to optimize it.

And since I'm assuming you actually want to run more than just the FreeNAS VM on your host, check out this blogpost here for some benchmarks of ZVOL vs QCOW2 with KVM:

http://jrs-s.net/2018/03/13/zvol-vs-qcow2-with-kvm/

 

[Ericloewe]

<moderator hat>
No need to make a mountain out of a molehill, everyone.

@nateynate: Your post was a bit vague about the use case - that's something we like hearing a lot about and it's where much advice typically originates. It's also a good idea to name at least the processor family, so we have a general idea.

@Chris Moore: There's no need to be so harsh on the OP. Although some of the points are a bit unclear, the post is far from gibberish. Next time, please ask and give people the chance to clear things up.
</moderator hat>

 

[kdragon75]

Physical CPU core count can be a huge factor in the vCPU count recommendation. 4 cores is a LOT on a host with only 8 physical cores. This could be detrimental to performance depending on the other VMs and core counts you would like to run. For FreeNAS I always recommend 2 vCPU cores to start with unless you have data that shows you NEED more.

 

[nateynate]

@kdragon75 : Good point. Apologies for leaving out the other uses of the host.

FreeNAS will be tied for the most important VM on the host, with the other being a pFSense firewall, which runs with 2 virtual cores and 4GB of RAM. There are 2 other VMs, both small linux machines doing nothing intensive.

Do you recommend I lower the core count for my FreeNAS install? I am ok with doing that, but also ok with it using 4 cores as it's likely anything else I install will be minimal. FreeNAS will by far be the most intensive workload on the host, and I want to make sure it has what it needs. I am not doing dedup or anything like that and my existing FreeNAS install (which I am replacing with this VM for memory (non-ecc) and hardware reasons) is a similar machine. It's currently an OptiPlex 990 with a pretty old dual core i5.

Thanks for the great advice!
N

 

[kdragon75]

The issue is that you can only run 8 (16ish) vCPUs in parallel. To illustrate imagine you have a quad core host @2GHz. Two 2 core VMs would run happily at full speed. Now if you have a 3 core and a 2 core, only one can run at a time effectively cutting you clock speed in half while 1 or 2 cores depending on the VM executing during that clock cycle site idle or "ready" with nothing to do.

This is why VMs should always be configured with the minimum core count to do the job needed. It just make the job of the VM CPU scheduler much easier while also allowing for higher consolidation ratios, better use of power, and an overall faster host.

 

[nateynate]

@kdragon75 : gotcha. This makes sense to me.

So in my case, with 8 real cores and 16 virtual cores: I assume I would hit the swapping you describe when running 10 virtual cores? (4) on FreeNAS, (2) linux 1, (2) linux2, (2) pFSense?

It sounds like based on your description, that even if the cores are multi-threaded then I can still not assign more virtual cores than there are real cores without hitting this limit. I would guess this counts even if the host(s) assigned those extra cores were idle. If this is the case, I could set my linux machines easily at 1 virtual core, since they are minimal machines anyway and are almost always idle (one runs BIND, the other used for occasional web dev). Likely could to the same with pFSense.

Thanks for this. Interesting.

N

 

[HoneyBadger]

It's a little more complicated. While hyper-threading/SMT lets you execute two threads on one physical CPU, many resources are shared between them (eg: cache, instruction load, execution) so if the loads are too diverse, they'll have to wait. If Thread1 needs to do an instruction load and Thread2 is running an operation, they can execute simultaneously - but if they both need to execute, someone's waiting.

Now with that said, if your VMs aren't anywhere near a "sustained, simultaneous load" then you can oversubscribe your CPUs pretty hard. But I'd make that decision based on your physical cores (8).

KVM doesn't have the same easy options for CPU reservation that VMware does, but you could consider setting core affinities of your VMs so that they aren't being shared.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/virtualization/ch33s08

Example: Pin FreeNAS VM to 0/1, pfSense to 2, Linuxes to 3/4, and then you can create VMs that are allowed to share 5/6/7.

 

[nateynate]

It's a little more complicated. While hyper-threading/SMT lets you execute two threads on one physical CPU, many resources are shared between them (eg: cache, instruction load, execution) so if the loads are too diverse, they'll have to wait. If Thread1 needs to do an instruction load and Thread2 is running an operation, they can execute simultaneously - but if they both need to execute, someone's waiting.

Now with that said, if your VMs aren't anywhere near a "sustained, simultaneous load" then you can oversubscribe your CPUs pretty hard. But I'd make that decision based on your physical cores (8).

KVM doesn't have the same easy options for CPU reservation that VMware does, but you could consider setting core affinities of your VMs so that they aren't being shared.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/5/html/virtualization/ch33s08

Example: Pin FreeNAS VM to 0/1, pfSense to 2, Linuxes to 3/4, and then you can create VMs that are allowed to share 5/6/7.

Fascinating. Very good information. I think in my use-case, I'll likely never hit barriers load-wise where this will ever come into play. The VMs I have running now will likely be the only ones, since this setup basically runs my home network. Most of the time all the machines including FreeNAS are essentially idle. The exception would be when copying files to the FreeNAS machine and that would likely be the biggest load.

I do want to set up periodic snapshot tasks, so that might need some consideration as well as far as load goes.

Thanks for the great advice and info!
N

 

[nateynate]

@HoneyBadger : My thoughts here are I just want to make sure my FreeNAS has what it needs. I don't mind if other VMs (or FreeNAS for that matter) have to wait a little on processing, as they will never be too intensive. I just want to make sure the FreeNAS box is as stable as possible and make sure my data is as consistent and error-free as possible. I'm really using the FreeNAS for backup/storage of some studio multi-track session files and my family stuff, like pics.

The current machine I am using (the old Opti990) has three 1TB drives in RAIDZ1 (I know... I know..). That was the purpose of this exercise; to build a better-suited machine, even though that one has been running faithfully for several years. Another added benefit was that I consolidated two machines into a single one. I was using another Opti990 for a Xen hypervisor with the other minimal VMs on it. FreeNAS was it's own physical server until I moved into the Proxmox setup.

 

[jgreco]

The issue is that you can only run 8 (16ish) vCPUs in parallel. To illustrate imagine you have a quad core host @2GHz. Two 2 core VMs would run happily at full speed. Now if you have a 3 core and a 2 core, only one can run at a time effectively cutting you clock speed in half while 1 or 2 cores depending on the VM executing during that clock cycle site idle or "ready" with nothing to do.

This is why VMs should always be configured with the minimum core count to do the job needed. It just make the job of the VM CPU scheduler much easier while also allowing for higher consolidation ratios, better use of power, and an overall faster host.

This is correct, and usually for a typical home user fileserver-only FreeNAS you are not that likely to exceed two cores being needed that often. Better to set it to two and then find out you need to add a third (or fourth).