Virtually FreeNAS ... an alternative for those seeking virtualization

Status
Not open for further replies.

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
[---- 2014/12/24: Note, there is another post describing how to create a full-sized FreeNAS VM directly managing a bunch of disks. THIS post is aimed at people wanting to use FreeNAS to provide a little basic file sharing off of virtual disks on shared VM storage. ----]

There's been a lot of firefights over virtualizing FreeNAS over the past few years. These are well-intentioned and intended to help people not get into unrecoverable catastroph**ks.

The typical argument goes that "well making a VM is simple" - which is more or less true enough. However, your typical web server VM or other single disk VM is a much simpler beast than most FreeNAS systems. People come expecting to be able to allow FreeNAS to "do ZFS" off a bunch of disks. The devil is in the details, and what happens when things go wrong. The only sane way to do that is to attach the disks to the FreeNAS VM directly, via PCI-Passthrough, as documented here.

However - a number of discussions have also suggested that not everyone necessarily wants or needs a massive FreeNAS system with lots of storage and lots of disks. This has also been discussed in the forums, but here's a new resource to help you understand and accomplish this.

A Small Virtualized FreeNAS VM

So, consider the case where you have a big virtualization host in your office with a lot of storage for VM's. You also have some PC's or Macs or WhatHaveYou's, and you'd like to be able to share documents and maybe a little scratch space. You really don't want a big huge massiveNAS. You're not looking to serve VM disk files to ESXi, you're not looking to do anything super-intensive. You just need a small, competent NAS.

FreeNAS can do this for you, and it comes complete with my blessing, as long as you follow the guidelines herein and understand the caveats as well.

To do this, rather than fighting the virtualization and trying to figure out how to get what we want by subverting the platform to work as we want, we're going to play strictly by the virtualization game and do everything RIGHT.

Key Concept 1:

The virtualization host requires redundant datastores. No RAID0 datastores for the NAS VM files or for the NAS VM virtual disks. One of the problems with virtual machines is that the I/O subsystem of a VM will stall when the host hypervisor experiences an unrecoverable storage problem. It can be RAID1, RAID10, RAID5, whatever, so long as the storage is redundant and the VM does not hang on error.

Key Concept 2:

ZFS supports error detection and correction, but it requires multiple disk devices in a vdev to do so (I realize there's an edge case with copies= but let's keep this simple!). You cannot just use a single virtual disk, because doing so means that ZFS will not be able to correct errors, and you may lose data as a result.

You will need at least two VM virtual disks for your ZFS pool. It becomes progressively riskier as the amount of storage and number of VM virtual disks increases.

You CAN put two virtual disks on a single redundant datastore and it'll work, but it'll probably perform poorly. This is sufficient to meet the ZFS requirement for someplace to get redundancy from in the event of bit errors.

Key Concept 3:

If you didn't immediately catch that the implication of combining the two concepts above is that in order to get 1GB of NAS storage you may need 4GB of raw space on the virtualization datastore, please go read those again.

If this is a dealbreaker for you, then your only sane option is to go for the VT-d virtualization option. That is NOT for beginners and requires specific hardware capabilities.

Key Concept 4:

This virtualization technique works on basically any hypervisor that is capable of running FreeNAS as a VM in a stable fashion. There may be caveats. Under ESXi and Proxmox, this is known to be perfectly stable.

Key Concept 5:

Your FreeNAS data disks adopt the reliability characteristics of the underlying VM datastore(s). This should be a "well, DUH" type statement. But among those things, note that FreeNAS loses the ability to monitor the health of the storage, and you are now dependent on monitoring the storage health through whatever means the virtualization and storage platforms offer. If your VM storage goes south, so goes your FreeNAS. Of course, so do all your other VM's, so you probably have other issues at that point.

So here it is. You set up a VM. You play by actual VM rules. You create a small virtual boot disk and then two virtual data disks on datastores that are each redundant. Make sure you've allocated sufficient resources; 8GB RAM is the minimum for FreeNAS regardless of how little you're storing.

Gotchas

ZFS normally supports capacity expansion of a vdev through replacement of a disk with a larger disk and resilvering. You may be tempted to try configuring an existing disk "larger", which the hypervisor will allow, but which will cause a failure and lots of problems for FreeNAS. Do not do this. You can create a new larger virtual data disk and replace the old disk, mimicking the set of actions that'd happen on a physical machine. That works.

Scrubs will drive heavy I/O towards the VM datastores, which may cause undesirable performance impacts. Attempting to correct this through mechanisms such as limits or Storage I/O Control will cause the NAS VM performance to suck. If the I/O demands of a scrub are too high, the only truly viable option is to place the FreeNAS data disks on datastores that are only lightly used for other purposes, or to disable scrubs.

And invariably the biggest gotcha is the user who thinks they can disregard the things I've said in here. For those of you old enough to own your own mistakes and not come here begging for help, good for you! Please experiment and even feel free to share. But I'm probably only going to offer advice and commentary to those who have followed the design concepts I've outlined above. I've explained at least some of the key reasons for what I've said, and variances are not to be made lightly.

Benefits

A high performance RAID controller for the virtualization host may help performance. I'm not actually convinced of that and I don't think it should be a major issue for small NAS type deployments, but it is a good idea for other VM's and may help mitigate the stresses on datastores during scrubs. Most people seem to put a high performance RAID controller in their virtualization hosts anyways.

Of course, a VM solution offers you the ability to assign resources as needed. Do not under-resource your FreeNAS box, but definitely feel free to experiment!

Communications through the virtualization host's virtual switch to other VM's of course carries with it the possibility of communicating at greater-than-gigabit speeds.
 
Status
Not open for further replies.
Top