Virtualized FreeNAS 11 on FreeNAS 11

[lungj]

Hello all! I'm trying to store multiple XenServer VM disk images shared via NFS but need the data to be encrypted at rest. My threat model is to deal only with physical theft of the drives or server. Currently using Solaris for the job, but I have some hardware incompatibility. 25-disk storage server runs in a shared closet that I have limited access to. Unfortunately, FreeNAS' rekeying process scares me and the need to reboot the server after a disk replacement is not ideal. Also, native ZFS encryption isn't ready for prime-time in FreeNAS (or possibly even ported).

What I'm thinking of doing is running FreeNAS 11 on bare metal with access to the hardware HBA with unencrypted data. This should give me all the nice ZFS correctness guarantees about the data written to disk. Then, I plan to install FreeNAS as a Bhyve VM and then use FreeNAS' FDE in the VM, presenting the underlying pool-protected storage as a single drive. Since this VM will live on some ZVOLs, I believe this means that individual disk failures will be hidden from the virtualized FreeNAS but I can transparently (to the VM) hot-swap drives, managing the underlying pool from the bare-metal FreeNAS. Thus, I can avoid the rekeying process.

What I am concerned about is whether I will still get the correctness guarantees of ZFS if I run my storage in this fashion. I know ZFS requires the underlying storage to not "lie" about when and what data is written. I'm guessing I may also see a performance increase since the documentation states that performance may suffer when running FDE on systems with more than 8 disks. Since that doesn't refer to throughput of individual disks or the pool capacity, I'm guessing this has to do with the fact that disks are keyed differently. I do not need the additional protection afforded by FreeNAS re: disposing of disks; I am okay with running a magnet over non-functioning disks and then taking a sledgehammer to them.

So does a virtualized FreeNAS 11 with FDE on FreeNAS 11 bare-metal provide the same integrity guarantees as FreeNAS 11 on bare-metal? Must I avoid scrubbing from within the VM? Are there any settings I need to set? Anything else I should be aware of for this setup?

 

[lungj]

P.S. Thanks in advance (I can't edit my last message because the forum is flagging my change as spam)!

 

[Chris Moore]

and the need to reboot the server after a disk replacement is not ideal.

Why would you need to reboot?

 

[Chris Moore]

Overall, I would say this is a bad idea. You should not run another instance of FreeNAS virtualized within FreeNAS.
There may be some other way to accomplish a similar functionality, but this specific plan sounds like it would fail badly.

 

[jgreco]

https://forums.freenas.org/index.ph...nas-in-production-as-a-virtual-machine.12484/

 

[lungj]

@ChrisMoore Thanks for your input. I thought there was a shutdown involved (or at least an off-lining of the target) before performing the replacement when using encrypted pools/volumes. Guess I was mistaken. In any case, the possibility of data loss during a replacement is not great from an uptime and headache perspective -- having to monitor disk health (I'm at a startup for which I am the defacto IT person; downtime and mindshare detracts from my primary role). Right now, I let Solaris' ZFS hot spares do their thing and only check in on the server once in a while. I've never used hot spares in my other FreeNAS boxes and don't know how they interact with FreeNAS' use of geli which doesn't itself require a rekey for a new disk.

@jgreco Thanks; I've read the link, but as far as I can tell, the reasons for the warnings on the page are to warn people against relying on the virtualized FreeNAS for providing protection of one's data. I'm primarily trying to use the virtualized FreeNAS for the encryption and NFS sharing. I would expect the lack of things like SMART data are unimportant if we assume the underlying disk to be error-free (which I should be able to, given that a supposedly-error free ZVOL-backed disk is the underlying storage of the VM. I specifically replaced the RAID card in our server that presented disks as a series of RAID0s with an LSI card for JBOD to provide that SMART data and things like TLER control).

Though it sounds like this particular setup is a no-go, I'm still hoping someone will have a suggestion for using FreeNAS as an encrypted NFS share :)

 

[jgreco]

So I think you're probably going about this kinda wrong. Fine if you want to use FreeNAS for the pool capabilities. You probably want to run a virtualized FreeBSD on top of it with pretty standard disk encryption and NFS.

There are some examples at the bottom of the geli man page. I use one of them whenever I have a need for an NFS VM that doesn't store plaintext on disk, and it actually requires the manual entry of a key at the console on reboot, so an unscrupulous cloud provider can't just snarf the data at rest. There are several possibilities depending on your specific needs.