ESXi All-in-One Help

[livegorilla]

Hi,

I'm planning on running FreeNAS as a VM on an ESXi host, and wanted some feedback on my plan. The ESXi host is a Dell R710 with 2xL5640 CPUs, 96GB of RAM, and an H200 HBA with 2x250GB SSDs and 4x3TB hard drives.

ESXi will be installed on a USB flash drive. Since you can't create an ESXi datastore on a USB device, I'll install FreeNAS on a 250GB hard drive I have laying around. I will use a Dell H200 flashed to IT mode and passed through to the FreeNAS VM. Connected to the H200 will be 4x3TB hard drives and 2x250GB SSDs. I will have two zpools. The first will be 2 vdevs of 2 mirrored hard drives each. The second will be the 2 SSDs each in their own vdev (RAID 0). The SSD zpool will be used for fast VM storage shared via iSCSI to ESXi. A portion of the hard drive zpool will also be used for less I/O intensive VMs via iSCSI, with the rest allocated for backups, network AFP shares, etc.

1) Any issues with installing FreeNAS to a hard drive? Is mirroring the boot drive necessary or can I get away without it?

2) Is having 2 separate zpools the best way of doing this?

3) Is there a significant performance hit for using iSCSI to ESXi versus just local ESXi datastores? Would it be better to just have a local datastore on 1 large SSD for all fast VMs and just use FreeNAS for the hard drive zpool?

Any advice is greatly appreciated!

Thanks!

 

[Spearfoot]

Hi,

I'm planning on running FreeNAS as a VM on an ESXi host, and wanted some feedback on my plan. The ESXi host is a Dell R710 with 2xL5640 CPUs, 96GB of RAM, and an H200 HBA with 2x250GB SSDs and 4x3TB hard drives.

ESXi will be installed on a USB flash drive. Since you can't create an ESXi datastore on a USB device, I'll install FreeNAS on a 250GB hard drive I have laying around. I will use a Dell H200 flashed to IT mode and passed through to the FreeNAS VM. Connected to the H200 will be 4x3TB hard drives and 2x250GB SSDs. I will have two zpools. The first will be 2 vdevs of 2 mirrored hard drives each. The second will be the 2 SSDs each in their own vdev (RAID 0). The SSD zpool will be used for fast VM storage shared via iSCSI to ESXi. A portion of the hard drive zpool will also be used for less I/O intensive VMs via iSCSI, with the rest allocated for backups, network AFP shares, etc.

1) Any issues with installing FreeNAS to a hard drive? Is mirroring the boot drive necessary or can I get away without it?

2) Is having 2 separate zpools the best way of doing this?

3) Is there a significant performance hit for using iSCSI to ESXi versus just local ESXi datastores? Would it be better to just have a local datastore on 1 large SSD for all fast VMs and just use FreeNAS for the hard drive zpool?

Any advice is greatly appreciated!

Thanks!

How many VMs do you plan to run? Since you're planning on using iSCSI, you'll want to dedicate at least 32GB of RAM to FreeNAS, more if you can spare it. Be sure to reserve all of the memory when you set up the FreeNAS VM.

I recommend using redundant storage for your ESXi datastore. So if you want ~500GB of SSD-based storage, use a single 2 or 3-way mirror vdev of 500GB drives instead of the non-redundant design you described. Also, bear in mind that iSCSI works best on sparsely populated pools, so you may actually need more storage and therefore larger disks, depending on the number and size of VMs you plan to run.

Whether you use iSCSI or NFS-based datastores, you will need a ZIL SLOG device or else you will suffer abysmally slow transfer rates. Search the forum for recommendations, but a good entry-level choice is an Intel DC S3700/S3710 SSD. Or you can live on the wild side and just turn off synchronous writes on your VM datastores...

Answering your questions:

1) Any issues with installing FreeNAS to a hard drive? Is mirroring the boot drive necessary or can I get away without it?

No, and you can get away without a mirror as long as you backup your configuration. But I recommend installing FreeNAS mirrored on a pair of small, cheap SSDs. BTW, these will need to be plugged in to the motherboard's SATA ports, not the H200 HBA, because you'll be passing the HBA through to ESXi and FreeNAS itself needs to reside on a local datastore.

2) Is having 2 separate zpools the best way of doing this?

Your 2-pool design is fine, provided you use a redundant topography for your SSD-based VM storage pool as discussed above.

3) Is there a significant performance hit for using iSCSI to ESXi versus just local ESXi datastores? Would it be better to just have a local datastore on 1 large SSD for all fast VMs and just use FreeNAS for the hard drive zpool?

I haven't benchmarked the difference, so I can't answer this one; perhaps one of the forum members with more ESXi experience will chime in. My initial impression is that a local datastore would be faster, all else being equal... but I could be wrong. And you would need an additional RAID controller to implement any kind of redundancy on a local datastore, as ESXi doesn't support RAID configurations based on motherboard SATA ports.

Also, note that you can install both ESXi and FreeNAS to a local non-USB datastore, so going that route, using a separate RAID controller for example, is a viable alternative to booting ESXi from a USB stick and storing FreeNAS on a separate disk or disks. See @joeschmuck's 'Dream System' thread for insight about this.

Be sure to set up a separate virtual switch and subnet for your FreeNAS-based datastores, as described in @Benjamin Bryan's excellent article, FreeNAS 9.10 on VMware ESXi 6.0 Guide.

Good luck!

 

[livegorilla]

Thanks for your reply!

I recommend using redundant storage for your ESXi datastore. So if you want ~500GB of SSD-based storage, use a single 2 or 3-way mirror vdev of 500GB drives instead of the non-redundant design you described.

In an ideal world, I would do exactly this, but cost is a major factor. I can't really afford 2 500GB SSDs right now, without compromising elsewhere. My reasoning was that the VMs will be regularly backed up to the hard drive zpool, so I could relatively easily restore them even if my datastore SSDs were to crap out. Thinking more about it, I may go with a single 500GB SSD for now, with the intention of adding another one for a mirror later down the line.

Whether you use iSCSI or NFS-based datastores, you will need a ZIL SLOG device or else you will suffer abysmally slow transfer rates.

I was under the impression that synchronous writes were disabled by default with ISCSI, so I had no intention of enabling them. This is something I need to read up on.

But I recommend installing FreeNAS mirrored on a pair of small, cheap SSDs. BTW, these will need to be plugged in to the motherboard's SATA ports, not the H200 HBA, because you'll be passing the HBA through to ESXi and FreeNAS itself needs to reside on a local datastore.

Again, ideally I would do this, but my ESXi host is a pre-built rack mount server, which makes adding more than one drive directly to the motherboard VERY non-trivial, so I'm pretty limited in that regard.

Also, note that you can install both ESXi and FreeNAS to a local non-USB datastore, so going that route, using a separate RAID controller for example, is a viable alternative to booting ESXi from a USB stick and storing FreeNAS on a separate disk or disks.

Interesting, I wasn't aware of this. What exactly would the advantages of this be? It seems like you're just consolidating the hypervisor, the FreeNAS VM and the ESXI datastore all into a single point of failure. Although, I guess you would probably be using some form of redundancy with this setup.

Thanks for all the advice and resources! I think, for now at least, I'm just going to go with a single 500GB SSD for a datastore. I think I'll also snag one of those PCI-E SSD adapters to test out local datastore vs iSCSI performance.

 

[Spearfoot]

Thanks for your reply!

You're welcome!

In an ideal world, I would do exactly this, but cost is a major factor.

I understand the cost factor very well; I put two daughters through college! :)

I was under the impression that synchronous writes were disabled by default with ISCSI, so I had no intention of enabling them. This is something I need to read up on.

Ummm... you're right! However, the inimitable @jgreco recommends turning on synchronous writes for VM datastores, which means you'd need a SLOG device.

Realistically... provided you're not running a production system, it's okay to run with synchronous writes turned off; I did so for nearly a year.

 

[jgreco]

You're welcome!
I understand the cost factor very well; I put two daughters through college! :)
Ummm... you're right! However, the inimitable @jgreco recommends turning on synchronous writes for VM datastores, which means you'd need a SLOG device.

Realistically... provided you're not running a production system, it's okay to run with synchronous writes turned off; I did so for nearly a year.

This is absolutely and totally a function of whether or not you would be inconvenienced when a VM might be corrupted by the hopefully-incredibly-unlikely event that your filer crashes and reboots. For production use, you need a SLOG. For home lab use, or anywhere that you wouldn't mind restoring from backup, quite possibly not.