Exsi hdd pass through advice

[Girth]

Hi all,

I have had a read through the how to's and understand that it is generally frowned upon, but due to hardware constraints and my own requirements of having a single server I have a few questions.

The server is a HP microserver Gen 8, xeon 1265L, an SSD, 2 1TB test drives and 16gigs of ram running exsi booting from a usb stick. I wanted a virtualised instance of Pfsense, so have installed a dual port intel nic and vt-d to the vm, this occupies the single pcie port, so no additional hdd controller available.

I have two options available for passing the hdd's through to FreeNAS,

1. rdm the hdd's within exsi?
2. Pass the stata controller through to FreeNas, then create iScsi targets for the other VM's to run on?

Which would be the lesser of two evils?

Also, this is just for home use, storing movies, music, photo's etc... Which, the important ones are/will also be replicated to an offsite backup

Thanks

Garth

 

[depasseg]

Option c - buy a small cheap pfsense box.
If for no other reason than it's nice to have your internet working when you do maintenance on ESX or your FreeNAS.

 

[Girth]

I already have one, but the Boss (wife) doesn't want the entire of the understairs cupboard full of stuff.

 

[jgreco]

Both solutions are evil, and will result in various problems. RDM is a terrible idea because people often find out that a failed hard drive leads to an unrecoverable pool, and those that suffer that fate invariably don't have the ESXi-fu to analyze exactly what has gone awry, so all I ever do is just tell people not to do this. VT-d with iSCSI targets from FreeNAS for your VM's runs into a bootstrap issue, where ESXi may wedge for many minutes. More recent ESXi's may have some mitigating options available (since there's some options in there to deal with vSAN style storage technologies) but you need to be certain that you're okay with maybe needing to go in there manually and whack things over the head with a hammer to get your VM's started if something goes awry.

https://forums.freenas.org/index.ph...nas-in-production-as-a-virtual-machine.12484/
https://forums.freenas.org/index.ph...ide-to-not-completely-losing-your-data.12714/

etc. I know the idea of an AIO box is extremely tempting, but if I had a nickel for every time we saw a trainwreck from people thinking that somehow they were "special" and that these problems didn't apply to them, and then it blew up in their faces, I'd be able to go out and enjoy a nice steak dinner. On the other hand, if you actually FOLLOW the advice therein, I'll say that I cannot recall any totally unrecoverable situation... it may require lots of beatings with a hammer to fix, but fixable. Do note that you still need a datastore on which to store the FreeNAS VM, ESXi won't let you store it on the USB (without some hacking). The all-or-nothing aspect of VT-d kind of sucks.

 

[clarknova]

RDM is a terrible idea because people often find out that a failed hard drive leads to an unrecoverable pool

Would you please explain this point, or provide an informative link? I've read many of the relevant discussions, and I know that RDM is considered to be a recipe for disaster, but I can't imagine why, and I haven't seen any details on it.

 

[jgreco]

Would you please explain this point, or provide an informative link? I've read many of the relevant discussions, and I know that RDM is considered to be a recipe for disaster, but I can't imagine why, and I haven't seen any details on it.

There's a wonderful thread called "Please do not run FreeNAS in production as a virtual machine" in which it's all discussed, actually read the whole thing for details. There's the VMware Knowledge Base #1017530 which explains why not to use local storage. I originally wrote the virtualization warning sticky because we had a very steady stream of users showing up in the forum who had lost their pools due to various blog posts that had told them RDM was the bee's knees. The general problem is that none of them had sufficient ESXi-fu to figure out the *why*, and I quite frankly have better things to do than to try to figure out *why*, but the general result is that mappings get trashed when something fails and then suddenly the pool becomes unrecoverable.

I will refer you to the Forum Rules which note that virtualization topics may go unanswered: "Virtualization of FreeNAS on Type-1 Hypervisors such as ESXi, XenServer, and Hyper-V. In short, if you can't do this without help, you shouldn't be doing it."

Beyond this point, you are on your own and left to your own devices. You're free to research and experiment. If, by some chance, you're an ESXi technomage who's interested in figuring out specifically what the hell happens when a disk fails and why it takes the pool with it, be my guest. I could figure it out but I don't have the time, or a failed system, or the desire to set up a system, wait for it to fail, and then do the work. But if you do, great. I'll even be moderately interested in hearing the explanation, and I'll be happy to include whatever remediation steps you figure out in some convenient place for others to reference. Otherwise this is pretty much the end of the line for me for this topic.

 

[clarknova]

There's a wonderful thread called "Please do not run FreeNAS in production as a virtual machine" in which it's all discussed, actually read the whole thing for details.

Thanks for the link. I'd read the first page recently, but only saw warnings without details. I just went back and skimmed the whole thing, and more carefully read the context of every mention of RDM.

There's the VMware Knowledge Base #1017530 which explains why not to use local storage.

No, that article explains that RDM usage has three requirements, and then outlines how to set it up. It's funny that KB was referred to many times in the FN forum thread that you linked, and everybody talked about it as if a)VMWare was about to end RDM support (2 years ago), and b) RDM use is discouraged or unsupported by VMWare. Neither conclusion is supported by anything to be found in the KB article as it exists today. Perhaps its contents have changed since 2013.

the general result is that mappings get trashed when something fails and then suddenly the pool becomes unrecoverable.

If there's a record of that happening, that's the kind of thing I'd like to read. Still hoping somebody can post a link.

I will refer you to the Forum Rules which note that virtualization topics may go unanswered: "Virtualization of FreeNAS on Type-1 Hypervisors such as ESXi, XenServer, and Hyper-V. In short, if you can't do this without help, you shouldn't be doing it."

Fine by me. We all agreed to those rules.

To be fair, I did pick up the following from your thread. Let's take a closer look:

1. All SMART support is gone.

From FreeNAS, but still available to the ESXi

2. Proper read/write errors to/from the disk are gone.

I don't know if I believe that.

3. RDM is not supported any longer in ESXi and will likely be removed in the future.

Not true.

4. Troubleshooting issues that may be related to virtualizing are no longer possible.

Through this forum, anyway.

5. if its not able to mount your zpool automatically there is literally zero chance you'll ever see your data again

This statement, even in its original context, is vague, and even if it's true, I don't see how virtualization makes it worse.

6. hypervisors and their I/O and CPU scheduling can in certain fluke scenarios...re-order I/O or CPU threads that can break some assumptions that kernel writers may have made. VMware optimized RDMs for use specifically with MS clustering it appears, NOT for ZFS's concerns - this can really mess with the assumptions of writes by ZFS

This last point at least sounds plausible.

So to summarize what appears to be the official position of the grinches and dogs of the forum,

1. Virtualize at your own risk. We won't help.
2. Virtualization of FreeNAS is a bad idea, and unless you know what you're doing, there are all kinds of pitfalls, including but not limited to RDM.

I get #1. Every support channel has its limits.

#2, in my view, appears to be dogma. I'm not saying it isn't true. I'm saying that it gets repeated a lot, mostly by the same two users as far as I can tell. If it's true then I'd like to see either a reasoned explanation or some sort of body count. As I've already pointed out, at least some of the warnings that get tossed about on these forums simply don't hold water. Drop those and your argument becomes immediately stronger, more concise, and more compelling. Users will be more likely to heed the warnings, and perhaps give meaningful contributions to the question of why virtualization has a habit of trashing data, if that is indeed the case.

 

[jgreco]

The VMware article clearly explains why local storage isn't suitable. As for the rest, you can call it dogma if you want. I call it fatigue. After you see something happen dozens of times, and the affected people are angry, and taking it out on the volunteers here, well, the obvious conclusion is that if you see people being ejected through the windshield often enough, is to tell them to buckle the hell up, and not to listen to the whines about how a seatbelt never saved them before.

I am *the* virtualization guy here. I've been doing it longer and documenting it better than anyone. You think you can do it better? Go ahead, be my guest. But don't expect me to do your homework for you. I have given you what I know. You're free to take it or leave it. I've been nice enough to write it up. Please be considerate enough not to call all my effort "dogma" when in fact all I'm trying to do is guide you to the one known path to success. I assume you give a crap about your data or you wouldn't be using ZFS. I assume losing your data would be a PITA. If those assumptions are in error, feel free to do as you wish.