"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[Alvin]

FreeBSD 9.2 will also contain virtio drivers. That should ease virtualisation on KVM or bhyve. (source: http://forums.freebsd.org/showthread.php?t=41246 )

 

[pbucher]

It becomes ever more challenging to figure out the whole VMware ecosystem as they keep adding and changing various features and modules. Has there been any estimate as to when 5.5 will actually hit FCS?

I hear developers just got RC1 recently, so I'd say a few weeks or so depending on feedback. I agree the moving around of features and modules has become crazy, in fact I'm in a pissing match with vmware support over if I should have a license for VSA, tech support says I should have one, license supports agrees, sales would like a PO & money for a license. Between FreeNAS & now the new vsan I no longer see a much future for VSA for myself.

Also what's your take on vsan, it smells a little ZFS like to me?

 

[jgreco]

Bear in mind we're a 4.1 shop here so my comments are that of outsider-looking-in. Haven't done much with it.

VSA seemed like a fairly straightforward attempt to create a NAS VM they could generate licensing revenue from. I thought, based on early hype, that there was supposed to be a good degree of integration, but I haven't heard much that seemed compelling about VSA. I believe it's a SLES VM plus distributed storage system (similar to HAST+UFS or DRBD+Ext3) layered on top. This would have capabilities that FreeNAS doesn't, such as failover in the event of failure, since FreeNAS lacks an effective realtime redundancy solution. But of course FreeNAS also has some features of its own...

The new vSAN thing, however, seems potentially meaningful. My first nasty thought was "well they finally figured out how to install nfsd", but it appears to be substantially more than that. All I can really make out is that it is some sort of unconventional object-based datastore implementation. Actual design is unclear, so making intelligent discussion is difficult. Admittedly I was on vacation and so may have missed some more in-depth descriptions. Basically some handwaving about using some spinny rust and an SSD on each host, with some promising-sounding stuff about read and write caching on the SSD. Experience tells me that glossing over the nitty gritty bits of how it actually works means that it isn't likely to be particularly high performance, but there are lots of environments where that'll still be very useful.

From my PoV, the real win isn't any of that but rather Flash Read Cache (people here, think of it as L2ARC for your ESXi). And I really have to say, about fscking time, VMware.

There seem to be two basic schools of thought about VM designs; one is to take existing physical servers and just "transition" them onto a virtualization platform. This extends to the creation of new VM's as well, where an admin treats a virtualized machine being installed no differently than if it were a physical server. The other is to design a VM for the virtualization environment. For example: Since writes are problematic for VMware and many storage systems (including ZFS), but traditionally UNIX had assumed it had the disks all to itself, disabling little things like file atime updates on your VM's base OS files can reduce the inevitable fragmentation of your ZFS iSCSI extent hosting the datastore. Or, for another example, when the periodic crons run daily, if you have several dozen VM's all suddenly becoming active and saturating your SAN with relatively unnecessary "find" commands, staggering the run of those cron jobs makes for storage happiness. Obviously the "design it for virtualization" strategy is going to be more friendly on your virtualization platform.

A lot of the things I've seen over the years boil down to storage subsystem problems, and I hope FRC turns out to be a viable solution to accelerate reads and reduce stress on the SAN, which simultaneously means more potential write capacity since it isn't as busy with reads. And while it would be easy and pleasant to say "but FreeNAS has done that for years through the L2ARC", which is absolutely true, the fact is that if VMware has done FRC well, then a smaller cache on each host, each attached via 6Gbps, is going to function with more speed and less latency than centralized iSCSI SAN or NFS requests over 1Gbps or even 10Gbps to even a very big, fast FreeNAS server.

 

[pbucher]

From my PoV, the real win isn't any of that but rather Flash Read Cache (people here, think of it as L2ARC for your ESXi). And I really have to say, about fscking time, VMware.

FRC looks very cool, but you need the vSphere Enterprise Plus edition in order to use it :( I hate that the new shiny features are always only for the high end editions. Standard at least now gets hot add included so I'll get a least something new.

Your right about VSA, my interest was mostly to offload my FreeNAS array onto my local RAID array and take advantage of it's mirroring ability and some of the automation for failure overs and such.

I did dig into VSA some and it's mirroring is actually done using iSCSI and mdadm which seems painful compared to using DRBD. At this point I'm probably going to simply roll my own linux NFS server with DRBD+ext4 or xfs and pass through my RAID controller and get a much better performing vsan then VSA. My main goal is to get the low bandwidth but constant i/o stuff off my FreeNAS san so I can dedicate it for my VMs that need high bandwidth(they are all Linux servers and I use mounted NFS shares directly from the VM to FreeNAS and skip the whole ESXi layer).

 

[sincar]

Hello all,
this is my first post on this forum and I thank you in advance for the help that you give me.
I don't know in depth FreeNAS and I have basic knowledge of vSphere , GNU / Linux and SAN and NAS systems.
But above all I am an enthusiast of OSS and I would use it in the project which I will describe now:

Next month I will have 40 users (mainly iMAC use with Adobe Photoshop / InDesign , but also some Windows clients ) that will interact with
a publishing system that I will install on CentOS using the LAMP platform.
I would like to :

• install LAMP stack on CentOS Box (definitely ok!)
• handle authentication via Samba 4 / LDAP through the distribution Zentyal (seems ok);
• bind iMac client via LDAP and Win client via AD to Zentyal server;
• bind a FreeNAS VM via LDAP to Zentyal server to export AFP shares to MAC users (seems ok, some issue on use LDAP group for ro/rw access);

I followed the posts "Please do not run FreeNAS in production as a Virtual Machine!" and "Absolutely must virtualize FreeNAS!" and now I am ashamed to have thought, before reading the post, to manage ZFS volumes by adding more and more virtual hard disk to the FreeNAS VM ...

My questions are :

• Having fiber SAN architecture and using NPIV, I have a better (or different) chance of successfully virtualize?
• VMware documentation says that I should use RDM to have vMotion, it means that I can't follow the directives of jgreco if not sacrificing vMotion ?
• In my server I have 2 hba to connect with redundancy to storage through 2 fiber switch, using the PCI- passthrough, I map HBA direct to a VM , but this means that my esxi host cannot use them to access a LUN for the vSphere DataStore?
• I guess the best solution is a third server with 2 other HBA where install FreeNAS directly. But I need to purchase additional hardware and I haven't hw redundancy...

I apologize for the long post and the bad English, but I am really anxious to put in production a fully open environment! (except vSphere)

thanks!
sincar

 

[pbucher]

My questions are :

• Having fiber SAN architecture and using NPIV, I have a better (or different) chance of successfully virtualize?
• VMware documentation says that I should use RDM to have vMotion, it means that I can't follow the directives of jgreco if not sacrificing vMotion ?
• In my server I have 2 hba to connect with redundancy to storage through 2 fiber switch, using the PCI- passthrough, I map HBA direct to a VM , but this means that my esxi host cannot use them to access a LUN for the vSphere DataStore?
• I guess the best solution is a third server with 2 other HBA where install FreeNAS directly. But I need to purchase additional hardware and I haven't hw redundancy...

I really can't say if your FC channel SAN might help or hinder your setup. Here is what I do for my production vSAN. I put ESXi onto a nice little SSD drive(though it could be just about anything) with enough space to have a small data store to hold my FreeNAS OS volume. I then pass through my HBA(a high end LSI SAS card) and build out FreeNAS using the physical drives off the HBA. I create a 2nd network on each ESXi box that is dedicated for vMotion and SAN traffic. I attach the FreeNAS to both virtual networks so I can manage it from the office net and then serve NFS to the SAN network and host my ESXi VMs on that.

Could you possibly add a 2nd HBA to each server and pass that through to the FreeNAS VM? You'll want to look at what FC HBAs are supported in FreeBSD 9.1.

 

[jgreco]

[*]Having fiber SAN architecture and using NPIV, I have a better (or different) chance of successfully virtualize?

To some extent, yes. Your SAN infrastructure is presumably monitored for faults, and since your non-NAS data is at risk too, it all shares the same fate. That compares favorably to SMART monitoring (assuming you actually ARE monitoring the SAN).

[*]VMware documentation says that I should use RDM to have vMotion, it means that I can't follow the directives of jgreco if not sacrificing vMotion ?
[*]In my server I have 2 hba to connect with redundancy to storage through 2 fiber switch, using the PCI- passthrough, I map HBA direct to a VM , but this means that my esxi host cannot use them to access a LUN for the vSphere DataStore?
[*]I guess the best solution is a third server with 2 other HBA where install FreeNAS directly. But I need to purchase additional hardware and I haven't hw redundancy...
[/LIST]
I apologize for the long post and the bad English, but I am really anxious to put in production a fully open environment! (except vSphere)

thanks!
sincar

Well, I think the question is how much data are you planning to store this way? If it wasn't a huge amount (ie could fit on a single VMware virtual disk) and you are prepared to rely on the SAN for reliability properties, then you probably could just make a conventional VM. Using RDM in a VMware blessed manner may also be okay, maybe pbucher will stop in with some comments.

Anyways: The guide I've given outlines many of the common issues. You need to give them thought. I cannot promise success with alternative strategies, but I strongly encourage you to devise a reliable way to back up your system ... and then your options are better, because even if disaster strikes, ...

 

[pbucher]

Using RDM in a VMware blessed manner may also be okay, maybe pbucher will stop in with some comments

Since you have real SAN you should be able to use RDM in a vmware officially supported way which will help things greatly. The key is your SAN needs to give the LUNs a unique serial # and have it stick with the LUN. This usually isn't a problem, except under locally attached storage using certain HBAs and brands of hard drives(there isn't a list vmware just forbids the whole setup to be safe).

I've used RDM physical disks successfully in the past, though once I got the HBA to pass through and work correctly I swtiched to PCI passthrough of the HBA(in fact I went back and forth a few times without having to rebuild the pool). I've heard other people have some really bad problems with RDM setups when a disk had to be replaced, I suspect though they had run of the mill consumer hardware and had issues with the drive serial #s. I just brought up a RDM setup using ESXi 5.5 and FN 9.1.1 and it seems to be stable and is holding up ok so far(I hope I will be able to convince the company to buy a server that supports pass-through in the near future).

 

[cyberjock]

Since you have real SAN you should be able to use RDM in a vmware officially supported way which will help things greatly. The key is your SAN needs to give the LUNs a unique serial # and have it stick with the LUN. This usually isn't a problem, except under locally attached storage using certain HBAs and brands of hard drives(there isn't a list vmware just forbids the whole setup to be safe).

Actually, it seems to be more complex than that. I will agree that your reasoning is very sound, but it actually looks like RDM'd disks in FreeNAS are being zeroed out. At least, for some people that's the case.

For others the RDM mapping suddenly changes and the disks are 0 bytes as read from gpart list(I believe that's where I got it from). Again, not all, but some.

Still, for others it seems like neither of those, though what is actually wrong hasn't really been investigated by anyone. Generally we("we" being some of the more knowledgable people here) just say "too bad so sad" when we get cases of RDM gone bad because recovery is pretty much impossible from what has been researched on the topic before.

Additionally, even when RDM mappings go bad, if you redo the RDM mappings as if something went wrong for a FreeNAS disk things don't get better. But if you redo the RDM mapping on any other OS it seems to work just fine. So something fishy and unexplained is going on.

I've used RDM physical disks successfully in the past, though once I got the HBA to pass through and work correctly I swtiched to PCI passthrough of the HBA(in fact I went back and forth a few times without having to rebuild the pool). I've heard other people have some really bad problems with RDM setups when a disk had to be replaced, I suspect though they had run of the mill consumer hardware and had issues with the drive serial #s. I just brought up a RDM setup using ESXi 5.5 and FN 9.1.1 and it seems to be stable and is holding up ok so far(I hope I will be able to convince the company to buy a server that supports pass-through in the near future).

Hehe. That last sentence has been many people's epitaph. The biggest problems with RDM(and with the whole ECC versus non-ECC RAM) is that things will appear to work fine. You'll be able to create it, performance will be good, disk failure testing will be just fine. There will be no indication that anything is not right with the setup or that you should be concerned at all. It might work great for weeks or months(some people for more than a year).

But then one day they reboot the server, reboot FreeNAS, or whatever else you can think of that is very non-intrusive and not out of the ordinary, but something goes wrong. That's where you press the "o ****" button and realize you've been had by this bug.

I'll tell you something though. I live for figuring out the weird things in the world. I'm like Dr. House from the TV show in that I live for solving unsolvable puzzles. But being that I'm not a programmer and the issue is not easily reproducible by anyone really makes me wish I had the skills to find and solve this issue. It fascinates me that RDM should work well, and does for other VMs, but goes horribly horribly wrong with FreeNAS even in situations where you'd think its non possible.

 

[pbucher]

While I'm not a fan of using RDM for a FreeNAS install, I don't see that if done properly you have any problem doing it. It's very similar to the argument about virtualizingFreeNAS in the first place. If it's done improperly with non-enterprise hardware bad things lie in your future. Far too many folks using RDM are doing it via hacks and worse often choose the hack of doing a virtual RDM and not a physical one. Even so no one doing the hack has any idea if their hardware combo actually will work the way VMware needs it to work. Which the a none working combo would give you the mapping getting changed and the wheels falling off on reboot. Which is why vmware blocks it out by default and a hack is required in the 1st place.

Like I said I've run it without problem(though I prefer not to but I don't always get a choice). I think Sincar's got the right hardware to do it, there is a big difference between RDMs from a FC san vs some cheap consumer drives hooked to a SAS expander and then into a SAS HBA. The key thing is does the drive serial # that vmware counts on for the mapping come from the physical drive or SAN controllers vs a #s assigned by a driver or some firmware have the drive. I may be wrong but I'd think any FC would have to handout serial #s for a LUN that remain consistent, though I fear just like their are NAS boxes built by _____ that have ZFS on top of a single hardware raid volume there are probably bad FC san boxes also. I guess I'd ask what kind of drives are in the FC san box? Is it full of SATA drives with some kind of FC convertor or are they real enterprise FC drives. Sincar's chances are much better if they are real FC drives.

 

[cyberjock]

Far too many folks using RDM are doing it via hacks and worse often choose the hack of doing a virtual RDM and not a physical one.

I know that every person I've tried to help did physical RDM. Note that I'm not implying that physical must be bad. I'm just saying that the 4 people I've helped via Teamviewer all did physical and none got their data back.

The key thing is does the drive serial # that vmware counts on for the mapping come from the physical drive or SAN controllers vs a #s assigned by a driver or some firmware have the drive.

My M1015 reflashed to IT mod provided the serial number in the device ID when I did an RDM map for a linux VM. I can't confirm any other configurations myself. Currently I'm doing RDM via the onboard SATAIII port on my Supermicro X9SCM-F motherboard with ESXi 5.1 and it definitely has the serial number too.

My problem with the whole RDM thing for FreeNAS is that in theory you should be able to pull out those RDMed disks and boot up bare metal on another machine and be a zpool import away from accessing your pool. But it never works out that way. For all of them the partition table is wiped out with random data(or at least it appears to be random). Just very puzzling for me(and everyone else). I've done exactly this with several linux disks without any problem on my X9SCM-F board. /shrug

It's just a roll of the dice from what I've seen. We don't have a pathology for what goes wrong. We don't have a working solution. So this thread was created to hopefully push people away from trying it. Of course most people that do it despite the warnings cite "cost savings" as their justification(which you kind of eluded to with your comment that you "didn't have much choice") until they lose their pool. Then they realize that the cost savings wasn't worth the gamble. Suddenly they are begging for help, offering to throw money at the problem which doesn't change the outcome(but not the $25k+ that some of the professional data recovery experts ask) and they realize that they really didn't properly prioritize the benefits versus risks when they made their choice. Whoops!

Of course, if you keep religious backups then its not that big of a deal. Additionally if you just want some temp storage space and you don't care about the data then its no big deal either. But lets be honest with ourselves. How often have we created a temp storage location and then put critical non backed up data on it? I bet everyone in this forum is guilty of doing that at some time.

 

[jgreco]

Is it fool of SATA drives with some kind of FC convertor

Please don't fool around, tell us what you really think!

 

[pbucher]

Please don't fool around, tell us what you really think!

Opps I meant to say full....but only a fool would put SATA drives in a FC SAN........

 

[pbucher]

Generally we("we" being some of the more knowledgable people here) just say "too bad so sad" when we get cases of RDM gone bad because recovery is pretty much impossible from what has been researched on the topic before.

Agreed....RDM is indeed playing with fire...some times you can cook with it and other times you get burned bad.

 

[cyberjock]

In my Mr. T impersonation...

I petty the full?

 

[KMR]

Quick question here. I'll be moving to a virtual setup soon using a 1230v2 CPU, 24GB ECC RAM, and an M1015 in passthrough mode for ZFS drives. The VM will host a 6x3tb RAIDZ2 pool and a 2x1TB mirror pool. The Mirrored pool will be for downloading and whatnot and the primary storage pool will basically just be for streaming. If I allocate 12GB of RAM to the VM will this be enough for these tasks or should I allocate a full 16GB of RAM? I know that under-resourcing VMs is a primary danger in a virtual setup but I thought this would be a decent compromise until the price of RAM falls back down to normal levels.
I can probably skimp on the RAM for other VMs until I upgrade to 32GB if the full 16GB is required for the FreeNAS VM, it would just be nice to have more than 8GB to play with for the rest of them.

 

[jgreco]

Under-resourcing is primarily problematic for the guy desperate to run ESXi on a 16GB N54L and can't swallow allocating 8GB for FreeNAS, so he thinks he's special and goes with 4GB, which works until it does not.

Please remember two things:

1) For the size of disk under discussion, 8GB results in stable operation .... speed may be impacted but not stability.

2) The wonders of virtualization allow you to twiddle RAM by a shutdown, edit, reboot.

So you are ideally situated in that you can try 8 or 12 as you wish, and change it trivially if it doesn't meet expectations.

Remember those things and you don't need to overthink this.

 

[jgreco]

P.s. remember you can do fun stuff like allocating odd amounts of memory like 10.52 GB as well...

 

[KMR]

Excellent! Thank you for you speedy and thorough response, as always. I have 32GB of RAM currently but I ordered another motherboard with IMPI for the ESXi build and will be using my current motherboard and one 8GB stick of RAM for another FreeNAS build that I'm going to keep at my brothers house as a backup server / Christmas gift; two birds, one stone, and whatnot.

 

[jgreco]

How big is "big enough" is always a tough question. Virtualization is both a danger (because of the inherent desire to under-resource so as to fit in more yummy other-VM's) and a blessing (because you can trivially experiment without having to shuffle sticks of physical RAM in a physical box).

I do intend to scare people more than a little bit, but the goal is simply to make sure people don't paint their way into a corner. My main thought is that the dual pool setup is possibly a little stressy for 8GB. However, if it turns into a problem you can push the RAM out to 16 or even 20GB by temporarily shutting down other VM's. The normal problem people seem to run into on "too-small" memory systems is that it'll panic and then will not have sufficient memory to import the pool after the crash. So you have a solution to address that latter problem. That's really why I think you can get away with it. But I will note that there aren't a lot of people doing dual pools, so please keep us all informed as to how it works out and what you learn.