Storage for VMware ESX -- QNAP or FreeNAS?

[ttblum]

Hi,

I have a small company that has about 30 ESX VMs running. If we want to expand our current storage, which would be better - building a FreeNAS box, or purchasing a commercial unit such as QNAP?

Do the benefits of having VMWare supported and certified hardware outweigh the benefits of using ZFS? I spoke to several VM techs who said that FreeNAS was only supported for making backups, not be supported for production use. Not that we can afford a VMWare support contract right now anyways.

Switching to Proxmox doesn't seem to make sense, because it doesn't look like a very smooth transition from VMWare to Proxmox. It doesn't look running them both at the same time would be fun either.

From looking, it does not look like QNAP has many features to transfer filesystem snapshots offsite like ZFS has, which would be a nice feature to have.

 

[jgreco]

The QNAP will be very nice until it runs out of steam, which with many of the desktop NAS devices is sooner rather than later.

You may have better luck with ZFS, but please do consider Intel ethernets, 32GB ECC, and mirror vdevs to be necessary. FreeNAS with a good configuration will blow away the QNAP. It also requires a lot more hardware just to deliver similar performance levels.

 

[ser_rhaegar]

What is your budget? How much storage space? What kind of downtime is allowed if there is an issue?

Personally, for work I would look at what is supported by the platform and for a product that has its own support/warranty. Especially if I was working at a small company. With the exception of a single IBM box, I use all EMC boxes at work for the support on both sides. As much as I like FreeNAS, I would need to know it a hell of a lot better before I could use it for the company I work at.

Also this is probably best asked in an ESXi forum.

 

[jgreco]

I disagree, this is a good place to ask, largely because a lot of the people who answer these questions on an ESXi forum will be home lab guys who built themselves a great 16GB lab box on an N54L, which is a poor starting point for discussing what actually needs to be in place for serious VM storage.

I've been trying to derive a solution to do VM storage on FreeNAS for awhile, and I have a fairly clear idea of both the ins and outs of it all. There are others here with some expertise as well.

 

[ser_rhaegar]

Most forums I visit regarding servers, storage and virtualization are mixed between home and commercial users, including this site.

Regarding the OP, I also think if you have to ask, the answer is a commercial system with phone/onsite support such as IBM, HP, EMC, etc rather than relying on a forum when something breaks.

 

[jgreco]

There are relatively few commercial users here as active participants. I can count them on one hand. Even if I include myself.

Anyone who is contemplating a QNAP for their VM storage needs is already well out of the realm of "IBM, HP, EMC, etc". There is no reason that a well-built FreeNAS system couldn't be made to work, and there are plenty of clues as to how to go about doing just that. You have already stated your opinion twice. If you are uninterested in answering the question the OP posed, please feel free to move along.

 

[ttblum]

Our budget is $3000 not including drives.
Storage space about 20 TB.
In case of chassis failure, an hour of downtime or less would be good in order to switch to backup chassis.

I called QNAP, and the rep. recommended the TS870U-RP.

My question is, is using FreeNAS worth the possibility of losing support in future?
Or are the amount of storage related problems few enough that it doesn't really matter?

 

[diehard]

For 3K grab a supermicro chassis and socket 2011 motherboard along with your favorite HBA (people here like the IBM 1015). We have one of these in production with 36 drives and run some VM's on it. Most disappointing thing probably is the iscsi fragmentation issues when running on a COW filesystem.. jgreco knows more about this than i do.

 

[cyberjock]

I helped someone 3 weeks ago put together a system that services over 30VMs. Total server cost including drives was about $4000 if I remember correctly.

 

[ttblum]

Hi, yes, ser_rhaegar seems to be the only one answering my question. I'm fairly certain that a well built and configured unit running FreeNAS could outperform more expensive commercial units.

I'm asking those that have been running FreeNAS with VMWare, have you ever had to call VMWare for support? If so, did running FreeNAS diminish your level of support with them at all?

 

[cyberjock]

Well, 99% of the people in this forum use the free version of ESXi. Which means "no support even if we wanted it". So you aren't likely to see any of your answers here. I agree with ser_rhaegar that this isn't the best place to ask this question. Usually businesses show up, get their setup going, then disappear and never return. Or, they show up and either pay someone like me for some consultation or go to iX and buy a pre-built pre-configured system. In any case, they don't ever show up here again after they've finished building the system unless something stops working.

 

[jgreco]

QUOTE] In any case, they don't ever show up here again after they've finished building the system unless something stops working. [/QUOTE]

Unless they're strange like me. But basically the system that QNAP pointed you at is a relatively small box. It might work. But if you want to do ZFS, you should probably do some homework first. Actually you should do it regardless.

The big thing here is that VM traffic means very different things to different people. I can host a thousand VM's worth of low activity VM's on a 32GB Haswell box and still have plenty of resources left over, but I can also trivially load a single major VM onto a filer and cause it to melt. This is where vendors might (or might not) help you out, figuring out what you need, though it isn't that unusual for a vendor to sell you something and then leave you hanging when it only kinda works.

You're not going to get much help with the trivial amount of information you've provided. You have to give us some idea of what sort of VM's these are, how much disk space they take, what's the working set size, how much read vs how much write, etc.

Dollar for dollar, FreeNAS and ZFS will typically do better than commercial units, but whereas that little 2U QNAP unit has a 2.4GHz dual core CPU and 4GB RAM for $2400, we're likely to suggest something with much larger specs that will still come in cheaper. But we have no insight into your requirements.

 

[bigphil]

I agree with jgreco on this. I think you should step back a second and look at some other options. To best serve you, I think we would need a little more infrastructure background, specifically:

How many hosts do you have running?
What type and version of hardware are they running on? i.e. HP Proliant, etc.
What version of ESX or ESXi are they running?
If licensed, what vSphere license do you have?
Do you have a vCenter server?
Network backbone speed?
Managed or unmanged switches?
Guest OS's you're supporting and what are they serving?

The reason I ask is that I would hate for you to have a nice storage system but be lacking in host redundancy which is just as important. With that being said, I think for your budget, you're in a tough position. For 3k, you could build a pretty kick butt system on FreeNAS without drives. I would second a Supermicro chassis build and Intel system for a FreeNAS build. I have quite a lot of experience with Qnap and have had little to no problems with them. I have used several of the larger 8 bay units for NFS and iSCSI storage and a lot of the smaller 2 bay units for backing up standalone servers. Another good one to look at for VMware storage would be Drobo. Very nice smb SAN's (B800i and B1200i).

So if you provide the requested information, I think we could better answer your question in the most appropriate manner for your situation.

 

[ttblum]

Ok, here are the results of the survey:

___________________________________________________________________________________
How many hosts do you have running? 22 VMs

What type and version of hardware are they running on? i.e. HP Proliant, etc. -

MFSYS25 with built-in SAN, plus a locally SAS-attached Promise SAN.
Both SANs are configured with a single RAID6 array. They are now both full.

What version of ESX or ESXi are they running?
ESX 4.1.0

If licensed, what vSphere license do you have?
vSphere 4 Essentials 6 CPUs
vSphere 4 Standard 6 CPUs
Currently running (5) 2-CPU blades, leaving 2 CPU licenses unallocated
vCenter Server 4 Essentials administration console (1 instance)
vCenter Server 4 Foundation (1 instance), unused

Do you have a vCenter server?

Yes, we have two, one for each license (both are VMs)

Network backbone speed?
Gigabit

Managed or unmanged switches?
Managed, two 3750s

Guest OS's you're supporting and what are they serving?

(10) are Win2k8 running Sybase SQL Anywhere, (1) running Win2k3 running MS SQL, the rest 10-user terminal servers or miscellaneous.
Most of the database VMs are between 300 and 500 GB in size.
___________________________________________________________________________________

Three of the Sybase VMs, as well as the MS SQL VMs are performing poorly, and the only reason seems to be storage latency (spindle contention?).

My objective would be to migrate the four problem VMs to better performing storage, and make room to house new VMs as they are needed in the future.

 

[bigphil]

I should have added another question. What type of VM and/or application backups do you have and where do they reside? My opinion for your situation is a Drobo 1200i. Its just over budget at $3700 though. Sure you could build a nice FreeNAS system, but if it will be a little more difficult to support if you're the only guy in the shop. That particular Drobo offers field replaceable parts, including the controller, so you could really minimize downtime in the event of a hardware failure. Not that you couldn't do the same with hardware running FreeNAS. Ultimately, its your decision. No matter what you decide, I would be damn sure to have some sort of good backup plan. I would highly recommend looking into Veeam Backup & Replication if you don't already have something to backup your VM's.

 

[ttblum]

Yes, we have no VM backups right now - we are currently only making SQL backups to an SMB share on a physical Win2k3 server.
Right now, to restore, we have to create a VM from a template, then import the backed up database.
But unfortunately, the increasing amount of SMB traffic to the Win2k3 backup server is beginning to overload it.

I am currently building a FreeNAS server (Supermicro X7DBN/SC826TQ-R800LPB) in order to provide NFS storage to the ESX hosts for backups.

I looked at Veeam and it seemed like the regular products were over-budget, and the free ones were pretty scaled down?

 

[bigphil]

Veeam is a bit pricey when you have lots of populated sockets on your ESXi hosts...its like $750 per socket for the standard version. The free version can only create VeeamZIP backups (still a Full crash-consistent VM backup) and you cant schedule it. Other than that, its quite powerful for a free product! Have you looked into upgrading your hosts to the latest release of ESXi? I would at the very least, install the free version of Veeam and create backups of your important VM's and store them on your NFS share you're building. I'm trying to find out of if the VMware VDR appliance 2.0.3 is compatible with ESX 4.1. (Just saw the matrix and VDR 2.0.3 is compatible with ESX 4.1 U1-U3). It would be a free backup solution that supports VMFS datastore targets as well as CIFS targets for storing its backups. It does however require the vSphere Essentials Plus license.

 

[Got2GoLV]

I'm asking those that have been running FreeNAS with VMWare, have you ever had to call VMWare for support? If so, did running FreeNAS diminish your level of support with them at all?

Ive been involved with the design/implementation/maintenance of several ESX projects that use FN for back end storage, and in use commercially for the last 3 years, and I have yet to run into any FN specific issues with ESX. The issues I have run into are general ESX issues that would happen on any storage back end.

The only thing FreeNAS/FreeBSD fails miserably is in automatic failed drive replacement from available spares. It basically doesn't. It is strictly a manual affair. Not horrible, but not automatic. And with room for errors (user errors).
This, I find a little ridiculous at this stage in the development/life of FN/FreeBSD. This is not v1.0 anymore.


Budget of $3k + drives ?
22VMs ?

Build yourself a nice Supermicro 24bay server with plenty of RAM, SLOG (sync=always on iSCSI), and L2ARC, redundant power supplies, redundant NICs.
It will blow away the Promise and built-in arrays.

Then, use the built in array and Promise arrays for disk to disk backups of the VMs (either at VM level, or at OS level as needed).

You would have a lot of options to play with different tiers of storage performance as needed (Ex: SSD based zvols for heave DB use), slower for regular OS/file server...slowest for backups/sequential stuff.

Drobo? Pfft....

Just my opinion here...
But what do I know...I once shit my pants at a county fair...

 

[ccdoggy]

I might be a bit late to the conversation but alas can speak to the VMWare support model around storage issues. For my job I interface with them weekly for storage support calls with customers for troubleshooting issues.

Their support teams are great, except roadblocks occur when something is not on their support matrix or they believe it is a storage issue. If you do not have a support team to call for the storage and you are not very familiar with the underlying technology it could place you in a position you really do not want to be in. VMWare support will not dig into your possible storage issues beyond looking at the ESXi host logs and pointing the finger depending on what they find.

I am all for building your own machines for an environment as the budget can go so much further, but given the important data which will be living on it the support model and resources are paramount if something unfortunate were to occur. Sounds like there might be some FreeNas support resources out there, have not looked myself, but it might be a good idea to ensure whatever you build follows their best practices for equipment to ensure it is built 100% supported to their best practices.