Available Space difference from FreeNAS and VMware

[soulaero]

I created a zvol on FreeNAS for VMware.
For my storage server, it's 4*2TB, after mirroring, there are 3.6 TB available. zvol is 3.1TiB, and there are still 2.5 TiB left.
In Vmware, Capacity is 3.00 TB, However, only 1.27 TB Free.

I believe, Compress might cause space difference between FreeNAS and Vmware. My questions are
1) Can VMware get the correct information for free space?
2) Since From FreeNAS, there are still 2.5 TiB available, however, when I try to create another 1 TB zvol, it's failed duo to lack of space. why?

 

[mav@]

You've mixed all possible unrelated things together. Not even sure where to start.

Free space reported by VMware is a space on free on VMFS file system, space which is not logically allocated to any virtual macnine, and which has nothing to do with real storage space use.

Available space you've marked on FreeNAS is space available for writing for this specific zvol, taking into account its space reservation. If you look couple lines higher, you'll see that for other datasets and zvols available only 420GB, since all other space is reserved by zvol_1. That is why creation of new zvol fails (unless you try to create it is thin-provisioned, which may be not a good way for our level of understanding of all this mechanics).

 

[soulaero]

You've mixed all possible unrelated things together. Not even sure where to start.

Free space reported by VMware is a space on free on VMFS file system, space which is not logically allocated to any virtual macnine, and which has nothing to do with real storage space use.

Available space you've marked on FreeNAS is space available for writing for this specific zvol, taking into account its space reservation. If you look couple lines higher, you'll see that for other datasets and zvols available only 420GB, since all other space is reserved by zvol_1. That is why creation of new zvol fails (unless you try to create it is thin-provisioned, which may be not a good way for our level of understanding of all this mechanics).

I've known the situation, VMFS vol is reserved 3TB and there are only 420 GB free.
However, is that mean Compress totally meaningless for VMFS? you saved space but can not use it?

 

[jgreco]

You don't have any business trying to use all the space. ZFS is a copy on write filesystem, and needs significant amounts of space free in order to keep performing at acceptable levels. Your pool should probably never be filled more than 50% if you want ESXi to continue to like your FreeNAS ZFS datastore.

With that in mind, your VMFS zvol shouldn't be larger than maybe 1.8TB, and that's only if you were to wipe out the jails. With the jails, you should only have about a 1TB zvol.

Within that space, the space you win via compression should probably be viewed as a performance enhancer rather than "wow look more space for me to waste."

Your existing configuration is a disaster waiting to happen.

 

[soulaero]

You don't have any business trying to use all the space. ZFS is a copy on write filesystem, and needs significant amounts of space free in order to keep performing at acceptable levels. Your pool should probably never be filled more than 50% if you want ESXi to continue to like your FreeNAS ZFS datastore.

With that in mind, your VMFS zvol shouldn't be larger than maybe 1.8TB, and that's only if you were to wipe out the jails. With the jails, you should only have about a 1TB zvol.

Within that space, the space you win via compression should probably be viewed as a performance enhancer rather than "wow look more space for me to waste."

Your existing configuration is a disaster waiting to happen.

Thanks for your suggestion. In my current situation, much space is suitable for compress. Even the data in VMFS reach to 3TB, the actual usage in ZFS pool should be larger then 50% or 1.8TB.

 

[jgreco]

Thanks for your suggestion. In my current situation, much space is suitable for compress. Even the data in VMFS reach to 3TB, the actual usage in ZFS pool should be larger then 50% or 1.8TB.

That's a dangerous game to play. If your assumptions turn out wrong, the whole thing can come to a screeching halt and a crash. Your datastore, that is.

 

[zimmy6996]

That's a dangerous game to play. If your assumptions turn out wrong, the whole thing can come to a screeching halt and a crash. Your datastore, that is.

Hey jgreco, I'm sorry to bother you. But I sort of have the same questions as the original poster, though I'm not trying to create another zvol. I just am asking about the free space reported by ESXi.

In my situation ....

So like the original poster, my question is:

I have a 12.1TB ZVOL created for iSCSI, presented to my ESXi hosts. I have thin-provisioned 7.29TB of guests, of which, it looks like those represent about 4.71TB of actual raw data. Leaving 8.20TB of RAW available storage from ESXi's perspective.

When you look at the FreeNAS side, you can see you actually are getting 1.32x compression, so FreeNAS figures that I have 11.5 TB of the 12.1TB available. If i'm following this right, the math here is that the FreeNAS is calculating that based on zvol is 12.1TB * 1.32 for current compression, figuring that the ZVOL net size with compression is actually about 15.972TB. When you subtract the 4.71TB of RAW usage that is being reported from the VMware side from this number, you get 11.262TB. Numbers don't completely match up, but it's close. I'm guessing this is where the math is coming together?

So ... Assuming that is the case, I think the posters original question, which is the same question I have, is ... With compression enabled, in the way this setup is running today, I would imagine (not saying it's smart to fill to 100%), but lets just assuming this situation for a second. What happens on the ESXi side of things, when I fill to 12.1TB of RAW data? Right now, I would assume, ESXi is going to say that the filesystem is full. And yet, even though it's full, with compression enabled, theoretically, FreeNAS is going to report that it sees (assuming 1.32x compression were to hold) that there is about 3TB of free space still available on the ZVOL.

So I think the question here is, if you are only using FreeNAS as a ESXi store, (and and my case, I am ... I know I have the CIFS mount there, but i'm using about 15GB of space for some ISO files and text config backups from some network gear, of which this amount of usage is never going to grow). But again, if you are only using FreeNAS for iSCSI ESXI storage, is there a benefit to using compression? It would seem to me that at the end of the day, compression doesn't really "buy" you anything, because any net savings based on compression is negated by the fact that the ESXi won't see that net savings. It would seem to me, in order to actually see the net savings in this manner, you'd almost have to be using NFS over iSCSI.

Sorry for the newb sort of question, but if I could beg you for a "teach me something new" response, I would greatly appreciate it!

 

[jgreco]

Okay. So what you do is you create a 12.1TB ZVOL. A ZVOL is a block device, so what you've actually asked is for ZFS to set aside 3,248,069,017 4K sized blocks for you (or thereabouts). ESXi has used VMFS to lay these out into a nice filesystem for you to use, yes?

So when you've written zeroes to all 3.2 billion sectors on the VMFS filesystem, what do you think VMFS is going to see? It's going to see 3.2 billion sectors filled with zeroes.

The fact that the 3.2 billion sectors might be highly compressed underneath the sheets is irrelevant. It's a block access protocol. You created a storage service with 3.2 billion blocks. You got it. No amount of compression will increase the number of blocks in the storage device you created, it's giving you what you specified. This is an inherent property of a block storage device. You experience this on an SSD as well. All those SandForce controllers out there do aggressive compression. Yet you don't throw a fit that your 120GB SSD doesn't magically grow to 150GB when the SandForce controller's managed a modest job of compression. Instead, that savings is used to the advantage of the device, providing higher performance. It's just harder for you to log into an SSD and see what's actually going on.

Suddenly feel like "oh duh?" If so, good, you probably got the point. ;-)

So. Moving on. Compression is ABSOLUTELY a great idea. First, a compressed block will transfer from disk more quickly, and CPU decompression is gobs faster than SATA/SAS transfer of a larger sized uncompressed block of data. Second, compression increases the pool free space. Since ZFS write performance is loosely tied to the pool occupancy rate, having more free space tends to increase write performance.

Next thing. Your pool looks like it is made out of RAIDZ2, am I guessing right when I say ... is that 6 x 4TB in a RAIDZ2 pool? This is highly suboptimal for VM storage because ZFS may not be using an ideal amount of disk to handle the parity data, plus your whole zvol will have approximately the IOPS capacity of a single underlying disk.

So the thing is, you probably never want to fill even a well-designed mirror vdev pool past maybe the 50% point, because at 50% you're at a point where the pool will eventually settle into a relatively low performance profile. But if you leave compression enabled and deliberately create your zvol such that it's 50% of the size of the pool, then between the free space in the VMFS and things like compression, you're more likely to end up at some reasonable percentage of fill that keeps performance pretty adequate.

 

[zimmy6996]

So the thing is, you probably never want to fill even a well-designed mirror vdev pool past maybe the 50% point, because at 50% you're at a point where the pool will eventually settle into a relatively low performance profile. But if you leave compression enabled and deliberately create your zvol such that it's 50% of the size of the pool, then between the free space in the VMFS and things like compression, you're more likely to end up at some reasonable percentage of fill that keeps performance pretty adequate.

Jgreco -

OH DUH!!!! Honestly that is pretty much what I was expecting in terms of the block file system, and the fact that it remains the same regardless of compression.

What you are saying is the benefit to compression is that I currently provisioned 80% of my pool, with a ZVOL for block storage, and with compression, assuming the compression rates stay the same, the pool is going to find that when I'm at 100% usage of the iSCSI volume, the ZVOL will look like it's 50-60% full, which at the end of the day, keeps ZFS happy, on top of the other benefits of compression with speed, etc ...

Do I have my head around this correctly now?

As to your comment about RAIDZ2, you are correct, currently the volume is setup as RAIDZ2. I've been too chickensh*t to do mirrors, due to really having a desire to be able to sustain dual disk failure. I guess I need to put my faith in a 10 disk 5 mirrors strip set, and realize the statistical odds of 2 disks in the same mirror failing is pretty slim, and the performance gains out weigh the risks if proper backups are being performed. (does this sound like I have my head around it now?)

 

[jgreco]

Well, ZFS won't be super happy at 50-60%. Over time, what happens is that fragmentation increases on the pool and the ability of ZFS to rapidly find contiguous ranges of free space drops, which impacts write performance. You won't see this right away... some people fill their pool to 80% and say "oh speeds are great, I'll just do this then" but then as time passes and they do a lot of writes to their pool, the performance falls like a rock, because fragmentation has increased. ZFS fools you at first because it can be VERY fast even out to 95% the first time around.

Over time, there is more or less a bottom to where performance falls to. If you're not doing a lot of pool writes, you won't get there. If you are, you'll eventually get there. So the guys at Delphix actually took a single disk and tested this, and came up with what follows:

These are random writes to the pool. Notice that at 10% occupancy, a single disk is sustaining 6000KB/sec, or about 1500 IOPS/sec. Now the first words out of any storage admin's mouth would be "horses***, a single disk can't do that for random writes." True 'nuff. But what ZFS is doing is grouping writes together into contiguous blocks of free space, so even though userland may have written to sectors 3, 9, 77, and 121, ZFS writes those sequentially to a set of blocks. No seeks. But on the flip side, out there past 50%, ZFS isn't able to find large contiguous runs of free space, so not only the random writes suffer, but so do the sequentials.

So at the end of the day, for VM storage, what you really want is to fall into the window between 25-50% (or even lower) pool occupancy. ZFS can work some real performance magic the more free space is on the pool.

What *I* suggest is on a pool that's exclusively for iSCSI, you use 50% of the size for a zvol. That way, because you probably don't actually fill your VMFS datastore, you're forced into the happy regions of that graph. And it's better to not obsess over WHERE you fall; view the thing like you would an SSD. I have a VM filer here with 52TB of disk that's used to deliver 7TB of reliable VM storage. I don't get all OCD about it.

That VM filer, by the way, is made up of 2TB drives. You take 24 2TB drives (plus two spare). You make eight three-way mirrors. That's 16TB of pool space from 52TB of raw space, but includes two spares and dual disk redundancy for everything. Then you decide you really want to stay well within the happy parts of that graph above. So you only allocate 7TB. But you can use up to all of it, knowing it'll be reasonably zippy even when full.

I've written a lot about this and you can probably learn more by searching for me and the word "fragmentation" and "mirrors".