Compression and free space in pool

francomartin33

Dabbler
Joined
Aug 23, 2018
Messages
16
Hi, I have a pool with a single vdev with three 240gb ssds in raidz1 that I use for primary storage for ESXi. The problem is that I want to use as much space as I can but freenas doesnt let me create more zvols in the dataset because it says its full.
According to the zpool I used 345GB of space with only 82GB left,
according to the zvol I used 345GB of space with 234GB left and
according to ESXi, I used 268GB of space (in that datastore) with 71Gb left.
If I had to guess id say ESXi is just calculating the size of the vms in the datastore and calculating the difference with the datastore capacity, thats why it is not aware of compression in the storage backend but the difference between the zpool and the zvol is bugging me.
Quick note, this is not a production system. It has daily backups and nothing depends on this system.
To begin with, who should I trust? and then, how bad of an idea is it to force size of a new zvol to make use of the space I have.
I have a solid monitoring system so I could monitor how much space Im using, so Id like to know what metric I should be looking at.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
Who do you trust? All three, because they're telling you three different but all important things.

Wall of text incoming.

Your assumption on ESXi is correct as it has no visibility into the ZFS level - it can only see what is "free" at the VMFS partition level (71GB) as well as the unpartitioned space (you assigned 268GB to the datastore from a 345GB ZVOL, correct?)

For the ZVOL, ZFS will apply compression to the blocks written so it is able to squash the VMDKs and write a smaller total amount of data. That's why it sees a larger amount of free space (234GB) from the full size of the ZVOL.

And finally at the zpool level, this is not a sparse (thin) ZVOL, so the space was fully allocated. That's why the pool considers itself as 80% full already (it's full of ZVOL) and won't let you make more.

And then at an even lower layer, you have the SSD NAND itself which is constantly going through a state of programmed/dirty/free.

Is it a good idea to override the warning - definitely no. This is here to preserve the write performance on the pool, especially with block storage which fragments free space more easily.

Is it possible to get out of this? Yes. I would create a new datastore on another set of drives, migrate your data (or just torch it if your backups are recent enough) and then destroy the zpool entirely and recreate it. (This will also TRIM the devices by default, which will get you to a 100% fresh state)

Then create your zpool again, and when you provision the ZVOL, set it up for a smaller size (say, 200GB) and create it as a sparse volume. Think "thin disk" in VMware parlance - the blocks in the ZVOL won't consume pool space until they're written to, and blocks that get marked as free will be TRIMmed by ZFS (unless you've got a hardware implementation that broke TRIM). The smaller size will mean that you can gauge how much actual space on the pool will be taken up by the logical data.

Once you've presented the ZVOL, make sure you format it with VMFS6 from ESXi. This will let VMware do automatic space reclamation on the volume (which will translate down to UNMAP instructions on the ZVOL) and migrate/restore your VMs.

Monitor all three free space levels. If any of them hit 0% your VMs will "experience issues."
 

francomartin33

Dabbler
Joined
Aug 23, 2018
Messages
16
Great! Ill try that. Ill come back with the results soon
 

francomartin33

Dabbler
Joined
Aug 23, 2018
Messages
16
So that fixed it. Im now monitoring zpool free space. Thank you!!
 
Top