Question about Zvol Space Management for VM

[Jack99]

Hi,

totally new to TrueNAS Scale and playing arround with it for the first time, so forgive me for my dumb beginner questions as i couldn't really find an answer on the forum yet. I am running version 22.12.4.2

I have a test VM running with PfSense which is stored on a (thick) 120GiB Zvol (see attachments). The VM itself uses about 1GB diskspace, but according to the "Zvol Space Management" the total data written is 52GiB, and about everyday it adds 1GiB to it so this is growing fast!
(yesterday it was 51GiB, the day before 50GiB and so on).

How is this possible, where is all this "Data Written" coming from and what can i do about it? I don't use any snapshots, just installed the VM with PfSense that's all.

Additional question:
I made the 120GiB Zvol way too big and want to make it smaller, but already understood that it isn't possible to shrink. Is it possible to create a new (smaller) Zvol and move the VM to the new Zvol, and delete the old one? And can i use ZFS send command for that?

 

[NugentS]

how much space does pfsense think is being used in the zvol?

I don't think so as ZFS / TrueNAS does not understand the data inside the ZVOL.

 

[Patrick M. Hausen]

Also what is the filesystem pfSense is using inside the VM?

If it's ZFS due to the copy-on-write nature your Zvol will always blow up to its maximum size. Any block that is written at least once in the guest OS will be "used" viewed from ZFS outside the VM. TrueNAS/ZFS cannot tell how much space your VM is really using, only which blocks have been touched and which have not.

Inside VMs UFS/Ext4 are much better choices than ZFS. You can always do snapshots and backup on the outside.

And no, you cannot shrink a Zvol, not even with ZFS send/receive. If you copy the Zvol with send/receive you will get an identically sized copy.

Backup your pfSense config, create a smaller Zvol, reinstall, restore config. 30-40 G should be plenty.

 

[Jack99]

how much space does pfsense think is being used in the zvol?

I don't think so as ZFS / TrueNAS does not understand the data inside the ZVOL.

How or where can i check that?

 

[Jack99]

Also what is the filesystem pfSense is using inside the VM?

If it's ZFS due to the copy-on-write nature your Zvol will always blow up to its maximum size. Any block that is written at least once in the guest OS will be "used" viewed from ZFS outside the VM. TrueNAS/ZFS cannot tell how much space your VM is really using, only which blocks have been touched and which have not.

Inside VMs UFS/Ext4 are much better choices than ZFS. You can always do snapshots and backup on the outside.

And no, you cannot shrink a Zvol, not even with ZFS send/receive. If you copy the Zvol with send/receive you will get an identically sized copy.

Backup your pfSense config, create a smaller Zvol, reinstall, restore config. 30-40 G should be plenty.

PfSense uses ZFS indeed, as that is the prefered filesystem for PfSense according to documentation. I already found some posts where you mentioned this before about the ZFS filesystem taking all of the space. But is that really a problem if it "blows up"to maximum size?

Good tip about recreating a smaller Zvol and reinstall PfSense, creating a new 30-40G was exactly what i was planning to do.

 

[Patrick M. Hausen]

But is that really a problem if it "blows up"to maximum size?

Not in general but frequently people overprovision VM storage expecting a behaviour similar to VMware "thin" images. These blow up, too, if the guest OS uses ZFS.

If you have the space, go ahead.

 

[Whattteva]

pfSense shouldn't use much at all. I've been running OPNsense (pfSense fork) for the last 9 months on an 8GiB ZVOL and it's only using 50% of it. I am using UFS though.

 

[Jack99]

Not in general but frequently people overprovision VM storage expecting a behaviour similar to VMware "thin" images. These blow up, too, if the guest OS uses ZFS.

If you have the space, go ahead.

I was more refering to if it will "blow up" until the available space in the thick Zvol (the 120GiB) is taken or will it grow outside of the Zvol and take additional space from the storagepool, with risk of no available space in the storagepool at some point? Because that the reason i choose a thick volume so it cannot exceed that.

 

[Jack99]

pfSense shouldn't use much at all. I've been running OPNsense (pfSense fork) for the last 9 months on an 8GiB ZVOL and it's only using 50% of it.

I know, i have run PfSense for many years on the Qnap QTS OS with Virtualisation Station and it was at maximum a few gigis, that's why i didn;t understand why this thing was growing till 53GB as we speak.

 

[NugentS]

@Jack99. I use OpnSense but on the front panel its got a disk usage widgit that tells me I what I am using as a %

pfsense and opnsense are not dissimilar and mine is using the square root of bugger all in disk space. Mine is physical BTW. Due (as @Patrick M. Hausen says) to the nature of COW the whole ZVOL could eventually be thought of as used by TrueNAS. But whats important is what the application using the ZVOL thinks.

 

[Jack99]

I know, i have run PfSense for many years on the Qnap QTS OS with Virtualisation Station and it was at maximum a few gigis, that's why i didn;t understand why this thing was growing till 53GB as we speak.

PS: I got tired of Qnap QTS with all their bloatware these days and that's why i choose to give Truenas a shot, which i like very much so far. But it has a learning curve... New world for me

 

[Jack99]

@NugentS I believe i posted a similar widget from PfSense in my starting post, see attachment there

 

[NugentS]

It mentions zvol, so I assumed its was from TrueNAS. a VM surely wouldn't know its using a ZVOL unless thats just a naming trick. I do not have pfsense.

 

[sfatula]

You can create sparse zvols. I don't believe the Scale UI allows that yet, just from command line. That way, it doesn't pre-allocate the space. I didn't do that on my original creation of my zvols, which is ok for me. But I DID use the technique on my remote backup machine so that space was not wasted there. So, the replicated copy uses less space.

 

[Patrick M. Hausen]

Create, then use zfs inherit -S refreservation <pool>/path/to/<zvol>.

 

[Jack99]

Allright update:

I removed the old Zvol and created a new one of 50GB and reinstalled the Pfsense box with UFS filesystem instead of ZFS to stop the growing (as @Patrick M. Hausen suggested). I left the Zvol as "Thick" instead of "Sparse". Only reason i use Thick volume is because i am used to do that in other hypervisors like Vmware (slightly better performance), but maybe that doesn't matter with ZFS i don't know.

Here are the results, Zvol says 793MB written and Pfsense says 1,6GB in use. I still find it weird why there is a difference between those two, i guess i just don't understand yet how ZFS / Zvols works under the hood... But as long as the growing stops i am ok with that for now.