pool and dataset reporting higher usage than what's actually stored

[Derek Storvik]

FreeNAS-9.1.1-RELEASE-x64 (a752d35)
I have a storage pool storage1 that has a capacity of 13.6 TB 10 drives in a "raid 10" setup ( mirrors and stripes.)

The zpool reports
size = 13.6 tb
free = 2.63 tb
allocated = 11.0TB

zfs get all reports
storage1 type filesystem -
storage1 used 11.0T -
storage1 available 2.42T -
storage1 referenced 152K -
storage1 quota none default
storage1 reservation none default
storage1 recordsize 128K default
on this volume/pol is a dataset called S1-Dataset
zfs get all reports on it
storage1/S1-Dataset type filesystem -
storage1/S1-Dataset used 11.0T -
storage1/S1-Dataset available 2.42T -
storage1/S1-Dataset referenced 11.0T -
storage1/S1-Dataset quota none default
storage1/S1-Dataset reservation none default
storage1/S1-Dataset recordsize 128K default
storage1/S1-Dataset usedbydataset 11.0T -
on this dataset sits a file extent for iscsi that is connected to vmware hosts
this all seems fine until vmware reports 8TB free on this volume and only 5Tb of actual machines and data are on the iscsi volume.
I"m trying to figure out why it seems that zfs thinks it is using or IS using so much more space than the data I have put on the NAS from VMWARE.
Any help would be appreciated.
Thanks

 

[bigphil]

Do you have any snapshots or clones that are holding onto data? Whats the output of "zfs list -t all -r storage1" ?

 

[Derek Storvik]

No snapshots or clones

NAME USED AVAIL REFER MOUNTPOINT
storage1 11.0T 2.42T 152K /mnt/storage1
storage1/S1-Dataset 11.0T 2.42T 11.0T /mnt/storage1/S1-Dataset

 

[Derek Storvik]

wait, you referring to ZFS snapshots and clones right? Don't have any of those.

Thanks

 

[bigphil]

Yes, I was referring to ZFS snaps and clones. What is the output of "zfs get copies" ?

 

[Derek Storvik]

[root@nas3] /mnt/storage1/S1-Dataset# zfs get copies
NAME PROPERTY VALUE SOURCE
storage1 copies 1 default
storage1/S1-Dataset copies 1 default
storage2 copies 1 default
storage2/S2-dataset copies 1 default
storage2 is set up the same way but slightly smaller. it seems to be starting to exhibit the same issues but not to the same degree because it's under less use. So I only focused on providing details on storage1.
Thanks for your help.

 

[bigphil]

Just to double check the snapshots, what's the output of "zfs list -t all -o space" ? What's probably happening is that I don't think FreeNAS has current support for SCSI UNMAP. Your space will never decrease in this case because as data is written, the blocks never get freed by the system and space utilization just keeps increasing. I believe the roadmap for future versions of FreeNAS is to have a new iSCSI kernel module and maybe that will support reclaiming space. You might consider using NFS for hosting your VM's instead of iSCSI. If you do this, a SLOG would advisable as the NFS writes are synchronous.

Edit: To add to this, I found this passage from a few posts on the net that might zero out the blocks if you have compression on the dataset/volume:
---------------------------
"One way of reclaiming space with iSCSI ZVOL is to enable compression on the ZVOL and then "fill" the disk with zero file from the host that is using the iSCSI ZVOL. If freenas reports you have 1G left storage but the host using that ZVOL reports 89 GB free. just run in the host 'dd if=/dev/zero of=bigfile bs=1g count=88' then remove bigfile

This will cause the freenas host to free up 88G as all the 0's get compressed"
---------------------------

Up to you to test this out to see if it'll work in your case because you have a file extent and not a device extent/zvol ;-)

 

[Derek Storvik]

Seriously?!?!? that seems like that would make iSCSI useless on freenas yet I haven't seen that kind of issues raised in my searching so far. I did manage to sort of confirm that by deleting a couple small VM's ( 60Gb ) on storage 2 and the free/used numbers did not change.... but that could practically be lost in rounding on 10TB volumes....
so is that space not reclaimable? or is it not reporting as free? For example on storage 1 VMware says i have over 7tb free. ZFS says i have 2.4tb free. If I tried to write 5tb of data to a new VM would that work and i would just be "using" more than i have? or would the operation fail and then my drive would be "full" freezing everything with little recourse?

zfs list -t all -o space
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
storage1 2.41T 11.0T 0 152K 0 11.0T
storage1/S1-Dataset 2.41T 11.0T 0 11.0T 0 0
storage2 5.05T 5.65T 0 152K 0 5.65T
storage2/S2-dataset 5.05T 5.65T 0 5.65T 0 0

 

[HoneyBadger]

I'm going to try recreating this, but I can't recall this happening for me re: SCSI UNMAP. Mind you, I've always used file-based extents. With file-based extents the underlying ZFS system will be "full" of the iSCSI file, but "inside the file" there are unused/dirty blocksthat the VMFS should be able to overwrite with new data.

 

[bigphil]

Yeah, it should be a fairly easy test to see if the dirty/unused space can be reallocated. I would be curious to see the results of this test on a device extent as well as a file extent to see if there is a difference.

 

[bigphil]

Ok, so I did a test and am seeing the same results as you...I do have a little bit different setup though and will perform another test tomorrow that mimics your setup.
Setup > Current Pool used space = 280 GB. Created a 2GB zvol and used as iSCSI device extent. Pool used space is now 282GB. zvol has lz4 compression and no dedupe. Filled the LUN from a VMware ESXi 5.5 host to near max capacity. Deleted all of the files and filled the LUN again. I did this about 5 times and each time it was about 1.2GB worth of data. The entire time my pool used space remained at 282GB. This tells me that the dirty blocks were reused. The used space of the zvol never decreased though (USEDDS field), so it definitely doesn't unmap the unused blocks when the files are deleted by the ESXi host, but it does reuse them.

I'll test a file extent tomorrow.

 

[cyberjock]

FreeBSD's iSCSI implementation does not reclaim space once it's been assigned. ;) It's something that is to be fixed in the future. Not sure what the timetable is.

 

[HoneyBadger]

Thanks @bigphil for verifying the remapping of dirty blocks, I got busy with other work.

As long as VMFS will reuse blocks, the lack of UNMAP isn't a huge deal except against an all-SSD pool; without the space being reclaimed, the drive can't TRIM the blocks and performance will suffer. Until it's patched I'd say use NFS instead.

 

[cyberjock]

Actually, performance and SSD lifespan could suffer.

The recommendation is to either use NFS(which often isn't a "good" compromise) or continue to use iSCSI, but ensure you don't fill a pool excessively full so the drive can do its own maintenance. I'd guess that 80% full is probably a good maximum place to go.

 

[HoneyBadger]

Actually, performance and SSD lifespan could suffer.

The recommendation is to either use NFS(which often isn't a "good" compromise) or continue to use iSCSI, but ensure you don't fill a pool excessively full so the drive can do its own maintenance. I'd guess that 80% full is probably a good maximum place to go.

Underprovisioning can help, but yes I agree that lifespan will be hurt as well. Using all available blocks on an SSD is bad regardless of what's causing it, that's why people shouldn't use DBAN or other tools designed to wipe magnetic media - use ATA Secure Erase or the vendor's toolbox.

NFS is definitely a compromise especially if you only have GbE backbone; no MPIO means a single GbE can become your bottleneck.

I assume that since UNMAP support is on the roadmap there's no point in filing a bug, or is there already a case open?

 

[cyberjock]

Not aware of any open bug reports, but if someone put one in it would be closed to "3rd party to resolve".

 

[Serverbaboon]

Head hurts :(,

I think this inefficient space usage is mentioned elsewhere although it may have been due to block size, but I am struggling to understand from the space usage issue which is best, file or device extents?

 

[cyberjock]

Head hurts :(,

I think this inefficient space usage is mentioned elsewhere although it may have been due to block size, but I am struggling to understand from the space usage issue which is best, file or device extents?

Both will suffer in the exact same way, and for the exact same reasons.

 

[bigphil]

@Derek Storvik, out of curiosity, what is the output when you run this command "zdb -C storage1 | grep ashift"? I ask because you say you're running 9.1.1 and there might have been a fix applied in 9.2.1, here. Unfortunately for you, it probably wont do you any good because you cant change your ashift values after the pool is created, but it could further explain your situation.

 

[Derek Storvik]

Sorry for my long departure form the thread. the ashift is 12 on everything. So essentially we are syaing the usage in freenas will expand to full size eventually and it really is a non useful value. as their are dirty bits in the file that iscsi and vmware are aware of and the freespace vmware reports is actually accurate?