Total disk space not as expected/mismatch with CLI

[overkill]

Hi

I made a raidz2 array originally with 9 3TB disks and 1 1.5TB disk. I replaced the 1.5TB this morning with a 3TB, autoexpand on. The array finished resilvering and now the web GUI is showing 20.4TB.
I should have ~24TB. Taking into account the base10 that manufacturers use I should have ~22351 GB or 21.82TB.
I went in SSH and get this
zpool get free
NAME PROPERTY VALUE SOURCE
Volume0 free 23.7T -

zpool get size
NAME PROPERTY VALUE SOURCE
Volume0 size 27.2T -

What's weirder is this

The dataset inside the volume has larger size than the volume...
Running scrub on the volume at the moment, not sure if it'll help.

Any other reasons for the size differences?
Thanks for the help.

 

[dlavigne]

Did you figure this out?

 

[overkill]

nope. the scrub didn't do any justice, I'm still wondering why a dataset inside a volume can have larger space than the volume itself, and still not seeing as many TB as I expected.

 

[cyberjock]

There is nothing weird or creepy going on. Read up on snapshots, reservations, quotas, and datasets and it will all make sense to you.

Or, alternately go searching the forums as this has been explained in pain-staking detail at least 10 times this year. ;)

 

[overkill]

I suspected snapshots taking up room as well but this is a brand new array where I've only put data, not delete data. I checked snapshots and they're not taking up even a GB on their own.
Didn't set any quotas, not on the array, not on the dataset, so not sure where that enters the scene.
Reservations I shall brush up on once more. I remember looking over that possibility, don't remember how much top of my head but I didn't believe it would take 1.4TB for ZFS reservations. I had a conversation with my coworker who has a FreeNAS setup at his home and he didn't think reservations would take up that much either. His array is comprise of 2TB disks and needless to say he's not seeing the capacity discrepancy I'm seeing on mine.
I'd like to say I'm not the kind that go asking question without searching the forum first, but I may have done it wrong/missed some keywords. I did a google search with site:forum.freenas.org for "space" , "storage" and not finding much in the same situation that I'm having. Some users forget to subtract the number of disks, some other only missing a small amount which can be explained with reservations but nothing I find that's missing as much as mine. Some pointers/links would be appreciated.

 

[cyberjock]

Reservations can take up as little as 512 bytes or as much as 100% of the pool. ;)

 

[overkill]

So I went and read up on this https://docs.oracle.com/cd/E19253-01/819-5461/gazvb/index.html
Then did a zfs list command, correct me if I'm wrong, REFER is the reserve amount right? Here's what I got.
~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
Volume0 2.77T 17.7T 420K /mnt/Volume0
Volume0/.system 1000M 17.7T 384K /mnt/Volume0/.sy stem
Volume0/.system/cores 3.71M 17.7T 3.71M /mnt/Volume0/.sy stem/cores
Volume0/.system/samba4 833M 17.7T 4.00M /mnt/Volume0/.sy stem/samba4
Volume0/.system/syslog 163M 17.7T 1.95M /mnt/Volume0/.sy stem/syslog
Volume0/LDrive 2.72T 17.7T 2.72T /mnt/Volume0/LDr ive

If I add 17.7 and 2.72 I get 20.4TB, for LDrive dataset, just like in the GUI shows, if it's reserved for LDrive then it makes sense why Volume0 only shows 17.7T of total space.
What doesn't make sense is that I didn't set a reservation for LDrive dataset.

Other datasets settings are pretty much the same, 0 Quota and 0 reserved space.
I know ZFS takes some space on its own and that's the number I don't remember, but it's not much, shouldn't be much.
http://serverfault.com/questions/445639/how-much-overhead-does-zfs-have

So what gives?