Weird space usage on ZFS pool after destroyed datasets

[Krautmaster]

Dear all,

I noticed that there exist a few threads mentioning that anything is weird on ZFS and deletion but I cant get behind so far.

What I did and have.

-> freenas 11.2 most actual
-> backupped from one RaidZ1 pool to a second, several TB
-> backup pool was reported to be full so i deleted several TB of that backup pool. The "rm" command took like years to delete the folder so I destroyed the datasets

2019-09-24.00:03:32 zpool set cachefile=/data/zfs/zpool.cache RaidZWDGreen [user 0 (root) on ]
2019-09-26.07:44:06 zfs destroy -r RaidZWDGreen/ncdata [user 0 (root) on freenas.local]
2019-09-26.07:45:58 zfs destroy -r RaidZWDGreen/cloud [user 0 (root) on freenas.local]
2019-09-26.07:47:36 zfs destroy -r RaidZWDGreen/Software [user 0 (root) on freenas.local]
2019-09-27.07:26:10 zfs destroy -r RaidZWDGreen/Backup [user 0 (root) on freenas.local]
2019-09-27.13:18:14 zpool import -c /data/zfs/zpool.cache.saved -o cachefile=none -R /mnt -f 4492606138290332196 [user 0 (root) on ]

any how, the free space is still way less than it should be related to size and the usage.

RaidZWDGreen 17.1T 475G 166K /RaidZWDGreen
RaidZWDGreen/Benutzerdaten 3.83T 475G 3.83T /RaidZWDGreen/Benutzerdaten
RaidZWDGreen/Medien 6.65T 475G 6.65T /RaidZWDGreen/Medien

17TB, not even 10TB used and only 475 GB free? It can be that i may take a few time to free that TB of data but not several days or??

How can that be. And no, there are no known snapshots. Thanks for your help.

Edit. Btw, it would be helpful if i could get one of that destroyed datasets restored, wanted to check something in there. Is that possible? -the nfFx import did not help. But as the space seem to be used after destruction of the dataset, the data might be still there?

 

[Krautmaster]

oh my. The free space seem to be growing but pretty slow.

RaidZWDGreen 17.1T 489G 166K /RaidZWDGreen
RaidZWDGreen/Benutzerdaten 3.83T 489G 3.83T /RaidZWDGreen/Benutzerdaten
RaidZWDGreen/Medien 6.65T 489G 6.65T /RaidZWDGreen/Medien

that is crazy. <500GB in 30h.

Edit: da fu*k is it doing. I can set the "zfs list" command several times and the space seem to fluctuate.

root@freenas[/mnt]# zfs list
...
RaidZWDGreen 17.1T 487G 166K /RaidZWDGreen
RaidZWDGreen/Benutzerdaten 3.83T 487G 3.83T /RaidZWDGreen/Benutzerdaten
RaidZWDGreen/Medien 6.65T 487G 6.65T /RaidZWDGreen/Medien

root@freenas[/mnt]# zfs list
...
RaidZWDGreen 17.1T 489G 166K /RaidZWDGreen
RaidZWDGreen/Benutzerdaten 3.83T 489G 3.83T /RaidZWDGreen/Benutzerdaten
RaidZWDGreen/Medien 6.65T 489G 6.65T /RaidZWDGreen/Medien

root@freenas[/mnt]# zfs list
...
RaidZWDGreen 17.1T 477G 166K /RaidZWDGreen
RaidZWDGreen/Benutzerdaten 3.83T 477G 3.83T /RaidZWDGreen/Benutzerdaten
RaidZWDGreen/Medien 6.65T 477G 6.65T /RaidZWDGreen/Medien
freenas-boot 3.16G 27.4G 176K none


it rises and drops again? there is not a single action by my ongoing on this pool. What the heck happens here? ;)

Edit. Seem to rise slowly but it does even if the reporting messes it sometimes

 

[garm]

You are told not to fill the pool for a reason.. by filling it past 97% ZFS is severely crippled. What you are seeing is probably the effects of there only being a handful of GB left on each drive and ZFS struggles to clear the unused space.

 

[Krautmaster]

okay, so all i can do i wait correct? It freed 50GB in the past 6 hours... still pretty slow tough.