After 9.1->9.3 upgrade, dataset full. Nulling files reduces dataset size.

[Osiris]

Hi guys,

I've got a weird problem.
After an upgrade form FreeNas 9.1 to 9.3 (3 days ago since this post) one of my datasets has become full, while it was at 97% before.
It's a 6disk setup, ZFS2, with 3TB WD Reds.
The dataset was 100% of the volume.
I've filled up 97% if the available 10.5 TB (I know I know).

After the upgrade, the volume / dataset displayed this:

I admit no quota were set. (Feel free to advise here)

I've read on these fora and other FreeBSD sources that a full dataset is a problem when it comes to deleting files. Imho a major flaw of ZFS that needs to be countered by some default quota setting from FreeNas.

When I 'null' some files using for example
echo > some.file.avi
all that happens is
1. The file is set to 1B
2. the total size of the dataset is reduced !!

- df before:
raid6 11281717016 11281717016 0 100% /mnt/raid6
- I null a file here
- df after
raid6 11279547713 11279547713 0 100% /mnt/raid6

What can I do besides copying those 10 TB to another dataset/volume/nas ?

 

[9C1 Newbee]

Sooooooooooo...................what is your hardware specs?

 

[Osiris]

Nas3:
CPU: AMD A10-6700 APU with Radeon(tm) HD Graphics (3693.35-MHz K8-class CPU)
RAM: 8GB 1333 - 2 x 4GB Corsair 667 DDR3 DIMM 64b ECC
MB: ASRock FM2A85X-ITX
Gfx: Integrated AMD Radeon HD 7000 series graphics in A-series APU
HD: 6 x 3 TB WD Red SATA-600 - buffer: 64 MB cache

 

[mav@]

ZFS in FreeNAS 9.3 reserves about 3.5% of pool space for internal purposes, such as an ability to free some space at all, that was not guarantied before. It is any way highly not recommended to use more then 90% of pool space, so you should free at least another 1-7% of space.

 

[Osiris]

So this is something worth mentioning before upgrading, no ?
A recommendation isn't a requirement, imho.
Secondly, you say to 'free up some space'. This would mean that I would have to manually 'null' a lot of files via command line.

This is the first time I'm not happy about FreeNas

 

[mav@]

Secondly, you say to 'free up some space'. This would mean that I would have to manually 'null' a lot of files via command line.

There could be other ways to get space: delete old snapshots, add another disk/vdev to the pool, ...

 

[Osiris]

Thanks for the hint.
No snapshots were available.
Before adding another disk to the pool, I continued setting some files I no longer needed to 1B using commandline.
Although the the pool appears to remain saturated ...

Code:

raid6                                             11266039285 11266039285        0   100%    /mnt/raid6

I can remove files again via my share.
I'll do a gigantic cleanup first.

The problem appears solved for now, but the extra 3.5% of the pool requirement should be something mentioned when doing the upgrade.

Update: after deleting a few files, I once again get the
No space left on device
when doing rm on a file

 

[Osiris]

60 GB freed up. I'm in the clear after a lot of commandline work.

Code:

raid6                                                 10T     10T     56G    99%    /mnt/raid6

What is the use of reserving another 3,5% if actions on the dataset become restricted when the cap is reached ?

 

[mav@]

Is there a way to adapt the 80% warning to something manually set ?

There are GUI-configurable space thresholds for iSCSI volumes, but those general 80/90% alert thresholds are hardcoded IIRC. You may create feature request to make them configurable too.

But I would still recommend you to avoid filling pool too high. Extra fragmentation may hurt performance a lot, and may be problematic to get rid of.