Help on full readonly pool needed

[icos66]

I have a pool that was set readonly a long time ago. Recently I discovered that scrubbing did not run anymore. On CLI I get 'cannot scrub pool: out of space'.
Then I tried to turn readonly off but get the same error: cannot set property for 'pool': out of space
zpool list reports some free space of 4GB:

Code:

NAME     SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
pool    16.4T  16.4T  4.34G        -         -    68%    99%  1.00x    ONLINE  /mnt

I have played with tunable vfs.zfs.spa.slop_shift and had set it higher, but that changes nothing.
Is there any way to delete some files on the pool to get it back in 'normal readonly' mode? My ultimate resort would be to copy all data (which works) to another pool and setup that pool anew.

Cord

 

[c77dk]

Maybe you can free a little space by disabling swap (if there's any?) But wait to see if some of the "older" members chime in, and see if they say my thought is a no-go.

 

[Etorix]

With a Copy-on-Write file system, freeing space requires space to write updated metadata. It seems you've found out the hard way why a pool should never, never ever, be allowed to reach 99-100% full.

If this is a mirror pool, you can add an extra vdev to have extra space and be able to work with the pool. If you're ready to delete files to get space back, this might even be a single drive—which should then be removed as soon as the pool is repaired and before any new data is thrown in.
If this is raidz, you're stuck. Backup, destroy the pool, create a bigger one and restore.

 

[danb35]

You should have started getting alerts when your pool hit 80% capacity, which would have given you some time to address this. Now that you're effectively 100% full, your options are much more limited. But see if you can export the pool, followed by importing it again: zpool export pool; zpool import pool. If the latter succeeds, see what you have in the way of snapshots: zfs list -t snapshot. If you have snapshots, destroying a few of them may free up enough space to unbork your pool. If you don't, deleting some files may be possible.

Maybe you can free a little space by disabling swap (if there's any?)

Since the partition would still be there, I wouldn't think this would help.

 

[icos66]

Thanks for your thoughts!

You should have started getting alerts when your pool hit 80% capacity, which would have given you some time to address this. Now that you're effectively 100% full, your options are much more limited. But see if you can export the pool, followed by importing it again: zpool export pool; zpool import pool. If the latter succeeds, see what you have in the way of snapshots: zfs list -t snapshot. If you have snapshots, destroying a few of them may free up enough space to unbork your pool.

export / import acatually works. But there are no snapshots at all.

If you don't, deleting some files may be possible.

It's readonly and trying to turn this off results already in 'out of space'
Historically, I filled the disk up to a few GB and set it readonly. This worked for 1,5 years and suddenly it is 'full' - quite unexpected.

With a Copy-on-Write file system, freeing space requires space to write updated metadata. It seems you've found out the hard way why a pool should never, never ever, be allowed to reach 99-100% full.

I bet I did ..

 

[danb35]

It's readonly

I wonder if you can import it while explicitly disabling readonly--I think the syntax would be something like zpool import -o readonly=off pool

 

[icos66]

Correction:

I wonder if you can import it while explicitly disabling readonly--I think the syntax would be something like zpool import -o readonly=off pool

The import gave no error but it's still readonly, can't delete a file.
Any other ideas left? Otherwise I will set it up from scratch..

 

[joeschmuck]

My ultimate resort would be to copy all data (which works) to another pool and setup that pool anew.

I think this is where you are at.

 

[Arwen]

You might check if any dataset or zVol has unused space in a Reservation. Then clearing the reservation might solve the problem.