Cannot delete folders on imported pool after dataset was deleted

rokyo

Dabbler
Joined
May 12, 2023
Messages
11
Hi,

I have converted my old PC to a TrueNAS Scale server yesterday and ran into the following issue today:

The PC was previously running Debian 12 and had a ZFS pool called "tank" made up of 3x 1TB drives in RAIDZ1 with several datasets on it including one named "home" which had a sub-dataset named "rokyo" containing my home folder. Since my account was the only account on that machine, it had the UID 1000 and a group also named "rokyo" with GID 1000.

---------------
Backstory (maybe tl;dr and not needed, although any of these steps here could be part of the problem):
To convert the Debian machine to a TrueNAS server, I just installed TrueNAS on the SSD which previously had the Debian installation and imported the "tank" pool afterwards in the GUI.
This didn't work because I had not properly exported it in Debian before, so I imported it via the TrueNAS shell with `zpool import -f tank` which also didn't work because the pool had a log device before which I took out before installing TrueNAS (without detaching/deleting it from the pool first).
So, I found this thread (https://www.truenas.com/community/t...with-missing-log-device-missing-device.82200/) and used the exact command from there to import the pool on the command line in TrueNAS: `zpool import -f -F -m tank` which worked.
Then I removed the missing log device from the pool so it wasn't degraded anymore (and later attached a different SSD as SLOG again).
Now I had the pool successfully imported and it showed in the shell when running `zpool list` and the datasets with `zfs list`. However, they were not visible in the GUI and were gone again after a reboot.
So, after the reboot, I imported the pool again on the shell with `-f -F -m`, then immediately exported it again on the shell and imported it through the GUI (which now worked, since the pool was properly exported now.
Now I had the pool and datasets showing up in the GUI and this persisted through a reboot as well.
Only problem left: All datasets in "tank" were encrypted with a raw key in the Debian machine, so they all showed as "locked" in the GUI. Trying to unlock them via the GUI failed because, apparently, TrueNAS expects a JSON file as the unlock key and not RAW (? - at least that's what the error message implied). So I unlocked them by copying the RAW key files to the location where they were on the Debian machine (which was stored in all dataset properties under "keylocation" and then unlocked them in the shell with `zfs load-key -a` which worked. The datasets were now all showing as "unlocked" in the GUI and I could `cd` into them in the shell under `/mnt/tank`.
Since I did not want to keep the data in these encrypted datasets, I created a new (unencrypted) dataset `tank/home2` and moved all data from the encrypted (but unlocked) `tank/home/rokyo` to this new dataset with `sudo mv * /mnt/tank/home2`, which took quite some time (for ~1TB).
After the move command was finished, all data was present correctly in `/mnt/tank/home2` and was gone from `/mnt/tank/home/rokyo` EXCEPT the empty base folders. These were not deleted in `/mnt/tank/home/rokyo` because of a "permission error" but were empty. They were not deleted even though the move command was done via `sudo` so as `root` I assume... all data inside the folders were actually moved, so deleted at the source and now only present at the destination of the move command. Why not the "base folders"?
After this, I deleted the `tank/home/rokyo` and `tank/home` datasets completely in the GUI, along with any other excrypted datasets which were in `tank` (which were all empty file/folder-wise anyways), so only `tank/home2` was now remaining.
---------------

This is where my current problem comes in:

These non-deletable folders (normal Linux home subfolders like "Documents", "Downloads", "Pictures", etc) still exist at `/mnt/tank/home/rokyo` now, even after this dataset and its parent dataset were deleted and the machine was rebooted several times. Why is this still here and "mounted" (`mount` does not show this)?

The folders all show owner as 1000 and group as 1000 in `ls -hl`, as UID and GID which does not exist on the TrueNAS and was my old UID/GID from Debian. I cannot delete these folders with `sudo rm -rf Documents` for example and also not transfer ownership to admin via `chown -R admin:admin Documents`, both result in "operation not permitted".

I even created a group called "rokyo" with GID 1000 and a user named "rokyo" with UID 1000, gave them "all sudo" permissions and tried deleting the folders as "rokyo", which also gives "operation not permitted" even though `ls -hl` now shows owner/group as "rokyo/rokyo" instead of "1000/1000". What is happening here?

The "base folder" of the old `tank/home` dataset at `/mnt/tank/home` is showing "root/root" as owner/group by the way, but this also cannot be deleted via `sudo rm -rf /mnt/tank/home` and shows "rm: cannot remove '/mnt/tank/home/rokyo/Pictures': Permission denied" for every one of the subfolders under `../home/rokyo`. I always thought that root can deleted everything?

I can also not create a new dataset `tank/home` anymore now, because the mount position is not empty (obviously, because the un-deletable folders are still there).

Did I bork my install completely? Why, though? Is there any way to delete `/mnt/tank/home` including everything below it, so that I can re-create a new `tank/home`?
 
Top