Operation not permitted on new pool after replication from Truenas 12

[delta_alpha_november]

Hi, I am trying to migrate from TrueNAS core 12 to a new machine running Truenas scale bluefin which I just installed.

I have set up the new machine with a mirrord special device and a single 8TB HDD for now.
I replicated all datasets under the root level to my new machine using the UI recursively, which seemed to have worked.
Here are my replication settings:




The pool is encrypted and unlocked.

Code:

root@truenas[/mnt/Daten]# zfs list | grep -v boot
NAME                                                         USED  AVAIL     REFER  MOUNTPOINT
Daten                                                       4.78T  2.36T      336K  /mnt/Daten
Daten/Backups                                                916G  2.36T      184K  /mnt/Daten/Backups
Daten/Backups/location                                         916G  2.36T      244K  /mnt/Daten/Backups/location
Daten/Backups/location/Public                                 9.37G  2.36T     8.84G  /mnt/Daten/Backups/location/Public
Daten/Backups/location/Name3                                 30.2M  2.36T      240K  /mnt/Daten/Backups/location/Name3
Daten/Backups/location/admin                                   907G  2.36T      807G  /mnt/Daten/Backups/location/admin
Daten/Downloads                                              108G  2.36T      108G  /mnt/Daten/Downloads
Daten/Name1SSD                                               151G  2.36T      151G  -
Daten/gemeinsam                                             1.11T  2.36T      893G  /mnt/Daten/gemeinsam
Daten/home                                                  1.76T  2.36T      248K  /mnt/Daten/home
Daten/home/Name2                                           1.45T  2.36T      695G  /mnt/Daten/home/Name2
Daten/home/Name1                                             314G  2.36T      206G  /mnt/Daten/home/Name1
Daten/retropie                                               800M  2.36T      797M  /mnt/Daten/retropie
Daten/location                                                 775G  2.36T      774G  /mnt/Daten/location

However, the root dataset behaves very strange and I can't figure out why.
It will not let me mount some of the datasets or even just make files and/or directories in the pool directly, as per below.

Code:

root@truenas[/mnt/Daten]# mount | grep Daten
Daten on /mnt/Daten type zfs (rw,noatime,xattr,posixacl,casesensitive)
Daten/Backups on /mnt/Daten/Backups type zfs (rw,noatime,xattr,posixacl,casesensitive)
Daten/Backups/location on /mnt/Daten/Backups/location type zfs (rw,noatime,xattr,posixacl,casesensitive)
Daten/Backups/location/Name3 on /mnt/Daten/Backups/location/Name3 type zfs (rw,noatime,xattr,posixacl,casesensitive)
Daten/Backups/location/Public on /mnt/Daten/Backups/location/Public type zfs (rw,noatime,xattr,posixacl,casesensitive)
Daten/Backups/location/admin on /mnt/Daten/Backups/location/admin type zfs (rw,noatime,xattr,posixacl,casesensitive)
root@truenas[/mnt/Daten]# zfs mount -a
cannot mount '/mnt/Daten/Downloads': failed to create mountpoint: Operation not permitted
cannot mount '/mnt/Daten/gemeinsam': failed to create mountpoint: Operation not permitted
cannot mount '/mnt/Daten/retropie': failed to create mountpoint: Operation not permitted
cannot mount '/mnt/Daten/home': failed to create mountpoint: Operation not permitted
cannot mount '/mnt/Daten/location': failed to create mountpoint: Operation not permitted
root@truenas[/mnt/Daten]# zpool status Daten
  pool: Daten
 state: ONLINE
  scan: scrub repaired 0B in 06:41:02 with 0 errors on Fri Sep 22 19:22:12 2023
config:

        NAME                                      STATE     READ WRITE CKSUM
        Daten                                     ONLINE       0     0     0
          e8dee78e-de34-49d2-b4e5-4abdb79a1912    ONLINE       0     0     0
        special
          mirror-1                                ONLINE       0     0     0
            852f08f6-25e2-424c-87c6-42a53b75af4c  ONLINE       0     0     0
            a10e4cd4-31cf-4a7a-b370-202a08885446  ONLINE       0     0     0

errors: No known data errors
root@truenas[/mnt/Daten]# mkdir test
mkdir: cannot create directory ‘test’: Operation not permitted
root@truenas[/mnt/Daten]# ll
total 35
drwxr-xr-x 5 root 5 Sep 20 17:39 ./
drwxr-xr-x 3 root 3 Sep 22 12:26 ../
drwxr-xr-x 4 root 4 Sep 22 12:32 Backups/
drwxrwxrwx 2 root 2 Sep 20 14:40 OldDaten/
drwxr-xr-x 2 root 2 Sep 22 12:43 ix-applications/

The pool is not mounted read-only. The datasets I'm trying to mount aren't either:

Code:

root@truenas[/mnt/Daten]# zfs get readonly Daten/gemeinsam
NAME             PROPERTY  VALUE   SOURCE
Daten/gemeinsam  readonly  off     inherited from Daten

On the other hand some of the datasets I did replicate over work just fine:

Code:

root@truenas[/mnt/Daten/Backups]# mkdir test
root@truenas[/mnt/Daten/Backups]# ls
WindowsBackups  test  location
root@truenas[/mnt/Daten/Backups]# cd test
root@truenas[/mnt/Daten/Backups/test]# ls
root@truenas[/mnt/Daten/Backups/test]# touch file
root@truenas[/mnt/Daten/Backups/test]# ll
total 18
drwxr-xr-x 2 root 3 Sep 22 23:05 ./
drwxr-xr-x 5 root 5 Sep 22 23:05 ../
-rw-r--r-- 1 root 0 Sep 22 23:05 file
root@truenas[/mnt/Daten/Backups/test]# rm file
root@truenas[/mnt/Daten/Backups/test]# cd ..
root@truenas[/mnt/Daten/Backups]# rmdir test

I have tried changing the ACL type through the UI without luck.
I have checked that these datasets are not read-only.
I have tried exporting and reimporting the pool.

I'm pretty much out of ideas where the problem might be. The pool has been scrubbed today without any errors.


I could nuke the pool and start over but I'd really like to understand what went wrong. I think all of the affected datasets are datasets without POSIX ACLs on my old truenas core machine which have been used as SMB shares.

 

[delta_alpha_november]

I could not figure it out but I recreated the pool and the problem does no longer exist.