Pool datasets missing from mountpoint

[Boneshark]

After a reboot earlier today the pool and all the datasets are missing from the mountpoint (/mnt). The ZPOOL and ZFS listings (as well as the 11.2 GUI storage display show them to be fine. I have tried various things such as export/import the pool to no avail. I am able to create a new dataset in the pool and it shows in the mountpoint. I also coped (from and RSYNC backup) a few of the datasets back into the pool and they are accessible from a windows share.

Output of the list commands all look normal with the correct sizes:
root@freenas[/mnt/NAS_Backup_USB/NAS_5TB]# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
NAS_5TB 10.9T 4.87T 6.01T - - 4% 44% 1.00x ONLINE -
NAS_Backup_USB 5.44T 3.36T 2.07T - - 1% 61% 1.00x ONLINE -
freenas-boot 114G 849M 113G - - - 0% 1.00x ONLINE -
root@freenas[/mnt/NAS_Backup_USB/NAS_5TB]# zfs list
NAME USED AVAIL REFER MOUNTPOINT
NAS_5TB 2.36T 2.75T 1.92M /NAS_5TB
NAS_5TB/Acronis 816G 2.75T 816G /NAS_5TB/Acronis
NAS_5TB/MacBackup 41.0G 2.75T 41.0G /NAS_5TB/MacBackup
NAS_5TB/Mike 163K 2.75T 163K /NAS_5TB/Mike
NAS_5TB/Music 1.09T 2.75T 1.09T /NAS_5TB/Music
NAS_5TB/Win10 414G 2.75T 414G /NAS_5TB/Win10
NAS_5TB/iocage 22.8G 2.75T 3.97M /NAS_5TB/iocage
NAS_5TB/iocage/download 391M 2.75T 128K /NAS_5TB/iocage/download
NAS_5TB/iocage/download/11.1-RELEASE 119M 2.75T 119M /NAS_5TB/iocage/download/11.1-RELEASE
NAS_5TB/iocage/download/11.2-RELEASE 272M 2.75T 272M /NAS_5TB/iocage/download/11.2-RELEASE
NAS_5TB/iocage/images 128K 2.75T 128K /NAS_5TB/iocage/images
NAS_5TB/iocage/jails 21.2G 2.75T 128K /NAS_5TB/iocage/jails
NAS_5TB/iocage/jails/plex 21.2G 2.75T 256K /NAS_5TB/iocage/jails/plex
NAS_5TB/iocage/jails/plex/root 21.2G 2.75T 22.1G /NAS_5TB/iocage/jails/plex/root
NAS_5TB/iocage/log 134K 2.75T 134K /NAS_5TB/iocage/log
NAS_5TB/iocage/releases 1.22G 2.75T 128K /NAS_5TB/iocage/releases
NAS_5TB/iocage/releases/11.2-RELEASE 1.22G 2.75T 128K /NAS_5TB/iocage/releases/11.2-RELEASE
NAS_5TB/iocage/releases/11.2-RELEASE/root 1.22G 2.75T 1.19G /NAS_5TB/iocage/releases/11.2-RELEASE/root
NAS_5TB/iocage/templates 128K 2.75T 128K /NAS_5TB/iocage/templates
NAS_5TB/test 128K 2.75T 128K /NAS_5TB/test
NAS_Backup_USB 3.36T 1.90T 2.35T /mnt/NAS_Backup_USB
NAS_Backup_USB/NAS_Backup_USB 1.01T 1.90T 1.01T /mnt
freenas-boot 849M 110G 64K none
freenas-boot/.system 80.4M 110G 40K legacy
freenas-boot/.system/configs-31d3c4dc156b45b8879a33cd927b3068 1.80M 110G 1.80M legacy
freenas-boot/.system/cores 1.58M 110G 1.58M legacy
freenas-boot/.system/rrd-31d3c4dc156b45b8879a33cd927b3068 15.9M 110G 15.9M legacy
freenas-boot/.system/samba4 90K 110G 90K legacy
freenas-boot/.system/syslog-31d3c4dc156b45b8879a33cd927b3068 61.0M 110G 61.0M legacy
freenas-boot/.system/webui 29K 110G 29K legacy
freenas-boot/ROOT 767M 110G 29K none
freenas-boot/ROOT/Initial-Install 1K 110G 759M legacy
freenas-boot/ROOT/default 767M 110G 760M legacy
root@freenas[/mnt/NAS_Backup_USB/NAS_5TB]#

The contents of /mnt however show none of these datasets (other than the ones I copied including the pool root NAS_5TB:
root@freenas[/mnt/NAS_Backup_USB/NAS_5TB]# cd /mnt
root@freenas[/mnt]# ls
NAS_5TB NAS_Backup_USB
root@freenas[/mnt]# ls NAS_5TB
Acronis iocage Mike
root@freenas[/mnt]#

I have a backup on USB that will take me a few days to copy back and I am trying to avoid that. Any suggestions on how I can remount the pool/datasets back into the mountpoint. I have tried various zpool/zfs mount commands but nada. Should I delete the pool and reload everything (includes my plex server) or is there an easier way?

I am fairly literate in UNIX but have only been running Freenas for about 6 months and any help is appreciated

 

[sretalla]

what does zfs mount show?

 

[Boneshark]

root@freenas[/mnt]# zfs mount
freenas-boot/ROOT/default /
NAS_Backup_USB/NAS_Backup_USB /mnt
NAS_Backup_USB /mnt/NAS_Backup_USB
freenas-boot/.system /var/db/system
freenas-boot/.system/cores /var/db/system/cores
freenas-boot/.system/samba4 /var/db/system/samba4
freenas-boot/.system/syslog-31d3c4dc156b45b8879a33cd927b3068 /var/db/system/syslog-31d3c4dc156b45b8879a33cd927b3068
freenas-boot/.system/rrd-31d3c4dc156b45b8879a33cd927b3068 /var/db/system/rrd-31d3c4dc156b45b8879a33cd927b3068
freenas-boot/.system/configs-31d3c4dc156b45b8879a33cd927b3068 /var/db/system/configs-31d3c4dc156b45b8879a33cd927b3068
freenas-boot/.system/webui /var/db/system/webui
NAS_5TB /NAS_5TB
NAS_5TB/Acronis /NAS_5TB/Acronis
NAS_5TB/MacBackup /NAS_5TB/MacBackup
NAS_5TB/Mike /NAS_5TB/Mike
NAS_5TB/Music /NAS_5TB/Music
NAS_5TB/Win10 /NAS_5TB/Win10
NAS_5TB/iocage /NAS_5TB/iocage
NAS_5TB/iocage/download /NAS_5TB/iocage/download
NAS_5TB/iocage/download/11.1-RELEASE /NAS_5TB/iocage/download/11.1-RELEASE
NAS_5TB/iocage/download/11.2-RELEASE /NAS_5TB/iocage/download/11.2-RELEASE
NAS_5TB/iocage/images /NAS_5TB/iocage/images
NAS_5TB/iocage/jails /NAS_5TB/iocage/jails
NAS_5TB/iocage/jails/plex /NAS_5TB/iocage/jails/plex
NAS_5TB/iocage/jails/plex/root /NAS_5TB/iocage/jails/plex/root
NAS_5TB/iocage/log /NAS_5TB/iocage/log
NAS_5TB/iocage/releases /NAS_5TB/iocage/releases
NAS_5TB/iocage/releases/11.2-RELEASE /NAS_5TB/iocage/releases/11.2-RELEASE
NAS_5TB/iocage/releases/11.2-RELEASE/root /NAS_5TB/iocage/releases/11.2-RELEASE/root
NAS_5TB/iocage/templates /NAS_5TB/iocage/templates
NAS_5TB/test /NAS_5TB/test
root@freenas[/mnt]#

 

[dlavigne]

What is the mountpoint used for? Is something else mounted there?

 

[Boneshark]

Basically a simple setup of a single pool with a plex installation. The pool contains several directories of media files as well as the plex jail. Structured like this (some directories left out)
/mnt/NAS_5TB
NAS_5TB/Acronis
NAS_5TB/MacBackup
NAS_5TB/Mike
NAS_5TB/Music (this has all the media files)
NAS_5TB/Win10
NAS_5TB/iocage
NAS_5TB/iocage/jails
NAS_5TB/iocage/jails/plex
NAS_5TB/iocage/jails/plex/root
NAS_5TB/iocage/jails/plex/root /media
NAS_5TB/iocage/jails/plex/root/media/mountpoint back to NAS_5TB/Music

(looks recursive but follows the plex recommendation for mounting media files via the Jails section of the GUI

To complicate it further I have an rsync job that copies NAS_5TB nightly to a USB drive. (itself a second ZFS pool) When I dismounted the USB drive (using the 11.2 GUI) the other day (included the option to destroy the pool as I was rebuilding it) the main NAS_5TB mount point (the main pool) disappeared. The curious thing was that all the listings showed the pool was still there even though the mount point (in /mnt) was gone. Even if I had made a mistake an deleted the data from the main pool instead of the USB backup I would have expected the statistics in ZFS to show it as gone

I have since copied all the data back from the USB drive and essentially rebuilt the main pool)

At any rate I have rebuilt this now and all seems normal again.

Thanks for responding

 

[sretalla]

So it seems to me that all (or most) of your mountpoints have moved up one level... e.g.
NAS_5TB /NAS_5TB

Can you get to it like that?

If so, I have seen this happen to me once before when transitioning a drive between systems.

You can perhaps try zfs get mountpoint NAS_5TB as a start just to see if we're right.

Then you could use zfs set mountpoint=/mnt/NAS_5TB NAS_5TB to correct it (although you now need to do that for all the broken mountpoints.

I don't strictly recommend doing this as it would be completely unsupported by IX, but the other solution is backup your data, destroy the whole pool and rebuild it with the GUI, then restore. This would be the official fix... I'll leave it to you to think about how you want to go with it.

Although, now fully reading your last post, it's all moot.

 

[Boneshark]

Thanks for the reply - it does look that way but the real issue was that the ZFS view of the world did not match the physical. Even though ZFS shows those mounts the datasets are not there. No explanation as to why they disappeared after a reboot. At any rate I completely rebuilt it from a USB backup as you outlined. Took almost 30 hours to recover the 3.5TB