Booting stuck at "Import ZFS pools" on Scale 21.06

[masru]

I got the same issue, I was updating from BETA 2.

I am able to boot into the BETA.2 -- everything looks shiny there, all pools accessible.

 

[masru]

Ok, I booted into BETA.2 and checked the pool status. Only one of several pools shows a warning:

Code:

  pool: somename01
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 17:06:34 with 0 errors on Tue Sep 28 06:50:53 2021

Could it be that this warning prevents RC1 from importing the pools and finish booting?

 

[JJGadgets]

I am new to TrueNAS, and wanted to use it virtualized in Proxmox (passthrough HBA via IOMMU) instead of letting Proxmox handle my data drives (Proxmox & VMs are on a mirror NVMe). Installed SCALE, and this is my maybe 6th reboot when this occurred to me too. Just registered an account to throw my experience in.

I then booted the ISO since it also has ZFS, and forcefully imported the pool then exported it. However, this didn't fix it.

After some digging and brainstorming, I booted the TrueNAS install disk, added "init=/bin/bash" to the boot arguments in Grub, and went hunting for the systemd service responsible for the "Importing ZFS pools".

I then found that TrueNAS uses its own import pool service & command, since it was ix-zfs.service instead of the usual zfs-import.service or whatever that other desktop distros use (I use ZFS on Void which isn't systemd, so I have forgotten what the original service is called for systemd). ix-zfs.service uses the command midclt call -job --job-print description pool.import_on_boot which uses midclt, which is part of TrueNAS Scale's middlewared.

Manually calling the command imported my pools, since zpool list shows them as imported, and zpool status -v seems to show they are working just fine, but the midclt process didn't seem to finish and exit and instead got stuck on "Importing pool [pool name]" despite the pools being imported (seemingly) cleanly and already being in usage. Manually killing it and re-running the command with the pools already imported, the command seems to exit cleanly after a little while, showing "Pools import completed".

This leads me to believe the issue lies within middlewared's pool.import_on_boot code, which seems to be in Python, not properly recognizing that the pool import is finished when it is indeed actually finished.

Is there somewhere I can report this to, or is the forum the correct place?

 

[c77dk]

FYI - a featureflag has been removed, so if it's there, it's time to rebuild pool.

 

[JJGadgets]

FYI - a featureflag has been removed, so if it's there, it's time to rebuild pool.

Thanks for the notice, however my pool was originally created in Proxmox, so this feature is completely disabled for me.

 

[riptide_]

@c77dk Thanks for the information, it helped me pinpoint my issue.

I checked the feature flag and one of my pools has this flag. I created that pool (pool1) under the TrueNAS SCALE 21.08-BETA.2 release just a week prior to upgrading to RC1. The other pool (pool3) was created using the RC.1 release and show no trace of the flag.

Code:

root@truenas[~]# zpool get all boot-pool | grep xattr_compat
root@truenas[~]# zpool get all pool3 | grep xattr_compat
root@truenas[~]# zpool get all pool1 | grep xattr_compat
pool1  unsupported@com.ixsystems:xattr_compat  inactive                                local
root@truenas[~]# 

I ran into this issue on TrueNAS SCALE 22.02-RC.1 and it seems to only occur after I add an SMB share from one of my pools. I can make a lot of other changes to the configuration and reboot with a working system, even activate the SMB service. But as soon as I add an SMB share and reboot the system, it ends up hanging during startup with the "A start job is running for Import ZFS pools" message.

Luckily I still have the original data on another NAS, but it will take a couple of days to move those 18TB on the gigabit network I am stuck with.

 

[jkjkjk]

I am new to TrueNAS, and wanted to use it virtualized in Proxmox (passthrough HBA via IOMMU) instead of letting Proxmox handle my data drives (Proxmox & VMs are on a mirror NVMe). Installed SCALE, and this is my maybe 6th reboot when this occurred to me too. Just registered an account to throw my experience in.

I then booted the ISO since it also has ZFS, and forcefully imported the pool then exported it. However, this didn't fix it.

After some digging and brainstorming, I booted the TrueNAS install disk, added "init=/bin/bash" to the boot arguments in Grub, and went hunting for the systemd service responsible for the "Importing ZFS pools".

I then found that TrueNAS uses its own import pool service & command, since it was ix-zfs.service instead of the usual zfs-import.service or whatever that other desktop distros use (I use ZFS on Void which isn't systemd, so I have forgotten what the original service is called for systemd). ix-zfs.service uses the command midclt call -job --job-print description pool.import_on_boot which uses midclt, which is part of TrueNAS Scale's middlewared.

Manually calling the command imported my pools, since zpool list shows them as imported, and zpool status -v seems to show they are working just fine, but the midclt process didn't seem to finish and exit and instead got stuck on "Importing pool [pool name]" despite the pools being imported (seemingly) cleanly and already being in usage. Manually killing it and re-running the command with the pools already imported, the command seems to exit cleanly after a little while, showing "Pools import completed".

This leads me to believe the issue lies within middlewared's pool.import_on_boot code, which seems to be in Python, not properly recognizing that the pool import is finished when it is indeed actually finished.

Is there somewhere I can report this to, or is the forum the correct place?

Exactly the situation. Running TrueNAS in proxmox.
Have you solved the problem?

 

[riptide_]

Just wanted to mention that there are two issues resulting in the same error behaviour, halting at the "Import FZS pools" message. And I got bitten by both, but not at the same time.

I finally figured that out when I found a message in another thread.

There's an update here. https://www.truenas.com/community/threads/scale-22-02-rc-1-feature-flags-and-import.96461/

In short there are two separate issues:
1. xattr_compat feature flag - if this is "active" the pool needs to be backed-up and recreated (described in release notes)
2. there is a pool import issues with encrypted datasets and SMB -- that will be fixed in an RC1.1 planned for this week. Described in this bug report: https://jira.ixsystems.com/browse/NAS-113052

 

[Robert Thomspon]

Hey gents, I just ran into the same issue updating from 22.02.00.01 to 22.02.1, Mine happened to hang because of the number fo snapshots (31k vs the 10k recommended...) Mine booted on its own eventually, but that seems to have been the issue on my end at least.

Just in case that helps anyone.