SOLVED Pool unavailable after power off/on USB HDD (purpose offline backup)

[Alfnie]

Hi All,

This is my first post on the forum after I have been reading a lot (!) of very helpfull posts in building and configuring a TrueNAS system.
I moved from Netgear, to Synology, and now to TrueNAS. I am very happy with this system, often it feels too easy and intuïtive that it feels too good to be true.

Thanks in advance for the help.

In short, my problem is that I had setup my (offline) backup and it worked perfectly for me.
USB-HDD 16TB, Import Disk, Create Pool#2 with 2 datasets: 1 for Replicating Pool#1, and 1 for SMB-sharing to backup my Windows 10 System Image.
Then I powered down the USB-HDD to change the power-outlet. Then after Power On the USB-HDD, the problem arises "Pool UNAVAIL".
However, the SMB drive with the windows 10 system image still works, I can access the share. But I can't get the Pool#2 and the replicated Pool#1 to work, I don't see what I can do.

I have read a lot in the forum

• Don't use USB drives (link): I was doing this previously, and it kinda worked for my. My Netgear/Synology NAS did recognice the USB-HDD as a seperate device (NTFS/EXT4) formatted.
• How-to USB Drive as offline backup (link): I did it like this 'create a single-disk pool on the external USB drive'.
• Same usecase 'Manually (on-demand) replicating entire main pool to external USB drive' (link): learning something about offline/online drives
• How to Restore Disconnected Pool (link): I tried import existing Pool, can't see the Pool#2 to import
• Other source (link lost): I learned that one should put Pool offline before disconnecting USB-drive. If true, this would not fit my usecase

I hope one can help me out:

1. Get the Pool back online
2. Advise firstly how I could get this to work through the TrueNAS (comparable to Netgear/Synology)
3. Secondly, advice, if this really should not be used like that, what I best could do next (I want a 'scheduled' offline backup, in case of emergency)

Version: TrueNAS-13.0-U5.3 (Core)
Mainboard: Gigabyte B360M D3H
CPU: Intel(R) Pentium(R) Gold G5400T
RAM: 8GB
Pool1: RAIDZ2, 4x NVMe 1TB
Pool2: Striped, 1x USB-HDD 16TB

 

[Arwen]

It appears you did not export your USB pool #2 before you powered it down. This can be a problem, as the USB to SATA controller chip might not perform the writes in order. ZFS wants writes to disks in order that it gives the write request. This allows the copy on write to work perfectly.

Did you export your USB pool #2 before powering down the USB drive?

Next, if you can supply the output of the following commands in CODE tags, it would be helpful;
zpool import zpool list zpool status

 

[Alfnie]

It appears you did not export your USB pool #2 before you powered it down. This can be a problem, as the USB to SATA controller chip might not perform the writes in order. ZFS wants writes to disks in order that it gives the write request. This allows the copy on write to work perfectly.

Did you export your USB pool #2 before powering down the USB drive?

Next, if you can supply the output of the following commands in CODE tags, it would be helpful;
zpool import zpool list zpool status

Thanks for your reply!

I did not export my USB pool #2 before powering down... oops

Code:

root@truenas[~]# zpool import
no pools available to import

root@truenas[~]# zpool list
NAME         SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
NAS_BACKUP  14.5T   938G  13.6T        -         -     0%     6%  1.00x   UNAVAIL  /mnt
NAS_STORE   3.62T  1.30T  2.32T        -         -     0%    35%  1.00x    ONLINE  /mnt
boot-pool    111G  1.30G   110G        -         -     0%     1%  1.00x    ONLINE  -

Code:

root@truenas[~]# zpool status
  pool: NAS_BACKUP
 state: UNAVAIL
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-JQ
config:

        NAME                                          STATE     READ WRITE CKSUM
        NAS_BACKUP                                    UNAVAIL      0     0     0  insufficient replicas
          gptid/04aa97bf-6de6-11ee-b4b3-e0d55ee1cca2  REMOVED      0     0     0
errors: List of errors unavailable: pool I/O is currently suspended

errors: 8 data errors, use '-v' for a list

  pool: NAS_STORE
 state: ONLINE
config:

        NAME                                            STATE     READ WRITE CKSUM
        NAS_STORE                                       ONLINE       0     0 0
          raidz2-0                                      ONLINE       0     0 0
            gptid/b96b5d2a-5d70-11ee-82ea-e0d55ee1cca2  ONLINE       0     0 0
            gptid/b96cba11-5d70-11ee-82ea-e0d55ee1cca2  ONLINE       0     0 0
            gptid/b974f599-5d70-11ee-82ea-e0d55ee1cca2  ONLINE       0     0 0
            gptid/b9782ac8-5d70-11ee-82ea-e0d55ee1cca2  ONLINE       0     0 0

errors: No known data errors

  pool: boot-pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:05 with 0 errors on Fri Oct 20 03:45:05 2023
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          ada0p2    ONLINE       0     0     0

errors: No known data errors

 

[Alfnie]

I restarted the system, now the GUI at Pools read "Export/Disconnect"

I redo the commands above, see output below.

Now the zpool import command does show the NAS_BACKUP pool (this is pool #2 which is lost)

Code:

root@truenas[~]# zpool import
   pool: NAS_BACKUP
     id: 13537780390545233988
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

        NAS_BACKUP                                    ONLINE
          gptid/04aa97bf-6de6-11ee-b4b3-e0d55ee1cca2  ONLINE

Code:

root@truenas[~]# zpool list
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
NAS_STORE  3.62T  1.30T  2.32T        -         -     0%    35%  1.00x    ONLINE  /mnt
boot-pool   111G  1.30G   110G        -         -     0%     1%  1.00x    ONLINE  -
root@truenas[~]# zpool status
  pool: NAS_STORE
 state: ONLINE
config:

        NAME                                            STATE     READ WRITE CKSUM
        NAS_STORE                                       ONLINE       0     0 0
          raidz2-0                                      ONLINE       0     0 0
            gptid/b96b5d2a-5d70-11ee-82ea-e0d55ee1cca2  ONLINE       0     0 0
            gptid/b96cba11-5d70-11ee-82ea-e0d55ee1cca2  ONLINE       0     0 0
            gptid/b974f599-5d70-11ee-82ea-e0d55ee1cca2  ONLINE       0     0 0
            gptid/b9782ac8-5d70-11ee-82ea-e0d55ee1cca2  ONLINE       0     0 0

errors: No known data errors

  pool: boot-pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:05 with 0 errors on Fri Oct 20 03:45:05 2023
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          ada0p2    ONLINE       0     0     0

errors: No known data errors

Code:

root@truenas[~]# zpool status
  pool: NAS_STORE
 state: ONLINE
config:

        NAME                                            STATE     READ WRITE CKSUM
        NAS_STORE                                       ONLINE       0     0 0
          raidz2-0                                      ONLINE       0     0 0
            gptid/b96b5d2a-5d70-11ee-82ea-e0d55ee1cca2  ONLINE       0     0 0
            gptid/b96cba11-5d70-11ee-82ea-e0d55ee1cca2  ONLINE       0     0 0
            gptid/b974f599-5d70-11ee-82ea-e0d55ee1cca2  ONLINE       0     0 0
            gptid/b9782ac8-5d70-11ee-82ea-e0d55ee1cca2  ONLINE       0     0 0

errors: No known data errors

  pool: boot-pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:05 with 0 errors on Fri Oct 20 03:45:05 2023
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          ada0p2    ONLINE       0     0     0

errors: No known data errors

 

[Arwen]

You can try and import the missing pool with;
zpool import NAS_BACKUP

If it won't import, don't panic, their are recovery options beyond a simple import.


In the future, think of a ZFS export as un-mounting the file systems, readying the ZFS pool for movement elsewhere. USB and or hardware RAID controllers complicate the situation a bit, because you really want to properly export a ZFS pool before removal. ZFS was designed for direct access to the disks. USB and hardware RAID controllers don't always give that direct access.

 

[Alfnie]

You can try and import the missing pool with;
zpool import NAS_BACKUP

If it won't import, don't panic, their are recovery options beyond a simple import.


In the future, think of a ZFS export as un-mounting the file systems, readying the ZFS pool for movement elsewhere. USB and or hardware RAID controllers complicate the situation a bit, because you really want to properly export a ZFS pool before removal. ZFS was designed for direct access to the disks. USB and hardware RAID controllers don't always give that direct access.

This worked!
I already tried this yesterday, I assumed that would be the next step, but I had an error (not sure what is was).
Today, I look, and pool #2 is back.

I understand your writing about un-mounting file systems, and that USB is not the proper way to do this wit a BACKUP pool.

Is there an other option that you could suggest given I would like to use USB-HDD, but then not needed to unmount it everytime, being unsure if the Backup is working later on?

Can I use the USB-HDD as NTFS or something like that from Truenas? Without making it a Pool?

 

[NugentS]

A USB is fine. Indeed @Arwen has posted many times extolling this mechanism. I think there are even some scripts somewhere to help.
Just make sure you export the pool before removing the disks. After that you should be able to import just fine.

Note that this applies to single disk pools for backup purposes.

TrueNAS only writes to ZFS - nothing else. It can read from (import) some other filing systems - but writes to ZFS only

 

[Arwen]

This worked!
I already tried this yesterday, I assumed that would be the next step, but I had an error (not sure what is was).
Today, I look, and pool #2 is back.

I understand your writing about un-mounting file systems, and that USB is not the proper way to do this wit a BACKUP pool.

Is there an other option that you could suggest given I would like to use USB-HDD, but then not needed to unmount it everytime, being unsure if the Backup is working later on?

Can I use the USB-HDD as NTFS or something like that from Truenas? Without making it a Pool?

Glad that helped. In general, you want to import or export a pool through the GUI. Somethings done outside the GUI means that the middleware, (aka TrueNAS software, but not the OS), does not know about the change.


Yes, you can use a USB attached drive for backup purposes. That has working fine for me, for years.

The problems can start when leaving a USB attached drive for long periods of time. Then, problems can start cropping up like loose USB cables or connectors, excessive heat, etc...

 

[Alfnie]

Glad that helped. In general, you want to import or export a pool through the GUI. Somethings done outside the GUI means that the middleware, (aka TrueNAS software, but not the OS), does not know about the change.


Yes, you can use a USB attached drive for backup purposes. That has working fine for me, for years.

The problems can start when leaving a USB attached drive for long periods of time. Then, problems can start cropping up like loose USB cables or connectors, excessive heat, etc...

Thanks, I will install 2x SATA drives and keep em in the system as ZFS mirror for backup.
I will use the USB HDD over the network connected to my PC for backups.