One pool lost VDEV

[simdim]

After upgrade to 23.10.0 from latest Bluefin one of the pools lost VDEV - I can reimport that pool but concerned that data will be lost

Do we know why pool had become exported and is it safe to import it back?

 

[winnielinnie]

What pools are currently imported?

Code:

zpool list

What pools can be imported?

Code:

zpool import

 

[simdim]

Update: was not able to import trough GUI, however CLI zpool import worked.
Now I am missing all Apps - config.json is missing

Application(s) have failed to start: [EFAULT] Missing "pool/ix-applications/releases, pool/ix-applications/k3s" dataset(s) required for starting kubernetes.

One step at a time :)

 

[simdim]

I think the problem is:
pool used to mount to /mnt/pool
now it mounts to /pool

 

[winnielinnie]

That's why you you should import only with the GUI, except for emergencies.

Code:

zpool get altroot pool

What does the above command reveal?


You can try to export the pool, and then re-import it with the GUI.

 

[simdim]

I did:
]zfs set mountpoint=/mnt/pool pool
zfs get mountpoint pool
NAME PROPERTY VALUE SOURCE
pool mountpoint /mnt/pool local


zpool get altroot pool
NAME PROPERTY VALUE SOURCE
pool altroot - default[/CODE]

Apps show as "running" but they are not - should I export pool and import it again from GUI (the reason I did from CLI - it was not working from GUI)?

 

[winnielinnie]

The "altroot" property must be set to /mnt

Code:

zpool set altroot=/mnt pool

Then export it (from the GUI) and re-import it (from the GUI).

I did:
]zfs set mountpoint=/mnt/pool pool

Don't mess with these. Let TrueNAS handle it. However, TrueNAS expects the "altroot" (pool property) to be exactly /mnt

 

[simdim]

Getting an error there:

zpool set altroot=/mnt pool
cannot set property for 'pool': property 'altroot' can only be set during pool creation or import

 

[winnielinnie]

My bad.

Export it with the GUI, then re-import it as such (which I thought the GUI was supposed to by by default):

Code:

zpool import -R /mnt pool

Actually, try to import it normally with the GUI, without the above command. It should automatically set the altroot.

 

[simdim]

Well the saga continues:
I can not export it via GUI

It claims processes are using pool

And the task fails

Also dashboard is not really trustworthy it reports that pool is down and network is down

However storage view shows green status:

From cli:

Code:

zpool list
NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
BigDisk       9.06T  6.17T  2.89T        -         -     8%    68%  1.00x    ONLINE  /mnt
USBBAckup     10.9T  6.43T  4.47T        -         -     1%    58%  1.00x    ONLINE  /mnt
freenas-boot   112G  40.2G  71.3G        -         -      -    36%  1.00x    ONLINE  -
pool          3.62T   300G  3.33T        -         -    10%     8%  1.00x    ONLINE  -

I had stopped all apps and unset pool this did not help

After reboot since altroot is not set we are back to mounting at /

 

[winnielinnie]

Is it possible reboot into an earlier "Boot Environment"?

 

[simdim]

Unfortunately I had "upgraded" pool thinking that this may be an issue - so according to documentation this is one way deal no going back ...

Should I try to export from cli?

 

[winnielinnie]

Try to move the System Dataset back to the boot-pool, and then try to export it from the GUI.

I'm assuming you stop all other services? (SMB, NFS, etc)

 

[simdim]

It is already at boot pool

 

[winnielinnie]

I don't know the effects of forcing an export on SCALE, in regards to "Apps". (Excluding the System Dataset, since it's not on the storage pool.)

@morganL or @Kris Moore

Is it safe to forcefully (with the command-line) export a storage pool which holds "Apps"?

They need to be able to export the pool, so they can import it properly (with the GUI), so that it may use "/mnt" as the altroot.

 

[simdim]

Attempted to
zpool export pool
cannot export 'pool': pool is busy

lsof |grep pool

not returning anything


zpool events -vHf pool
Oct 26 2023 17:08:50.602541832 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "pool"
pool_guid = 0x363894e2a47e3568
pool_state = 0x0
pool_context = 0x0
history_hostname = "freenas"
history_internal_str = "pool version 5000; software version zfs-2.1.99-2204-gfa2d8bd27; uts freenas 6.1.55-production+truenas #2 SMP PREEMPT_DYNAMIC Mon Oct 23 13:41:33 UTC 2023 x86_64"
history_internal_name = "open"
history_txg = 0x8564f3
history_time = 0x653ad562
time = 0x653ad562 0x23ea0f08
eid = 0x6

Oct 26 2023 17:08:50.942541836 sysevent.fs.zfs.config_sync
version = 0x0
class = "sysevent.fs.zfs.config_sync"
pool = "pool"
pool_guid = 0x363894e2a47e3568
pool_state = 0x0
pool_context = 0x0
time = 0x653ad562 0x382e0c0c
eid = 0x7

Oct 26 2023 17:08:50.942541836 sysevent.fs.zfs.pool_import
version = 0x0
class = "sysevent.fs.zfs.pool_import"
pool = "pool"
pool_guid = 0x363894e2a47e3568
pool_state = 0x0
pool_context = 0x0
time = 0x653ad562 0x382e0c0c
eid = 0x8

Oct 26 2023 17:08:51.018541837 sysevent.fs.zfs.history_event
version = 0x0
class = "sysevent.fs.zfs.history_event"
pool = "pool"
pool_guid = 0x363894e2a47e3568
pool_state = 0x0
pool_context = 0x0
history_hostname = "freenas"
history_internal_str = "pool version 5000; software version zfs-2.1.99-2204-gfa2d8bd27; uts freenas 6.1.55-production+truenas #2 SMP PREEMPT_DYNAMIC Mon Oct 23 13:41:33 UTC 2023 x86_64"
history_internal_name = "import"
history_txg = 0x8564f5
history_time = 0x653ad562
time = 0x653ad563 0x11aed0d
eid = 0x9

Oct 26 2023 17:08:51.554541845 sysevent.fs.zfs.config_sync
version = 0x0
class = "sysevent.fs.zfs.config_sync"
pool = "pool"
pool_guid = 0x363894e2a47e3568
pool_state = 0x0
pool_context = 0x0
time = 0x653ad563 0x210da315
eid = 0xa

 

[simdim]

Still could not figure out what was claiming to be using pool that I could not export and went for desperate measures:
1. Rebooted back to bluefin
2. In Bluefin the problematic pool could not be found at all
3. Rebooted back to Cobia
4. All services and apps came up - nothing is complaining ....

ALTROOT on problem pool is still not set

Code:

zpool list
NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
BigDisk       9.06T  6.17T  2.89T        -         -     8%    68%  1.00x    ONLINE  /mnt
USBBAckup     10.9T  6.43T  4.48T        -         -     1%    58%  1.00x    ONLINE  /mnt
freenas-boot   112G  40.2G  71.3G        -         -      -    36%  1.00x    ONLINE  -
pool          3.62T   301G  3.33T        -         -    10%     8%  1.00x    ONLINE  -

However at this point I am not going to make any changes and will wait for 23.10.1
I guess stepping on the rake of .0 code is genetic and can not be helped :)

 

[sfatula]

I guess stepping on the rake of .0 code is genetic and can not be helped :)

Lol, see a lot of regret here from posters in different threads! I wait until .2, always, every time, every application/os/appliance/whatever. Doing nothing is surprisingly easy!

I would wager if you submit a trouble ticket they can get you going correctly. It might help someone else as well who may be in the same boat.

Alternatively if brave, you could try setting the "canmount" property and rebooting. In theory, it would not mount the pool allowing you to adjust, set canmount back to on, and then reboot.

 

[simdim]

Lol, see a lot of regret here from posters in different threads! I wait until .2, always, every time, every application/os/appliance/whatever. Doing nothing is surprisingly easy!

I would wager if you submit a trouble ticket they can get you going correctly. It might help someone else as well who may be in the same boat.

Alternatively if brave, you could try setting the "canmount" property and rebooting. In theory, it would not mount the pool allowing you to adjust, set canmount back to on, and then reboot.

canmount is a good suggestion, however half of my house is not talking to the other half (I am running HomeBridge, AgentDVR, Scrypted, Calibre on this box) and was given an ultimatum, by people that mater, to "Get this shit going as is :)
It is all up and running now and survives reboot so I will deal with ALTROOT issue next time around on .3 version perhaps.