ZFS/gptid has disappeared

NASbox

Guru
Joined
May 8, 2012
Messages
644
Sorry if this question is a bit off topic, but it requires detailed knowledge of FreeNAS/FreeBSD/ZFS which only developers would know/care about.

Where does this gptid/735dxxxx-xxxx-xxxx-xxxx-xxxx17d2xxxx come from when zpool import is executed on a single disk pool?

More importantly, how can I restore this behaviour (or if necessary create a new gptid that displays).

Reason for question: Backup script automatically recovers the pool name and mounts the pool when a removable disk is installed at /dev/ada3 and the backup script is run.

Background:

Originally zpool import displayed the gptid of my pool when there was a pool available for import:
Code:
#>zpool import
   pool: BACKUP02
     id: 1487853121776782679
  state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:

        BACKUP02                                      ONLINE
          gptid/735dxxxx-xxxx-xxxx-xxxx-xxxx17d2xxxx  ONLINE


I forgot to export the pool over a reboot and somehow messed up this behaviour so:
Code:
#>zpool import
   pool: BACKUP02
     id: 1487853121776782679
  state: ONLINE
status: Some supported features are not enabled on the pool.
action: The pool can be imported using its name or numeric identifier, though
        some features will not be available without an explicit 'zpool upgrade'.
config:

        BACKUP02    ONLINE
          ada3p2    ONLINE
no longer shows the gptid.
(Pool created under FreeNAS 11.1-currently running 11.2-U7. I have tried rolling back to 11.1-U7 (the point I upgraded to 11.2-U7) and it doesn't make any difference.)

Another bit of information that might be helpful is: glabel status and ls -la /dev/gptid still show the gptid until the pool is imported (zpool import BACKUP02).

Once I import the pool, ada3p2 no longer shows up in glabel status and the gptid disappears from /dev/gptid. The pool is mounted in the file system at the expected location and functions correctly otherwise.

After I export BACKUP02, glabel status and ls -la /dev/gptid show the gptid again.

Any assistance/suggestions are much appreciated. I don't know if the original behaviour is due to FreeNAS or FreeBSD, or even what to Google to find a resolution on my own.

I would really prefer not to have destroy/recreate the pool from scratch as it contains about 7-8TB which is going to take a day or so.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
How are you exporting and importing the pool for your testing?
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
It might be helpful if you could post your script.
 

NASbox

Guru
Joined
May 8, 2012
Messages
644
It might be helpful if you could post your script.
The script would be TMI and only confuse the situation. The issue was I did something really stupid, I left BACKUP02 in the mount and imported under 11.1-U7 when I started my upgrade to 11.2-U7. (Disconnected it during the first reboot). That messed up something on the disk (BACKUP02).

In a nutshell, the script does the automount by doing a glabel status, searching for ada3p2, extracting the gptid, and then grepping against zpool import to extract the name BACKUP02 which is used in a zpool import BACKUP02 command. The only variable in the equation that is a known is ada3 because that is where the removable drive is connected.

How are you exporting and importing the pool for your testing?

For purposes of this question, I have simply been running these commands
Code:
glabel status
zpool import
ls -la /dev/gptid
zpool import BACKUP02
glabel status
ls -la /dev/gptid
zpool export BACKUP02
glabel status
ls -la /dev/gptid
in the shell.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
You should try importing the pool from within the GUI one time to see if it gives different results.
 

NASbox

Guru
Joined
May 8, 2012
Messages
644
You should try importing the pool from within the GUI one time to see if it gives different results.
I guess I should have mentioned I tried that already. It isn't an issue because the WebUI dropdown shows BACKUP02|1487853121776782679, but
because the decision to import is manual one FreeNAS doesn't need the gptid to function.

I did a normal import/export with the WebUI, but that didn't change anything-everything described.

I started playing with gpart show/list to compare two pools BACKUP01 (Works properly) and BACKUP02 (behaves differntly) and I can't see any significant difference.

Any other commands I should look at? What other info can I dump looking for differences?

Does FreeNAS do something special when it creates vdevs so that the gptid is available, or is this something that a nomal FreeBSD does too?

BACKUP01 (Working Device)
Code:
#>zpool import
   pool: BACKUP01
     id: 5978781716510586925
  state: ONLINE
status: Some supported features are not enabled on the pool.
action: The pool can be imported using its name or numeric identifier, though
        some features will not be available without an explicit 'zpool upgrade'.
config:

        BACKUP01                                      ONLINE
          gptid/8d86d52b-edf9-11e7-aadb-001517d2677a  ONLINE

#>gpart show -p ada3    
=>         40  15628053088    ada3  GPT  (7.3T)
           40      4194392          - free -  (2.0G)
      4194432  15623858688  ada3p2  freebsd-zfs  (7.3T)
  15628053120            8          - free -  (4.0K)

#>gpart show -l ada3    
=>         40  15628053088  ada3  GPT  (7.3T)
           40      4194392        - free -  (2.0G)
      4194432  15623858688     2  (null)  (7.3T)
  15628053120            8        - free -  (4.0K)

#>gpart show -r ada3
=>         40  15628053088  ada3  GPT  (7.3T)
           40      4194392        - free -  (2.0G)
      4194432  15623858688     2  516e7cba-6ecf-11d6-8ff8-00022d09712b  (7.3T)
  15628053120            8        - free -  (4.0K)

#>gpart list ada3
Geom name: ada3
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 15628053127
first: 40
entries: 152
scheme: GPT
Providers:
1. Name: ada3p2
   Mediasize: 7999415648256 (7.3T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   efimedia: HD(2,GPT,8d86d52b-edf9-11e7-aadb-001517d2677a,0x400080,0x3a3412a00)
   rawuuid: 8d86d52b-edf9-11e7-aadb-001517d2677a
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 7999415648256
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 15628053119
   start: 4194432
Consumers:
1. Name: ada3
   Mediasize: 8001563222016 (7.3T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0

#>glabel status
                                      Name  Status  Components
gptid/8d86d52b-edf9-11e7-aadb-001517d2677a     N/A  ada3p2

#>zpool get all BACKUP01
NAME      PROPERTY                       VALUE                          SOURCE
BACKUP01  size                           7.25T                          -
BACKUP01  capacity                       92%                            -
BACKUP01  altroot                        -                              default
BACKUP01  health                         ONLINE                         -
BACKUP01  guid                           5978781716510586925            default
BACKUP01  version                        -                              default
BACKUP01  bootfs                         -                              default
BACKUP01  delegation                     on                             default
BACKUP01  autoreplace                    off                            default
BACKUP01  cachefile                      -                              default
BACKUP01  failmode                       continue                       local
BACKUP01  listsnapshots                  off                            default
BACKUP01  autoexpand                     on                             local
BACKUP01  dedupditto                     0                              default
BACKUP01  dedupratio                     1.00x                          -
BACKUP01  free                           545G                           -
BACKUP01  allocated                      6.72T                          -
BACKUP01  readonly                       off                            -
BACKUP01  comment                        -                              default
BACKUP01  expandsize                     -                              -
BACKUP01  freeing                        0                              default
BACKUP01  fragmentation                  1%                             -
BACKUP01  leaked                         0                              default
BACKUP01  bootsize                       -                              default
BACKUP01  checkpoint                     -                              -
BACKUP01  feature@async_destroy          enabled                        local
BACKUP01  feature@empty_bpobj            active                         local
BACKUP01  feature@lz4_compress           active                         local
BACKUP01  feature@multi_vdev_crash_dump  enabled                        local
BACKUP01  feature@spacemap_histogram     active                         local
BACKUP01  feature@enabled_txg            active                         local
BACKUP01  feature@hole_birth             active                         local
BACKUP01  feature@extensible_dataset     enabled                        local
BACKUP01  feature@embedded_data          active                         local
BACKUP01  feature@bookmarks              enabled                        local
BACKUP01  feature@filesystem_limits      enabled                        local
BACKUP01  feature@large_blocks           enabled                        local
BACKUP01  feature@sha512                 enabled                        local
BACKUP01  feature@skein                  enabled                        local
BACKUP01  feature@device_removal         disabled                       local
BACKUP01  feature@obsolete_counts        disabled                       local
BACKUP01  feature@zpool_checkpoint       disabled                       local

#>zpool status BACKUP01
  pool: BACKUP01
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0 in 0 days 13:35:26 with 0 errors on Fri Jun 28 01:09:33 2019
config:

        NAME                                          STATE     READ WRITE CKSUM
        BACKUP01                                      ONLINE       0     0     0
          gptid/8d86d52b-edf9-11e7-aadb-001517d2677a  ONLINE       0     0     0

BACKUP02 (Problem Device)
Code:
#>zpool import
   pool: BACKUP02
     id: 8430597905561431551
  state: ONLINE
status: Some supported features are not enabled on the pool.
action: The pool can be imported using its name or numeric identifier, though
        some features will not be available without an explicit 'zpool upgrade'.
config:

        BACKUP02    ONLINE
          ada3p2    ONLINE

#>gpart show -p ada3
=>         40  19532873648    ada3  GPT  (9.1T)
           40      4194392          - free -  (2.0G)
      4194432  19528679248  ada3p2  freebsd-zfs  (9.1T)
  19532873680            8          - free -  (4.0K)

#>gpart show -l ada3
=>         40  19532873648  ada3  GPT  (9.1T)
           40      4194392        - free -  (2.0G)
      4194432  19528679248     2  (null)  (9.1T)
  19532873680            8        - free -  (4.0K)

#>gpart show -r ada3
=>         40  19532873648  ada3  GPT  (9.1T)
           40      4194392        - free -  (2.0G)
      4194432  19528679248     2  516e7cba-6ecf-11d6-8ff8-00022d09712b  (9.1T)
  19532873680            8        - free -  (4.0K)


#>gpart list ada3
Geom name: ada3
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 19532873687
first: 40
entries: 152
scheme: GPT
Providers:
1. Name: ada3p2
   Mediasize: 9998683774976 (9.1T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(2,GPT,735dc305-6477-11e8-8a07-001517d2677a,0x400080,0x48bffff50)
   rawuuid: 735dc305-6477-11e8-8a07-001517d2677a
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 9998683774976
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 19532873679
   start: 4194432
Consumers:
1. Name: ada3
   Mediasize: 10000831348736 (9.1T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2

#>glabel status
                                      Name  Status  Components
gptid/735dc305-6477-11e8-8a07-001517d2677a     N/A  ada3p2

#>zpool get all BACKUP02
NAME      PROPERTY                       VALUE                          SOURCE
BACKUP02  size                           9.06T                          -
BACKUP02  capacity                       74%                            -
BACKUP02  altroot                        -                              default
BACKUP02  health                         ONLINE                         -
BACKUP02  guid                           8430597905561431551            default
BACKUP02  version                        -                              default
BACKUP02  bootfs                         -                              default
BACKUP02  delegation                     on                             default
BACKUP02  autoreplace                    off                            default
BACKUP02  cachefile                      -                              default
BACKUP02  failmode                       continue                       local
BACKUP02  listsnapshots                  off                            default
BACKUP02  autoexpand                     on                             local
BACKUP02  dedupditto                     0                              default
BACKUP02  dedupratio                     1.00x                          -
BACKUP02  free                           2.34T                          -
BACKUP02  allocated                      6.72T                          -
BACKUP02  readonly                       off                            -
BACKUP02  comment                        -                              default
BACKUP02  expandsize                     -                              -
BACKUP02  freeing                        0                              default
BACKUP02  fragmentation                  0%                             -
BACKUP02  leaked                         0                              default
BACKUP02  bootsize                       -                              default
BACKUP02  checkpoint                     -                              -
BACKUP02  feature@async_destroy          enabled                        local
BACKUP02  feature@empty_bpobj            active                         local
BACKUP02  feature@lz4_compress           active                         local
BACKUP02  feature@multi_vdev_crash_dump  enabled                        local
BACKUP02  feature@spacemap_histogram     active                         local
BACKUP02  feature@enabled_txg            active                         local
BACKUP02  feature@hole_birth             active                         local
BACKUP02  feature@extensible_dataset     enabled                        local
BACKUP02  feature@embedded_data          active                         local
BACKUP02  feature@bookmarks              enabled                        local
BACKUP02  feature@filesystem_limits      enabled                        local
BACKUP02  feature@large_blocks           enabled                        local
BACKUP02  feature@sha512                 enabled                        local
BACKUP02  feature@skein                  enabled                        local
BACKUP02  feature@device_removal         disabled                       local
BACKUP02  feature@obsolete_counts        disabled                       local
BACKUP02  feature@zpool_checkpoint       disabled                       local

#>zpool status BACKUP02
  pool: BACKUP02
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0 in 0 days 13:12:18 with 0 errors on Sat Nov 30 06:09:30 2019
config:

        NAME        STATE     READ WRITE CKSUM
        BACKUP02    ONLINE       0     0     0
          ada3p2    ONLINE       0     0     0
 

NASbox

Guru
Joined
May 8, 2012
Messages
644
Are there any other commands that I can run to help determine what may be causing this issue?

Does FreeBSD have some sort of volume label (set by FreeNAS?) that may have been wiped/changed?
 

NASbox

Guru
Joined
May 8, 2012
Messages
644
Any suggestions? Is this "extended" label standard FreeBSD or is it something unique set by FreeNAS?
 

NASbox

Guru
Joined
May 8, 2012
Messages
644
@dlavigne can you suggest anyplace I might be able to get an answer to this one? I know very little about FreeBSD internals, but if I had to guess I would suspect that it is something a kin to a volume label like NTFS/FAT uses, but this is just a wild guess and I have no idea where to go next with this short of blowing away the volume and rebuilding it from scratch.
 

lostpacket

Cadet
Joined
Dec 27, 2019
Messages
3
/dev/gptid/<diskid> is populated by FreeBSD when gpt disk is created, ie
Code:
gpart create -s da0

/dev/gpt/<labelname> is populated by FreeBSD when gpt partition labels are created, ie
Code:
gpart add -a 1m -s 100g -l <labelname> da0

/dev/label/<label> is populated by freebsd when you label a disk with glabel (built-in)
Code:
glabel create -v labelname da0
 

HolyK

Ninja Turtle
Moderator
Joined
May 26, 2011
Messages
653
Not directly related as you're doing the exports/imports via CLI but could you check following tables if anything suspicious? Try it first w/o imported disk. Then import it via GUI and check again. Look for things like missing/duplicate records or wrong mapping to (a)da#, diskID, etc...

Code:
sqlite3 /data/freenas-v1.db
select * from storage_disk;
select * from storage_volume;
 

NASbox

Guru
Joined
May 8, 2012
Messages
644
Not directly related as you're doing the exports/imports via CLI but could you check following tables if anything suspicious? Try it first w/o imported disk. Then import it via GUI and check again. Look for things like missing/duplicate records or wrong mapping to (a)da#, diskID, etc...

Code:
sqlite3 /data/freenas-v1.db
select * from storage_disk;
select * from storage_volume;
@HolyK Thanks for the suggestion, I gave it a try, but the gptid isn't in that part of the database. I did however learn how to peek/extract info from the database which was interesting and may possible be useful.

/dev/gptid/<diskid> is populated by FreeBSD when gpt disk is created, ie
Code:
gpart create -s da0

/dev/gpt/<labelname> is populated by FreeBSD when gpt partition labels are created, ie
Code:
gpart add -a 1m -s 100g -l <labelname> da0

/dev/label/<label> is populated by freebsd when you label a disk with glabel (built-in)
Code:
glabel create -v labelname da0
@lostpacket I think you are on the right track here. Can you suggest how I can revert the disk shown below back to it's original (gptid/735dxxxx-xxxx-xxxx-xxxx-xxxx17d2xxxx) without destroying the data on it? I was looking at the man page for gpart and it looks like gpart modify might be the right command, but with my current level of FreeBSD knowledge I can't figure out how to accomplish what I need to accomplish (and I'm afraid of blowing up my data).
Code:
Original:
#>zpool import
   pool: BACKUP02
     id: 1487853121776782679
  state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:

        BACKUP02                                      ONLINE
          gptid/735dxxxx-xxxx-xxxx-xxxx-xxxx17d2xxxx  ONLINE

Changed:
#>zpool import
   pool: BACKUP02
     id: 1487853121776782679
  state: ONLINE
status: Some supported features are not enabled on the pool.
action: The pool can be imported using its name or numeric identifier, though
        some features will not be available without an explicit 'zpool upgrade'.
config:

        BACKUP02    ONLINE
          ada3p2    ONLINE


Am I correct in assuming that the part in bold is a partition label gptid/735dxxxx-xxxx-xxxx-xxxx-xxxx17d2xxxx ONLINE. If so, then it is just a matter of relabeling a partition. Can someone give me some guidance. I have read the man page, but don't have the background necessary to use/understand what I have read. Thanks.

I just found this https://lists.freebsd.org/pipermail/freebsd-fs/2011-October/012855.html post, the relevant part being:
GEOM shows all providers for a disk/partition that is not in use. Once you
acces a disk/partition via a particular provider, all others are hidden.
This is to prevent you from accessing a particular disk/paprtition via
multiple names.

For example, a GPT-partitioned disk could be accessed via the following GEOM
providers:
/dev/ada0p1
/dev/gptid/somelongstring
/dev/gpt/some-label
/dev/ufsid/someotherlongstring
/dev/ufs/some-other-label

As soon as you mount the filesystem via one of those paths, all the rest are
hidden.
Does that explain anything?
I have also noticed that in a normally set up FreeNAS, zpool status shows the boot drive as ada2p2, while zpool status of the pool drives drives shows the gptid. WHY? Does FreeNAS label the pools so that they are easier to identify/re-import? I'm just guessing that since the boot drive was originally a single flash drive there was no need to identify it in that manner and hence nobody bothered to label it.

IIUC then all I need to do is write a new label to ada3p2 with the gptid.

Would gpart modify -l 735dxxxx-xxxx-xxxx-xxxx-xxxx17d2xxxx -i2 ada3 rewrite the label?
 
Last edited:
Top