SOLVED Corrupt or invalid GPT detected | GPT rejected -- may not be recoverable

[Aero-Graphics]

I had a little SNAFU with my hardware and it appears that the partition table on my drives has been corrupted to the point that TrueNAS will no longer recognize them. I was able to boot a Linux live environment and it will recognize the partitions and mount the ZFS pool. So, I was able to get a copy of the data off just in case. I am now attempting to rebuild the partition table on TrueNAS, but I'm running into a little trouble. Gdisk on Linux shows this for a partition table:

Code:

Number    Start (sector)    End (sector)    Size    Code
1                  128         4194431     2.0 GiB  A502
2              4194432     23437770711     10.9 TiB A504

So, I thought that the following commands in TrueNAS might do the trick:

Code:

gpart create -s gpt <device>
gpart add -b 128 -s 4194303 -t freebsd-swap <device>
gpart add -b 4194432 -s 23433576279 -t freebsd-zfs <device>

But, I don't even get past the first command:

Code:

root@truenas[~]# gpart create -s gpt da0
gpart: geom 'da0': Operation not permitted

'geom disk list' shows all the physical drives, but because of the corrupt partition tables I appear unable to perform any operations on them. I'm much more familiar with Linux, what do I need to do to be able to write a new partition table in TrueNAS/FreeBSD?

Thanks,

Rob

 

[sretalla]

gpart create -s gpt da0

gpart create -s gpt /dev/da0

You might find you need to first destroy the partition table though...

 

[Aero-Graphics]

FYI:

Code:

root@truenas[~]# geom disk list da0
Geom name: da0
Providers:
1. Name: da0
   Mediasize: 12000138625024 (11T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   descr: SEAGATE ST12000NM0027
   lunid: 5000c500a67fe423
   ident: ZJV1REWY0000C851C5WN
   rotationrate: 7200
   fwsectors: 63
   fwheads: 255

root@truenas[~]# gpart create -s gpt /dev/da0
gpart: geom 'da0': Operation not permitted
root@truenas[~]# gpart destroy da0
gpart: arg0 'da0': Invalid argument
root@truenas[~]# gpart destroy /dev/da0
gpart: arg0 'da0': Invalid argument

Because it refuses to recognize the partition the system appears not to allow operations on the disk...

 

[unseen]

If there's already a swap partition on the drives, are you sure that swap isn't already in use? That would stop your attempts to partition the disks.
If so:

swapoff -a
gpart destroy -F da0

Should make it possible to carry on...

 

[Aero-Graphics]

There is a swap partition on the drives, but it is not being used by the system for swap. Since TrueNAS refuses to recognize there is anything on the drive, it could not be used as swap. The swap partition was created by TrueNAS when the Zpool was created, so I'm fairly certain it has something to do with ZFS and nothing to do with system swap. But, it didn't make any difference:

Code:

root@truenas[~]# swapoff -a
root@truenas[~]# gpart destroy -F da0
gpart: arg0 'da0': Invalid argument

 

[unseen]

Boot Linux from USB, run gparted, destroy partition table, create new GPT partition table.

Reboot into FreeBSD and go from there via the TrueNAS GUI.

I've had troubles like this before and always found the least painful method was using Linux to fix things. You may even need to overwrite the end of the disk with dd so that FreeBSD can't find any ZFS signatures on the drives. See also: https://www.truenas.com/community/threads/operation-not-permitted-when-trying-to-wipe-disks.69240/

 

[Aero-Graphics]

I figured there had to be a way to do this in FreeBSD and because of my unfamiliarity, I was just missing it. I can definitely remove the partition table in Linux, but hopefully that will allow it to be recreated in FreeBSD. I'm trying to rebuild the partition table without wiping the ZFS pool. While I do have a backup copy just in case, it still takes a long time to write ~150TB of data...

 

[unseen]

There's a success story in one of the "Similar threads": https://www.truenas.com/community/threads/gpt-rejected-may-not-be-recoverable.43813/

Maybe that will help? I've not had to solve this before myself, so I'll let someone more experienced comment.

 

[Aero-Graphics]

Thanks for your input. Unfortunately those links will only help if I abandon my hope to only recreate the partition table and not destroy any of the underlying ZFS data.

 

[Aero-Graphics]

It looks like dd may be the only option. I booted into GParted-live, backed up the existing partition table of sdc (da0) with sgdisk, created a new empty GPT table on the disk, and then rebooted into TrueNAS:

Code:

Jul 14 09:53:55 truenas GEOM_MULTIPATH: da0 added to disk1
Jul 14 09:53:55 truenas GEOM: multipath/disk1: corrupt or invalid GPT detected.
Jul 14 09:53:55 truenas GEOM: multipath/disk1: GPT rejected -- may not be recoverable.

Code:

root@truenas[~]# gpart create -s gpt da0
gpart: geom 'da0': Operation not permitted

So it would appear that FreeBSD will not recognize an empty GPT table created by Linux and nothing short of a clean slate will allow it to be written to by gpart...

 

[Aero-Graphics]

A quick follow-up in case anyone runs across this looking for answers to a similar issue.

I found the tidbit I needed to progress forward here:
https://www.truenas.com/community/t...ath-and-reenables-itself-after-reboot.101484/

Once issuing a 'sysctl kern.geom.debugflags=0x10' I was then able to work with the disks without unnecessarily deleting or recreating any partition tables. There was one that was corrupted to the point that this was necessary, and other that the sizes being off by one, my initial proposed partition recreation seemed to do the trick.

Another help was setting up a FreeBSD 13.1 VM, installing gdisk on it and then copying the installed files to the TrueNAS boxes. I checked that all the prerequisites were installed on TrueNAS (albeit with slightly older versions). Note that gdisk requires the use of the full path of the devices even though gpart and other built-in system utilities don't.

Multipathing was another complication in my case, because you apparently need to work with the multipath devices and not the underlying individual disks. I started by wiping out all the multipaths, because I knew the underlying disks were okay (because I had mounted the pool and made a backup copy from Linux). Then I was able to do an 'sgdisk -e' and/or gpart recover and then gpart status showed that the system saw the partitions on the disks. I then rebuilt the multipaths and everything seems to be happy.

Rob