Upgraded 8.0.4 to 8.3 - now have a zpool with a duplicate name

[devsforev]

Hey everyone.

Last night I finally upgraded to FreeNAS 8.3-RELEASE from 8.0.4. Since then, my zpool "main" has not come online automatically on boot. I did some digging, and found something interesting...

Code:

[root@roku] /data/zfs# zpool import
   pool: main
     id: 5551600229828797753
  state: UNAVAIL
 status: The pool was last accessed by another system.
 action: The pool cannot be imported due to damaged devices or data.
   see: http://www.sun.com/msg/ZFS-8000-EY
 config:

	main                   UNAVAIL  insufficient replicas
	  4790801146886228091  UNAVAIL  cannot open
	  2805911385260330571  UNAVAIL  cannot open
	  651395760516798873   UNAVAIL  cannot open
	  1799081482536981512  UNAVAIL  cannot open

   pool: main
     id: 9440702576656588814
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

	main                                            ONLINE
	  raidz1-0                                      ONLINE
	    gptid/92469923-51ef-11e0-8ae7-001b219e8995  ONLINE
	    gptid/9257de68-51ef-11e0-8ae7-001b219e8995  ONLINE
	    gptid/92691460-51ef-11e0-8ae7-001b219e8995  ONLINE
	    gptid/927ab515-51ef-11e0-8ae7-001b219e8995  ONLINE

So it appears that zpool import thinks that I have two pools with the same name... I can assure you, I have one pool. Since they have the same name, I tried importing them based on ID. Lets try the first one that will surely fail:

Code:

[root@roku] /data/zfs# zpool import 5551600229828797753
cannot import 'main': pool may be in use from other system, it was last accessed by unset (hostid: 0xeaafaa42) on Wed Dec 31 19:00:00 1969
use '-f' to import anyway

That date... okay... Something's fishy here. Lets do the second one that looks healthy and online, presumably the one with my data:

Code:

[root@roku] /data/zfs# zpool import -R /mnt 9440702576656588814
[root@roku] /data/zfs# zpool status
  pool: main
 state: ONLINE
  scan: none requested
config:

	NAME                                            STATE     READ WRITE CKSUM
	main                                            ONLINE       0     0     0
	  raidz1-0                                      ONLINE       0     0     0
	    gptid/92469923-51ef-11e0-8ae7-001b219e8995  ONLINE       0     0     0
	    gptid/9257de68-51ef-11e0-8ae7-001b219e8995  ONLINE       0     0     0
	    gptid/92691460-51ef-11e0-8ae7-001b219e8995  ONLINE       0     0     0
	    gptid/927ab515-51ef-11e0-8ae7-001b219e8995  ONLINE       0     0     0

errors: No known data errors
[root@roku] /data/zfs# ls /mnt/
./       ../      .snap/   main/    md_size
[root@roku] /data/zfs# ls /mnt/main/
./                    .AppleDesktop/        .TemporaryItems/      Network Trash Folder/ home/
../                   .AppleDouble/         .apdisk*              Temporary Items/
.AppleDB/             .DS_Store             .freenas/             data/

There's my data. Phew!

The web GUI was complaining about a volume with a duplicate name when I tried to do the auto-import. Now have found this stale/rouge zpool, I would imagine the next thing to do would be to destroy it and then try the import again.

Thing is, zpool destroy does not accept an ID as a parameter, only a name... I have two zpools with the same name, so now what?

Anyway, somebody please tell me if I'm on the right path with this logic. And if I am, can somebody please advise me with the next step on how to nuke this stale zpool (id: 5551600229828797753) and proceed on to getting my system online again.

Thank you all for your time and have a great night,
-- devsforev

 

[cyberjock]

Yeah, whatever you do be very wary of anything involving "destroying" a zpool.

I would try to do a detach of your zpool. Make sure(and double check and triple check) that marking disks as new and deleting shares are NOT checked. Let it export, then rebooted, then reboot again, then try autoimport.

That might fix it.

 

[devsforev]

noobsauce80,

Thanks for the quick reply. I did notice that the Web GUI still listed the pool as attached. Detached, rebooted as suggested, did the auto-import and still failed claiming the same name. zpool import displays the same info as my OP.

Good call about being wary of destroying the first zpool. It seemed like a brute force approach... Anyway any other ideas are welcome. Thanks again!
-- devsforev

 

[paleoN]

So it appears that zpool import thinks that I have two pools with the same name... I can assure you, I have one pool. Since they have the same name, I tried importing them based on ID.

What's the history of these drives?

Thing is, zpool destroy does not accept an ID as a parameter, only a name... I have two zpools with the same name, so now what?

You can always rename the existing pool.

Output of:

Code:

camcontrol devlist

glabel status

gpart show

 

[devsforev]

paleoN,

Thanks for the reply. History of the drives... Nothing too exciting; picked them up 03/10/2011, installed them in this system right off the bat. Here's the exact model I bought off NewEgg. Been hummin' along ever since. Here are the specs you requested:

camcontrol devlist

Code:

[root@roku] ~# camcontrol devlist
<SAMSUNG HD204UI 1AQ10001>         at scbus6 target 0 lun 0 (pass0,ada0)
<SAMSUNG HD204UI 1AQ10001>         at scbus7 target 0 lun 0 (pass1,ada1)
<SAMSUNG HD204UI 1AQ10001>         at scbus8 target 0 lun 0 (pass2,ada2)
<SAMSUNG HD204UI 1AQ10001>         at scbus9 target 0 lun 0 (pass3,ada3)
<SanDisk U3 Cruzer Micro 8.02>     at scbus10 target 0 lun 0 (pass4,da0)

glabel status

Code:

[root@roku] ~# glabel status
                                      Name  Status  Components
gptid/92469923-51ef-11e0-8ae7-001b219e8995     N/A  ada0p2
gptid/9257de68-51ef-11e0-8ae7-001b219e8995     N/A  ada1p2
gptid/92691460-51ef-11e0-8ae7-001b219e8995     N/A  ada2p2
gptid/927ab515-51ef-11e0-8ae7-001b219e8995     N/A  ada3p2
                             ufs/FreeNASs3     N/A  da0s3
                             ufs/FreeNASs4     N/A  da0s4
                    ufsid/4f860b046ec32dc6     N/A  da0s1a
                            ufs/FreeNASs1a     N/A  da0s1a
                            ufs/FreeNASs2a     N/A  da0s2a

gpart show

Code:

[root@roku] ~# gpart show
=>        34  3907029101  ada0  GPT  (1.8T)
          34          94        - free -  (47k)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  3902834703     2  freebsd-zfs  (1.8T)

=>        34  3907029101  ada1  GPT  (1.8T)
          34          94        - free -  (47k)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  3902834703     2  freebsd-zfs  (1.8T)

=>        34  3907029101  ada2  GPT  (1.8T)
          34          94        - free -  (47k)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  3902834703     2  freebsd-zfs  (1.8T)

=>        34  3907029101  ada3  GPT  (1.8T)
          34          94        - free -  (47k)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  3902834703     2  freebsd-zfs  (1.8T)

=>     63  3907520  da0  MBR  (1.9G)
       63  1930257    1  freebsd  (942M)
  1930320       63       - free -  (31k)
  1930383  1930257    2  freebsd  [active]  (942M)
  3860640     3024    3  freebsd  (1.5M)
  3863664    41328    4  freebsd  (20M)
  3904992     2591       - free -  (1.3M)

=>      0  1930257  da0s1  BSD  (942M)
        0       16         - free -  (8.0k)
       16  1930241      1  !0  (942M)

=>      0  1930257  da0s2  BSD  (942M)
        0       16         - free -  (8.0k)
       16  1930241      1  !0  (942M)

Thanks again for your help. If I can provide any further information to help get this resolved please let me know!

-- devsforev

 

[paleoN]

History of the drives... Nothing too exciting; picked them up 03/10/2011, installed them in this system right off the bat.

Hmm, so none of them were ever part of a different pool named main?

Let's see if we can track down where this alternate pool is coming from:

Code:

zdb -l /dev/ada0

zdb -l /dev/ada0p1

zdb -l /dev/ada0p2

Repeat for ada1-ada3. The last line should be the only one to have labels, but the first may reveal some. You will need to check the four labels against each other, they should be self-consistent, and against the other drives. Feel free to only post differences as the output is long.

 

[devsforev]

paleoN,

The only time that these drives may have been part of another pool called "main" would be when I was initially configuring the pool to begin with. I may have created and destroyed it a few times because I didn't like my initial configuration of it for whatever reason. But that's the only thing I can think of -- they didn't come from a different system, nor did I ever intentionally try to run two main pools in the same system at once.

Thanks for your continued help. As requested, here is the dump from those commands. I put the output in a pastebin because as you said it is rather verbose :)

As always please let me know what I can do to help get this resolved. Thank you very much for your time!

-- devsforev

 

[paleoN]

The only time that these drives may have been part of another pool called "main" would be when I was initially configuring the pool to begin with. I may have created and destroyed it a few times because I didn't like my initial configuration of it for whatever reason

Perhaps you tried out the older .7 series, NAS4Free or created from the CLI. Doesn't really matter at this point. The problem is an older "main" pool was created using whole disks and there is still a phantom 3[sup]rd[/sup] label, or part of it, on the disks.

As always please let me know what I can do to help get this resolved. Thank you very much for your time!

The simplest & safest thing would be to rename the existing pool to something else, e.g.:

Code:

zpool import -R /mnt 9440702576656588814 primary

This doesn't remove the phantom label, but should allow the pool to be automatically imported.

Alternatively, you can use dd and zero out the beginning of the phantom label from each disk. Given you would be using dd you need to be exact or else you will damage information on the disk, pool/GPT. Given you have a raidz1 pool you can safely do this 1 disk at a time regardless. If you want to wipe the label, info to start:

Code:

diskinfo ada0
diskinfo ada1
diskinfo ada2
diskinfo ada3

 

[devsforev]

paleoN,

Renaming the pool worked without issue. I would imagine it would be easy to auto-import the pool now :)

However, now you got me thinking that wiping out this phantom label might be the way to go to avoid future issues like this. If you think it's prudent, then I'll be willing to follow along.

Code:

[root@roku] /mnt# diskinfo ada0
ada0	512	2000398934016	3907029168	4096	0	3876021	16	63
[root@roku] /mnt# diskinfo ada1
ada1	512	2000398934016	3907029168	4096	0	3876021	16	63
[root@roku] /mnt# diskinfo ada2
ada2	512	2000398934016	3907029168	4096	0	3876021	16	63
[root@roku] /mnt# diskinfo ada3
ada3	512	2000398934016	3907029168	4096	0	3876021	16	63

Thanks for all your help so far. I owe you a beer after this!

 

[paleoN]

If you think it's prudent, then I'll be willing to follow along.

You just need to be careful. I would scrub the pool & check the partition info after you wipe the label on the first disk. At most there might be some metadata in that area, which I doubt would be in the same place on the other disks, but you can simply scrub in-between for each label you wipe.


Alternatively, you can save the last meg or so off each drive and scrub once afterwards. If there was an issue than you can restore what you had originally.



With the pool exported:

Code:

dd if=/dev/zero of=/dev/ada0 bs=1024 seek=1953514088 count=1

This will wipe the first 1K of the 112K Name-Value pairs of the phantom label, though we don't know how much is actually left. It should be sufficient to 'erase' even a complete label and minimizes overwriting something else.

To double check the partition info:

Code:

gpart list /dev/ada0

Import pool and scrub pool. Then repeat for remaining drives.

 

[devsforev]

paleoN,

Got it. So what we are doing here is just zeroing out the label, and hopefully that should eliminate it from existence. I'm doing a scrub right now, before I run any of these commands just out of paranoia. I trust you, just dd makes me worried :P

FYI, I just ran gpart list /dev/ada0, which returned "gpart: No such geom: /dev/ada0." I take it you meant "gpart list ada0." Here is the output before running the dd command:

Code:

[root@roku] ~# gpart list ada0
Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e0
   rawuuid: 92447225-51ef-11e0-8ae7-001b219e8995
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: swap-ad12
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada0p2
   Mediasize: 1998251367936 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   rawuuid: 92469923-51ef-11e0-8ae7-001b219e8995
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: ad12
   length: 1998251367936
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 3907029134
   start: 4194432
Consumers:
1. Name: ada0
   Mediasize: 2000398934016 (1.8T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e4

If dd is successful this output will say the same, right? And finally, is there anything else I should do as a precaution before running dd?

Sorry for being verbose. I figure it's probably better to get these things out of the way first. You've been incredibly helpful so far, thanks!
-- devsforev

 

[paleoN]

I'm doing a scrub right now, before I run any of these commands just out of paranoia.

Good idea.

FYI, I just ran gpart list /dev/ada0, which returned "gpart: No such geom: /dev/ada0." I take it you meant "gpart list ada0."

Works either way for me. The secondary GPT table is at the very end of the drive and gpart list will complain if it's damaged. We will be operating at least few hundred K away from it, but you can double check everything still looks good.

If dd is successful this output will say the same, right? And finally, is there anything else I should do as a precaution before running dd

Yes, and just do it one disk at a time and scrub in-between disks. Your pool can survive one whole disk being wiped if your copy/paste skills leave something to be desired. ;)

 

[devsforev]

Awesome, thanks!

So I decided to bite the bullet and run the dd command as you suggested. I was immediately halted:

Code:

[root@roku] ~# zpool status
no pools available
[root@roku] ~# dd if=/dev/zero of=/dev/ada0 bs=1024 seek=1953514088 count=1
dd: /dev/ada0: Operation not permitted
[root@roku] ~# sudo dd if=/dev/zero of=/dev/ada0 bs=1024 seek=1953514088 count=1
Sorry, user root is not allowed to execute '/bin/dd if=/dev/zero of=/dev/ada0 bs=1024 seek=1953514088 count=1' as root on roku.local.

It appears my root user isn't root enough :P. I found another thread on here suggesting to fiddle with debug flags and try again... But you've gotten me this far and I just wanted to see if I should try anything else first.

Thanks as always.
-- devsforev

 

[paleoN]

I found another thread on here suggesting to fiddle with debug flags

It's no doubt the dumb swap partition still being opened:

Code:

swapctl -ls

swapctl -d /dev/ada0p1

Otherwise yes, enabling the debug flags will allow you to write directly to the drive.

 

[paleoN]

Wonderful, my post was just auto-moderated. The stupid spam filter needs some work.

 

[devsforev]

Wonderful, my post was just auto-moderated. The stupid spam filter needs some work.

same thing happened to me last night too :(

 

[tingo]

FWIW, the command

Code:

zpool history

can also tell you a bit on the history of a pool. I find it useful.

 

[paleoN]

Not sure if you noticed devsforev, but my post finally showed up. Did you wind up trying this or are you leaving it as is?

can also tell you a bit on the history of a pool. I find it useful.

I agree useful, but not relevant in this instance.