ZFS - Recover overwritten pool

yourmate

Contributor
Joined
Apr 4, 2016
Messages
105
I know I will be smashed and trashed to bits asking this question but I am a brave idiot ;)

The storage setup consisted of 6 4TB WD Reds (storing data - called "ocean") and two mirrored 240GB SSD (storing jails - called "whirlpool"). A few months ago I bought 3 8TB WD Reds and changed 3 4TB disks over, one by one re-silvering them. All went fine.
The new 8TB WD Reds were running much hotter (13-18 C hotter) than the 4TBs so I switched the system off and investigated. WD said they are good up to 65C but I just wanted to make sure the case has adequate cooling so I made some mods.

Anyway, new FreeNAS came around (11.3 U1) and my Kodi box had write authentication issues I thought why not reinstall from scratch.

After some initial settings, I was ready to import my ZFS pools. Normally, I start by importing ocean first and then whirlpool. But now I thought I will scrap my old jails and start with a brand new pool for the SSD so I started with NEW POOL, named it whirlpool and choose the two SSD. But then I clicked on "recommended setup" to see what I will be offered and got distracted by a phone call. Then click on create and voila whirlpool has been created (literally :( ).
I then moved to import my data and I was offered to import whirlpool...
Then it struck me, that the "recommended setup" choose the 3 new 8TB WD Reds as whirlpool so literally half of my data pool was lost.

The ocean is setup 4+2 so losing two disks is not the end of the world but now I lost three. Nothing was written to the new pool, I simply shut down the system and crawled into the darkest corner to die. As you might already figured, there is no backup of the NAS.

This was two weeks ago, the box just stands there switched off and I slowly coming to terms with myself and the losses.
Guys, is there any way to go back from here?

Edit: I still have 2 of the 4TB WD Reds I replaced if that would help (I reckon it wouldn't, but hey)
Edit2: I would like to ask especially @Ericloewe and @cyberjock to be easy on me :)
 
Last edited:

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,703
Could you share the output of zpool status -v please? we need to see where it's at.
 

yourmate

Contributor
Joined
Apr 4, 2016
Messages
105
Could you share the output of zpool status -v please? we need to see where it's at.

Thanks for the help!

Code:
root@freenas[~]# zpool status -v
  pool: freenas-boot
state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            da0p2   ONLINE       0     0     0
            da1p2   ONLINE       0     0     0

errors: No known data errors

  pool: whirlpool
 state: ONLINE
  scan: none requested
config:

        NAME                                            STATE     READ WRITE CKSUM
        whirlpool                                       ONLINE       0     0 0
          raidz1-0                                      ONLINE       0     0 0
            gptid/c6fbf984-5cd3-11ea-bf13-0025907aa820  ONLINE       0     0 0
            gptid/c6ff85b1-5cd3-11ea-bf13-0025907aa820  ONLINE       0     0 0
            gptid/c701d157-5cd3-11ea-bf13-0025907aa820  ONLINE       0     0 0

errors: No known data errors
root@freenas[~]#


Would this help in any way? This only shows the "newly created" whirlpool pool on the three 8TB WDs
Currently, all 6 WDs are connected (previously "ocean", but the two SSDs are disconnected (original whirlpool)

Edit2: upodated - now full output shown
 
Last edited:

yourmate

Contributor
Joined
Apr 4, 2016
Messages
105
Here are the connected disks:
disks.PNG
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,703
How about anything shown by zpool import ?
 

yourmate

Contributor
Joined
Apr 4, 2016
Messages
105
How about anything shown by zpool import ?

Code:
root@freenas[~]# zpool import
   pool: ocean
     id: 5961047394514547352
  state: UNAVAIL
 status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: http://illumos.org/msg/ZFS-8000-3C
 config:

        ocean                                           UNAVAIL  insufficient replicas
          raidz2-0                                      UNAVAIL  insufficient replicas
            gptid/374ac9e5-7a97-11e6-b2e9-0025907aa820  ONLINE
            gptid/387baa11-7a97-11e6-b2e9-0025907aa820  ONLINE
            gptid/39af8191-7a97-11e6-b2e9-0025907aa820  ONLINE
            3578683748334357503                         UNAVAIL  cannot open
            17017105264701968685                        UNAVAIL  cannot open
            15071426524228456616                        UNAVAIL  cannot open
root@freenas[~]#
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,703
OK, so the drives still there know they are part of a pool with 3 drives missing... you "may" get it to mount by running zpool import -F ocean .

You will certainly be missing some (most likely all) of the data even if it will mount.

I'm afraid that in the situation you're in, that's the best you can hope for.
 

yourmate

Contributor
Joined
Apr 4, 2016
Messages
105
OK, so the drives still there know they are part of a pool with 3 drives missing... you "may" get it to mount by running zpool import -F ocean .

You will certainly be missing some (most likely all) of the data even if it will mount.

I'm afraid that in the situation you're in, that's the best you can hope for.

Excellent stuff sretalla, really appreciate your help!

ONE FINAL THOUGHT:
What would happen if I connected the three old/replaced 4TB WDs?!? Two of them still untouched since the re-silvering, so ideally it would be 4+1 (and one which is already used elsewhere). Obviously, some data were modified since then, but would it be worth trying?
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,703
What would happen if I connected the three old/replaced 4TB WDs?!? Two of them still untouched since the re-silvering, so ideally it would be 4+1 (and one which is already used elsewhere). Obviously, some data were modified since then, but would it be worth trying?
I suspect it can't hurt to try (with no inflated level of expectation). Depending on how much has changed and the state of those drives, you may get it to mount with enough replicas to make a full copy of some of the files... best of luck.

You can add the drives back and see what zpool import shows without even risking anything at all.
 

yourmate

Contributor
Joined
Apr 4, 2016
Messages
105
with no inflated level of expectation


LOL, this was rather excellent.

Not much to lose, as I already dealt with the loss emotionally :D
Also, I just bought another 3 8TB WD Reds, so none of the 4TBs would be in use (except the one I already used in a Windows system) so I can try after replacing them all.
I will report back - although it takes a bit of time as one of the drives is currently 1,200 miles away...

Thanks again. Stay safe :)
 
Top