Power Loss During Resilvering

MikeV

Cadet
Joined
Mar 19, 2017
Messages
2
I have 6x WD Red drives in a z2 configuration. It's been running great for five years. Yesterday I had my first SMART read failure but the pool remained "healthy." I puchased a 12GB WD Easystore and shucked the drive to find a WD wd120emaz which is fine for my purposes and the price. I followed the instructions in the FreeNAS manual. The only alteration is that FreeNAS complained that there was already a GPT present when I got to the actual replacement step, so I ran a gpart -F, rebooted and was able to replace the drive. Resilvering started. While the resilvering was proceeding, I was tidying up cables and must have either held the power button or placed something on top of the power button (its a Fractal Design case with the button on top) but I discovered the power was off a few minutes later. I powered it back on but the resilvering was still proceeding.

It is now the next day and the resilvering is complete but the pool is still degraded. The zpool status has also changed in that now all the disks are reported as "gptid" labels instead of the "daX" labels from before. I also seem to have two entries under the "replacing" entry and they are both "Unavail" It looks like something may have gone wrong but I'm at a loss as to what.

Any ideas as to how I can reset this process and get the new drive in line with the others?

Code:
admin@freenas:~ % zpool status -v
  pool: Storage1
 state: DEGRADED
  scan: resilvered 0 in 0 days 10:16:29 with 0 errors on Sat Jan 25 06:43:21 2020
config:

        NAME                                            STATE     READ WRITE CKSUM
        Storage1                                        DEGRADED     0     0     0
          raidz2-0                                      DEGRADED     0     0     0
            gptid/ca021fee-ba09-11e3-b119-002590f00770  ONLINE       0     0     0
            gptid/7578fc08-c256-11e3-b930-002590f00770  ONLINE       0     0     0
            replacing-2                                 UNAVAIL      0     0     0
              3976306838224556563                       UNAVAIL      0     0     0  was /dev/gptid/26b3141f-cb2d-11e3-8e6b-002590f00770
              12300288536700044800                      UNAVAIL      0     0     0  was /dev/gptid/c12de6da-3f11-11ea-b499-002590f00770
            gptid/cb2f11aa-ba09-11e3-b119-002590f00770  ONLINE       0     0     0
            gptid/cb82865b-ba09-11e3-b119-002590f00770  ONLINE       0     0     0
            gptid/cbdbd264-ba09-11e3-b119-002590f00770  ONLINE       0     0     0

errors: No known data errors


Note below that there are only five storage disks shown (da0-da4). The sixth new disk doesn't appear.
Code:
admin@freenas:~ % gpart show
=>       40  234441568  ada0  GPT  (112G)
         40     204800     1  efi  (100M)
     204840  234236760     2  freebsd-zfs  (112G)
  234441600          8        - free -  (4.0K)

=>        34  7814037101  da0  GPT  (3.6T)
          34          94       - free -  (47K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  7809842696    2  freebsd-zfs  (3.6T)
  7814037128           7       - free -  (3.5K)

=>        34  7814037101  da1  GPT  (3.6T)
          34          94       - free -  (47K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  7809842696    2  freebsd-zfs  (3.6T)
  7814037128           7       - free -  (3.5K)

=>        34  7814037101  da2  GPT  (3.6T)
          34          94       - free -  (47K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  7809842696    2  freebsd-zfs  (3.6T)
  7814037128           7       - free -  (3.5K)

=>        34  7814037101  da3  GPT  (3.6T)
          34          94       - free -  (47K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  7809842696    2  freebsd-zfs  (3.6T)
  7814037128           7       - free -  (3.5K)

=>        34  7814037101  da4  GPT  (3.6T)
          34          94       - free -  (47K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  7809842696    2  freebsd-zfs  (3.6T)
  7814037128           7       - free -  (3.5K)


Motherboard make and model: Supermicro MBD-X10SL7-F-O
CPU make and model: Xeon E3-1230v3
RAM quantity: 32GB DDR3 ECC
Hard drives, quantity, model numbers, and RAID configuration, including boot drives: 6x WD red, RAIDZ2, booting off Sandisk 120GB SSD
Hard disk controllers: mobo onboard controller
Network cards: mobo onboard NIC
FreeNAS 11.2-U7
 

MikeV

Cadet
Joined
Mar 19, 2017
Messages
2
The problem is solved. I shut down and took the disk out of the system then ran a gpart -F on another computer. Put it back and started up. No change in the zpool status. So, I chose the top UNAVAIL volume in the GUI and replaced it with the newly cleaned disk. The system took 10 hours to resilver but then everything was as it should be. The GUI shows the daX names once again and the pool is healthy.

Code:
admin@freenas:~ % zpool status
  pool: Storage1
 state: ONLINE
  scan: resilvered 2.62T in 0 days 10:18:24 with 0 errors on Sun Jan 26 03:03:29 2020
config:

        NAME                                            STATE     READ WRITE CKSUM
        Storage1                                        ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/ca021fee-ba09-11e3-b119-002590f00770  ONLINE       0     0     0
            gptid/7578fc08-c256-11e3-b930-002590f00770  ONLINE       0     0     0
            gptid/f0b46957-3fbb-11ea-af4d-002590f00770  ONLINE       0     0     0
            gptid/cb2f11aa-ba09-11e3-b119-002590f00770  ONLINE       0     0     0
            gptid/cb82865b-ba09-11e3-b119-002590f00770  ONLINE       0     0     0
            gptid/cbdbd264-ba09-11e3-b119-002590f00770  ONLINE       0     0     0

errors: No known data errors


Code:
admin@freenas:~ % gpart show
=>       40  234441568  ada0  GPT  (112G)
         40     204800     1  efi  (100M)
     204840  234236760     2  freebsd-zfs  (112G)
  234441600          8        - free -  (4.0K)

=>        34  7814037101  da0  GPT  (3.6T)
          34          94       - free -  (47K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  7809842696    2  freebsd-zfs  (3.6T)
  7814037128           7       - free -  (3.5K)

=>        34  7814037101  da1  GPT  (3.6T)
          34          94       - free -  (47K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  7809842696    2  freebsd-zfs  (3.6T)
  7814037128           7       - free -  (3.5K)

=>        34  7814037101  da2  GPT  (3.6T)
          34          94       - free -  (47K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  7809842696    2  freebsd-zfs  (3.6T)
  7814037128           7       - free -  (3.5K)

=>        34  7814037101  da3  GPT  (3.6T)
          34          94       - free -  (47K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  7809842696    2  freebsd-zfs  (3.6T)
  7814037128           7       - free -  (3.5K)

=>        34  7814037101  da4  GPT  (3.6T)
          34          94       - free -  (47K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  7809842696    2  freebsd-zfs  (3.6T)
  7814037128           7       - free -  (3.5K)

=>         40  23437770672  da5  GPT  (11T)
           40           88       - free -  (44K)
          128      4194304    1  freebsd-swap  (2.0G)
      4194432  23433576280    2  freebsd-zfs  (11T)
 
Top