Help with gptzfsboot error 16

[Steve_C]

Recently my iXMini went to degraded state. I was able to log into the GUI an noticed that there was no information for device ada1. I looked at the indicator lights, and the activity light for the ada1 disk was on continuously. There was no flickering. I used to power off drop down from the GUI to shit down the system. I reseated the ada1 drive and powered it back on. I was greeted by a series scrolling series of gptzfsboot: error 16 lba XXXXXXXXX where XXXXXXX is a set of increasing numbers. It's been scrolling for awhile now, and I'm wondering if it will resolve itselft, or I need to intervene. A search for error 16 did not turn up anything.

Thanks in advance

 

[Steve_C]

Update: it finished booting. ada1 is missing from the disk list.

Is it safe to power it down and switch that disk position with another to determine if it is a disk issue or a iXMini issue?

Any other debugging tips would be appreciated.

 

[Redcoat]

Yes, It should be safe to do that.. Be aware that the logical disk numbers may be reassigned - look for the activity lights on the disk caddys to see if that same caddy content behavior transfers.

 

[Steve_C]

Thanks, that should be easy to tell. The disk activity light on it is on solid.

Here's what I have now.

And here's a bit of the messages log file.

Jul 28 20:54:13 freenas Trying to mount root from zfs:freenas-boot/ROOT/11.3-U3.2 []...
Jul 28 20:54:13 freenas (ada1:ahcich10:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 af f4 a0 40 ba 02 00 00 00 00
Jul 28 20:54:13 freenas (ada1:ahcich10:0:0:0): CAM status: ATA Status Error
Jul 28 20:54:13 freenas (ada1:ahcich10:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul 28 20:54:13 freenas (ada1:ahcich10:0:0:0): RES: 41 40 af f4 a0 00 ba 02 00 00 00
Jul 28 20:54:13 freenas (ada1:ahcich10:0:0:0): Retrying command
Jul 28 20:54:13 freenas (ada1:ahcich10:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 af f4 a0 40 ba 02 00 00 00 00
Jul 28 20:54:13 freenas (ada1:ahcich10:0:0:0): CAM status: ATA Status Error
Jul 28 20:54:13 freenas (ada1:ahcich10:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul 28 20:54:13 freenas (ada1:ahcich10:0:0:0): RES: 41 40 af f4 a0 00 ba 02 00 00 00
Jul 28 20:54:13 freenas (ada1:ahcich10:0:0:0): Retrying command
Jul 28 20:54:13 freenas (ada1:ahcich10:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 af f4 a0 40 ba 02 00 00 00 00
Jul 28 20:54:13 freenas (ada1:ahcich10:0:0:0): CAM status: ATA Status Error
Jul 28 20:54:13 freenas (ada1:ahcich10:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul 28 20:54:13 freenas (ada1:ahcich10:0:0:0): RES: 41 40 af f4 a0 00 ba 02 00 00 00
Jul 28 20:54:13 freenas (ada1:ahcich10:0:0:0): Retrying command
Jul 28 20:54:13 freenas (ada1:ahcich10:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 af f4 a0 40 ba 02 00 00 00 00
Jul 28 20:54:13 freenas (ada1:ahcich10:0:0:0): CAM status: ATA Status Error
Jul 28 20:54:13 freenas (ada1:ahcich10:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul 28 20:54:13 freenas (ada1:ahcich10:0:0:0): RES: 41 40 af f4 a0 00 ba 02 00 00 00
Jul 28 20:54:13 freenas (ada1:ahcich10:0:0:0): Retrying command
Jul 28 20:54:13 freenas (ada1:ahcich10:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 af f4 a0 40 ba 02 00 00 00 00
......
Jul 28 20:54:13 freenas (ada1:ahcich10:0:0:0): Error 5, Retries exhausted
Jul 28 20:54:13 freenas GEOM: ada1: the secondary GPT table is corrupt or invalid.
Jul 28 20:54:13 freenas GEOM: ada1: using the primary only -- recovery suggested.
Jul 28 20:54:13 freenas (ada1:ahcich10:0:0:0): READ_FPDMA_QUEUED. ACB: 60 01 40 00 00 40 00 00 00 00 00 00
Jul 28 20:54:13 freenas (ada1:ahcich10:0:0:0): CAM status: ATA Status Error
Jul 28 20:54:13 freenas (ada1:ahcich10:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
Jul 28 20:54:13 freenas (ada1:ahcich10:0:0:0): RES: 41 40 40 00 00 00 00 00 00 00 00


Still going to swap the two caddies, but from the looks of the log, it knows that ada1 is there.

Hold on.....

 

[Steve_C]

Some more info.

root@freenas:/var/log # gpart show
=> 34 31277165 ada0 GPT (15G)
34 6 - free - (3.0K)
40 1024 1 freebsd-boot (512K)
1064 31276128 2 freebsd-zfs (15G)
31277192 7 - free - (3.5K)

=> 40 11721045088 ada2 GPT (5.5T)
40 88 - free - (44K)
128 4194304 1 freebsd-swap (2.0G)
4194432 11716850696 2 freebsd-zfs (5.5T)

=> 40 11721045088 ada3 GPT (5.5T)
40 88 - free - (44K)
128 4194304 1 freebsd-swap (2.0G)
4194432 11716850696 2 freebsd-zfs (5.5T)

=> 40 11721045088 ada4 GPT (5.5T)
40 88 - free - (44K)
128 4194304 1 freebsd-swap (2.0G)
4194432 11716850696 2 freebsd-zfs (5.5T)

root@freenas:/var/log # gpart show ada1
gpart: No such geom: ada1.
root@freenas:/var/log #

 

[Steve_C]

root@freenas:/var/log # zpool status
pool: FreeNAS
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://illumos.org/msg/ZFS-8000-2Q
scan: scrub repaired 0 in 0 days 10:55:46 with 0 errors on Sun Jul 12 10:55:47 2020
config:

NAME STATE READ WRITE CKSUM
FreeNAS DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
344171107366985803 UNAVAIL 0 0 0 was /dev/gptid/3dff1796-b8ba-11ea-a8a7-d050992ed042
gptid/a0feb954-0e63-11ea-988e-d050992ed042 ONLINE 0 0 0
gptid/36c4a3dd-a303-11ea-afba-d050992ed042 ONLINE 0 0 0
gptid/8a5bfe8b-7314-11ea-bd33-d050992ed042 ONLINE 0 0 0

errors: No known data errors

pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0 days 00:01:11 with 0 errors on Mon Jul 27 03:46:12 2020
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
ada0p2 ONLINE 0 0 0

errors: No known data errors

 

[Steve_C]

Caddies swapped, and the always on light moved with it. All the swap did was change ada1 to ada2 in the error messages. Swapped the caddies back to where they were originally.

From the above, do I need to rebuild it somehow? Is a replacement the best option?

 

[Redcoat]

Caddies swapped, and the always on light moved with it. All the swap did was change ada1 to ada2 in the error messages. Swapped the caddies back to where they were originally.

From the above, do I need to rebuild it somehow? Is a replacement the best option?

Looks like a replacement drive is called for.

 

[Steve_C]

Thanks for the help.

Hopefully Western Digital will send an actual new drive not a refurbished one.

 

[Steve_C]

Ended up just buying a new drive. If I get one from WD, it will be a spare.

During the resilvering, the message log is filling up with the following: messages. Is this normal?

Jul 30 23:05:18 freenas (ada1:ahcich10:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 c8 b6 39 40 5f 00 00 01 00 00
Jul 30 23:05:18 freenas (ada1:ahcich10:0:0:0): CAM status: ATA Status Error
Jul 30 23:05:18 freenas (ada1:ahcich10:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF )
Jul 30 23:05:18 freenas (ada1:ahcich10:0:0:0): RES: 41 10 c8 b6 39 00 5f 00 00 00 00
Jul 30 23:05:18 freenas (ada1:ahcich10:0:0:0): Retrying command
Jul 30 23:05:25 freenas (ada1:ahcich10:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 d0 b7 39 40 5f 00 00 01 00 00
Jul 30 23:05:25 freenas (ada1:ahcich10:0:0:0): CAM status: ATA Status Error
Jul 30 23:05:25 freenas (ada1:ahcich10:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF )
Jul 30 23:05:25 freenas (ada1:ahcich10:0:0:0): RES: 41 10 d0 b7 39 00 5f 00 00 00 00
Jul 30 23:05:25 freenas (ada1:ahcich10:0:0:0): Retrying command
Jul 30 23:05:43 freenas (ada1:ahcich10:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 38 71 43 40 5f 00 00 01 00 00
Jul 30 23:05:43 freenas (ada1:ahcich10:0:0:0): CAM status: ATA Status Error
Jul 30 23:05:43 freenas (ada1:ahcich10:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF )
Jul 30 23:05:43 freenas (ada1:ahcich10:0:0:0): RES: 41 10 38 71 43 00 5f 00 00 00 00
Jul 30 23:05:43 freenas (ada1:ahcich10:0:0:0): Retrying command
Jul 30 23:05:50 freenas (ada1:ahcich10:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 38 73 43 40 5f 00 00 01 00 00
Jul 30 23:05:50 freenas (ada1:ahcich10:0:0:0): CAM status: ATA Status Error
Jul 30 23:05:50 freenas (ada1:ahcich10:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF )
Jul 30 23:05:50 freenas (ada1:ahcich10:0:0:0): RES: 41 10 38 73 43 00 5f 00 00 00 00
Jul 30 23:05:50 freenas (ada1:ahcich10:0:0:0): Retrying command

I'm guessing it's just the write thread getting blocked due to the write vs read speed, but checking in to be sure.

Thanks again for the help.

 

[ttippl]

H

Ended up just buying a new drive. If I get one from WD, it will be a spare.

During the resilvering, the message log is filling up with the following: messages. Is this normal?

Jul 30 23:05:18 freenas (ada1:ahcich10:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 c8 b6 39 40 5f 00 00 01 00 00
Jul 30 23:05:18 freenas (ada1:ahcich10:0:0:0): CAM status: ATA Status Error
Jul 30 23:05:18 freenas (ada1:ahcich10:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF )
Jul 30 23:05:18 freenas (ada1:ahcich10:0:0:0): RES: 41 10 c8 b6 39 00 5f 00 00 00 00
Jul 30 23:05:18 freenas (ada1:ahcich10:0:0:0): Retrying command
Jul 30 23:05:25 freenas (ada1:ahcich10:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 d0 b7 39 40 5f 00 00 01 00 00
Jul 30 23:05:25 freenas (ada1:ahcich10:0:0:0): CAM status: ATA Status Error
Jul 30 23:05:25 freenas (ada1:ahcich10:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF )
Jul 30 23:05:25 freenas (ada1:ahcich10:0:0:0): RES: 41 10 d0 b7 39 00 5f 00 00 00 00
Jul 30 23:05:25 freenas (ada1:ahcich10:0:0:0): Retrying command
Jul 30 23:05:43 freenas (ada1:ahcich10:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 38 71 43 40 5f 00 00 01 00 00
Jul 30 23:05:43 freenas (ada1:ahcich10:0:0:0): CAM status: ATA Status Error
Jul 30 23:05:43 freenas (ada1:ahcich10:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF )
Jul 30 23:05:43 freenas (ada1:ahcich10:0:0:0): RES: 41 10 38 71 43 00 5f 00 00 00 00
Jul 30 23:05:43 freenas (ada1:ahcich10:0:0:0): Retrying command
Jul 30 23:05:50 freenas (ada1:ahcich10:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 00 38 73 43 40 5f 00 00 01 00 00
Jul 30 23:05:50 freenas (ada1:ahcich10:0:0:0): CAM status: ATA Status Error
Jul 30 23:05:50 freenas (ada1:ahcich10:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF )
Jul 30 23:05:50 freenas (ada1:ahcich10:0:0:0): RES: 41 10 38 73 43 00 5f 00 00 00 00
Jul 30 23:05:50 freenas (ada1:ahcich10:0:0:0): Retrying command

I'm guessing it's just the write thread getting blocked due to the write vs read speed, but checking in to be sure.

Thanks again for the help.

Hi, did you solved this problem? I have same issue with same error and i don´t know what to do. I dont think, its broken disk, coz if i swap them, everytime is missing different disk, but everytime on same possition ada03. And Sometime i have problem to boot coz problem with ada01, but sometime it boots normaly but in degraded state coz removed ada03. What was your solution please?