Errors on new boot pool ssd

simdim

Explorer
Joined
Mar 12, 2019
Messages
75
Hi All.
Running TrueNAS-SCALE-22.02.1
I had recently converted from single usb stick boot to mirrored SSD ( PNY CS900 https://www.amazon.com/gp/product/B0722XPTL6/)
The are driven by SAS2008 (https://www.amazon.com/gp/product/B01M2AC40Y/) controller
The same controller is driving 2 pools of WD RED 2x2 mirrored drives.

Last Friday I had got errors on one of SSDs
root@freenas[~]# zpool status
pool: BigDisk
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: resilvered 5.78T in 15:12:46 with 0 errors on Tue Apr 26 01:59:13 2022
config:

NAME STATE READ WRITE CKSUM
BigDisk ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ad805547-c2aa-11e8-abea-50e54950d301 ONLINE 0 0 0
ef16e8ca-55fa-4496-b29e-a791c878295e ONLINE 0 0 0

errors: No known data errors

pool: freenas-boot
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
scan: resilvered 1.03M in 00:00:00 with 0 errors on Fri May 13 22:45:38 2022
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sda2 ONLINE 0 0 0
sdb2 ONLINE 3 280 0 <---------------------------------------!

errors: No known data errors

pool: pool
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: scrub repaired 0B in 01:39:43 with 0 errors on Tue Apr 26 15:26:09 2022
config:

NAME STATE READ WRITE CKSUM
pool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
d6f0fbc8-651f-11e9-9eb7-a0369f85bf5c ONLINE 0 0 0
17d4aa63-c4ee-11e8-9cd3-001b21028156 ONLINE 0 0 0

errors: No known data errors

Seems like there was a timeout:

May 13 22:43:41 freenas kernel: sd 0:0:1:0: attempting task abort!scmd(0x00000000811943f4), outstanding for 30556 ms & timeout 30000 ms
May 13 22:43:41 freenas kernel: sd 0:0:1:0: [sdb] tag#622 CDB: Write(10) 2a 00 02 59 76 8e 00 00 05 00
May 13 22:43:41 freenas kernel: scsi target0:0:1: handle(0x0009), sas_address(0x4433221102000000), phy(2)
May 13 22:43:41 freenas kernel: scsi target0:0:1: enclosure logical id(0x500605b002c8f5e1), slot(1)
May 13 22:43:45 freenas kernel: sd 0:0:1:0: task abort: SUCCESS scmd(0x00000000811943f4)
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#622 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=34s
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#622 CDB: Write(10) 2a 00 02 59 76 8e 00 00 05 00
May 13 22:43:45 freenas kernel: zio pool=freenas-boot vdev=/dev/sdb2 error=5 type=2 offset=20181208064 size=2560 flags=180880
May 13 22:43:45 freenas kernel: sd 0:0:1:0: attempting task abort!scmd(0x0000000079e54da6), outstanding for 34560 ms & timeout 30000 ms
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#630 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#621 CDB: Write(10) 2a 00 02 3b b7 f5 00 00 03 00
May 13 22:43:45 freenas kernel: scsi target0:0:1: handle(0x0009), sas_address(0x4433221102000000), phy(2)
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#630 Sense Key : Not Ready [current]
May 13 22:43:45 freenas kernel: scsi target0:0:1: enclosure logical id(0x500605b002c8f5e1), slot(1)
May 13 22:43:45 freenas kernel: sd 0:0:1:0: No reference found at driver, assuming scmd(0x0000000079e54da6) might have completed
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#630 Add. Sense: Logical unit not ready, cause not reportable
May 13 22:43:45 freenas kernel: sd 0:0:1:0: task abort: SUCCESS scmd(0x0000000079e54da6)
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#621 FAILED Result: hostbyte=DID_TIME_OUT driverbyte=DRIVER_OK cmd_age=34s
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#630 CDB: Write(10) 2a 00 02 5c 1f 0d 00 00 0a 00
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#621 CDB: Write(10) 2a 00 02 3b b7 f5 00 00 03 00
May 13 22:43:45 freenas kernel: zio pool=freenas-boot vdev=/dev/sdb2 error=5 type=2 offset=20270402048 size=5120 flags=40080c80
May 13 22:43:45 freenas kernel: zio pool=freenas-boot vdev=/dev/sdb2 error=5 type=2 offset=19183147520 size=1536 flags=180880
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#631 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#631 Sense Key : Not Ready [current]
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#631 Add. Sense: Logical unit not ready, cause not reportable
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#631 CDB: Read(10) 28 00 00 00 06 38 00 00 10 00
May 13 22:43:45 freenas kernel: zio pool=freenas-boot vdev=/dev/sdb2 error=5 type=1 offset=270336 size=8192 flags=b08c1
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#632 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#632 Sense Key : Not Ready [current]
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#632 Add. Sense: Logical unit not ready, cause not reportable
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#632 CDB: Read(10) 28 00 0d f9 46 38 00 00 10 00
May 13 22:43:45 freenas kernel: zio pool=freenas-boot vdev=/dev/sdb2 error=5 type=1 offset=120032862208 size=8192 flags=b08c1
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#633 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#633 Sense Key : Not Ready [current]
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#633 Add. Sense: Logical unit not ready, cause not reportable
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#633 CDB: Read(10) 28 00 0d f9 48 38 00 00 10 00
May 13 22:43:45 freenas kernel: zio pool=freenas-boot vdev=/dev/sdb2 error=5 type=1 offset=120033124352 size=8192 flags=b08c1
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#634 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#634 Sense Key : Not Ready [current]
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#634 Add. Sense: Logical unit not ready, cause not reportable
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#634 CDB: Write(10) 2a 00 02 99 3e 11 00 00 05 00
May 13 22:43:45 freenas kernel: zio pool=freenas-boot vdev=/dev/sdb2 error=5 type=2 offset=22321287680 size=2560 flags=180880
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#635 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#635 Sense Key : Not Ready [current]
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#635 Add. Sense: Logical unit not ready, cause not reportable
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#635 CDB: Write(10) 2a 00 02 9b 43 8c 00 00 0a 00
May 13 22:43:45 freenas kernel: zio pool=freenas-boot vdev=/dev/sdb2 error=5 type=2 offset=22389114880 size=5120 flags=40080c80
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#641 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#641 Sense Key : Not Ready [current]
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#641 Add. Sense: Logical unit not ready, cause not reportable
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#641 CDB: Read(10) 28 00 00 00 06 38 00 00 10 00
May 13 22:43:45 freenas kernel: zio pool=freenas-boot vdev=/dev/sdb2 error=5 type=1 offset=270336 size=8192 flags=b0ac1
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#642 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#642 Sense Key : Not Ready [current]
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#642 Add. Sense: Logical unit not ready, cause not reportable
May 13 22:43:45 freenas kernel: sd 0:0:1:0: [sdb] tag#642 CDB: Read(10) 28 00 0d f9 46 38 00 00 10 00
May 13 22:43:45 freenas kernel: zio pool=freenas-boot vdev=/dev/sdb2 error=5 type=1 offset=120032862208 size=8192 flags=b0ac1
May 13 22:43:45 freenas kernel: zio pool=freenas-boot vdev=/dev/sdb2 error=5 type=1 offset=120033124352 size=8192 flags=b0ac1
May 13 22:44:21 freenas kernel: sd 0:0:1:0: device_block, handle(0x0009)
May 13 22:44:23 freenas kernel: sd 0:0:1:0: device_unblock and setting to running, handle(0x0009)
May 13 22:44:23 freenas kernel: sd 0:0:1:0: [sdb] Synchronizing SCSI cache
May 13 22:44:23 freenas kernel: sd 0:0:1:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
May 13 22:44:23 freenas kernel: mpt2sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x4433221102000000)
May 13 22:44:23 freenas kernel: mpt2sas_cm0: removing handle(0x0009), sas_addr(0x4433221102000000)
May 13 22:44:23 freenas kernel: mpt2sas_cm0: enclosure logical id(0x500605b002c8f5e1), slot(1)
May 13 22:45:38 freenas kernel: scsi 0:0:6:0: Direct-Access ATA PNY CS900 120GB 0615 PQ: 0 ANSI: 6
May 13 22:45:38 freenas kernel: scsi 0:0:6:0: SATA: handle(0x0009), sas_addr(0x4433221102000000), phy(2), device_name(0x5f8db4c2144007a9)
May 13 22:45:38 freenas kernel: scsi 0:0:6:0: enclosure logical id (0x500605b002c8f5e1), slot(1)
May 13 22:45:38 freenas kernel: scsi 0:0:6:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
May 13 22:45:38 freenas kernel: scsi 0:0:6:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
May 13 22:45:38 freenas kernel: sd 0:0:6:0: Attached scsi generic sg1 type 0
May 13 22:45:38 freenas kernel: end_device-0:6: add: handle(0x0009), sas_addr(0x4433221102000000)
May 13 22:45:38 freenas kernel: sd 0:0:6:0: [sdb] 234441648 512-byte logical blocks: (120 GB/112 GiB)
May 13 22:45:38 freenas kernel: sd 0:0:6:0: [sdb] Write Protect is off
May 13 22:45:38 freenas kernel: sd 0:0:6:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
May 13 22:45:38 freenas kernel: sdb: sdb1 sdb2
May 13 22:45:38 freenas kernel: sd 0:0:6:0: [sdb] Attached SCSI disk


Should I suspect disk or controller?
Also should I upgrade my pools as message states:

"status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details."
 

simdim

Explorer
Joined
Mar 12, 2019
Messages
75
Did zpool clear freenas-pool
And rerun scrub of freenas-pool - no errors, should I replace this disk just incase?
 

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,947
I wouldn't bother. Just make sure you have a backup of the config file
 
Top