Swapped out the 2nd disk with SMART errors this morning following the exact same procedure as above. Only this time, when I inserted the new disk into the bay previously occupied by the old disk, all the red fault lights came on in that chassis, and then ZFS started rediscovering all 50 disks!
Huge amount of log data spewed out in a few seconds. Here are some pieces of it: (forum wouldn't let me post the whole thing)
Code:
Jun 16 06:56:13 freenas (da36:mps2:0:18:0): READ(10). CDB: 28 00 66 88 b1 b0 00 00 20 00 length 16384 SMID 889 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da36:mps2:0:18:0): READ(10). CDB: 28 00 66 88 b1 f8 00 00 20 00 length 16384 SMID 207 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da35:mps2:0:17:0): READ(10). CDB: 28 00 66 88 b1 f8 00 00 20 00 length 16384 SMID 170 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da34:mps2:0:16:0): READ(10). CDB: 28 00 66 88 b1 f8 00 00 20 00 length 16384 SMID 470 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da33:mps2:0:15:0): READ(10). CDB: 28 00 66 88 b1 f8 00 00 20 00 length 16384 SMID 908 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da41:mps2:0:23:0): READ(10). CDB: 28 00 66 88 b1 b0 00 00 20 00 length 16384 SMID 265 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da41:mps2:0:23:0): READ(10). CDB: 28 00 66 88 b1 f0 00 00 20 00 length 16384 SMID 1017 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da40:mps2:0:22:0): READ(10). CDB: 28 00 66 88 b1 b0 00 00 20 00 length 16384 SMID 935 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da40:mps2:0:22:0): READ(10). CDB: 28 00 66 88 b1 f0 00 00 20 00 length 16384 SMID 551 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da39:mps2:0:21:0): READ(10). CDB: 28 00 66 88 b1 b0 00 00 20 00 length 16384 SMID 404 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da39:mps2:0:21:0): READ(10). CDB: 28 00 66 88 b1 f0 00 00 20 00 length 16384 SMID 136 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da38:mps2:0:33:0): READ(10). CDB: 28 00 66 88 b1 b0 00 00 20 00 length 16384 SMID 698 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da38:mps2:0:33:0): READ(10). CDB: 28 00 66 88 b1 f0 00 00 20 00 length 16384 SMID 130 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da37:mps2:0:19:0): READ(10). CDB: 28 00 66 88 b1 d0 00 00 20 00 length 16384 SMID 648 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da37:mps2:0:19:0): READ(10). CDB: 28 00 66 88 b1 b0 00 00 20 00 length 16384 SMID 1018 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): READ(10). CDB: 28 00 51 b0 ed a0 00 00 20 00 length 16384 SMID 754 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): READ(10). CDB: 28 00 51 b0 ed 80 00 00 20 00 length 16384 SMID 937 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da27:mps2:0:9:0): READ(10). CDB: 28 00 51 b0 ed c0 00 00 20 00 length 16384 SMID 666 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da27:mps2:0:9:0): READ(10). CDB: 28 00 51 b0 ed a0 00 00 20 00 length 16384 SMID 642 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da27:mps2:0:9:0): READ(10). CDB: 28 00 51 b0 ed 80 00 00 20 00 length 16384 SMID 583 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da35:mps2:0:17:0): READ(10). CDB: 28 00 66 88 b1 b0 00 00 20 00
Jun 16 06:56:13 freenas (da35:mps2:0:17:0): CAM status: SCSI Status Error
Jun 16 06:56:13 freenas (da35:mps2:0:17:0): SCSI status: Check Condition
Jun 16 06:56:13 freenas (da35:mps2:0:17:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun 16 06:56:13 freenas (da35:mps2:0:17:0): Retrying command (per sense data)
Jun 16 06:56:13 freenas (da34:mps2:0:16:0): READ(10). CDB: 28 00 66 88 b1 b0 00 00 20 00
Jun 16 06:56:13 freenas (da34:mps2:0:16:0): CAM status: SCSI Status Error
Jun 16 06:56:13 freenas (da34:mps2:0:16:0): SCSI status: Check Condition
Jun 16 06:56:13 freenas (da34:mps2:0:16:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun 16 06:56:13 freenas (da34:mps2:0:16:0): Retrying command (per sense data)
Jun 16 06:56:13 freenas (da33:mps2:0:15:0): READ(10). CDB: 28 00 66 88 b1 b0 00 00 20 00
Jun 16 06:56:13 freenas (da33:mps2:0:15:0): CAM status: SCSI Status Error
Jun 16 06:56:13 freenas (da33:mps2:0:15:0): SCSI status: Check Condition
Jun 16 06:56:13 freenas (da33:mps2:0:15:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun 16 06:56:13 freenas (da33:mps2:0:15:0): Retrying command (per sense data)
Jun 16 06:56:13 freenas (da34:mps2:0:16:0): READ(10). CDB: 28 00 66 88 b1 d8 00 00 20 00
Jun 16 06:56:13 freenas (da34:mps2:0:16:0): CAM status: SCSI Status Error
Jun 16 06:56:13 freenas (da34:mps2:0:16:0): SCSI status: Check Condition
Jun 16 06:56:13 freenas (da34:mps2:0:16:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun 16 06:56:13 freenas (da34:mps2:0:16:0): Retrying command (per sense data)
Jun 16 06:56:13 freenas (da33:mps2:0:15:0): READ(10). CDB: 28 00 66 88 b1 d8 00 00 20 00
Jun 16 06:56:13 freenas (da33:mps2:0:15:0): CAM status: SCSI Status Error
snip
Jun 16 06:56:13 freenas (da40:mps2:0:22:0): CAM status: SCSI Status Error
Jun 16 06:56:13 freenas (da40:mps2:0:22:0): SCSI status: Check Condition
Jun 16 06:56:13 freenas (da40:mps2:0:22:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun 16 06:56:13 freenas (da40:mps2:0:22:0): Retrying command (per sense data)
Jun 16 06:56:13 freenas (da40:mps2:0:22:0): READ(10). CDB: 28 00 66 88 b1 b0 00 00 20 00
Jun 16 06:56:13 freenas (da40:mps2:0:22:0): CAM status: SCSI Status Error
Jun 16 06:56:13 freenas (da40:mps2:0:22:0): SCSI status: Check Condition
snip
Jun 16 06:56:13 freenas (da44:mps2:0:27:0): READ(10). CDB: 28 00 34 29 37 98 00 00 20 00 length 16384 SMID 345 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da45:mps2:0:28:0): READ(10). CDB: 28 00 34 29 37 58 00 00 20 00 length 16384 SMID 225 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da45:mps2:0:28:0): READ(10). CDB: 28 00 34 29 37 78 00 00 20 00 length 16384 SMID 184 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da45:mps2:0:28:0): READ(10). CDB: 28 00 34 29 37 98 00 00 20 00 length 16384 SMID 778 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da46:mps2:0:29:0): READ(10). CDB: 28 00 34 29 37 58 00 00 20 00 length 16384 SMID 134 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da46:mps2:0:29:0): READ(10). CDB: 28 00 34 29 37 78 00 00 20 00 length 16384 SMID 450 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da46:mps2:0:29:0): READ(10). CDB: 28 00 34 29 37 98 00 00 20 00 length 16384 SMID 201 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da47:mps2:0:30:0): READ(10). CDB: 28 00 34 29 37 58 00 00 20 00 length 16384 SMID 578 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da47:mps2:0:30:0): READ(10). CDB: 28 00 34 29 37 78 00 00 20 00 length 16384 SMID 755 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da47:mps2:0:30:0): READ(10). CDB: 28 00 34 29 37 98 00 00 20 00 length 16384 SMID 994 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da49:mps2:0:32:0): READ(10). CDB: 28 00 34 29 37 78 00 00 20 00 length 16384 SMID 174 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da49:mps2:0:32:0): READ(10). CDB: 28 00 34 29 37 98 00 00 20 00 length 16384 SMID 961 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da49:mps2:0:32:0): READ(10). CDB: 28 00 34 29 37 b8 00 00 20 00 length 16384 SMID 446 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): READ(10). CDB: 28 00 51 b0 ed 80 00 00 20 00
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): CAM status: SCSI Status Error
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): SCSI status: Check Condition
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): Retrying command (per sense data)
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): READ(10). CDB: 28 00 51 b0 ed a0 00 00 20 00
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): CAM status: SCSI Status Error
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): SCSI status: Check Condition
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): Retrying command (per sense data)
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): READ(10). CDB: 28 00 51 b0 ed c0 00 00 20 00
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): CAM status: SCSI Status Error
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): SCSI status: Check Condition
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): Retrying command (per sense data)
Jun 16 06:56:13 freenas (da36:mps2:0:18:0): CAM status: SCSI Status Error
Jun 16 06:56:13 freenas (da36:mps2:0:18:0): SCSI status: Check Condition
Jun 16 06:56:13 freenas (da36:mps2:0:18:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun 16 06:56:13 freenas (da36:mps2:0:18:0): Retrying command (per sense data)
Jun 16 06:56:13 freenas (da36:mps2:0:18:0): READ(10). CDB: 28 00 66 88 b1 d0 00 00 20 00
Jun 16 06:56:13 freenas (da36:mps2:0:18:0): CAM status: SCSI Status Error
Jun 16 06:56:13 freenas (da36:mps2:0:18:0): SCSI status: Check Condition
Jun 16 06:56:13 freenas (da36:mps2:0:18:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun 16 06:56:13 freenas (da36:mps2:0:18:0): Retrying command (per sense data)
Jun 16 06:56:14 freenas (da30:mps2:0:12:0): READ(10). CDB: 28 00 51 b0 ed 80 00 00 20 00 length 16384 SMID 617 terminated ioc 804b scsi 0 state 0 xfer 0
Jun 16 06:56:14 freenas (da30:mps2:0:12:0): READ(10). CDB: 28 00 51 b0 ed a0 00 00 20 00 length 16384 SMID 695 terminated ioc 804b scsi 0 state 0 xfer 0
Jun 16 06:56:14 freenas (da30:mps2:0:12:0): READ(10). CDB: 28 00 51 b0 ed c0 00 00 20 00 length 16384 SMID 388 terminated ioc 804b scsi 0 state 0 xfer 0
Jun 16 06:56:14 freenas (da36:mps2:0:18:0): READ(10). CDB: 28 00 66 88 b1 d0 00 00 20 00 length 16384 SMID 266 terminated ioc 804b scsi 0 state 0 xfer 0
Jun 16 06:56:14 freenas (da37:mps2:0:19:0): READ(10). CDB: 28 00 66 88 b1 f0 00 00 20 00 length 16384 SMID 671 terminated ioc 804b scsi 0 state 0 xfer 0
Jun 16 06:56:14 freenas (da37:mps2:0:19:0): READ(10). CDB: 28 00 66 88 b1 b0 00 00 20 00 length 16384 SMID 686 terminated ioc 804b scsi 0 state 0 xfer 0
Jun 16 06:56:14 freenas (da37:mps2:0:19:0): READ(10). CDB: 28 00 66 88 b1 d0 00 00 20 00 length 16384 SMID 236 terminated ioc 804b scsi 0 state 0 xfer 0
snip
Jun 16 06:56:14 freenas (da28:mps2:0:10:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jun 16 06:56:14 freenas (da28:mps2:0:10:0): Retrying command (per sense data)
Jun 16 06:56:14 freenas (da29:mps2:0:11:0): READ(10). CDB: 28 00 51 b0 ed a0 00 00 20 00
Jun 16 06:56:14 freenas (da29:mps2:0:11:0): CAM status: SCSI Status Error
Jun 16 06:56:14 freenas (da29:mps2:0:11:0): SCSI status: Check Condition
Jun 16 06:56:14 freenas (da29:mps2:0:11:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jun 16 06:56:14 freenas (da29:mps2:0:11:0): Retrying command (per sense data)
Jun 16 06:56:15 freenas (da30:mps2:0:12:0): READ(10). CDB: 28 00 51 b0 ed 80 00 00 20 00
Jun 16 06:56:15 freenas (da30:mps2:0:12:0): CAM status: SCSI Status Error
Jun 16 06:56:15 freenas (da30:mps2:0:12:0): SCSI status: Check Condition
Jun 16 06:56:15 freenas (da30:mps2:0:12:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jun 16 06:56:15 freenas (da30:mps2:0:12:0): Retrying command (per sense data)
Jun 16 06:56:15 freenas (da31:mps2:0:13:0): READ(10). CDB: 28 00 51 b0 ed 60 00 00 20 00
Jun 16 06:56:15 freenas (da31:mps2:0:13:0): CAM status: SCSI Status Error
Jun 16 06:56:15 freenas (da31:mps2:0:13:0): SCSI status: Check Condition
Jun 16 06:56:15 freenas (da31:mps2:0:13:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jun 16 06:56:15 freenas (da31:mps2:0:13:0): Retrying command (per sense data)
Jun 16 06:56:15 freenas (da33:mps2:0:15:0): READ(10). CDB: 28 00 66 88 b1 f8 00 00 20 00
Jun 16 06:56:15 freenas (da33:mps2:0:15:0): CAM status: SCSI Status Error
Jun 16 06:56:15 freenas (da33:mps2:0:15:0): SCSI status: Check Condition
Jun 16 06:56:15 freenas (da33:mps2:0:15:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jun 16 06:56:15 freenas (da33:mps2:0:15:0): Retrying command (per sense data)
Jun 16 06:56:15 freenas (da34:mps2:0:16:0): READ(10). CDB: 28 00 66 88 b1 f8 00 00 20 00
Jun 16 06:56:15 freenas (da34:mps2:0:16:0): CAM status: SCSI Status Error
Jun 16 06:56:15 freenas (da34:mps2:0:16:0): SCSI status: Check Condition
Jun 16 06:56:15 freenas (da34:mps2:0:16:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jun 16 06:56:15 freenas (da34:mps2:0:16:0): Retrying command (per sense data)
Jun 16 06:56:15 freenas (da35:mps2:0:17:0): READ(10). CDB: 28 00 66 88 b1 f8 00 00 20 00
Jun 16 06:56:15 freenas (da35:mps2:0:17:0): CAM status: SCSI Status Error
Jun 16 06:56:15 freenas (da35:mps2:0:17:0): SCSI status: Check Condition
Jun 16 06:56:15 freenas (da35:mps2:0:17:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jun 16 06:56:15 freenas (da35:mps2:0:17:0): Retrying command (per sense data)
Jun 16 06:56:15 freenas (da36:mps2:0:18:0): READ(10). CDB: 28 00 66 88 b1 f8 00 00 20 00
Jun 16 06:56:15 freenas (da36:mps2:0:18:0): CAM status: SCSI Status Error
Jun 16 06:56:15 freenas (da36:mps2:0:18:0): SCSI status: Check Condition
Jun 16 06:56:15 freenas (da36:mps2:0:18:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jun 16 06:56:15 freenas (da36:mps2:0:18:0): Retrying command (per sense data)
Jun 16 06:56:15 freenas (da37:mps2:0:19:0): READ(10). CDB: 28 00 66 88 b1 f0 00 00 20 00
Jun 16 06:56:15 freenas (da37:mps2:0:19:0): CAM status: SCSI Status Error
Jun 16 06:56:15 freenas (da37:mps2:0:19:0): SCSI status: Check Condition
Jun 16 06:56:15 freenas (da37:mps2:0:19:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jun 16 06:56:15 freenas (da37:mps2:0:19:0): Retrying command (per sense data)
Jun 16 06:56:15 freenas (da42:mps2:0:25:0): READ(10). CDB: 28 00 34 29 37 c0 00 00 20 00
Jun 16 06:56:15 freenas (da42:mps2:0:25:0): CAM status: CCB request aborted by the host
Jun 16 06:56:15 freenas (da42:mps2:0:25:0): Retrying command
Jun 16 06:56:15 freenas zfsd: Event exceeds event size limit of 8192 bytes.
Jun 16 06:56:15 freenas zfsd: Truncated 7 characters from event.
Jun 16 06:56:15 freenas (da42:mps2:0:25:0): READ(10). CDB: 28 00 34 29 37 78 00 00 20 00
Jun 16 06:56:15 freenas (da42:mps2:0:25:0): CAM status: CCB request aborted by the host
Jun 16 06:56:15 freenas (da42:mps2:0:25:0): Retrying command
Jun 16 06:56:15 freenas (da42:mps2:0:25:0): READ(10). CDB: 28 00 34 29 37 58 00 00 20 00
Jun 16 06:56:15 freenas (da42:mps2:0:25:0): CAM status: CCB request aborted by the host
Jun 16 06:56:15 freenas (da42:mps2:0:25:0): Retrying command
Jun 16 06:56:15 freenas (da43:mps2:0:26:0): READ(10). CDB: 28 00 34 29 37 c0 00 00 20 00
Jun 16 06:56:15 freenas (da43:mps2:0:26:0): CAM status: CCB request aborted by the host
Jun 16 06:56:15 freenas (da43:mps2:0:26:0): Retrying command
Jun 16 06:56:15 freenas (da43:mps2:0:26:0): READ(10). CDB: 28 00 34 29 37 78 00 00 20 00
Jun 16 06:56:15 freenas (da43:mps2:0:26:0): CAM status: CCB request aborted by the host
Jun 16 06:56:15 freenas (da43:mps2:0:26:0): Retrying command
Jun 16 06:56:15 freenas (da43:mps2:0:26:0): READ(10). CDB: 28 00 34 29 37 58 00 00 20 00
Jun 16 06:56:15 freenas (da43:mps2:0:26:0): CAM status: CCB request aborted by the host
Jun 16 06:56:15 freenas (da43:mps2:0:26:0): Retrying command
Jun 16 06:56:15 freenas (da44:mps2:0:27:0): READ(10). CDB: 28 00 34 29 37 98 00 00 20 00
Jun 16 06:56:15 freenas (da44:mps2:0:27:0): CAM status: CCB request aborted by the host
Jun 16 06:56:15 freenas (da44:mps2:0:27:0): Retrying command
Jun 16 06:56:15 freenas (da44:mps2:0:27:0): READ(10). CDB: 28 00 34 29 37 78 00 00 20 00
Jun 16 06:56:15 freenas (da44:mps2:0:27:0): CAM status: CCB request aborted by the host
Jun 16 06:56:15 freenas (da44:mps2:0:27:0): Retrying command
Jun 16 06:56:15 freenas (da44:mps2:0:27:0): READ(10). CDB: 28 00 34 29 37 58 00 00 20 00
Jun 16 06:56:15 freenas (da44:mps2:0:27:0): CAM status: CCB request aborted by the host
Jun 16 06:56:15 freenas (da44:mps2:0:27:0): Retrying command
Jun 16 06:56:15 freenas (da45:mps2:0:28:0): READ(10). CDB: 28 00 34 29 37 98 00 00 20 00
Jun 16 06:56:15 freenas (da45:mps2:0:28:0): CAM status: CCB request aborted by the host
Jun 16 06:56:15 freenas (da45:mps2:0:28:0): Retrying command
Jun 16 06:56:15 freenas (da45:mps2:0:28:0): READ(10). CDB: 28 00 34 29 37 78 00 00 20 00
Jun 16 06:56:15 freenas (da45:mps2:0:28:0): CAM status: CCB request aborted by the host
Jun 16 06:56:15 freenas (da45:mps2:0:28:0): Retrying command
Jun 16 06:56:15 freenas (da45:mps2:0:28:0): READ(10). CDB: 28 00 34 29 37 58 00 00 20 00
Jun 16 06:56:15 freenas (da45:mps2:0:28:0): CAM status: CCB request aborted by the host
Jun 16 06:56:15 freenas (da45:mps2:0:28:0): Retrying command
Jun 16 06:56:15 freenas (da46:mps2:0:29:0): READ(10). CDB: 28 00 34 29 37 98 00 00 20 00
Jun 16 06:56:15 freenas (da46:mps2:0:29:0): CAM status: CCB request aborted by the host
snip
Jun 16 06:56:15 freenas da42 at mps2 bus 0 scbus2 target 25 lun 0
Jun 16 06:56:15 freenas da42: <ATA ST4000DM000-1F21 CC51> s/n Z3004TYS detached
Jun 16 06:56:15 freenas da43 at mps2 bus 0 scbus2 target 26 lun 0
Jun 16 06:56:15 freenas da43: <ATA ST4000DM000-1F21 CC52> s/n Z301N0RN detached
Jun 16 06:56:15 freenas da44 at mps2 bus 0 scbus2 target 27 lun 0
Jun 16 06:56:15 freenas da44: <ATA ST4000DM000-1F21 CC52> s/n Z301N06S detached
Jun 16 06:56:15 freenas da45 at mps2 bus 0 scbus2 target 28 lun 0
Jun 16 06:56:15 freenas da45: <ATA ST4000DM000-1F21 CC52> s/n W300SLZ2 detached
Jun 16 06:56:15 freenas da46 at mps2 bus 0 scbus2 target 29 lun 0
Jun 16 06:56:15 freenas da46: <ATA ST4000DM000-1F21 CC52> s/n W300STS6 detached
Jun 16 06:56:15 freenas da47 at mps2 bus 0 scbus2 target 30 lun 0
Jun 16 06:56:15 freenas da47: <ATA ST4000DM000-1F21 CC52> s/n W300STBJ detached
Jun 16 06:56:15 freenas da48 at mps2 bus 0 scbus2 target 31 lun 0
Jun 16 06:56:15 freenas da48: <ATA ST4000DM000-1F21 CC54> s/n S300HZMR detached
Jun 16 06:56:15 freenas da49 at mps2 bus 0 scbus2 target 32 lun 0
Jun 16 06:56:15 freenas da49: <ATA ST4000DM000-1F21 CC54> s/n S300HZNM detached
Jun 16 06:56:15 freenas GEOM_ELI: Device da42p1.eli destroyed.
Jun 16 06:56:15 freenas GEOM_ELI: Detached da42p1.eli on last close.
Jun 16 06:56:15 freenas GEOM_ELI: Device da43p1.eli destroyed.
Jun 16 06:56:15 freenas GEOM_ELI: Detached da43p1.eli on last close.
Jun 16 06:56:15 freenas GEOM_ELI: Device da44p1.eli destroyed.
Jun 16 06:56:15 freenas GEOM_ELI: Detached da44p1.eli on last close.
Jun 16 06:56:15 freenas GEOM_ELI: Device da45p1.eli destroyed.
Jun 16 06:56:15 freenas GEOM_ELI: Detached da45p1.eli on last close.
Jun 16 06:56:15 freenas GEOM_ELI: Device da46p1.eli destroyed.
Jun 16 06:56:15 freenas GEOM_ELI: Detached da46p1.eli on last close.
Jun 16 06:56:15 freenas GEOM_ELI: Device da47p1.eli destroyed.
Jun 16 06:56:15 freenas GEOM_ELI: Detached da47p1.eli on last close.
Jun 16 06:56:15 freenas GEOM_ELI: Device da48p1.eli destroyed.
Jun 16 06:56:15 freenas GEOM_ELI: Detached da48p1.eli on last close.
Jun 16 06:56:15 freenas GEOM_ELI: Device da49p1.eli destroyed.
Jun 16 06:56:15 freenas GEOM_ELI: Detached da49p1.eli on last close.
Jun 16 06:56:16 freenas (da38:mps2:0:33:0): READ(10). CDB: 28 00 66 88 b1 f0 00 00 20 00
Jun 16 06:56:16 freenas (da38:mps2:0:33:0): CAM status: SCSI Status Error
Jun 16 06:56:16 freenas (da38:mps2:0:33:0): SCSI status: Check Condition
Jun 16 06:56:16 freenas (da38:mps2:0:33:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jun 16 06:56:16 freenas (da38:mps2:0:33:0): Retrying command (per sense data)
Jun 16 06:56:16 freenas (da39:mps2:0:21:0): READ(10). CDB: 28 00 66 88 b1 f0 00 00 20 00
Jun 16 06:56:16 freenas (da39:mps2:0:21:0): CAM status: SCSI Status Error
Jun 16 06:56:16 freenas (da39:mps2:0:21:0): SCSI status: Check Condition
Jun 16 06:56:16 freenas (da39:mps2:0:21:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jun 16 06:56:16 freenas (da39:mps2:0:21:0): Retrying command (per sense data)
Jun 16 06:56:16 freenas (da40:mps2:0:22:0): READ(10). CDB: 28 00 66 88 b1 f0 00 00 20 00
Jun 16 06:56:16 freenas (da40:mps2:0:22:0): CAM status: SCSI Status Error
Jun 16 06:56:16 freenas (da40:mps2:0:22:0): SCSI status: Check Condition
Jun 16 06:56:16 freenas (da40:mps2:0:22:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jun 16 06:56:16 freenas (da40:mps2:0:22:0): Retrying command (per sense data)
Jun 16 06:56:16 freenas (da41:mps2:0:23:0): READ(10). CDB: 28 00 66 88 b1 f0 00 00 20 00
Jun 16 06:56:16 freenas (da41:mps2:0:23:0): CAM status: SCSI Status Error
Jun 16 06:56:16 freenas (da41:mps2:0:23:0): SCSI status: Check Condition
Jun 16 06:56:16 freenas (da41:mps2:0:23:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jun 16 06:56:16 freenas (da41:mps2:0:23:0): Retrying command (per sense data)
Jun 16 06:56:16 freenas (da42:mps2:0:25:0): Periph destroyed
Jun 16 06:56:16 freenas (da49:mps2:0:32:0): Periph destroyed
Jun 16 06:56:16 freenas (da48:mps2:0:31:0): Periph destroyed
Jun 16 06:56:16 freenas (da47:mps2:0:30:0): Periph destroyed
Jun 16 06:56:16 freenas (da46:mps2:0:29:0): Periph destroyed
Jun 16 06:56:16 freenas (da45:mps2:0:28:0): Periph destroyed
Jun 16 06:56:16 freenas (da44:mps2:0:27:0): Periph destroyed
Jun 16 06:56:16 freenas (da43:mps2:0:26:0): Periph destroyed
Jun 16 06:56:17 freenas da32 at mps2 bus 0 scbus2 target 27 lun 0
Jun 16 06:56:17 freenas da32: <ATA ST4000DM000-1F21 CC52> Fixed Direct Access SCSI-6 device
Jun 16 06:56:17 freenas da32: Serial Number Z301N06S
Jun 16 06:56:17 freenas da32: 600.000MB/s transfers
Jun 16 06:56:17 freenas da32: Command Queueing enabled
Jun 16 06:56:17 freenas da32: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
Jun 16 06:56:17 freenas da32: quirks=0x8<4K>
Jun 16 06:56:17 freenas da42 at mps2 bus 0 scbus2 target 28 lun 0
Jun 16 06:56:17 freenas da42: <ATA ST4000DM000-1F21 CC52> Fixed Direct Access SCSI-6 device
Jun 16 06:56:17 freenas da42: Serial Number W300SLZ2
Jun 16 06:56:17 freenas da42: 600.000MB/s transfers
Jun 16 06:56:17 freenas da42: Command Queueing enabled
Jun 16 06:56:17 freenas da42: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
Jun 16 06:56:17 freenas da42: quirks=0x8<4K>
Jun 16 06:56:17 freenas da43 at mps2 bus 0 scbus2 target 30 lun 0
Jun 16 06:56:17 freenas da43: <ATA ST4000DM000-1F21 CC52> Fixed Direct Access SCSI-6 device
Jun 16 06:56:17 freenas da43: Serial Number W300STBJ
Jun 16 06:56:17 freenas da43: 600.000MB/s transfers
Jun 16 06:56:17 freenas da43: Command Queueing enabled
Jun 16 06:56:17 freenas da43: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
Jun 16 06:56:17 freenas da43: quirks=0x8<4K>
Jun 16 06:56:17 freenas da44 at mps2 bus 0 scbus2 target 26 lun 0
Jun 16 06:56:17 freenas da44: <ATA ST4000DM000-1F21 CC52> Fixed Direct Access SCSI-6 device
Jun 16 06:56:17 freenas da44: Serial Number Z301N0RN
Jun 16 06:56:17 freenas da44: 600.000MB/s transfers
Jun 16 06:56:17 freenas da44: Command Queueing enabled
Jun 16 06:56:17 freenas da44: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
Jun 16 06:56:17 freenas da44: quirks=0x8<4K>
Jun 16 06:56:17 freenas da45 at mps2 bus 0 scbus2 target 29 lun 0
Jun 16 06:56:17 freenas da45: <ATA ST4000DM000-1F21 CC52> Fixed Direct Access SCSI-6 device
Jun 16 06:56:17 freenas da45: Serial Number W300STS6
Jun 16 06:56:17 freenas da45: 600.000MB/s transfers
Jun 16 06:56:17 freenas da45: Command Queueing enabled
Jun 16 06:56:17 freenas da45: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
Jun 16 06:56:17 freenas da45: quirks=0x8<4K>
Jun 16 06:56:17 freenas da46 at mps2 bus 0 scbus2 target 25 lun 0
Jun 16 06:56:17 freenas da46: <ATA ST4000DM000-1F21 CC51> Fixed Direct Access SCSI-6 device
Jun 16 06:56:17 freenas da46: Serial Number Z3004TYS
Jun 16 06:56:17 freenas da46: 600.000MB/s transfers
Jun 16 06:56:17 freenas da46: Command Queueing enabled
Jun 16 06:56:17 freenas da46: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
Jun 16 06:56:17 freenas da46: quirks=0x8<4K>
Jun 16 06:56:17 freenas da47 at mps2 bus 0 scbus2 target 32 lun 0
Jun 16 06:56:17 freenas da47: <ATA ST4000DM000-1F21 CC54> Fixed Direct Access SCSI-6 device
Jun 16 06:56:17 freenas da47: Serial Number S300HZNM
Jun 16 06:56:17 freenas da47: 600.000MB/s transfers
Jun 16 06:56:17 freenas da47: Command Queueing enabled
Jun 16 06:56:17 freenas da47: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
Jun 16 06:56:17 freenas da47: quirks=0x8<4K>
Jun 16 06:56:17 freenas da48 at mps2 bus 0 scbus2 target 31 lun 0
Jun 16 06:56:17 freenas da48: <ATA ST4000DM000-1F21 CC54> Fixed Direct Access SCSI-6 device
Jun 16 06:56:17 freenas da48: Serial Number S300HZMR
Jun 16 06:56:17 freenas da48: 600.000MB/s transfers
Jun 16 06:56:17 freenas da48: Command Queueing enabled
Jun 16 06:56:17 freenas da48: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
Jun 16 06:56:17 freenas da48: quirks=0x8<4K>
Jun 16 06:56:27 freenas da49 at mps2 bus 0 scbus2 target 34 lun 0
Jun 16 06:56:27 freenas da49: <ATA Hitachi HUA72302 A840> Fixed Direct Access SCSI-6 device
Jun 16 06:56:27 freenas da49: Serial Number YGGNZ0GD
Jun 16 06:56:27 freenas da49: 600.000MB/s transfers
Jun 16 06:56:27 freenas da49: Command Queueing enabled
Jun 16 06:56:27 freenas da49: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
Jun 16 06:56:46 freenas zfsd: Replace vdev(v1/10367877350599139386) by physical path: Unable to allocate spare target data.
Jun 16 06:56:46 freenas zfsd: Replace vdev(v1/7245229600930455534) by physical path: Unable to allocate spare target data.
Jun 16 06:56:46 freenas zfsd: Replace vdev(v1/11159745782969864676) by physical path: Unable to allocate spare target data.
Jun 16 06:56:46 freenas zfsd: Replace vdev(v1/10288687537002700716) by physical path: Unable to allocate spare target data.
Jun 16 06:56:46 freenas zfsd: Replace vdev(v1/7112856598500417308) by physical path: Unable to allocate spare target data.
Jun 16 06:56:46 freenas zfsd: Replace vdev(v1/13565106232670091350) by physical path: Unable to allocate spare target data.
Jun 16 06:56:46 freenas zfsd: Replace vdev(v1/2083712239136955752) by physical path: Unable to allocate spare target data.
Jun 16 06:56:46 freenas zfsd: Replace vdev(v1/8599137421765933395) by physical path: Unable to allocate spare target data.
[root@freenas] ~/scripts#
Fortunately everything was fine again within 30 seconds and I was able to ONLINE the new disk and the array was back in a normal state.
No idea what cause the fault lights to light up in that 2nd chassis and why zfs dropped all the drives and rediscovered them.
May when I do the next one I'll power down the whole system instead. That was quire a scare but I was impressed how quickly the system recovered.