BUILD SuperMicro X10SRL-F + 3 846 Chassis + 72 Disks

[Bidule0hm]

Offlining it makes sure the software is aware that you unplug the drive but I'm not sure it's required.

 

[pclausen]

That's what I was wondering. I have noticed when testing things out, that within a few seconds of pulling a disk out of a bay, I get a "peripheral destroyed" message on the console. I guess maybe by offlining it first, FreeNAS freaks out a little less. :D

 

[mjws00]

Offlining it first handles swap correctly.

 

[pclausen]

Great, that worked perfectly.

Once the resilver completed, the new disk (da51) took over the spot previously occupied by the old (da38) disk.

Then as soon as I offlined the new disk, I got this:

The I pulled out the old drive (da38) and moved the new drive into that bay.

And finally, I onlined the new drive, and presto, everything was back to normal and the new drive even inherited the da assignment of the old drive.

 

[pclausen]

Swapped out the 2nd disk with SMART errors this morning following the exact same procedure as above. Only this time, when I inserted the new disk into the bay previously occupied by the old disk, all the red fault lights came on in that chassis, and then ZFS started rediscovering all 50 disks!

Huge amount of log data spewed out in a few seconds. Here are some pieces of it: (forum wouldn't let me post the whole thing)

Code:

Jun 16 06:56:13 freenas  (da36:mps2:0:18:0): READ(10). CDB: 28 00 66 88 b1 b0 00 00 20 00 length 16384 SMID 889 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da36:mps2:0:18:0): READ(10). CDB: 28 00 66 88 b1 f8 00 00 20 00 length 16384 SMID 207 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da35:mps2:0:17:0): READ(10). CDB: 28 00 66 88 b1 f8 00 00 20 00 length 16384 SMID 170 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da34:mps2:0:16:0): READ(10). CDB: 28 00 66 88 b1 f8 00 00 20 00 length 16384 SMID 470 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da33:mps2:0:15:0): READ(10). CDB: 28 00 66 88 b1 f8 00 00 20 00 length 16384 SMID 908 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da41:mps2:0:23:0): READ(10). CDB: 28 00 66 88 b1 b0 00 00 20 00 length 16384 SMID 265 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da41:mps2:0:23:0): READ(10). CDB: 28 00 66 88 b1 f0 00 00 20 00 length 16384 SMID 1017 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da40:mps2:0:22:0): READ(10). CDB: 28 00 66 88 b1 b0 00 00 20 00 length 16384 SMID 935 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da40:mps2:0:22:0): READ(10). CDB: 28 00 66 88 b1 f0 00 00 20 00 length 16384 SMID 551 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da39:mps2:0:21:0): READ(10). CDB: 28 00 66 88 b1 b0 00 00 20 00 length 16384 SMID 404 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da39:mps2:0:21:0): READ(10). CDB: 28 00 66 88 b1 f0 00 00 20 00 length 16384 SMID 136 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da38:mps2:0:33:0): READ(10). CDB: 28 00 66 88 b1 b0 00 00 20 00 length 16384 SMID 698 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da38:mps2:0:33:0): READ(10). CDB: 28 00 66 88 b1 f0 00 00 20 00 length 16384 SMID 130 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da37:mps2:0:19:0): READ(10). CDB: 28 00 66 88 b1 d0 00 00 20 00 length 16384 SMID 648 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da37:mps2:0:19:0): READ(10). CDB: 28 00 66 88 b1 b0 00 00 20 00 length 16384 SMID 1018 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da30:mps2:0:12:0): READ(10). CDB: 28 00 51 b0 ed a0 00 00 20 00 length 16384 SMID 754 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da30:mps2:0:12:0): READ(10). CDB: 28 00 51 b0 ed 80 00 00 20 00 length 16384 SMID 937 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da27:mps2:0:9:0): READ(10). CDB: 28 00 51 b0 ed c0 00 00 20 00 length 16384 SMID 666 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da27:mps2:0:9:0): READ(10). CDB: 28 00 51 b0 ed a0 00 00 20 00 length 16384 SMID 642 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da27:mps2:0:9:0): READ(10). CDB: 28 00 51 b0 ed 80 00 00 20 00 length 16384 SMID 583 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da35:mps2:0:17:0): READ(10). CDB: 28 00 66 88 b1 b0 00 00 20 00
Jun 16 06:56:13 freenas (da35:mps2:0:17:0): CAM status: SCSI Status Error
Jun 16 06:56:13 freenas (da35:mps2:0:17:0): SCSI status: Check Condition
Jun 16 06:56:13 freenas (da35:mps2:0:17:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun 16 06:56:13 freenas (da35:mps2:0:17:0): Retrying command (per sense data)
Jun 16 06:56:13 freenas (da34:mps2:0:16:0): READ(10). CDB: 28 00 66 88 b1 b0 00 00 20 00
Jun 16 06:56:13 freenas (da34:mps2:0:16:0): CAM status: SCSI Status Error
Jun 16 06:56:13 freenas (da34:mps2:0:16:0): SCSI status: Check Condition
Jun 16 06:56:13 freenas (da34:mps2:0:16:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun 16 06:56:13 freenas (da34:mps2:0:16:0): Retrying command (per sense data)
Jun 16 06:56:13 freenas (da33:mps2:0:15:0): READ(10). CDB: 28 00 66 88 b1 b0 00 00 20 00
Jun 16 06:56:13 freenas (da33:mps2:0:15:0): CAM status: SCSI Status Error
Jun 16 06:56:13 freenas (da33:mps2:0:15:0): SCSI status: Check Condition
Jun 16 06:56:13 freenas (da33:mps2:0:15:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun 16 06:56:13 freenas (da33:mps2:0:15:0): Retrying command (per sense data)
Jun 16 06:56:13 freenas (da34:mps2:0:16:0): READ(10). CDB: 28 00 66 88 b1 d8 00 00 20 00
Jun 16 06:56:13 freenas (da34:mps2:0:16:0): CAM status: SCSI Status Error
Jun 16 06:56:13 freenas (da34:mps2:0:16:0): SCSI status: Check Condition
Jun 16 06:56:13 freenas (da34:mps2:0:16:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun 16 06:56:13 freenas (da34:mps2:0:16:0): Retrying command (per sense data)
Jun 16 06:56:13 freenas (da33:mps2:0:15:0): READ(10). CDB: 28 00 66 88 b1 d8 00 00 20 00
Jun 16 06:56:13 freenas (da33:mps2:0:15:0): CAM status: SCSI Status Error
snip
Jun 16 06:56:13 freenas (da40:mps2:0:22:0): CAM status: SCSI Status Error
Jun 16 06:56:13 freenas (da40:mps2:0:22:0): SCSI status: Check Condition
Jun 16 06:56:13 freenas (da40:mps2:0:22:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun 16 06:56:13 freenas (da40:mps2:0:22:0): Retrying command (per sense data)
Jun 16 06:56:13 freenas (da40:mps2:0:22:0): READ(10). CDB: 28 00 66 88 b1 b0 00 00 20 00
Jun 16 06:56:13 freenas (da40:mps2:0:22:0): CAM status: SCSI Status Error
Jun 16 06:56:13 freenas (da40:mps2:0:22:0): SCSI status: Check Condition
snip
Jun 16 06:56:13 freenas  (da44:mps2:0:27:0): READ(10). CDB: 28 00 34 29 37 98 00 00 20 00 length 16384 SMID 345 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da45:mps2:0:28:0): READ(10). CDB: 28 00 34 29 37 58 00 00 20 00 length 16384 SMID 225 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da45:mps2:0:28:0): READ(10). CDB: 28 00 34 29 37 78 00 00 20 00 length 16384 SMID 184 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da45:mps2:0:28:0): READ(10). CDB: 28 00 34 29 37 98 00 00 20 00 length 16384 SMID 778 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da46:mps2:0:29:0): READ(10). CDB: 28 00 34 29 37 58 00 00 20 00 length 16384 SMID 134 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da46:mps2:0:29:0): READ(10). CDB: 28 00 34 29 37 78 00 00 20 00 length 16384 SMID 450 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da46:mps2:0:29:0): READ(10). CDB: 28 00 34 29 37 98 00 00 20 00 length 16384 SMID 201 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da47:mps2:0:30:0): READ(10). CDB: 28 00 34 29 37 58 00 00 20 00 length 16384 SMID 578 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da47:mps2:0:30:0): READ(10). CDB: 28 00 34 29 37 78 00 00 20 00 length 16384 SMID 755 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da47:mps2:0:30:0): READ(10). CDB: 28 00 34 29 37 98 00 00 20 00 length 16384 SMID 994 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da49:mps2:0:32:0): READ(10). CDB: 28 00 34 29 37 78 00 00 20 00 length 16384 SMID 174 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da49:mps2:0:32:0): READ(10). CDB: 28 00 34 29 37 98 00 00 20 00 length 16384 SMID 961 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas  (da49:mps2:0:32:0): READ(10). CDB: 28 00 34 29 37 b8 00 00 20 00 length 16384 SMID 446 terminated ioc 804b scsi 0 state c xfer 0
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): READ(10). CDB: 28 00 51 b0 ed 80 00 00 20 00
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): CAM status: SCSI Status Error
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): SCSI status: Check Condition
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): Retrying command (per sense data)
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): READ(10). CDB: 28 00 51 b0 ed a0 00 00 20 00
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): CAM status: SCSI Status Error
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): SCSI status: Check Condition
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): Retrying command (per sense data)
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): READ(10). CDB: 28 00 51 b0 ed c0 00 00 20 00
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): CAM status: SCSI Status Error
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): SCSI status: Check Condition
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun 16 06:56:13 freenas (da30:mps2:0:12:0): Retrying command (per sense data)
Jun 16 06:56:13 freenas (da36:mps2:0:18:0): CAM status: SCSI Status Error
Jun 16 06:56:13 freenas (da36:mps2:0:18:0): SCSI status: Check Condition
Jun 16 06:56:13 freenas (da36:mps2:0:18:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun 16 06:56:13 freenas (da36:mps2:0:18:0): Retrying command (per sense data)
Jun 16 06:56:13 freenas (da36:mps2:0:18:0): READ(10). CDB: 28 00 66 88 b1 d0 00 00 20 00
Jun 16 06:56:13 freenas (da36:mps2:0:18:0): CAM status: SCSI Status Error
Jun 16 06:56:13 freenas (da36:mps2:0:18:0): SCSI status: Check Condition
Jun 16 06:56:13 freenas (da36:mps2:0:18:0): SCSI sense: NOT READY asc:4,0 (Logical unit not ready, cause not reportable)
Jun 16 06:56:13 freenas (da36:mps2:0:18:0): Retrying command (per sense data)
Jun 16 06:56:14 freenas  (da30:mps2:0:12:0): READ(10). CDB: 28 00 51 b0 ed 80 00 00 20 00 length 16384 SMID 617 terminated ioc 804b scsi 0 state 0 xfer 0
Jun 16 06:56:14 freenas  (da30:mps2:0:12:0): READ(10). CDB: 28 00 51 b0 ed a0 00 00 20 00 length 16384 SMID 695 terminated ioc 804b scsi 0 state 0 xfer 0
Jun 16 06:56:14 freenas  (da30:mps2:0:12:0): READ(10). CDB: 28 00 51 b0 ed c0 00 00 20 00 length 16384 SMID 388 terminated ioc 804b scsi 0 state 0 xfer 0
Jun 16 06:56:14 freenas  (da36:mps2:0:18:0): READ(10). CDB: 28 00 66 88 b1 d0 00 00 20 00 length 16384 SMID 266 terminated ioc 804b scsi 0 state 0 xfer 0
Jun 16 06:56:14 freenas  (da37:mps2:0:19:0): READ(10). CDB: 28 00 66 88 b1 f0 00 00 20 00 length 16384 SMID 671 terminated ioc 804b scsi 0 state 0 xfer 0
Jun 16 06:56:14 freenas  (da37:mps2:0:19:0): READ(10). CDB: 28 00 66 88 b1 b0 00 00 20 00 length 16384 SMID 686 terminated ioc 804b scsi 0 state 0 xfer 0
Jun 16 06:56:14 freenas  (da37:mps2:0:19:0): READ(10). CDB: 28 00 66 88 b1 d0 00 00 20 00 length 16384 SMID 236 terminated ioc 804b scsi 0 state 0 xfer 0
snip
Jun 16 06:56:14 freenas (da28:mps2:0:10:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jun 16 06:56:14 freenas (da28:mps2:0:10:0): Retrying command (per sense data)
Jun 16 06:56:14 freenas (da29:mps2:0:11:0): READ(10). CDB: 28 00 51 b0 ed a0 00 00 20 00
Jun 16 06:56:14 freenas (da29:mps2:0:11:0): CAM status: SCSI Status Error
Jun 16 06:56:14 freenas (da29:mps2:0:11:0): SCSI status: Check Condition
Jun 16 06:56:14 freenas (da29:mps2:0:11:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jun 16 06:56:14 freenas (da29:mps2:0:11:0): Retrying command (per sense data)
Jun 16 06:56:15 freenas (da30:mps2:0:12:0): READ(10). CDB: 28 00 51 b0 ed 80 00 00 20 00
Jun 16 06:56:15 freenas (da30:mps2:0:12:0): CAM status: SCSI Status Error
Jun 16 06:56:15 freenas (da30:mps2:0:12:0): SCSI status: Check Condition
Jun 16 06:56:15 freenas (da30:mps2:0:12:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jun 16 06:56:15 freenas (da30:mps2:0:12:0): Retrying command (per sense data)
Jun 16 06:56:15 freenas (da31:mps2:0:13:0): READ(10). CDB: 28 00 51 b0 ed 60 00 00 20 00
Jun 16 06:56:15 freenas (da31:mps2:0:13:0): CAM status: SCSI Status Error
Jun 16 06:56:15 freenas (da31:mps2:0:13:0): SCSI status: Check Condition
Jun 16 06:56:15 freenas (da31:mps2:0:13:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jun 16 06:56:15 freenas (da31:mps2:0:13:0): Retrying command (per sense data)
Jun 16 06:56:15 freenas (da33:mps2:0:15:0): READ(10). CDB: 28 00 66 88 b1 f8 00 00 20 00
Jun 16 06:56:15 freenas (da33:mps2:0:15:0): CAM status: SCSI Status Error
Jun 16 06:56:15 freenas (da33:mps2:0:15:0): SCSI status: Check Condition
Jun 16 06:56:15 freenas (da33:mps2:0:15:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jun 16 06:56:15 freenas (da33:mps2:0:15:0): Retrying command (per sense data)
Jun 16 06:56:15 freenas (da34:mps2:0:16:0): READ(10). CDB: 28 00 66 88 b1 f8 00 00 20 00
Jun 16 06:56:15 freenas (da34:mps2:0:16:0): CAM status: SCSI Status Error
Jun 16 06:56:15 freenas (da34:mps2:0:16:0): SCSI status: Check Condition
Jun 16 06:56:15 freenas (da34:mps2:0:16:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jun 16 06:56:15 freenas (da34:mps2:0:16:0): Retrying command (per sense data)
Jun 16 06:56:15 freenas (da35:mps2:0:17:0): READ(10). CDB: 28 00 66 88 b1 f8 00 00 20 00
Jun 16 06:56:15 freenas (da35:mps2:0:17:0): CAM status: SCSI Status Error
Jun 16 06:56:15 freenas (da35:mps2:0:17:0): SCSI status: Check Condition
Jun 16 06:56:15 freenas (da35:mps2:0:17:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jun 16 06:56:15 freenas (da35:mps2:0:17:0): Retrying command (per sense data)
Jun 16 06:56:15 freenas (da36:mps2:0:18:0): READ(10). CDB: 28 00 66 88 b1 f8 00 00 20 00
Jun 16 06:56:15 freenas (da36:mps2:0:18:0): CAM status: SCSI Status Error
Jun 16 06:56:15 freenas (da36:mps2:0:18:0): SCSI status: Check Condition
Jun 16 06:56:15 freenas (da36:mps2:0:18:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jun 16 06:56:15 freenas (da36:mps2:0:18:0): Retrying command (per sense data)
Jun 16 06:56:15 freenas (da37:mps2:0:19:0): READ(10). CDB: 28 00 66 88 b1 f0 00 00 20 00
Jun 16 06:56:15 freenas (da37:mps2:0:19:0): CAM status: SCSI Status Error
Jun 16 06:56:15 freenas (da37:mps2:0:19:0): SCSI status: Check Condition
Jun 16 06:56:15 freenas (da37:mps2:0:19:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jun 16 06:56:15 freenas (da37:mps2:0:19:0): Retrying command (per sense data)
Jun 16 06:56:15 freenas (da42:mps2:0:25:0): READ(10). CDB: 28 00 34 29 37 c0 00 00 20 00
Jun 16 06:56:15 freenas (da42:mps2:0:25:0): CAM status: CCB request aborted by the host
Jun 16 06:56:15 freenas (da42:mps2:0:25:0): Retrying command
Jun 16 06:56:15 freenas zfsd: Event exceeds event size limit of 8192 bytes.
Jun 16 06:56:15 freenas zfsd: Truncated 7 characters from event.
Jun 16 06:56:15 freenas (da42:mps2:0:25:0): READ(10). CDB: 28 00 34 29 37 78 00 00 20 00
Jun 16 06:56:15 freenas (da42:mps2:0:25:0): CAM status: CCB request aborted by the host
Jun 16 06:56:15 freenas (da42:mps2:0:25:0): Retrying command
Jun 16 06:56:15 freenas (da42:mps2:0:25:0): READ(10). CDB: 28 00 34 29 37 58 00 00 20 00
Jun 16 06:56:15 freenas (da42:mps2:0:25:0): CAM status: CCB request aborted by the host
Jun 16 06:56:15 freenas (da42:mps2:0:25:0): Retrying command
Jun 16 06:56:15 freenas (da43:mps2:0:26:0): READ(10). CDB: 28 00 34 29 37 c0 00 00 20 00
Jun 16 06:56:15 freenas (da43:mps2:0:26:0): CAM status: CCB request aborted by the host
Jun 16 06:56:15 freenas (da43:mps2:0:26:0): Retrying command
Jun 16 06:56:15 freenas (da43:mps2:0:26:0): READ(10). CDB: 28 00 34 29 37 78 00 00 20 00
Jun 16 06:56:15 freenas (da43:mps2:0:26:0): CAM status: CCB request aborted by the host
Jun 16 06:56:15 freenas (da43:mps2:0:26:0): Retrying command
Jun 16 06:56:15 freenas (da43:mps2:0:26:0): READ(10). CDB: 28 00 34 29 37 58 00 00 20 00
Jun 16 06:56:15 freenas (da43:mps2:0:26:0): CAM status: CCB request aborted by the host
Jun 16 06:56:15 freenas (da43:mps2:0:26:0): Retrying command
Jun 16 06:56:15 freenas (da44:mps2:0:27:0): READ(10). CDB: 28 00 34 29 37 98 00 00 20 00
Jun 16 06:56:15 freenas (da44:mps2:0:27:0): CAM status: CCB request aborted by the host
Jun 16 06:56:15 freenas (da44:mps2:0:27:0): Retrying command
Jun 16 06:56:15 freenas (da44:mps2:0:27:0): READ(10). CDB: 28 00 34 29 37 78 00 00 20 00
Jun 16 06:56:15 freenas (da44:mps2:0:27:0): CAM status: CCB request aborted by the host
Jun 16 06:56:15 freenas (da44:mps2:0:27:0): Retrying command
Jun 16 06:56:15 freenas (da44:mps2:0:27:0): READ(10). CDB: 28 00 34 29 37 58 00 00 20 00
Jun 16 06:56:15 freenas (da44:mps2:0:27:0): CAM status: CCB request aborted by the host
Jun 16 06:56:15 freenas (da44:mps2:0:27:0): Retrying command
Jun 16 06:56:15 freenas (da45:mps2:0:28:0): READ(10). CDB: 28 00 34 29 37 98 00 00 20 00
Jun 16 06:56:15 freenas (da45:mps2:0:28:0): CAM status: CCB request aborted by the host
Jun 16 06:56:15 freenas (da45:mps2:0:28:0): Retrying command
Jun 16 06:56:15 freenas (da45:mps2:0:28:0): READ(10). CDB: 28 00 34 29 37 78 00 00 20 00
Jun 16 06:56:15 freenas (da45:mps2:0:28:0): CAM status: CCB request aborted by the host
Jun 16 06:56:15 freenas (da45:mps2:0:28:0): Retrying command
Jun 16 06:56:15 freenas (da45:mps2:0:28:0): READ(10). CDB: 28 00 34 29 37 58 00 00 20 00
Jun 16 06:56:15 freenas (da45:mps2:0:28:0): CAM status: CCB request aborted by the host
Jun 16 06:56:15 freenas (da45:mps2:0:28:0): Retrying command
Jun 16 06:56:15 freenas (da46:mps2:0:29:0): READ(10). CDB: 28 00 34 29 37 98 00 00 20 00
Jun 16 06:56:15 freenas (da46:mps2:0:29:0): CAM status: CCB request aborted by the host
snip
Jun 16 06:56:15 freenas da42 at mps2 bus 0 scbus2 target 25 lun 0
Jun 16 06:56:15 freenas da42: <ATA ST4000DM000-1F21 CC51> s/n  Z3004TYS detached
Jun 16 06:56:15 freenas da43 at mps2 bus 0 scbus2 target 26 lun 0
Jun 16 06:56:15 freenas da43: <ATA ST4000DM000-1F21 CC52> s/n  Z301N0RN detached
Jun 16 06:56:15 freenas da44 at mps2 bus 0 scbus2 target 27 lun 0
Jun 16 06:56:15 freenas da44: <ATA ST4000DM000-1F21 CC52> s/n  Z301N06S detached
Jun 16 06:56:15 freenas da45 at mps2 bus 0 scbus2 target 28 lun 0
Jun 16 06:56:15 freenas da45: <ATA ST4000DM000-1F21 CC52> s/n  W300SLZ2 detached
Jun 16 06:56:15 freenas da46 at mps2 bus 0 scbus2 target 29 lun 0
Jun 16 06:56:15 freenas da46: <ATA ST4000DM000-1F21 CC52> s/n  W300STS6 detached
Jun 16 06:56:15 freenas da47 at mps2 bus 0 scbus2 target 30 lun 0
Jun 16 06:56:15 freenas da47: <ATA ST4000DM000-1F21 CC52> s/n  W300STBJ detached
Jun 16 06:56:15 freenas da48 at mps2 bus 0 scbus2 target 31 lun 0
Jun 16 06:56:15 freenas da48: <ATA ST4000DM000-1F21 CC54> s/n  S300HZMR detached
Jun 16 06:56:15 freenas da49 at mps2 bus 0 scbus2 target 32 lun 0
Jun 16 06:56:15 freenas da49: <ATA ST4000DM000-1F21 CC54> s/n  S300HZNM detached
Jun 16 06:56:15 freenas GEOM_ELI: Device da42p1.eli destroyed.
Jun 16 06:56:15 freenas GEOM_ELI: Detached da42p1.eli on last close.
Jun 16 06:56:15 freenas GEOM_ELI: Device da43p1.eli destroyed.
Jun 16 06:56:15 freenas GEOM_ELI: Detached da43p1.eli on last close.
Jun 16 06:56:15 freenas GEOM_ELI: Device da44p1.eli destroyed.
Jun 16 06:56:15 freenas GEOM_ELI: Detached da44p1.eli on last close.
Jun 16 06:56:15 freenas GEOM_ELI: Device da45p1.eli destroyed.
Jun 16 06:56:15 freenas GEOM_ELI: Detached da45p1.eli on last close.
Jun 16 06:56:15 freenas GEOM_ELI: Device da46p1.eli destroyed.
Jun 16 06:56:15 freenas GEOM_ELI: Detached da46p1.eli on last close.
Jun 16 06:56:15 freenas GEOM_ELI: Device da47p1.eli destroyed.
Jun 16 06:56:15 freenas GEOM_ELI: Detached da47p1.eli on last close.
Jun 16 06:56:15 freenas GEOM_ELI: Device da48p1.eli destroyed.
Jun 16 06:56:15 freenas GEOM_ELI: Detached da48p1.eli on last close.
Jun 16 06:56:15 freenas GEOM_ELI: Device da49p1.eli destroyed.
Jun 16 06:56:15 freenas GEOM_ELI: Detached da49p1.eli on last close.
Jun 16 06:56:16 freenas (da38:mps2:0:33:0): READ(10). CDB: 28 00 66 88 b1 f0 00 00 20 00
Jun 16 06:56:16 freenas (da38:mps2:0:33:0): CAM status: SCSI Status Error
Jun 16 06:56:16 freenas (da38:mps2:0:33:0): SCSI status: Check Condition
Jun 16 06:56:16 freenas (da38:mps2:0:33:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jun 16 06:56:16 freenas (da38:mps2:0:33:0): Retrying command (per sense data)
Jun 16 06:56:16 freenas (da39:mps2:0:21:0): READ(10). CDB: 28 00 66 88 b1 f0 00 00 20 00
Jun 16 06:56:16 freenas (da39:mps2:0:21:0): CAM status: SCSI Status Error
Jun 16 06:56:16 freenas (da39:mps2:0:21:0): SCSI status: Check Condition
Jun 16 06:56:16 freenas (da39:mps2:0:21:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jun 16 06:56:16 freenas (da39:mps2:0:21:0): Retrying command (per sense data)
Jun 16 06:56:16 freenas (da40:mps2:0:22:0): READ(10). CDB: 28 00 66 88 b1 f0 00 00 20 00
Jun 16 06:56:16 freenas (da40:mps2:0:22:0): CAM status: SCSI Status Error
Jun 16 06:56:16 freenas (da40:mps2:0:22:0): SCSI status: Check Condition
Jun 16 06:56:16 freenas (da40:mps2:0:22:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jun 16 06:56:16 freenas (da40:mps2:0:22:0): Retrying command (per sense data)
Jun 16 06:56:16 freenas (da41:mps2:0:23:0): READ(10). CDB: 28 00 66 88 b1 f0 00 00 20 00
Jun 16 06:56:16 freenas (da41:mps2:0:23:0): CAM status: SCSI Status Error
Jun 16 06:56:16 freenas (da41:mps2:0:23:0): SCSI status: Check Condition
Jun 16 06:56:16 freenas (da41:mps2:0:23:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Jun 16 06:56:16 freenas (da41:mps2:0:23:0): Retrying command (per sense data)
Jun 16 06:56:16 freenas (da42:mps2:0:25:0): Periph destroyed
Jun 16 06:56:16 freenas (da49:mps2:0:32:0): Periph destroyed
Jun 16 06:56:16 freenas (da48:mps2:0:31:0): Periph destroyed
Jun 16 06:56:16 freenas (da47:mps2:0:30:0): Periph destroyed
Jun 16 06:56:16 freenas (da46:mps2:0:29:0): Periph destroyed
Jun 16 06:56:16 freenas (da45:mps2:0:28:0): Periph destroyed
Jun 16 06:56:16 freenas (da44:mps2:0:27:0): Periph destroyed
Jun 16 06:56:16 freenas (da43:mps2:0:26:0): Periph destroyed
Jun 16 06:56:17 freenas da32 at mps2 bus 0 scbus2 target 27 lun 0
Jun 16 06:56:17 freenas da32: <ATA ST4000DM000-1F21 CC52> Fixed Direct Access SCSI-6 device
Jun 16 06:56:17 freenas da32: Serial Number  Z301N06S
Jun 16 06:56:17 freenas da32: 600.000MB/s transfers
Jun 16 06:56:17 freenas da32: Command Queueing enabled
Jun 16 06:56:17 freenas da32: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
Jun 16 06:56:17 freenas da32: quirks=0x8<4K>
Jun 16 06:56:17 freenas da42 at mps2 bus 0 scbus2 target 28 lun 0
Jun 16 06:56:17 freenas da42: <ATA ST4000DM000-1F21 CC52> Fixed Direct Access SCSI-6 device
Jun 16 06:56:17 freenas da42: Serial Number  W300SLZ2
Jun 16 06:56:17 freenas da42: 600.000MB/s transfers
Jun 16 06:56:17 freenas da42: Command Queueing enabled
Jun 16 06:56:17 freenas da42: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
Jun 16 06:56:17 freenas da42: quirks=0x8<4K>
Jun 16 06:56:17 freenas da43 at mps2 bus 0 scbus2 target 30 lun 0
Jun 16 06:56:17 freenas da43: <ATA ST4000DM000-1F21 CC52> Fixed Direct Access SCSI-6 device
Jun 16 06:56:17 freenas da43: Serial Number  W300STBJ
Jun 16 06:56:17 freenas da43: 600.000MB/s transfers
Jun 16 06:56:17 freenas da43: Command Queueing enabled
Jun 16 06:56:17 freenas da43: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
Jun 16 06:56:17 freenas da43: quirks=0x8<4K>
Jun 16 06:56:17 freenas da44 at mps2 bus 0 scbus2 target 26 lun 0
Jun 16 06:56:17 freenas da44: <ATA ST4000DM000-1F21 CC52> Fixed Direct Access SCSI-6 device
Jun 16 06:56:17 freenas da44: Serial Number  Z301N0RN
Jun 16 06:56:17 freenas da44: 600.000MB/s transfers
Jun 16 06:56:17 freenas da44: Command Queueing enabled
Jun 16 06:56:17 freenas da44: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
Jun 16 06:56:17 freenas da44: quirks=0x8<4K>
Jun 16 06:56:17 freenas da45 at mps2 bus 0 scbus2 target 29 lun 0
Jun 16 06:56:17 freenas da45: <ATA ST4000DM000-1F21 CC52> Fixed Direct Access SCSI-6 device
Jun 16 06:56:17 freenas da45: Serial Number  W300STS6
Jun 16 06:56:17 freenas da45: 600.000MB/s transfers
Jun 16 06:56:17 freenas da45: Command Queueing enabled
Jun 16 06:56:17 freenas da45: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
Jun 16 06:56:17 freenas da45: quirks=0x8<4K>
Jun 16 06:56:17 freenas da46 at mps2 bus 0 scbus2 target 25 lun 0
Jun 16 06:56:17 freenas da46: <ATA ST4000DM000-1F21 CC51> Fixed Direct Access SCSI-6 device
Jun 16 06:56:17 freenas da46: Serial Number  Z3004TYS
Jun 16 06:56:17 freenas da46: 600.000MB/s transfers
Jun 16 06:56:17 freenas da46: Command Queueing enabled
Jun 16 06:56:17 freenas da46: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
Jun 16 06:56:17 freenas da46: quirks=0x8<4K>
Jun 16 06:56:17 freenas da47 at mps2 bus 0 scbus2 target 32 lun 0
Jun 16 06:56:17 freenas da47: <ATA ST4000DM000-1F21 CC54> Fixed Direct Access SCSI-6 device
Jun 16 06:56:17 freenas da47: Serial Number  S300HZNM
Jun 16 06:56:17 freenas da47: 600.000MB/s transfers
Jun 16 06:56:17 freenas da47: Command Queueing enabled
Jun 16 06:56:17 freenas da47: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
Jun 16 06:56:17 freenas da47: quirks=0x8<4K>
Jun 16 06:56:17 freenas da48 at mps2 bus 0 scbus2 target 31 lun 0
Jun 16 06:56:17 freenas da48: <ATA ST4000DM000-1F21 CC54> Fixed Direct Access SCSI-6 device
Jun 16 06:56:17 freenas da48: Serial Number  S300HZMR
Jun 16 06:56:17 freenas da48: 600.000MB/s transfers
Jun 16 06:56:17 freenas da48: Command Queueing enabled
Jun 16 06:56:17 freenas da48: 3815447MB (7814037168 512 byte sectors: 255H 63S/T 486401C)
Jun 16 06:56:17 freenas da48: quirks=0x8<4K>
Jun 16 06:56:27 freenas da49 at mps2 bus 0 scbus2 target 34 lun 0
Jun 16 06:56:27 freenas da49: <ATA Hitachi HUA72302 A840> Fixed Direct Access SCSI-6 device
Jun 16 06:56:27 freenas da49: Serial Number YGGNZ0GD
Jun 16 06:56:27 freenas da49: 600.000MB/s transfers
Jun 16 06:56:27 freenas da49: Command Queueing enabled
Jun 16 06:56:27 freenas da49: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
Jun 16 06:56:46 freenas zfsd: Replace vdev(v1/10367877350599139386) by physical path: Unable to allocate spare target data.
Jun 16 06:56:46 freenas zfsd: Replace vdev(v1/7245229600930455534) by physical path: Unable to allocate spare target data.
Jun 16 06:56:46 freenas zfsd: Replace vdev(v1/11159745782969864676) by physical path: Unable to allocate spare target data.
Jun 16 06:56:46 freenas zfsd: Replace vdev(v1/10288687537002700716) by physical path: Unable to allocate spare target data.
Jun 16 06:56:46 freenas zfsd: Replace vdev(v1/7112856598500417308) by physical path: Unable to allocate spare target data.
Jun 16 06:56:46 freenas zfsd: Replace vdev(v1/13565106232670091350) by physical path: Unable to allocate spare target data.
Jun 16 06:56:46 freenas zfsd: Replace vdev(v1/2083712239136955752) by physical path: Unable to allocate spare target data.
Jun 16 06:56:46 freenas zfsd: Replace vdev(v1/8599137421765933395) by physical path: Unable to allocate spare target data.
[root@freenas] ~/scripts#

Fortunately everything was fine again within 30 seconds and I was able to ONLINE the new disk and the array was back in a normal state.

No idea what cause the fault lights to light up in that 2nd chassis and why zfs dropped all the drives and rediscovered them.

May when I do the next one I'll power down the whole system instead. That was quire a scare but I was impressed how quickly the system recovered.

 

[pclausen]

It's nice to finally be down to only single digit sector allocation errors on a few drives:

 

[Bidule0hm]

Only this time, when I inserted the new disk into the bay previously occupied by the old disk, all the red fault lights came on in that chassis, and then ZFS started rediscovering all 50 disks!

Exactly why I hate hot-swapping :D

I see that some drives are (very) old and with this number of drives you'll probably have more failures relatively soon, I recommend to have more tested spares on the shelf than what you currently need to replace the failed drives ;)

 

[pclausen]

Yeah, after the experience this morning, I tend to agree! I've never had it happen to me before hot swapping with these SuperMicro 846 enclosures, and I've used them for 7 years now, but then I have never had 3 24 port expanders on different HBAs all running on a single motherboard before. I'm going to double check that I did indeed flash the firmware on all 3 expanders to the same version.

Old? You think 5.3 years of power on years is old? I hope to get at least another 5 years out of all these drives. You should see the power on hour counts on all my 1TB drives that I decided not to implement in this volume. :D

In all seriousness, I do plan to pick up some more of those $50 2TB Hitachi drives. They were brand new btw according to the SMART power on hours and start/stop count.

My long term strategy is to begin swapping 2TB drives with 4TB Reds one VDev at a time, starting with the ones with the oldest drives. When I get close to having replaced all 10 drives, I'll probably go ahead and do the last few so that I can expand that particular VDev to twice its original capacity.

Hopefully by the time I get done with the first VDevs, the 4TB Reds will be selling for $50 new. :D

 

[CheckYourSix]

Yeah, after the experience this morning, I tend to agree! I've never had it happen to me before hot swapping with these SuperMicro 846 enclosures, and I've used them for 7 years now, but then I have never had 3 24 port expanders on different HBAs all running on a single motherboard before. I'm going to double check that I did indeed flash the firmware on all 3 expanders to the same version.

Old? You think 5.3 years of power on years is old? I hope to get at least another 5 years out of all these drives. You should see the power on hour counts on all my 1TB drives that I decided not to implement in this volume. :D

In all seriousness, I do plan to pick up some more of those $50 2TB Hitachi drives. They were brand new btw according to the SMART power on hours and start/stop count.

My long term strategy is to begin swapping 2TB drives with 4TB Reds one VDev at a time, starting with the ones with the oldest drives. When I get close to having replaced all 10 drives, I'll probably go ahead and do the last few so that I can expand that particular VDev to twice its original capacity.

Hopefully by the time I get done with the first VDevs, the 4TB Reds will be selling for $50 new. :D

Just an FYI, I was looking at drives 2-3 days ago. Cost per GB on 4TB Reds was like $0.0399 and for 6 TB it was like $0.041. The difference in price is insignificant considering you would get ~50% more space over the 4TB models and not have to replace for that much longer.

On another note, what $50 Hitachi drives? I need to get a drive or 2 for my space PC.

 

[pclausen]

Good point on the 4TB vs 6TB Reds. Of course it means it will cost me 50% more to upgrade each vdev. But looking long term, yes it definitely makes the most sense.

On another note, what $50 Hitachi drives? I need to get a drive or 2 for my space PC.

These

 

[Bidule0hm]

It's not that old (I have a IBM drive from the 90s which still works fine...) but 5 years is generally the lifetime you can hope for recent drives.

 

[pclausen]

So I had another drive that had SMART error counts that started to increase, so following the exact same procedure as before, but when I went to ONLINE the replacement disk, I got the following error:

Code:

Jun 26 12:10:01 freenas notifier: swapoff: /dev/da28p1.eli: No such file or directory
Jun 26 12:10:01 freenas notifier: geli: No such device: /dev/da28p1.
Jun 26 12:10:01 freenas notifier: 1+0 records in
Jun 26 12:10:01 freenas notifier: 1+0 records out
Jun 26 12:10:01 freenas notifier: 1048576 bytes transferred in 0.471564 secs (2223613 bytes/sec)
Jun 26 12:10:01 freenas notifier: dd: /dev/da28: short write on character device
Jun 26 12:10:01 freenas notifier: dd: /dev/da28: end of device
Jun 26 12:10:01 freenas notifier: 5+0 records in
Jun 26 12:10:01 freenas notifier: 4+1 records out
Jun 26 12:10:01 freenas notifier: 4284416 bytes transferred in 0.012649 secs (338714200 bytes/sec)
Jun 26 12:10:06 freenas notifier: swapoff: /dev/da28p1.eli: No such file or directory
Jun 26 12:10:06 freenas notifier: geli: No such device: /dev/da28p1.
Jun 26 12:10:06 freenas manage.py: [middleware.exceptions:38] [MiddlewareError: Disk replacement failed: "invalid vdev specification, use '-f' to override the following errors:, /dev/gptid/cdba2e5c-1c1d-11e5-8714-0cc47a3311b4 is part of active pool 'v1', "]

Any ideas?

Here's the action I'm trying to perform that throws the above error:

Should I reboot the server and try again?

Resilver itself completed with no errors. In fact, it was flying this time:

Code:

 pool: v1
state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
  continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Jun 26 06:22:25 2015
  17.0T scanned out of 58.4T at 3.82G/s, 3h5m to go
  208G resilvered, 29.10% done

 

[maglin]

Should I reboot the server and try again?

Did you manage to figure it out? I love your set-up as well as your power solution. I wish I had 3 phase 440 in my house.

 

[pclausen]

Yep I'm all good. I ended up shutting down the server and replacing the new drive with another one and then doing another resilver.

I won't be doing any more hot swapping and should have listened to BiduleOhm the first time around. :)

I don't have 3 phase, just 400A 220V service. :)

 

[depasseg]

Do the drives in the enclosures that are just JBOD's light up only during activity, or when there is power? In mine, the drives in my servers have their lights on normally and blink during activity, and the enclosure operates differently (lights are normally off, and blink with activity). I'm trying to figure out if I wired something wrong.

 

[Ericloewe]

Do the drives in the enclosures that are just JBOD's light up only during activity, or when there is power? In mine, the drives in my servers have their lights on normally and blink during activity, and the enclosure operates differently (lights are normally off, and blink with activity). I'm trying to figure out if I wired something wrong.

Different firmware versions, maybe? Drive LEDs should be controlled via SAS' sideband stuff anyway, so that sounds really weird.

 

[pclausen]

My enclosures all behave the same. That is, the LEDs only light during disk activity. My understanding is that this is normal with SATA drives. For SAS drives, the blue LEDs are always LIT when the drives are powered and blink during disk activity.

Speaking of SuperMicro LEDs, I really wish my red LEDs would light up when there's a fault detected with a given drive. My understanding is that this is implemented in TrueNAS for SuperMicro chassis. I wonder what the odds are that feature will trickle down into FreeNAS?

 

[depasseg]

My enclosures all behave the same. That is, the LEDs only light during disk activity. My understand is that this is normal with SATA drives. For SAS drives, the blue LEDs are always LIT when the drives are powered and blink during disk activity.

That could explain it. The server has all SAS drives and my enclosure is all SATA.

 

[Scampicfx]

I hope you removed all excessive standoffs. Common error it seems..

Wait a second... :D What do you mean with "excessive standoffs"?

I'm a bit worried right now, because I remember mounting a lot of mainboards with these 1cm tall mounting / standoff things... :D I thought the purpose of them is to establish a tiny distance between the solder joints on the backside of the mainboard and the chassis?

 

[jgreco]

Wait a second... :D What do you mean with "excessive standoffs"?

I'm a bit worried right now, because I remember mounting a lot of mainboards with these 1cm tall mounting / standoff things... :D I thought the purpose of them is to establish a tiny distance between the solder joints on the backside of the mainboard and the chassis?

Sure, but they actually have to align with screw holes. If you install a board with a standoff that's instead making contact with traces or components on the board, you're shorting them to ground.