No, I'm waiting for the 4th scrub to complete (with out errors) on my 2nd PSU (Only). I will then re-add drives to restore redundancy in the pool.You put all your drives back yet?
Jun 13 10:39:35 ultraman zfsd: CaseFile::Serialize: Unable to open /etc/zfs/cases/pool_16006326459371220184_vdev_17769119138064296099.case. Jun 13 10:39:35 ultraman (da11:isci0:0:33:0): WRITE(16). CDB: 8a 00 00 00 00 01 64 be e7 60 00 00 00 c0 00 00 Jun 13 10:39:35 ultraman (da11:isci0:0:33:0): CAM status: SCSI Status Error Jun 13 10:39:35 ultraman (da11:isci0:0:33:0): SCSI status: Check Condition Jun 13 10:39:35 ultraman (da11:isci0:0:33:0): SCSI sense: ILLEGAL REQUEST asc:ffffffff,ffffffff (Reserved ASC/ASCQ pair) Jun 13 10:39:35 ultraman (da11:isci0:0:33:0): Error 22, Unretryable error
Did you look into updating the backplanes firmware?
Jun 13 16:43:31 ultraman zfsd: CaseFile::Serialize: Unable to open /etc/zfs/cases/pool_16006326459371220184_vdev_13826095765754444936.case. Jun 13 16:43:31 ultraman (da18:isci0:0:37:0): WRITE(10). CDB: 2a 00 43 45 ac 60 00 01 00 00 Jun 13 16:43:31 ultraman (da18:isci0:0:37:0): CAM status: SCSI Status Error Jun 13 16:43:31 ultraman (da18:isci0:0:37:0): SCSI status: Check Condition Jun 13 16:43:31 ultraman (da18:isci0:0:37:0): SCSI sense: ILLEGAL REQUEST asc:ffffffff,ffffffff (Reserved ASC/ASCQ pair) ... ... Jun 14 01:47:52 ultraman zfsd: CaseFile::Serialize: Unable to open /etc/zfs/cases/pool_16006326459371220184_vdev_9506731245968758733.case. Jun 14 01:47:52 ultraman (da20:isci0:0:38:0): WRITE(10). CDB: 2a 00 a9 42 84 08 00 00 c0 00 Jun 14 01:47:52 ultraman (da20:isci0:0:38:0): CAM status: SCSI Status Error Jun 14 01:47:52 ultraman (da20:isci0:0:38:0): SCSI status: Check Condition Jun 14 01:47:52 ultraman (da20:isci0:0:38:0): SCSI sense: ILLEGAL REQUEST asc:ffffffff,ffffffff (Reserved ASC/ASCQ pair) Jun 14 01:47:52 ultraman (da20:isci0:0:38:0): Error 22, Unretryable error ... ...Jun 14 05:54:06 ultraman zfsd: CaseFile::Serialize: Unable to open /etc/zfs/cases/pool_16006326459371220184_vdev_15352642184540883744.case. Jun 14 05:54:06 ultraman zfsd: CaseFile::Serialize: Unable to open /etc/zfs/cases/pool_16006326459371220184_vdev_15352642184540883744.case. Jun 14 05:54:06 ultraman (da22:isci0:0:39:0): WRITE(10). CDB: 2a 00 06 c6 58 90 00 00 90 00 Jun 14 05:54:06 ultraman (da22:isci0:0:39:0): CAM status: SCSI Status Error Jun 14 05:54:06 ultraman (da22:isci0:0:39:0): SCSI status: Check Condition Jun 14 05:54:06 ultraman (da22:isci0:0:39:0): SCSI sense: ILLEGAL REQUEST asc:ffffffff,ffffffff (Reserved ASC/ASCQ pair) Jun 14 05:54:06 ultraman (da22:isci0:0:39:0): Error 22, Unretryable error
Jun 14 12:52:16 ultraman zfsd: CaseFile::Serialize: Unable to open /etc/zfs/cases/pool_16006326459371220184_vdev_7446801437149806016.case. Jun 14 12:52:16 ultraman (da4:isci0:0:30:0): WRITE(10). CDB: 2a 00 91 76 91 c8 00 01 00 00 Jun 14 12:52:16 ultraman (da4:isci0:0:30:0): CAM status: SCSI Status Error Jun 14 12:52:16 ultraman (da4:isci0:0:30:0): SCSI status: Check Condition Jun 14 12:52:16 ultraman (da4:isci0:0:30:0): SCSI sense: ILLEGAL REQUEST asc:ffffffff,ffffffff (Reserved ASC/ASCQ pair) Jun 14 12:52:16 ultraman (da4:isci0:0:30:0): Error 22, Unretryable error
Jun 22 16:46:50 ultraman (da6:mps0:0:17:0): READ(10). CDB: 28 00 80 08 86 80 00 01 00 00 length 131072 SMID 781 terminated ioc 804b scsi 0 state 0 xfer 0 Jun 22 16:46:50 ultraman (da6:mps0:0:17:0): READ(10). CDB: 28 00 80 08 86 80 00 01 00 00 Jun 22 16:46:50 ultraman (da6:mps0:0:17:0): CAM status: CCB request completed with an error Jun 22 16:46:50 ultraman (da6:mps0:0:17:0): Retrying command Jun 22 16:46:50 ultraman (da6:mps0:0:17:0): READ(10). CDB: 28 00 80 08 85 b8 00 00 c8 00 Jun 22 16:46:50 ultraman (da6:mps0:0:17:0): CAM status: SCSI Status Error Jun 22 16:46:50 ultraman (da6:mps0:0:17:0): SCSI status: Check Condition Jun 22 16:46:50 ultraman (da6:mps0:0:17:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error) Jun 22 16:46:50 ultraman (da6:mps0:0:17:0): Info: 0x800885b8 Jun 22 16:46:50 ultraman (da6:mps0:0:17:0): Error 5, Unretryable error
Have you made some calculations on how much power do you need from the PSUs for all your drives and components? The max power needed must not exceed 80% of the rated PSU power.
You can also try adding one drive at a time starting from the 12 config you did the tests without errors.
Can you post the make/model of PSUs?
Are you on 110V or a 220V grid?
Usually for something beyond 12 drives, you want to get a redundant power supply. The fact of the matter is that 12 drives are unlikely to all spin up at exactly the same time and actually consume all your power. You'll notice that in most rackmount designs that there are two power supplies, either of which are capable of holding the load, often just barely, but when teamed they are both just lazily feeding power.