Steven Sedory
Explorer
- Joined
- Apr 7, 2014
- Messages
- 96
Hi,
We're currently on FreeNAS-9.3-STABLE-201604041648, with dual E5-2620 V3, 128GB ECC RAM, Avago SAS 9300-8i on IT firmware version 9 and BIOS 8.21. We have 44x 4TB NSSAS drives, 15 300GB 15K SAS, and 3x Intel PCI 400GB SSD drives.
If need be, I can get more details on our setup.
Things have been fine for the past 8 months or so, but just last week, we started seeing errors like below:
filesan.domain.com kernel log messages:
> (da1:mpr0:0:9:0): Info: 0x400388
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 98 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaf98
> (da1:mpr0:0:9:0): Actual Retry Count: 19
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af a0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafa3
> (da1:mpr0:0:9:0): Actual Retry Count: 18
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 04 08 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400408
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 04 38 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400438
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafb0
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af c8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafc8
> (da1:mpr0:0:9:0): Actual Retry Count: 19
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 04 38 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400438
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 ba bb 70 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x18babb70
> (da1:mpr0:0:9:0): Actual Retry Count: 19
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 98 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400398
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 ba c2 38 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x18bac238
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 04 08 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x400408
> (da1:mpr0:0:9:0): Actual Retry Count: 18
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 98 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaf98
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 04 38 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400438
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 88 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaf88
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 1d ec 85 18 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x1dec8518
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 1d ec 85 70 00 00 10 00
> (da1:mpr0:0:9:0): Info: 0x1dec8570
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 1d ec 85 88 00 00 10 00
> (da1:mpr0:0:9:0): Info: 0x1dec8588
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 1d ec 85 98 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x1dec8598
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 04 58 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x400458
> (da1:mpr0:0:9:0): Actual Retry Count: 18
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 ba f9 10 00 01 00 00
> (da1:mpr0:0:9:0): Info: 0x18baf9b4
> (da1:mpr0:0:9:0): Actual Retry Count: 21
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 bb 00 b0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x18bb00b0
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 bb 06 b8 00 01 00 00
> (da1:mpr0:0:9:0): Info: 0x18bb0758
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 80 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400380
> (da1:mpr0:0:9:0): Actual Retry Count: 18
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 bb 14 f0 00 01 00 00
> (da1:mpr0:0:9:0): Info: 0x18bb15d0
> (da1:mpr0:0:9:0): Field Replaceable Unit: 8
> (da1:mpr0:0:9:0): Actual Retry Count: 18
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af c8 00 00 08 00
> (da1:mpr0:0:9:0): CAM status: SCSI Status Error
> (da1:mpr0:0:9:0): SCSI status: Check Condition
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecafc8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 0d 13 94 68 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0xd139468
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 04 18 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400418
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 b8 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x4003b8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafb9
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 c0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003c0
> (da1:mpr0:0:9:0): Actual Retry Count: 20
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af c0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafc0
> (da1:mpr0:0:9:0): Actual Retry Count: 19
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 d8 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x4003d8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 bb 38 10 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x18bb3810
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 bb 38 10 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x18bb3810
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 bb 38 18 00 01 00 00
> (da1:mpr0:0:9:0): Info: 0x18bb3818
> (da1:mpr0:0:9:0): Actual Retry Count: 5
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b0 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: HARDWARE FAILURE asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecafb0
> (da1:mpr0:0:9:0): Actual Retry Count: 23
> (da1:mpr0:0:9:0): Retrying command (per sense data)
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 d8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003d8
> (da1:mpr0:0:9:0): Actual Retry Count: 18
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 0d 3e 05 e8 00 00 58 00
> (da1:mpr0:0:9:0): Info: 0xd3e063f
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 e8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003e8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 b8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003b8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafb8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 bb 60 f0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x18bb60f0
> (da1:mpr0:0:9:0): Actual Retry Count: 18
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af f0 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecaff0
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 b0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003b0
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafb0
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 d8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003d8
> (da1:mpr0:0:9:0): Actual Retry Count: 18
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 bb 8a b8 00 00 10 00
> (da1:mpr0:0:9:0): Info: 0x18bb8ab8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 bb 8a b8 00 00 10 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x18bb8ab8
> (da1:mpr0:0:9:0): Actual Retry Count: 8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 88 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x400388
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 e8 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x4003e8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af e8 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecafe8
> (da1:mpr0:0:9:0): Actual Retry Count: 6
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af f8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaff8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 98 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaf99
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 c5 b2 70 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x18c5b270
> (da1:mpr0:0:9:0): Actual Retry Count: 18
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 c6 43 00 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x18c64300
> (da1:mpr0:0:9:0): Actual Retry Count: 18
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 c6 52 60 00 00 50 00
> (da1:mpr0:0:9:0): SCSI sense: HARDWARE FAILURE asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x18c652a1
> (da1:mpr0:0:9:0): Actual Retry Count: 23
> (da1:mpr0:0:9:0): Retrying command (per sense data)
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af a8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafa8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 c6 52 e8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x18c652e8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafb8
> (da1:mpr0:0:9:0): Actual Retry Count: 7
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 98 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaf9f
> (da1:mpr0:0:9:0): Actual Retry Count: 23
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af c8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafc8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af d8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafd8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 c8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003c8
> (da1:mpr0:0:9:0): Actual Retry Count: 20
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 e8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003e8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 c7 b6 90 00 00 40 00
> (da1:mpr0:0:9:0): Info: 0x18c7b690
> (da1:mpr0:0:9:0): Actual Retry Count: 19
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 88 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400388
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 c8 f6 d8 00 00 b0 00
> (da1:mpr0:0:9:0): Info: 0x18c8f72a
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 c8 ff 48 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x18c8ff48
> (da1:mpr0:0:9:0): Actual Retry Count: 18
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 90 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaf90
> (da1:mpr0:0:9:0): Field Replaceable Unit: 8
> (da1:mpr0:0:9:0): Actual Retry Count: 21
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af a8 00 00 08 00
> (da1:mpr0:0:9:0): CAM status: SCSI Status Error
> (da1:mpr0:0:9:0): SCSI status: Check Condition
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecafa8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafb8
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 98 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400398
> (da1:mpr0:0:9:0): Actual Retry Count: 4
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 98 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaf98
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 88 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400388
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 cf 97 e0 00 00 28 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x18cf97e0
> (da1:mpr0:0:9:0): Actual Retry Count: 9
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 c8 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x4003c8
> (da1:mpr0:0:9:0): Actual Retry Count: 19
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af c8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafc8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 f8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003f8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b8 00 00 08 00
> (da1:mpr0:0:9:0): CAM status: SCSI Status Error
> (da1:mpr0:0:9:0): SCSI status: Check Condition
> (da1:mpr0:0:9:0): SCSI sense: HARDWARE FAILURE asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecafb8
> (da1:mpr0:0:9:0): Field Replaceable Unit: 8
> (da1:mpr0:0:9:0): Actual Retry Count: 23
> (da1:mpr0:0:9:0): Retrying command (per sense data)
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 88 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaf88
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 a8 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x4003a8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafb8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec b0 38 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecb038
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 98 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecaf98
> (da1:mpr0:0:9:0): Actual Retry Count: 18
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af c8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafc8
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 d8 f3 a8 00 01 00 00
> (da1:mpr0:0:9:0): Info: 0x18d8f3ef
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af f8 00 00 08 00
> (da1:mpr0:0:9:0): CAM status: SCSI Status Error
> (da1:mpr0:0:9:0): SCSI status: Check Condition
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecaff8
> (da1:mpr0:0:9:0): Field Replaceable Unit: 8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): Info: 0x22ecafb0
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 d9 03 d8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x18d903d8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 d9 03 b0 00 00 20 00
> (da1:mpr0:0:9:0): Info: 0x18d903b0
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 d9 03 e8 00 01 00 00
> (da1:mpr0:0:9:0): CAM status: SCSI Status Error
> (da1:mpr0:0:9:0): SCSI status: Check Condition
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x18d903e8
> (da1:mpr0:0:9:0): Field Replaceable Unit: 8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafb8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 04 48 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400448
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 98 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400398
> (da1:mpr0:0:9:0): Actual Retry Count: 19
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 d9 27 80 00 00 a8 00
> (da1:mpr0:0:9:0): Info: 0x18d92780
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b0 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: HARDWARE FAILURE asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecafb0
> (da1:mpr0:0:9:0): Actual Retry Count: 23
> (da1:mpr0:0:9:0): Retrying command (per sense data)
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafb1
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 0d 3f 6b b8 00 00 58 00
> (da1:mpr0:0:9:0): Info: 0xd3f6c06
> (da1:mpr0:0:9:0): Actual Retry Count: 8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 0d 3f 6c 48 00 01 00 00
> (da1:mpr0:0:9:0): Info: 0xd3f6c48
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 04 10 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400410
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec b0 10 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecb010
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 04 28 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400428
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec b0 28 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecb028
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 88 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400388
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 b8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003b8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafb8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 0d 3f 7c 90 00 00 b8 00
> (da1:mpr0:0:9:0): Info: 0xd3f7cc5
> (da1:mpr0:0:9:0): Actual Retry Count: 22
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 f8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003f8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af f8 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecaff8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af c0 00 00 08 00
> (da1:mpr0:0:9:0): CAM status: SCSI Status Error
> (da1:mpr0:0:9:0): SCSI status: Check Condition
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecafc0
> (da1:mpr0:0:9:0): Field Replaceable Unit: 8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af d8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafd8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af e8 00 00 08 00
> (da1:mpr0:0:9:0): CAM status: SCSI Status Error
> (da1:mpr0:0:9:0): SCSI status: Check Condition
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecafe8
> (da1:mpr0:0:9:0): Field Replaceable Unit: 8
> (da1:mpr0:0:9:0): Actual Retry Count: 18
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 0d 3f 91 a0 00 00 08 00
> (da1:mpr0:0:9:0): CAM status: SCSI Status Error
> (da1:mpr0:0:9:0): SCSI status: Check Condition
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0xd3f91a3
> (da1:mpr0:0:9:0): Field Replaceable Unit: 8
> (da1:mpr0:0:9:0): Actual Retry Count: 8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 0d 3f 91 a0 00 00 30 00
> (da1:mpr0:0:9:0): Info: 0xd3f91a3
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 0d 3f 91 a0 00 00 30 00
> (da1:mpr0:0:9:0): Info: 0xd3f91a8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 0d 3f 92 38 00 00 60 00
> (da1:mpr0:0:9:0): Info: 0xd3f9238
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 80 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x400380
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 98 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaf98
> (da1:mpr0:0:9:0): Actual Retry Count: 3
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 a6 bb b0 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: HARDWARE FAILURE asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x18a6bbb0
> (da1:mpr0:0:9:0): Actual Retry Count: 23
> (da1:mpr0:0:9:0): Retrying command (per sense data)
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 a6 bb b0 00 00 08 00
> (da1:mpr0:0:9:0): CAM status: SCSI Status Error
> (da1:mpr0:0:9:0): SCSI status: Check Condition
> (da1:mpr0:0:9:0): SCSI sense: HARDWARE FAILURE asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x18a6bbb0
> (da1:mpr0:0:9:0): Field Replaceable Unit: 8
> (da1:mpr0:0:9:0): Actual Retry Count: 23
> (da1:mpr0:0:9:0): Retrying command (per sense data)
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 a6 bc 18 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x18a6bc18
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 c9 12 b0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x18c912b1
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 c9 12 e0 00 00 18 00
> (da1:mpr0:0:9:0): Info: 0x18c912e0
> (da1:mpr0:0:9:0): Field Replaceable Unit: 8
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 98 00 00 08 00
> (da1:mpr0:0:9:0): CAM status: SCSI Status Error
> (da1:mpr0:0:9:0): SCSI status: Check Condition
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecaf98
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 a0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003a0
> (da1:mpr0:0:9:0): Actual Retry Count: 21
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af a0 00 00 08 00
> (da1:mpr0:0:9:0): CAM status: SCSI Status Error
> (da1:mpr0:0:9:0): SCSI status: Check Condition
> (da1:mpr0:0:9:0): SCSI sense: HARDWARE FAILURE asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecafa0
> (da1:mpr0:0:9:0): Field Replaceable Unit: 8
> (da1:mpr0:0:9:0): Actual Retry Count: 23
> (da1:mpr0:0:9:0): Retrying command (per sense data)
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af d0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafd0
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 c9 16 98 00 00 18 00
> (da1:mpr0:0:9:0): Info: 0x18c916a1
> (da1:mpr0:0:9:0): Actual Retry Count: 21
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 0d 3f b2 a8 00 00 b8 00
> (da1:mpr0:0:9:0): Info: 0xd3fb320
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 c9 16 e8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x18c916e9
> (da1:mpr0:0:9:0): Actual Retry Count: 9
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af d0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafd0
> (da1:mpr0:0:9:0): Actual Retry Count: 21
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 e8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003e8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af e8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafe8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af f0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaff0
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 1d ec 95 98 00 00 10 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x1dec9598
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Actual Retry Count: 21
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af d8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafd8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 04 08 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400408
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 04 10 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400410
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec b0 10 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecb010
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 88 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400388
> (da1:mpr0:0:9:0): Actual Retry Count: 23
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 88 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaf88
> (da1:mpr0:0:9:0): Actual Retry Count: 22
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 d0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003d0
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af d0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafd0
> (da1:mpr0:0:9:0): Actual Retry Count: 7
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af f8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaff8
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 04 18 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x400418
> (da1:mpr0:0:9:0): Actual Retry Count: 7
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec b0 18 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecb018
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 90 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaf92
> (da1:mpr0:0:9:0): Actual Retry Count: 20
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af d8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafd8
> (da1:mpr0:0:9:0): Actual Retry Count: 7
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af a8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafa8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 c8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003c8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af c8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafc8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 aa c4 70 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x18aac474
> (da1:mpr0:0:9:0): Actual Retry Count: 19
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af c8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafc8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec b0 18 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecb018
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 88 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x400388
> (da1:mpr0:0:9:0): Actual Retry Count: 21
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 88 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaf88
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af a8 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecafa8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafb8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec b0 08 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecb008
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 b0 65 f0 00 00 18 00
> (da1:mpr0:0:9:0): Info: 0x18b065f0
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af c8 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: HARDWARE FAILURE asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecafc8
> (da1:mpr0:0:9:0): Actual Retry Count: 23
> (da1:mpr0:0:9:0): Retrying command (per sense data)
> (da1:mpr0:0:9:0): Actual Retry Count: 22
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af f0 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: HARDWARE FAILURE asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecaff0
> (da1:mpr0:0:9:0): Actual Retry Count: 23
> (da1:mpr0:0:9:0): Retrying command (per sense data)
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 98 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaf98
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af a8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafa8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af d8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafd8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
-- End of security output --
I see that it says "RECOVERED ERROR", but we keep getting these, so I'm a bit nervious. What might be causing this, and how can I best mitigate it? The GUI says "green light - all is well".
We're currently on FreeNAS-9.3-STABLE-201604041648, with dual E5-2620 V3, 128GB ECC RAM, Avago SAS 9300-8i on IT firmware version 9 and BIOS 8.21. We have 44x 4TB NSSAS drives, 15 300GB 15K SAS, and 3x Intel PCI 400GB SSD drives.
If need be, I can get more details on our setup.
Things have been fine for the past 8 months or so, but just last week, we started seeing errors like below:
filesan.domain.com kernel log messages:
> (da1:mpr0:0:9:0): Info: 0x400388
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 98 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaf98
> (da1:mpr0:0:9:0): Actual Retry Count: 19
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af a0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafa3
> (da1:mpr0:0:9:0): Actual Retry Count: 18
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 04 08 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400408
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 04 38 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400438
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafb0
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af c8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafc8
> (da1:mpr0:0:9:0): Actual Retry Count: 19
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 04 38 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400438
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 ba bb 70 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x18babb70
> (da1:mpr0:0:9:0): Actual Retry Count: 19
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 98 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400398
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 ba c2 38 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x18bac238
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 04 08 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x400408
> (da1:mpr0:0:9:0): Actual Retry Count: 18
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 98 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaf98
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 04 38 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400438
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 88 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaf88
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 1d ec 85 18 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x1dec8518
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 1d ec 85 70 00 00 10 00
> (da1:mpr0:0:9:0): Info: 0x1dec8570
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 1d ec 85 88 00 00 10 00
> (da1:mpr0:0:9:0): Info: 0x1dec8588
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 1d ec 85 98 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x1dec8598
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 04 58 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x400458
> (da1:mpr0:0:9:0): Actual Retry Count: 18
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 ba f9 10 00 01 00 00
> (da1:mpr0:0:9:0): Info: 0x18baf9b4
> (da1:mpr0:0:9:0): Actual Retry Count: 21
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 bb 00 b0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x18bb00b0
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 bb 06 b8 00 01 00 00
> (da1:mpr0:0:9:0): Info: 0x18bb0758
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 80 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400380
> (da1:mpr0:0:9:0): Actual Retry Count: 18
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 bb 14 f0 00 01 00 00
> (da1:mpr0:0:9:0): Info: 0x18bb15d0
> (da1:mpr0:0:9:0): Field Replaceable Unit: 8
> (da1:mpr0:0:9:0): Actual Retry Count: 18
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af c8 00 00 08 00
> (da1:mpr0:0:9:0): CAM status: SCSI Status Error
> (da1:mpr0:0:9:0): SCSI status: Check Condition
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecafc8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 0d 13 94 68 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0xd139468
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 04 18 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400418
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 b8 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x4003b8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafb9
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 c0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003c0
> (da1:mpr0:0:9:0): Actual Retry Count: 20
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af c0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafc0
> (da1:mpr0:0:9:0): Actual Retry Count: 19
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 d8 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x4003d8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 bb 38 10 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x18bb3810
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 bb 38 10 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x18bb3810
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 bb 38 18 00 01 00 00
> (da1:mpr0:0:9:0): Info: 0x18bb3818
> (da1:mpr0:0:9:0): Actual Retry Count: 5
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b0 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: HARDWARE FAILURE asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecafb0
> (da1:mpr0:0:9:0): Actual Retry Count: 23
> (da1:mpr0:0:9:0): Retrying command (per sense data)
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 d8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003d8
> (da1:mpr0:0:9:0): Actual Retry Count: 18
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 0d 3e 05 e8 00 00 58 00
> (da1:mpr0:0:9:0): Info: 0xd3e063f
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 e8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003e8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 b8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003b8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafb8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 bb 60 f0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x18bb60f0
> (da1:mpr0:0:9:0): Actual Retry Count: 18
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af f0 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecaff0
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 b0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003b0
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafb0
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 d8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003d8
> (da1:mpr0:0:9:0): Actual Retry Count: 18
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 bb 8a b8 00 00 10 00
> (da1:mpr0:0:9:0): Info: 0x18bb8ab8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 bb 8a b8 00 00 10 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x18bb8ab8
> (da1:mpr0:0:9:0): Actual Retry Count: 8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 88 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x400388
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 e8 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x4003e8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af e8 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecafe8
> (da1:mpr0:0:9:0): Actual Retry Count: 6
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af f8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaff8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 98 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaf99
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 c5 b2 70 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x18c5b270
> (da1:mpr0:0:9:0): Actual Retry Count: 18
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 c6 43 00 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x18c64300
> (da1:mpr0:0:9:0): Actual Retry Count: 18
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 c6 52 60 00 00 50 00
> (da1:mpr0:0:9:0): SCSI sense: HARDWARE FAILURE asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x18c652a1
> (da1:mpr0:0:9:0): Actual Retry Count: 23
> (da1:mpr0:0:9:0): Retrying command (per sense data)
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af a8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafa8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 c6 52 e8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x18c652e8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafb8
> (da1:mpr0:0:9:0): Actual Retry Count: 7
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 98 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaf9f
> (da1:mpr0:0:9:0): Actual Retry Count: 23
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af c8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafc8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af d8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafd8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 c8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003c8
> (da1:mpr0:0:9:0): Actual Retry Count: 20
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 e8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003e8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 c7 b6 90 00 00 40 00
> (da1:mpr0:0:9:0): Info: 0x18c7b690
> (da1:mpr0:0:9:0): Actual Retry Count: 19
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 88 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400388
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 c8 f6 d8 00 00 b0 00
> (da1:mpr0:0:9:0): Info: 0x18c8f72a
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 c8 ff 48 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x18c8ff48
> (da1:mpr0:0:9:0): Actual Retry Count: 18
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 90 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaf90
> (da1:mpr0:0:9:0): Field Replaceable Unit: 8
> (da1:mpr0:0:9:0): Actual Retry Count: 21
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af a8 00 00 08 00
> (da1:mpr0:0:9:0): CAM status: SCSI Status Error
> (da1:mpr0:0:9:0): SCSI status: Check Condition
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecafa8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafb8
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 98 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400398
> (da1:mpr0:0:9:0): Actual Retry Count: 4
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 98 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaf98
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 88 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400388
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 cf 97 e0 00 00 28 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x18cf97e0
> (da1:mpr0:0:9:0): Actual Retry Count: 9
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 c8 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x4003c8
> (da1:mpr0:0:9:0): Actual Retry Count: 19
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af c8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafc8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 f8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003f8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b8 00 00 08 00
> (da1:mpr0:0:9:0): CAM status: SCSI Status Error
> (da1:mpr0:0:9:0): SCSI status: Check Condition
> (da1:mpr0:0:9:0): SCSI sense: HARDWARE FAILURE asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecafb8
> (da1:mpr0:0:9:0): Field Replaceable Unit: 8
> (da1:mpr0:0:9:0): Actual Retry Count: 23
> (da1:mpr0:0:9:0): Retrying command (per sense data)
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 88 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaf88
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 a8 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x4003a8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafb8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec b0 38 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecb038
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 98 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecaf98
> (da1:mpr0:0:9:0): Actual Retry Count: 18
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af c8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafc8
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 d8 f3 a8 00 01 00 00
> (da1:mpr0:0:9:0): Info: 0x18d8f3ef
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af f8 00 00 08 00
> (da1:mpr0:0:9:0): CAM status: SCSI Status Error
> (da1:mpr0:0:9:0): SCSI status: Check Condition
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecaff8
> (da1:mpr0:0:9:0): Field Replaceable Unit: 8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): Info: 0x22ecafb0
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 d9 03 d8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x18d903d8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 d9 03 b0 00 00 20 00
> (da1:mpr0:0:9:0): Info: 0x18d903b0
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 d9 03 e8 00 01 00 00
> (da1:mpr0:0:9:0): CAM status: SCSI Status Error
> (da1:mpr0:0:9:0): SCSI status: Check Condition
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x18d903e8
> (da1:mpr0:0:9:0): Field Replaceable Unit: 8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafb8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 04 48 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400448
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 98 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400398
> (da1:mpr0:0:9:0): Actual Retry Count: 19
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 d9 27 80 00 00 a8 00
> (da1:mpr0:0:9:0): Info: 0x18d92780
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b0 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: HARDWARE FAILURE asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecafb0
> (da1:mpr0:0:9:0): Actual Retry Count: 23
> (da1:mpr0:0:9:0): Retrying command (per sense data)
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafb1
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 0d 3f 6b b8 00 00 58 00
> (da1:mpr0:0:9:0): Info: 0xd3f6c06
> (da1:mpr0:0:9:0): Actual Retry Count: 8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 0d 3f 6c 48 00 01 00 00
> (da1:mpr0:0:9:0): Info: 0xd3f6c48
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 04 10 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400410
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec b0 10 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecb010
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 04 28 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400428
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec b0 28 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecb028
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 88 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400388
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 b8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003b8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafb8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 0d 3f 7c 90 00 00 b8 00
> (da1:mpr0:0:9:0): Info: 0xd3f7cc5
> (da1:mpr0:0:9:0): Actual Retry Count: 22
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 f8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003f8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af f8 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecaff8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af c0 00 00 08 00
> (da1:mpr0:0:9:0): CAM status: SCSI Status Error
> (da1:mpr0:0:9:0): SCSI status: Check Condition
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecafc0
> (da1:mpr0:0:9:0): Field Replaceable Unit: 8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af d8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafd8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af e8 00 00 08 00
> (da1:mpr0:0:9:0): CAM status: SCSI Status Error
> (da1:mpr0:0:9:0): SCSI status: Check Condition
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecafe8
> (da1:mpr0:0:9:0): Field Replaceable Unit: 8
> (da1:mpr0:0:9:0): Actual Retry Count: 18
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 0d 3f 91 a0 00 00 08 00
> (da1:mpr0:0:9:0): CAM status: SCSI Status Error
> (da1:mpr0:0:9:0): SCSI status: Check Condition
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0xd3f91a3
> (da1:mpr0:0:9:0): Field Replaceable Unit: 8
> (da1:mpr0:0:9:0): Actual Retry Count: 8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 0d 3f 91 a0 00 00 30 00
> (da1:mpr0:0:9:0): Info: 0xd3f91a3
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 0d 3f 91 a0 00 00 30 00
> (da1:mpr0:0:9:0): Info: 0xd3f91a8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 0d 3f 92 38 00 00 60 00
> (da1:mpr0:0:9:0): Info: 0xd3f9238
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 80 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x400380
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 98 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaf98
> (da1:mpr0:0:9:0): Actual Retry Count: 3
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 a6 bb b0 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: HARDWARE FAILURE asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x18a6bbb0
> (da1:mpr0:0:9:0): Actual Retry Count: 23
> (da1:mpr0:0:9:0): Retrying command (per sense data)
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 a6 bb b0 00 00 08 00
> (da1:mpr0:0:9:0): CAM status: SCSI Status Error
> (da1:mpr0:0:9:0): SCSI status: Check Condition
> (da1:mpr0:0:9:0): SCSI sense: HARDWARE FAILURE asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x18a6bbb0
> (da1:mpr0:0:9:0): Field Replaceable Unit: 8
> (da1:mpr0:0:9:0): Actual Retry Count: 23
> (da1:mpr0:0:9:0): Retrying command (per sense data)
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 a6 bc 18 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x18a6bc18
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 c9 12 b0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x18c912b1
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 c9 12 e0 00 00 18 00
> (da1:mpr0:0:9:0): Info: 0x18c912e0
> (da1:mpr0:0:9:0): Field Replaceable Unit: 8
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 98 00 00 08 00
> (da1:mpr0:0:9:0): CAM status: SCSI Status Error
> (da1:mpr0:0:9:0): SCSI status: Check Condition
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecaf98
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 a0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003a0
> (da1:mpr0:0:9:0): Actual Retry Count: 21
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af a0 00 00 08 00
> (da1:mpr0:0:9:0): CAM status: SCSI Status Error
> (da1:mpr0:0:9:0): SCSI status: Check Condition
> (da1:mpr0:0:9:0): SCSI sense: HARDWARE FAILURE asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecafa0
> (da1:mpr0:0:9:0): Field Replaceable Unit: 8
> (da1:mpr0:0:9:0): Actual Retry Count: 23
> (da1:mpr0:0:9:0): Retrying command (per sense data)
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af d0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafd0
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 c9 16 98 00 00 18 00
> (da1:mpr0:0:9:0): Info: 0x18c916a1
> (da1:mpr0:0:9:0): Actual Retry Count: 21
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 0d 3f b2 a8 00 00 b8 00
> (da1:mpr0:0:9:0): Info: 0xd3fb320
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 c9 16 e8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x18c916e9
> (da1:mpr0:0:9:0): Actual Retry Count: 9
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af d0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafd0
> (da1:mpr0:0:9:0): Actual Retry Count: 21
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 e8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003e8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af e8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafe8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af f0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaff0
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 1d ec 95 98 00 00 10 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x1dec9598
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Actual Retry Count: 21
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af d8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafd8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 04 08 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400408
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 04 10 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400410
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec b0 10 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecb010
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 88 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x400388
> (da1:mpr0:0:9:0): Actual Retry Count: 23
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 88 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaf88
> (da1:mpr0:0:9:0): Actual Retry Count: 22
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 d0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003d0
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af d0 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafd0
> (da1:mpr0:0:9:0): Actual Retry Count: 7
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af f8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaff8
> (da1:mpr0:0:9:0): Actual Retry Count: 17
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 04 18 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x400418
> (da1:mpr0:0:9:0): Actual Retry Count: 7
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec b0 18 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecb018
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 90 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaf92
> (da1:mpr0:0:9:0): Actual Retry Count: 20
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af d8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafd8
> (da1:mpr0:0:9:0): Actual Retry Count: 7
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af a8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafa8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 c8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x4003c8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af c8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafc8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 aa c4 70 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x18aac474
> (da1:mpr0:0:9:0): Actual Retry Count: 19
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af c8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafc8
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec b0 18 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecb018
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 00 40 03 88 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x400388
> (da1:mpr0:0:9:0): Actual Retry Count: 21
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 88 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaf88
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af a8 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: RECOVERED ERROR asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecafa8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af b8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafb8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec b0 08 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecb008
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 18 b0 65 f0 00 00 18 00
> (da1:mpr0:0:9:0): Info: 0x18b065f0
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af c8 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: HARDWARE FAILURE asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecafc8
> (da1:mpr0:0:9:0): Actual Retry Count: 23
> (da1:mpr0:0:9:0): Retrying command (per sense data)
> (da1:mpr0:0:9:0): Actual Retry Count: 22
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af f0 00 00 08 00
> (da1:mpr0:0:9:0): SCSI sense: HARDWARE FAILURE asc:3,0 (Peripheral device write fault)
> (da1:mpr0:0:9:0): Info: 0x22ecaff0
> (da1:mpr0:0:9:0): Actual Retry Count: 23
> (da1:mpr0:0:9:0): Retrying command (per sense data)
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af 98 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecaf98
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af a8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafa8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
> (da1:mpr0:0:9:0): WRITE(10). CDB: 2a 00 22 ec af d8 00 00 08 00
> (da1:mpr0:0:9:0): Info: 0x22ecafd8
> (da1:mpr0:0:9:0): Actual Retry Count: 10
-- End of security output --
I see that it says "RECOVERED ERROR", but we keep getting these, so I'm a bit nervious. What might be causing this, and how can I best mitigate it? The GUI says "green light - all is well".