CAM status: CCB request completed with an error

[dfriestedt]

I'm getting a recurring error message: CAM status: CCB request completed with an error

The system is rebooting every 2 hours or so. Any thought on what could be happening here from the error logs?

Code:

Nov 12 16:08:35 freenas         (da13:mpr0:0:37:0): READ(16). CDB: 88 00 00 00 00 02 c0 84 3a d0 00 00 00 08 00 00 length 4096 SMID 959 terminated ioc 804b loginfo 31110e03 scsi 0 state c xfer                 0
Nov 12 16:08:35 freenas         (da13:mpr0:0:37:0): READ(10). CDB: 28 00 f0 31 b5 70 00 00 30 00 length 24576 SMID 794 terminated ioc 804b loginfo 31110e03 scsi 0 state c xfer 0
Nov 12 16:08:35 freenas         (da13:mpr0:0:37:0): READ(10). CDB: 28 00 f0 31 b5 38 00 00 30 00 length 24576 SMID 604 terminated ioc 804b loginfo 31110e03 scsi 0 state c xfer 0
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): READ(16). CDB: 88 00 00 00 00 02 c0 84 3a d0 00 00 00 08 00 00
Nov 12 16:08:35 freenas         (da13:mpr0:0:37:0): READ(10). CDB: 28 00 79 40 1b b8 00 00 f0 00 length 122880 SMID 738 terminated ioc 804b loginfo 31110e03 (da13:mpr0:0:37:0): CAM status: CCB                 request completed with an error
Nov 12 16:08:35 freenas (da13:scsi 0 state c xfer 72712
Nov 12 16:08:35 freenas mpr0:0:37:0): Retrying command
Nov 12 16:08:35 freenas         (da13:mpr0:0:37:0): READ(10). CDB: 28 00 79 40 23 a8 00 00 f0 00 length 122880 SMID 364 terminated ioc 804b loginfo 31110e03 scsi 0 state c xfer 0
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): READ(10). CDB: 28 00 f0 31 b5 70 00 00 30 00
Nov 12 16:08:35 freenas         (da13:mpr0:0:37:0): READ(10). CDB: 28 00 79 40 22 a8 00 01 00 00 length 131072 SMID 34 terminated ioc 804b loginfo 31110e03 s(da13:mpr0:0:37:0): CAM status: CCB                 request completed with an error
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): Retrying command
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): READ(10). CDB: 28 00 f0 31 b5 38 00 00 30 00
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): CAM status: CCB request completed with an error
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): Retrying command
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): READ(10). CDB: 28 00 79 40 1b b8 00 00 f0 00
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): CAM status: CCB request completed with an error
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): Retrying command
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): READ(10). CDB: 28 00 79 40 23 a8 00 00 f0 00
Nov 12 16:08:35 freenas csi 0 state c xfer 0
Nov 12 16:08:35 freenas         (da13:mpr0:0:37:0): READ(10). CDB: 28 00 79 40 21 a8 00 01 00 00 length 131072 SMID 561 terminated ioc 804b loginfo 31110e03 (da13:mpr0:0:37:0): CAM status: CCB                 request completed with an error
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): Retrying command
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): READ(10). CDB: 28 00 79 40 22 a8 00 01 00 00
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): CAM status: CCB request completed with an error
Nov 12 16:08:35 freenas scsi 0 state c xfer 0
Nov 12 16:08:35 freenas         (da13:mpr0:0:37:0): READ(10). CDB: 28 00 79 40 20 a8 00 01 00 00 length 131072 SMID 262 terminated ioc 804b loginfo 31110e03 (da13:scsi 0 state c xfer 0
Nov 12 16:08:35 freenas mpr0:0:37:0): Retrying command
Nov 12 16:08:35 freenas         (da13:mpr0:0:37:0): READ(10). CDB: 28 00 79 40 1f a8 00 01 00 00 length 131072 SMID 879 terminated ioc 804b loginfo 31110e03 (da13:mpr0:0:37:0): READ(10). CDB:                 28 00 79 40 21 a8 00 01 00 00
Nov 12 16:08:35 freenas scsi 0 state c xfer 0
Nov 12 16:08:35 freenas         (da13:mpr0:0:37:0): READ(10). CDB: 28 00 79 40 1e a8 00 01 00 00 length 131072 SMID 122 terminated ioc 804b loginfo 31110e03 (da13:mpr0:0:37:0): CAM status: CCB                 request completed with an error
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): Retrying command
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): READ(10). CDB: 28 00 79 40 20 a8 00 01 00 00
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): CAM status: CCB request completed with an error
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): Retrying command
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): READ(10). CDB: 28 00 79 40 1f a8 00 01 00 00
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): CAM status: CCB request completed with an error
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): Retrying command
Nov 12 16:08:35 freenas scsi 0 state c xfer 0
Nov 12 16:08:35 freenas         (da13:mpr0:0:37:0): READ(10). CDB: 28 00 79 40 1d a8 00 01 00 00 length 131072 SMID 509 terminated ioc 804b loginfo 31110e03 scsi 0 state c xfer 0
Nov 12 16:08:35 freenas         (da13:mpr0:0:37:0): READ(10). CDB: 28 00 79 40 1c a8 00 01 00 00 length 131072 SMID 320 terminated ioc 804b loginfo 31110e03 scsi 0 state c xfer 0
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): READ(10). CDB: 28 00 79 40 1e a8 00 01 00 00
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): CAM status: CCB request completed with an error
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): Retrying command
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): READ(10). CDB: 28 00 79 40 1d a8 00 01 00 00
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): CAM status: CCB request completed with an error
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): Retrying command
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): READ(10). CDB: 28 00 79 40 1c a8 00 01 00 00
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): CAM status: CCB request completed with an error
Nov 12 16:08:35 freenas (da13:mpr0:0:37:0): Retrying command
Nov 12 16:08:36 freenas (da13:mpr0:0:37:0): READ(16). CDB: 88 00 00 00 00 02 c0 84 3a d0 00 00 00 08 00 00
Nov 12 16:08:36 freenas (da13:mpr0:0:37:0): CAM status: SCSI Status Error
Nov 12 16:08:36 freenas (da13:mpr0:0:37:0): SCSI status: Check Condition
Nov 12 16:08:36 freenas (da13:mpr0:0:37:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Nov 12 16:08:36 freenas (da13:mpr0:0:37:0): Retrying command (per sense data)
Nov 12 16:12:08 freenas         (da7:mpr0:0:31:0): READ(10). CDB: 28 00 ef ef e3 30 00 00 30 00 length 24576 SMID 476 terminated ioc 804b loginfo 31110e03 scsi 0 state c xfer 0
Nov 12 16:12:08 freenas         (da7:mpr0:0:31:0): READ(10). CDB: 28 00 ef ef e2 f8 00 00 38 00 length 28672 SMID 188 terminated ioc 804b loginfo 31110e03 scsi 0 state c xfer 0
Nov 12 16:12:08 freenas (da7:mpr0:0:31:0): READ(10). CDB: 28 00 ef ef e3 30 00 00 30 00
Nov 12 16:12:08 freenas         (da7:mpr0:0:31:0): READ(10). CDB: 28 00 ef ef e2 c0 00 00 38 00 length 28672 SMID 650 terminated ioc 804b loginfo 31110e03 sc(da7:mpr0:0:31:0): CAM status: CCB                 request completed with an error
Nov 12 16:12:08 freenas si 0 state c xfer 0
Nov 12 16:12:08 freenas         (da7:mpr0:0:31:0): READ(10). CDB: 28 00 ef ef e2 20 00 00 30 00 length 24576 SMID 952 terminated ioc 804b loginfo 31110e03 scsi 0 state c xfer 0
Nov 12 16:12:08 freenas         (da7:mpr0:0:31:0): READ(10). CDB: 28 00 ef ef e1 e8 00 00 30 00 length 24576 SMID 892 terminated ioc 804b loginfo 31110e03 scsi 0 state c xfer 0
Nov 12 16:12:08 freenas         (da7:mpr0:0:31:0): READ(10). CDB: 28 00 ef ef e1 b0 00 00 30 00 length 24576 SMID 625 terminated ioc 804b loginfo 31110e03 scsi 0 state c xfer 0
Nov 12 16:12:08 freenas         (da7:mpr0:0:31:0): READ(10). CDB: 28 00 c2 9c cc 00 00 00 38 00 length 28672 SMID 446 terminated ioc 804b loginfo 31110e03 scsi 0 state c xfer 27656
Nov 12 16:12:08 freenas         (da7:mpr0:0:31:0): READ(10). CDB: 28 00 c2 9c cc 38 00 00 30 00 length 24576 SMID 131 terminated ioc 804b loginfo 31110e03 scsi 0 state c xfer 0
Nov 12 16:12:08 freenas (da7:mpr0:0:31:0): Retrying command
Nov 12 16:12:08 freenas (da7:mpr0:0:31:0): READ(10). CDB: 28 00 ef ef e2 f8 00 00 38 00
Nov 12 16:12:08 freenas (da7:mpr0:0:31:0): CAM status: CCB request completed with an error
Nov 12 16:12:08 freenas (da7:mpr0:0:31:0): Retrying command
Nov 12 16:12:08 freenas (da7:mpr0:0:31:0): READ(10). CDB: 28 00 ef ef e2 c0 00 00 38 00
Nov 12 16:12:08 freenas (da7:mpr0:0:31:0): CAM status: CCB request completed with an error
Nov 12 16:12:08 freenas (da7:mpr0:0:31:0): Retrying command
Nov 12 16:12:08 freenas (da7:mpr0:0:31:0): READ(10). CDB: 28 00 ef ef e2 20 00 00 30 00
Nov 12 16:12:08 freenas (da7:mpr0:0:31:0): CAM status: CCB request completed with an error
Nov 12 16:12:08 freenas (da7:mpr0:0:31:0): Retrying command
Nov 12 16:12:08 freenas (da7:mpr0:0:31:0): READ(10). CDB: 28 00 ef ef e1 e8 00 00 30 00
Nov 12 16:12:08 freenas (da7:mpr0:0:31:0): CAM status: CCB request completed with an error
Nov 12 16:12:08 freenas (da7:mpr0:0:31:0): Retrying command
Nov 12 16:12:08 freenas (da7:mpr0:0:31:0): READ(10). CDB: 28 00 ef ef e1 b0 00 00 30 00
Nov 12 16:12:08 freenas (da7:mpr0:0:31:0): CAM status: CCB request completed with an error
Nov 12 16:12:08 freenas (da7:mpr0:0:31:0): Retrying command
Nov 12 16:12:08 freenas (da7:mpr0:0:31:0): READ(10). CDB: 28 00 c2 9c cc 00 00 00 38 00
Nov 12 16:12:08 freenas (da7:mpr0:0:31:0): CAM status: CCB request completed with an error
Nov 12 16:12:08 freenas (da7:mpr0:0:31:0): Retrying command
Nov 12 16:12:08 freenas (da7:mpr0:0:31:0): READ(10). CDB: 28 00 c2 9c cc 38 00 00 30 00
Nov 12 16:12:08 freenas (da7:mpr0:0:31:0): CAM status: CCB request completed with an error
Nov 12 16:12:08 freenas (da7:mpr0:0:31:0): Retrying command
Nov 12 16:12:09 freenas (da7:mpr0:0:31:0): READ(10). CDB: 28 00 ef ef e3 30 00 00 30 00
Nov 12 16:12:09 freenas (da7:mpr0:0:31:0): CAM status: SCSI Status Error
Nov 12 16:12:09 freenas (da7:mpr0:0:31:0): SCSI status: Check Condition
Nov 12 16:12:09 freenas (da7:mpr0:0:31:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Nov 12 16:12:09 freenas (da7:mpr0:0:31:0): Retrying command (per sense data)

 

[JohnDigital]

One of the recurring themes on these errors always are replacing and or reseating cables and HBA. These errors seem to be caused by just about everything. It could be bad/unseated cable, bad disk, HBA, power supply. Here is about the closest I can find for you here. Good luck!

 

[dfriestedt]

One of the recurring themes on these errors always are replacing and or reseating cables and HBA. These errors seem to be caused by just about everything. It could be bad/unseated cable, bad disk, HBA, power supply. Here is about the closest I can find for you here. Good luck!

I was seeing these articles too but was hoping there was a "silver bullet". Looks like I'm going fishing! Thank you for taking a look. I'll report back when I figure it out.

 

[JohnDigital]

Yeah there is no panacea unfortunately. My advice is to start with the cheapest thing first. If the system is rebooting you might look to power first. Just a thought.

 

[sretalla]

You might want to look at your recent SMART data from da7 and da13... if SMART is clean, then check cabling.

 

[dfriestedt]

You might want to look at your recent SMART data from da7 and da13... if SMART is clean, then check cabling.

Both PASSED the test just now. I'll start by replacing the cable. This is a new setup and with it I've been TEARING through hard drives at an alarming rate. They are the Western Digital 8TB Ultrastar DC HC320 SATA HDD - 7200 RPM Class, SATA 6 Gb/s, 256MB Cache, 3.5"

I bought them online from Amazon and figured they were selling me crappy drives. I literally replaced about 10 of them through the WD warranty service. But even the replaced drives from WD are going bad quickly. I have 1 go bad every week. No kidding. I'm wondering if the failure rate has something to do with the errors I'm seeing in the logs.

I will replace the cable in question. It's this one: "CableCreation 6.6FT External HD Mini SAS SFF-8644 to SFF-8644 Cable"

 

[Fredda]

Please also check the status of you HBA. Make sure you have flashed it to IT mode and the firmware is not too old.
If you are unsure, post the output of sas3flash -list

 

[dfriestedt]

Here is the output. Now that I think about it I should list 2 controller cards in my signature. One for the Expander Chassis and one for the local drives. The expander card is: LSI Logic Controller Card 05-25704-00 9305-16e 16-Port External SAS 12Gb/s PCI-Express 3.0 Host Bus Adapter Single

I need to add the new local card, for the drives in the Freenas Chassis. I will do that this AM. But the output below looks to be for the 9305 card.

Code:

root@freenas[~]# sas3flash -list
Avago Technologies SAS3 Flash Utility
Version 16.00.00.00 (2017.05.02)
Copyright 2008-2017 Avago Technologies. All rights reserved.

    Adapter Selected is a Avago SAS: SAS3216(A1)

    Controller Number              : 0
    Controller                     : SAS3216(A1)
    PCI Address                    : 00:03:00:00
    SAS Address                    : 500062b-2-00f3-dd20
    NVDATA Version (Default)       : 0b.04.00.05
    NVDATA Version (Persistent)    : 0b.04.00.05
    Firmware Product ID            : 0x2228 (IT)
    Firmware Version               : 14.00.00.00
    NVDATA Vendor                  : LSI
    NVDATA Product ID              : SAS9305-16e
    BIOS Version                   : 08.33.00.00
    UEFI BSD Version               : 13.00.00.00
    FCODE Version                  : N/A
    Board Name                     : SAS9305-16e
    Board Assembly                 : 03-25704-02005
    Board Tracer Number            : SV62837418

    Finished Processing Commands Successfully.
    Exiting SAS3Flash.

 

[sretalla]

I'm wondering if the failure rate has something to do with the errors I'm seeing in the logs.

It's more likely the inverse... the errors in the logs are to do with the failures you're seeing.

Certainly finding a root cause is a good idea with that very high failure rate.

 

[JohnDigital]

Looks as though there might be a newer firmware for that 9305. I show one dated 5.20.2018

 

[dfriestedt]

I will update the firmware. Thank you for all the help!

Initial read is something is wrong with a SFF-8644 Cable. I pulled it and a piece of shielding came lose. I "manually" repaired it and popped it back in and errors are going away.... For now. I ordered 2 new cables. I'll report back after replacing them and confirming it was a cable issue. I also noticed the cable was not properly seated. I was able to remove the cable without pulling the restraining "thing"... Highly technical here. More to follow.

 

[dfriestedt]

Just wanted to report back in case anyone else has this issue. It was a lose SFF-8644 Cable. I was able to properly seat the cable and the errors and unexpected reboots stopped immediately. Thanks for all the help!