Terminated ioc 804b problem

[Mack Hankins]

So i've got a big NAS running a 9207-8i and 9207-8e on FreeNAS-9.10.2-U3 (e1497f269) which I admit wasn't running all the smart test I should of been, but I'm having trouble diagnosing this set to errors.

Code:

(probe52:mps1:0:59:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 924 terminated ioc 804b scsi 0 state c xfer 0
(probe52:mps1:0:59:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe52:mps1:0:59:0): CAM status: CCB request completed with an error
(probe52:mps1:0:59:0): Retrying command
(probe52:mps1:0:59:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 925 terminated ioc 804b scsi 0 state c xfer 0
(probe52:mps1:0:59:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe52:mps1:0:59:0): CAM status: CCB request completed with an error
(probe52:mps1:0:59:0): Retrying command
(probe52:mps1:0:59:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 926 terminated ioc 804b scsi 0 state c xfer 0
(probe52:mps1:0:59:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe52:mps1:0:59:0): CAM status: CCB request completed with an error
(probe52:mps1:0:59:0): Error 5, Retries exhausted

I'm currently batching out short and long test on all the drives, but I'm just looking for some general guidance. I saw everything on the forums from bad drives to failed controllers.

 

[Dice]

Please provide detailed hardware information.

 

[Mack Hankins]

Board: Supermicro X9-DRH-7TF
Proc: E5-2603V2 x2
Memory: 32GB ECC
HBA(S): LSI 9207-8i & 9207-8e (v20 firmware - latest)

Code:

<ATA SanDisk SD6SB1M1 600>		 at scbus0 target 0 lun 0 (pass0,da0)
<ATA SanDisk SD6SB1M1 600>		 at scbus0 target 1 lun 0 (pass1,da1)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus0 target 8 lun 0 (pass2,da2)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus0 target 9 lun 0 (pass3,da3)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus0 target 10 lun 0 (pass4,da4)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus0 target 11 lun 0 (pass5,da5)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus0 target 12 lun 0 (pass6,da6)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus0 target 13 lun 0 (pass7,da7)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus0 target 14 lun 0 (pass8,da8)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus0 target 15 lun 0 (pass9,da9)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus0 target 17 lun 0 (pass10,da10)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus0 target 18 lun 0 (pass11,da11)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus0 target 19 lun 0 (pass12,da12)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus0 target 20 lun 0 (pass13,da13)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus0 target 21 lun 0 (pass14,da14)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus0 target 22 lun 0 (pass15,da15)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus0 target 23 lun 0 (pass16,da16)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus0 target 24 lun 0 (pass17,da17)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus0 target 25 lun 0 (pass18,da18)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus0 target 26 lun 0 (pass19,da19)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus0 target 27 lun 0 (pass20,da20)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus0 target 28 lun 0 (pass21,da21)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus0 target 29 lun 0 (pass22,da22)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus0 target 30 lun 0 (pass23,da23)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus0 target 31 lun 0 (pass24,da24)
<LSI SAS2X36 0e12>				 at scbus0 target 32 lun 0 (pass25,ses0)
<ATA ST2000VN000-1HJ1 SC60>		at scbus0 target 33 lun 0 (pass26,da25)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 9 lun 0 (pass27,da26)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 10 lun 0 (pass28,da27)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 11 lun 0 (pass29,da28)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 12 lun 0 (pass30,da29)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 13 lun 0 (pass31,da30)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 15 lun 0 (pass32,da31)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 16 lun 0 (pass33,da32)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 17 lun 0 (pass34,da33)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 18 lun 0 (pass35,da34)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 19 lun 0 (pass36,da35)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 20 lun 0 (pass37,da36)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 21 lun 0 (pass38,da37)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 22 lun 0 (pass39,da38)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 23 lun 0 (pass40,da39)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 24 lun 0 (pass41,da40)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 25 lun 0 (pass42,da41)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 26 lun 0 (pass43,da42)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 27 lun 0 (pass44,da43)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 28 lun 0 (pass45,da44)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 29 lun 0 (pass46,da45)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 30 lun 0 (pass47,da46)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 31 lun 0 (pass48,da47)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 32 lun 0 (pass49,da48)
<LSI SAS2X36 0e12>				 at scbus8 target 33 lun 0 (pass50,ses1)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 34 lun 0 (pass51,da49)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 35 lun 0 (pass52,da50)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 36 lun 0 (pass53,da51)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 37 lun 0 (pass54,da52)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 38 lun 0 (pass55,da53)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 39 lun 0 (pass56,da54)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 40 lun 0 (pass57,da55)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 41 lun 0 (pass58,da56)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 42 lun 0 (pass59,da57)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 43 lun 0 (pass60,da58)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 44 lun 0 (pass61,da59)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 45 lun 0 (pass62,da60)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 46 lun 0 (pass63,da61)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 47 lun 0 (pass64,da62)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 48 lun 0 (pass65,da63)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 49 lun 0 (pass66,da64)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 50 lun 0 (pass67,da65)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 51 lun 0 (pass68,da66)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 52 lun 0 (pass69,da67)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 53 lun 0 (pass70,da68)
<ATA TOSHIBA DT01ACA2 ABB0>		at scbus8 target 54 lun 0 (pass71,da69)
<LSI SAS2X36 0e12>				 at scbus8 target 55 lun 0 (pass72,ses2)
<PNY USB 2.0 FD 1100>			  at scbus10 target 0 lun 0 (pass73,da70)

 

[Vito Reiter]

Personally looks like a failing drive somewhere, you can run the test individually on each drive until only one shoots out errors. If it's only one then replace it with a known-good, and if errors are gone it was just the drive, if not then it may be the controller.

 

[Mack Hankins]

Personally looks like a failing drive somewhere, you can run the test individually on each drive until only one shoots out errors. If it's only one then replace it with a known-good, and if errors are gone it was just the drive, if not then it may be the controller.

Thanks, I'm thinking the same. However that error was keeping me from booting yesterday and it just probed for hours. I had to drop in another usb just to get it to boot, but if I reboot it the probing will return. So, right now I'm just running long tests throughout the night.

It's the strangest issue, but I'll return with any updates.

 

[Vito Reiter]

Maybe the boot drive is the failing one?

 

[Dice]

Once you've established that all connected drives pass SMART tests with flying colors, further investigation can happen.
A drive failing to initialize, or having potential connectivity issues I *guess* could generate this type of error.
A couple of useful links for you to rule out any malfunctioning drives:
https://forums.freenas.org/index.ph...bleshooting-guide-all-versions-of-freenas.17/
https://en.wikipedia.org/wiki/S.M.A.R.T.

 

[Mack Hankins]

I'm not really sure what to make of this test result. @Dice Ah, those links are very helpful. I have spare drives of course and could afford to replace all 3 that have pretty much the same error.

Code:

~# smartctl -a /dev/da22
smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:	 Toshiba 3.5" DT01ACA... Desktop HDD
Device Model:	 TOSHIBA DT01ACA200
Serial Number:	643GW9DGS
LU WWN Device Id: 5 000039 ffac6c3b0
Firmware Version: MX4OABB0
User Capacity:	2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:	 512 bytes logical, 4096 bytes physical
Rotation Rate:	7200 rpm
Form Factor:	  3.5 inches
Device is:		In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:	Fri Apr 28 09:15:45 2017 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x84)	Offline data collection activity
					was suspended by an interrupting command from host.
					Auto Offline Data Collection: Enabled.
Self-test execution status:	  (   0)	The previous self-test routine completed
					without error or no self-test has ever
					been run.
Total time to complete Offline
data collection:		 (14726) seconds.
Offline data collection
capabilities:			 (0x5b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					No Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:			(0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:		(0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine
recommended polling time:	 (   1) minutes.
Extended self-test routine
recommended polling time:	 ( 246) minutes.
SCT capabilities:			(0x003d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME		  FLAG	 VALUE WORST THRESH TYPE	  UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate	 0x000b   100   100   016	Pre-fail  Always	   -	   0
  2 Throughput_Performance  0x0005   139   139   054	Pre-fail  Offline	  -	   70
  3 Spin_Up_Time			0x0007   130   130   024	Pre-fail  Always	   -	   290 (Average 293)
  4 Start_Stop_Count		0x0012   100   100   000	Old_age   Always	   -	   50
  5 Reallocated_Sector_Ct   0x0033   100   100   005	Pre-fail  Always	   -	   0
  7 Seek_Error_Rate		 0x000b   100   100   067	Pre-fail  Always	   -	   0
  8 Seek_Time_Performance   0x0005   126   126   020	Pre-fail  Offline	  -	   32
  9 Power_On_Hours		  0x0012   098   098   000	Old_age   Always	   -	   19262
10 Spin_Retry_Count		0x0013   100   100   060	Pre-fail  Always	   -	   0
12 Power_Cycle_Count	   0x0032   100   100   000	Old_age   Always	   -	   50
192 Power-Off_Retract_Count 0x0032   100   100   000	Old_age   Always	   -	   114
193 Load_Cycle_Count		0x0012   100   100   000	Old_age   Always	   -	   114
194 Temperature_Celsius	 0x0002   200   200   000	Old_age   Always	   -	   30 (Min/Max 16/37)
196 Reallocated_Event_Count 0x0032   100   100   000	Old_age   Always	   -	   0
197 Current_Pending_Sector  0x0022   100   100   000	Old_age   Always	   -	   0
198 Offline_Uncorrectable   0x0008   100   100   000	Old_age   Offline	  -	   0
199 UDMA_CRC_Error_Count	0x000a   200   200   000	Old_age   Always	   -	   2

SMART Error Log Version: 1
ATA Error Count: 2
	CR = Command Register [HEX]
	FR = Features Register [HEX]
	SC = Sector Count Register [HEX]
	SN = Sector Number Register [HEX]
	CL = Cylinder Low Register [HEX]
	CH = Cylinder High Register [HEX]
	DH = Device/Head Register [HEX]
	DC = Device Command Register [HEX]
	ER = Error register [HEX]
	ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 2 occurred at disk power-on lifetime: 1547 hours (64 days + 11 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 d1 af fc 3f 00  Error: ICRC, ABRT at LBA = 0x003ffcaf = 4193455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 00 80 fc 3f 40 00	  01:27:23.749  READ FPDMA QUEUED
  60 00 00 80 03 00 40 00	  01:27:23.749  READ FPDMA QUEUED
  60 00 00 80 02 00 40 00	  01:27:23.748  READ FPDMA QUEUED
  60 00 00 80 01 00 40 00	  01:27:23.748  READ FPDMA QUEUED
  60 00 00 80 00 00 40 00	  01:27:23.747  READ FPDMA QUEUED

Error 1 occurred at disk power-on lifetime: 1546 hours (64 days + 10 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  84 51 81 ff 01 00 00  Error: ICRC, ABRT at LBA = 0x000001ff = 511

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 00 80 01 00 40 00	  01:02:12.913  READ FPDMA QUEUED
  60 00 00 80 00 00 40 00	  01:02:12.913  READ FPDMA QUEUED
  60 00 00 80 01 40 40 00	  01:02:12.609  READ FPDMA QUEUED
  60 00 00 80 00 40 40 00	  01:02:12.609  READ FPDMA QUEUED
  60 00 00 80 01 40 40 00	  01:02:12.608  READ FPDMA QUEUED

SMART Self-test log structure revision number 1
Num  Test_Description	Status				  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline	Completed without error	   00%	 19247		 -
# 2  Short offline	   Completed without error	   00%	  5494		 -
# 3  Short offline	   Completed without error	   00%	  4774		 -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
	1		0		0  Not_testing
	2		0		0  Not_testing
	3		0		0  Not_testing
	4		0		0  Not_testing
	5		0		0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[Dice]

AFAIK, that drive looks healthy but with a few quirks to compare to other drives in the system.
That is, value of ID#2 (might be vendor specific?) and ID#19.
Still a very low value, but commonly caused by miscommunication between drive and controller. I silently predicted one or a couple of drives displaying higher counts. If this is the worst you have, you're fine on ID#199 and if that was the main cause for the error post in OP to occur "a few times" as opposed to "regularly" you're fine.

However, it is too soon to rule out any other issue.
Report back once you've looked through all drives.

Sidenote: you need to configure SMART schedules on the box. Do you have scrub schedules?

 

[Mack Hankins]

With the exception of da6, da16, and da22 that had the UDMA_CRC_Error_Count of 2 or less. Everything else passed and was 0 on Current_Pending_Sector / Reallocated_Sector_Ct.

As I mentioned smart wasn't set up as often as needed which has been fixed, but scrubs are only happening once a month. I've bumped it to 1st and 15th now.

Smart sched short is 3rd, 10th, 20th
Smart sched long is 8th, 27th

 

[Dice]

To me you're now at a comforting enough position to not worry.
Most would probably stop investigating at this point.

Here is a scenario:
There might have been a glitch somewhere causing the ID#199 to generate the error message which indicates some sort of miscommunication to a drive.
However, it is also possible the SMART errors were instilled prior to the FreeNAS error.

Could you describe a bit further the characteristics of the error message's occurrences?

 

[Mack Hankins]

We have to back up a little bit for that and I'm sorry I don't have the actual probe messages, but I've seen similar ones on this board in a related topic.

On a previous 9.3 install during boot (of freenas) it would simply probe a drive for hours without ever actually starting services. So, I unplugged the JBOD it booted right up. Then I installed the latest version of FN on a new usb and restored config, it booted right up with everything plugged in. I updated the firmware on the LSI hbas to 20 instead of 16 and then (the next day) updated freenas. After the reboot for the update, it started that probe again and never booted.

I defaulted back to initial install and here we are doing tests. The next thing I'm going to try after backing up my config again is a reboot to see if it probes. If that happens, I'm not even sure how to get back into the system w/o reverting to initial install.

 

[Mack Hankins]

Although there are updates available I'm not going to apply them. I've rebooted it 2 times this morning and the only anomalies are the messages below.

Thanks for the help.

Code:

ukbd1: <HID Keyboard> on usbus1
kbd2 at ukbd1
		(probe30:mps1:0:59:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 922 terminated ioc 804b scsi 0 state c xfer 0
(probe30:mps1:0:59:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe30:mps1:0:59:0): CAM status: CCB request completed with an error
(probe30:mps1:0:59:0): Retrying command
		(probe30:mps1:0:59:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 923 terminated ioc 804b scsi 0 state c xfer 0
(probe30:mps1:0:59:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe30:mps1:0:59:0): CAM status: CCB request completed with an error
(probe30:mps1:0:59:0): Retrying command
		(probe30:mps1:0:59:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 924 terminated ioc 804b scsi 0 state c xfer 0
(probe30:mps1:0:59:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe30:mps1:0:59:0): CAM status: CCB request completed with an error
(probe30:mps1:0:59:0): Error 5, Retries exhausted
ses0 at mps0 bus 0 scbus0 target 32 lun 0
ses0: <LSI SAS2X36 0e12> Fixed Enclosure Services SPC-3 SCSI device
ses0: Serial Number
ses0: 600.000MB/s transfers
ses0: Command Queueing enabled
ses0: SCSI-3 ENC Device
ses1 at mps1 bus 0 scbus8 target 33 lun 0
ses1: <LSI SAS2X36 0e12> Fixed Enclosure Services SPC-3 SCSI device
ses1: Serial Number
ses1: 600.000MB/s transfers
ses1: Command Queueing enabled
ses1: SCSI-3 ENC Device
ses2 at mps1 bus 0 scbus8 target 55 lun 0
ses2: <LSI SAS2X36 0e12> Fixed Enclosure Services SPC-3 SCSI device
ses2: Serial Number
ses2: 600.000MB/s transfers
ses2: Command Queueing enabled
ses2: SCSI-3 ENC Device
da2 at mps0 bus 0 scbus0 target 8 lun 0
da3 at mps0 bus 0 scbus0 target 9 lun 0
da2: <ATA TOSHIBA DT01ACA2 ABB0> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number			Y4A4X79TS
da2: 600.000MB/s transfers

Code:

(probe52:mps1:0:59:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 535 terminated ioc 804b scsi 0 state c xfer 0
(probe52:mps1:0:59:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe52:mps1:0:59:0): CAM status: CCB request completed with an error
(probe52:mps1:0:59:0): Retrying command
ugen0.2: <vendor 0x8087> at usbus0
uhub2: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0
ugen1.2: <vendor 0x8087> at usbus1
uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus1
uhub2: 6 ports with 6 removable, self powered
uhub3: 8 ports with 8 removable, self powered
ugen0.3: <Winbond Electronics Corp> at usbus0
ukbd0: <Winbond Electronics Corp Hermon USB hidmouse Device, class 0/0, rev 1.10/0.01, addr 3> on usbus0
kbd0 at ukbd0
(probe52:mps1:0:59:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 921 terminated ioc 804b scsi 0 state c xfer 0
(probe52:mps1:0:59:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe52:mps1:0:59:0): CAM status: CCB request completed with an error
(probe52:mps1:0:59:0): Retrying command
ugen1.3: <PNY Technologies> at usbus1
umass0: <PNY Technologies USB 2.0 FD, class 0/0, rev 2.00/11.00, addr 3> on usbus1
umass0:  SCSI over Bulk-Only; quirks = 0x8100
umass0:10:0:-1: Attached to scbus10
ugen1.4: <ATEN> at usbus1
ukbd1: <HID Keyboard> on usbus1
kbd2 at ukbd1
(probe52:mps1:0:59:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 922 terminated ioc 804b scsi 0 state c xfer 0
(probe52:mps1:0:59:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe52:mps1:0:59:0): CAM status: CCB request completed with an error
(probe52:mps1:0:59:0): Retrying command
(probe52:mps1:0:59:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 923 terminated ioc 804b scsi 0 state c xfer 0
(probe52:mps1:0:59:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe52:mps1:0:59:0): CAM status: CCB request completed with an error
(probe52:mps1:0:59:0): Retrying command
(probe52:mps1:0:59:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 924 terminated ioc 804b scsi 0 state c xfer 0
(probe52:mps1:0:59:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe52:mps1:0:59:0): CAM status: CCB request completed with an error
(probe52:mps1:0:59:0): Error 5, Retries exhausted
ses0 at mps0 bus 0 scbus0 target 32 lun 0
ses0: <LSI SAS2X36 0e12> Fixed Enclosure Services SPC-3 SCSI device