Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

SYNCHRONIZE CACHE command timeout error

mloiterman

Newbie
Joined
Jan 30, 2013
Messages
45
I'm having similar issues and have been battling for MONTHS with no progress. So, so frustrating. Ready to take this server and chuck it. I've tried the LSI and IT firmware with no changes. I've tried the following LSI firmware versions from SuperMicro:

  1. 3008_FW_PH16.00.01.00.zip (IT and IR - currently on IR and staying)
  2. 3008_FW_PH10.00.03.00.zip (IT)
  3. Original firmware, but cannot determine version, but I don't recall having issue at that time. This was pre 11.2 which I believe is the source of the error.
Here is a similar thread I started:

https://www.ixsystems.com/community/threads/failed-to-read-smart-attribute-data.72468/#post-502263

I believe the issue is rooted in something in 11.2 and the timeouts for the LSI driver, but I have no data and this is only speculation.

Here are the kind of errors I'm getting
Code:
pid 2232 (syslog-ng), uid 0: exited on signal 6 (core dumped)
(da6:mpr0:0:6:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 length 0 SMID 449 Aborting command 0xfffffe0001077570
mpr0: Sending reset from mprsas_send_abort for target ID 6
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 52 b0 00 00 00 80 00 00 length 65536 SMID 978 terminated ioc 804b loginfo 31130000 scsi 0 state c xfer 0
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 51 a8 00 00 00 80 00 00 length 65536 SMID 316 terminated ioc 804b loginfo 31130000 scsi 0 state c xfer 0
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 52 b0 00 00 00 80 00 00
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 52 28 00 00 00 80 00 00 length 65536 SMID 729 terminated ioc 804b l(da6:mpr0:0:6:0): CAM status: CCB request completed with an error
oginfo 31130000 scsi 0 state c xfer 0
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 51 20 00 00 00 80 00 00 length 65536 SMID 930 terminated ioc 804b l(da6:mpr0:0:6:0): Retrying command
oginfo 31130000 scsi 0 state c xfer 0
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 51 a8 00 00 00 80 00 00
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 50 a0 00 00 00 80 00 00 length 65536 SMID 886 terminated ioc 804b l(da6:mpr0:0:6:0): CAM status: CCB request completed with an error
oginfo 31130000 scsi 0 state c xfer 0
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 50 18 00 00 00 80 00 00 length 65536 SMID 802 terminated ioc 804b l(da6:mpr0:0:6:0): Retrying command
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 52 28 00 00 00 80 00 00
oginfo 31130000 scsi 0 state c xfer 0
(da6:mpr0:0:6:0): CAM status: CCB request completed with an error
(da6:mpr0:0:6:0): Retrying command
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 51 20 00 00 00 80 00 00
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 4f 98 00 00 00 80 00 00 length 65536 SMID 558 terminated ioc 804b l(da6:mpr0:0:6:0): CAM status: CCB request completed with an error
oginfo 31130000 scsi 0 state c xfer 0
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 4f 10 00 00 00 80 00 00 length 65536 SMID 588 terminated ioc 804b l(da6:mpr0:0:6:0): Retrying command
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 50 a0 00 00 00 80 00 00
oginfo 31130000 scsi 0 state c xfer 0
(da6:mpr0:0:6:0): CAM status: CCB request completed with an error
(da6:mpr0:0:6:0): Retrying command
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 50 18 00 00 00 80 00 00
(pass6:mpr0:0:6:0): ATA COMMAND PASS THROUGH(16). CDB: 85 08 0e 00 d0 00 01 00 00 00 4f 00 c2 00 b0 00 length 512 SMID 480 te(da6:mpr0:0:6:0): CAM status: CCB request completed with an error
(da6:mpr0:0:6:0): Retrying command
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 4f 98 00 00 00 80 00 00
rminated ioc 804b loginfo 31130000 scsi 0 state c xfer 0
(da6:mpr0:0:6:0): CAM status: CCB request completed with an error
(da6:mpr0:0:6:0): Retrying command
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 4f 10 00 00 00 80 00 00
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 4f 08 00 00 00 08 00 00 length 4096 SMID 778 terminated ioc 804b lo(da6:mpr0:0:6:0): CAM status: CCB request completed with an error
ginfo 31130000 scsi 0 state c xfer 0
(da6:mpr0:0:6:0): Retrying command
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 4e 08 00 00 01 00 00 00 length 131072 SMID 765 terminated ioc 804b (da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 4f 08 00 00 00 08 00 00
loginfo 31130000 scsi 0 state c xfer 0
(da6:mpr0:0:6:0): CAM status: CCB request completed with an error
(da6:mpr0:0:6:0): Retrying command
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 4d 48 00 00 00 c0 00 00 length 98304 SMID 867 terminated ioc 804b l(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 4e 08 00 00 01 00 00 00
oginfo 31130000 scsi 0 state c xfer 0
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 4c 48 00 00 01 00 00 00 length 131072 SMID 905 terminated ioc 804b (da6:mpr0:0:6:0): CAM status: CCB request completed with an error
loginfo 31130000 scsi 0 state c xfer 0
(da6:mpr0:0:    (da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 4b 48 00 00 01 00 00 00 length 131072 SMID 671 terminated ioc 804b 6:loginfo 31130000 scsi 0 state c xfer 0
0): Retrying command
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 4a 48 00 00 01 00 00 00 length 131072 SMID 596 terminated ioc 804b (da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 4d 48 00 00 00 c0 00 00
loginfo 31130000 scsi 0 state c xfer 0
(da6:mpr0:0:6:0): CAM status: CCB request completed with an error
(da6:mpr0:0:6:0): Retrying command
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 4c 48 00 00 01 00 00 00
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 49 48 00 00 01 00 00 00 length 131072 SMID 731 terminated ioc 804b (da6:mpr0:0:6:0): CAM status: CCB request completed with an error
loginfo 31130000 scsi 0 state c xfer 0
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 48 48 00 00 01 00 00 00 length 131072 SMID 293 terminated ioc 804b (da6:loginfo 31130000 scsi 0 state c xfer 0
mpr0:0:6:0): Retrying command
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 47 48 00 00 01 00 00 00 length 131072 SMID 805 terminated ioc 804b (da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 4b 48 00 00 01 00 00 00
loginfo 31130000 scsi 0 state c xfer 0
(da6:mpr0:0:6:0): CAM status: CCB request completed with an error
(da6:mpr0:0:6:0): Retrying command
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 4a 48 00 00 01 00 00 00
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 46 48 00 00 01 00 00 00 length 131072 SMID 186 terminated ioc 804b (da6:mpr0:0:6:0): CAM status: CCB request completed with an error
loginfo 31130000 scsi 0 state c xfer 0
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 58 ac 00 70 00 00 00 f8 00 00 length 126976 SMID 983 terminated ioc 804b (da6:loginfo 31130000 scsi 0 state c xfer 0
mpr0: Unfreezing devq for target ID 6
mpr0:0:6:0): Retrying command
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 49 48 00 00 01 00 00 00
(da6:mpr0:0:6:0): CAM status: CCB request completed with an error
(da6:mpr0:0:6:0): Retrying command
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 48 48 00 00 01 00 00 00
(da6:mpr0:0:6:0): CAM status: CCB request completed with an error
(da6:mpr0:0:6:0): Retrying command
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 47 48 00 00 01 00 00 00
(da6:mpr0:0:6:0): CAM status: CCB request completed with an error
(da6:mpr0:0:6:0): Retrying command
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 1c da 46 48 00 00 01 00 00 00
(da6:mpr0:0:6:0): CAM status: CCB request completed with an error
(da6:mpr0:0:6:0): Retrying command
(da6:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 58 ac 00 70 00 00 00 f8 00 00
(da6:mpr0:0:6:0): CAM status: CCB request completed with an error
(da6:mpr0:0:6:0): Retrying command
(da6:mpr0:0:6:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da6:mpr0:0:6:0): CAM status: Command timeout
(da6:mpr0:0:6:0): Retrying command
(da6:mpr0:0:6:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da6:mpr0:0:6:0): CAM status: SCSI Status Error
(da6:mpr0:0:6:0): SCSI status: Check Condition
(da6:mpr0:0:6:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
(da6:mpr0:0:6:0): Error 6, Retries exhausted
(da6:mpr0:0:6:0): Invalidating pack


Code:
    (da1:mpr0:0:1:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 length 0 SMID 422 Aborting command 0xfffffe0001074ea0
mpr0: Sending reset from mprsas_send_abort for target ID 1
    (da1:mpr0:0:1:0): READ(10). CDB: 28 00 57 e3 25 10 00 00 08 00 length 4096 SMID 435 terminated ioc 804b loginfo 31130000 scsi 0 state c xfer 0
    (pass1:mpr0:0:1:0): ATA COMMAND PASS THROUGH(16). CDB: 85 08 0e 00 d5 00 01 00 00 00 4f 00 c2 00 b0 00 length 512 SMID 640 terminated ioc 804b loginfo 31130000 scsi 0 state c xfer 0
(da1:mpr0:0:1:0): READ(10). CDB: 28 00 57 e3 25 10 00 00 08 00
    (da1:mpr0:0:1:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 length 0 SMID 220 terminated ioc 804b loginfo 311(da1:mpr0:0:1:0): CAM status: CCB request completed with an error
30000 scsi 0 state c xfer 0
(da1:mpr0:0:1:0): Retrying command
    (da1:mpr0:0:1:0): WRITE(16). CDB: 8a 00 00 00 00 01 04 bb 5c a8 00 00 00 08 00 00 length 4096 SMID 139 terminated ioc 804b lo(da1:mpr0:0:1:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
ginfo 31130000 scsi 0 state c xfer 0
mpr0: Unfreezing devq for target ID 1
(da1:mpr0:0:1:0): CAM status: CCB request completed with an error
(da1:mpr0:0:1:0): Retrying command
(da1:mpr0:0:1:0): WRITE(16). CDB: 8a 00 00 00 00 01 04 bb 5c a8 00 00 00 08 00 00
(da1:mpr0:0:1:0): CAM status: CCB request completed with an error
(da1:mpr0:0:1:0): Retrying command
(da1:mpr0:0:1:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da1:mpr0:0:1:0): CAM status: Command timeout
(da1:mpr0:0:1:0): Retrying command
(da1:mpr0:0:1:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da1:mpr0:0:1:0): CAM status: SCSI Status Error
(da1:mpr0:0:1:0): SCSI status: Check Condition
(da1:mpr0:0:1:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
(da1:mpr0:0:1:0): Error 6, Retries exhausted
(da1:mpr0:0:1:0): Invalidating pack

  • motherboard make and model
    • SuperMicro X11SSH-CTF
      • Firmware Revision : 01.48
      • Firmware Build Time : 06/22/2018
      • BIOS Version: 2.2
      • BIOS Build Time: 05/23/2018
      • Redfish Version : 1.0.1
  • CPU make and model
    • CPU: Intel(R) Xeon(R) CPU E3-1275 v5 @ 3.60GHz (3600.18-MHz K8-class CPU)
      • Origin="GenuineIntel" Id=0x506e3 Family=0x6 Model=0x5e Stepping=3
  • RAM quantity
    • 32 GiB
      • Crucial 2-16GB DDR4-2400 EUDIMM 1.2V CL17
  • boot drive
    • Intel SSD 600p Series SSDPEKKW128G7X1 (128 GB, M.2 80mm PCIe NVMe 3.0 x4, 3D1, TLC)
  • hard drives, quantity, model numbers, and RAID configuration
    • 8 x ST4000LM024
    • RAIDZ2
  • hard disk controllers
    • Avago Technologies (LSI) SAS3008
    • Code:
      sas3flash -c 0 -list
      Avago Technologies SAS3 Flash Utility
      Version 16.00.00.00 (2017.05.02)
      Copyright 2008-2017 Avago Technologies. All rights reserved.
      
          Adapter Selected is a Avago SAS: SAS3008(C0)
      
          Controller Number              : 0
          Controller                     : SAS3008(C0)
          PCI Address                    : 00:01:00:00
          SAS Address                    : 5003048-0-1e04-6000
          NVDATA Version (Default)       : 0e.01.31.27
          NVDATA Version (Persistent)    : 0e.01.31.27
          Firmware Product ID            : 0x2721 (IR)
          Firmware Version               : 16.00.01.00
          NVDATA Vendor                  : LSI
          NVDATA Product ID              : LSI3008-IR
          BIOS Version                   : 08.37.00.00
          UEFI BSD Version               : 18.00.00.00
          FCODE Version                  : N/A
          Board Name                     : LSI3008-IR
          Board Assembly                 : N/A
          Board Tracer Number            : N/A
      
          Finished Processing Commands Successfully.
          Exiting SAS3Flash.
      
  • network cards
    • ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k> mem 0xd0200000-0xd03fffff,0xd0404000-0xd0407fff irq 16 at device 0.0 on pci4
    • ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k> mem 0xd0000000-0xd01fffff,0xd0400000-0xd0403fff irq 17 at device 0.1 on pci4
  • FreeNAS-11.2-U4 (Build Date: May 7, 2019 0:46)
 
Last edited:

mgittelman

FreeNAS Experienced
Joined
Dec 8, 2017
Messages
171
@mloiterman Same motherboard, same problem since about November last year. Tried different firmware versions, both IR and IT mode. All my drives are ST8000NM0055 though.
 

mloiterman

Newbie
Joined
Jan 30, 2013
Messages
45
@mloiterman Same motherboard, same problem since about November last year. Tried different firmware versions, both IR and IT mode. All my drives are ST8000NM0055 though.
That's probably about when mine started as well. Like I said, I believe it started with 11.2:
11.2 was released in 12 2018 and that's when my first post was.
 

mloiterman

Newbie
Joined
Jan 30, 2013
Messages
45
This is so frustrating. Any kind of significant writes are causing these errors.

This basically renders FreeNAS useless to me and I'm sorry I'm continuing to waste my time. I've been using FreeNAS since it was an early fork of M0n0wall and this is ridiculous. So disappointed and frustrated that I cannot rely on this any more.

Code:
(da4:mpr0:0:6:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 length 0 SMID 732 Aborting command 0xfffffe0001090c40
mpr0: Sending reset from mprsas_send_abort for target ID 6
    (pass4:mpr0:0:6:0): ATA COMMAND PASS THROUGH(16). CDB: 85 08 0e 00 d5 00 01 00 00 00 4f 00 c2 00 b0 00 length 512 SMID 185 terminated ioc 804b loginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 21 c0 00 00 00 80 00 00 length 65536 SMID 822 terminated ioc 804b loginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 21 40 00 00 00 80 00 00 length 65536 SMID 666 terminated ioc 804b loginfo 31130000 scsi 0 state c xfer 0
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 21 c0 00 00 00 80 00 00
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 20 b8 00 00 00 80 00 00 length 65536 SMID 320 terminated ioc 804b l(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
oginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 20 38 00 00 00 80 00 00 length 65536 SMID 256 terminated ioc 804b l(da4:oginfo 31130000 scsi 0 state c xfer 0
mpr0:0:6:0): Retrying command
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1f b0 00 00 00 80 00 00 length 65536 SMID 694 terminated ioc 804b loginfo 31130000 scsi 0 state c xfer 0
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 21 40 00 00 00 80 00 00
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1f 30 00 00 00 80 00 00 length 65536 SMID 1111 terminated ioc 804b (da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 20 b8 00 00 00 80 00 00
loginfo 31130000 scsi 0 state c xfer 0
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 20 38 00 00 00 80 00 00
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1e a8 00 00 00 80 00 00 length 65536 SMID 993 terminated ioc 804b l(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
oginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1d f8 00 00 00 28 00 00 length 20480 SMID 450 terminated ioc 804b l(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1f b0 00 00 00 80 00 00
oginfo 31130000 scsi 0 state c xfer 0
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1f 30 00 00 00 80 00 00
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1c f8 00 00 01 00 00 00 length 131072 SMID 1064 terminated ioc 804b(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1e a8 00 00 00 80 00 00
loginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1b f8 00 00 01 00 00 00 length 131072 SMID 519 terminated ioc 804b (da4:mpr0:0:6:0): CAM status: CCB request completed with an error
loginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1a f8 00 00 01 00 00 00 length 131072 SMID 500 terminated ioc 804b (da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1d f8 00 00 00 28 00 00
loginfo 31130000 scsi 0 state c xfer 0
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1c f8 00 00 01 00 00 00
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 19 f8 00 00 01 00 00 00 length 131072 SMID 349 terminated ioc 804b (da4:mpr0:0:6:0): CAM status: CCB request completed with an error
loginfo 31130000 scsi 0 state c xfer 0
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1b f8 00 00 01 00 00 00
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 18 f8 00 00 01 00 00 00 length 131072 SMID 413 terminated ioc 804b (da4:mpr0:0:6:0): CAM status: CCB request completed with an error
loginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 17 f8 00 00 01 00 00 00 length 131072 SMID 797 terminated ioc 804b (da4:loginfo 31130000 scsi 0 state c xfer 0
mpr0:0:6:0): Retrying command
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 17 38 00 00 00 c0 00 00 length 98304 SMID 439 terminated ioc 804b l(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1a f8 00 00 01 00 00 00
oginfo 31130000 scsi 0 state c xfer 0
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 19 f8 00 00 01 00 00 00
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 16 38 00 00 01 00 00 00 length 131072 SMID 708 terminated ioc 804b (da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:loginfo 31130000 scsi 0 state c xfer 0
mpr0:0:6:0): Retrying command
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 15 38 00 00 01 00 00 00 length 131072 SMID 779 terminated ioc 804b (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 18 f8 00 00 01 00 00 00
loginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 14 38 00 00 01 00 00 00 length 131072 SMID 143 terminated ioc 804b (da4:mpr0:0:6:0): CAM status: CCB request completed with an error
loginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 13 38 00 00 01 00 00 00 length 131072 SMID 523 terminated ioc 804b (da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 17 f8 00 00 01 00 00 00
loginfo 31130000 scsi 0 state c xfer 0
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 12 38 00 00 01 00 00 00 length 131072 SMID 879 terminated ioc 804b (da4:mpr0:0:loginfo 31130000 scsi 0 state c xfer 0
6:0): Retrying command
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 11 38 00 00 01 00 00 00 length 131072 SMID 505 terminated ioc 804b (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 17 38 00 00 00 c0 00 00
loginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 10 38 00 00 01 00 00 00 length 131072 SMID 1032 terminated ioc 804b(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
loginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 0f 78 00 00 00 c0 00 00 length 98304 SMID 544 terminated ioc 804b l(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 16 38 00 00 01 00 00 00
oginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 0e 78 00 00 01 00 00 00 length 131072 SMID 582 terminated ioc 804b (da4:mpr0:0:6:0): CAM status: CCB request completed with an error
loginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 0d 78 00 00 01 00 00 00 length 131072 SMID 823 terminated ioc 804b (da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 15 38 00 00 01 00 00 00
loginfo 31130000 scsi 0 state c xfer 0
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 14 38 00 00 01 00 00 00
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 0c 78 00 00 01 00 00 00 length 131072 SMID 840 terminated ioc 804b (da4:mpr0:0:6:0): CAM status: CCB request completed with an error
loginfo 31130000 scsi 0 state c xfer 0
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 13 38 00 00 01 00 00 00
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 0b 78 00 00 01 00 00 00 length 131072 SMID 935 terminated ioc 804b (da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 12 38 00 00 01 00 00 00
loginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 0a 78 00 00 01 00 00 00 length 131072 SMID 715 terminated ioc 804b (da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 11 38 00 00 01 00 00 00
loginfo 31130000 scsi 0 state c xfer 0
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 10 38 00 00 01 00 00 00
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 09 78 00 00 01 00 00 00 length 131072 SMID 1126 terminated ioc 804b(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
loginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 08 78 00 00 01 00 00 00 length 131072 SMID 1086 terminated ioc 804b(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 0f 78 00 00 00 c0 00 00
loginfo 31130000 scsi 0 state c xfer 0
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 0e 78 00 00 01 00 00 00
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 0d 78 00 00 01 00 00 00
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 0c 78 00 00 01 00 00 00
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 28 4c cc f0 00 00 00 10 00 00 length 8192 SMID 771 terminated ioc 804b lo(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 0b 78 00 00 01 00 00 00
ginfo 31130000 scsi 0 state c xfer 0
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
mpr0: Unfreezing devq for target ID 6
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 0a 78 00 00 01 00 00 00
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 09 78 00 00 01 00 00 00
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 08 78 00 00 01 00 00 00
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 28 4c cc f0 00 00 00 10 00 00
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da4:mpr0:0:6:0): CAM status: Command timeout
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 08 78 00 00 01 00 00 00
(da4:mpr0:0:6:0): CAM status: SCSI Status Error
(da4:mpr0:0:6:0): SCSI status: Check Condition
(da4:mpr0:0:6:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
(da4:mpr0:0:6:0): Retrying command (per sense data)
 

mgittelman

FreeNAS Experienced
Joined
Dec 8, 2017
Messages
171
@mloiterman Have you tried connecting any of your disks to the SATA connectors on that MB? That's one thing I haven't tried yet as mine isn't that unstable. I have a drive dropping out every month or so on average and I just re-join it to the vdev.
 

mloiterman

Newbie
Joined
Jan 30, 2013
Messages
45
@mloiterman Have you tried connecting any of your disks to the SATA connectors on that MB? That's one thing I haven't tried yet as mine isn't that unstable. I have a drive dropping out every month or so on average and I just re-join it to the vdev.
I have not done that and it's a bit difficult due to my chassis. I would have to pull all the drives and lay them out with individual SATA cables. Mine are connected with the SAS backplane in my chassis.
 

pro lamer

FreeNAS Guru
Joined
Feb 16, 2018
Messages
624
lay them out with individual SATA cables.
Alternatively you can use a breakout cable (just make sure if it has to be a reverse breakout cable or a forward one since I can't remember from the top of my head) and connect the 8087 cable connector to the backplane...

Sent from my phone
 

soulburn

FreeNAS Experienced
Joined
Jul 6, 2014
Messages
100
This is so frustrating. Any kind of significant writes are causing these errors.

This basically renders FreeNAS useless to me and I'm sorry I'm continuing to waste my time. I've been using FreeNAS since it was an early fork of M0n0wall and this is ridiculous. So disappointed and frustrated that I cannot rely on this any more.

Code:
(da4:mpr0:0:6:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 length 0 SMID 732 Aborting command 0xfffffe0001090c40
mpr0: Sending reset from mprsas_send_abort for target ID 6
    (pass4:mpr0:0:6:0): ATA COMMAND PASS THROUGH(16). CDB: 85 08 0e 00 d5 00 01 00 00 00 4f 00 c2 00 b0 00 length 512 SMID 185 terminated ioc 804b loginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 21 c0 00 00 00 80 00 00 length 65536 SMID 822 terminated ioc 804b loginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 21 40 00 00 00 80 00 00 length 65536 SMID 666 terminated ioc 804b loginfo 31130000 scsi 0 state c xfer 0
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 21 c0 00 00 00 80 00 00
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 20 b8 00 00 00 80 00 00 length 65536 SMID 320 terminated ioc 804b l(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
oginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 20 38 00 00 00 80 00 00 length 65536 SMID 256 terminated ioc 804b l(da4:oginfo 31130000 scsi 0 state c xfer 0
mpr0:0:6:0): Retrying command
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1f b0 00 00 00 80 00 00 length 65536 SMID 694 terminated ioc 804b loginfo 31130000 scsi 0 state c xfer 0
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 21 40 00 00 00 80 00 00
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1f 30 00 00 00 80 00 00 length 65536 SMID 1111 terminated ioc 804b (da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 20 b8 00 00 00 80 00 00
loginfo 31130000 scsi 0 state c xfer 0
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 20 38 00 00 00 80 00 00
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1e a8 00 00 00 80 00 00 length 65536 SMID 993 terminated ioc 804b l(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
oginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1d f8 00 00 00 28 00 00 length 20480 SMID 450 terminated ioc 804b l(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1f b0 00 00 00 80 00 00
oginfo 31130000 scsi 0 state c xfer 0
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1f 30 00 00 00 80 00 00
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1c f8 00 00 01 00 00 00 length 131072 SMID 1064 terminated ioc 804b(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1e a8 00 00 00 80 00 00
loginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1b f8 00 00 01 00 00 00 length 131072 SMID 519 terminated ioc 804b (da4:mpr0:0:6:0): CAM status: CCB request completed with an error
loginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1a f8 00 00 01 00 00 00 length 131072 SMID 500 terminated ioc 804b (da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1d f8 00 00 00 28 00 00
loginfo 31130000 scsi 0 state c xfer 0
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1c f8 00 00 01 00 00 00
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 19 f8 00 00 01 00 00 00 length 131072 SMID 349 terminated ioc 804b (da4:mpr0:0:6:0): CAM status: CCB request completed with an error
loginfo 31130000 scsi 0 state c xfer 0
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1b f8 00 00 01 00 00 00
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 18 f8 00 00 01 00 00 00 length 131072 SMID 413 terminated ioc 804b (da4:mpr0:0:6:0): CAM status: CCB request completed with an error
loginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 17 f8 00 00 01 00 00 00 length 131072 SMID 797 terminated ioc 804b (da4:loginfo 31130000 scsi 0 state c xfer 0
mpr0:0:6:0): Retrying command
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 17 38 00 00 00 c0 00 00 length 98304 SMID 439 terminated ioc 804b l(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 1a f8 00 00 01 00 00 00
oginfo 31130000 scsi 0 state c xfer 0
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 19 f8 00 00 01 00 00 00
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 16 38 00 00 01 00 00 00 length 131072 SMID 708 terminated ioc 804b (da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:loginfo 31130000 scsi 0 state c xfer 0
mpr0:0:6:0): Retrying command
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 15 38 00 00 01 00 00 00 length 131072 SMID 779 terminated ioc 804b (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 18 f8 00 00 01 00 00 00
loginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 14 38 00 00 01 00 00 00 length 131072 SMID 143 terminated ioc 804b (da4:mpr0:0:6:0): CAM status: CCB request completed with an error
loginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 13 38 00 00 01 00 00 00 length 131072 SMID 523 terminated ioc 804b (da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 17 f8 00 00 01 00 00 00
loginfo 31130000 scsi 0 state c xfer 0
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 12 38 00 00 01 00 00 00 length 131072 SMID 879 terminated ioc 804b (da4:mpr0:0:loginfo 31130000 scsi 0 state c xfer 0
6:0): Retrying command
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 11 38 00 00 01 00 00 00 length 131072 SMID 505 terminated ioc 804b (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 17 38 00 00 00 c0 00 00
loginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 10 38 00 00 01 00 00 00 length 131072 SMID 1032 terminated ioc 804b(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
loginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 0f 78 00 00 00 c0 00 00 length 98304 SMID 544 terminated ioc 804b l(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 16 38 00 00 01 00 00 00
oginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 0e 78 00 00 01 00 00 00 length 131072 SMID 582 terminated ioc 804b (da4:mpr0:0:6:0): CAM status: CCB request completed with an error
loginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 0d 78 00 00 01 00 00 00 length 131072 SMID 823 terminated ioc 804b (da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 15 38 00 00 01 00 00 00
loginfo 31130000 scsi 0 state c xfer 0
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 14 38 00 00 01 00 00 00
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 0c 78 00 00 01 00 00 00 length 131072 SMID 840 terminated ioc 804b (da4:mpr0:0:6:0): CAM status: CCB request completed with an error
loginfo 31130000 scsi 0 state c xfer 0
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 13 38 00 00 01 00 00 00
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 0b 78 00 00 01 00 00 00 length 131072 SMID 935 terminated ioc 804b (da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 12 38 00 00 01 00 00 00
loginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 0a 78 00 00 01 00 00 00 length 131072 SMID 715 terminated ioc 804b (da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 11 38 00 00 01 00 00 00
loginfo 31130000 scsi 0 state c xfer 0
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 10 38 00 00 01 00 00 00
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 09 78 00 00 01 00 00 00 length 131072 SMID 1126 terminated ioc 804b(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
loginfo 31130000 scsi 0 state c xfer 0
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 08 78 00 00 01 00 00 00 length 131072 SMID 1086 terminated ioc 804b(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 0f 78 00 00 00 c0 00 00
loginfo 31130000 scsi 0 state c xfer 0
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 0e 78 00 00 01 00 00 00
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 0d 78 00 00 01 00 00 00
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 0c 78 00 00 01 00 00 00
    (da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 28 4c cc f0 00 00 00 10 00 00 length 8192 SMID 771 terminated ioc 804b lo(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 0b 78 00 00 01 00 00 00
ginfo 31130000 scsi 0 state c xfer 0
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
mpr0: Unfreezing devq for target ID 6
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 0a 78 00 00 01 00 00 00
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 09 78 00 00 01 00 00 00
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 08 78 00 00 01 00 00 00
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 28 4c cc f0 00 00 00 10 00 00
(da4:mpr0:0:6:0): CAM status: CCB request completed with an error
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da4:mpr0:0:6:0): CAM status: Command timeout
(da4:mpr0:0:6:0): Retrying command
(da4:mpr0:0:6:0): WRITE(16). CDB: 8a 00 00 00 00 01 95 9a 08 78 00 00 01 00 00 00
(da4:mpr0:0:6:0): CAM status: SCSI Status Error
(da4:mpr0:0:6:0): SCSI status: Check Condition
(da4:mpr0:0:6:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
(da4:mpr0:0:6:0): Retrying command (per sense data)
You posted in another thread where I said the same thing: LSI (Avago) 9207-8i with Seagate 10TB Enterprise (ST10000NM0016).

I'm at the point where I ordered another HBA to test it out such as an SAS 3200 based card (SAS 3224 and SAS 3216). I'll try to remember to report back with my findings as to if this helps because I have no other ideas at this point.
 

DGenerateKane

Newbie
Joined
Sep 4, 2014
Messages
88
I completed testing the WD White drive, and will have to replace the Ironwolf drives with them unless somebody else finds an HBA that doesn't hate them. I was able to transfer ~8tb of data back and forth from my old server to the White drive that is in my current problematic server a half dozen times without issue. When I first set the new machine up, I had a hell of a time transferring large chunks of data, basically anything over 1tb would choke and I'd have to cancel the transfer and start over again. I also frequently had one or more drives error when transferring that much data as well on a regular basis. Yet it didn't happen once with the WD White. I really don't want to drop two grand on replacement drives. This sucks.
 

BlackMoonWolf

Newbie
Joined
Dec 26, 2016
Messages
11
I'm going to throw in on this. I've been beating my head against this stupid issue for the last 36 freakin' hours. I've had a ridiculous streak of issues with my server - finally resulting in a motherboard replacement. I moved FreeNAS over to an R710 while I waited for the new Dell R720 V7 motherboard to arrive.

I put it all back together in my R720 chassis thinking, "FINALLY... I'll have everything working again!"

Yeah. RIGHT.

Now I'm met with these "CAM status: CCB request completed with an error" messages littering dmesg and console - on ALL FOUR of my large drives. (Although my two SSDs aren't having any issues...)

I know that I didn't just experience a failure on 4 drives at the same time. I replaced da0, resilvered and still getting these messages so I KNOW the drives are NOT the issue.

I've got a SAS9300-8i. Firmware in every point of the system is up-to-date with latest available.

I've flip-flopped my SAS cables. I've moved the controller between 3 different slots. I've changed power settings all the way from conservative to maximum performance in the BIOS. I've enabled/disabled features in numerous combinations of on and off. I've moved my drives to the farthest slots away from their original position. I've booted between 4 different boot environments. I've even backed up, deleted, recreated and re-migrated all of my data to the drives again.

NOTHING CHANGES THIS PROBLEM. The issues seem to scroll the console exponentially faster when you do a larger number of writes to the pool.

I'm about to lose my mind when I try to use my console.

WHAT is going ON here?!?!?
 

BlackMoonWolf

Newbie
Joined
Dec 26, 2016
Messages
11

mloiterman

Newbie
Joined
Jan 30, 2013
Messages
45
For anyone that cares, I've given up on FreeNAS and have purchased a Synology RS1219+. After three weeks, I have thrown everything I can at it and so far it's been absolutely rock solid.

I'm certain that's not the answer anyone is looking for and it was by far the most expensive way out, but I just don't have the time or energy to wrestle with this any more.
 

soulburn

FreeNAS Experienced
Joined
Jul 6, 2014
Messages
100
Did you happen to find out anything?
Yes but I changed too many things to be of any meaningful help. I did purchase an SAS 3200 card, updated the firmware, and put it in a new, different Supermicro motherboard and it is rock solid now.
 

cobrakiller58

FreeNAS Experienced
Joined
Jan 18, 2017
Messages
321
@soulburn what is your current setup that is working properly? Any information on functioning configurations I would consider meaningful.
 

soulburn

FreeNAS Experienced
Joined
Jul 6, 2014
Messages
100
@soulburn what is your current setup that is working properly? Any information on functioning configurations I would consider meaningful.
@cobrakiller58 So as far as pertinent details go, the box in question is a Supermicro SuperStorage 6048R-E1CR36L server. It includes an X10DRH-iT motherboard and an AOC-S3008L-L8e HBA (which is an SAS3 via Broadcom 3008 controller in IT mode). I removed the AOC-S3008L-L8e HBA and purchased and installed an SAS 9305-16i. I updated the firmware for the HBA from the Broadcom site. The only other thing I did was replace my Mellanox ConnectX-3 (MCX354A-FCBT) NIC with a Chelsio T580-LP-CR as I was getting some messages about errors from the NIC in the console on occasion and I had to manually set it to Ethernet as it would default to Infiniband.

I thought it was a long shot to replace the HBA because they both use the same drivers but once I did this all the errors stopped and dropped drives went away. I don't think the NIC replacement had anything to do with it but at this point who knows.
 

El Al

Newbie
Joined
Oct 30, 2016
Messages
10
Try this:

Disable NCQ on the seagate drives and you will find that the problem disappears!

camcontrol tags daX -N 1

It took me way to long to find the solution.
 

Pheran

FreeNAS Experienced
Joined
Jul 14, 2015
Messages
261
Try this:

Disable NCQ on the seagate drives and you will find that the problem disappears!

camcontrol tags daX -N 1

It took me way to long to find the solution.
I've been out of touch with this thread for a while but if this works I'm going to have to do some kind of dance. Do you know if there's a performance hit for disabling NCQ? Also, where did you find this solution?
 

El Al

Newbie
Joined
Oct 30, 2016
Messages
10
No obvious performance hit in my day to day use. But based on the nature of the feature that will be disabled i would not be surprised. Certainly will increase wear on your drives and if you are at the cutting edge of performance it might impact you. Easy enough to test and I don't see any potential harm.

I was suffering from all kinds of scsi status cam problems that would happen intermittently with no clear pattern in terms of load or time or scripts going off at the time. This would eventually lead to all the problems described in this thread and more. Degradation etc etc

The discussions on this related to freenas were all circling around hardware, cables, hbas, ram, psu everything under the sun except for freenas having a problem itself. (Or seagate having a problem with freenas...) My wd reds in the same server have never shown any issues.

So i got ready to switch to zol and did some research. Search for that issue on their mailing lists and you will find people that have identified that issue a long time ago and found that disabling ncq will get rid of the errors. If you spend some time on reading those you will find its a very different community over there. None of that toxic low skilled seniority stuff that's going on here but i digress... I vaguely remember someone in the mailing list mentioning that they had no performance hit using this.

Obviously I take no credit for this solution!

I have a script that runs on startup to make sure all installed seagate drives have ncq disabled and that has been enough for me. The issues have disappeared completely. Smarter people than me could probably use this as a starting point to get to the bottom of it. Personally I will never buy another seagate disk in my life and thats the end of it. This was by far the most frustrating issue i have dealt with concerning freenas. I saw a thread on reddit mentioning this thread which i had also seen before and thought i share what i had found because i know how annoying it is. I usually avoid coming here.

the script:

#!/usr/local/bin/bash

for i in `camcontrol devlist | grep "ST14" | cut -d"," -f2 | cut -d")" -f1` ; do camcontrol tags $i -N 1 ; done

simple enough!
 

Pheran

FreeNAS Experienced
Joined
Jul 14, 2015
Messages
261
Thanks very much for your help. I have manually disabled NCQ on my 6 10TB Ironwolfs but it's too soon to tell if the problem is fixed yet. Usually I lose a drive a least once a week (sometimes one a day), but it will probably take a month to be certain of the fix.

I'm totally with you on the never buying Seagate again. This problem has been a big thorn in my side for 2 years now.
 

Pheran

FreeNAS Experienced
Joined
Jul 14, 2015
Messages
261
Sadly, this fix does not appear to work. I lost a drive today with NCQ disabled.

Code:
# camcontrol tag da6 -v
(pass6:mpr0:0:6:0): dev_openings  1
(pass6:mpr0:0:6:0): dev_active    0
(pass6:mpr0:0:6:0): allocated     0
(pass6:mpr0:0:6:0): queued        0
(pass6:mpr0:0:6:0): held          0
(pass6:mpr0:0:6:0): mintags       2
(pass6:mpr0:0:6:0): maxtags       255

Aug  1 03:18:40 vault   (da6:mpr0:0:6:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00 length 0 SMID 975 Aborting command 0xfffffe00010a6990
Aug  1 03:18:40 vault mpr0: Sending reset from mprsas_send_abort for target ID 6
Aug  1 03:18:40 vault mpr0: Unfreezing devq for target ID 6
Aug  1 03:18:40 vault (da6:mpr0:0:6:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
Aug  1 03:18:40 vault (da6:mpr0:0:6:0): CAM status: Command timeout
Aug  1 03:18:40 vault (da6:mpr0:0:6:0): Retrying command
Aug  1 03:18:40 vault (da6:mpr0:0:6:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
Aug  1 03:18:40 vault (da6:mpr0:0:6:0): CAM status: SCSI Status Error
Aug  1 03:18:40 vault (da6:mpr0:0:6:0): SCSI status: Check Condition
Aug  1 03:18:40 vault (da6:mpr0:0:6:0): SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)
Aug  1 03:18:40 vault (da6:mpr0:0:6:0): Error 6, Retries exhausted
Aug  1 03:18:40 vault (da6:mpr0:0:6:0): Invalidating pack
Aug  1 03:18:40 vault ZFS: vdev state changed, pool_guid=8222513563341353211 vdev_guid=2829914938609022262
Aug  1 03:18:40 vault ZFS: vdev state changed, pool_guid=8222513563341353211 vdev_guid=2829914938609022262
 
Top