Console showing CAM status: SCSI Status Error -- SCSI status: OK

[12pharoh21]

Hello all,

I am not new to TrueNAS/FreeNAS and have another server that I have been running for several years. Getting this new server setup has been a challenge so far though. My first server started on FreeNAS 9 (if I remember correct, might have been 8), and has been upgraded and is now running TrueNAS-12.0-U6.

I have done a ton of searching and can't really find anything like what I'm seeing. I am having a two fold problem with my TrueNAS install. This is a brand new setup that I am just getting started. Here is my server config:

TrueNAS-12.0-U7
MB - ASUS SaberTooth X58
CPU - Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz
RAM - 24GB
LSI MegaRAID SAS 9361-4i
LSI LSI00412 0.6 Meter Internal Cable SFF8643 to x4 SAS8482
-Disks - 4x8TB HGST SAS NAS drives in RAIDZ1

The first part of this is I was able to get the TrueNAS server setup fine, but when I rebooted the machine it goes through a process of doing disk reads from sectors and took almost 2 hours to complete. If I power off the server and disconnect the SAS cable from the LSI card then boot it goes right into the TrueNAS GUI on the console. With the cable connected it does the read from sectors thing. (I can get an output of this data, but want to follow up on the second part first before I reboot again)

I let that complete and was able to get into the GUI using the IP address so I could start the configration, but as soon as I add the pool to the config in TrueNAS the console continually shows the following

Code:

(da2:mrsas0:1:10:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da2:mrsas0:1:10:0): CAM status: SCSI Status Error
(da2:mrsas0:1:10:0): SCSI status: OK
(da2:mrsas0:1:10:0): WRITE(10). CDB: 2a 00 00 41 76 d0 00 00 08 00
(da2:mrsas0:1:10:0): CAM status: SCSI Status Error
(da2:mrsas0:1:10:0): SCSI status: OK
(da1:mrsas0:1:9:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da1:mrsas0:1:9:0): CAM status: SCSI Status Error
(da1:mrsas0:1:9:0): SCSI status: OK
(da2:mrsas0:1:10:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da2:mrsas0:1:10:0): CAM status: SCSI Status Error
(da2:mrsas0:1:10:0): SCSI status: OK
(da2:mrsas0:1:10:0): WRITE(10). CDB: 2a 00 00 40 02 18 00 00 08 00
(da2:mrsas0:1:10:0): CAM status: SCSI Status Error
(da2:mrsas0:1:10:0): SCSI status: OK
(da1:mrsas0:1:9:0): WRITE(10). CDB: 2a 00 00 40 02 18 00 00 08 00
(da1:mrsas0:1:9:0): CAM status: SCSI Status Error
(da1:mrsas0:1:9:0): SCSI status: OK
(da1:mrsas0:1:9:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da1:mrsas0:1:9:0): CAM status: SCSI Status Error
(da1:mrsas0:1:9:0): SCSI status: OK
(da1:mrsas0:1:9:0): WRITE(10). CDB: 2a 00 00 c1 4b 58 00 00 08 00
(da1:mrsas0:1:9:0): CAM status: SCSI Status Error
(da1:mrsas0:1:9:0): SCSI status: OK
(da3:mrsas0:1:11:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da3:mrsas0:1:11:0): CAM status: SCSI Status Error
(da3:mrsas0:1:11:0): SCSI status: OK
(da3:mrsas0:1:11:0): WRITE(10). CDB: 2a 00 00 40 02 28 00 00 08 00
(da3:mrsas0:1:11:0): CAM status: SCSI Status Error
(da3:mrsas0:1:11:0): SCSI status: OK
(da0:mrsas0:1:8:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da0:mrsas0:1:8:0): CAM status: SCSI Status Error
(da0:mrsas0:1:8:0): SCSI status: OK
(da0:mrsas0:1:8:0): WRITE(10). CDB: 2a 00 00 40 02 38 00 00 08 00
(da0:mrsas0:1:8:0): CAM status: SCSI Status Error
(da0:mrsas0:1:8:0): SCSI status: OK
(da0:mrsas0:1:8:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da0:mrsas0:1:8:0): CAM status: SCSI Status Error
(da0:mrsas0:1:8:0): SCSI status: OK
(da0:mrsas0:1:8:0): WRITE(10). CDB: 2a 00 00 41 78 48 00 00 08 00
(da0:mrsas0:1:8:0): CAM status: SCSI Status Error
(da0:mrsas0:1:8:0): SCSI status: OK
(da3:mrsas0:1:11:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da3:mrsas0:1:11:0): CAM status: SCSI Status Error
(da3:mrsas0:1:11:0): SCSI status: OK
(da0:mrsas0:1:8:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da0:mrsas0:1:8:0): CAM status: SCSI Status Error
(da0:mrsas0:1:8:0): SCSI status: OK
(da3:mrsas0:1:11:0): WRITE(10). CDB: 2a 00 00 40 02 40 00 00 08 00
(da3:mrsas0:1:11:0): CAM status: SCSI Status Error
(da3:mrsas0:1:11:0): SCSI status: OK
(da0:mrsas0:1:8:0): WRITE(10). CDB: 2a 00 00 40 02 40 00 00 08 00
(da0:mrsas0:1:8:0): CAM status: SCSI Status Error
(da0:mrsas0:1:8:0): SCSI status: OK
(da0:mrsas0:1:8:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da0:mrsas0:1:8:0): CAM status: SCSI Status Error
(da0:mrsas0:1:8:0): SCSI status: OK
(da0:mrsas0:1:8:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da0:mrsas0:1:8:0): CAM status: SCSI Status Error
(da0:mrsas0:1:8:0): SCSI status: OK
(da0:mrsas0:1:8:0): WRITE(10). CDB: 2a 00 00 40 01 b8 00 00 08 00
(da0:mrsas0:1:8:0): CAM status: SCSI Status Error
(da0:mrsas0:1:8:0): SCSI status: OK
(da0:mrsas0:1:8:0): SYNCHRONIZE CACHE(10). CDB: 35 00 00 00 00 00 00 00 00 00
(da0:mrsas0:1:8:0): CAM status: SCSI Status Error
(da0:mrsas0:1:8:0): SCSI status: OK
(da0:mrsas0:1:8:0): WRITE(10). CDB: 2a 00 00 41 79 60 00 00 08 00
(da0:mrsas0:1:8:0): CAM status: SCSI Status Error
(da0:mrsas0:1:8:0): SCSI status: OK

At first I thought it was doing a scrub of the pool, but it's been running for almost 24 hours at this point and everything in the GUI is showing stable and OK. I have not rebooted the server since I added the pool and it started doing this because I wanted to see if it would stop on it's own.

Also of note, I can modify the configuration of the TrueNAS server without issue, I have been able to add a dataset to the pool and was able to add an SMB share to that as well as an NFS share. There are some issues with NFS share, but I will probably start a new thread for that one once this is resolved. Let me know if you need any other info from my system or if you have suggestions on what to try next. Thanks in advance for your help.

 

[Ericloewe]

Your first problem is that you're using a RAID controller. There may be more, but that's something to review afterwards.

 

[12pharoh21]

Your first problem is that you're using a RAID controller. There may be more, but that's something to review afterwards.

The RAID controller has all disks set to JBOD mode, and they are all detected and working in TrueNAS, so why would that be a problem?

 

[Samuel Tai]

What's all the noise about HBA's, and why can't I use a RAID controller?

This is relevant to FreeNAS and TrueNAS CORE. Some parts of it might also be relevant to Scale, but I don't really know how reliable the Linux drivers are. 1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with...

www.truenas.com

 

[12pharoh21]

What's all the noise about HBA's, and why can't I use a RAID controller?

This is relevant to FreeNAS and TrueNAS CORE. Some parts of it might also be relevant to Scale, but I don't really know how reliable the Linux drivers are. 1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with...

www.truenas.com

I reviewed that, but based on my research into the RAID card I'm using, it's acting like it should be unless I'm missing something. I am able to see the disk information just fine when I run "camcontrol devlist"

Code:

<HGST HUS728T8TAL5204 HC23>        at scbus9 target 8 lun 0 (pass0,da0)
<HGST HUS728T8TAL5204 HC23>        at scbus9 target 9 lun 0 (pass1,da1)
<HGST HUS728T8TAL5204 HC23>        at scbus9 target 10 lun 0 (pass2,da2)
<HGST HUS728T8TAL5204 HC23>        at scbus9 target 11 lun 0 (pass3,da3)
<OCZ-AGILITY3 2.22>                at scbus12 target 0 lun 0 (pass4,ada0)
<Optiarc DVD RW AD-7200S 1.06>     at scbus13 target 0 lun 0 (cd0,pass5)
<AHCI SGPIO Enclosure 2.00 0001>   at scbus18 target 0 lun 0 (pass6,ses0)

And I can see the SMART data when I run "smartctl -i /dev/da0" unless I'm reading something wrong here.

Code:

smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p11 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               HGST
Product:              HUS728T8TAL5204
Revision:             HC23
Compliance:           SPC-4
User Capacity:        8,001,563,222,016 bytes [8.00 TB]
Logical block size:   512 bytes
Physical block size:  4096 bytes
LU is fully provisioned
Rotation Rate:        7200 rpm
Form Factor:          3.5 inches
Logical Unit id:      0x5000cca0bb53cc08
Serial number:        
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Mon Dec 20 10:49:16 2021 MST
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Disabled or Not Supported

I would really prefer not to have to buy another card, but I can if I absolutely have to. From everything I'm seeing, this card should be handling everything the way I need it to, but I'm not sure why I'm constantly seeing the messages from my first post in the console.

 

[Ericloewe]

Well, selling the RAID controller and buying an HBA would be a net profit. But anyway, what might we be dealing with here? All disks are complaining. Is the SMART data for these disks good? What happens if you just use the Intel SATA ports? It looks to me like you can just use those instead of the SAS controller, too.
Of course, updating the card's firmware should be high on the list of things to do, as well.

 

[12pharoh21]

Well, selling the RAID controller and buying an HBA would be a net profit. But anyway, what might we be dealing with here? All disks are complaining. Is the SMART data for these disks good? What happens if you just use the Intel SATA ports? It looks to me like you can just use those instead of the SAS controller, too.
Of course, updating the card's firmware should be high on the list of things to do, as well.

These are SAS disks, so I can't use the internal SATA ports on the MB. I will work on upgrading the firmware on the card, and look at getting a new HBA card.

That being said, based off what I have posted, this card should at least work for the time being, correct? It's passing through the disks as I expect it to, but I'm not sure why I'm seeing all the messages and stuff that I'm seeing. I will try a reboot and see if I can grab the data that is displayed during the boot process. This should all resolve when I swap to the HBA card, correct?

 

[Ericloewe]

This should all resolve when I swap to the HBA card, correct?

I would hope so, but it's hard to definitively say so without testing the remaining components. If the errors are gone with updated firmware, that would bode well for the HBA as well.

As a sidenote: why SAS HDDs?

 

[12pharoh21]

I would hope so, but it's hard to definitively say so without testing the remaining components. If the errors are gone with updated firmware, that would bode well for the HBA as well.

As a sidenote: why SAS HDDs?

They were originally supposed to go in my Synology NAS, but I didn't do enough research to know that it didn't support SAS HDDs, and SAS drives have the 12Gb/s speeds that you can't get with SATA. When I went to return them, it would have cost me almost $150 in restocking fees, so I decided to keep them and use them in another application. I was originally going to use these drives and card in another type of system, but I found I like TrueNAS better anyway, so I went that route... and here we are now. LOL

I will report back after I get the firmware updated and let you know how things are at that point.

I'm having another issue with an NFS share, but will open a new thread for that one.

 

[Ericloewe]

Well, 12 Gb/s isn't very relevant with spinning rust. Even 6 Gb/s is overkill. The only advantage is with expanders, to reduce contention, and even then the disks themselves can be SATA with no real downside (caveat: some fancy self-encrypting options are SAS-only, as are selectable sector sizes).

 

[12pharoh21]

Well, 12 Gb/s isn't very relevant with spinning rust. Even 6 Gb/s is overkill. The only advantage is with expanders, to reduce contention, and even then the disks themselves can be SATA with no real downside (caveat: some fancy self-encrypting options are SAS-only, as are selectable sector sizes).

Gotcha, well it seems I don't know as much about hardware as I thought I did... LOL... At the same time, I didn't want to send them back if I could ultimately use them, so I'm not too hurt by it.

I'll upgrade the firmware, and keep an eye on the system to make sure everything keeps working. I'll let you know once I get the new card and get it replaced to make sure everything is working better. Thank you kindly for your help so far, it is much appreciated.

 

[12pharoh21]

OK, so the LSI SAS9300-8i HBA card got her WAY faster than the estimated shipping date so I got it installed today and fully upgraded to the latest version of firmware and BIOS. Even after getting the new card put in place, I'm still getting errors on the console, and they are different than before.

Code:

(da3:mpr0:0:3:0): READ(10). CDB: 28 00 00 40 00 7f 00 00 01 00
(da3:mpr0:0:3:0): CAM status: SCSI Status Error
(da3:mpr0:0:3:0): SCSI status: Check Condition
(da3:mpr0:0:3:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da3:mpr0:0:3:0): Info: 0x40007f
(da3:mpr0:0:3:0): Actual Retry Count: 1382
(da3:mpr0:0:3:0): Descriptor 0x80: f7 2d 00 00 02 1e 00 00 00 00 00 00 00 00
(da3:mpr0:0:3:0): Descriptor 0x81: 00 00 9d 02 02 1d
(da3:mpr0:0:3:0): Error 5, Unretryable error
(da3:mpr0:0:3:0): READ(10). CDB: 28 00 00 40 00 7e 00 00 01 00
(da3:mpr0:0:3:0): CAM status: SCSI Status Error
(da3:mpr0:0:3:0): SCSI status: Check Condition
(da3:mpr0:0:3:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da3:mpr0:0:3:0): Info: 0x40007e
(da3:mpr0:0:3:0): Actual Retry Count: 1382
(da3:mpr0:0:3:0): Descriptor 0x80: f7 2d 00 00 02 1e 00 00 00 00 00 00 00 00
(da3:mpr0:0:3:0): Descriptor 0x81: 00 00 9d 02 02 1d
(da3:mpr0:0:3:0): Error 5, Unretryable error
(da3:mpr0:0:3:0): READ(10). CDB: 28 00 00 40 00 7f 00 00 01 00
(da3:mpr0:0:3:0): CAM status: SCSI Status Error
(da3:mpr0:0:3:0): SCSI status: Check Condition
(da3:mpr0:0:3:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da3:mpr0:0:3:0): Info: 0x40007f
(da3:mpr0:0:3:0): Actual Retry Count: 1382
(da3:mpr0:0:3:0): Descriptor 0x80: f7 2d 00 00 02 1e 00 00 00 00 00 00 00 00
(da3:mpr0:0:3:0): Descriptor 0x81: 00 00 9d 02 02 1d
(da3:mpr0:0:3:0): Error 5, Unretryable error
(da3:mpr0:0:3:0): READ(10). CDB: 28 00 00 40 00 7f 00 00 01 00
(da3:mpr0:0:3:0): CAM status: SCSI Status Error
(da3:mpr0:0:3:0): SCSI status: Check Condition
(da3:mpr0:0:3:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da3:mpr0:0:3:0): Info: 0x40007f
(da3:mpr0:0:3:0): Actual Retry Count: 1382
(da3:mpr0:0:3:0): Descriptor 0x80: f7 2d 00 00 02 1e 00 00 00 00 00 00 00 00
(da3:mpr0:0:3:0): Descriptor 0x81: 00 00 9d 02 02 1d
(da3:mpr0:0:3:0): Error 5, Unretryable error
(da3:mpr0:0:3:0): READ(10). CDB: 28 00 00 40 00 7f 00 00 01 00
(da3:mpr0:0:3:0): CAM status: SCSI Status Error
(da3:mpr0:0:3:0): SCSI status: Check Condition
(da3:mpr0:0:3:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da3:mpr0:0:3:0): Info: 0x40007f
(da3:mpr0:0:3:0): Actual Retry Count: 1382
(da3:mpr0:0:3:0): Descriptor 0x80: f7 2d 00 00 02 1e 00 00 00 00 00 00 00 00
(da3:mpr0:0:3:0): Descriptor 0x81: 00 00 9d 02 02 1d
(da3:mpr0:0:3:0): Error 5, Unretryable error
(da3:mpr0:0:3:0): READ(10). CDB: 28 00 00 40 00 7f 00 00 01 00
(da3:mpr0:0:3:0): CAM status: SCSI Status Error
(da3:mpr0:0:3:0): SCSI status: Check Condition
(da3:mpr0:0:3:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da3:mpr0:0:3:0): Info: 0x40007f
(da3:mpr0:0:3:0): Actual Retry Count: 1382
(da3:mpr0:0:3:0): Descriptor 0x80: f7 2d 00 00 02 1e 00 00 00 00 00 00 00 00
(da3:mpr0:0:3:0): Descriptor 0x81: 00 00 9d 02 02 1d
(da3:mpr0:0:3:0): Error 5, Unretryable error

The disks seems to be reporting fine

Code:

root@headsh0t-fs03[~]# camcontrol devlist
<HGST HUS728T8TAL5204 HC23>        at scbus8 target 0 lun 0 (pass0,da0)
<HGST HUS728T8TAL5204 HC23>        at scbus8 target 1 lun 0 (pass1,da1)
<HGST HUS728T8TAL5204 HC23>        at scbus8 target 2 lun 0 (pass2,da2)
<HGST HUS728T8TAL5204 HC23>        at scbus8 target 3 lun 0 (pass3,da3)
<OCZ-AGILITY3 2.22>                at scbus11 target 0 lun 0 (pass4,ada0)
<Optiarc DVD RW AD-7200S 1.06>     at scbus12 target 0 lun 0 (cd0,pass5)
<AHCI SGPIO Enclosure 2.00 0001>   at scbus17 target 0 lun 0 (pass6,ses0)

SMART of the first drive seems fine as well, but I'm seeing the errors above on all 4 drives

Code:

smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p11 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               HGST
Product:              HUS728T8TAL5204
Revision:             HC23
Compliance:           SPC-4
User Capacity:        8,001,563,222,016 bytes [8.00 TB]
Logical block size:   512 bytes
Physical block size:  4096 bytes
LU is fully provisioned
Rotation Rate:        7200 rpm
Form Factor:          3.5 inches
Logical Unit id:      0x5000cca0bb527400
Serial number:       
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Thu Dec 23 19:18:20 2021 MST
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Disabled or Not Supported

Any ideas why I'm seeing those lines in my console?

 

[12pharoh21]

When I reboot the TrueNAS server it is taking forever to come up. It's currently been running for almost an hour and what I'm seeing on the console is

Code:

disk1: Read 32 sector(s) from xxxxxxx to 0xxxxxx (0x00000): 0x10
disk2: Read 32 sector(s) from xxxxxxx to 0xxxxxx (0x00000): 0x10

It is doing this for all 4 disks, and looks to be going through all of the sectors, but seems to be repeating. Looking at it right now, I'm seeing the following repeating over and over on disk2.

Code:

disk2: Read 32 sector(s) from 4195432 to 0x576b4 (0x616b4): 0x10
disk2: Read 32 sector(s) from 4195440 to 0x576b4 (0x616b4): 0x10
disk2: Read 32 sector(s) from 4195448 to 0x576b4 (0x616b4): 0x10
disk2: Read 32 sector(s) from 4195456 to 0x576b4 (0x616b4): 0x10

It's incrementing, but it's taking forever and seems to be looping right now. I can't access the TrueNAS GUI and I can't interact with the console at all. Anyone seen this before? It finally just moved on to disk4 which should be the final one, but this is taking way too long for normal operation. My other TrueNAS server doesn't do this, but it's also using the SATA connections on the motherboard. Can't use the SATA connections on this one as these are SAS drives.

 

[Ericloewe]

MART of the first drive seems fine as well, but I'm seeing the errors above on all 4 drives

Is that smartctl -x /dev/da0 ?

 

[12pharoh21]

Is that smartctl -x /dev/da0 ?

No, it was smartctl -a /dev/da0. The output of smartctl -x /dev/da0 is

Code:

smartctl -x /dev/da0
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p11 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               HGST
Product:              HUS728T8TAL5204
Revision:             HC23
Compliance:           SPC-4
User Capacity:        8,001,563,222,016 bytes [8.00 TB]
Logical block size:   512 bytes
Physical block size:  4096 bytes
LU is fully provisioned
Rotation Rate:        7200 rpm
Form Factor:          3.5 inches
Logical Unit id:      0x5000cca0bb527400
Serial number:      
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Fri Dec 24 02:50:19 2021 MST
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Disabled or Not Supported
Read Cache is:        Enabled
Writeback Cache is:   Disabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Grown defects during certification = 0
Total blocks reassigned during format = 0
Total new blocks reassigned = 0
Power on minutes since format = 7664
Current Drive Temperature:     33 C
Drive Trip Temperature:        85 C

Manufactured in week 03 of year 2020
Specified cycle count over device lifetime:  50000
Accumulated start-stop cycles:  39
Specified load-unload count over device lifetime:  600000
Accumulated load-unload cycles:  40
Elements in grown defect list: 0

Vendor (Seagate Cache) information
  Blocks sent to initiator = 15840107823104

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0        0         0         0    1056144          0.334        5501
write:         0        0         0         0        519          8.841           0
verify:        0        0         0         0     140530          0.009           0

Non-medium error count:       46

SMART Self-test log
Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]
     Description                              number   (hours)
# 1  Background long   Failed in segment -->       6      43        4070793720 [0x3 0x5d 0x1]

Long (extended) Self-test duration: 59271 seconds [987.9 minutes]

Background scan results log
  Status: halted - vendor specific cause
    Accumulated power on time, hours:minutes 127:46 [7666 minutes]
    Number of background scans performed: 0,  scan progress: 0.00%
    Number of background medium scans performed: 0

   #  when        lba(hex)    [sk,asc,ascq]    reassign_status
   1  120:33  0000000000014848  [3,11,0]   Require Write or Reassign Blocks command
   2  120:33  0000000000014840  [3,11,0]   Require Write or Reassign Blocks command
   3  120:33  0000000000014838  [3,11,0]   Require Write or Reassign Blocks command
   4  120:33  0000000000014830  [3,11,0]   Require Write or Reassign Blocks command
   5  120:33  0000000000014828  [3,11,0]   Require Write or Reassign Blocks command
   6  120:33  0000000000014820  [3,11,0]   Require Write or Reassign Blocks command
   7  120:33  0000000000014818  [3,11,0]   Require Write or Reassign Blocks command
   8  120:32  0000000000014810  [3,11,0]   Require Write or Reassign Blocks command
   9  120:32  0000000000014808  [3,11,0]   Require Write or Reassign Blocks command
  10  120:32  0000000000014800  [3,11,0]   Require Write or Reassign Blocks command
  11  120:32  00000000000147f8  [3,11,0]   Require Write or Reassign Blocks command
  12  120:32  00000000000147f0  [3,11,0]   Require Write or Reassign Blocks command
  13  120:32  00000000000147e8  [3,11,0]   Require Write or Reassign Blocks command
  14  120:32  00000000000147e0  [3,11,0]   Require Write or Reassign Blocks command
  15  120:32  00000000000147d8  [3,11,0]   Require Write or Reassign Blocks command
  16  120:32  00000000000147d0  [3,11,0]   Require Write or Reassign Blocks command
  17  120:32  00000000000147c8  [3,11,0]   Require Write or Reassign Blocks command
  18  120:32  00000000000147c0  [3,11,0]   Require Write or Reassign Blocks command
  19  120:32  00000000000147b8  [3,11,0]   Require Write or Reassign Blocks command
  20  120:31  00000000000147b0  [3,11,0]   Require Write or Reassign Blocks command
  21  120:31  00000000000147a8  [3,11,0]   Require Write or Reassign Blocks command
  22  120:31  00000000000147a0  [3,11,0]   Require Write or Reassign Blocks command
  23  120:31  0000000000014798  [3,11,0]   Require Write or Reassign Blocks command
  24  120:31  0000000000014790  [3,11,0]   Require Write or Reassign Blocks command
  25  120:31  0000000000014788  [3,11,0]   Require Write or Reassign Blocks command
  26  120:31  0000000000014780  [3,11,0]   Require Write or Reassign Blocks command
  27  120:31  0000000000014778  [3,11,0]   Require Write or Reassign Blocks command
  28  120:31  0000000000014770  [3,11,0]   Require Write or Reassign Blocks command
  29  120:31  0000000000014768  [3,11,0]   Require Write or Reassign Blocks command
  30  120:31  0000000000014760  [3,11,0]   Require Write or Reassign Blocks command
  31  120:31  0000000000014758  [3,11,0]   Require Write or Reassign Blocks command
  32  120:31  0000000000014750  [3,11,0]   Require Write or Reassign Blocks command
  33  120:30  0000000000014748  [3,11,0]   Require Write or Reassign Blocks command
  34  120:30  0000000000014740  [3,11,0]   Require Write or Reassign Blocks command
  35  120:30  0000000000014738  [3,11,0]   Require Write or Reassign Blocks command
  36  120:30  0000000000014730  [3,11,0]   Require Write or Reassign Blocks command
  37  120:30  0000000000014728  [3,11,0]   Require Write or Reassign Blocks command
  38  120:30  0000000000014720  [3,11,0]   Require Write or Reassign Blocks command
  39  120:30  0000000000014718  [3,11,0]   Require Write or Reassign Blocks command
  40  120:30  0000000000014710  [3,11,0]   Require Write or Reassign Blocks command
  41  120:30  0000000000014708  [3,11,0]   Require Write or Reassign Blocks command
  42  120:30  0000000000014700  [3,11,0]   Require Write or Reassign Blocks command
  43  120:30  00000000000146f8  [3,11,0]   Require Write or Reassign Blocks command
  44  120:30  00000000000146f0  [3,11,0]   Require Write or Reassign Blocks command
  45  120:30  00000000000146e8  [3,11,0]   Require Write or Reassign Blocks command
  46  120:30  00000000000146e0  [3,11,0]   Require Write or Reassign Blocks command
  47  120:29  00000000000146d8  [3,11,0]   Require Write or Reassign Blocks command
  48  120:29  00000000000146d0  [3,11,0]   Require Write or Reassign Blocks command
  49  120:29  00000000000146c8  [3,11,0]   Require Write or Reassign Blocks command
  50  120:29  00000000000146c0  [3,11,0]   Require Write or Reassign Blocks command
  51  120:29  00000000000146b8  [3,11,0]   Require Write or Reassign Blocks command
  52  120:29  00000000000146b0  [3,11,0]   Require Write or Reassign Blocks command
  53  120:29  00000000000146a8  [3,11,0]   Require Write or Reassign Blocks command
  54  120:29  00000000000146a0  [3,11,0]   Require Write or Reassign Blocks command
  55  120:29  0000000000014698  [3,11,0]   Require Write or Reassign Blocks command
  56  120:29  0000000000014690  [3,11,0]   Require Write or Reassign Blocks command
  57  120:29  0000000000014688  [3,11,0]   Require Write or Reassign Blocks command
  58  120:29  0000000000014680  [3,11,0]   Require Write or Reassign Blocks command
  59  120:29  0000000000014678  [3,11,0]   Require Write or Reassign Blocks command
  60  120:29  0000000000014670  [3,11,0]   Require Write or Reassign Blocks command
  61  120:28  0000000000014668  [3,11,0]   Require Write or Reassign Blocks command
  62  120:28  0000000000014660  [3,11,0]   Require Write or Reassign Blocks command
  63  120:28  0000000000014658  [3,11,0]   Require Write or Reassign Blocks command
  64  120:28  0000000000014650  [3,11,0]   Require Write or Reassign Blocks command
  65  120:28  0000000000014648  [3,11,0]   Require Write or Reassign Blocks command
  66  120:28  0000000000014640  [3,11,0]   Require Write or Reassign Blocks command
  67  120:28  0000000000014638  [3,11,0]   Require Write or Reassign Blocks command
  68  120:28  0000000000014630  [3,11,0]   Require Write or Reassign Blocks command
  69  120:28  0000000000014628  [3,11,0]   Require Write or Reassign Blocks command
  70  120:28  0000000000014620  [3,11,0]   Require Write or Reassign Blocks command
  71  120:28  0000000000014618  [3,11,0]   Require Write or Reassign Blocks command
  72  120:28  0000000000014610  [3,11,0]   Require Write or Reassign Blocks command
  73  120:28  0000000000014608  [3,11,0]   Require Write or Reassign Blocks command
  74  120:28  0000000000014600  [3,11,0]   Require Write or Reassign Blocks command
  75  120:27  00000000000145f8  [3,11,0]   Require Write or Reassign Blocks command
  76  120:27  00000000000145f0  [3,11,0]   Require Write or Reassign Blocks command
  77  120:27  00000000000145e8  [3,11,0]   Require Write or Reassign Blocks command
  78  120:27  00000000000145e0  [3,11,0]   Require Write or Reassign Blocks command
  79  120:27  00000000000145d8  [3,11,0]   Require Write or Reassign Blocks command
  80  120:27  00000000000145d0  [3,11,0]   Require Write or Reassign Blocks command
  81  120:27  00000000000145c8  [3,11,0]   Require Write or Reassign Blocks command
  82  120:27  00000000000145c0  [3,11,0]   Require Write or Reassign Blocks command
  83  120:27  00000000000145b8  [3,11,0]   Require Write or Reassign Blocks command
  84  120:27  00000000000145b0  [3,11,0]   Require Write or Reassign Blocks command
  85  120:27  00000000000145a8  [3,11,0]   Require Write or Reassign Blocks command
  86  120:27  00000000000145a0  [3,11,0]   Require Write or Reassign Blocks command
  87  120:27  0000000000014598  [3,11,0]   Require Write or Reassign Blocks command
  88  120:27  0000000000014590  [3,11,0]   Require Write or Reassign Blocks command
  89  120:26  0000000000014588  [3,11,0]   Require Write or Reassign Blocks command
  90  120:26  0000000000014580  [3,11,0]   Require Write or Reassign Blocks command
  91  120:26  0000000000014578  [3,11,0]   Require Write or Reassign Blocks command
  92  120:26  0000000000014570  [3,11,0]   Require Write or Reassign Blocks command
  93  120:26  0000000000014568  [3,11,0]   Require Write or Reassign Blocks command
  94  120:26  0000000000014560  [3,11,0]   Require Write or Reassign Blocks command
  95  120:26  0000000000014558  [3,11,0]   Require Write or Reassign Blocks command
  96  120:26  0000000000014550  [3,11,0]   Require Write or Reassign Blocks command
  97  120:26  0000000000014548  [3,11,0]   Require Write or Reassign Blocks command
  98  120:26  0000000000014540  [3,11,0]   Require Write or Reassign Blocks command
  99  120:26  0000000000014538  [3,11,0]   Require Write or Reassign Blocks command
 100  120:26  0000000000014530  [3,11,0]   Require Write or Reassign Blocks command
>>>> log truncated, fetched 16124 of 49172 available bytes
------This came up over 600 times before the system truncated it

Protocol Specific port log page for SAS SSP
relative target port id = 1
  generation code = 5
  number of phys = 1
  phy identifier = 0
    attached device type: SAS or SATA device
    attached reason: unknown
    reason: unknown
    negotiated logical link rate: phy enabled; 12 Gbps
    attached initiator port: ssp=1 stp=1 smp=1
    attached target port: ssp=0 stp=0 smp=0
    SAS address = 0x5000cca0bb527401
    attached SAS address = 0x500605b00a33b652
    attached phy identifier = 2
    Invalid DWORD count = 4
    Running disparity error count = 0
    Loss of DWORD synchronization = 1
    Phy reset problem = 0
    Phy event descriptors:
     Invalid word count: 4
     Running disparity error count: 0
     Loss of dword synchronization count: 1
     Phy reset problem count: 0
relative target port id = 2
  generation code = 5
  number of phys = 1
  phy identifier = 1
    attached device type: no device attached
    attached reason: unknown
    reason: power on
    negotiated logical link rate: phy enabled; unknown
    attached initiator port: ssp=0 stp=0 smp=0
    attached target port: ssp=0 stp=0 smp=0
    SAS address = 0x5000cca0bb527402
    attached SAS address = 0x0
    attached phy identifier = 0
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization = 0
    Phy reset problem = 0
    Phy event descriptors:
     Invalid word count: 0
     Running disparity error count: 0
     Loss of dword synchronization count: 0
     Phy reset problem count: 0

Admittedly, I don't know a whole lot about SMART, so I'm not sure if any of this is saying there are errors, or what I'm really looking at. Give me a network and I can go to town, but this is not my forte.

 

[Ericloewe]

Okay, at least da0 is failing SMART tests:

Code:

SMART Self-test log
Num Test            Status           segment       LifeTime   LBA_first_err [SK  ASC  ASQ] 
    Description                      number        (hours) 
# 1 Background long Failed in segment --> 6           43         4070793720 [0x3 0x5d 0x1]

That's one drive to immediately replace under warranty. I suspect the others will reveal similar issues.

The next step is to run a long test on all of them (smartctl -t long /dev/daX), let it finish, and then examine the results of the tests at the end. I haven't heard of any vendors with the gall to refuse an RMA on a disk that's failing SMART tests and is within warranty.

 

[12pharoh21]

Okay, at least da0 is failing SMART tests:

That's one drive to immediately replace under warranty. I suspect the others will reveal similar issues.

The next step is to run a long test on all of them (smartctl -t long /dev/daX), let it finish, and then examine the results of the tests at the end. I haven't heard of any vendors with the gall to refuse an RMA on a disk that's failing SMART tests and is within warranty.

That's super weird because these are brand new drives, but that being said, they were purchased a while ago. Let me double check and make sure they're still in warranty and get that process started. I will run the long SMART test on that drive, and do the same on the other 3 as well.

 

[Ericloewe]

That's super weird because these are brand new drives

It is, maybe they received special care and attention from the delivery service on their way to you and they liked it so much they want to go back to the factory ASAP.

 

[12pharoh21]

It is, maybe they received special care and attention from the delivery service on their way to you and they liked it so much they want to go back to the factory ASAP.

Well, I started the RMA process with the company I purchased them from and sent them back. They have performed their testing and found them all to be defective so they are giving me a refund since they don't have replacement drives to send me. I guess I will get some more ordered and probably just go with SATA this time around. Thanks for all your help.