SCSI errors from 2 different drives on SAS 2308, a week apart

[jms703]

I have a new FreeNAS system and have noticed the these error messages below. I can't seem to figure out what the issue could be since it's a new install and I haven't done much more than replicate my data from my old FreeNAS box. I don't see any other errors. All hardware is brand new. Drives report Power_On_Hours to be 1453. Any thoughts or suggestions?

Things I notice:

1. I'm on version 16 of the firmware and driver and version 18 is out. I wonder if I'm hitting a bug that has been fixed.
2. The time these errors occur is on Sunday, after 20:00 PDT. There is a default scrub job that runs on Sunday at midnight, but that is all that I can tell happens on Sunday. I don't have any systems pushing anything to FreeNAS at that time.
3. I have high Load_Cycle_Count (approx 43k on all 6 drives), possibly due the 8 second park issue. I read about this in a thread about about the WD Red drives. I will fix that later if not related.
4. I have 6x WD RED drives and there is concern about vibration in a case with more than 4 or 5 of these drives. Each drive is in a removable tray with rubber/silicon dampeners. Is that sufficient?

Error Messages:

Code:

Apr  6 20:02:16 freenas kernel: (da2:mps0:0:2:0): WRITE(10). CDB: 2a 00 18 fc 8b d8 00 00 08 00
Apr  6 20:02:16 freenas kernel: (da2:mps0:0:2:0): CAM status: SCSI Status Error
Apr  6 20:02:16 freenas kernel: (da2:mps0:0:2:0): SCSI status: Check Condition
Apr  6 20:02:16 freenas kernel: (da2:mps0:0:2:0): SCSI sense: ILLEGAL REQUEST asc:21,0 (Logical block address out of range)
Apr  6 20:02:16 freenas kernel: (da2:mps0:0:2:0): Info: 0x18fc8bd8
Apr  6 20:02:16 freenas kernel: (da2:mps0:0:2:0): Error 22, Unretryable error

Code:

Apr 13 20:49:40 freenas kernel: (da0:mps0:0:0:0): WRITE(10). CDB: 2a 00 1e 66 df 90 00 00 08 00
Apr 13 20:49:40 freenas kernel: (da0:mps0:0:0:0): CAM status: SCSI Status Error
Apr 13 20:49:40 freenas kernel: (da0:mps0:0:0:0): SCSI status: Check Condition
Apr 13 20:49:40 freenas kernel: (da0:mps0:0:0:0): SCSI sense: ILLEGAL REQUEST asc:21,0 (Logical block address out of range)
Apr 13 20:49:40 freenas kernel: (da0:mps0:0:0:0): Info: 0x1e66df90
Apr 13 20:49:40 freenas kernel: (da0:mps0:0:0:0): Error 22, Unretryable error

Code:

ZFS Pool output:
 
  pool: vol1
state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
  see: http://illumos.org/msg/ZFS-8000-9P
  scan: resilvered 16K in 0h0m with 0 errors on Sun Apr 13 20:49:47 2014
config:

Hardware
MB: Supermicro X10SL7 w/LSI 2308 onboard controller
CPU: Intel Xeon E3-1230 v3
RAM: Kingston 32 GB ECC
DISKS: 6x WDC WD40EFRX-68WT0N0 in a single RAIDZ2 volume
PSU: Sea Sonic S12G 550W ATX
CASE: ARC Midi R2 - Fractal Design

Versions
OS: FreeNAS-9.2.1.3-RELEASE-x64 (dc0c46b)
LSI 2308 Firmware: 16.00.01.00 (IT version)
LSI 2308 Driver: 16.00.00.00-fbsd
Disk firmware: 80.00A80

 

[dlavigne]

1. I'm on version 16 of the firmware and driver and version 18 is out. I wonder if I'm hitting a bug that has been fixed.

Version 16 is the current driver being used in FreeNAS. You could make a feature request at bugs.freenas.org which includes the URL to the version 18 driver. If you do, post the issue number here.

see: http://illumos.org/msg/ZFS-8000-9

Does following the "action" at that URL resolve the issue?

 

[cyberjock]

Version 16 is the current driver being used in FreeNAS. You could make a feature request at bugs.freenas.org which includes the URL to the version 18 driver. If you do, post the issue number here.

Yeah, except I already did that.. and they didn't upgrade to the latest...

https://bugs.freenas.org/issues/2400

Phase17 drivers came out(which I mentioned in the ticket) yet they still only upgraded to P16. Not sure why but they didn't even upgrade to the latest at the time. /smh

 

[dlavigne]

Yeah, looks like they missed comment #6 and just went with the title of the bug. Now that 17 has been surpassed by 18, can one of you create a new feature request and post the issue number here?

 

[cyberjock]

That makes it even worse if they couldn't be inconvenienced to read the comments.

 

[cyberjock]

https://bugs.freenas.org/issues/4825

 

[jms703]

Does following the "action" at that URL resolve the issue?

The action cleared the error, but today I noticed this:

Apr 22 04:22:18 freenas kernel: (da1:mps0:0:1:0): WRITE(10). CDB: 2a 00 1e 76 0f 68 00 00 08 00
Apr 22 04:22:18 freenas kernel: (da1:mps0:0:1:0): CAM status: SCSI Status Error
Apr 22 04:22:18 freenas kernel: (da1:mps0:0:1:0): SCSI status: Check Condition
Apr 22 04:22:18 freenas kernel: (da1:mps0:0:1:0): SCSI sense: ILLEGAL REQUEST asc:21,0 (Logical block address out of range)
Apr 22 04:22:18 freenas kernel: (da1:mps0:0:1:0): Info: 0x1e760f68
Apr 22 04:22:18 freenas kernel: (da1:mps0:0:1:0): Error 22, Unretryable error

A third, brand new disk with this same error.

 

[jms703]

Is there a better place to ask questions like this?

 

[cyberjock]

Not really. I just don't have an answer for you.

Can you do smartctl -a -q noserial /dev/XXXXX for each of your disks and post it to the forums?

 

[jms703]

Can you do smartctl -a -q noserial /dev/XXXXX for each of your disks and post it to the forums?

Text file uploaded.

 

[cyberjock]

Everything looks fine in regards to your problem. Assuming you reflashed it to IT mode I have no recommendations. v16 is what you want though because that's what FreeNAS uses. firmware and drivers need to stay in lock step. If you were to find out that the bug is fixed on v17 or v18, you'd need to update the driver in FreeNAS before going after the firmware update. Failure to do so can be costly(aka data loss).

You do need to take care of the high load cycle though. 1700 hours and 50k cycles is a bad thing. That problem isn't related to your current problem though(at least, it *shouldn't* be).

Other than that, I have no recommendations. Maybe your PSU sucks/is failing. I really don't have any good ideas.

 

[jms703]

Assuming you reflashed it to IT mode I have no recommendations.

Firmware is flashed to IT mode.

You do need to take care of the high load cycle though. 1700 hours and 50k cycles is a bad thing.

This morning, I disabled the idle park timer using idle3-tools. (I tried to set it to 300s, but this tool doesn't seem to accept values greater than 250. I decided to just disable idle altogether.)

Other than that, I have no recommendations. Maybe your PSU sucks/is failing. I really don't have any good ideas.

The power supply is brand new, so I hope it's not failing already. I will start monitoring the voltage settings to see if anything exceeds or drops below the thresholds. So far the 3.3v, 5v, and 12v readings look fine.

 

[cyberjock]

A few people have said that the idle3-tools doesn't last through power cycles... so use it at your own risk.

 

[ser_rhaegar]

You have to power cycle the drives for the new timers to apply, so it would have to last through a power cycle. I used idle3-tools for mine as well without issue.

Regarding the 300s, the value is not 300. See the table at the bottom of this page: http://idle3-tools.sourceforge.net/ If I remember correctly from when I did mine, the idle3-tools uses the same values as wdidle3 v1.05. The value you would need to set it to for 300s is 138.

 

[jms703]

I just wanted to follow up on this. I have not seen this error again.

The only change(s) I made to the system was disabling the idle park timer using idle3-tools.

 

[dewie]

Hi jms, i have the exact same setup as you only 6* WD RE 4 disks.
But i also get

Code:

Jan 29 18:11:33 freenas (da3:mps0:0:3:0): WRITE(10). CDB: 2a 00 10 47 56 68 00 00 10 00
Jan 29 18:11:33 freenas (da3:mps0:0:3:0): CAM status: SCSI Status Error
Jan 29 18:11:33 freenas (da3:mps0:0:3:0): SCSI status: Check Condition
Jan 29 18:11:33 freenas (da3:mps0:0:3:0): SCSI sense: ILLEGAL REQUEST asc:21,0 (Logical block address out of range)
Jan 29 18:11:33 freenas (da3:mps0:0:3:0): Info: 0x10475668
Jan 29 18:11:33 freenas (da3:mps0:0:3:0): Error 22, Unretryable error

did you use the idel3-tools on freebsd ?

I'm having problems compiling these

 

[jms703]

did you use the idel3-tools on freebsd ?
I'm having problems compiling these

I can't remember how I ran these tools. I didn't compile them though. I may have used a bootable archlinux cd. Either way, this error went away after I made the change, which is strange, given that these two things should not be related.

 

[dewie]

Checked with idle3-tools, all disabled.
Will try to replace the sata cabling :(

 

[ggoldfingerd]

Checked with idle3-tools, all disabled.
Will try to replace the sata cabling :(

What ended up being the problem?